[PR #6965] [management, proxy] Management-owned LLM pricing: file-backed defaults + #30053

Closed
opened 2026-08-05 08:09:54 -04:00 by saavagebueno · 0 comments
Owner

Original Pull Request: https://github.com/netbirdio/netbird/pull/6965

State: closed
Merged: Yes


operator prices, delivered via cost_meter middleware config

Describe your changes

Operator-pinned model prices (ProviderModel.InputPer1k/OutputPer1k) were
stored and round-tripped through the API but never consumed for billing:
the proxy priced exclusively from a defaults_pricing.yaml embedded in
its binary. Any model outside that file recorded $0 spend, price changes
required a proxy release, and the catalog and the YAML were two
hand-synced copies of the same rates, guarded only by a manually
maintained coverage list that had already drifted.

Management now owns pricing end to end and ships the full table inside
cost_meter's ConfigJSON on the existing mapping stream:

{"pricing": {"defaults":  {surface: {model: rates}},
             "providers": {providerRecordID: {model: rates}}}}

Pricing resolves through three layers, highest precedence first:

1. per-provider operator prices (dashboard/API, per account)
2. defaults_llm_pricing.yaml on management disk (hot-reloaded)
3. compiled-in defaults (plain Go: catalog rate literals + a small
   supplemental list; no go:embed remains anywhere)

The defaults file (AgentNetwork.PricingDefaultsFile in management.json,
or /defaults_llm_pricing.yaml by convention; same schema as the
retired proxy file) is loaded at boot and re-read every minute, merged
entry-whole over the compiled-in fallback. Absent file = layer 3 serves
and the path stays watched, so dropping the file in later loads without
a restart. Strict at boot for an explicitly configured path (typo fails
startup); lenient at runtime (a malformed save keeps the previous table,
deletion reverts to built-ins). The live table feeds the synthesizer
(what proxies bill with) and the dashboard's catalog endpoint (what
model rows prefill with), so the two can never disagree. A generated
defaults_llm_pricing.example.yaml ships as the operator starting point,
kept byte-in-sync with the compiled-in table by a golden test +
go generate.

  • The catalog remains the structural source: catalog.Model gains the
    three cache rates (cached_input for the OpenAI subset shape,
    cache_read/cache_creation for the additive Anthropic shape),
    catalog.Provider gains PricingSurfaces mapping its models onto
    cost-meter surfaces (not derivable from ParserID: bedrock/vertex/kimi
    leave it empty). The agentnetwork/pricing package folds it into the
    default table plus supplementals for priced-but-not-selectable ids
    (gpt-5 family, claude-opus-5, kimi-k3[1m]).
  • The synthesizer materializes per-provider entries from the operator's
    stored model rows: ids normalized exactly as the proxy's request
    parser normalizes them (new shared/llm package, extracted from
    proxy/internal/llm so both sides compare equal), default cache rates
    folded in from the live table, operator input/output verbatim
    (explicit 0 = free), nil cache pointer = inherit vs explicit 0 = no
    discount. Orphan providers (no authorising policy) ship no prices;
    empty-Models gateways rely on the defaults.
  • The API gains optional cached_input_per_1k / cache_read_per_1k /
    cache_creation_per_1k on AgentNetworkProviderModel (stored as
    pointers so nil stays distinct from 0) and, read-only, on
    AgentNetworkCatalogModel; AgentNetworkCatalogProvider exposes
    pricing_surfaces so the dashboard renders shape-aware price fields.
    The providers handler validates all rates finite and non-negative —
    the single ingress guard for operator pricing.
  • The proxy becomes a dumb consumer: embedded YAML, DefaultTable,
    mergeOver, and the (unreachable) file-override Loader with its mtime
    poll are deleted, along with the dead FactoryContext.DataDir /
    Server.MiddlewareDataDir plumbing whose only consumer they were.
    cost_meter resolves per-record (llm.resolved_provider_id, already in
    the metadata bag) then surface defaults, sharing one EntryCosts
    formula; an empty config (old management) builds an all-skip instance
    with a build-time warning instead of failing.

Price updates propagate with zero per-request cost: provider saves ride
the existing reconcile -> synth -> push -> chain rebuild (milliseconds);
defaults-file edits enter the table within a minute and reach proxies on
the next reconcile or reconnect. Requests do two in-memory map lookups,
exactly as before.

Tests: the drift-prone coverage list is replaced by programmatic
catalog<->table equality plus a collision test (providers sharing a
surface+model must agree on rates); cost_calculation_matrix_test feeds
the real management DefaultTable() through the real parser pipeline,
making the exact-USD vendor-price assertions a cross-stack contract
test; the realstack test pins an operator price different from the
catalog rate and asserts the request bills exactly from it end to end
(store -> synth -> wire -> per-record lookup -> RecordLLMUsage); the
reload lifecycle (edit, broken save, removal, re-creation) is covered
under -race.

Deployment: upgrade management before proxies. A new proxy under an old
management receives "{}" and records $0 for every request (USD budget
caps inert) until management ships the table.

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)
  • This change does not modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — OR I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See CONTRIBUTING.md.

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

  • I added/updated documentation for this change
  • Documentation is not needed for this change (explain why)

Docs PR URL (required if "docs added" is checked)

https://github.com/netbirdio/docs/pull/896


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Summary by CodeRabbit

New Features

  • Added configurable agent-network pricing defaults with automatic reload support.
  • Added per-model cache pricing and provider pricing-surface metadata to catalog and provider APIs.
  • Propagated default and operator-defined pricing into proxy cost calculations, including custom model pricing.

Bug Fixes

  • Added validation for finite, non-negative pricing values and explicit zero-rate handling.
  • Improved Bedrock and Vertex model matching.
  • Preserved the last valid pricing configuration when reloads fail.

Documentation

  • Updated configuration guidance, example pricing YAML, and API schemas.
**Original Pull Request:** https://github.com/netbirdio/netbird/pull/6965 **State:** closed **Merged:** Yes --- operator prices, delivered via cost_meter middleware config ## Describe your changes Operator-pinned model prices (ProviderModel.InputPer1k/OutputPer1k) were stored and round-tripped through the API but never consumed for billing: the proxy priced exclusively from a defaults_pricing.yaml embedded in its binary. Any model outside that file recorded $0 spend, price changes required a proxy release, and the catalog and the YAML were two hand-synced copies of the same rates, guarded only by a manually maintained coverage list that had already drifted. Management now owns pricing end to end and ships the full table inside cost_meter's ConfigJSON on the existing mapping stream: {"pricing": {"defaults": {surface: {model: rates}}, "providers": {providerRecordID: {model: rates}}}} Pricing resolves through three layers, highest precedence first: 1. per-provider operator prices (dashboard/API, per account) 2. defaults_llm_pricing.yaml on management disk (hot-reloaded) 3. compiled-in defaults (plain Go: catalog rate literals + a small supplemental list; no go:embed remains anywhere) The defaults file (AgentNetwork.PricingDefaultsFile in management.json, or <datadir>/defaults_llm_pricing.yaml by convention; same schema as the retired proxy file) is loaded at boot and re-read every minute, merged entry-whole over the compiled-in fallback. Absent file = layer 3 serves and the path stays watched, so dropping the file in later loads without a restart. Strict at boot for an explicitly configured path (typo fails startup); lenient at runtime (a malformed save keeps the previous table, deletion reverts to built-ins). The live table feeds the synthesizer (what proxies bill with) and the dashboard's catalog endpoint (what model rows prefill with), so the two can never disagree. A generated defaults_llm_pricing.example.yaml ships as the operator starting point, kept byte-in-sync with the compiled-in table by a golden test + go generate. - The catalog remains the structural source: catalog.Model gains the three cache rates (cached_input for the OpenAI subset shape, cache_read/cache_creation for the additive Anthropic shape), catalog.Provider gains PricingSurfaces mapping its models onto cost-meter surfaces (not derivable from ParserID: bedrock/vertex/kimi leave it empty). The agentnetwork/pricing package folds it into the default table plus supplementals for priced-but-not-selectable ids (gpt-5 family, claude-opus-5, kimi-k3[1m]). - The synthesizer materializes per-provider entries from the operator's stored model rows: ids normalized exactly as the proxy's request parser normalizes them (new shared/llm package, extracted from proxy/internal/llm so both sides compare equal), default cache rates folded in from the live table, operator input/output verbatim (explicit 0 = free), nil cache pointer = inherit vs explicit 0 = no discount. Orphan providers (no authorising policy) ship no prices; empty-Models gateways rely on the defaults. - The API gains optional cached_input_per_1k / cache_read_per_1k / cache_creation_per_1k on AgentNetworkProviderModel (stored as pointers so nil stays distinct from 0) and, read-only, on AgentNetworkCatalogModel; AgentNetworkCatalogProvider exposes pricing_surfaces so the dashboard renders shape-aware price fields. The providers handler validates all rates finite and non-negative — the single ingress guard for operator pricing. - The proxy becomes a dumb consumer: embedded YAML, DefaultTable, mergeOver, and the (unreachable) file-override Loader with its mtime poll are deleted, along with the dead FactoryContext.DataDir / Server.MiddlewareDataDir plumbing whose only consumer they were. cost_meter resolves per-record (llm.resolved_provider_id, already in the metadata bag) then surface defaults, sharing one EntryCosts formula; an empty config (old management) builds an all-skip instance with a build-time warning instead of failing. Price updates propagate with zero per-request cost: provider saves ride the existing reconcile -> synth -> push -> chain rebuild (milliseconds); defaults-file edits enter the table within a minute and reach proxies on the next reconcile or reconnect. Requests do two in-memory map lookups, exactly as before. Tests: the drift-prone coverage list is replaced by programmatic catalog<->table equality plus a collision test (providers sharing a surface+model must agree on rates); cost_calculation_matrix_test feeds the real management DefaultTable() through the real parser pipeline, making the exact-USD vendor-price assertions a cross-stack contract test; the realstack test pins an operator price different from the catalog rate and asserts the request bills exactly from it end to end (store -> synth -> wire -> per-record lookup -> RecordLLMUsage); the reload lifecycle (edit, broken save, removal, re-creation) is covered under -race. Deployment: upgrade management before proxies. A new proxy under an old management receives "{}" and records $0 for every request (USD budget caps inert) until management ships the table. ## Issue ticket number and link ## Stack <!-- branch-stack --> ### Checklist - [x] Is it a bug fix - [ ] Is a typo/documentation fix - [ ] Is a feature enhancement - [x] It is a refactor - [ ] Created tests that fail without the change (if possible) - [ ] This change does **not** modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — **OR** I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See [CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#discuss-changes-with-the-netbird-team-first). > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [x] I added/updated documentation for this change - [ ] Documentation is **not needed** for this change (explain why) ### Docs PR URL (required if "docs added" is checked) https://github.com/netbirdio/docs/pull/896 <!-- codesmith:footer --> --- <a href="https://app.blacksmith.sh/netbirdio/codesmith/netbird/pr/6965"><picture><source media="(prefers-color-scheme: dark)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v2.svg"><source media="(prefers-color-scheme: light)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-light-v2.svg"><img alt="View with [code]smith" src="https://pr-comments-assets.blacksmith.sh/codesmith/view-with-codesmith-dark-v2.svg"></picture></a> <a href="https://backend.blacksmith.sh/track/enable-autofix?expires=1787928705&installation_model_id=427504&pr_number=6965&repository=netbirdio%2Fnetbird&return_to=https%3A%2F%2Fgithub.com%2Fnetbirdio%2Fnetbird%2Fpull%2F6965&signature=eb3e31ba9e1d1753f6dff842ac8aa26bd252a81ff04a9bebbf39ae1ec2987565"><picture><source media="(prefers-color-scheme: dark)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-light.svg"><img alt="Autofix with [code]smith" src="https://pr-comments-assets.blacksmith.sh/codesmith/autofix-with-codesmith-dark.svg"></picture></a> <sup>Need help on this PR? Tag <code>@codesmith-bot</code> with what you need. Autofix is disabled.</sup> <!-- codesmith:autofix:disabled --> <!-- /codesmith:footer --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## New Features * Added configurable agent-network pricing defaults with automatic reload support. * Added per-model cache pricing and provider pricing-surface metadata to catalog and provider APIs. * Propagated default and operator-defined pricing into proxy cost calculations, including custom model pricing. ## Bug Fixes * Added validation for finite, non-negative pricing values and explicit zero-rate handling. * Improved Bedrock and Vertex model matching. * Preserved the last valid pricing configuration when reloads fail. ## Documentation * Updated configuration guidance, example pricing YAML, and API schemas. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
saavagebueno added the pull-request label 2026-08-05 08:09:54 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#30053