mirror of
https://github.com/netbirdio/netbird.git
synced 2026-08-03 03:08:43 -04:00
[docs] Update agent-network docs for management-owned pricing
The docs still described the retired proxy-side pricing: pricing.Loader, pricing_path, MiddlewareDataDir, embedded defaults_pricing.yaml, and the symlink-safe Unix loader. Rewrite them for the current design — management synthesizes the whole table and ships it in cost_meter's ConfigJSON, so the proxy carries no price list and has nothing to reload.
This commit is contained in:
braginini
@@ -115,7 +115,7 @@ sequenceDiagram
|
||||
Resp->>Resp: parse usage tokens, completion
|
||||
Note over Resp: capture_completion gates raw<br/>completion capture
|
||||
Resp->>Cost: tokens
|
||||
Cost->>Cost: lookup pricing.yaml + compute cost
|
||||
Cost->>Cost: lookup rates from config-delivered<br/>pricing table + compute cost
|
||||
Cost->>Rec: tokens + cost
|
||||
Rec->>MgmtGrpc: RecordLLMUsage(provider, model, prompt_t, completion_t, cost, groups, user)
|
||||
Rec-->>Log: emit access-log entry<br/>(if EnableLogCollection)
|
||||
|
||||
@@ -15,6 +15,10 @@ Inside the package: `manager.go` is the CRUD + permissions-gated facade; `synthe
|
||||
| ---- | ---- |
|
||||
| `agentnetwork/manager.go` | Manager interface + CRUD + permission gates + bootstrap-settings + reconcile trigger |
|
||||
| `agentnetwork/synthesizer.go` | Settings/policy → wire-format synthesis; sole writer of the proxy middleware chain |
|
||||
| `agentnetwork/synthesizer_pricing.go` | `buildCostMeterConfigJSON` — default table + per-provider prices → `cost_meter` config |
|
||||
| `agentnetwork/pricing/defaults.go` | Default pricing table derived from the catalog + supplementals; `DefaultTable`, `LookupDefault`, wire `Entry` |
|
||||
| `agentnetwork/pricing/override.go` | `LoadFile`/`StartReloader` for `AgentNetwork.PricingDefaultsFile` (mtime poll, merge over compiled-in base) |
|
||||
| `agentnetwork/pricing/{exampleyaml,gen}.go` | Generates `defaults_llm_pricing.example.yaml` from the compiled-in table (golden-tested) |
|
||||
| `agentnetwork/policyselect.go` | Per-request policy attribution + account-budget ceiling (min-wins) |
|
||||
| `agentnetwork/reconcile.go` | Per-account synth diff vs in-memory cache → Create/Update/Delete |
|
||||
| `agentnetwork/catalog/catalog.go` | Static provider catalogue (auth headers, identity-injection shapes) |
|
||||
@@ -48,6 +52,8 @@ flowchart TD
|
||||
I --> J[indexProviderGroups: providerID -> sorted source groups]
|
||||
J --> K[buildRouterConfigJSON drops orphan providers]
|
||||
J --> L[buildIdentityInjectConfigJSON per catalog entry]
|
||||
J --> K2[buildCostMeterConfigJSON: default table + per-provider prices]
|
||||
K2 --> P
|
||||
H --> M[mergeGuardrails: union allowlist, OR redact]
|
||||
M --> N[applyAccountCollectionControls account toggle = SOLE capture control]
|
||||
N --> O[marshalGuardrailConfig]
|
||||
@@ -60,6 +66,84 @@ flowchart TD
|
||||
R --> T[accountManager.UpdateAccountPeers — fans synth ACLs into network map]
|
||||
```
|
||||
|
||||
### LLM pricing (management is the sole authority)
|
||||
|
||||
**The proxy carries no price list.** Management synthesizes the entire pricing
|
||||
table and ships it inside `cost_meter`'s `ConfigJSON`, so a price change reaches
|
||||
the proxies as an ordinary mapping push — the chain rebuild installs a fresh
|
||||
table and there is nothing to reload on the proxy side.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[catalog.All — PricingSurfaces x Models] --> B[buildDefaultTable + supplementalDefaults]
|
||||
B --> C{AgentNetwork.PricingDefaultsFile}
|
||||
C -- absent --> D[compiled-in table serves]
|
||||
C -- loaded --> E[LoadFile: merge file entries WHOLE over compiled base]
|
||||
E --> F[mergedTable atomic.Pointer]
|
||||
D --> G[DefaultTable]
|
||||
F --> G
|
||||
G --> H[buildCostMeterConfigJSON — pricing.defaults]
|
||||
I[types.Provider.Models operator prices] --> J[normalizePricingModelID<br/>bedrock ARN/region/version, vertex @version]
|
||||
J --> K[materializeEntry: default entry as base,<br/>operator input/output verbatim,<br/>cache pointers only when non-nil]
|
||||
K --> L[pricing.providers keyed by provider record ID]
|
||||
H --> M[cost_meter ConfigJSON]
|
||||
L --> M
|
||||
G --> N[GET /catalog — applyDefaultPricing prefills dashboard rows]
|
||||
O[StartReloader: mtime poll every ReloadInterval 1m] --> E
|
||||
```
|
||||
|
||||
**Two tiers, resolved per request on the proxy** (`synthesizer_pricing.go:22-35`):
|
||||
|
||||
- `pricing.defaults` — surface (`openai`/`anthropic`/`bedrock`) → normalized model
|
||||
id → rates. The **full** default table ships to every account: it is small
|
||||
(~10 KB) and it is what keeps gateway-style providers (which enumerate no
|
||||
models, so they claim every model) priced.
|
||||
- `pricing.providers` — provider **record** id → normalized model id → rates,
|
||||
matched against the `llm.resolved_provider_id` the router stamps. Entries are
|
||||
**fully materialized here**, at synth time: `materializeEntry` starts from the
|
||||
default entry for that model so cache rates the operator didn't state are
|
||||
inherited, overlays operator `input`/`output` verbatim (**including an explicit
|
||||
0**, which prices a self-hosted or internal endpoint as free rather than
|
||||
silently reverting to list price), and overlays cache-rate **pointers only when
|
||||
non-nil** — `nil` means "inherit the default", an explicit `0` means "no
|
||||
discount, bill this bucket at the input rate". The proxy therefore does two map
|
||||
lookups and no merging.
|
||||
|
||||
Same orphan rule as the router: a provider no enabled policy authorises is
|
||||
unreachable, so its prices aren't shipped. Model ids are normalized with the
|
||||
**same** functions the request parser uses (`NormalizeBedrockModel` /
|
||||
`NormalizeVertexModel`), which is what makes the per-record lookup key compare
|
||||
equal to the `llm.model` the proxy meters. Post-normalization duplicates resolve
|
||||
first-occurrence-wins, matching the routing dedup order.
|
||||
|
||||
**`AgentNetwork.PricingDefaultsFile`** (`config.go:190-207`) lets an operator
|
||||
replace default rates without a rebuild. Schema is `surface → model → rates`
|
||||
(`input_per_1k`, `output_per_1k`, and optional `cached_input_per_1k` /
|
||||
`cache_read_per_1k` / `cache_creation_per_1k`). Semantics:
|
||||
|
||||
- A **relative** path resolves against `<Datadir>`, so a bare filename lands
|
||||
alongside the store. Empty config probes `<Datadir>/defaults_llm_pricing.yaml`.
|
||||
- An **explicitly configured** path is *required to load*: a typo or malformed
|
||||
file fails startup, because the operator believes those rates are live. The
|
||||
conventional probe is optional — an absent file just serves compiled-in
|
||||
defaults, and the path stays watched in case it appears later.
|
||||
- File entries **replace** the compiled-in entry for the same (surface, model)
|
||||
**whole** — they are not field-merged, so an entry must repeat the cache rates
|
||||
it wants to keep. Everything the file doesn't mention keeps built-in rates.
|
||||
- Unknown YAML fields are rejected (`KnownFields(true)`) and every rate must be
|
||||
finite and non-negative — the same constraints the HTTP API enforces on
|
||||
operator per-provider prices.
|
||||
- Reload is an mtime poll (`ReloadInterval`, 1 min) and is **lenient at runtime**:
|
||||
a parse error keeps the previous table, a deleted file reverts to compiled-in
|
||||
defaults. A mid-edit save can never take pricing down.
|
||||
|
||||
The live table feeds **both** consumers, which is what keeps them consistent: the
|
||||
synthesizer (what proxies actually bill with) and `GET /api/agent-network/catalog`
|
||||
via `applyDefaultPricing` (what the dashboard's model-row prices prefill with).
|
||||
`defaults_llm_pricing.example.yaml` is generated from the compiled-in table
|
||||
(`go generate ./management/internals/modules/agentnetwork/pricing`) and
|
||||
golden-tested, so operators start from a file matching the built-in rates exactly.
|
||||
|
||||
### Budget rule resolution (min-wins, group+user bound)
|
||||
|
||||
```mermaid
|
||||
@@ -124,7 +208,7 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
|
||||
| on_request | 3 | `llm_identity_inject` | `{"providers":[{provider_id, header_pair?, json_metadata?, extra_headers?}]}` | **true** |
|
||||
| on_request | 4 | `llm_guardrail` | `{"provider_allowlists"?: {providerID: []model}, "prompt_capture":{enabled,redact_pii}}` | – |
|
||||
| on_response | 5 | `llm_limit_record` | `{}` (runs LAST at runtime) | – |
|
||||
| on_response | 6 | `cost_meter` | `{}` | – |
|
||||
| on_response | 6 | `cost_meter` | `{"pricing":{"defaults":{surface:{model:rates}},"providers"?:{providerRecordID:{model:rates}}}}` — rates are `{input_per_1k, output_per_1k, cached_input_per_1k?, cache_read_per_1k?, cache_creation_per_1k?}` | – |
|
||||
| on_response | 7 | `llm_response_parser` | `{"capture_completion": <bool>, "redact_pii"?: true}` | – |
|
||||
- **Synthesized service shape** (`synthesizer.go:739`): `Mode=HTTP`, `Private=true`, `Domain=<subdomain>.<cluster>`, `AccessGroups=unionSourceGroups(enabledPolicies)`, one `TargetTypeCluster` target with `Host=noop.invalid:443` (router rewrites per request), `Options.{DirectUpstream,AgentNetwork}=true`, `DisableAccessLog=!settings.EnableLogCollection`, `CaptureMax{Req,Resp}Bytes=1<<20`, `CaptureContentTypes=["application/json","text/event-stream"]`.
|
||||
|
||||
@@ -139,6 +223,12 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
|
||||
- **Orphan providers (no enabled policy authorises them) NEVER reach the router** (`synthesizer.go:351-357`); skipped from `identity_inject` for symmetry.
|
||||
- **Provider creation refuses empty `api_key`** (`manager.go:175`); **deletion refuses while any policy still references it** (`manager.go:265-273`).
|
||||
- **Session keypair stability across provider edits** (`manager.go:226-228`) — server-managed, copied through every `UpdateProvider`, never API-surfaced.
|
||||
- **Management is the sole pricing authority.** The proxy has no embedded price list, so an account whose `cost_meter` config carries no `pricing` block bills **nothing** (`cost.skipped=unknown_model`, $0) rather than falling back to stale built-ins. The top-level `pricing` wrapper is also the feature-detection signal in both directions: an old proxy ignores it as an unknown field, and a new proxy reads its absence as "old management".
|
||||
- **Per-provider prices are materialized at synth time, not merged on the proxy** (`synthesizer_pricing.go:114-131`). A per-record entry is always complete, so the proxy's lookup is per-record-then-defaults with no field-level fallback between tiers.
|
||||
- **An explicit operator price of `0` prices the model as free** — it must not be treated as "unset" and reverted to list price (`synthesizer_pricing.go:49-54`). Only *cache*-rate fields distinguish unset from zero, via `*float64`.
|
||||
- **Pricing model ids are normalized with the same functions the request parser uses** (`normalizePricingModelID`). If the two ever diverge, per-record prices silently stop matching and every request falls through to surface defaults.
|
||||
- **The default table's coverage is structural, not curated.** It is derived from the catalog via each provider's `PricingSurfaces`; `TestDefaultTable_CoversEveryCatalogModel` fails on an unpriced catalog model and `TestDefaultTable_NoConflictingContributions` fails if two providers contribute the same (surface, model) at different rates.
|
||||
- **A pricing-defaults file failure is fatal only at startup, and only for an explicitly configured path.** Runtime reload failures keep the previous table; a deleted file reverts to compiled-in defaults (`pricing/override.go:62-81, 113-148`).
|
||||
|
||||
## Things to scrutinize
|
||||
|
||||
@@ -176,10 +266,12 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
|
||||
- **Capture-pointer semantics (restated):** non-agent-network callers see no field → legacy nil-default emit, identical to pre-PR. Agent-network targets always carry an explicit `capture_*` value.
|
||||
- **`TestSynthesizeServices_HappyPath` was updated:** request-parser config moved from `{}` to `{"capture_prompt":false}` (`synthesizer_test.go:174`). External snapshot tests against synth output need updating.
|
||||
- **`MergedGuardrails` retains zeroed `TokenLimits`/`Budget`/`Retention`** even though `Policy.Limits` carries the real values now; `llm_limit_check` is the authoritative enforcement. Comment at `synthesizer.go:940-948` calls this out.
|
||||
- **`cost_meter`'s `pricing` block is version-skew-safe in both directions.** A proxy predating config-delivered pricing ignores the field as unknown JSON (it previously priced from its own embedded table, so it keeps billing — at its own rates, which is the skew to be aware of during a rolling upgrade). A current proxy paired with old management sees no `pricing` block, logs one warning at chain-build time, and records `cost.skipped=unknown_model` — token counting and cap enforcement are unaffected, only the USD annotation goes to $0.
|
||||
|
||||
### Performance
|
||||
|
||||
- **`SynthesizeServices` runs on every controller tick / mutation reconcile.** Cost: 4 store reads + optional per-provider keypair backfill. Sort + index + merge are O(N log N) / O(P × G); dominant cost is JSON marshalling. No nested loops escape these dimensions.
|
||||
- **The full default pricing table is marshalled into every account's `cost_meter` config on every synth** (~10 KB serialized). This is a deliberate trade: it keeps gateway-style providers priced for every catalog model, and it is the largest single contributor to the synth JSON. `DefaultTable()` itself is a pointer load (or a `sync.Once`-built map) — the cost is the marshal, not the build.
|
||||
- **`reconcile.diffMappings` is O(N + M)** with N=M=1 per account today — effectively constant.
|
||||
- **`SynthesizeServicesForCluster`** (`synthesizer.go:71`) walks every account on a cluster; per-account failures are **swallowed** (`synthesizer.go:91-93`) so a single misconfigured account doesn't drop the cluster. Runs per proxy reconnect.
|
||||
|
||||
@@ -188,6 +280,7 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
|
||||
- **Activity codes:** `AgentNetwork{Provider,Policy,Guardrail,BudgetRule}{Created,Updated,Deleted}`; `AgentNetworkSettingsUpdated` with `log_collection/prompt_collection/redact_pii` payload (`manager.go:567-571`). **No activity code for `SelectPolicyForRequest` denies** — surfaced via proxy access log only (likely intentional given volume).
|
||||
- **Deny codes** namespaced: `llm_policy.{token,budget}_cap_exceeded`, `llm_account.{token,budget}_cap_exceeded` (`policyselect.go:18-26`).
|
||||
- **Reconcile failures are logged at warn and swallowed** (`reconcile.go:42-44`). Persistent synth failures (e.g. unknown catalog id) silently keep the proxy out of sync — consider a manager-level synth-health surface if this becomes a support burden.
|
||||
- **Pricing-file lifecycle logs at info** (load, reload, revert-to-built-ins) and **at warn** for a runtime reload failure; the mtime check itself is `Debugf`. There is no metric on reload failures, so an operator who breaks the file mid-flight keeps billing at the previous table with only a log line to show it (`pricing/override.go:113-148`).
|
||||
|
||||
## Test coverage
|
||||
|
||||
@@ -198,6 +291,9 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest
|
||||
| `synthesizer_guardrail_realstore_test.go` | `PromptCaptureAccountIsSoleControl`; `PromptCaptureFlowsWhenAccountOptsIn`; `AccountRedactWithoutGuardrailRedact`; `NoGuardrail_CaptureOff`. |
|
||||
| `synthesizer_log_collection_realstore_test.go` | `LogCollection{Off_SuppressesAccessLog,On_PermitsAccessLog}` — verifies `DisableAccessLog` propagation through `ToProtoMapping`. |
|
||||
| `synthesizer_parser_redact_realstore_test.go` | **Capture-pointer regression suite:** `ParserConfigsCarryRedactPii`; `ParserConfigsSuppressCaptureWhenLogCollectionOnly` (log=on/prompt=off ⇒ both capture flags false); `ParserConfigsOmitRedactPiiWhenOff`. |
|
||||
| `synthesizer_pricing_test.go` | `BuildCostMeterConfig_{BedrockModelNormalization,CacheRateNilVsZero,OrphanAndGatewayProviders}` — the per-record tier's three load-bearing rules: keys normalized like the parser's, `nil` cache pointer inherits vs explicit `0` bills at input rate, and orphan / gateway (empty `Models`) providers ship no per-record entry. |
|
||||
| `pricing/defaults_test.go` | `DefaultTable_{CoversEveryCatalogModel,NoConflictingContributions,AllRatesFiniteNonNegative,PinnedRates}`; `LookupDefault_SurfaceOrder`. Catalog-derived coverage + rate sanity are structural, not curated. |
|
||||
| `pricing/override_test.go` | `LoadFile_{MergesOverCompiledDefaults,MissingPath,RejectsInvalid}`; `Reload_LifeCycle` (mtime detect, parse error keeps previous, delete reverts to built-ins); `ExampleYAML_InSyncWithBuiltins` golden. |
|
||||
| `policyselect_test.go` | Mock-store: `NoApplicablePolicies`; `AllowWithLowestGroupAttribution`; `LargerPoolWinsAcrossUsageLevels`; `StaysOnLargerPoolAfterPartialDrain`; `FallsThroughToSmallerPoolWhenLargerExhausted`; `TiebreakBy{LargerGroupPool,CreatedAt}`; `DeniesWhenAllExhausted`; `UncappedPolicyAlwaysWinsAgainstCapped`; `DisabledPolicyIgnored`; `StoreErrorPropagates`; `RejectsEmptyAccount`; `SharesGroupCounterAcrossPolicies`; `AntiFallThroughOnLowestGroup`; `BudgetOnlyExhaustionDenies`; `BudgetTighterThanTokenWins`. |
|
||||
| `policyselect_realstore_test.go` | Real-sqlite regression guard: `NoApplicablePolicies`; `AllowAndLowestGroupAttribution`; `LargerPoolWins_FallsThroughWhenExhausted`; `BudgetCapDenies`; `GroupCounterSharedAcrossPolicies`; `DisabledPolicyIgnored`. |
|
||||
| `policyselect_account_realstore_test.go` | Account budget rules: `AccountCeilingBindsEvenWithUncappedPolicy` (min-wins); `AccountGroupCeiling`; `AccountTargetUsersBindsOnlyThatUser`; `AccountRuleRecordsToOwnWindow`. |
|
||||
|
||||
@@ -5,7 +5,7 @@ LLM request. The two highest-blast-radius areas are the **capture-pointer
|
||||
semantics** and the **limit_check ⇒ limit_record** record-once invariant.
|
||||
|
||||
Sibling module: [32-proxy-llm-parsers.md](./32-proxy-llm-parsers.md) — the SDK
|
||||
adapters + pricing catalog this chain delegates to.
|
||||
adapters + pricing table and cost formula this chain delegates to.
|
||||
|
||||
---
|
||||
|
||||
@@ -34,7 +34,7 @@ rewrites.
|
||||
| `llm_identity_inject` | OnRequest | `llm.{resolved_provider_id,authorising_groups}`, `Input.{UserEmail,UserID,UserGroups,UserGroupNames}` | none | header strip/inject + optional body rewrite |
|
||||
| `llm_guardrail` | OnRequest | `llm.{model,request_prompt_raw}` | `llm_policy.{decision,reason}`, `llm.request_prompt` | none (model allowlist deny) |
|
||||
| `llm_response_parser` | OnResponse | `llm.provider`, `Input.{RespHeaders,RespBody,Status}` | `llm.{input,output,total,cached_input,cache_creation}_tokens`, `llm.response_completion` | none |
|
||||
| `cost_meter` | OnResponse | `llm.{provider,model}`, token buckets | `cost.usd_total` or `cost.skipped` | pricing lookup |
|
||||
| `cost_meter` | OnResponse | `llm.{provider,model,resolved_provider_id}`, token buckets | `cost.usd_{input,cached_input,cache_creation,output,total,cache}` or `cost.skipped` | none (in-memory pricing lookup) |
|
||||
| `llm_limit_record` | OnResponse | `llm.{attribution_group_id,attribution_window_seconds,input_tokens,output_tokens}`, `cost.usd_total` | none | gRPC `RecordLLMUsage` |
|
||||
|
||||
[all_test.go:26–40](../../../proxy/internal/middleware/builtin/all_test.go)
|
||||
@@ -44,7 +44,7 @@ locks the ID set; adding or removing one is a conscious extension.
|
||||
|
||||
| File | LOC | Notes |
|
||||
|---|---:|---|
|
||||
| `builtin.go` | 86 | Registry + `FactoryContext` (ctx, data dir, meter, logger, mgmt client) |
|
||||
| `builtin.go` | 90 | Registry + `FactoryContext` (ctx, meter, logger, mgmt client) |
|
||||
| `all_test.go` | 41 | Locks the 8-ID registry surface |
|
||||
| `agentnetwork_chain_integration_test.go` | 319 | Live sqlite + real gRPC bufconn; gate→recorder wire path |
|
||||
| `llm_request_parser/*` | 162 / 66 / 356 | Provider detection, body parse, prompt extraction with capture-pointer gating |
|
||||
@@ -53,7 +53,7 @@ locks the ID set; adding or removing one is a conscious extension.
|
||||
| `llm_identity_inject/*` | 440 / 108 / 666 | HeaderPair (LiteLLM) + JSONMetadata (Portkey) + ExtraHeaders |
|
||||
| `llm_guardrail/*` | 176 / 82 / 75 / 219 / 217 | Model allowlist + optional prompt capture with PII redaction |
|
||||
| `llm_response_parser/*` | 258 / 222 / 43 / 433 / 169 / 111 | Buffered + SSE accumulation; AWS event-stream accumulator (`streaming_bedrock.go`) for Bedrock; capture-pointer gates completion emit |
|
||||
| `cost_meter/*` | 181 / 84 / 439 | Token → USD via `proxy/internal/llm/pricing` |
|
||||
| `cost_meter/*` | 236 / 98 / 586 | Token → USD via `proxy/internal/llm/pricing`; both pricing tiers arrive in the middleware config |
|
||||
| `llm_limit_record/*` | 144 / 35 / 191 | Post-flight `RecordLLMUsage` (5s, debug-on-error) |
|
||||
|
||||
## Per-middleware
|
||||
@@ -168,12 +168,46 @@ token schema.
|
||||
|
||||
### cost_meter
|
||||
|
||||
Reads `llm.provider` + `llm.model` + token buckets, looks up per-1k rate via
|
||||
`pricing.Loader`, emits `cost.usd_total` or a closed-set `cost.skipped`
|
||||
reason (`missing_provider/model/tokens`, `unparseable_tokens`, `zero_tokens`,
|
||||
`unknown_model`). Loader's hot-reload goroutine is bound to proxy-lifetime
|
||||
context via `startReloader`. **Key invariant:** provider-shape switch lives
|
||||
in `pricing.Table.Cost` (sibling doc) — `cost_meter` stays provider-agnostic.
|
||||
Reads `llm.provider` + `llm.model` + token buckets, looks up the per-1k rates,
|
||||
and emits the full `cost.usd_*` breakdown (four per-bucket values plus the
|
||||
`_total` and `_cache` aggregates) or a closed-set `cost.skipped` reason
|
||||
(`missing_provider/model/tokens`, `unparseable_tokens`, `zero_tokens`,
|
||||
`unknown_model`).
|
||||
|
||||
**Management owns pricing.** The proxy carries no embedded price list: the whole
|
||||
table arrives in this middleware's `ConfigJSON` as
|
||||
`{pricing: {defaults, providers}}`, synthesized by management from the catalog
|
||||
plus the operator's stored per-provider prices
|
||||
([factory.go:13–34](../../../proxy/internal/middleware/builtin/cost_meter/factory.go)).
|
||||
Both tiers are validated by `pricing.NewTable` / `pricing.NewEntries` at
|
||||
construction, so a non-finite or negative rate fails the chain build. A price
|
||||
change is an ordinary mapping push — the chain rebuild yields a fresh instance
|
||||
over a fresh immutable table, so there is no data dir, no pricing file, no
|
||||
reload goroutine, and nothing to invalidate.
|
||||
|
||||
**Two-tier lookup**
|
||||
([middleware.go:165–183](../../../proxy/internal/middleware/builtin/cost_meter/middleware.go)):
|
||||
|
||||
1. **Per-provider-record** — the operator's stored price for the route that
|
||||
actually served the request, keyed by the `llm.resolved_provider_id` that
|
||||
`llm_router` stamped on the allow path, then by normalized model id. Entries
|
||||
arrive fully materialized (management folds default cache rates in at synth
|
||||
time), so there is no merging here. Absent metadata — no router in the chain
|
||||
— skips this tier.
|
||||
2. **Surface defaults** — the catalog-derived table keyed by `llm.provider`
|
||||
(`openai`/`anthropic`/`bedrock`). This is also what prices gateway-style
|
||||
providers, which enumerate no models and therefore get no per-record entry.
|
||||
|
||||
**Backward compatibility:** a config with no `pricing` block means management
|
||||
predates config-delivered pricing. The factory logs one warning at build time
|
||||
and the instance records `cost.skipped=unknown_model` ($0) for every request
|
||||
rather than falling back to a stale built-in price list
|
||||
([factory.go:55–60](../../../proxy/internal/middleware/builtin/cost_meter/factory.go)).
|
||||
|
||||
**Key invariant:** the provider-shape switch lives in `pricing.EntryCosts`
|
||||
(sibling doc) and is selected by the **surface**, not by which tier the entry
|
||||
came from — `cost_meter` stays provider-agnostic, and a per-record override on
|
||||
an Anthropic route still bills its cache buckets additively.
|
||||
|
||||
### llm_limit_record
|
||||
|
||||
@@ -246,12 +280,14 @@ no mocks. Tests: `TestChain_AllowPath_StampsAttributionAndRecordsCounter`
|
||||
| `llm_identity_inject` | `{providers: [{provider_id, header_pair?|json_metadata?, extra_headers?}]}` |
|
||||
| `llm_guardrail` | `{provider_allowlists: {providerID: []string}, prompt_capture: {enabled, redact_pii}}` — allowlist keyed by resolved provider id; a provider absent from the map is unrestricted (fail-closed backstop; authoritative per-policy/group check is management's `CheckLLMPolicyLimits`) |
|
||||
| `llm_response_parser` | `{redact_pii?, capture_completion?: *bool}` |
|
||||
| `cost_meter` | `{pricing_path?}` (basename inside data-dir; defaults `pricing.yaml`) |
|
||||
| `cost_meter` | `{pricing: {defaults: {surface: {model: rates}}, providers: {providerRecordID: {model: rates}}}}` — rates are `{input_per_1k, output_per_1k, cached_input_per_1k?, cache_read_per_1k?, cache_creation_per_1k?}`. A missing `pricing` key means "management predates config-delivered pricing": every request records `cost.skipped=unknown_model` |
|
||||
| `llm_limit_record` | `{}` — same pattern as `llm_limit_check` |
|
||||
|
||||
All factories accept empty / null / `{}` / whitespace as zero-value config;
|
||||
only structurally invalid JSON is rejected so misconfig surfaces at chain
|
||||
build time.
|
||||
build time. `cost_meter` adds a semantic check on top of that: a `pricing`
|
||||
block carrying a negative or non-finite rate fails the build too, rather than
|
||||
mispricing live traffic.
|
||||
|
||||
## Invariants
|
||||
|
||||
@@ -320,10 +356,11 @@ non-object `metadata` field
|
||||
— header path still attributes, but body-level tag-budget enforcement
|
||||
doesn't run for that request.
|
||||
|
||||
**Concurrency.** `cost_meter` shares a `pricing.Loader` via
|
||||
`atomic.Pointer[Table]`; readers always see a consistent table. Every
|
||||
middleware is a stateless value receiver. Integration test uses real bufconn
|
||||
gRPC — race detector is the meaningful bar.
|
||||
**Concurrency.** `cost_meter`'s two pricing tables are built once from the
|
||||
middleware config and never mutated, so the lookup path needs no lock or atomic
|
||||
swap — a price change replaces the whole instance. Every middleware is
|
||||
otherwise a stateless value receiver. Integration test uses real bufconn gRPC —
|
||||
race detector is the meaningful bar.
|
||||
|
||||
**Perf.** Hot path is `lookupKV` linear scan over <10 KVs; `cost_meter.Cost`
|
||||
is O(1); SSE accumulation is single-pass. No map allocation per call.
|
||||
@@ -349,13 +386,13 @@ counter accuracy.
|
||||
| `llm_guardrail/redact_test.go` | 15 | Email, SSN, phone (E.164 + NA), bearer, IPv4; fixture-driven |
|
||||
| `llm_response_parser/middleware_test.go` | 18 | Buffered OAI+Anthro, capture-pointer, redact, truncation |
|
||||
| `llm_response_parser/streaming_test.go` | 7 | OAI usage frame, Anthro message_delta, truncated body best-effort |
|
||||
| `cost_meter/middleware_test.go` | 17 | Each skip reason, provider-shape, pricing loader integration |
|
||||
| `cost_meter/middleware_test.go` | 22 | Each skip reason, provider-shape formulas, config-delivered defaults, per-record-beats-defaults + miss-falls-back, per-record uses surface formula, nil-pricing skips everything, invalid-rate rejection |
|
||||
| `llm_limit_record/middleware_test.go` | 7 | Skip-on-no-signal, skip-on-missing-attribution, RPC failure swallowed |
|
||||
|
||||
## Cross-references
|
||||
|
||||
- Sibling: [32-proxy-llm-parsers.md](./32-proxy-llm-parsers.md) — SDK adapters
|
||||
+ SSE framer + pricing loader.
|
||||
+ SSE framer + pricing table and cost formula.
|
||||
- Path-routed providers (Vertex AI + Bedrock), `keyfile::` credential, GCP
|
||||
token minting, `/bedrock` prefix:
|
||||
[50-path-routed-providers.md](./50-path-routed-providers.md).
|
||||
|
||||
@@ -9,7 +9,7 @@ pricing table's per-provider cost formula is the highest-leverage place a
|
||||
small bug would silently mis-bill operators.
|
||||
|
||||
Sibling module: [31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md)
|
||||
— the 8 middlewares that consume this package's parsers + pricing loader.
|
||||
— the 8 middlewares that consume this package's parsers + pricing table.
|
||||
|
||||
---
|
||||
|
||||
@@ -24,8 +24,9 @@ proxy-framework dependencies:
|
||||
- `openai.go` / `anthropic.go` / `bedrock.go` — per-provider `Parser` impls.
|
||||
- `sse.go` — SSE scanner (`Scanner`, `Event`, `NewScanner`).
|
||||
- `errors.go` — sentinels callers branch on with `errors.Is`.
|
||||
- `pricing/` — embedded-default + hot-reload override table with
|
||||
symlink-safe Unix loader (build-tagged stub elsewhere).
|
||||
- `pricing/` — immutable pricing table + the per-surface cost formula. The
|
||||
rates themselves come from management inside `cost_meter`'s middleware
|
||||
config; this package holds no price list and reads no files.
|
||||
- `fixtures/` — captured request/response/stream bodies the tests replay.
|
||||
|
||||
The package carries zero proxy-framework dependencies so the same parsers can
|
||||
@@ -47,12 +48,9 @@ be reused later by a WASM adapter
|
||||
| `sse_test.go` | 175 | 12 tests; fixture replay + multiline + size limits |
|
||||
| `parser_test.go` | 53 | `Parsers()`, `DetectParser`, provider enum values |
|
||||
| `errors.go` | 31 | 6 sentinels: `Err{Unknown,Unsupported}Provider/Model`, `Err{NotLLM,Malformed}Response`, `ErrStreamingUnsupported`, `ErrMalformedRequest` |
|
||||
| `pricing/pricing.go` | 421 | `Loader`, `Table`, `Entry`; embedded defaults + atomic swap + mtime reload |
|
||||
| `pricing/pricing_unix.go` | 69 | `O_NOFOLLOW` + fstat-from-FD + 1 MiB cap |
|
||||
| `pricing/pricing_other.go` | 21 | Stub returning "not supported on this platform" |
|
||||
| `pricing/pricing_test.go` | 432 | 21 tests — symlink rejection, reload race, path traversal, oversize |
|
||||
| `pricing/defaults_pricing.yaml` | 85 | go:embed source of truth |
|
||||
| `fixtures/*` | 21–59 | OAI chat/responses/stream + Anthro messages/stream + pricing starter |
|
||||
| `pricing/pricing.go` | 234 | `Table`, `Entry`, `EntryJSON`, `Costs`; `NewTable`/`NewEntries` validation + `EntryCosts` formula. No I/O, no reload, no embedded rates |
|
||||
| `pricing/pricing_test.go` | 177 | 10 tests — provider-shape formulas, cached clamp, rate fallback, nil-safety, rate validation |
|
||||
| `fixtures/*` | 21–59 | OAI chat/responses/stream + Anthro messages/stream |
|
||||
|
||||
## Request body → parser dispatch
|
||||
|
||||
@@ -188,9 +186,11 @@ response leg, covering both Bedrock body shapes:
|
||||
`totalTokens`). `firstNonZero` folds the two naming conventions into one
|
||||
`Usage`; when Converse omits `totalTokens` the parser sums the buckets.
|
||||
|
||||
`ProviderName()` returns `"bedrock"` — its own `defaults_pricing.yaml` block,
|
||||
keyed by the **normalised** model id (region prefix + version suffix stripped by
|
||||
the request parser). `ParseResponse` returns `ErrStreamingUnsupported` for an
|
||||
`ProviderName()` returns `"bedrock"` — its own pricing surface in the table
|
||||
management ships, keyed by the **normalised** model id (region prefix + version
|
||||
suffix stripped by the request parser; management normalises its keys the same
|
||||
way at synth time so the two compare equal). `ParseResponse` returns
|
||||
`ErrStreamingUnsupported` for an
|
||||
AWS binary event-stream content-type (`application/vnd.amazon.eventstream`,
|
||||
`isAWSEventStream`) so the caller routes to the streaming accumulator instead.
|
||||
|
||||
@@ -205,11 +205,34 @@ response body. Streaming accumulators live in the middleware package
|
||||
([llm_response_parser/streaming.go](../../../proxy/internal/middleware/builtin/llm_response_parser/streaming.go))
|
||||
but use `llm.NewScanner` so the framing contract stays here.
|
||||
|
||||
### Pricing catalog
|
||||
### Pricing table
|
||||
|
||||
`Table.Cost`
|
||||
([pricing.go:129–174](../../../proxy/internal/llm/pricing/pricing.go))
|
||||
is the cost formula — most security-relevant math in this module:
|
||||
**Management is the sole pricing authority.** The proxy carries no embedded
|
||||
price list and reads no pricing file: the whole table arrives inside
|
||||
`cost_meter`'s `ConfigJSON` on the ordinary mapping push, and a price change
|
||||
is just another push — the chain rebuild constructs a fresh `Table`, so there
|
||||
is nothing to reload
|
||||
([pricing.go:1–7](../../../proxy/internal/llm/pricing/pricing.go)). The
|
||||
management side of the contract (catalog defaults, the operator's stored
|
||||
per-provider prices, and `AgentNetwork.PricingDefaultsFile`) is covered in the
|
||||
management-side module guide; `cost_meter`'s wire shape is in
|
||||
[31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md).
|
||||
|
||||
`EntryJSON`
|
||||
([pricing.go:36–45](../../../proxy/internal/llm/pricing/pricing.go)) is the
|
||||
management→proxy contract — five USD-per-1k rates under `input_per_1k`,
|
||||
`output_per_1k`, `cached_input_per_1k`, `cache_read_per_1k`,
|
||||
`cache_creation_per_1k`. Management's `pricing.Entry` marshals the identical
|
||||
names, and `EntryJSON`/`Entry` are field-identical so `NewEntries` converts by
|
||||
direct struct conversion rather than field-by-field copying (a new rate can't
|
||||
be silently dropped in transit).
|
||||
|
||||
`EntryCosts`
|
||||
([pricing.go:183–234](../../../proxy/internal/llm/pricing/pricing.go))
|
||||
is the cost formula — most security-relevant math in this module. The
|
||||
**surface** (the `llm.provider` value the request parser stamped) selects the
|
||||
formula, never the tier the entry came from: a per-provider-record override on
|
||||
an Anthropic route still bills its cache buckets additively.
|
||||
|
||||
| Provider | Formula |
|
||||
|---|---|
|
||||
@@ -218,7 +241,7 @@ is the cost formula — most security-relevant math in this module:
|
||||
| default | `inTokens × InputPer1K + outTokens × OutputPer1K` |
|
||||
|
||||
`bedrock` shares the Anthropic additive-cache formula
|
||||
([pricing.go:172-174](../../../proxy/internal/llm/pricing/pricing.go)):
|
||||
([pricing.go:214–229](../../../proxy/internal/llm/pricing/pricing.go)):
|
||||
Anthropic-on-Bedrock reports the same additive cache buckets, while non-Anthropic
|
||||
Bedrock models (Nova, Llama) simply report zero in those buckets so cost reduces
|
||||
to `input + output`.
|
||||
@@ -226,15 +249,12 @@ to `input + output`.
|
||||
Each per-bucket rate falls back to `InputPer1K` when zero — operators opt in
|
||||
to discounts by setting the field.
|
||||
|
||||
`Loader`
|
||||
([pricing.go:212–268](../../../proxy/internal/llm/pricing/pricing.go))
|
||||
overlays an optional `pricing.yaml` from data-dir on top of the go:embed
|
||||
defaults. Atomic pointer swap means readers never observe a partial update.
|
||||
The mtime-poll reloader (30s default cadence) keeps the previous table on
|
||||
parse failure so cost annotation never goes blank during a botched edit.
|
||||
|
||||
`defaults_pricing.yaml` is the source of truth for built-in pricing.
|
||||
Operator overrides only carry the entries they want to change.
|
||||
`Costs`
|
||||
([pricing.go:143–163](../../../proxy/internal/llm/pricing/pricing.go)) is the
|
||||
per-request split. The four per-bucket fields are the base; `TotalUSD` and
|
||||
`CacheUSD` are **derived** in `newCosts` so the aggregates can never drift from
|
||||
the breakdown. `InputUSD` is always the non-cached input bucket on both
|
||||
provider shapes, so input and cached-input never double-count.
|
||||
|
||||
## Public contracts
|
||||
|
||||
@@ -264,29 +284,38 @@ Order matters: `DetectFromURL` ties resolve by registration order.
|
||||
`ProviderBedrock = 3`. Numeric values are persisted in nothing today but treat
|
||||
them as wire-stable — new providers must take fresh numbers.
|
||||
|
||||
**`Pricing` lookup**
|
||||
([pricing.go:129](../../../proxy/internal/llm/pricing/pricing.go)):
|
||||
**`Pricing` construction + lookup**
|
||||
([pricing.go:60–130](../../../proxy/internal/llm/pricing/pricing.go)):
|
||||
|
||||
```go
|
||||
func NewEntries(raw map[string]map[string]EntryJSON) (map[string]map[string]Entry, error)
|
||||
func NewTable(raw map[string]map[string]EntryJSON) (*Table, error)
|
||||
|
||||
func (t *Table) Lookup(provider, model string) (Entry, bool)
|
||||
func (t *Table) Cost(provider, model string, inTokens, outTokens, cachedInput, cacheCreation int64) (float64, bool)
|
||||
func (t *Table) Costs(provider, model string, inTokens, outTokens, cachedInput, cacheCreation int64) (Costs, bool)
|
||||
func EntryCosts(entry Entry, surface string, inTokens, outTokens, cachedInput, cacheCreation int64) Costs
|
||||
```
|
||||
|
||||
Nil-safe: `t.Cost` on a nil receiver returns `(0, false)`
|
||||
([pricing.go:130–132](../../../proxy/internal/llm/pricing/pricing.go)).
|
||||
`ok=false` means provider or model is absent from the loaded table; the caller
|
||||
emits `cost.skipped=unknown_model`.
|
||||
`NewTable` is the surface-keyed defaults table; `NewEntries` returns the raw
|
||||
two-level map `cost_meter` uses for the per-provider-record tier (it looks up an
|
||||
`Entry` directly and calls `EntryCosts`, so it needs no `Table` wrapper). Both
|
||||
reject any non-finite or negative rate, so a corrupt config fails the chain
|
||||
build rather than mispricing silently. Nil input yields an empty,
|
||||
never-matching table.
|
||||
|
||||
Nil-safe: `t.Cost`/`t.Lookup` on a nil receiver returns `ok=false`
|
||||
([pricing.go:96–99](../../../proxy/internal/llm/pricing/pricing.go)).
|
||||
`ok=false` means the surface or model is absent from the table management sent;
|
||||
the caller emits `cost.skipped=unknown_model`.
|
||||
|
||||
## Invariants
|
||||
|
||||
1. **Cross-platform pricing build.** `pricing_unix.go` carries the only
|
||||
functional `loadPricing` (uses `syscall.O_NOFOLLOW` and `f.Stat()` on an
|
||||
open descriptor — both Unix-only). `pricing_other.go` is a build-tag
|
||||
fallback that returns `"not supported on this platform"`
|
||||
([pricing_other.go:14–16](../../../proxy/internal/llm/pricing/pricing_other.go)).
|
||||
The proxy is Linux-only in production today; a Windows port needs an
|
||||
equivalent path-as-handle implementation. Reviewers building on Windows
|
||||
should expect this surface to return an error at startup if an override
|
||||
file is configured.
|
||||
1. **The pricing package is pure and platform-independent.** No file I/O, no
|
||||
`//go:embed`, no goroutines, no build tags — the rates arrive as config, so
|
||||
there is nothing platform-specific left to port. Anything reintroducing a
|
||||
read-from-disk path here re-splits pricing authority between management and
|
||||
the proxy, which is exactly what this design removed.
|
||||
|
||||
2. **SSE scanner handles partial chunks.** A buffered prefix that doesn't end
|
||||
in `\n\n` still yields its accumulated event before `io.EOF`
|
||||
@@ -298,38 +327,45 @@ emits `cost.skipped=unknown_model`.
|
||||
usage rather than aborting
|
||||
([streaming.go:68–73, 144–150](../../../proxy/internal/middleware/builtin/llm_response_parser/streaming.go)).
|
||||
|
||||
3. **`defaults_pricing.yaml` is the source of truth.** Compiled into the
|
||||
binary via `//go:embed`
|
||||
([pricing.go:29–30](../../../proxy/internal/llm/pricing/pricing.go)).
|
||||
`DefaultTable()` parses once and panics on parse failure
|
||||
([pricing.go:42–49](../../../proxy/internal/llm/pricing/pricing.go))
|
||||
— by design: a broken embedded YAML must not ship to production.
|
||||
3. **Management is the only source of rates.** `Table` has no constructor that
|
||||
invents prices: the only way in is `NewTable`/`NewEntries` over the wire map
|
||||
management sent. A missing or empty `pricing` block therefore means *no
|
||||
prices at all* (`cost_meter` records `cost.skipped=unknown_model`, $0) —
|
||||
never a stale built-in fallback that would silently bill list price.
|
||||
|
||||
4. **Loader path validation.** `resolveMiddlewareDataPath`
|
||||
([pricing.go:370–394](../../../proxy/internal/llm/pricing/pricing.go))
|
||||
rejects absolute paths, traversal segments, and basenames that fail
|
||||
`basenameRegex = ^[a-zA-Z0-9._-]+$`. The resolved path must remain
|
||||
inside `baseDir` even after `filepath.Clean`. Tests:
|
||||
`TestNewLoader_PathValidation`, `TestNewLoader_PathValidation_Extended`,
|
||||
`TestNewLoader_SymlinkOutsideBaseDirRejected`, `TestNewLoader_SymlinkRejected`.
|
||||
4. **Tables are immutable once built.** `Table.entries` is written only in
|
||||
`NewEntries` and never mutated afterwards, and `cost_meter`'s `perRecord`
|
||||
map is likewise build-time-only
|
||||
([pricing.go:47–52](../../../proxy/internal/llm/pricing/pricing.go)). This
|
||||
is what makes the no-reload design safe: a price change arrives as a mapping
|
||||
push that builds a new middleware instance over a new table, so concurrent
|
||||
readers can't observe a half-updated price list and no atomic swap or lock
|
||||
is needed on the hot path.
|
||||
|
||||
5. **Unix loader symlink safety.** `O_NOFOLLOW` on open, `f.Stat()` on the
|
||||
open descriptor (never re-stat by path), `info.Mode().IsRegular()` check,
|
||||
`io.LimitReader(f, maxPricingBytes+1)` with a final size assertion
|
||||
([pricing_unix.go:25–57](../../../proxy/internal/llm/pricing/pricing_unix.go)).
|
||||
A mid-read symlink swap is detected because the fstat is on the original
|
||||
fd. Test: `TestNewLoader_RejectsOversizedFile_FixesM4`.
|
||||
5. **Rate validation happens at chain-build time, not per request.**
|
||||
`NewEntries` rejects negative, NaN, and ±Inf rates field by field
|
||||
([pricing.go:60–83](../../../proxy/internal/llm/pricing/pricing.go)), naming
|
||||
the offending surface/model/field in the error. Management enforces the same
|
||||
constraints at its API boundary and in its YAML parser, so this is
|
||||
defense-in-depth — but it means a corrupt push fails loudly at build instead
|
||||
of producing negative costs on live traffic. Test:
|
||||
`TestNewTable_ValidatesRates`.
|
||||
|
||||
6. **`yaml.NewDecoder(...).KnownFields(true)`**
|
||||
([pricing.go:397–398](../../../proxy/internal/llm/pricing/pricing.go))
|
||||
rejects YAML files that carry fields not in the schema. A typo in an
|
||||
operator override file fails loud instead of silently zeroing rates.
|
||||
6. **New rates must be added to `Entry`, `EntryJSON`, *and* management's
|
||||
`pricing.Entry` together.** `NewEntries` converts by direct struct
|
||||
conversion `Entry(e)`
|
||||
([pricing.go:76–78](../../../proxy/internal/llm/pricing/pricing.go)), which
|
||||
only compiles while the two structs stay field-identical — so the proxy half
|
||||
is compiler-enforced. The management half is not: a rate added there but not
|
||||
here unmarshals into nothing and prices that bucket at `InputPer1K`.
|
||||
|
||||
## Things to scrutinise
|
||||
|
||||
**Correctness.** Verify OpenAI cached-prompt clamp at
|
||||
[pricing.go:147–149](../../../proxy/internal/llm/pricing/pricing.go)
|
||||
short-circuits before subtraction. `Anthropic.TotalTokens` sums all four
|
||||
**Correctness.** Verify the OpenAI cached-prompt clamp at
|
||||
[pricing.go:203–206](../../../proxy/internal/llm/pricing/pricing.go)
|
||||
short-circuits before subtraction. Negative token counts are clamped to zero up
|
||||
front ([pricing.go:186–197](../../../proxy/internal/llm/pricing/pricing.go)) so
|
||||
no formula can yield a negative cost. `Anthropic.TotalTokens` sums all four
|
||||
buckets (in + out + cache_read + cache_creation) — downstream dashboards
|
||||
need to know this differs from `input + output`.
|
||||
`OpenAIParser.ExtractPrompt` falls through `messages → input → prompt`; a
|
||||
@@ -338,22 +374,27 @@ noting).
|
||||
|
||||
**Security.** `Scanner.maxLine = 1 MiB`; a 2 MiB single-line `data:` event
|
||||
errors from `Scanner.Next` and both accumulators stop with partial usage.
|
||||
Pricing file 1 MiB cap is orders of magnitude larger than realistic. Confirm
|
||||
new schema additions are mirrored in both `pricingFile` and `Entry`;
|
||||
`KnownFields(true)` will reject silently-typo'd operator overrides
|
||||
otherwise.
|
||||
Pricing is no longer file-backed, so the loader's path-traversal / symlink /
|
||||
oversize surface is gone entirely — the config channel (an authenticated
|
||||
mapping push from management) is now the only way rates enter the proxy, and
|
||||
`NewEntries` is the validation boundary on it. A new rate added to management's
|
||||
`pricing.Entry` but not to `EntryJSON` here is the remaining silent-mispricing
|
||||
path (see invariant 6).
|
||||
|
||||
**Concurrency.** `Loader.table` is `atomic.Pointer[Table]`; readers never
|
||||
block or see a torn table. `Loader.Reload` is one goroutine, cancelled via
|
||||
context (`TestLoader_ReloadBackgroundLoopCancellation`). `DefaultTable()`
|
||||
uses `sync.Once`. Per-call `Scanner` instances mean no shared state across
|
||||
concurrent response-parser calls.
|
||||
**Concurrency.** Nothing in this package is shared mutable state: tables are
|
||||
built once and never written again, so `cost_meter`'s hot path is lock-free by
|
||||
construction rather than by atomic swap. Per-call `Scanner` instances mean no
|
||||
shared state across concurrent response-parser calls.
|
||||
|
||||
**Perf.** `Table.Cost` is two map lookups + multiplications, O(1).
|
||||
`Scanner.Next` is one `ReadString('\n')` per line. Pricing reload poll 30s.
|
||||
**Perf.** `Table.Cost` is two map lookups + multiplications, O(1); the
|
||||
per-provider-record tier adds at most one more lookup. `Scanner.Next` is one
|
||||
`ReadString('\n')` per line. No background goroutines and no per-request
|
||||
allocation of pricing state.
|
||||
|
||||
**Observability.** Reload failures count via `metric.Int64Counter` keyed
|
||||
`plugin`; warning log rate-limited at 5 min so a broken file doesn't flood.
|
||||
**Observability.** A config carrying no `pricing` block logs one warning at
|
||||
chain-build time (`cost_meter` factory) and then records
|
||||
`cost.skipped=unknown_model` per request, so an old-management deployment is
|
||||
visible in both logs and the access log rather than quietly reporting $0.
|
||||
Parser errors return sentinels — middleware uses `errors.Is` to map to the
|
||||
right `cost.skipped` reason.
|
||||
|
||||
@@ -365,7 +406,7 @@ right `cost.skipped` reason.
|
||||
| `openai_test.go` | 11 | Chat Completions + Responses API + legacy `prompt`; cached-tokens subset for both naming conventions; fixture replays |
|
||||
| `anthropic_test.go` | 7 | Messages + legacy `/v1/complete`; streaming REJECTED on `ParseResponse` (must use scanner); fixture replays |
|
||||
| `sse_test.go` | 12 | Fixture replay both providers; multiline `data:`; CRLF; comment skip; trailing-event-without-blank-line; oversize rejection |
|
||||
| `pricing/pricing_test.go` | 21 | Provider-shape switch; cached-rate fallback; cached-clamp; symlink rejection (target outside basedir + symlink to file); path validation matrix; oversize rejection; reload-keeps-previous-on-parse-error; mtime change detection; goroutine cancellation |
|
||||
| `pricing/pricing_test.go` | 10 | Provider-shape switch (surface selects the formula); cached-rate + cache-read/creation fallback to `InputPer1K`; cached-clamp; negative-token clamp; nil-receiver safety; rate validation (negative / NaN / Inf rejected); nil + empty table |
|
||||
|
||||
**Fixtures** ([proxy/internal/llm/fixtures/](../../../proxy/internal/llm/fixtures/)):
|
||||
`openai_chat_completion.json` (chat.completions with usage),
|
||||
@@ -373,14 +414,15 @@ right `cost.skipped` reason.
|
||||
`openai_stream.txt` (3 deltas + usage + `[DONE]`),
|
||||
`anthropic_messages.json` (Messages API non-streaming),
|
||||
`anthropic_stream.txt` (full 7-event sequence: message_start →
|
||||
content_block_{start,delta×2,stop} → message_delta (usage) → message_stop),
|
||||
`pricing.yaml` (realistic-pricing starter for operator overrides).
|
||||
content_block_{start,delta×2,stop} → message_delta (usage) → message_stop).
|
||||
No pricing fixture: the table is config-delivered, so pricing tests construct
|
||||
it in-process from a wire-shape map.
|
||||
|
||||
## Cross-references
|
||||
|
||||
- Sibling: [31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md)
|
||||
— the chain that calls `llm.Parsers()`, `llm.ParserByName`,
|
||||
`llm.NewScanner`, `pricing.NewLoader`.
|
||||
`llm.NewScanner`, `pricing.NewTable` / `pricing.NewEntries`.
|
||||
- Path-routed providers (Vertex AI + Bedrock), credential syntax, and the
|
||||
Bedrock AWS event-stream accumulator:
|
||||
[50-path-routed-providers.md](./50-path-routed-providers.md).
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# proxy/runtime — translate + serve + log
|
||||
|
||||
> **Risk level:** High — every config push from management is translated here, and the chain runs on every HTTP request to a synth target.
|
||||
> **Backward-compat impact:** Additive at the wire (`PathTargetOptions.middlewares`, `agent_network`, `disable_access_log`, capture caps) and on the proxy `Server` struct (`MiddlewareDataDir`, `MiddlewareCaptureBudgetBytes`). Non-agent-network targets stay on the no-middleware fast path.
|
||||
> **Backward-compat impact:** Additive at the wire (`PathTargetOptions.middlewares`, `agent_network`, `disable_access_log`, capture caps) and on the proxy `Server` struct (`MiddlewareCaptureBudgetBytes`). Non-agent-network targets stay on the no-middleware fast path. Middleware config is entirely wire-delivered — no proxy-side data dir is involved, including for LLM pricing, which management ships inside `cost_meter`'s config.
|
||||
|
||||
## Module boundary
|
||||
|
||||
@@ -114,8 +114,7 @@ At **request time** the access-log middleware stamps `CapturedData`; the auth ch
|
||||
|
||||
## Public contracts touched
|
||||
|
||||
- `proxy.Server.MiddlewareDataDir` (string) — base dir for file-backed middleware config (server.go:238-241).
|
||||
- `proxy.Server.MiddlewareCaptureBudgetBytes` (int64) — process-wide capture cap; defaults to 256 MiB (server.go:248-250).
|
||||
- `proxy.Server.MiddlewareCaptureBudgetBytes` (int64) — process-wide capture cap; defaults to 256 MiB (server.go:249-253). There is no `MiddlewareDataDir`: no built-in middleware reads config from disk, so `builtin.FactoryContext` carries only the proxy-lifetime context, meter, logger, and management client.
|
||||
- `proxy/internal/proxy.WithMiddlewareManager(*middleware.Manager) Option` — new option on `NewReverseProxy`; nil keeps the fast path (reverseproxy.go:48-56).
|
||||
- `proxy/internal/proxy.PathTarget` adds `Middlewares`, `CaptureConfig`, `AgentNetwork`, `DisableAccessLog` (servicemapping.go:27-51), all zero-default.
|
||||
- `proxy/internal/proxy.CapturedData` adds `agentNetwork`, `suppressAccessLog`, `userGroupNames` behind `sync.RWMutex`; slices deep-copied (context.go:47-66, 183-258).
|
||||
|
||||
@@ -87,9 +87,9 @@ strips the `@version` suffix from the model, and maps the publisher to a parser
|
||||
surface via `vertexPublisherVendor`:
|
||||
|
||||
- `anthropic` → `llm.provider="anthropic"` → metered through the Anthropic
|
||||
parser, priced under the **`anthropic`** block in `defaults_pricing.yaml`
|
||||
(the parser emits the standard Anthropic provider label, so Vertex Claude
|
||||
reuses first-party Anthropic prices).
|
||||
parser, priced under the **`anthropic`** surface of the pricing table
|
||||
management ships (the parser emits the standard Anthropic provider label, so
|
||||
Vertex Claude reuses first-party Anthropic prices).
|
||||
- `openai` → `llm.provider="openai"` (reserved; not in the catalog lineup
|
||||
today).
|
||||
- anything else (notably `google` / Gemini) → empty vendor → **no parser**.
|
||||
@@ -104,8 +104,9 @@ is omitted from the catalog.
|
||||
|
||||
> Caveat: cross-region inference profiles in `eu` / `apac` carry a ~10% price
|
||||
> premium that the base per-token rates do **not** model — cost annotations for
|
||||
> those regions read low. Operators who need exact regional billing override
|
||||
> the affected entries in `pricing.yaml`.
|
||||
> those regions read low. Operators who need exact regional billing set the
|
||||
> affected models' prices on the provider record, or replace the default entries
|
||||
> via management's `AgentNetwork.PricingDefaultsFile`.
|
||||
|
||||
## AWS Bedrock (`bedrock_api`)
|
||||
|
||||
@@ -211,15 +212,19 @@ so a model-listing call can't be rewritten onto an upstream that would 404 it.
|
||||
## Catalog ↔ pricing cross-check
|
||||
|
||||
Catalog prices and context windows are cross-checked against LiteLLM's
|
||||
`model_prices_and_context_window.json`. The proxy's embedded
|
||||
`defaults_pricing.yaml` covers **every metered first-party model** the catalog
|
||||
enumerates — guarded by
|
||||
`TestDefaultTable_FirstPartyModelCoverage`
|
||||
([pricing/defaults_coverage_test.go](../../../proxy/internal/llm/pricing/defaults_coverage_test.go)),
|
||||
which fails if a catalog model has no embedded price. Bedrock entries are keyed
|
||||
by the **normalised** id the request parser emits (region prefix + version
|
||||
suffix stripped). Vertex Claude carries no Bedrock-style prefix, so it prices
|
||||
straight off the `anthropic` block.
|
||||
`model_prices_and_context_window.json`. The **catalog is the source of default
|
||||
prices**: management's `pricing.DefaultTable` folds every catalog provider's
|
||||
models into the surfaces that provider declares (`PricingSurfaces`), so coverage
|
||||
is structural rather than maintained in a parallel file
|
||||
([pricing/defaults.go](../../../management/internals/modules/agentnetwork/pricing/defaults.go)).
|
||||
`TestDefaultTable_CoversEveryCatalogModel` fails if a catalog model ends up
|
||||
unpriced, and `TestDefaultTable_NoConflictingContributions` fails if two
|
||||
providers contribute the same (surface, model) at different rates. Bedrock
|
||||
entries are keyed by the **normalised** id the request parser emits (region
|
||||
prefix + version suffix stripped) — management applies the same normalisation to
|
||||
per-provider prices at synth time, so the two keys compare equal. Vertex Claude
|
||||
carries no Bedrock-style prefix, so it prices straight off the `anthropic`
|
||||
surface.
|
||||
|
||||
## Things to scrutinise
|
||||
|
||||
@@ -232,16 +237,17 @@ operator-misconfigured Vertex provider and unmetered Gemini traffic; verify
|
||||
publishers).
|
||||
|
||||
**Correctness.** `normalizeBedrockModel` is the join between the wire id and the
|
||||
pricing key — a model that normalises to something not in `defaults_pricing.yaml`
|
||||
meters at `cost.skipped=unknown_model` rather than failing the request. The
|
||||
pricing key — a model that normalises to something absent from the shipped
|
||||
pricing table meters at `cost.skipped=unknown_model` rather than failing the
|
||||
request. The
|
||||
`/bedrock` prefix strip must run on both the parser side (so the model is
|
||||
extracted) and the router side (so the upstream path is native); a regression in
|
||||
either silently breaks the other.
|
||||
|
||||
**Metering caveats.** eu/apac cross-region Bedrock + Vertex profiles carry a
|
||||
~10% premium not modelled by base pricing — flagged in both the catalog comment
|
||||
and `defaults_pricing.yaml`. Operators needing exact regional billing override
|
||||
the relevant entries.
|
||||
~10% premium not modelled by base pricing — flagged in the catalog comment.
|
||||
Operators needing exact regional billing set per-provider prices on the model
|
||||
rows (or replace the default entries via `AgentNetwork.PricingDefaultsFile`).
|
||||
|
||||
## Cross-references
|
||||
|
||||
|
||||
Reference in New Issue
Block a user