From b27ef7ff76c4bb59129f136c295e043e5aa97cc4 Mon Sep 17 00:00:00 2001 From: braginini Date: Sat, 1 Aug 2026 13:46:59 +0200 Subject: [PATCH] [docs] Update agent-network docs for management-owned pricing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The docs still described the retired proxy-side pricing: pricing.Loader, pricing_path, MiddlewareDataDir, embedded defaults_pricing.yaml, and the symlink-safe Unix loader. Rewrite them for the current design — management synthesizes the whole table and ships it in cost_meter's ConfigJSON, so the proxy carries no price list and has nothing to reload. --- docs/agent-networks/01-end-to-end-flows.md | 2 +- .../modules/21-management-agentnetwork.md | 98 +++++++- .../modules/31-proxy-middleware-builtin.md | 73 ++++-- .../modules/32-proxy-llm-parsers.md | 210 +++++++++++------- .../modules/33-proxy-runtime.md | 5 +- .../modules/50-path-routed-providers.md | 44 ++-- 6 files changed, 306 insertions(+), 126 deletions(-) diff --git a/docs/agent-networks/01-end-to-end-flows.md b/docs/agent-networks/01-end-to-end-flows.md index b8891001b..0de6b4c33 100644 --- a/docs/agent-networks/01-end-to-end-flows.md +++ b/docs/agent-networks/01-end-to-end-flows.md @@ -115,7 +115,7 @@ sequenceDiagram Resp->>Resp: parse usage tokens, completion Note over Resp: capture_completion gates raw
completion capture Resp->>Cost: tokens - Cost->>Cost: lookup pricing.yaml + compute cost + Cost->>Cost: lookup rates from config-delivered
pricing table + compute cost Cost->>Rec: tokens + cost Rec->>MgmtGrpc: RecordLLMUsage(provider, model, prompt_t, completion_t, cost, groups, user) Rec-->>Log: emit access-log entry
(if EnableLogCollection) diff --git a/docs/agent-networks/modules/21-management-agentnetwork.md b/docs/agent-networks/modules/21-management-agentnetwork.md index cc74206e9..f91c369f7 100644 --- a/docs/agent-networks/modules/21-management-agentnetwork.md +++ b/docs/agent-networks/modules/21-management-agentnetwork.md @@ -15,6 +15,10 @@ Inside the package: `manager.go` is the CRUD + permissions-gated facade; `synthe | ---- | ---- | | `agentnetwork/manager.go` | Manager interface + CRUD + permission gates + bootstrap-settings + reconcile trigger | | `agentnetwork/synthesizer.go` | Settings/policy → wire-format synthesis; sole writer of the proxy middleware chain | +| `agentnetwork/synthesizer_pricing.go` | `buildCostMeterConfigJSON` — default table + per-provider prices → `cost_meter` config | +| `agentnetwork/pricing/defaults.go` | Default pricing table derived from the catalog + supplementals; `DefaultTable`, `LookupDefault`, wire `Entry` | +| `agentnetwork/pricing/override.go` | `LoadFile`/`StartReloader` for `AgentNetwork.PricingDefaultsFile` (mtime poll, merge over compiled-in base) | +| `agentnetwork/pricing/{exampleyaml,gen}.go` | Generates `defaults_llm_pricing.example.yaml` from the compiled-in table (golden-tested) | | `agentnetwork/policyselect.go` | Per-request policy attribution + account-budget ceiling (min-wins) | | `agentnetwork/reconcile.go` | Per-account synth diff vs in-memory cache → Create/Update/Delete | | `agentnetwork/catalog/catalog.go` | Static provider catalogue (auth headers, identity-injection shapes) | @@ -48,6 +52,8 @@ flowchart TD I --> J[indexProviderGroups: providerID -> sorted source groups] J --> K[buildRouterConfigJSON drops orphan providers] J --> L[buildIdentityInjectConfigJSON per catalog entry] + J --> K2[buildCostMeterConfigJSON: default table + per-provider prices] + K2 --> P H --> M[mergeGuardrails: union allowlist, OR redact] M --> N[applyAccountCollectionControls account toggle = SOLE capture control] N --> O[marshalGuardrailConfig] @@ -60,6 +66,84 @@ flowchart TD R --> T[accountManager.UpdateAccountPeers — fans synth ACLs into network map] ``` +### LLM pricing (management is the sole authority) + +**The proxy carries no price list.** Management synthesizes the entire pricing +table and ships it inside `cost_meter`'s `ConfigJSON`, so a price change reaches +the proxies as an ordinary mapping push — the chain rebuild installs a fresh +table and there is nothing to reload on the proxy side. + +```mermaid +flowchart TD + A[catalog.All — PricingSurfaces x Models] --> B[buildDefaultTable + supplementalDefaults] + B --> C{AgentNetwork.PricingDefaultsFile} + C -- absent --> D[compiled-in table serves] + C -- loaded --> E[LoadFile: merge file entries WHOLE over compiled base] + E --> F[mergedTable atomic.Pointer] + D --> G[DefaultTable] + F --> G + G --> H[buildCostMeterConfigJSON — pricing.defaults] + I[types.Provider.Models operator prices] --> J[normalizePricingModelID
bedrock ARN/region/version, vertex @version] + J --> K[materializeEntry: default entry as base,
operator input/output verbatim,
cache pointers only when non-nil] + K --> L[pricing.providers keyed by provider record ID] + H --> M[cost_meter ConfigJSON] + L --> M + G --> N[GET /catalog — applyDefaultPricing prefills dashboard rows] + O[StartReloader: mtime poll every ReloadInterval 1m] --> E +``` + +**Two tiers, resolved per request on the proxy** (`synthesizer_pricing.go:22-35`): + +- `pricing.defaults` — surface (`openai`/`anthropic`/`bedrock`) → normalized model + id → rates. The **full** default table ships to every account: it is small + (~10 KB) and it is what keeps gateway-style providers (which enumerate no + models, so they claim every model) priced. +- `pricing.providers` — provider **record** id → normalized model id → rates, + matched against the `llm.resolved_provider_id` the router stamps. Entries are + **fully materialized here**, at synth time: `materializeEntry` starts from the + default entry for that model so cache rates the operator didn't state are + inherited, overlays operator `input`/`output` verbatim (**including an explicit + 0**, which prices a self-hosted or internal endpoint as free rather than + silently reverting to list price), and overlays cache-rate **pointers only when + non-nil** — `nil` means "inherit the default", an explicit `0` means "no + discount, bill this bucket at the input rate". The proxy therefore does two map + lookups and no merging. + +Same orphan rule as the router: a provider no enabled policy authorises is +unreachable, so its prices aren't shipped. Model ids are normalized with the +**same** functions the request parser uses (`NormalizeBedrockModel` / +`NormalizeVertexModel`), which is what makes the per-record lookup key compare +equal to the `llm.model` the proxy meters. Post-normalization duplicates resolve +first-occurrence-wins, matching the routing dedup order. + +**`AgentNetwork.PricingDefaultsFile`** (`config.go:190-207`) lets an operator +replace default rates without a rebuild. Schema is `surface → model → rates` +(`input_per_1k`, `output_per_1k`, and optional `cached_input_per_1k` / +`cache_read_per_1k` / `cache_creation_per_1k`). Semantics: + +- A **relative** path resolves against ``, so a bare filename lands + alongside the store. Empty config probes `/defaults_llm_pricing.yaml`. +- An **explicitly configured** path is *required to load*: a typo or malformed + file fails startup, because the operator believes those rates are live. The + conventional probe is optional — an absent file just serves compiled-in + defaults, and the path stays watched in case it appears later. +- File entries **replace** the compiled-in entry for the same (surface, model) + **whole** — they are not field-merged, so an entry must repeat the cache rates + it wants to keep. Everything the file doesn't mention keeps built-in rates. +- Unknown YAML fields are rejected (`KnownFields(true)`) and every rate must be + finite and non-negative — the same constraints the HTTP API enforces on + operator per-provider prices. +- Reload is an mtime poll (`ReloadInterval`, 1 min) and is **lenient at runtime**: + a parse error keeps the previous table, a deleted file reverts to compiled-in + defaults. A mid-edit save can never take pricing down. + +The live table feeds **both** consumers, which is what keeps them consistent: the +synthesizer (what proxies actually bill with) and `GET /api/agent-network/catalog` +via `applyDefaultPricing` (what the dashboard's model-row prices prefill with). +`defaults_llm_pricing.example.yaml` is generated from the compiled-in table +(`go generate ./management/internals/modules/agentnetwork/pricing`) and +golden-tested, so operators start from a file matching the built-in rates exactly. + ### Budget rule resolution (min-wins, group+user bound) ```mermaid @@ -124,7 +208,7 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest | on_request | 3 | `llm_identity_inject` | `{"providers":[{provider_id, header_pair?, json_metadata?, extra_headers?}]}` | **true** | | on_request | 4 | `llm_guardrail` | `{"provider_allowlists"?: {providerID: []model}, "prompt_capture":{enabled,redact_pii}}` | – | | on_response | 5 | `llm_limit_record` | `{}` (runs LAST at runtime) | – | - | on_response | 6 | `cost_meter` | `{}` | – | + | on_response | 6 | `cost_meter` | `{"pricing":{"defaults":{surface:{model:rates}},"providers"?:{providerRecordID:{model:rates}}}}` — rates are `{input_per_1k, output_per_1k, cached_input_per_1k?, cache_read_per_1k?, cache_creation_per_1k?}` | – | | on_response | 7 | `llm_response_parser` | `{"capture_completion": , "redact_pii"?: true}` | – | - **Synthesized service shape** (`synthesizer.go:739`): `Mode=HTTP`, `Private=true`, `Domain=.`, `AccessGroups=unionSourceGroups(enabledPolicies)`, one `TargetTypeCluster` target with `Host=noop.invalid:443` (router rewrites per request), `Options.{DirectUpstream,AgentNetwork}=true`, `DisableAccessLog=!settings.EnableLogCollection`, `CaptureMax{Req,Resp}Bytes=1<<20`, `CaptureContentTypes=["application/json","text/event-stream"]`. @@ -139,6 +223,12 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest - **Orphan providers (no enabled policy authorises them) NEVER reach the router** (`synthesizer.go:351-357`); skipped from `identity_inject` for symmetry. - **Provider creation refuses empty `api_key`** (`manager.go:175`); **deletion refuses while any policy still references it** (`manager.go:265-273`). - **Session keypair stability across provider edits** (`manager.go:226-228`) — server-managed, copied through every `UpdateProvider`, never API-surfaced. +- **Management is the sole pricing authority.** The proxy has no embedded price list, so an account whose `cost_meter` config carries no `pricing` block bills **nothing** (`cost.skipped=unknown_model`, $0) rather than falling back to stale built-ins. The top-level `pricing` wrapper is also the feature-detection signal in both directions: an old proxy ignores it as an unknown field, and a new proxy reads its absence as "old management". +- **Per-provider prices are materialized at synth time, not merged on the proxy** (`synthesizer_pricing.go:114-131`). A per-record entry is always complete, so the proxy's lookup is per-record-then-defaults with no field-level fallback between tiers. +- **An explicit operator price of `0` prices the model as free** — it must not be treated as "unset" and reverted to list price (`synthesizer_pricing.go:49-54`). Only *cache*-rate fields distinguish unset from zero, via `*float64`. +- **Pricing model ids are normalized with the same functions the request parser uses** (`normalizePricingModelID`). If the two ever diverge, per-record prices silently stop matching and every request falls through to surface defaults. +- **The default table's coverage is structural, not curated.** It is derived from the catalog via each provider's `PricingSurfaces`; `TestDefaultTable_CoversEveryCatalogModel` fails on an unpriced catalog model and `TestDefaultTable_NoConflictingContributions` fails if two providers contribute the same (surface, model) at different rates. +- **A pricing-defaults file failure is fatal only at startup, and only for an explicitly configured path.** Runtime reload failures keep the previous table; a deleted file reverts to compiled-in defaults (`pricing/override.go:62-81, 113-148`). ## Things to scrutinize @@ -176,10 +266,12 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest - **Capture-pointer semantics (restated):** non-agent-network callers see no field → legacy nil-default emit, identical to pre-PR. Agent-network targets always carry an explicit `capture_*` value. - **`TestSynthesizeServices_HappyPath` was updated:** request-parser config moved from `{}` to `{"capture_prompt":false}` (`synthesizer_test.go:174`). External snapshot tests against synth output need updating. - **`MergedGuardrails` retains zeroed `TokenLimits`/`Budget`/`Retention`** even though `Policy.Limits` carries the real values now; `llm_limit_check` is the authoritative enforcement. Comment at `synthesizer.go:940-948` calls this out. +- **`cost_meter`'s `pricing` block is version-skew-safe in both directions.** A proxy predating config-delivered pricing ignores the field as unknown JSON (it previously priced from its own embedded table, so it keeps billing — at its own rates, which is the skew to be aware of during a rolling upgrade). A current proxy paired with old management sees no `pricing` block, logs one warning at chain-build time, and records `cost.skipped=unknown_model` — token counting and cap enforcement are unaffected, only the USD annotation goes to $0. ### Performance - **`SynthesizeServices` runs on every controller tick / mutation reconcile.** Cost: 4 store reads + optional per-provider keypair backfill. Sort + index + merge are O(N log N) / O(P × G); dominant cost is JSON marshalling. No nested loops escape these dimensions. +- **The full default pricing table is marshalled into every account's `cost_meter` config on every synth** (~10 KB serialized). This is a deliberate trade: it keeps gateway-style providers priced for every catalog model, and it is the largest single contributor to the synth JSON. `DefaultTable()` itself is a pointer load (or a `sync.Once`-built map) — the cost is the marshal, not the build. - **`reconcile.diffMappings` is O(N + M)** with N=M=1 per account today — effectively constant. - **`SynthesizeServicesForCluster`** (`synthesizer.go:71`) walks every account on a cluster; per-account failures are **swallowed** (`synthesizer.go:91-93`) so a single misconfigured account doesn't drop the cluster. Runs per proxy reconnect. @@ -188,6 +280,7 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest - **Activity codes:** `AgentNetwork{Provider,Policy,Guardrail,BudgetRule}{Created,Updated,Deleted}`; `AgentNetworkSettingsUpdated` with `log_collection/prompt_collection/redact_pii` payload (`manager.go:567-571`). **No activity code for `SelectPolicyForRequest` denies** — surfaced via proxy access log only (likely intentional given volume). - **Deny codes** namespaced: `llm_policy.{token,budget}_cap_exceeded`, `llm_account.{token,budget}_cap_exceeded` (`policyselect.go:18-26`). - **Reconcile failures are logged at warn and swallowed** (`reconcile.go:42-44`). Persistent synth failures (e.g. unknown catalog id) silently keep the proxy out of sync — consider a manager-level synth-health surface if this becomes a support burden. +- **Pricing-file lifecycle logs at info** (load, reload, revert-to-built-ins) and **at warn** for a runtime reload failure; the mtime check itself is `Debugf`. There is no metric on reload failures, so an operator who breaks the file mid-flight keeps billing at the previous table with only a log line to show it (`pricing/override.go:113-148`). ## Test coverage @@ -198,6 +291,9 @@ At request time the path is independent: the proxy calls `SelectPolicyForRequest | `synthesizer_guardrail_realstore_test.go` | `PromptCaptureAccountIsSoleControl`; `PromptCaptureFlowsWhenAccountOptsIn`; `AccountRedactWithoutGuardrailRedact`; `NoGuardrail_CaptureOff`. | | `synthesizer_log_collection_realstore_test.go` | `LogCollection{Off_SuppressesAccessLog,On_PermitsAccessLog}` — verifies `DisableAccessLog` propagation through `ToProtoMapping`. | | `synthesizer_parser_redact_realstore_test.go` | **Capture-pointer regression suite:** `ParserConfigsCarryRedactPii`; `ParserConfigsSuppressCaptureWhenLogCollectionOnly` (log=on/prompt=off ⇒ both capture flags false); `ParserConfigsOmitRedactPiiWhenOff`. | +| `synthesizer_pricing_test.go` | `BuildCostMeterConfig_{BedrockModelNormalization,CacheRateNilVsZero,OrphanAndGatewayProviders}` — the per-record tier's three load-bearing rules: keys normalized like the parser's, `nil` cache pointer inherits vs explicit `0` bills at input rate, and orphan / gateway (empty `Models`) providers ship no per-record entry. | +| `pricing/defaults_test.go` | `DefaultTable_{CoversEveryCatalogModel,NoConflictingContributions,AllRatesFiniteNonNegative,PinnedRates}`; `LookupDefault_SurfaceOrder`. Catalog-derived coverage + rate sanity are structural, not curated. | +| `pricing/override_test.go` | `LoadFile_{MergesOverCompiledDefaults,MissingPath,RejectsInvalid}`; `Reload_LifeCycle` (mtime detect, parse error keeps previous, delete reverts to built-ins); `ExampleYAML_InSyncWithBuiltins` golden. | | `policyselect_test.go` | Mock-store: `NoApplicablePolicies`; `AllowWithLowestGroupAttribution`; `LargerPoolWinsAcrossUsageLevels`; `StaysOnLargerPoolAfterPartialDrain`; `FallsThroughToSmallerPoolWhenLargerExhausted`; `TiebreakBy{LargerGroupPool,CreatedAt}`; `DeniesWhenAllExhausted`; `UncappedPolicyAlwaysWinsAgainstCapped`; `DisabledPolicyIgnored`; `StoreErrorPropagates`; `RejectsEmptyAccount`; `SharesGroupCounterAcrossPolicies`; `AntiFallThroughOnLowestGroup`; `BudgetOnlyExhaustionDenies`; `BudgetTighterThanTokenWins`. | | `policyselect_realstore_test.go` | Real-sqlite regression guard: `NoApplicablePolicies`; `AllowAndLowestGroupAttribution`; `LargerPoolWins_FallsThroughWhenExhausted`; `BudgetCapDenies`; `GroupCounterSharedAcrossPolicies`; `DisabledPolicyIgnored`. | | `policyselect_account_realstore_test.go` | Account budget rules: `AccountCeilingBindsEvenWithUncappedPolicy` (min-wins); `AccountGroupCeiling`; `AccountTargetUsersBindsOnlyThatUser`; `AccountRuleRecordsToOwnWindow`. | diff --git a/docs/agent-networks/modules/31-proxy-middleware-builtin.md b/docs/agent-networks/modules/31-proxy-middleware-builtin.md index efe1bc4ce..ad56feb77 100644 --- a/docs/agent-networks/modules/31-proxy-middleware-builtin.md +++ b/docs/agent-networks/modules/31-proxy-middleware-builtin.md @@ -5,7 +5,7 @@ LLM request. The two highest-blast-radius areas are the **capture-pointer semantics** and the **limit_check ⇒ limit_record** record-once invariant. Sibling module: [32-proxy-llm-parsers.md](./32-proxy-llm-parsers.md) — the SDK -adapters + pricing catalog this chain delegates to. +adapters + pricing table and cost formula this chain delegates to. --- @@ -34,7 +34,7 @@ rewrites. | `llm_identity_inject` | OnRequest | `llm.{resolved_provider_id,authorising_groups}`, `Input.{UserEmail,UserID,UserGroups,UserGroupNames}` | none | header strip/inject + optional body rewrite | | `llm_guardrail` | OnRequest | `llm.{model,request_prompt_raw}` | `llm_policy.{decision,reason}`, `llm.request_prompt` | none (model allowlist deny) | | `llm_response_parser` | OnResponse | `llm.provider`, `Input.{RespHeaders,RespBody,Status}` | `llm.{input,output,total,cached_input,cache_creation}_tokens`, `llm.response_completion` | none | -| `cost_meter` | OnResponse | `llm.{provider,model}`, token buckets | `cost.usd_total` or `cost.skipped` | pricing lookup | +| `cost_meter` | OnResponse | `llm.{provider,model,resolved_provider_id}`, token buckets | `cost.usd_{input,cached_input,cache_creation,output,total,cache}` or `cost.skipped` | none (in-memory pricing lookup) | | `llm_limit_record` | OnResponse | `llm.{attribution_group_id,attribution_window_seconds,input_tokens,output_tokens}`, `cost.usd_total` | none | gRPC `RecordLLMUsage` | [all_test.go:26–40](../../../proxy/internal/middleware/builtin/all_test.go) @@ -44,7 +44,7 @@ locks the ID set; adding or removing one is a conscious extension. | File | LOC | Notes | |---|---:|---| -| `builtin.go` | 86 | Registry + `FactoryContext` (ctx, data dir, meter, logger, mgmt client) | +| `builtin.go` | 90 | Registry + `FactoryContext` (ctx, meter, logger, mgmt client) | | `all_test.go` | 41 | Locks the 8-ID registry surface | | `agentnetwork_chain_integration_test.go` | 319 | Live sqlite + real gRPC bufconn; gate→recorder wire path | | `llm_request_parser/*` | 162 / 66 / 356 | Provider detection, body parse, prompt extraction with capture-pointer gating | @@ -53,7 +53,7 @@ locks the ID set; adding or removing one is a conscious extension. | `llm_identity_inject/*` | 440 / 108 / 666 | HeaderPair (LiteLLM) + JSONMetadata (Portkey) + ExtraHeaders | | `llm_guardrail/*` | 176 / 82 / 75 / 219 / 217 | Model allowlist + optional prompt capture with PII redaction | | `llm_response_parser/*` | 258 / 222 / 43 / 433 / 169 / 111 | Buffered + SSE accumulation; AWS event-stream accumulator (`streaming_bedrock.go`) for Bedrock; capture-pointer gates completion emit | -| `cost_meter/*` | 181 / 84 / 439 | Token → USD via `proxy/internal/llm/pricing` | +| `cost_meter/*` | 236 / 98 / 586 | Token → USD via `proxy/internal/llm/pricing`; both pricing tiers arrive in the middleware config | | `llm_limit_record/*` | 144 / 35 / 191 | Post-flight `RecordLLMUsage` (5s, debug-on-error) | ## Per-middleware @@ -168,12 +168,46 @@ token schema. ### cost_meter -Reads `llm.provider` + `llm.model` + token buckets, looks up per-1k rate via -`pricing.Loader`, emits `cost.usd_total` or a closed-set `cost.skipped` -reason (`missing_provider/model/tokens`, `unparseable_tokens`, `zero_tokens`, -`unknown_model`). Loader's hot-reload goroutine is bound to proxy-lifetime -context via `startReloader`. **Key invariant:** provider-shape switch lives -in `pricing.Table.Cost` (sibling doc) — `cost_meter` stays provider-agnostic. +Reads `llm.provider` + `llm.model` + token buckets, looks up the per-1k rates, +and emits the full `cost.usd_*` breakdown (four per-bucket values plus the +`_total` and `_cache` aggregates) or a closed-set `cost.skipped` reason +(`missing_provider/model/tokens`, `unparseable_tokens`, `zero_tokens`, +`unknown_model`). + +**Management owns pricing.** The proxy carries no embedded price list: the whole +table arrives in this middleware's `ConfigJSON` as +`{pricing: {defaults, providers}}`, synthesized by management from the catalog +plus the operator's stored per-provider prices +([factory.go:13–34](../../../proxy/internal/middleware/builtin/cost_meter/factory.go)). +Both tiers are validated by `pricing.NewTable` / `pricing.NewEntries` at +construction, so a non-finite or negative rate fails the chain build. A price +change is an ordinary mapping push — the chain rebuild yields a fresh instance +over a fresh immutable table, so there is no data dir, no pricing file, no +reload goroutine, and nothing to invalidate. + +**Two-tier lookup** +([middleware.go:165–183](../../../proxy/internal/middleware/builtin/cost_meter/middleware.go)): + +1. **Per-provider-record** — the operator's stored price for the route that + actually served the request, keyed by the `llm.resolved_provider_id` that + `llm_router` stamped on the allow path, then by normalized model id. Entries + arrive fully materialized (management folds default cache rates in at synth + time), so there is no merging here. Absent metadata — no router in the chain + — skips this tier. +2. **Surface defaults** — the catalog-derived table keyed by `llm.provider` + (`openai`/`anthropic`/`bedrock`). This is also what prices gateway-style + providers, which enumerate no models and therefore get no per-record entry. + +**Backward compatibility:** a config with no `pricing` block means management +predates config-delivered pricing. The factory logs one warning at build time +and the instance records `cost.skipped=unknown_model` ($0) for every request +rather than falling back to a stale built-in price list +([factory.go:55–60](../../../proxy/internal/middleware/builtin/cost_meter/factory.go)). + +**Key invariant:** the provider-shape switch lives in `pricing.EntryCosts` +(sibling doc) and is selected by the **surface**, not by which tier the entry +came from — `cost_meter` stays provider-agnostic, and a per-record override on +an Anthropic route still bills its cache buckets additively. ### llm_limit_record @@ -246,12 +280,14 @@ no mocks. Tests: `TestChain_AllowPath_StampsAttributionAndRecordsCounter` | `llm_identity_inject` | `{providers: [{provider_id, header_pair?|json_metadata?, extra_headers?}]}` | | `llm_guardrail` | `{provider_allowlists: {providerID: []string}, prompt_capture: {enabled, redact_pii}}` — allowlist keyed by resolved provider id; a provider absent from the map is unrestricted (fail-closed backstop; authoritative per-policy/group check is management's `CheckLLMPolicyLimits`) | | `llm_response_parser` | `{redact_pii?, capture_completion?: *bool}` | -| `cost_meter` | `{pricing_path?}` (basename inside data-dir; defaults `pricing.yaml`) | +| `cost_meter` | `{pricing: {defaults: {surface: {model: rates}}, providers: {providerRecordID: {model: rates}}}}` — rates are `{input_per_1k, output_per_1k, cached_input_per_1k?, cache_read_per_1k?, cache_creation_per_1k?}`. A missing `pricing` key means "management predates config-delivered pricing": every request records `cost.skipped=unknown_model` | | `llm_limit_record` | `{}` — same pattern as `llm_limit_check` | All factories accept empty / null / `{}` / whitespace as zero-value config; only structurally invalid JSON is rejected so misconfig surfaces at chain -build time. +build time. `cost_meter` adds a semantic check on top of that: a `pricing` +block carrying a negative or non-finite rate fails the build too, rather than +mispricing live traffic. ## Invariants @@ -320,10 +356,11 @@ non-object `metadata` field — header path still attributes, but body-level tag-budget enforcement doesn't run for that request. -**Concurrency.** `cost_meter` shares a `pricing.Loader` via -`atomic.Pointer[Table]`; readers always see a consistent table. Every -middleware is a stateless value receiver. Integration test uses real bufconn -gRPC — race detector is the meaningful bar. +**Concurrency.** `cost_meter`'s two pricing tables are built once from the +middleware config and never mutated, so the lookup path needs no lock or atomic +swap — a price change replaces the whole instance. Every middleware is +otherwise a stateless value receiver. Integration test uses real bufconn gRPC — +race detector is the meaningful bar. **Perf.** Hot path is `lookupKV` linear scan over <10 KVs; `cost_meter.Cost` is O(1); SSE accumulation is single-pass. No map allocation per call. @@ -349,13 +386,13 @@ counter accuracy. | `llm_guardrail/redact_test.go` | 15 | Email, SSN, phone (E.164 + NA), bearer, IPv4; fixture-driven | | `llm_response_parser/middleware_test.go` | 18 | Buffered OAI+Anthro, capture-pointer, redact, truncation | | `llm_response_parser/streaming_test.go` | 7 | OAI usage frame, Anthro message_delta, truncated body best-effort | -| `cost_meter/middleware_test.go` | 17 | Each skip reason, provider-shape, pricing loader integration | +| `cost_meter/middleware_test.go` | 22 | Each skip reason, provider-shape formulas, config-delivered defaults, per-record-beats-defaults + miss-falls-back, per-record uses surface formula, nil-pricing skips everything, invalid-rate rejection | | `llm_limit_record/middleware_test.go` | 7 | Skip-on-no-signal, skip-on-missing-attribution, RPC failure swallowed | ## Cross-references - Sibling: [32-proxy-llm-parsers.md](./32-proxy-llm-parsers.md) — SDK adapters - + SSE framer + pricing loader. + + SSE framer + pricing table and cost formula. - Path-routed providers (Vertex AI + Bedrock), `keyfile::` credential, GCP token minting, `/bedrock` prefix: [50-path-routed-providers.md](./50-path-routed-providers.md). diff --git a/docs/agent-networks/modules/32-proxy-llm-parsers.md b/docs/agent-networks/modules/32-proxy-llm-parsers.md index 0376bc988..52faeaac1 100644 --- a/docs/agent-networks/modules/32-proxy-llm-parsers.md +++ b/docs/agent-networks/modules/32-proxy-llm-parsers.md @@ -9,7 +9,7 @@ pricing table's per-provider cost formula is the highest-leverage place a small bug would silently mis-bill operators. Sibling module: [31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md) -— the 8 middlewares that consume this package's parsers + pricing loader. +— the 8 middlewares that consume this package's parsers + pricing table. --- @@ -24,8 +24,9 @@ proxy-framework dependencies: - `openai.go` / `anthropic.go` / `bedrock.go` — per-provider `Parser` impls. - `sse.go` — SSE scanner (`Scanner`, `Event`, `NewScanner`). - `errors.go` — sentinels callers branch on with `errors.Is`. -- `pricing/` — embedded-default + hot-reload override table with - symlink-safe Unix loader (build-tagged stub elsewhere). +- `pricing/` — immutable pricing table + the per-surface cost formula. The + rates themselves come from management inside `cost_meter`'s middleware + config; this package holds no price list and reads no files. - `fixtures/` — captured request/response/stream bodies the tests replay. The package carries zero proxy-framework dependencies so the same parsers can @@ -47,12 +48,9 @@ be reused later by a WASM adapter | `sse_test.go` | 175 | 12 tests; fixture replay + multiline + size limits | | `parser_test.go` | 53 | `Parsers()`, `DetectParser`, provider enum values | | `errors.go` | 31 | 6 sentinels: `Err{Unknown,Unsupported}Provider/Model`, `Err{NotLLM,Malformed}Response`, `ErrStreamingUnsupported`, `ErrMalformedRequest` | -| `pricing/pricing.go` | 421 | `Loader`, `Table`, `Entry`; embedded defaults + atomic swap + mtime reload | -| `pricing/pricing_unix.go` | 69 | `O_NOFOLLOW` + fstat-from-FD + 1 MiB cap | -| `pricing/pricing_other.go` | 21 | Stub returning "not supported on this platform" | -| `pricing/pricing_test.go` | 432 | 21 tests — symlink rejection, reload race, path traversal, oversize | -| `pricing/defaults_pricing.yaml` | 85 | go:embed source of truth | -| `fixtures/*` | 21–59 | OAI chat/responses/stream + Anthro messages/stream + pricing starter | +| `pricing/pricing.go` | 234 | `Table`, `Entry`, `EntryJSON`, `Costs`; `NewTable`/`NewEntries` validation + `EntryCosts` formula. No I/O, no reload, no embedded rates | +| `pricing/pricing_test.go` | 177 | 10 tests — provider-shape formulas, cached clamp, rate fallback, nil-safety, rate validation | +| `fixtures/*` | 21–59 | OAI chat/responses/stream + Anthro messages/stream | ## Request body → parser dispatch @@ -188,9 +186,11 @@ response leg, covering both Bedrock body shapes: `totalTokens`). `firstNonZero` folds the two naming conventions into one `Usage`; when Converse omits `totalTokens` the parser sums the buckets. -`ProviderName()` returns `"bedrock"` — its own `defaults_pricing.yaml` block, -keyed by the **normalised** model id (region prefix + version suffix stripped by -the request parser). `ParseResponse` returns `ErrStreamingUnsupported` for an +`ProviderName()` returns `"bedrock"` — its own pricing surface in the table +management ships, keyed by the **normalised** model id (region prefix + version +suffix stripped by the request parser; management normalises its keys the same +way at synth time so the two compare equal). `ParseResponse` returns +`ErrStreamingUnsupported` for an AWS binary event-stream content-type (`application/vnd.amazon.eventstream`, `isAWSEventStream`) so the caller routes to the streaming accumulator instead. @@ -205,11 +205,34 @@ response body. Streaming accumulators live in the middleware package ([llm_response_parser/streaming.go](../../../proxy/internal/middleware/builtin/llm_response_parser/streaming.go)) but use `llm.NewScanner` so the framing contract stays here. -### Pricing catalog +### Pricing table -`Table.Cost` -([pricing.go:129–174](../../../proxy/internal/llm/pricing/pricing.go)) -is the cost formula — most security-relevant math in this module: +**Management is the sole pricing authority.** The proxy carries no embedded +price list and reads no pricing file: the whole table arrives inside +`cost_meter`'s `ConfigJSON` on the ordinary mapping push, and a price change +is just another push — the chain rebuild constructs a fresh `Table`, so there +is nothing to reload +([pricing.go:1–7](../../../proxy/internal/llm/pricing/pricing.go)). The +management side of the contract (catalog defaults, the operator's stored +per-provider prices, and `AgentNetwork.PricingDefaultsFile`) is covered in the +management-side module guide; `cost_meter`'s wire shape is in +[31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md). + +`EntryJSON` +([pricing.go:36–45](../../../proxy/internal/llm/pricing/pricing.go)) is the +management→proxy contract — five USD-per-1k rates under `input_per_1k`, +`output_per_1k`, `cached_input_per_1k`, `cache_read_per_1k`, +`cache_creation_per_1k`. Management's `pricing.Entry` marshals the identical +names, and `EntryJSON`/`Entry` are field-identical so `NewEntries` converts by +direct struct conversion rather than field-by-field copying (a new rate can't +be silently dropped in transit). + +`EntryCosts` +([pricing.go:183–234](../../../proxy/internal/llm/pricing/pricing.go)) +is the cost formula — most security-relevant math in this module. The +**surface** (the `llm.provider` value the request parser stamped) selects the +formula, never the tier the entry came from: a per-provider-record override on +an Anthropic route still bills its cache buckets additively. | Provider | Formula | |---|---| @@ -218,7 +241,7 @@ is the cost formula — most security-relevant math in this module: | default | `inTokens × InputPer1K + outTokens × OutputPer1K` | `bedrock` shares the Anthropic additive-cache formula -([pricing.go:172-174](../../../proxy/internal/llm/pricing/pricing.go)): +([pricing.go:214–229](../../../proxy/internal/llm/pricing/pricing.go)): Anthropic-on-Bedrock reports the same additive cache buckets, while non-Anthropic Bedrock models (Nova, Llama) simply report zero in those buckets so cost reduces to `input + output`. @@ -226,15 +249,12 @@ to `input + output`. Each per-bucket rate falls back to `InputPer1K` when zero — operators opt in to discounts by setting the field. -`Loader` -([pricing.go:212–268](../../../proxy/internal/llm/pricing/pricing.go)) -overlays an optional `pricing.yaml` from data-dir on top of the go:embed -defaults. Atomic pointer swap means readers never observe a partial update. -The mtime-poll reloader (30s default cadence) keeps the previous table on -parse failure so cost annotation never goes blank during a botched edit. - -`defaults_pricing.yaml` is the source of truth for built-in pricing. -Operator overrides only carry the entries they want to change. +`Costs` +([pricing.go:143–163](../../../proxy/internal/llm/pricing/pricing.go)) is the +per-request split. The four per-bucket fields are the base; `TotalUSD` and +`CacheUSD` are **derived** in `newCosts` so the aggregates can never drift from +the breakdown. `InputUSD` is always the non-cached input bucket on both +provider shapes, so input and cached-input never double-count. ## Public contracts @@ -264,29 +284,38 @@ Order matters: `DetectFromURL` ties resolve by registration order. `ProviderBedrock = 3`. Numeric values are persisted in nothing today but treat them as wire-stable — new providers must take fresh numbers. -**`Pricing` lookup** -([pricing.go:129](../../../proxy/internal/llm/pricing/pricing.go)): +**`Pricing` construction + lookup** +([pricing.go:60–130](../../../proxy/internal/llm/pricing/pricing.go)): ```go +func NewEntries(raw map[string]map[string]EntryJSON) (map[string]map[string]Entry, error) +func NewTable(raw map[string]map[string]EntryJSON) (*Table, error) + +func (t *Table) Lookup(provider, model string) (Entry, bool) func (t *Table) Cost(provider, model string, inTokens, outTokens, cachedInput, cacheCreation int64) (float64, bool) +func (t *Table) Costs(provider, model string, inTokens, outTokens, cachedInput, cacheCreation int64) (Costs, bool) +func EntryCosts(entry Entry, surface string, inTokens, outTokens, cachedInput, cacheCreation int64) Costs ``` -Nil-safe: `t.Cost` on a nil receiver returns `(0, false)` -([pricing.go:130–132](../../../proxy/internal/llm/pricing/pricing.go)). -`ok=false` means provider or model is absent from the loaded table; the caller -emits `cost.skipped=unknown_model`. +`NewTable` is the surface-keyed defaults table; `NewEntries` returns the raw +two-level map `cost_meter` uses for the per-provider-record tier (it looks up an +`Entry` directly and calls `EntryCosts`, so it needs no `Table` wrapper). Both +reject any non-finite or negative rate, so a corrupt config fails the chain +build rather than mispricing silently. Nil input yields an empty, +never-matching table. + +Nil-safe: `t.Cost`/`t.Lookup` on a nil receiver returns `ok=false` +([pricing.go:96–99](../../../proxy/internal/llm/pricing/pricing.go)). +`ok=false` means the surface or model is absent from the table management sent; +the caller emits `cost.skipped=unknown_model`. ## Invariants -1. **Cross-platform pricing build.** `pricing_unix.go` carries the only - functional `loadPricing` (uses `syscall.O_NOFOLLOW` and `f.Stat()` on an - open descriptor — both Unix-only). `pricing_other.go` is a build-tag - fallback that returns `"not supported on this platform"` - ([pricing_other.go:14–16](../../../proxy/internal/llm/pricing/pricing_other.go)). - The proxy is Linux-only in production today; a Windows port needs an - equivalent path-as-handle implementation. Reviewers building on Windows - should expect this surface to return an error at startup if an override - file is configured. +1. **The pricing package is pure and platform-independent.** No file I/O, no + `//go:embed`, no goroutines, no build tags — the rates arrive as config, so + there is nothing platform-specific left to port. Anything reintroducing a + read-from-disk path here re-splits pricing authority between management and + the proxy, which is exactly what this design removed. 2. **SSE scanner handles partial chunks.** A buffered prefix that doesn't end in `\n\n` still yields its accumulated event before `io.EOF` @@ -298,38 +327,45 @@ emits `cost.skipped=unknown_model`. usage rather than aborting ([streaming.go:68–73, 144–150](../../../proxy/internal/middleware/builtin/llm_response_parser/streaming.go)). -3. **`defaults_pricing.yaml` is the source of truth.** Compiled into the - binary via `//go:embed` - ([pricing.go:29–30](../../../proxy/internal/llm/pricing/pricing.go)). - `DefaultTable()` parses once and panics on parse failure - ([pricing.go:42–49](../../../proxy/internal/llm/pricing/pricing.go)) - — by design: a broken embedded YAML must not ship to production. +3. **Management is the only source of rates.** `Table` has no constructor that + invents prices: the only way in is `NewTable`/`NewEntries` over the wire map + management sent. A missing or empty `pricing` block therefore means *no + prices at all* (`cost_meter` records `cost.skipped=unknown_model`, $0) — + never a stale built-in fallback that would silently bill list price. -4. **Loader path validation.** `resolveMiddlewareDataPath` - ([pricing.go:370–394](../../../proxy/internal/llm/pricing/pricing.go)) - rejects absolute paths, traversal segments, and basenames that fail - `basenameRegex = ^[a-zA-Z0-9._-]+$`. The resolved path must remain - inside `baseDir` even after `filepath.Clean`. Tests: - `TestNewLoader_PathValidation`, `TestNewLoader_PathValidation_Extended`, - `TestNewLoader_SymlinkOutsideBaseDirRejected`, `TestNewLoader_SymlinkRejected`. +4. **Tables are immutable once built.** `Table.entries` is written only in + `NewEntries` and never mutated afterwards, and `cost_meter`'s `perRecord` + map is likewise build-time-only + ([pricing.go:47–52](../../../proxy/internal/llm/pricing/pricing.go)). This + is what makes the no-reload design safe: a price change arrives as a mapping + push that builds a new middleware instance over a new table, so concurrent + readers can't observe a half-updated price list and no atomic swap or lock + is needed on the hot path. -5. **Unix loader symlink safety.** `O_NOFOLLOW` on open, `f.Stat()` on the - open descriptor (never re-stat by path), `info.Mode().IsRegular()` check, - `io.LimitReader(f, maxPricingBytes+1)` with a final size assertion - ([pricing_unix.go:25–57](../../../proxy/internal/llm/pricing/pricing_unix.go)). - A mid-read symlink swap is detected because the fstat is on the original - fd. Test: `TestNewLoader_RejectsOversizedFile_FixesM4`. +5. **Rate validation happens at chain-build time, not per request.** + `NewEntries` rejects negative, NaN, and ±Inf rates field by field + ([pricing.go:60–83](../../../proxy/internal/llm/pricing/pricing.go)), naming + the offending surface/model/field in the error. Management enforces the same + constraints at its API boundary and in its YAML parser, so this is + defense-in-depth — but it means a corrupt push fails loudly at build instead + of producing negative costs on live traffic. Test: + `TestNewTable_ValidatesRates`. -6. **`yaml.NewDecoder(...).KnownFields(true)`** - ([pricing.go:397–398](../../../proxy/internal/llm/pricing/pricing.go)) - rejects YAML files that carry fields not in the schema. A typo in an - operator override file fails loud instead of silently zeroing rates. +6. **New rates must be added to `Entry`, `EntryJSON`, *and* management's + `pricing.Entry` together.** `NewEntries` converts by direct struct + conversion `Entry(e)` + ([pricing.go:76–78](../../../proxy/internal/llm/pricing/pricing.go)), which + only compiles while the two structs stay field-identical — so the proxy half + is compiler-enforced. The management half is not: a rate added there but not + here unmarshals into nothing and prices that bucket at `InputPer1K`. ## Things to scrutinise -**Correctness.** Verify OpenAI cached-prompt clamp at -[pricing.go:147–149](../../../proxy/internal/llm/pricing/pricing.go) -short-circuits before subtraction. `Anthropic.TotalTokens` sums all four +**Correctness.** Verify the OpenAI cached-prompt clamp at +[pricing.go:203–206](../../../proxy/internal/llm/pricing/pricing.go) +short-circuits before subtraction. Negative token counts are clamped to zero up +front ([pricing.go:186–197](../../../proxy/internal/llm/pricing/pricing.go)) so +no formula can yield a negative cost. `Anthropic.TotalTokens` sums all four buckets (in + out + cache_read + cache_creation) — downstream dashboards need to know this differs from `input + output`. `OpenAIParser.ExtractPrompt` falls through `messages → input → prompt`; a @@ -338,22 +374,27 @@ noting). **Security.** `Scanner.maxLine = 1 MiB`; a 2 MiB single-line `data:` event errors from `Scanner.Next` and both accumulators stop with partial usage. -Pricing file 1 MiB cap is orders of magnitude larger than realistic. Confirm -new schema additions are mirrored in both `pricingFile` and `Entry`; -`KnownFields(true)` will reject silently-typo'd operator overrides -otherwise. +Pricing is no longer file-backed, so the loader's path-traversal / symlink / +oversize surface is gone entirely — the config channel (an authenticated +mapping push from management) is now the only way rates enter the proxy, and +`NewEntries` is the validation boundary on it. A new rate added to management's +`pricing.Entry` but not to `EntryJSON` here is the remaining silent-mispricing +path (see invariant 6). -**Concurrency.** `Loader.table` is `atomic.Pointer[Table]`; readers never -block or see a torn table. `Loader.Reload` is one goroutine, cancelled via -context (`TestLoader_ReloadBackgroundLoopCancellation`). `DefaultTable()` -uses `sync.Once`. Per-call `Scanner` instances mean no shared state across -concurrent response-parser calls. +**Concurrency.** Nothing in this package is shared mutable state: tables are +built once and never written again, so `cost_meter`'s hot path is lock-free by +construction rather than by atomic swap. Per-call `Scanner` instances mean no +shared state across concurrent response-parser calls. -**Perf.** `Table.Cost` is two map lookups + multiplications, O(1). -`Scanner.Next` is one `ReadString('\n')` per line. Pricing reload poll 30s. +**Perf.** `Table.Cost` is two map lookups + multiplications, O(1); the +per-provider-record tier adds at most one more lookup. `Scanner.Next` is one +`ReadString('\n')` per line. No background goroutines and no per-request +allocation of pricing state. -**Observability.** Reload failures count via `metric.Int64Counter` keyed -`plugin`; warning log rate-limited at 5 min so a broken file doesn't flood. +**Observability.** A config carrying no `pricing` block logs one warning at +chain-build time (`cost_meter` factory) and then records +`cost.skipped=unknown_model` per request, so an old-management deployment is +visible in both logs and the access log rather than quietly reporting $0. Parser errors return sentinels — middleware uses `errors.Is` to map to the right `cost.skipped` reason. @@ -365,7 +406,7 @@ right `cost.skipped` reason. | `openai_test.go` | 11 | Chat Completions + Responses API + legacy `prompt`; cached-tokens subset for both naming conventions; fixture replays | | `anthropic_test.go` | 7 | Messages + legacy `/v1/complete`; streaming REJECTED on `ParseResponse` (must use scanner); fixture replays | | `sse_test.go` | 12 | Fixture replay both providers; multiline `data:`; CRLF; comment skip; trailing-event-without-blank-line; oversize rejection | -| `pricing/pricing_test.go` | 21 | Provider-shape switch; cached-rate fallback; cached-clamp; symlink rejection (target outside basedir + symlink to file); path validation matrix; oversize rejection; reload-keeps-previous-on-parse-error; mtime change detection; goroutine cancellation | +| `pricing/pricing_test.go` | 10 | Provider-shape switch (surface selects the formula); cached-rate + cache-read/creation fallback to `InputPer1K`; cached-clamp; negative-token clamp; nil-receiver safety; rate validation (negative / NaN / Inf rejected); nil + empty table | **Fixtures** ([proxy/internal/llm/fixtures/](../../../proxy/internal/llm/fixtures/)): `openai_chat_completion.json` (chat.completions with usage), @@ -373,14 +414,15 @@ right `cost.skipped` reason. `openai_stream.txt` (3 deltas + usage + `[DONE]`), `anthropic_messages.json` (Messages API non-streaming), `anthropic_stream.txt` (full 7-event sequence: message_start → -content_block_{start,delta×2,stop} → message_delta (usage) → message_stop), -`pricing.yaml` (realistic-pricing starter for operator overrides). +content_block_{start,delta×2,stop} → message_delta (usage) → message_stop). +No pricing fixture: the table is config-delivered, so pricing tests construct +it in-process from a wire-shape map. ## Cross-references - Sibling: [31-proxy-middleware-builtin.md](./31-proxy-middleware-builtin.md) — the chain that calls `llm.Parsers()`, `llm.ParserByName`, - `llm.NewScanner`, `pricing.NewLoader`. + `llm.NewScanner`, `pricing.NewTable` / `pricing.NewEntries`. - Path-routed providers (Vertex AI + Bedrock), credential syntax, and the Bedrock AWS event-stream accumulator: [50-path-routed-providers.md](./50-path-routed-providers.md). diff --git a/docs/agent-networks/modules/33-proxy-runtime.md b/docs/agent-networks/modules/33-proxy-runtime.md index f553473f8..54046b614 100644 --- a/docs/agent-networks/modules/33-proxy-runtime.md +++ b/docs/agent-networks/modules/33-proxy-runtime.md @@ -1,7 +1,7 @@ # proxy/runtime — translate + serve + log > **Risk level:** High — every config push from management is translated here, and the chain runs on every HTTP request to a synth target. -> **Backward-compat impact:** Additive at the wire (`PathTargetOptions.middlewares`, `agent_network`, `disable_access_log`, capture caps) and on the proxy `Server` struct (`MiddlewareDataDir`, `MiddlewareCaptureBudgetBytes`). Non-agent-network targets stay on the no-middleware fast path. +> **Backward-compat impact:** Additive at the wire (`PathTargetOptions.middlewares`, `agent_network`, `disable_access_log`, capture caps) and on the proxy `Server` struct (`MiddlewareCaptureBudgetBytes`). Non-agent-network targets stay on the no-middleware fast path. Middleware config is entirely wire-delivered — no proxy-side data dir is involved, including for LLM pricing, which management ships inside `cost_meter`'s config. ## Module boundary @@ -114,8 +114,7 @@ At **request time** the access-log middleware stamps `CapturedData`; the auth ch ## Public contracts touched -- `proxy.Server.MiddlewareDataDir` (string) — base dir for file-backed middleware config (server.go:238-241). -- `proxy.Server.MiddlewareCaptureBudgetBytes` (int64) — process-wide capture cap; defaults to 256 MiB (server.go:248-250). +- `proxy.Server.MiddlewareCaptureBudgetBytes` (int64) — process-wide capture cap; defaults to 256 MiB (server.go:249-253). There is no `MiddlewareDataDir`: no built-in middleware reads config from disk, so `builtin.FactoryContext` carries only the proxy-lifetime context, meter, logger, and management client. - `proxy/internal/proxy.WithMiddlewareManager(*middleware.Manager) Option` — new option on `NewReverseProxy`; nil keeps the fast path (reverseproxy.go:48-56). - `proxy/internal/proxy.PathTarget` adds `Middlewares`, `CaptureConfig`, `AgentNetwork`, `DisableAccessLog` (servicemapping.go:27-51), all zero-default. - `proxy/internal/proxy.CapturedData` adds `agentNetwork`, `suppressAccessLog`, `userGroupNames` behind `sync.RWMutex`; slices deep-copied (context.go:47-66, 183-258). diff --git a/docs/agent-networks/modules/50-path-routed-providers.md b/docs/agent-networks/modules/50-path-routed-providers.md index b7cda3a97..08c976c5f 100644 --- a/docs/agent-networks/modules/50-path-routed-providers.md +++ b/docs/agent-networks/modules/50-path-routed-providers.md @@ -87,9 +87,9 @@ strips the `@version` suffix from the model, and maps the publisher to a parser surface via `vertexPublisherVendor`: - `anthropic` → `llm.provider="anthropic"` → metered through the Anthropic - parser, priced under the **`anthropic`** block in `defaults_pricing.yaml` - (the parser emits the standard Anthropic provider label, so Vertex Claude - reuses first-party Anthropic prices). + parser, priced under the **`anthropic`** surface of the pricing table + management ships (the parser emits the standard Anthropic provider label, so + Vertex Claude reuses first-party Anthropic prices). - `openai` → `llm.provider="openai"` (reserved; not in the catalog lineup today). - anything else (notably `google` / Gemini) → empty vendor → **no parser**. @@ -104,8 +104,9 @@ is omitted from the catalog. > Caveat: cross-region inference profiles in `eu` / `apac` carry a ~10% price > premium that the base per-token rates do **not** model — cost annotations for -> those regions read low. Operators who need exact regional billing override -> the affected entries in `pricing.yaml`. +> those regions read low. Operators who need exact regional billing set the +> affected models' prices on the provider record, or replace the default entries +> via management's `AgentNetwork.PricingDefaultsFile`. ## AWS Bedrock (`bedrock_api`) @@ -211,15 +212,19 @@ so a model-listing call can't be rewritten onto an upstream that would 404 it. ## Catalog ↔ pricing cross-check Catalog prices and context windows are cross-checked against LiteLLM's -`model_prices_and_context_window.json`. The proxy's embedded -`defaults_pricing.yaml` covers **every metered first-party model** the catalog -enumerates — guarded by -`TestDefaultTable_FirstPartyModelCoverage` -([pricing/defaults_coverage_test.go](../../../proxy/internal/llm/pricing/defaults_coverage_test.go)), -which fails if a catalog model has no embedded price. Bedrock entries are keyed -by the **normalised** id the request parser emits (region prefix + version -suffix stripped). Vertex Claude carries no Bedrock-style prefix, so it prices -straight off the `anthropic` block. +`model_prices_and_context_window.json`. The **catalog is the source of default +prices**: management's `pricing.DefaultTable` folds every catalog provider's +models into the surfaces that provider declares (`PricingSurfaces`), so coverage +is structural rather than maintained in a parallel file +([pricing/defaults.go](../../../management/internals/modules/agentnetwork/pricing/defaults.go)). +`TestDefaultTable_CoversEveryCatalogModel` fails if a catalog model ends up +unpriced, and `TestDefaultTable_NoConflictingContributions` fails if two +providers contribute the same (surface, model) at different rates. Bedrock +entries are keyed by the **normalised** id the request parser emits (region +prefix + version suffix stripped) — management applies the same normalisation to +per-provider prices at synth time, so the two keys compare equal. Vertex Claude +carries no Bedrock-style prefix, so it prices straight off the `anthropic` +surface. ## Things to scrutinise @@ -232,16 +237,17 @@ operator-misconfigured Vertex provider and unmetered Gemini traffic; verify publishers). **Correctness.** `normalizeBedrockModel` is the join between the wire id and the -pricing key — a model that normalises to something not in `defaults_pricing.yaml` -meters at `cost.skipped=unknown_model` rather than failing the request. The +pricing key — a model that normalises to something absent from the shipped +pricing table meters at `cost.skipped=unknown_model` rather than failing the +request. The `/bedrock` prefix strip must run on both the parser side (so the model is extracted) and the router side (so the upstream path is native); a regression in either silently breaks the other. **Metering caveats.** eu/apac cross-region Bedrock + Vertex profiles carry a -~10% premium not modelled by base pricing — flagged in both the catalog comment -and `defaults_pricing.yaml`. Operators needing exact regional billing override -the relevant entries. +~10% premium not modelled by base pricing — flagged in the catalog comment. +Operators needing exact regional billing set per-provider prices on the model +rows (or replace the default entries via `AgentNetwork.PricingDefaultsFile`). ## Cross-references