[GH-ISSUE #6950] Reverse-proxy (Agent Network) never recovers after its ephemeral peer is garbage-collected — embedded client permanently exits retry loop, mapping updates can't heal it #13069

Open
opened 2026-08-05 02:07:23 -04:00 by saavagebueno · 1 comment
Owner

Originally created by @ant0n777-dev on GitHub (Jul 28, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/6950

Describe the problem

On a self-hosted deployment, a ~36 min network outage between the reverse-proxy host and the management server permanently disabled the Agent Network endpoint. The proxy never recovered on its own; a manual container restart 3.5 hours later fixed it immediately, with byte-identical configuration.

The stuck state is silent on the management side: once connectivity returned, the proxy re-established its ProxyService session, so management_proxy_connection_count went back to 1 and management_proxy_heartbeat_count_total resumed incrementing once per minute — while the endpoint stayed dead for the next 3.5 hours.

Environment

  • NetBird management / signal / relay: 0.75.0, self-hosted (Podman/Quadlet), single account
  • netbirdio/reverse-proxy:0.75.0, running on a separate VM in another country from management
  • Agent Network in private mode (NB_PROXY_PRIVATE=true), one custom provider, one policy
  • Endpoint domain served from NetBird's internal DNS (no public wildcard record)
  • Also checked v0.75.1 (latest at time of filing): release notes contain no related changes.
  • Code references below are from main; the file:line pairs match the log lines emitted by our 0.75.0 binaries verbatim.

Timeline (all times UTC)

  1. 13:05 — network path proxy-host → management broke. Signal, management and relay all became unreachable from the proxy host (code = Unavailable, keepalive ping failed, relay ws/quic dial timeouts). The proxy host itself stayed online; already-established WireGuard tunnels from that host to other peers kept working throughout.

  2. 13:09 — the ProxyService session dropped: management_proxy_connection_count0, heartbeats stopped. (So far, so observable.)

  3. ~13:19:30 — management's ephemeral cleanup deleted the proxy's overlay peer: management_ephemeral_peers_cleaned_counter_total incremented by 1, ~14.5 min after the proxy went unreachable. Attribution: three unrelated ephemeral browser clients (offline since ~13:15) were cleaned in a separate +3 increment at ~13:30:30, and the API-visible peer count dropped by exactly 3 — the fourth cleaned peer is the proxy's service peer, which never appears in /api/peers. The proxy peer being ephemeral is confirmed in source: CreateProxyPeer in management/internals/modules/peers/manager.go creates the peer with Ephemeral: true (~line 230).

  4. 13:40:16 — connectivity restored; the embedded client's first RPC that got through returned an application-level error, and the client gave up permanently:

WARN shared/management/client/grpc.go:232: exiting the Management service connection retry loop
     due to the unrecoverable error: rpc error: code = PermissionDenied desc = peer is not registered
ERRO shared/management/client/grpc.go:601: failed to login to Management Service: rpc error:
     code = PermissionDenied desc = no peer auth method provided, please use a setup key or
     interactive SSO login
  1. 13:41:45 — the proxy re-established its ProxyService session and logged Initial mapping sync complete. All management-side proxy metrics went green again. But the embedded client never retried: from here the proxy logged nothing for hours.

  2. Consequences for every client peer in the Agent Network policy group: the endpoint hostname stopped resolving (NXDOMAIN — the record is served by NetBird's internal DNS and depends on the proxy having an overlay peer), and nothing served the endpoint on the overlay. All LLM traffic through the gateway failed.

  3. 17:03:22 — we mutated the Agent Network config (toggled the provider) to force a reconcile. Management re-sent the mapping; the proxy logged:

WARN proxy/internal/roundtrip/netbird.go:290: failed to notify status for existing client

— and did not recover. See root cause below: mapping updates re-register the service against the existing dead client.

  1. 17:13:46systemctl restart of the proxy container fixed everything within seconds. The proxy re-enrolled via CreateProxyPeer, received a new overlay IP, and the endpoint resolved and served requests again. No configuration or token change was involved.

Root cause (from code, main)

Three pieces combine into a permanent dead-end:

  1. The embedded client treats peer is not registered as terminal. shared/management/client/grpc.go:232 exits the retry loop on PermissionDenied as an "unrecoverable error". The client object survives in a dead state.

  2. Dead clients are never reaped or re-created. In proxy/internal/roundtrip/netbird.go, the client entry is removed from n.clients only by RemovePeer (mapping REMOVED). Any mapping CREATE/MODIFIED goes through AddPeerregisterExistingClient, which finds the existing (dead) entry and returns early — CreateProxyPeer is never attempted again. This is exactly what we observed in step 7: the netbird.go:290 warning is emitted from that code path.

  3. The proxy's overlay peer is Ephemeral: true (management/internals/modules/peers/manager.go, CreateProxyPeer), so any control-plane outage longer than the ephemeral lifetime deletes it — guaranteeing (1) fires on reconnect.

So: outage > ephemeral lifetime → peer GC'd → client permanently dead → no code path short of a process restart can recover, even though the proxy holds everything it needs (its NB_PROXY_TOKEN and a live ProxyService session — which is how a restart with identical config succeeds).

Secondary issue: readiness does not reflect ability to serve

Checker.ReadinessProbe() (proxy/internal/health/health.go) returns only managementConnected — the ProxyService session state, not the embedded client that actually serves the private endpoint. handleFull derives its status code from the same probe. By inspection, during the 3.5 h stuck period /healthz/ready and /healthz return 200 (the health server binds localhost inside the container, so we did not sample it live during the incident). Only /healthz/startup — which calls CheckClientsConnected and would see management_connected: false on the dead client — would have reported failure.

For a private (NetBird-only) service the proxy cannot serve any traffic without its overlay client, so an orchestrator wired to the documented readiness probe will never restart a proxy stuck in this state.

Expected behaviour

  1. On peer is not registered, re-create the embedded client (fresh CreateProxyPeer over the live ProxyService session) instead of exiting the retry loop permanently — a process restart proves all required material is already at hand.
  2. Failing that, reap dead client entries so that a re-delivered mapping (registerExistingClient miss) triggers re-enrollment.
  3. /healthz/ready should include embedded-client health for private services, so a container restart policy can recover the service.
  4. Consider exempting proxy service peers from ephemeral cleanup while the owning proxy holds a valid token / ProxyService registration, or make the proxy tolerant of the deletion.

Also seen by

The same log signature is reported by another self-hosted user, with no reply:
https://forum.netbird.io/t/agent-network-cant-connect-to-endpoint-timeout/853

(Note: the NotFound desc = service agent-net-svc-<accountID> not found warning on notify certificate ready, visible in that thread and in our logs, appears to be cosmetic and unrelated — synthesized Agent Network services are in-memory only (SynthesizedServiceIDPrefix, agentnetwork/synthesizer.go), so the persisted-service lookup always misses. It appears on every proxy start, including fully working ones. It made triage considerably harder, so it may be worth downgrading or rewording.)

Originally created by @ant0n777-dev on GitHub (Jul 28, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/6950 ### Describe the problem On a self-hosted deployment, a ~36 min network outage between the `reverse-proxy` host and the management server permanently disabled the Agent Network endpoint. The proxy never recovered on its own; a manual container restart 3.5 hours later fixed it immediately, with byte-identical configuration. The stuck state is silent on the management side: once connectivity returned, the proxy re-established its ProxyService session, so `management_proxy_connection_count` went back to `1` and `management_proxy_heartbeat_count_total` resumed incrementing once per minute — while the endpoint stayed dead for the next 3.5 hours. ### Environment - NetBird management / signal / relay: **0.75.0**, self-hosted (Podman/Quadlet), single account - `netbirdio/reverse-proxy:0.75.0`, running on a separate VM in another country from management - Agent Network in private mode (`NB_PROXY_PRIVATE=true`), one custom provider, one policy - Endpoint domain served from NetBird's internal DNS (no public wildcard record) - Also checked **v0.75.1** (latest at time of filing): release notes contain no related changes. - Code references below are from `main`; the file:line pairs match the log lines emitted by our 0.75.0 binaries verbatim. ### Timeline (all times UTC) 1. `13:05` — network path proxy-host → management broke. Signal, management and relay all became unreachable from the proxy host (`code = Unavailable`, `keepalive ping failed`, relay `ws`/`quic` dial timeouts). The proxy host itself stayed online; already-established WireGuard tunnels from that host to other peers kept working throughout. 2. `13:09` — the ProxyService session dropped: `management_proxy_connection_count` → `0`, heartbeats stopped. (So far, so observable.) 3. `~13:19:30` — management's ephemeral cleanup deleted the proxy's overlay peer: `management_ephemeral_peers_cleaned_counter_total` incremented by 1, ~14.5 min after the proxy went unreachable. Attribution: three unrelated ephemeral browser clients (offline since ~13:15) were cleaned in a separate +3 increment at `~13:30:30`, and the API-visible peer count dropped by exactly 3 — the fourth cleaned peer is the proxy's service peer, which never appears in `/api/peers`. The proxy peer being ephemeral is confirmed in source: `CreateProxyPeer` in `management/internals/modules/peers/manager.go` creates the peer with `Ephemeral: true` (~line 230). 4. `13:40:16` — connectivity restored; the embedded client's first RPC that got through returned an application-level error, and the client gave up permanently: ``` WARN shared/management/client/grpc.go:232: exiting the Management service connection retry loop due to the unrecoverable error: rpc error: code = PermissionDenied desc = peer is not registered ERRO shared/management/client/grpc.go:601: failed to login to Management Service: rpc error: code = PermissionDenied desc = no peer auth method provided, please use a setup key or interactive SSO login ``` 5. `13:41:45` — the proxy re-established its ProxyService session and logged `Initial mapping sync complete`. All management-side proxy metrics went green again. But the embedded client never retried: from here the proxy logged nothing for hours. 6. Consequences for every client peer in the Agent Network policy group: the endpoint hostname stopped resolving (NXDOMAIN — the record is served by NetBird's internal DNS and depends on the proxy having an overlay peer), and nothing served the endpoint on the overlay. All LLM traffic through the gateway failed. 7. `17:03:22` — we mutated the Agent Network config (toggled the provider) to force a reconcile. Management re-sent the mapping; the proxy logged: ``` WARN proxy/internal/roundtrip/netbird.go:290: failed to notify status for existing client ``` — and did **not** recover. See root cause below: mapping updates re-register the service against the existing dead client. 8. `17:13:46` — `systemctl restart` of the proxy container fixed everything within seconds. The proxy re-enrolled via `CreateProxyPeer`, received a **new** overlay IP, and the endpoint resolved and served requests again. No configuration or token change was involved. ### Root cause (from code, main) Three pieces combine into a permanent dead-end: 1. **The embedded client treats `peer is not registered` as terminal.** `shared/management/client/grpc.go:232` exits the retry loop on `PermissionDenied` as an "unrecoverable error". The client object survives in a dead state. 2. **Dead clients are never reaped or re-created.** In `proxy/internal/roundtrip/netbird.go`, the client entry is removed from `n.clients` only by `RemovePeer` (mapping `REMOVED`). Any mapping `CREATE`/`MODIFIED` goes through `AddPeer` → `registerExistingClient`, which finds the existing (dead) entry and returns early — `CreateProxyPeer` is never attempted again. This is exactly what we observed in step 7: the `netbird.go:290` warning is emitted from that code path. 3. **The proxy's overlay peer is `Ephemeral: true`** (`management/internals/modules/peers/manager.go`, `CreateProxyPeer`), so any control-plane outage longer than the ephemeral lifetime deletes it — guaranteeing (1) fires on reconnect. So: outage > ephemeral lifetime → peer GC'd → client permanently dead → no code path short of a process restart can recover, even though the proxy holds everything it needs (its `NB_PROXY_TOKEN` and a live ProxyService session — which is how a restart with identical config succeeds). ### Secondary issue: readiness does not reflect ability to serve `Checker.ReadinessProbe()` (`proxy/internal/health/health.go`) returns only `managementConnected` — the ProxyService session state, not the embedded client that actually serves the private endpoint. `handleFull` derives its status code from the same probe. By inspection, during the 3.5 h stuck period `/healthz/ready` and `/healthz` return `200` (the health server binds localhost inside the container, so we did not sample it live during the incident). Only `/healthz/startup` — which calls `CheckClientsConnected` and would see `management_connected: false` on the dead client — would have reported failure. For a private (NetBird-only) service the proxy cannot serve any traffic without its overlay client, so an orchestrator wired to the documented readiness probe will never restart a proxy stuck in this state. ### Expected behaviour 1. On `peer is not registered`, re-create the embedded client (fresh `CreateProxyPeer` over the live ProxyService session) instead of exiting the retry loop permanently — a process restart proves all required material is already at hand. 2. Failing that, reap dead client entries so that a re-delivered mapping (`registerExistingClient` miss) triggers re-enrollment. 3. `/healthz/ready` should include embedded-client health for private services, so a container restart policy can recover the service. 4. Consider exempting proxy service peers from ephemeral cleanup while the owning proxy holds a valid token / ProxyService registration, or make the proxy tolerant of the deletion. ### Also seen by The same log signature is reported by another self-hosted user, with no reply: https://forum.netbird.io/t/agent-network-cant-connect-to-endpoint-timeout/853 (Note: the `NotFound desc = service agent-net-svc-<accountID> not found` warning on `notify certificate ready`, visible in that thread and in our logs, appears to be cosmetic and unrelated — synthesized Agent Network services are in-memory only (`SynthesizedServiceIDPrefix`, `agentnetwork/synthesizer.go`), so the persisted-service lookup always misses. It appears on every proxy start, including fully working ones. It made triage considerably harder, so it may be worth downgrading or rewording.)
Author
Owner

@linear-code[bot] commented on GitHub (Jul 28, 2026):

NET-1450

<!-- gh-comment-id:5108297894 --> @linear-code[bot] commented on GitHub (Jul 28, 2026): <!-- linear-linkback --> <p><a href="https://linear.app/netbird/issue/NET-1450">NET-1450</a></p>
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#13069