[GH-ISSUE #7019] Reverse-proxy peers are never garbage-collected despite ephemeral=true, stalling every DNS lookup by ~2s #12290

Open
opened 2026-08-05 01:32:48 -04:00 by saavagebueno · 2 comments
Owner

Originally created by @kevin3274 on GitHub (Aug 1, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/7019

Describe the bug

Peer rows created by the reverse-proxy component are never garbage-collected, even though they are correctly marked ephemeral = true. They accumulate with every proxy pod replacement and keep being pushed to every client in the network map.

Because the client's local resolver runs the lazy-connection warm-up before filtering disconnected peers out of the answer, those dead rows make every A lookup for a proxy service domain block for the full 2s warm-up budget.

The same cleanup works correctly for networkrouter and clusterproxy peers created by the Kubernetes operator — reverse-proxy is the only component affected.

Evidence

Self-hosted v0.76.0. Setup-key usage vs. rows left behind in the peers table:

component ephemeral registrations rows left in DB
networkrouter-* true 87 2
clusterproxy-* true 131 3
proxy-* (reverse-proxy) true 8 8

networkrouter registered 87 times and left only 2 rows — the ephemeral cleaner works fine. Reverse-proxy left every single one.

These rows do not appear in GET /api/peers, so they are invisible from the API. They show up only in the DB and in each client's peer list, where they sit permanently in Connecting:

Peers count: 6/13 Connected      # 7 of the 13 are dead reverse-proxy peers

DELETE /api/peers/<id> does work on them, which is the only workaround we found.

Root cause

management/internals/modules/peers/ephemeral/manager/ephemeral.go:

  • OnPeerDisconnected() adds the peer to the cleanup list with a deadline
  • cleanup() sweeps it on cleanupWindow = 1 * time.Minute
  • OnPeerConnected() removes it from the list

Reverse-proxy peers appear never to reach OnPeerDisconnected(). The proxy connects over its own gRPC ProxyService channel, and the management log only ever shows the proxy session ending:

proxy <id> heartbeat stopped: context canceled
Proxy <id> session <sid> disconnected

There is no corresponding peer-disconnect handling, so the cleanup timer never starts. This matches what clients observe: the stale peers remain in Connecting forever instead of transitioning to disconnected.

Impact: ~2s added to nearly every DNS lookup

client/internal/dns/local/local.go:

d.warmLazyPeers(question, result.records)   // "Warm before filtering"
...
if len(records) < 2 { return }              // short-circuit

Warm-up receives the unfiltered record set, so it blocks in ActivatePeersByIP waiting on a peer that can never connect, until defaultLazyWarmupTimeout (2s) expires. With TTL 5 on the answer, this lands on nearly every navigation.

Measured:

DB state A-query latency
8 proxy rows (1 live + 7 stale) 2001 ms
2 proxy rows (both live) 0 ms

Two observations that took us a long time to establish, in case they save someone else the trouble:

  • ActivatePeersByIP itself behaves correctly. With 2 live records the warm-up still triggers (len(records) >= 2) yet returns in 0 ms, honouring its "fast no-op for already-connected addresses" contract. The stalls come entirely from unreachable peers in the set.
  • dig shows only the live record, because disconnected ones are filtered right after warm-up. The problem is therefore completely invisible from the DNS answer — it just reads as "the VPN is slow", with nothing logged anywhere.

To Reproduce

  1. Self-hosted deployment with the reverse-proxy component and a private service registered under a proxy cluster domain.
  2. Restart / reschedule the reverse-proxy pod several times (rollout, node drain, spot reclaim...).
  3. Query the service domain: dig A <service>.<proxy-cluster-domain> @<netbird-client-dns>
  4. Every A query takes ~2001 ms; netbird status shows the dead proxy peers stuck in Connecting.
  5. DELETE /api/peers/<id> for each dead row → latency drops to 0 ms.

Expected behavior

Reverse-proxy peer rows should go through the same ephemeral cleanup path as networkrouter / clusterproxy peers — i.e. OnPeerDisconnected() should fire when the proxy session ends, so the 1-minute cleanup window applies.

Secondarily, it may be worth considering whether warmLazyPeers should skip addresses already known to be disconnected. The resolver filters them immediately afterwards anyway, so skipping them would bound the blast radius of any stale record regardless of how it got there.

Environment

  • NetBird server + client: v0.76.0 (self-hosted, all components pinned to the same version)
  • Kubernetes (kOps) on AWS, netbird kubernetes-operator v0.8.0
  • Postgres store
  • Reverse-proxy with 2 replicas, NB_PROXY_PRIVATE=true
Originally created by @kevin3274 on GitHub (Aug 1, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/7019 ## Describe the bug Peer rows created by the **reverse-proxy** component are never garbage-collected, even though they are correctly marked `ephemeral = true`. They accumulate with every proxy pod replacement and keep being pushed to every client in the network map. Because the client's local resolver runs the lazy-connection warm-up **before** filtering disconnected peers out of the answer, those dead rows make every A lookup for a proxy service domain block for the full 2s warm-up budget. The same cleanup works correctly for `networkrouter` and `clusterproxy` peers created by the Kubernetes operator — reverse-proxy is the only component affected. ## Evidence Self-hosted v0.76.0. Setup-key usage vs. rows left behind in the `peers` table: | component | `ephemeral` | registrations | rows left in DB | |---|---|---|---| | `networkrouter-*` | `true` | 87 | 2 | | `clusterproxy-*` | `true` | 131 | 3 | | `proxy-*` (reverse-proxy) | `true` | 8 | **8** | `networkrouter` registered 87 times and left only 2 rows — the ephemeral cleaner works fine. Reverse-proxy left every single one. These rows do **not** appear in `GET /api/peers`, so they are invisible from the API. They show up only in the DB and in each client's peer list, where they sit permanently in `Connecting`: ``` Peers count: 6/13 Connected # 7 of the 13 are dead reverse-proxy peers ``` `DELETE /api/peers/<id>` does work on them, which is the only workaround we found. ## Root cause `management/internals/modules/peers/ephemeral/manager/ephemeral.go`: - `OnPeerDisconnected()` adds the peer to the cleanup list with a deadline - `cleanup()` sweeps it on `cleanupWindow = 1 * time.Minute` - `OnPeerConnected()` removes it from the list Reverse-proxy peers appear never to reach `OnPeerDisconnected()`. The proxy connects over its own gRPC `ProxyService` channel, and the management log only ever shows the **proxy session** ending: ``` proxy <id> heartbeat stopped: context canceled Proxy <id> session <sid> disconnected ``` There is no corresponding peer-disconnect handling, so the cleanup timer never starts. This matches what clients observe: the stale peers remain in `Connecting` forever instead of transitioning to disconnected. ## Impact: ~2s added to nearly every DNS lookup `client/internal/dns/local/local.go`: ```go d.warmLazyPeers(question, result.records) // "Warm before filtering" ... if len(records) < 2 { return } // short-circuit ``` Warm-up receives the **unfiltered** record set, so it blocks in `ActivatePeersByIP` waiting on a peer that can never connect, until `defaultLazyWarmupTimeout` (2s) expires. With `TTL 5` on the answer, this lands on nearly every navigation. Measured: | DB state | A-query latency | |---|---| | 8 proxy rows (1 live + 7 stale) | **2001 ms** | | 2 proxy rows (both live) | **0 ms** | Two observations that took us a long time to establish, in case they save someone else the trouble: - **`ActivatePeersByIP` itself behaves correctly.** With 2 live records the warm-up still triggers (`len(records) >= 2`) yet returns in 0 ms, honouring its "fast no-op for already-connected addresses" contract. The stalls come entirely from unreachable peers in the set. - **`dig` shows only the live record**, because disconnected ones are filtered right after warm-up. The problem is therefore completely invisible from the DNS answer — it just reads as "the VPN is slow", with nothing logged anywhere. ## To Reproduce 1. Self-hosted deployment with the reverse-proxy component and a private service registered under a proxy cluster domain. 2. Restart / reschedule the reverse-proxy pod several times (rollout, node drain, spot reclaim...). 3. Query the service domain: `dig A <service>.<proxy-cluster-domain> @<netbird-client-dns>` 4. Every A query takes ~2001 ms; `netbird status` shows the dead proxy peers stuck in `Connecting`. 5. `DELETE /api/peers/<id>` for each dead row → latency drops to 0 ms. ## Expected behavior Reverse-proxy peer rows should go through the same ephemeral cleanup path as `networkrouter` / `clusterproxy` peers — i.e. `OnPeerDisconnected()` should fire when the proxy session ends, so the 1-minute cleanup window applies. Secondarily, it may be worth considering whether `warmLazyPeers` should skip addresses already known to be disconnected. The resolver filters them immediately afterwards anyway, so skipping them would bound the blast radius of any stale record regardless of how it got there. ## Environment - NetBird server + client: **v0.76.0** (self-hosted, all components pinned to the same version) - Kubernetes (kOps) on AWS, netbird kubernetes-operator v0.8.0 - Postgres store - Reverse-proxy with 2 replicas, `NB_PROXY_PRIVATE=true`
Author
Owner

@linear-code[bot] commented on GitHub (Aug 1, 2026):

NET-1463

<!-- gh-comment-id:5151297100 --> @linear-code[bot] commented on GitHub (Aug 1, 2026): <!-- linear-linkback --> <p><a href="https://linear.app/netbird/issue/NET-1463">NET-1463</a></p>
Author
Owner

@renne commented on GitHub (Aug 2, 2026):

Same happened in Docker Compose with 2 reverse-proxies. After a restart Netbird Management-Server had both + 1 stale. Stale was not cleared and distributed to clients causing wrong DNS resolutions and connection errors.

<!-- gh-comment-id:5159822867 --> @renne commented on GitHub (Aug 2, 2026): Same happened in Docker Compose with 2 reverse-proxies. After a restart Netbird Management-Server had both + 1 stale. Stale was not cleared and distributed to clients causing wrong DNS resolutions and connection errors.
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#12290