[GH-ISSUE #6467] Agent on a high-fan-out hub (~841 peers) wedges over time — some peers stuck Connecting (ICE -/-, no handshake) while daemon reports Connected; only a hub restart recovers #12218

Open
opened 2026-08-05 01:32:41 -04:00 by saavagebueno · 3 comments
Owner

Originally created by @yspanchal-ottu on GitHub (Jun 18, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/6467

netbird.debug.2047447731.SANITIZED.zip

Describe the problem

In a self-hosted setup, one peer acts as a hub — an SSH gateway that has an ACL policy allowing it to reach all ~841 peers in the network. Over time (days–weeks of uptime) the agent on this hub wedges: the daemon reports most peers as Connected, but it stops cleanly establishing connections to some peers. Affected peers (e.g. a laptop client) are stuck at Status: Connecting with ICE candidate -/-, no relay address, and no WireGuard handshake — and from those peers, traffic to the hub fails with Required key not available (ENOKEY).

A netbird down && netbird up on the hub fully recovers it (peers reconnect), but the problem returns after ~weeks of uptime. It happens in both lazy-connection modes (lazy on: on-demand activation wedges; lazy off: ~841 eager tunnels).

The host is not resource-starved and the management server is idle, so this appears to be an agent-side stability/scale issue on a single high-fan-out hub, not a management-side performance problem (it does not match #4488, which is management-side, and our management CPU is near-idle).

Are you using NetBird Cloud or self-hosted?

Self-hosted.

NetBird version

0.70.5 — management, signal, and all agents on the same version. (Management runs native TLS on :443; relays run on the two nodes.)

Environment

  • Hub peer: Linux, 2 vCPU / 3.8 GiB RAM, kernel WireGuard (Interface type: Kernel), uptime ~38 days when observed wedged.
  • ~841 total peers in the account. The hub has an ACL policy granting it access to essentially all of them (it's an SSH gateway). Most other peers are low-fan-out.
  • Management + Signal: Connected. Relays: 2/2 Available on both hub and clients.

Symptoms / diagnostics

On the HUB (while wedged):

Peers count: 834/841 Connected      # daemon claims connected
Management: Connected   Signal: Connected   Relays: 2/2 Available
Lazy connection: false   Interface type: Kernel
load average: 1.33, 1.06, 1.00      # 2 cores
mem: 1.1Gi used / 3.8Gi (2.8Gi available)   disk: 3%
netbird daemon: ~49.7% CPU (~1 core pegged constantly), ~430 MB RSS
conntrack: 973 / 65536   (no fd/conntrack pressure)
journalctl -u netbird: no errors logged in the window

On an AFFECTED CLIENT (cannot reach the hub):

sudo netbird status -d
 <hub>:
  Status: Connecting
  Connection type: -
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:            # empty
  Last WireGuard handshake: -      # never

ping <hub-overlay-ip>
  From <client-overlay-ip> icmp_seq=1 Destination Host Unreachable
  ping: sendmsg: Required key not available

The client connects to other peers fine — only the hub is stuck. The hub likewise lists this client among the few it isn't connected to.

What we ruled out

  • Host resources — hub has a free CPU core, ~2.8 GiB free RAM, 3% disk, no conntrack/fd pressure.
  • Management-side load — management nodes are near-idle (load ~0.2), so not #4488.
  • DNS / policy — DNS resolves the hub's overlay IP correctly; the ACL policy (all → hub, all ports) covers the affected client. Failure is at WireGuard tunnel establishment, not name resolution or ACL.
  • Lazy connection toggle — reproduced with lazy both ON and OFF.
  • Client sidenetbird down/up on the client does not help; only restarting the hub's daemon does.

Workaround

systemctl restart netbird on the hub restores all peer connections. Recurs after ~weeks.

Questions

  1. Is the agent known to degrade on a single peer that must hold/manage connections to many hundreds of peers (~841)? Is there a recommended ceiling / guidance for a high-fan-out hub?
  2. Is there a known fix or relevant change in a newer release (0.71.x / 0.72.x) for the agent getting stuck Connecting / not (re)establishing some peers at scale? The 0.72.x notes don't obviously mention this.
  3. Is there an agent setting (keepalive, connection-worker concurrency, relay vs P2P preference, inactivity) to make a high-fan-out hub stable without periodic restarts?
  4. What logs/metrics best capture the wedged state for diagnosis?

Debug bundle

A sanitized netbird debug bundle --anonymize is attached. Note: network_map.json and status.txt were removed because peer DNS labels/names are not anonymized by --anonymize (they contain our internal/customer hostnames). All profiling artifacts are retained — goroutine.prof, mutex.prof, block.prof, heap.prof, allocs.prof, stack_trace.txt, plus client.log, config.txt, wgshow.txt, metrics.txt. Happy to provide the topology/status privately if needed.

Originally created by @yspanchal-ottu on GitHub (Jun 18, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/6467 [netbird.debug.2047447731.SANITIZED.zip](https://github.com/user-attachments/files/29103746/netbird.debug.2047447731.SANITIZED.zip) Describe the problem In a self-hosted setup, one peer acts as a **hub** — an SSH gateway that has an ACL policy allowing it to reach **all ~841 peers** in the network. Over time (days–weeks of uptime) the **agent on this hub wedges**: the daemon reports most peers as `Connected`, but it stops cleanly establishing connections to *some* peers. Affected peers (e.g. a laptop client) are stuck at `Status: Connecting` with `ICE candidate -/-`, no relay address, and no WireGuard handshake — and from those peers, traffic to the hub fails with `Required key not available` (ENOKEY). **A `netbird down && netbird up` on the hub fully recovers it** (peers reconnect), but the problem returns after ~weeks of uptime. It happens in **both lazy-connection modes** (lazy on: on-demand activation wedges; lazy off: ~841 eager tunnels). The host is **not resource-starved** and the **management server is idle**, so this appears to be an **agent-side stability/scale issue on a single high-fan-out hub**, not a management-side performance problem (it does not match #4488, which is management-side, and our management CPU is near-idle). ### Are you using NetBird Cloud or self-hosted? Self-hosted. ### NetBird version `0.70.5` — management, signal, and all agents on the same version. (Management runs native TLS on :443; relays run on the two nodes.) ### Environment - Hub peer: Linux, **2 vCPU / 3.8 GiB RAM**, kernel WireGuard (`Interface type: Kernel`), uptime ~38 days when observed wedged. - ~841 total peers in the account. The hub has an ACL policy granting it access to essentially all of them (it's an SSH gateway). Most other peers are low-fan-out. - Management + Signal: `Connected`. Relays: `2/2 Available` on both hub and clients. ### Symptoms / diagnostics **On the HUB (while wedged):** ``` Peers count: 834/841 Connected # daemon claims connected Management: Connected Signal: Connected Relays: 2/2 Available Lazy connection: false Interface type: Kernel load average: 1.33, 1.06, 1.00 # 2 cores mem: 1.1Gi used / 3.8Gi (2.8Gi available) disk: 3% netbird daemon: ~49.7% CPU (~1 core pegged constantly), ~430 MB RSS conntrack: 973 / 65536 (no fd/conntrack pressure) journalctl -u netbird: no errors logged in the window ``` **On an AFFECTED CLIENT (cannot reach the hub):** ``` sudo netbird status -d <hub>: Status: Connecting Connection type: - ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: # empty Last WireGuard handshake: - # never ping <hub-overlay-ip> From <client-overlay-ip> icmp_seq=1 Destination Host Unreachable ping: sendmsg: Required key not available ``` The client connects to other peers fine — only the hub is stuck. The hub likewise lists this client among the few it isn't connected to. ### What we ruled out - **Host resources** — hub has a free CPU core, ~2.8 GiB free RAM, 3% disk, no conntrack/fd pressure. - **Management-side load** — management nodes are near-idle (load ~0.2), so not #4488. - **DNS / policy** — DNS resolves the hub's overlay IP correctly; the ACL policy (`all → hub`, all ports) covers the affected client. Failure is at WireGuard tunnel establishment, not name resolution or ACL. - **Lazy connection toggle** — reproduced with lazy both ON and OFF. - **Client side** — `netbird down/up` on the client does not help; only restarting the **hub's** daemon does. ### Workaround `systemctl restart netbird` on the hub restores all peer connections. Recurs after ~weeks. ### Questions 1. Is the agent known to degrade on a single peer that must hold/manage connections to many hundreds of peers (~841)? Is there a recommended ceiling / guidance for a high-fan-out hub? 2. Is there a known fix or relevant change in a newer release (0.71.x / 0.72.x) for the agent getting stuck `Connecting` / not (re)establishing some peers at scale? The 0.72.x notes don't obviously mention this. 3. Is there an agent setting (keepalive, connection-worker concurrency, relay vs P2P preference, inactivity) to make a high-fan-out hub stable without periodic restarts? 4. What logs/metrics best capture the wedged state for diagnosis? ### Debug bundle A **sanitized** `netbird debug bundle --anonymize` is attached. Note: `network_map.json` and `status.txt` were removed because peer **DNS labels/names are not anonymized** by `--anonymize` (they contain our internal/customer hostnames). All profiling artifacts are retained — `goroutine.prof`, `mutex.prof`, `block.prof`, `heap.prof`, `allocs.prof`, `stack_trace.txt`, plus `client.log`, `config.txt`, `wgshow.txt`, `metrics.txt`. Happy to provide the topology/status privately if needed.
Author
Owner

@linear-code[bot] commented on GitHub (Jun 18, 2026):

NET-1293

<!-- gh-comment-id:4744203660 --> @linear-code[bot] commented on GitHub (Jun 18, 2026): <!-- linear-linkback --> <p><a href="https://linear.app/netbird/issue/NET-1293">NET-1293</a></p>
Author
Owner

@lixmal commented on GitHub (Jun 19, 2026):

Please test with the latest versions of signal/client (0.73.0)

<!-- gh-comment-id:4750212504 --> @lixmal commented on GitHub (Jun 19, 2026): Please test with the latest versions of signal/client (0.73.0)
Author
Owner

@Technerd-SG commented on GitHub (Jun 27, 2026):

I have a wired issue myself since a few days I have connection issues although I didn't change anything about my setup. When trying to connect the clients say that they are connected but thats false. After 30 seconds the status changes to connecting and trys again this goes for 2-3 minutes after that the client says connected and p2p and then it is really connected.
I never had issues like that normally all 40 clients connect within seconds without any problems..

I have the most current version while writing this ({"dashboard_available_version":"2.80.0","management_available_version":"0.73.2","management_current_version":"0.73.2","management_update_available":false})

<!-- gh-comment-id:4818317938 --> @Technerd-SG commented on GitHub (Jun 27, 2026): I have a wired issue myself since a few days I have connection issues although I didn't change anything about my setup. When trying to connect the clients say that they are connected but thats false. After 30 seconds the status changes to connecting and trys again this goes for 2-3 minutes after that the client says connected and p2p and then it is really connected. I never had issues like that normally all 40 clients connect within seconds without any problems.. I have the most current version while writing this ({"dashboard_available_version":"2.80.0","management_available_version":"0.73.2","management_current_version":"0.73.2","management_update_available":false})
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#12218