[GH-ISSUE #6953] Client grows to 6 GB RSS and spins ~60% of a core while Management is unreachable (0.73.2, linux/arm64) #12286

Open
opened 2026-08-05 01:32:47 -04:00 by saavagebueno · 2 comments
Owner

Originally created by @howdude230 on GitHub (Jul 29, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/6953

Summary

On a self-hosted deployment, netbird up --foreground-mode grew to 6.0 GB RSS
and consumed 52 hours of CPU time over 87 hours of wall time (~60% of one
core, sustained) while it was unable to reach the Management service.

The client did not exit, back off to a steady state, or cap its retry cost. It
kept allocating until the host ran out of memory and the kernel/init OOM killer
began terminating unrelated workloads on the same machine.

The memory is released immediately and completely on restart: a freshly started
client on the same host with the same (still failing) Management endpoint sits at
125 MB RSS with 12 s of CPU time. So the growth tracks time-spent-failing,
not configuration or workload.

Environment

NetBird client 0.73.2, installed as an OS extension/container
Management server self-hosted (version not independently verified at runtime; source pin records 0.71.x)
OS Talos Linux v1.13.6, Go 1.26.5
Arch linux/arm64
Host 2 vCPU / 11.9 GB RAM
Peers in account 12
Client invocation netbird up --foreground-mode

Impact

The client and the Management service it depends on were on the same host. That
made the failure self-sustaining:

  1. Management becomes unreachable.
  2. Client retries; RSS and CPU climb without bound.
  3. Host exhausts memory; the OOM killer starts killing the lowest-priority
    cgroups.
  4. Among the victims is the reverse proxy that serves the Management
    endpoint
    .
  5. Management is now definitively unreachable → back to step 2, permanently.

Once tipped, this could not self-heal. It ran for over an hour and only
recovered by manually restarting the NetBird client, which released 5.6 GB
instantly and let everything else recover.

This co-location is a deployment choice, not a NetBird bug, and we are not
asking you to fix it. It is included only because it explains why unbounded
retry cost escalated into a total outage rather than a degraded tunnel. On a
host where NetBird is not co-located with its own dependencies, the same
behaviour would still eventually OOM the host — it would just take longer to
matter.

Evidence

Process table at peak (host had 11904 MB total, 800 MB available):

PID     THREADS  CPU-TIME    VIRT    RES     COMMAND
15848   12       188030.32   7.3 GB  6.0 GB  /usr/local/bin/netbird up --foreground-mode
  • Wall clock for that PID: 87 h 25 m
  • CPU time: 188,030 s = 52.2 h → ~60% of one core, sustained, for days

After a plain restart, same host, Management still unreachable:

PID     THREADS  CPU-TIME  VIRT    RES     COMMAND
195115  10       12.35     1.4 GB  125 MB  /usr/local/bin/netbird up --foreground-mode

Log signature

Two patterns dominate the log while Management is down.

1. gRPC channel counter climbing. Each reconnect attempt appears to create a
new channel rather than reusing one — note Channel #31 SubChannel #32:

WARNING: [core] [Channel #31 SubChannel #32] grpc: addrConn.createTransport
  failed to connect to {Addr: "mgmt.example.net:443", ServerName: "mgmt.example.net:443", }.
  Err: connection error: desc = "transport: Error while dialing:
  nbnet.NewDialer().DialContext: dial tcp 203.0.113.10:443: i/o timeout"

2. High-rate writes to closed UDP sockets. These arrive in bursts, many
within the same millisecond, referencing peer sockets that are already gone:

level=ERROR msg="Failed to send" pid="<REDACTED-PEER-KEY>"
  error="write udp [::]:48535-><PEER-MESH-IP>:59693: use of closed network connection"
level=ERROR msg="Failed to send" pid="<REDACTED-PEER-KEY>"
  error="write udp [::]:48535-><PEER-MESH-IP>:59693: use of closed network connection"
level=ERROR msg="Failed to send" pid="<REDACTED-PEER-KEY>"
  error="write udp [::]:48535-><PEER-MESH-IP>:59693: use of closed network connection"
  ... (repeating at high frequency, same source port, same destination)

The repeated use of closed network connection against a stable source port
(48535) suggests send paths for dead peer connections are retained and kept
being driven rather than torn down.

3. Login failure loop. In --foreground-mode the process exits on login
failure and is restarted by the supervisor:

ERRO client/grpc.go:121: failed creating connection to Management Service:
  create connection: dial context: context deadline exceeded
ERRO auth/auth.go:51: failed connecting to Management Service https://mgmt.example.net:443:
  create connection: dial context: context deadline exceeded
Error: foreground login failed: failed to create auth client: create connection:
  dial context: context deadline exceeded

Note this exit-and-restart path is what keeps memory low now. The 6 GB process
was a long-lived one that had already authenticated and then lost Management,
so it stayed resident and retried in-process instead of exiting.

What we did not capture

Being straightforward about the limits of this report:

  • No pprof heap or goroutine dump was taken. The incident was live, the
    host was minutes from full exhaustion, and the process was restarted to
    restore service. That evidence is gone.
  • We therefore cannot point at a specific allocation site, and we are not
    claiming a diagnosed root cause in your code. What is established is the
    observable behaviour: unbounded RSS + CPU growth strictly correlated with
    time-spent-unable-to-reach-Management, fully reset by restart.
  • Management-side logs for the same window were not preserved.

If it would help, we can attempt a controlled reproduction (below) with
pprof enabled and attach real profiles.

Suggested reproduction

  1. Bring up a client with netbird up --foreground-mode against a reachable
    self-hosted Management, with a non-trivial peer count (ours: 12).
  2. Let it fully connect and establish peer sessions.
  3. Make Management unreachable in a way that blackholes rather than
    refuses — drop traffic to the Management address so dials hit
    i/o timeout, not connection refused. (In our case the TCP path timed
    out; that distinction may matter for which retry path is taken.)
  4. Leave it. Sample RSS and CPU time hourly.

We observed growth over days rather than minutes, so this likely needs to run
for hours to be visible.

What we would expect

  • Retry cost bounded — reconnect attempts reuse a channel, or old channels are
    closed and collected.
  • Send paths for peers that are no longer reachable are torn down rather than
    driven at high frequency against closed sockets.
  • Idle-but-disconnected steady state that does not grow without bound, so an
    unreachable Management degrades the tunnel instead of eventually taking down
    the host.

Workaround for other operators

Restarting the client immediately releases all of it. If you run the client on
a host alongside anything you care about, constrain it — a memory limit on the
NetBird service turns "host dies" into "NetBird restarts", which is strictly
better and is self-correcting.


Identifiers in this report are anonymized: hostnames, public and mesh IP
addresses, and peer public keys have been replaced with placeholders. Version
numbers, byte counts, CPU times, thread counts, port numbers and log structure
are verbatim.

Originally created by @howdude230 on GitHub (Jul 29, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/6953 ## Summary On a self-hosted deployment, `netbird up --foreground-mode` grew to **6.0 GB RSS** and consumed **52 hours of CPU time over 87 hours of wall time** (~60% of one core, sustained) while it was unable to reach the Management service. The client did not exit, back off to a steady state, or cap its retry cost. It kept allocating until the host ran out of memory and the kernel/init OOM killer began terminating unrelated workloads on the same machine. The memory is released immediately and completely on restart: a freshly started client on the same host with the same (still failing) Management endpoint sits at **125 MB RSS** with **12 s** of CPU time. So the growth tracks time-spent-failing, not configuration or workload. ## Environment | | | |---|---| | NetBird client | **0.73.2**, installed as an OS extension/container | | Management server | self-hosted (version not independently verified at runtime; source pin records 0.71.x) | | OS | Talos Linux v1.13.6, Go 1.26.5 | | Arch | **linux/arm64** | | Host | 2 vCPU / 11.9 GB RAM | | Peers in account | 12 | | Client invocation | `netbird up --foreground-mode` | ## Impact The client and the Management service it depends on were on the same host. That made the failure self-sustaining: 1. Management becomes unreachable. 2. Client retries; RSS and CPU climb without bound. 3. Host exhausts memory; the OOM killer starts killing the lowest-priority cgroups. 4. Among the victims is the reverse proxy that **serves the Management endpoint**. 5. Management is now definitively unreachable → back to step 2, permanently. Once tipped, this could not self-heal. It ran for over an hour and only recovered by manually restarting the NetBird client, which released 5.6 GB instantly and let everything else recover. This co-location is a deployment choice, not a NetBird bug, and we are not asking you to fix it. It is included only because it explains why unbounded retry cost escalated into a total outage rather than a degraded tunnel. On a host where NetBird is *not* co-located with its own dependencies, the same behaviour would still eventually OOM the host — it would just take longer to matter. ## Evidence Process table at peak (host had 11904 MB total, 800 MB available): ``` PID THREADS CPU-TIME VIRT RES COMMAND 15848 12 188030.32 7.3 GB 6.0 GB /usr/local/bin/netbird up --foreground-mode ``` - Wall clock for that PID: **87 h 25 m** - CPU time: **188,030 s = 52.2 h** → ~60% of one core, sustained, for days After a plain restart, same host, Management **still** unreachable: ``` PID THREADS CPU-TIME VIRT RES COMMAND 195115 10 12.35 1.4 GB 125 MB /usr/local/bin/netbird up --foreground-mode ``` ### Log signature Two patterns dominate the log while Management is down. **1. gRPC channel counter climbing.** Each reconnect attempt appears to create a new channel rather than reusing one — note `Channel #31 SubChannel #32`: ``` WARNING: [core] [Channel #31 SubChannel #32] grpc: addrConn.createTransport failed to connect to {Addr: "mgmt.example.net:443", ServerName: "mgmt.example.net:443", }. Err: connection error: desc = "transport: Error while dialing: nbnet.NewDialer().DialContext: dial tcp 203.0.113.10:443: i/o timeout" ``` **2. High-rate writes to closed UDP sockets.** These arrive in bursts, many within the same millisecond, referencing peer sockets that are already gone: ``` level=ERROR msg="Failed to send" pid="<REDACTED-PEER-KEY>" error="write udp [::]:48535-><PEER-MESH-IP>:59693: use of closed network connection" level=ERROR msg="Failed to send" pid="<REDACTED-PEER-KEY>" error="write udp [::]:48535-><PEER-MESH-IP>:59693: use of closed network connection" level=ERROR msg="Failed to send" pid="<REDACTED-PEER-KEY>" error="write udp [::]:48535-><PEER-MESH-IP>:59693: use of closed network connection" ... (repeating at high frequency, same source port, same destination) ``` The repeated `use of closed network connection` against a **stable source port** (`48535`) suggests send paths for dead peer connections are retained and kept being driven rather than torn down. **3. Login failure loop.** In `--foreground-mode` the process exits on login failure and is restarted by the supervisor: ``` ERRO client/grpc.go:121: failed creating connection to Management Service: create connection: dial context: context deadline exceeded ERRO auth/auth.go:51: failed connecting to Management Service https://mgmt.example.net:443: create connection: dial context: context deadline exceeded Error: foreground login failed: failed to create auth client: create connection: dial context: context deadline exceeded ``` Note this exit-and-restart path is what keeps memory low *now*. The 6 GB process was a long-lived one that had already authenticated and then lost Management, so it stayed resident and retried in-process instead of exiting. ## What we did not capture Being straightforward about the limits of this report: - **No `pprof` heap or goroutine dump was taken.** The incident was live, the host was minutes from full exhaustion, and the process was restarted to restore service. That evidence is gone. - We therefore **cannot point at a specific allocation site**, and we are not claiming a diagnosed root cause in your code. What is established is the observable behaviour: unbounded RSS + CPU growth strictly correlated with time-spent-unable-to-reach-Management, fully reset by restart. - Management-side logs for the same window were not preserved. If it would help, we can attempt a controlled reproduction (below) with `pprof` enabled and attach real profiles. ## Suggested reproduction 1. Bring up a client with `netbird up --foreground-mode` against a reachable self-hosted Management, with a non-trivial peer count (ours: 12). 2. Let it fully connect and establish peer sessions. 3. Make Management unreachable in a way that **blackholes** rather than refuses — drop traffic to the Management address so dials hit `i/o timeout`, not `connection refused`. (In our case the TCP path timed out; that distinction may matter for which retry path is taken.) 4. Leave it. Sample RSS and CPU time hourly. We observed growth over days rather than minutes, so this likely needs to run for hours to be visible. ## What we would expect - Retry cost bounded — reconnect attempts reuse a channel, or old channels are closed and collected. - Send paths for peers that are no longer reachable are torn down rather than driven at high frequency against closed sockets. - Idle-but-disconnected steady state that does not grow without bound, so an unreachable Management degrades the tunnel instead of eventually taking down the host. ## Workaround for other operators Restarting the client immediately releases all of it. If you run the client on a host alongside anything you care about, constrain it — a memory limit on the NetBird service turns "host dies" into "NetBird restarts", which is strictly better and is self-correcting. --- *Identifiers in this report are anonymized: hostnames, public and mesh IP addresses, and peer public keys have been replaced with placeholders. Version numbers, byte counts, CPU times, thread counts, port numbers and log structure are verbatim.*
Author
Owner

@linear-code[bot] commented on GitHub (Jul 29, 2026):

NET-1452

<!-- gh-comment-id:5113615996 --> @linear-code[bot] commented on GitHub (Jul 29, 2026): <!-- linear-linkback --> <p><a href="https://linear.app/netbird/issue/NET-1452">NET-1452</a></p>
Author
Owner

@Optic00 commented on GitHub (Jul 31, 2026):

I found and locally reproduced a concrete root-cause candidate for the unbounded CPU/RSS growth.

The exact log signature from this report:

level=ERROR msg="Failed to send" pid="..." error="write udp ... use of closed network connection"

comes from cunicu.li/go-rosenpass v0.5.42. NetBird v0.73.2 used that version, and v0.74.6, v0.76.0, and current main still use the same version.

An initiator handshake schedules a self-rescheduling retransmission timer. In v0.5.42:

  • handshake expiry removes the handshake from the map without stopping that timer;
  • initiatorHandshake.Close() stops only the expiry timer;
  • the surviving timer retains the handshake, peer, server, handlers, and closed UDP connection while continuing to send roughly every 10 seconds.

I reproduced this with a black-holed peer. At handshake expiry the test had recorded 8 sends; one second later it had reached 15. This behavior is covered by the existing upstream fix:

https://codeberg.org/cunicu/go-rosenpass/pulls/231

During validation I found two additional orphan paths in the same lifecycle:

  • a retransmission timer is created before the initial send, so an initial send failure can leave an unregistered timer behind;
  • a pending peer rekey timer can fire after Server.Close() and start another handshake on the closed socket.

Focused tests reproduced both cases. With go-rosenpass PR #231's timer synchronization plus fixes for those two cases, all three tests pass under the Go race detector.

This is a strong explanation for the continuing Failed to send traffic and retained memory. The increasing gRPC channel IDs probably indicate repeated NetBird engine generations, which would amplify the Rosenpass leak by repeatedly creating and closing servers. It may not explain every part of the incident by itself.

Could you confirm whether quantum resistance/Rosenpass was enabled on the affected client? The pid log field strongly suggests that it was.

The proposed contribution path is:

  1. complete and merge the upstream go-rosenpass timer fix;
  2. publish a new go-rosenpass tag;
  3. submit a focused NetBird dependency-bump PR;
  4. reproduce once more with a patched client and compare baseline versus post-blackhole heap/goroutine profiles.

Debug bundles or profiles may contain sensitive deployment information and should be shared privately with the maintainers.

<!-- gh-comment-id:5142689656 --> @Optic00 commented on GitHub (Jul 31, 2026): I found and locally reproduced a concrete root-cause candidate for the unbounded CPU/RSS growth. The exact log signature from this report: ```text level=ERROR msg="Failed to send" pid="..." error="write udp ... use of closed network connection" ``` comes from `cunicu.li/go-rosenpass` v0.5.42. NetBird v0.73.2 used that version, and v0.74.6, v0.76.0, and current main still use the same version. An initiator handshake schedules a self-rescheduling retransmission timer. In v0.5.42: - handshake expiry removes the handshake from the map without stopping that timer; - `initiatorHandshake.Close()` stops only the expiry timer; - the surviving timer retains the handshake, peer, server, handlers, and closed UDP connection while continuing to send roughly every 10 seconds. I reproduced this with a black-holed peer. At handshake expiry the test had recorded 8 sends; one second later it had reached 15. This behavior is covered by the existing upstream fix: https://codeberg.org/cunicu/go-rosenpass/pulls/231 During validation I found two additional orphan paths in the same lifecycle: - a retransmission timer is created before the initial send, so an initial send failure can leave an unregistered timer behind; - a pending peer rekey timer can fire after `Server.Close()` and start another handshake on the closed socket. Focused tests reproduced both cases. With [go-rosenpass PR #231](https://codeberg.org/cunicu/go-rosenpass/pulls/231)'s timer synchronization plus fixes for those two cases, all three tests pass under the Go race detector. This is a strong explanation for the continuing `Failed to send` traffic and retained memory. The increasing gRPC channel IDs probably indicate repeated NetBird engine generations, which would amplify the Rosenpass leak by repeatedly creating and closing servers. It may not explain every part of the incident by itself. Could you confirm whether quantum resistance/Rosenpass was enabled on the affected client? The `pid` log field strongly suggests that it was. The proposed contribution path is: 1. complete and merge the upstream go-rosenpass timer fix; 2. publish a new go-rosenpass tag; 3. submit a focused NetBird dependency-bump PR; 4. reproduce once more with a patched client and compare baseline versus post-blackhole heap/goroutine profiles. Debug bundles or profiles may contain sensitive deployment information and should be shared privately with the maintainers.
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#12286