[GH-ISSUE #6973] One peer stuck in handshake-retry loop grows RSS into the cgroup MemoryHigh cap; daemon wedges unresponsive for days while systemd reports active (0.74.6, linux/arm64) #13086

Open
opened 2026-08-05 02:07:25 -04:00 by saavagebueno · 1 comment
Owner

Originally created by @howdude230 on GitHub (Jul 29, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/6973

Summary

A single peer stuck in a WireGuard-handshake-timeout reconnect loop grew the client's RSS until it hit the systemd MemoryHigh= cap, at which point the kernel throttled it into permanent reclaim. The daemon stayed alive but stopped answering its own control socket, so netbird status returned:

Error: failed to connect to daemon error: context deadline exceeded

while systemctl still reported active (running). Because the process never died, nothing restarted it. It sat wedged for 5 days 8 hours.

Management, Signal, and both relays were reachable and healthy throughout, and other peers were connected normally. This is the part that distinguishes it from #6953, where growth happened because Management was unreachable — here the control plane was fine.

Distinction from #6953

#6953 this report
Trigger Management unreachable Management healthy; one peer failing handshake
Version 0.73.2 0.74.6
Invocation netbird up --foreground-mode systemd service, netbird service run
Growth 6.0 GB RSS, unbounded ~384 MB, bounded by cgroup cap
Terminal state host OOM, kernel kills unrelated workloads daemon wedged, unresponsive socket, no OOM, no restart

They may share a root cause (unbounded retry cost), but the observable failure and the reason it goes unnoticed are different, so filing separately.

Environment

NetBird client 0.74.6 (daemon + CLI)
OS RHEL-family Linux, linux/arm64
Deployment self-hosted management + 2 relays
Invocation netbird service run --daemon-addr unix:///var/run/netbird.sock --log-file /var/log/netbird/client.log
Peers in account 12
Interface type Kernel
Rosenpass enabled

systemd drop-in on the unit:

[Service]
CPUQuota=25%
MemoryMax=512M
MemoryHigh=384M
StandardOutput=null
StandardError=null

Timeline

Service start T+0
First handshake timeout vs peer B T+11h
Last timeout (manual restart) T+5d 19h
Duration of loop ~5d 8h continuous

Evidence

Warning volume, one peer dominating by 12×:

33,900   peer B   (the stuck peer)
 2,682   peer C
   482   peer D
   410   peer E

Distinct repeating messages over the window:

13,779 × client/internal/peer/conn.go:682       WireGuard handshake timeout detected, closing current connection
13,773 × client/internal/peer/wg_watcher.go:158 WireGuard handshake timed out
 4,735 × client/internal/peer/worker_ice.go:163 ICE Agent is not initialized yet

Rate was flat at ~112/hour for days — no backoff. The 4,735 ICE Agent is not initialized yet suggest the reconnect loop was outrunning its own ICE agent setup/teardown.

Resource state immediately before restart:

Memory: 383.7M (high: 384.0M  max: 512.0M  available: 256.0K  peak: 389.7M)
CPU: 4h 24min   Tasks: 8   uptime 5 days

cgroup showed no OOM at all — it was throttled, never killed:

oom 0
oom_kill 0

After restart (same host, same peers, same relays, same Rosenpass)

Memory: 66M
peer B: Status: Connected, Connection type: P2P, ICE host/host
handshake timeouts since restart: 0

So the network path was never blocked. State that accumulated inside the daemon was the fault, and it clears completely on restart — same "growth tracks time-spent-failing" behaviour noted in #6953.

Impact

This host is the DNS nameserver for the account. When its agent wedged, every other peer reported Nameservers: 0/1 Available and all internal names stopped resolving via getaddrinfo, while direct dig against a non-NetBird resolver still worked. That split made it look like a DNS or upstream-service outage rather than a client fault, which is why it went undiagnosed for 5 days.

What would have caught it

  1. A liveness check on the daemon's own control socket. systemd only supervises the process, and the process never exits. Anything that periodically proved the socket answers would have caught this in minutes.
  2. Backoff on repeated handshake failure for a single peer. 13.8k reconnect cycles at a flat rate for one unreachable peer is the whole cost driver.
  3. Bounded retry state, so a long-failing peer cannot grow RSS without limit.

Note for anyone hitting this

If you cap the unit with MemoryHigh=, be aware it produces a wedge rather than a crash — MemoryMax= alone would have OOM-killed and let Restart= recover it. Also note StandardOutput=null sends everything to --log-file, so journalctl -u netbird is empty and the evidence is only in the log file.

Hostnames, peer keys, IPs, and domains removed.

Originally created by @howdude230 on GitHub (Jul 29, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/6973 ## Summary A single peer stuck in a WireGuard-handshake-timeout reconnect loop grew the client's RSS until it hit the systemd `MemoryHigh=` cap, at which point the kernel throttled it into permanent reclaim. The daemon stayed alive but **stopped answering its own control socket**, so `netbird status` returned: ``` Error: failed to connect to daemon error: context deadline exceeded ``` while `systemctl` still reported `active (running)`. Because the process never died, nothing restarted it. **It sat wedged for 5 days 8 hours.** Management, Signal, and both relays were reachable and healthy throughout, and other peers were connected normally. This is the part that distinguishes it from #6953, where growth happened *because* Management was unreachable — here the control plane was fine. ## Distinction from #6953 | | #6953 | this report | |---|---|---| | Trigger | Management unreachable | Management healthy; one peer failing handshake | | Version | 0.73.2 | **0.74.6** | | Invocation | `netbird up --foreground-mode` | systemd service, `netbird service run` | | Growth | 6.0 GB RSS, unbounded | ~384 MB, bounded by cgroup cap | | Terminal state | host OOM, kernel kills unrelated workloads | **daemon wedged, unresponsive socket, no OOM, no restart** | They may share a root cause (unbounded retry cost), but the observable failure and the reason it goes unnoticed are different, so filing separately. ## Environment | | | |---|---| | NetBird client | **0.74.6** (daemon + CLI) | | OS | RHEL-family Linux, `linux/arm64` | | Deployment | self-hosted management + 2 relays | | Invocation | `netbird service run --daemon-addr unix:///var/run/netbird.sock --log-file /var/log/netbird/client.log` | | Peers in account | 12 | | Interface type | Kernel | | Rosenpass | enabled | systemd drop-in on the unit: ```ini [Service] CPUQuota=25% MemoryMax=512M MemoryHigh=384M StandardOutput=null StandardError=null ``` ## Timeline | | | |---|---| | Service start | T+0 | | First handshake timeout vs peer B | T+11h | | Last timeout (manual restart) | T+5d 19h | | Duration of loop | **~5d 8h continuous** | ## Evidence Warning volume, one peer dominating by 12×: ``` 33,900 peer B (the stuck peer) 2,682 peer C 482 peer D 410 peer E ``` Distinct repeating messages over the window: ``` 13,779 × client/internal/peer/conn.go:682 WireGuard handshake timeout detected, closing current connection 13,773 × client/internal/peer/wg_watcher.go:158 WireGuard handshake timed out 4,735 × client/internal/peer/worker_ice.go:163 ICE Agent is not initialized yet ``` Rate was flat at ~112/hour for days — **no backoff**. The 4,735 `ICE Agent is not initialized yet` suggest the reconnect loop was outrunning its own ICE agent setup/teardown. Resource state immediately before restart: ``` Memory: 383.7M (high: 384.0M max: 512.0M available: 256.0K peak: 389.7M) CPU: 4h 24min Tasks: 8 uptime 5 days ``` cgroup showed **no OOM at all** — it was throttled, never killed: ``` oom 0 oom_kill 0 ``` ## After restart (same host, same peers, same relays, same Rosenpass) ``` Memory: 66M peer B: Status: Connected, Connection type: P2P, ICE host/host handshake timeouts since restart: 0 ``` So the network path was never blocked. State that accumulated inside the daemon was the fault, and it clears completely on restart — same "growth tracks time-spent-failing" behaviour noted in #6953. ## Impact This host is the DNS nameserver for the account. When its agent wedged, every other peer reported `Nameservers: 0/1 Available` and all internal names stopped resolving via `getaddrinfo`, while direct `dig` against a non-NetBird resolver still worked. That split made it look like a DNS or upstream-service outage rather than a client fault, which is why it went undiagnosed for 5 days. ## What would have caught it 1. **A liveness check on the daemon's own control socket.** systemd only supervises the process, and the process never exits. Anything that periodically proved the socket answers would have caught this in minutes. 2. **Backoff on repeated handshake failure for a single peer.** 13.8k reconnect cycles at a flat rate for one unreachable peer is the whole cost driver. 3. **Bounded retry state**, so a long-failing peer cannot grow RSS without limit. ## Note for anyone hitting this If you cap the unit with `MemoryHigh=`, be aware it produces a *wedge* rather than a crash — `MemoryMax=` alone would have OOM-killed and let `Restart=` recover it. Also note `StandardOutput=null` sends everything to `--log-file`, so `journalctl -u netbird` is empty and the evidence is only in the log file. *Hostnames, peer keys, IPs, and domains removed.*
Author
Owner

@linear-code[bot] commented on GitHub (Jul 29, 2026):

NET-1456

<!-- gh-comment-id:5124580555 --> @linear-code[bot] commented on GitHub (Jul 29, 2026): <!-- linear-linkback --> <p><a href="https://linear.app/netbird/issue/NET-1456">NET-1456</a></p>
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#13086