[PR #5857] [MERGED] [client] Replace WG interface monitor polling with netlink subscription on Linux #28654

Open
opened 2026-08-05 08:06:42 -04:00 by saavagebueno · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/netbirdio/netbird/pull/5857
Author: @alexsavio
Created: 4/11/2026
Status: Merged
Merged: 5/4/2026
Merged by: @lixmal

Base: mainHead: leakfix-wg-iface-monitor-netlink


📝 Commits (6)

  • f7341b5 [client] Replace WG interface monitor polling with netlink subscription on Linux
  • 962cd4a fix(client): address PR review comments on WG interface monitor
  • 371b8ce refactor(client): extract link-event inspection to satisfy SonarCloud
  • 7dad470 refactor(client): split RTM_DELLINK and RTM_NEWLINK handlers
  • c676043 fix(client): wrap context error with %w for proper error chain
  • 7133da0 fix(client): recover monitoring on LinkSubscribe failure

📊 Changes

3 files changed (+195 additions, -26 deletions)

View changed files

📝 client/internal/wg_iface_monitor.go (+5 -26)
client/internal/wg_iface_monitor_linux.go (+134 -0)
client/internal/wg_iface_monitor_other.go (+56 -0)

📄 Description

Describe your changes

The WireGuard interface monitor introduced in #4370 spawns a 2 s ticker that calls net.InterfaceByName(ifaceName) on every tick. On Linux, that function issues syscall.NetlinkRIB(RTM_GETLINK, ...) and dumps the entire kernel link table on every call, then linear-scans it for the matching name. On hosts with many veth interfaces (Docker, containerd, k8s) the per-call cost is dozens of KB and the ticker generates roughly 1 GB/day of allocation churn from this single source. On long-running clients the GC pressure plus span fragmentation manifests as a slow, monotonic RSS climb.

This is one of the persistent leak sources behind #3678.

Smoking gun

Observed on a Raspberry Pi (4 GB, ARM64) running v0.68.1: netbird RSS climbed from 60 MB to 1.84 GB over ~2 days (~920 MB/day), eventually starving every other service on the host.

A pprof allocs.prof from a freshly-restarted v0.68.1 daemon (~20 minutes uptime) shows the WG interface monitor as the single largest allocator at 513 MB / 29% of all allocations:

File: netbird
Type: alloc_space
Showing nodes accounting for 1633951.63kB, 92.81% of 1760619.57kB total
      flat  flat%   sum%        cum   cum%
 7174.75kB  0.41%  0.41% 593081.62kB 33.69%  net.interfaceTable
         0     0%  0.41% 513712.69kB 29.18%  github.com/netbirdio/netbird/client/internal.(*Engine).Start.func3
         0     0%  0.41% 513712.69kB 29.18%  github.com/netbirdio/netbird/client/internal.(*WGIfaceMonitor).Start
         0     0%  0.41% 513712.69kB 29.18%  github.com/netbirdio/netbird/client/internal.getInterfaceIndex
 1024.06kB 0.058%  0.47% 513712.69kB 29.18%  net.InterfaceByName
391640.27kB 22.24% 22.71% 421874.18kB 23.96%  syscall.NetlinkRIB
217867.56kB 12.37% 54.85% 217867.56kB 12.37%  syscall.ParseNetlinkRouteAttr

Math: 86,400 s/day ÷ 2 s = 43,200 calls/day × ~30 KB per call = ~1.3 GB/day allocated by this one ticker. That matches the observed leak rate.

Fix

Split the watcher by build tag.

  • wg_iface_monitor_linux.go subscribes to RTNLGRP_LINK via netlink.LinkSubscribe (already a transitive dependency through vishvananda/netlink) and reacts to RTM_DELLINK / RTM_NEWLINK events for the tracked interface index. Allocations between events drop to zero. The same pattern is already used by client/internal/networkmonitor/check_change_linux.go for route events, so the dependency, idiom, and review surface are familiar.

  • wg_iface_monitor_other.go keeps the original 2 s polling loop for darwin / windows / freebsd / android / ios. No behavior change on those platforms — they do not exhibit the NetlinkRIB cost (darwin uses sysctl(NET_RT_IFLIST2), windows uses GetIfTable).

  • wg_iface_monitor.go keeps the shared WGIfaceMonitor type, the early-return checks (mobile / netstack / empty name) and getInterfaceIndex, then dispatches to the platform-specific watchInterface.

A small race window between the initial getInterfaceIndex call in Start and LinkSubscribe completing its handshake is closed by re-checking the index after subscribing.

Follow-up

Windows is also reported as affected in #3678 (multi-GB peaks on Server 2016/2019). The same event-driven treatment can be applied there using NotifyIpInterfaceChange from iphlpapi. Left as a follow-up so this PR stays focused on the worst offender (Linux) with the smallest possible diff.

Refs: #3678
Original feature: #4370

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)
  • Extended the README / documentation, if necessary

Verification

Cross-compiled clean for linux/{amd64,arm64}, darwin/{amd64,arm64}, and windows/amd64 against main. The patched binary has been running on the affected Pi for the last hour with no functional regressions; an extended-duration RSS comparison will be added once a meaningful sample is available.

Summary by CodeRabbit

  • Refactor
    • Interface monitoring now uses event-driven detection on Linux for more efficient, lower-overhead updates; other platforms continue to use a low-frequency polling fallback to detect interface recreation or deletion.
  • Documentation
    • Expanded docs describing platform-specific monitoring behavior, restart triggers, and stop conditions for improved clarity.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/netbirdio/netbird/pull/5857 **Author:** [@alexsavio](https://github.com/alexsavio) **Created:** 4/11/2026 **Status:** ✅ Merged **Merged:** 5/4/2026 **Merged by:** [@lixmal](https://github.com/lixmal) **Base:** `main` ← **Head:** `leakfix-wg-iface-monitor-netlink` --- ### 📝 Commits (6) - [`f7341b5`](https://github.com/netbirdio/netbird/commit/f7341b5dcd46a2e8192cf4472e6bd1e2eb0841af) [client] Replace WG interface monitor polling with netlink subscription on Linux - [`962cd4a`](https://github.com/netbirdio/netbird/commit/962cd4a60325e715027658999880f076513241cb) fix(client): address PR review comments on WG interface monitor - [`371b8ce`](https://github.com/netbirdio/netbird/commit/371b8ce06811371d957b57d6cb4be9dae1fe38df) refactor(client): extract link-event inspection to satisfy SonarCloud - [`7dad470`](https://github.com/netbirdio/netbird/commit/7dad47076d61a0260e8fda4ffe28a918757535d0) refactor(client): split RTM_DELLINK and RTM_NEWLINK handlers - [`c676043`](https://github.com/netbirdio/netbird/commit/c67604393007160bfac5868ec0eb30247f098343) fix(client): wrap context error with %w for proper error chain - [`7133da0`](https://github.com/netbirdio/netbird/commit/7133da0e213c8bba2663d1fab73a191c3aaa532e) fix(client): recover monitoring on LinkSubscribe failure ### 📊 Changes **3 files changed** (+195 additions, -26 deletions) <details> <summary>View changed files</summary> 📝 `client/internal/wg_iface_monitor.go` (+5 -26) ➕ `client/internal/wg_iface_monitor_linux.go` (+134 -0) ➕ `client/internal/wg_iface_monitor_other.go` (+56 -0) </details> ### 📄 Description ## Describe your changes The WireGuard interface monitor introduced in #4370 spawns a 2 s ticker that calls `net.InterfaceByName(ifaceName)` on every tick. On Linux, that function issues `syscall.NetlinkRIB(RTM_GETLINK, ...)` and dumps the entire kernel link table on every call, then linear-scans it for the matching name. On hosts with many veth interfaces (Docker, containerd, k8s) the per-call cost is dozens of KB and the ticker generates roughly **1 GB/day of allocation churn from this single source**. On long-running clients the GC pressure plus span fragmentation manifests as a slow, monotonic RSS climb. This is one of the persistent leak sources behind #3678. ### Smoking gun Observed on a Raspberry Pi (4 GB, ARM64) running v0.68.1: netbird RSS climbed from 60 MB to 1.84 GB over ~2 days (~920 MB/day), eventually starving every other service on the host. A pprof `allocs.prof` from a freshly-restarted v0.68.1 daemon (~20 minutes uptime) shows the WG interface monitor as the **single largest allocator** at 513 MB / 29% of all allocations: ``` File: netbird Type: alloc_space Showing nodes accounting for 1633951.63kB, 92.81% of 1760619.57kB total flat flat% sum% cum cum% 7174.75kB 0.41% 0.41% 593081.62kB 33.69% net.interfaceTable 0 0% 0.41% 513712.69kB 29.18% github.com/netbirdio/netbird/client/internal.(*Engine).Start.func3 0 0% 0.41% 513712.69kB 29.18% github.com/netbirdio/netbird/client/internal.(*WGIfaceMonitor).Start 0 0% 0.41% 513712.69kB 29.18% github.com/netbirdio/netbird/client/internal.getInterfaceIndex 1024.06kB 0.058% 0.47% 513712.69kB 29.18% net.InterfaceByName 391640.27kB 22.24% 22.71% 421874.18kB 23.96% syscall.NetlinkRIB 217867.56kB 12.37% 54.85% 217867.56kB 12.37% syscall.ParseNetlinkRouteAttr ``` Math: 86,400 s/day ÷ 2 s = 43,200 calls/day × ~30 KB per call = ~1.3 GB/day allocated by this one ticker. That matches the observed leak rate. ## Fix Split the watcher by build tag. * **`wg_iface_monitor_linux.go`** subscribes to `RTNLGRP_LINK` via `netlink.LinkSubscribe` (already a transitive dependency through `vishvananda/netlink`) and reacts to `RTM_DELLINK` / `RTM_NEWLINK` events for the tracked interface index. **Allocations between events drop to zero.** The same pattern is already used by `client/internal/networkmonitor/check_change_linux.go` for route events, so the dependency, idiom, and review surface are familiar. * **`wg_iface_monitor_other.go`** keeps the original 2 s polling loop for darwin / windows / freebsd / android / ios. **No behavior change** on those platforms — they do not exhibit the `NetlinkRIB` cost (darwin uses `sysctl(NET_RT_IFLIST2)`, windows uses `GetIfTable`). * **`wg_iface_monitor.go`** keeps the shared `WGIfaceMonitor` type, the early-return checks (mobile / netstack / empty name) and `getInterfaceIndex`, then dispatches to the platform-specific `watchInterface`. A small race window between the initial `getInterfaceIndex` call in `Start` and `LinkSubscribe` completing its handshake is closed by re-checking the index after subscribing. ### Follow-up Windows is also reported as affected in #3678 (multi-GB peaks on Server 2016/2019). The same event-driven treatment can be applied there using `NotifyIpInterfaceChange` from `iphlpapi`. Left as a follow-up so this PR stays focused on the worst offender (Linux) with the smallest possible diff. ## Issue ticket number and link Refs: #3678 Original feature: #4370 ## Checklist - [x] Is it a bug fix - [ ] Is a typo/documentation fix - [ ] Is a feature enhancement - [ ] It is a refactor - [ ] Created tests that fail without the change (if possible) - [x] Extended the README / documentation, if necessary ## Verification Cross-compiled clean for `linux/{amd64,arm64}`, `darwin/{amd64,arm64}`, and `windows/amd64` against `main`. The patched binary has been running on the affected Pi for the last hour with no functional regressions; an extended-duration RSS comparison will be added once a meaningful sample is available. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Refactor** * Interface monitoring now uses event-driven detection on Linux for more efficient, lower-overhead updates; other platforms continue to use a low-frequency polling fallback to detect interface recreation or deletion. * **Documentation** * Expanded docs describing platform-specific monitoring behavior, restart triggers, and stop conditions for improved clarity. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
saavagebueno added the pull-request label 2026-08-05 08:06:42 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#28654