[PR #6616] [client] Tear down idle lazy connections in kernel mode #29735

Open
opened 2026-08-05 08:08:49 -04:00 by saavagebueno · 0 comments
Owner

Original Pull Request: https://github.com/netbirdio/netbird/pull/6616

State: closed
Merged: No


Describe your changes

Lazy connections have two halves: activation (establish a peer on first traffic) and inactivity teardown (drop a peer once it goes idle). Activation already works in both userspace and kernel-WireGuard mode, but teardown was gated to userspace only:

  • lazyconn/manager.NewManager created the inactivity monitor only if wgIface.IsUserspaceBind(), otherwise logging "inactivity manager not supported for kernel mode, wait for remote peer to close the connection".
  • KernelConfigurer.LastActivities() returned nil, so the monitor never saw a kernel-mode peer as idle.

As a result a kernel-mode peer relied entirely on the remote peer's GO_IDLE. Two kernel-mode peers (e.g. two Linux hosts) would connect on demand and then stay up indefinitely, since neither side ever idled out. This becomes more visible as lazy connections move to default-on (#6571).

This PR gives kernel mode a real per-peer activity signal and removes the gate:

  • KernelConfigurer.LastActivities() now infers activity from the aggregate Tx/Rx byte counters already exposed via wgctrl (reusing GetStats()). A peer counts as active when its combined counter grows past a small threshold between polls. The baseline advances every poll so the 25s persistent keepalive can't accumulate into a false positive; a counter reset (peer re-added) counts as activity; newly seen peers are seeded active, mirroring the userspace ActivityRecorder.
  • The inactivity manager is now created unconditionally, so kernel mode reaches parity with userspace.

Why byte counters and not packet capture: in the linked discussion I initially floated tapping packets via AF_PACKET (before deleting it). On closer look that's the wrong tool for an always-on signal — it runs a continuous userspace read loop copying every decrypted overlay packet (negating kernel mode's performance benefit), is Linux-only (kernel mode also runs on FreeBSD), and shares the single debug-capture slot. The wgctrl counter dump is a once-a-minute netlink read the inactivity monitor already polls at, works on Linux + FreeBSD, and needs no extra privileges.

The change is split into two bisect-friendly commits: (1) add the kernel activity signal (no behavior change on its own), (2) remove the gate (activates teardown).

Threshold, validated on real hardware

Kernel mode only exposes aggregate counters, so the idle noise floor is handled with a fixed activityByteThreshold. I measured it on idle kernel-WireGuard peers (Teltonika TRB500, ARMv7, OpenWrt 21.02, kernel 4.14) by sampling per-peer wg show transfer every 60s: a keepalive-only poll grows the combined Tx+Rx counter by ~64 bytes, and a poll that contains a WireGuard rekey handshake (which recurs ~every 2 min, kept alive by the persistent keepalive itself) grows it by ~400 bytes. The rekey overhead is the non-obvious part — handshake messages land in the per-peer counters too, so the floor is higher than keepalives alone. The threshold is set to 1024 bytes/poll, comfortably above the measured ~400-byte ceiling and far below any real traffic. The per-poll baseline advances every poll so the floor can't accumulate across intervals. Happy to make it configurable if maintainers prefer.

Discussed on the default-on PR: https://github.com/netbirdio/netbird/pull/6571#issuecomment-4841303542
Related connection-mode RFCs: #5989, #5990

Stack

N/A

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)
  • This change does not modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — OR I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). — This changes kernel-mode behavior (idle lazy connections now tear down); raised on #6571 (link above) before submitting.

Documentation

Select exactly one:

  • I added/updated documentation for this change
  • Documentation is not needed for this change (explain why) — No code-repo docs assert "userspace only" for inactivity. If the netbirdio/docs site notes that lazy/idle teardown is userspace-only, that note should be dropped — flag if so and I'll open a docs PR.

Docs PR URL (required if "docs added" is checked)

N/A


Test plan (not part of template — for our review)

Verified in a golang:1.25.5 container:

  • gofmt -l clean on all three changed files
  • go vet ./client/iface/configurer/... ./client/internal/lazyconn/... — OK
  • go build of both package trees — OK
  • go test ./client/iface/configurer/... ./client/internal/lazyconn/... — all pass, including new table-driven tests for the tracker (new peer seeded active, growth > threshold → active, sub-threshold growth across polls → stays idle, counter reset → active, removed-peer pruning)

Validated against live hardware (kernel WireGuard, lazy enabled):

  • Sampled per-peer wg show transfer on idle TRB500 peers over 6 min → idle floor ~64 B/poll (keepalive) rising to ~400 B/poll on rekey polls; drove the activityByteThreshold = 1024 choice (see above). Real traffic on the same interface was orders of magnitude higher, cleanly separable.

Not yet done (needs deploying a patched build to two peers):

  • End-to-end: two kernel-WireGuard peers with lazy enabled + low NB_LAZY_CONN_INACTIVITY_THRESHOLD, confirming connect-on-traffic → teardown-on-idle → re-activate. (The current released agent in kernel mode keeps idle peers connected indefinitely — reproduced live: a P2P peer idle for 6+ min with only keepalive/rekey traffic stayed Connected.)
**Original Pull Request:** https://github.com/netbirdio/netbird/pull/6616 **State:** closed **Merged:** No --- ## Describe your changes Lazy connections have two halves: **activation** (establish a peer on first traffic) and **inactivity teardown** (drop a peer once it goes idle). Activation already works in both userspace and kernel-WireGuard mode, but teardown was gated to userspace only: - `lazyconn/manager.NewManager` created the inactivity monitor only `if wgIface.IsUserspaceBind()`, otherwise logging `"inactivity manager not supported for kernel mode, wait for remote peer to close the connection"`. - `KernelConfigurer.LastActivities()` returned `nil`, so the monitor never saw a kernel-mode peer as idle. As a result a kernel-mode peer relied entirely on the *remote* peer's `GO_IDLE`. Two kernel-mode peers (e.g. two Linux hosts) would connect on demand and then stay up indefinitely, since neither side ever idled out. This becomes more visible as lazy connections move to default-on (#6571). **This PR** gives kernel mode a real per-peer activity signal and removes the gate: - `KernelConfigurer.LastActivities()` now infers activity from the aggregate Tx/Rx byte counters already exposed via `wgctrl` (reusing `GetStats()`). A peer counts as active when its combined counter grows past a small threshold between polls. The baseline advances every poll so the 25s persistent keepalive can't accumulate into a false positive; a counter reset (peer re-added) counts as activity; newly seen peers are seeded active, mirroring the userspace `ActivityRecorder`. - The inactivity manager is now created unconditionally, so kernel mode reaches parity with userspace. **Why byte counters and not packet capture:** in the linked discussion I initially floated tapping packets via AF_PACKET (before deleting it). On closer look that's the wrong tool for an always-on signal — it runs a continuous userspace read loop copying every decrypted overlay packet (negating kernel mode's performance benefit), is Linux-only (kernel mode also runs on FreeBSD), and shares the single debug-capture slot. The `wgctrl` counter dump is a once-a-minute netlink read the inactivity monitor already polls at, works on Linux + FreeBSD, and needs no extra privileges. The change is split into two bisect-friendly commits: (1) add the kernel activity signal (no behavior change on its own), (2) remove the gate (activates teardown). ### Threshold, validated on real hardware Kernel mode only exposes aggregate counters, so the idle noise floor is handled with a fixed `activityByteThreshold`. I measured it on idle kernel-WireGuard peers (Teltonika TRB500, ARMv7, OpenWrt 21.02, kernel 4.14) by sampling per-peer `wg show transfer` every 60s: a keepalive-only poll grows the combined Tx+Rx counter by ~64 bytes, and a poll that contains a WireGuard rekey handshake (which recurs ~every 2 min, kept alive by the persistent keepalive itself) grows it by ~400 bytes. The rekey overhead is the non-obvious part — handshake messages land in the per-peer counters too, so the floor is higher than keepalives alone. The threshold is set to **1024 bytes/poll**, comfortably above the measured ~400-byte ceiling and far below any real traffic. The per-poll baseline advances every poll so the floor can't accumulate across intervals. Happy to make it configurable if maintainers prefer. ## Issue ticket number and link Discussed on the default-on PR: https://github.com/netbirdio/netbird/pull/6571#issuecomment-4841303542 Related connection-mode RFCs: #5989, #5990 ## Stack N/A ### Checklist - [x] Is it a bug fix - [ ] Is a typo/documentation fix - [x] Is a feature enhancement - [ ] It is a refactor - [x] Created tests that fail without the change (if possible) - [x] This change does **not** modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — **OR** I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). — *This changes kernel-mode behavior (idle lazy connections now tear down); raised on #6571 (link above) before submitting.* ## Documentation Select exactly one: - [ ] I added/updated documentation for this change - [x] Documentation is **not needed** for this change (explain why) — *No code-repo docs assert "userspace only" for inactivity. If the netbirdio/docs site notes that lazy/idle teardown is userspace-only, that note should be dropped — flag if so and I'll open a docs PR.* ### Docs PR URL (required if "docs added" is checked) N/A --- ## Test plan (not part of template — for our review) Verified in a `golang:1.25.5` container: - `gofmt -l` clean on all three changed files - `go vet ./client/iface/configurer/... ./client/internal/lazyconn/...` — OK - `go build` of both package trees — OK - `go test ./client/iface/configurer/... ./client/internal/lazyconn/...` — all pass, including new table-driven tests for the tracker (new peer seeded active, growth > threshold → active, sub-threshold growth across polls → stays idle, counter reset → active, removed-peer pruning) Validated against live hardware (kernel WireGuard, lazy enabled): - Sampled per-peer `wg show transfer` on idle TRB500 peers over 6 min → idle floor ~64 B/poll (keepalive) rising to ~400 B/poll on rekey polls; drove the `activityByteThreshold = 1024` choice (see above). Real traffic on the same interface was orders of magnitude higher, cleanly separable. Not yet done (needs deploying a patched build to two peers): - End-to-end: two kernel-WireGuard peers with lazy enabled + low `NB_LAZY_CONN_INACTIVITY_THRESHOLD`, confirming connect-on-traffic → teardown-on-idle → re-activate. (The current released agent in kernel mode keeps idle peers connected indefinitely — reproduced live: a P2P peer idle for 6+ min with only keepalive/rekey traffic stayed `Connected`.)
saavagebueno added the pull-request label 2026-08-05 08:08:49 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#29735