[PR #6084] [client, management] Phase 3.7i hardening + activity-trigger fast-path + Codex-review fixes — closes #5989 (stack 4/4) #27515

Open
opened 2026-08-05 07:08:50 -04:00 by saavagebueno · 0 comments
Owner

Original Pull Request: https://github.com/netbirdio/netbird/pull/6084

State: open
Merged: No


[client+management] Phase 3.7i hardening + activity-trigger fast-path + Codex-review fixes (closes #5989)

Branch: pr/d-hardening-activity-trigger → base pr/c-phase3.7i-visibility (stacked PR — depends on PR-C landing first)
Compare: https://github.com/netbirdio/netbird/compare/MichaelUray:netbird:pr/c-phase3.7i-visibility...MichaelUray:netbird:pr/d-hardening-activity-trigger?expand=1


Summary

This PR completes #5989. It adds the activity-trigger fast-path (relay-state activity from the WG ActivityRecorder re-attaches ICE so a peer that fell to relay during quiet periods upgrades back to P2P on the next real packet), a rate-limited backoff override (real user activity is the strongest "I want this peer back" signal we have, allowed once per 5 minutes), the Guard activity-driven retry-budget reset that Codex-review identified as the missing piece for srflx-only peers after Idle wake, plus a substantial round of hardening: legacy-fallback (older clients without the new ConnectionMode field cleanly map to p2p-lazy), peer-visibility for the user RBAC role, the closed-PR pgx fix, the keepWgPeer routed-subnet fix, and several Codex-review-2/3 corrections.

This is the fourth and final stacked PR implementing #5989.

Why

After PR-C landed the visibility layer, real-world testing on 4-peer testbed (Windows 11 daemon + 3 OpenWrt routers spread across LAN, Internet, LTE) revealed two specific lifecycle gaps:

  1. Relay-state activity didn't trigger ICE re-attach. A peer that was P2P, fell to relay after the ICE-inactivity timer fired, then received fresh user traffic — stayed on relay. The ActivityRecorder saw the packet, but no callback re-attached ICE. The fast-path here closes that loop: WG bind callback → Conn.AttachICEOnRelayActivity → register a fresh ICE listener with the handshaker.

  2. After Idle wake the per-peer guard's 3-tries-then-hourly retry budget got consumed in the first ~5 seconds of pair-checking. For peers with non-LAN candidates (srflx Internet, LTE) cold mappings need several attempts to prime; the guard then declared the peer "exhausted" and escalated to hourly retry. Subsequent real user activity found the guard already in hourly mode and waited up to an hour before retrying.

Codex review identified the second one as a real PR-blocker: the iceBackoff state and the guard's retry counter are two parallel throttles, and only iceBackoff was reset on activity. The fix is Guard.NotifyPeerActivity() — a buffered channel signalled by user/transport activity, the reconnect loop treats it like a connection-disconnected event (ticker stop, fresh ticker, iceState.reset).

What's in this PR

Activity-trigger fast-path (the new feature)

  • client/iface/bind/activity: extend ActivityRecorder with an optional OnActivity callback; per-peer PublicKey field on PeerRecord so the dispatch knows which peer fired.
  • client/internal/lazyconn: AttachICE on activity wake-up so the lazy-mgr's "fake-IP packet detected" path actually drives a fresh ICE re-attach, not just an Open() call.
  • client/lazyconn+peer: ResetIceBackoff on activity-trigger wake. Updates the StatusRecorder snapshot after reset (Codex-review follow-up).
  • client/iface+peer+engine: relay-state ICE re-attach fast-path — engine wires ActivityRecorder.OnActivityConn.AttachICEOnRelayActivity(). Gates: mode = p2p-dynamic, conn open, current priority = Relay, listener detached, never-connected guard, activity-override rate limit.
  • client/peer: rate-limited backoff override on relay-state activity. iceBackoff.AllowActivityOverride() allows ONE override per 5 minutes per peer; this is the rate limit that protects the signal server from amplification while still unsticking peers on a transient pair-check drop.
  • client/peer: markSuccess stamps lastResetAt (Codex review caught: previously it didn't, so the rate-limit window never engaged).
  • client/peer: clarify backoff-intent comments + classify failure types (first-attempt, post-success-drop, re-attach) for log diagnostics + invariant test.
  • client/peer/guard: NotifyPeerActivity() + activity-driven reset of ICE retry budget — the headline Codex-review fix. New buffered channel; the reconnect loop treats it like SetICEConnDisconnected (ticker stop, fresh newReconnectTicker, iceState.reset). Conn.AttachICEOnRelayActivity and lazyconn.manager.onPeerActivity both call it.

Hardening / Codex review rounds 2+3

  • mgmt/types+store: LegacyLazyFallback{Enabled,TimeoutSeconds} settings.
  • mgmt/conversion: legacy-client p2p-dynamicp2p-lazy fallback when remote daemon's SupportedFeatures doesn't include the new flag.
  • proto+client+mgmt: SupportedFeatures capability advertisement on the wire.
  • client/internal: latch conn_state_pusher disabled on Unimplemented (older mgmt servers).
  • mgmt/http+activity: expose LegacyLazyFallback settings via API.
  • client/internal: defensive nil-receiver guard on pusher entry points.
  • mgmt: legacy-fallback defaults consistent across all construction paths (test+prod+migration).
  • mgmt/peer: restore policy-aware peer visibility for user role — without this, a non-admin user could not see the new peer-status counters because the visibility check was based on the old, pre-policy-aware code path.
  • client/ui: silent auto-refresh on Networks window when daemon IPC drops.
  • client/ui: peer-detail + network-range text selectable + copyable.

Closed-PR-style quick fixes (could have been their own PRs but depend on Phase-3.7i fields)

  • mgmt/store: pgx getPeers must SELECT meta_supported_features + meta_effective_* — without this the SQL row-marshal silently dropped the new columns and RemotePeerConfig came across as zero-values.
  • client/stdnet: case-insensitive ICE interface filter (Windows P2P fix). (NOTE: this is also #PR-Q2 — separate standalone PR for early review. If Q2 lands first, this commit can be cherry-pick-dropped here.)
  • fix: keep WG peer entry across lazy-suspend so routed-subnet AllowedIPs survive — without this the route-manager's appended AllowedIPs got discarded on the lazy deactivate path, and the next activity-wake re-opened the conn with the original (subset) AllowedIPs.

Round-2 follow-ups

  • codex-review fixes: settings push, RelayServer materiality, uint32 validation, offline debounce.
  • codex review round 2: debounce safety, pre-init drain, dashboard cache, docs, tests.
  • hardening: explicit cancel hooks + handover-order regression tests.
  • hardening: reconnect-guard inactivity-skip + UI ICE-backoff fields.
  • codex review: 4 findings — server build, store epoch, meta notify, ICE-backoff display.
  • codex follow-up: session_id epoch field for unary-RPC stale-delta safety.
  • fix: gate guard skip-offer on everConnected (regression from Item 1).
  • fix: WG-handshake-timeout recovery — push peer back to lazy-idle.
  • client/lazyconn: IsSupported also accepts 0.0.0-dev-… semver-padded form (so dev builds with 0.0.0-dev-<sha> versions are recognised as lazy-capable).
  • client/engine: nil-guard connStatePusher closures during shutdown.
  • client: reconnect-guard p2p-dynamic-aware + proactive close on remote-offline.
  • client: hybrid "Relayed (negotiating P2P)" UI label during wakeup window.
  • client/ui (Win): colored status swatch on each peer row.
  • client: Android refresh wg-stats on PeersList + bump default relay-timeout to 24h.
  • peer-status: live_online from peer.Status.Connected for accurate counter.

Codex post-rebase fixes (the very latest)

  • client/cmd/testutil_test.go: nbgrpc.NewServer call updated to current 13-arg signature (peer_connections.Store + *peer_connections.SnapshotRouter).
  • .gitignore: management/netbird-mgmt ignored. (A 51 MB ELF was accidentally committed in an earlier iteration of the relay-state ICE re-attach commit; removed from this branch via interactive rebase — verified no commit in this branch contains the binary blob.)
  • ResetIceBackoff: stamp the cleared snapshot into StatusRecorder so netbird status -d and the daemon RPC stop advertising stale "suspended"/"Failures=N" after reset.

Tests

  • go test ./client/internal/peer ./client/internal/peer/guard ./client/internal/lazyconn/... — pass.
  • go build ./client/... ./management/... — pass on linux/amd64, linux/arm64, windows/amd64.
  • go test -c ./client/cmd — compiles. (Full test run blocked by sandbox /etc/systemd permissions, not a code-level regression.)
  • Hardware-validated lifecycle test on 4-peer testbed:
    • All 5 transitions covered: Idle → P2P (cold start, 30 s), P2P → Relayed (4 min idle), Relayed → P2P (B→A fast-path, 5–10 s), Relayed → Idle (9 min idle, full close), Idle → P2P (C→A wake, 3 s for srflx-only peers).
    • Pre-Phase-3.7i activity-trigger fix: srflx-only peers (Internet, LTE) stuck on relay after Idle wake for >30 minutes due to guard hourly-mode.
    • With this PR: all 3 BM peers (LAN host, Internet srflx, LTE srflx) reach P2P binnen 30 s after Idle wake; ICE-backoff state shows correctly cleared after reset.

Known flake (not blocker)

TestICEBind_HandlesConcurrentMixedTraffic in client/iface/bind is upstream-owned (last touched in #5953) and reproducibly fails under -count=10 or -race on local UDP-loopback when IPv6 delivery dips below the threshold. Verified by git diff upstream/main -- client/iface/bind/ice_bind_test.go returning empty: no test code changes here. Not addressed in this PR; would be a separate stabilisation PR for the test owner.

Test plan

  • All Phase-3.7i unit tests (peer, guard, lazyconn/*).
  • iceBackoff full suite incl. activity-override + grace + markSuccess-stamps-lastResetAt + Reset-clears-hourly invariants.
  • Cherry-pick clean against upstream/main (no untracked binaries; .gitignore corrected).
  • Maintainer review of the activity-trigger gate ordering in Conn.AttachICEOnRelayActivity (mode + opened + currentConnPriority + listener-nil + iceBackoff override + everConnected — comments document each).
  • Maintainer review of the legacy-fallback path: a daemon without SupportedFeatures set must cleanly degrade to p2p-lazy (regression-tested in mgmt/grpc).

Use case

The full p2p-dynamic experience: a fleet of intermittently-active peers (mix of LAN, Internet srflx, LTE srflx). They start lazy. On real user traffic they go P2P. On idle they detach ICE then close. On the next packet they re-open and re-establish P2P — always within a couple of seconds, regardless of whether the candidate type is host or srflx-only. ICE-backoff is bounded so a chronically-broken NAT path doesn't spam the signal server, but real user activity always overrides once per 5 minutes.

Linked work

  • Stacks on PR-C.
  • Closes #5989.
  • Subsumes the closed PR #5805's intent (already addressed in PR-B's offer-skip-during-connecting fix).
  • Subsumes (and extends) the standalone PR-Q2 (Windows ICE filter case-insensitive); if Q2 lands first the equivalent commit here can be dropped.

Maintainers are welcome to push directly to this branch.

Summary by CodeRabbit

  • New Features

    • Connection mode chooser (P2P, Relay‑Forced, P2P‑Lazy, P2P‑Dynamic) with editable relay/P2P timeouts and retry limits, plus server‑pushed defaults and profile/CLI overrides.
    • On‑demand per‑peer connection snapshots and live connection‑map pushes.
    • ICE backoff, richer per‑peer telemetry, and new aggregate peer counters surfaced in status.
  • Bug Fixes

    • Restored policy‑based peer visibility for non‑admin users.
  • UI Improvements

    • New Peers tab with detailed per‑peer views and controls to view/set connection mode and timeouts.

Documentation

  • Documentation is not needed

These changes are internal lifecycle / behavioural improvements; no user-visible API or CLI flag added that warrants new public docs. Existing flags/Settings already documented at netbirdio/docs cover the surface area.

**Original Pull Request:** https://github.com/netbirdio/netbird/pull/6084 **State:** open **Merged:** No --- # [client+management] Phase 3.7i hardening + activity-trigger fast-path + Codex-review fixes (closes #5989) **Branch:** `pr/d-hardening-activity-trigger` → base **`pr/c-phase3.7i-visibility`** (stacked PR — depends on PR-C landing first) **Compare:** https://github.com/netbirdio/netbird/compare/MichaelUray:netbird:pr/c-phase3.7i-visibility...MichaelUray:netbird:pr/d-hardening-activity-trigger?expand=1 --- ## Summary This PR completes #5989. It adds the **activity-trigger fast-path** (relay-state activity from the WG `ActivityRecorder` re-attaches ICE so a peer that fell to relay during quiet periods upgrades back to P2P on the next real packet), a **rate-limited backoff override** (real user activity is the strongest "I want this peer back" signal we have, allowed once per 5 minutes), the **Guard activity-driven retry-budget reset** that Codex-review identified as the missing piece for srflx-only peers after Idle wake, plus a substantial round of hardening: legacy-fallback (older clients without the new `ConnectionMode` field cleanly map to `p2p-lazy`), peer-visibility for the user RBAC role, the closed-PR pgx fix, the keepWgPeer routed-subnet fix, and several Codex-review-2/3 corrections. This is the fourth and final stacked PR implementing #5989. ## Why After PR-C landed the visibility layer, real-world testing on 4-peer testbed (Windows 11 daemon + 3 OpenWrt routers spread across LAN, Internet, LTE) revealed two specific lifecycle gaps: 1. **Relay-state activity didn't trigger ICE re-attach.** A peer that was P2P, fell to relay after the ICE-inactivity timer fired, then received fresh user traffic — stayed on relay. The `ActivityRecorder` saw the packet, but no callback re-attached ICE. The fast-path here closes that loop: WG bind callback → `Conn.AttachICEOnRelayActivity` → register a fresh ICE listener with the handshaker. 2. **After Idle wake the per-peer guard's 3-tries-then-hourly retry budget got consumed in the first ~5 seconds of pair-checking.** For peers with non-LAN candidates (srflx Internet, LTE) cold mappings need several attempts to prime; the guard then declared the peer "exhausted" and escalated to hourly retry. Subsequent real user activity found the guard already in hourly mode and waited up to an hour before retrying. Codex review identified the second one as a real PR-blocker: the iceBackoff state and the guard's retry counter are two parallel throttles, and only iceBackoff was reset on activity. The fix is `Guard.NotifyPeerActivity()` — a buffered channel signalled by user/transport activity, the reconnect loop treats it like a connection-disconnected event (ticker stop, fresh ticker, `iceState.reset`). ## What's in this PR ### Activity-trigger fast-path (the new feature) - `client/iface/bind/activity`: extend `ActivityRecorder` with an optional `OnActivity` callback; per-peer `PublicKey` field on `PeerRecord` so the dispatch knows which peer fired. - `client/internal/lazyconn`: `AttachICE` on activity wake-up so the lazy-mgr's "fake-IP packet detected" path actually drives a fresh ICE re-attach, not just an Open() call. - `client/lazyconn+peer`: `ResetIceBackoff` on activity-trigger wake. Updates the StatusRecorder snapshot after reset (Codex-review follow-up). - `client/iface+peer+engine`: relay-state ICE re-attach fast-path — engine wires `ActivityRecorder.OnActivity` → `Conn.AttachICEOnRelayActivity()`. Gates: mode = `p2p-dynamic`, conn open, current priority = Relay, listener detached, never-connected guard, activity-override rate limit. - `client/peer`: rate-limited backoff override on relay-state activity. `iceBackoff.AllowActivityOverride()` allows ONE override per 5 minutes per peer; this is the rate limit that protects the signal server from amplification while still unsticking peers on a transient pair-check drop. - `client/peer`: `markSuccess` stamps `lastResetAt` (Codex review caught: previously it didn't, so the rate-limit window never engaged). - `client/peer`: clarify backoff-intent comments + classify failure types (`first-attempt`, `post-success-drop`, `re-attach`) for log diagnostics + invariant test. - `client/peer/guard`: **`NotifyPeerActivity()` + activity-driven reset of ICE retry budget** — the headline Codex-review fix. New buffered channel; the reconnect loop treats it like SetICEConnDisconnected (ticker stop, fresh `newReconnectTicker`, `iceState.reset`). `Conn.AttachICEOnRelayActivity` and `lazyconn.manager.onPeerActivity` both call it. ### Hardening / Codex review rounds 2+3 - `mgmt/types+store`: `LegacyLazyFallback{Enabled,TimeoutSeconds}` settings. - `mgmt/conversion`: legacy-client `p2p-dynamic` → `p2p-lazy` fallback when remote daemon's `SupportedFeatures` doesn't include the new flag. - `proto+client+mgmt`: `SupportedFeatures` capability advertisement on the wire. - `client/internal`: latch `conn_state_pusher` disabled on `Unimplemented` (older mgmt servers). - `mgmt/http+activity`: expose `LegacyLazyFallback` settings via API. - `client/internal`: defensive nil-receiver guard on pusher entry points. - `mgmt`: legacy-fallback defaults consistent across all construction paths (test+prod+migration). - `mgmt/peer`: **restore policy-aware peer visibility for user role** — without this, a non-admin user could not see the new peer-status counters because the visibility check was based on the old, pre-policy-aware code path. - `client/ui`: silent auto-refresh on Networks window when daemon IPC drops. - `client/ui`: peer-detail + network-range text selectable + copyable. ### Closed-PR-style quick fixes (could have been their own PRs but depend on Phase-3.7i fields) - `mgmt/store`: **pgx `getPeers` must SELECT `meta_supported_features` + `meta_effective_*`** — without this the SQL row-marshal silently dropped the new columns and `RemotePeerConfig` came across as zero-values. - `client/stdnet`: case-insensitive ICE interface filter (Windows P2P fix). *(NOTE: this is also #PR-Q2 — separate standalone PR for early review. If Q2 lands first, this commit can be cherry-pick-dropped here.)* - `fix`: keep WG peer entry across lazy-suspend so routed-subnet AllowedIPs survive — without this the route-manager's appended AllowedIPs got discarded on the lazy deactivate path, and the next activity-wake re-opened the conn with the original (subset) AllowedIPs. ### Round-2 follow-ups - `codex-review fixes`: settings push, RelayServer materiality, uint32 validation, offline debounce. - `codex review round 2`: debounce safety, pre-init drain, dashboard cache, docs, tests. - `hardening`: explicit cancel hooks + handover-order regression tests. - `hardening`: reconnect-guard inactivity-skip + UI ICE-backoff fields. - `codex review`: 4 findings — server build, store epoch, meta notify, ICE-backoff display. - `codex follow-up`: session_id epoch field for unary-RPC stale-delta safety. - `fix`: gate guard skip-offer on `everConnected` (regression from Item 1). - `fix`: WG-handshake-timeout recovery — push peer back to lazy-idle. - `client/lazyconn`: `IsSupported` also accepts `0.0.0-dev-…` semver-padded form (so dev builds with `0.0.0-dev-<sha>` versions are recognised as lazy-capable). - `client/engine`: nil-guard `connStatePusher` closures during shutdown. - `client`: reconnect-guard p2p-dynamic-aware + proactive close on remote-offline. - `client`: hybrid "Relayed (negotiating P2P)" UI label during wakeup window. - `client/ui (Win)`: colored status swatch on each peer row. - `client`: Android refresh wg-stats on PeersList + bump default relay-timeout to 24h. - `peer-status`: `live_online` from `peer.Status.Connected` for accurate counter. ### Codex post-rebase fixes (the very latest) - `client/cmd/testutil_test.go`: `nbgrpc.NewServer` call updated to current 13-arg signature (`peer_connections.Store` + `*peer_connections.SnapshotRouter`). - `.gitignore`: `management/netbird-mgmt` ignored. (A 51 MB ELF was accidentally committed in an earlier iteration of the relay-state ICE re-attach commit; removed from this branch via interactive rebase — verified no commit in this branch contains the binary blob.) - `ResetIceBackoff`: stamp the cleared snapshot into `StatusRecorder` so `netbird status -d` and the daemon RPC stop advertising stale "suspended"/"Failures=N" after reset. ## Tests - `go test ./client/internal/peer ./client/internal/peer/guard ./client/internal/lazyconn/...` — pass. - `go build ./client/... ./management/...` — pass on linux/amd64, linux/arm64, windows/amd64. - `go test -c ./client/cmd` — compiles. (Full test run blocked by sandbox `/etc/systemd` permissions, not a code-level regression.) - **Hardware-validated lifecycle test on 4-peer testbed:** - All 5 transitions covered: Idle → P2P (cold start, 30 s), P2P → Relayed (4 min idle), Relayed → P2P (B→A fast-path, 5–10 s), Relayed → Idle (9 min idle, full close), Idle → P2P (C→A wake, 3 s for srflx-only peers). - Pre-Phase-3.7i activity-trigger fix: srflx-only peers (Internet, LTE) stuck on relay after Idle wake for >30 minutes due to guard hourly-mode. - With this PR: all 3 BM peers (LAN host, Internet srflx, LTE srflx) reach P2P binnen 30 s after Idle wake; ICE-backoff state shows correctly cleared after reset. ## Known flake (not blocker) `TestICEBind_HandlesConcurrentMixedTraffic` in `client/iface/bind` is upstream-owned (last touched in #5953) and reproducibly fails under `-count=10` or `-race` on local UDP-loopback when IPv6 delivery dips below the threshold. Verified by `git diff upstream/main -- client/iface/bind/ice_bind_test.go` returning empty: no test code changes here. Not addressed in this PR; would be a separate stabilisation PR for the test owner. ## Test plan - [x] All Phase-3.7i unit tests (peer, guard, lazyconn/*). - [x] iceBackoff full suite incl. activity-override + grace + markSuccess-stamps-lastResetAt + Reset-clears-hourly invariants. - [x] Cherry-pick clean against `upstream/main` (no untracked binaries; .gitignore corrected). - [ ] Maintainer review of the activity-trigger gate ordering in `Conn.AttachICEOnRelayActivity` (mode + opened + currentConnPriority + listener-nil + iceBackoff override + everConnected — comments document each). - [ ] Maintainer review of the legacy-fallback path: a daemon without `SupportedFeatures` set must cleanly degrade to `p2p-lazy` (regression-tested in `mgmt/grpc`). ## Use case The full p2p-dynamic experience: a fleet of intermittently-active peers (mix of LAN, Internet srflx, LTE srflx). They start lazy. On real user traffic they go P2P. On idle they detach ICE then close. On the next packet they re-open and re-establish P2P — always within a couple of seconds, regardless of whether the candidate type is host or srflx-only. ICE-backoff is bounded so a chronically-broken NAT path doesn't spam the signal server, but real user activity always overrides once per 5 minutes. ## Linked work - Stacks on PR-C. - Closes #5989. - Subsumes the closed PR #5805's intent (already addressed in PR-B's offer-skip-during-connecting fix). - Subsumes (and extends) the standalone PR-Q2 (Windows ICE filter case-insensitive); if Q2 lands first the equivalent commit here can be dropped. **Maintainers are welcome to push directly to this branch.** <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Connection mode chooser (P2P, Relay‑Forced, P2P‑Lazy, P2P‑Dynamic) with editable relay/P2P timeouts and retry limits, plus server‑pushed defaults and profile/CLI overrides. * On‑demand per‑peer connection snapshots and live connection‑map pushes. * ICE backoff, richer per‑peer telemetry, and new aggregate peer counters surfaced in status. * **Bug Fixes** * Restored policy‑based peer visibility for non‑admin users. * **UI Improvements** * New Peers tab with detailed per‑peer views and controls to view/set connection mode and timeouts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> ## Documentation - [x] Documentation is **not needed** These changes are internal lifecycle / behavioural improvements; no user-visible API or CLI flag added that warrants new public docs. Existing flags/Settings already documented at netbirdio/docs cover the surface area.
saavagebueno added the pull-request label 2026-08-05 07:08:50 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#27515