[GH-ISSUE #6687] macOS: process posture checks always fail on 0.74.x — 15s gather timeout abandons lsof-based process enumeration, client syncs empty meta.Files #11618

Closed
opened 2026-08-05 01:30:11 -04:00 by saavagebueno · 3 comments
Owner

Originally created by @Officetechadmin on GitHub (Jul 7, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/6687

Describe the problem

After upgrading a macOS client from 0.73.2 to 0.74.2, a peer subject to a process posture check receives an empty network map (0 peers, no routes) while showing Connected. The management server (0.74.2, self-hosted) stores meta.Files = [] for the peer, so the posture check fails. Downgrading the client to 0.73.2 resolves it. Linux peers on 0.74.2 against the same server and policies are unaffected.

Root cause (from source)

  1. 0.74.0 bounds posture gathering with systemInfoTimeout = 15 * time.Second (client/internal/engine.go), added to prevent multi-minute sync-loop freezes.
  2. On macOS, release builds have CGO_ENABLED=0 (.goreleaser.yaml), so gopsutil v3's non-cgo darwin ExeWithContext shells out to one lsof -p <pid> -Fpfn per PID (process_darwin_nocgo.go). On a real Mac (~1,000 processes) the sweep takes minutes — the multi-minute freeze the timeout was added for is (at least partly) this same sweep.
  3. So the gather deterministically exceeds 15s. In receiveManagementEvents, the timeout path falls back to system.GetInfo(...)base info with Files empty — and syncs it. The server overwrites the peer's previously-populated meta.Files with [], and every process posture check fails.
  4. It never self-heals: updateChecksIfNew re-gathers only when the checks list changes (isChecksEqual short-circuit), so the peer stays gated out until downgrade.

Impact

Any macOS peer subject to a process posture check silently loses all access on upgrade to 0.74.x — the UI shows Connected, making this hard for end users to diagnose. For fleets using process checks as a device-trust gate, a fleet-wide client upgrade is an outage.

Environment

  • Client: 0.74.2 (official pkg, darwin/arm64), introduced in 0.74.0; the same code is present unchanged on main and v0.75.0-rc.5 as of 2026-07-07
  • Management: 0.74.2 self-hosted (docker, netbirdio/netbird-server:latest)
  • Posture check: process check with mac_path set (an MDM agent binary); path exists and the process is running on the client
  • Evidence: management store peers.meta_files = [] for the 0.74.2 macOS peer vs [{"Path":"…","Exist":true,"ProcessIsRunning":true}] for an identical 0.73.2 macOS peer against the same server/policies

Suggested directions

  • Enumerate once instead of per-PID: a single lsof -Fpn (all processes) or ps -axo comm=-style full-path listing collapses the sweep to one exec.
  • Or use libproc proc_pidpath via direct syscalls (no cgo needed) on darwin.
  • On gather timeout, keep the last known Files instead of syncing base info with an empty list (empty ≠ unknown).
  • Retry the gather after a timeout even when the checks list hasn't changed, so an unlucky timeout isn't permanent.

Happy to test a fix build against our fleet's posture-check setup.

Originally created by @Officetechadmin on GitHub (Jul 7, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/6687 ## Describe the problem After upgrading a macOS client from 0.73.2 to 0.74.2, a peer subject to a **process posture check** receives an **empty network map** (0 peers, no routes) while showing Connected. The management server (0.74.2, self-hosted) stores `meta.Files = []` for the peer, so the posture check fails. Downgrading the client to 0.73.2 resolves it. Linux peers on 0.74.2 against the same server and policies are unaffected. ## Root cause (from source) 1. 0.74.0 bounds posture gathering with `systemInfoTimeout = 15 * time.Second` (`client/internal/engine.go`), added to prevent multi-minute sync-loop freezes. 2. On macOS, release builds have `CGO_ENABLED=0` (`.goreleaser.yaml`), so gopsutil v3's non-cgo darwin `ExeWithContext` shells out to **one `lsof -p <pid> -Fpfn` per PID** (`process_darwin_nocgo.go`). On a real Mac (~1,000 processes) the sweep takes minutes — the multi-minute freeze the timeout was added for is (at least partly) this same sweep. 3. So the gather deterministically exceeds 15s. In `receiveManagementEvents`, the timeout path falls back to `system.GetInfo(...)` — **base info with `Files` empty** — and syncs it. The server overwrites the peer's previously-populated `meta.Files` with `[]`, and every process posture check fails. 4. It never self-heals: `updateChecksIfNew` re-gathers only when the checks list *changes* (`isChecksEqual` short-circuit), so the peer stays gated out until downgrade. ## Impact Any macOS peer subject to a process posture check silently loses **all** access on upgrade to 0.74.x — the UI shows Connected, making this hard for end users to diagnose. For fleets using process checks as a device-trust gate, a fleet-wide client upgrade is an outage. ## Environment - Client: 0.74.2 (official pkg, darwin/arm64), introduced in 0.74.0; the same code is present unchanged on `main` and `v0.75.0-rc.5` as of 2026-07-07 - Management: 0.74.2 self-hosted (docker, netbirdio/netbird-server:latest) - Posture check: process check with `mac_path` set (an MDM agent binary); path exists and the process is running on the client - Evidence: management store `peers.meta_files = []` for the 0.74.2 macOS peer vs `[{"Path":"…","Exist":true,"ProcessIsRunning":true}]` for an identical 0.73.2 macOS peer against the same server/policies ## Suggested directions - Enumerate once instead of per-PID: a single `lsof -Fpn` (all processes) or `ps -axo comm=`-style full-path listing collapses the sweep to one exec. - Or use libproc `proc_pidpath` via direct syscalls (no cgo needed) on darwin. - On gather timeout, **keep the last known `Files`** instead of syncing base info with an empty list (empty ≠ unknown). - Retry the gather after a timeout even when the checks list hasn't changed, so an unlucky timeout isn't permanent. Happy to test a fix build against our fleet's posture-check setup.
Author
Owner

@linear-code[bot] commented on GitHub (Jul 7, 2026):

NET-1372

<!-- gh-comment-id:4905088557 --> @linear-code[bot] commented on GitHub (Jul 7, 2026): <!-- linear-linkback --> <p><a href="https://linear.app/netbird/issue/NET-1372">NET-1372</a></p>
Author
Owner

@Officetechadmin commented on GitHub (Jul 8, 2026):

Confirming v0.74.3 fixes this on our fleet (validated on the originally-affected macOS peer, same self-hosted server + posture policies as reported):

  • netbird status -d shows the full network map again — peers connect within ~5 s of netbird up (on 0.73.2 the first post-connect gather took ~90 s; the gopsutil v4 sweep is effectively instant)
  • the management store records populated meta.Files for the 0.74.3 peer ({"Path":".../kandji-daemon","Exist":true,"ProcessIsRunning":true}, meta_wt_version: 0.74.3) — the process posture check passes and gated routes/exit nodes are back

Thanks @mlsmaycon for the fast turnaround on #6688.

For anyone else hitting this: main still imports gopsutil v3 in client/system/process.go, so 0.75.0-rc.5 (and a 0.75.0 cut from it) would re-ship the regression — the port to main is #6695. Leaving this issue open until that lands.

Two design notes from the original report remain relevant even after #6695: a gather that exceeds the 15 s timeout still syncs empty Files (empty ≠ unknown), and it isn't retried until the checks list changes — so any future slow-gather regression would reproduce this failure mode silently.

<!-- gh-comment-id:4916600497 --> @Officetechadmin commented on GitHub (Jul 8, 2026): Confirming **v0.74.3 fixes this** on our fleet (validated on the originally-affected macOS peer, same self-hosted server + posture policies as reported): - `netbird status -d` shows the full network map again — peers connect within ~5 s of `netbird up` (on 0.73.2 the first post-connect gather took ~90 s; the gopsutil v4 sweep is effectively instant) - the management store records populated `meta.Files` for the 0.74.3 peer (`{"Path":".../kandji-daemon","Exist":true,"ProcessIsRunning":true}`, `meta_wt_version: 0.74.3`) — the process posture check passes and gated routes/exit nodes are back Thanks @mlsmaycon for the fast turnaround on #6688. For anyone else hitting this: `main` still imports gopsutil v3 in `client/system/process.go`, so 0.75.0-rc.5 (and a 0.75.0 cut from it) would re-ship the regression — the port to main is #6695. Leaving this issue open until that lands. Two design notes from the original report remain relevant even after #6695: a gather that exceeds the 15 s timeout still syncs empty `Files` (empty ≠ unknown), and it isn't retried until the checks list changes — so any future slow-gather regression would reproduce this failure mode silently.
Author
Owner

@alexharries commented on GitHub (Jul 8, 2026):

#WrittenByClaudeCode? :)

<!-- gh-comment-id:4917776913 --> @alexharries commented on GitHub (Jul 8, 2026): #WrittenByClaudeCode? :)
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11618