[GH-ISSUE #5727] [Bug] macOS: DNS probe fires before WireGuard routes are established — false "Unable to reach DNS servers" warning on every reconnection #11918

Closed
opened 2026-08-05 01:31:38 -04:00 by saavagebueno · 3 comments
Owner

Originally created by @YouKyi on GitHub (Mar 29, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5727

Describe the problem

On macOS, the NetBird client displays a "Warning: DNS — Unable to reach one or more DNS servers" notification on every reconnection (network map update, sleep/wake, network change). The DNS servers are fully functional — the warning is a false positive caused by a race condition in upstream.go:293.

The DNS upstream probe fires before the WireGuard allowed IPs / routes to the DNS servers are re-established on the utun interface. The probe packets get i/o timeout, the WARNING event is published, macOS shows the notification — and a few seconds later routes are ready and DNS works perfectly.

This is the same bug that was fixed for iOS in PR #5360 ("Ensured route settlement on iOS before handling DNS responses"), but the fix was not ported to macOS.

Debug logs show the exact race condition on every reconnection:

12:37:26 DEBG  iface.go:213: Removing allowed IP ... 10.10.10.2/32    ← route to DNS server REMOVED
12:37:34 DEBG  iface.go:213: Removing allowed IP ... 10.10.10.2/32    ← route still not re-created

12:37:34.513 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: i/o timeout  ← PROBE FIRES, NO ROUTE
12:37:34.519 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: i/o timeout
12:37:34.522 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: i/o timeout

12:37:34.523 DEBG  status.go:1093: event published:
  severity: WARNING  category: DNS
  message: "All upstream servers failed (probe failed)"
  userMessage: "Unable to reach one or more DNS servers..."              ← NOTIFICATION SHOWN

# Seconds later: routes established, DNS works fine
# netbird status -d shows: [10.10.10.2:53] for [local.domain.net] is Available

This pattern repeats systematically — 13+ occurrences documented over 5 weeks of logs.

To Reproduce

  1. Configure a match domain nameserver pointing to an internal DNS server accessible via a routing peer (e.g., 10.10.10.2:53 for *.domain.net)
  2. Connect to NetBird on macOS
  3. Wait for any network map update (happens on: network changes, sleep/wake, peer connect/disconnect)
  4. Observe the macOS notification: "Warning: DNS — Unable to reach one or more DNS servers. This might affect your ability to connect to some services."
  5. Run netbird status -d — nameserver shows as Available
  6. Test DNS with dscacheutil -q host -a name <internal-domain> — works perfectly

Expected behavior

The DNS probe should wait for WireGuard routes to be fully established before testing upstream nameservers — the same behavior implemented for iOS in PR #5360. The WARNING event should not fire during the transient window between route teardown and route re-establishment.

Are you using NetBird Cloud?

No — self-hosted NetBird control plane (management v0.66.4, dashboard v2.34.2).

NetBird version

OS: darwin/arm64
Daemon version: 0.66.x
CLI version: 0.66.x
Interface type: Userspace
Lazy connection: true

Is any other VPN software installed?

No.

Debug output

netbird status -dA (after the warning — DNS is working):

Nameservers:
  [10.10.10.2:53] for [local.domain.net] is Available
  [10.10.10.2:53] for [int.domain.net, int.domain.online] is Available
FQDN: youkyi-macbook.vpn.domain.net
NetBird IP: 10.9.1.29/24
Interface type: Userspace
Quantum resistance: false
Lazy connection: true
Peers count: 2/19 Connected
13 occurrences over 5 weeks — all showing the same race condition
Date Time Servers probed Timeouts
2026-02-12 09:39:25 10.10.10.2, 10.11.10.3 ×8
2026-02-12 11:08:24 10.10.10.2, 10.11.10.3 ×4
2026-02-12 14:21:24 10.10.10.2, 10.11.10.3 ×6
2026-02-12 19:36:31 10.10.10.2, 10.11.10.3 ×4
2026-02-24 10:23:08 10.10.10.2, 10.11.10.3 ×6
2026-02-24 10:48:58 10.10.10.2, 10.11.10.3 ×6
2026-03-27 11:50:34 10.10.10.2, 10.11.10.3, 10.9.1.193 ×9
2026-03-27 12:25:40 10.10.10.2, 10.11.10.3, 10.9.1.193 ×9
2026-03-28 10:16:20 10.10.10.2, 10.11.10.3, 10.9.1.193 ×9
2026-03-28 12:11:24 10.10.10.2, 10.11.10.3, 10.9.1.193 ×9
2026-03-28 22:16:37 10.10.10.2, 10.11.10.3, 10.9.1.193 ×9
2026-03-29 10:01:08 10.10.10.2, 10.11.10.3, 10.9.1.193 ×9
2026-03-29 12:37:34 10.10.10.2 ×3 → WARNING event

Each occurrence is preceded by removing allowed IP ... 10.10.10.2/32 and followed by DNS working normally.

Full log sequence of the last occurrence (2026-03-29 12:37)
# Step 1: Routes torn down
12:37:26 INFO  host_darwin.go:142: removing match domains from system
12:37:26 INFO  host_darwin.go:142: removing search domains from system
12:37:26 DEBG  iface.go:189: Removing peer oKFEuU1+... from interface utun100
12:37:26 DEBG  iface.go:213: Removing allowed IP from interface utun100: allowed IP 10.10.10.2/32
12:37:26 DEBG  iface.go:213: Removing allowed IP from interface utun100: allowed IP 10.11.10.3/32

# Step 2: Reconnection starts
12:37:33 DEBG  handler_chain.go:166: removing handler pattern: domain=connect.domain.net. priority=150
12:37:34 DEBG  iface.go:189: Removing peer oKFEuU1+... from interface utun100
12:37:34 DEBG  iface.go:213: Removing allowed IP from interface utun100: allowed IP 10.10.10.2/32

# Step 3: Probe fires BEFORE routes are re-established
12:37:34.513 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: read udp 10.9.1.29:62893->10.10.10.2:53: i/o timeout
12:37:34.519 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: read udp 10.9.1.29:52994->10.10.10.2:53: i/o timeout
12:37:34.522 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: read udp 10.9.1.29:62500->10.10.10.2:53: i/o timeout

# Step 4: Warning published → macOS notification displayed
12:37:34.523 DEBG  status.go:1093: event published: severity:WARNING category:DNS message:"All upstream servers failed (probe failed)" userMessage:"Unable to reach one or more DNS servers. This might affect your ability to connect to some services." metadata:{key:"upstreams" value:"10.10.10.2:53"}
12:37:34.523 DEBG  status.go:1093: event published: severity:WARNING category:DNS message:"All upstream servers failed (probe failed)" userMessage:"Unable to reach one or more DNS servers. This might affect your ability to connect to some services." metadata:{key:"upstreams" value:"10.10.10.2:53"}
12:37:34.531 DEBG  status.go:1093: event published: severity:WARNING category:DNS message:"All upstream servers failed (probe failed)" userMessage:"Unable to reach one or more DNS servers. This might affect your ability to connect to some services." metadata:{key:"upstreams" value:"10.10.10.2:53"}

Screenshots

Additional context

This is the same class of bug fixed for iOS in #5360 ("Ensured route settlement on iOS before handling DNS responses"). The removing allowed IPprobing upstream sequence in the logs proves the probe runs during the window where routes don't exist yet — it's a timing issue, not a network issue.

Proposed fixes (any of these would resolve it):

  1. Port the #5360 fix to macOS: wait for allowed IPs to be configured on utun before probing upstreams
  2. Grace period: suppress WARNING events during the first N seconds after a network map update / reconnection
  3. Retry before warning: if the first probe fails post-reconnection, retry after a short delay — only publish the WARNING if the retry also fails

Related issues and PRs:

  • #5360[iOS fix] "Ensured route settlement on iOS before handling DNS responses" — same bug, fixed on iOS only. Requesting the same fix for macOS.
  • #5576[Partial fix] "Fix DNS probe thread safety" — improved probe concurrency but doesn't address the timing issue (probe before routes ready)
  • #3647[Same bug] "Incorrect error for DNS in netbird status debug output" — author identified the same root cause: "probe DNS before the communication between the peers is fully established"
  • #3367[Same symptom] "DNS Nameservers read udp timeout" — macOS, same upstream.go timeout pattern on every network map update
  • #4332[Related] "DNS resolution intermittent failure on macOS 15.5" — same intermittent behavior

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @YouKyi on GitHub (Mar 29, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5727 **Describe the problem** On macOS, the NetBird client displays a "Warning: DNS — Unable to reach one or more DNS servers" notification on **every reconnection** (network map update, sleep/wake, network change). The DNS servers are fully functional — the warning is a **false positive** caused by a race condition in `upstream.go:293`. The DNS upstream probe fires **before** the WireGuard allowed IPs / routes to the DNS servers are re-established on the `utun` interface. The probe packets get `i/o timeout`, the WARNING event is published, macOS shows the notification — and a few seconds later routes are ready and DNS works perfectly. This is the **same bug** that was fixed for iOS in PR #5360 ("Ensured route settlement on iOS before handling DNS responses"), but the fix was **not ported to macOS**. Debug logs show the exact race condition on every reconnection: ``` 12:37:26 DEBG iface.go:213: Removing allowed IP ... 10.10.10.2/32 ← route to DNS server REMOVED 12:37:34 DEBG iface.go:213: Removing allowed IP ... 10.10.10.2/32 ← route still not re-created 12:37:34.513 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: i/o timeout ← PROBE FIRES, NO ROUTE 12:37:34.519 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: i/o timeout 12:37:34.522 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: i/o timeout 12:37:34.523 DEBG status.go:1093: event published: severity: WARNING category: DNS message: "All upstream servers failed (probe failed)" userMessage: "Unable to reach one or more DNS servers..." ← NOTIFICATION SHOWN # Seconds later: routes established, DNS works fine # netbird status -d shows: [10.10.10.2:53] for [local.domain.net] is Available ``` This pattern repeats **systematically** — 13+ occurrences documented over 5 weeks of logs. **To Reproduce** 1. Configure a match domain nameserver pointing to an internal DNS server accessible via a routing peer (e.g., `10.10.10.2:53` for `*.domain.net`) 2. Connect to NetBird on macOS 3. Wait for any network map update (happens on: network changes, sleep/wake, peer connect/disconnect) 4. Observe the macOS notification: "Warning: DNS — Unable to reach one or more DNS servers. This might affect your ability to connect to some services." 5. Run `netbird status -d` — nameserver shows as **Available** 6. Test DNS with `dscacheutil -q host -a name <internal-domain>` — works perfectly **Expected behavior** The DNS probe should **wait for WireGuard routes to be fully established** before testing upstream nameservers — the same behavior implemented for iOS in PR #5360. The WARNING event should not fire during the transient window between route teardown and route re-establishment. **Are you using NetBird Cloud?** No — self-hosted NetBird control plane (management v0.66.4, dashboard v2.34.2). **NetBird version** ``` OS: darwin/arm64 Daemon version: 0.66.x CLI version: 0.66.x Interface type: Userspace Lazy connection: true ``` **Is any other VPN software installed?** No. **Debug output** `netbird status -dA` (after the warning — DNS is working): ``` Nameservers: [10.10.10.2:53] for [local.domain.net] is Available [10.10.10.2:53] for [int.domain.net, int.domain.online] is Available FQDN: youkyi-macbook.vpn.domain.net NetBird IP: 10.9.1.29/24 Interface type: Userspace Quantum resistance: false Lazy connection: true Peers count: 2/19 Connected ``` <details> <summary>13 occurrences over 5 weeks — all showing the same race condition</summary> | Date | Time | Servers probed | Timeouts | |------|------|----------------|----------| | 2026-02-12 | 09:39:25 | 10.10.10.2, 10.11.10.3 | ×8 | | 2026-02-12 | 11:08:24 | 10.10.10.2, 10.11.10.3 | ×4 | | 2026-02-12 | 14:21:24 | 10.10.10.2, 10.11.10.3 | ×6 | | 2026-02-12 | 19:36:31 | 10.10.10.2, 10.11.10.3 | ×4 | | 2026-02-24 | 10:23:08 | 10.10.10.2, 10.11.10.3 | ×6 | | 2026-02-24 | 10:48:58 | 10.10.10.2, 10.11.10.3 | ×6 | | 2026-03-27 | 11:50:34 | 10.10.10.2, 10.11.10.3, 10.9.1.193 | ×9 | | 2026-03-27 | 12:25:40 | 10.10.10.2, 10.11.10.3, 10.9.1.193 | ×9 | | 2026-03-28 | 10:16:20 | 10.10.10.2, 10.11.10.3, 10.9.1.193 | ×9 | | 2026-03-28 | 12:11:24 | 10.10.10.2, 10.11.10.3, 10.9.1.193 | ×9 | | 2026-03-28 | 22:16:37 | 10.10.10.2, 10.11.10.3, 10.9.1.193 | ×9 | | 2026-03-29 | 10:01:08 | 10.10.10.2, 10.11.10.3, 10.9.1.193 | ×9 | | 2026-03-29 | 12:37:34 | 10.10.10.2 | ×3 → WARNING event | Each occurrence is preceded by `removing allowed IP ... 10.10.10.2/32` and followed by DNS working normally. </details> <details> <summary>Full log sequence of the last occurrence (2026-03-29 12:37)</summary> ``` # Step 1: Routes torn down 12:37:26 INFO host_darwin.go:142: removing match domains from system 12:37:26 INFO host_darwin.go:142: removing search domains from system 12:37:26 DEBG iface.go:189: Removing peer oKFEuU1+... from interface utun100 12:37:26 DEBG iface.go:213: Removing allowed IP from interface utun100: allowed IP 10.10.10.2/32 12:37:26 DEBG iface.go:213: Removing allowed IP from interface utun100: allowed IP 10.11.10.3/32 # Step 2: Reconnection starts 12:37:33 DEBG handler_chain.go:166: removing handler pattern: domain=connect.domain.net. priority=150 12:37:34 DEBG iface.go:189: Removing peer oKFEuU1+... from interface utun100 12:37:34 DEBG iface.go:213: Removing allowed IP from interface utun100: allowed IP 10.10.10.2/32 # Step 3: Probe fires BEFORE routes are re-established 12:37:34.513 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: read udp 10.9.1.29:62893->10.10.10.2:53: i/o timeout 12:37:34.519 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: read udp 10.9.1.29:52994->10.10.10.2:53: i/o timeout 12:37:34.522 WARN upstream.go:293: probing upstream nameserver 10.10.10.2:53: read udp 10.9.1.29:62500->10.10.10.2:53: i/o timeout # Step 4: Warning published → macOS notification displayed 12:37:34.523 DEBG status.go:1093: event published: severity:WARNING category:DNS message:"All upstream servers failed (probe failed)" userMessage:"Unable to reach one or more DNS servers. This might affect your ability to connect to some services." metadata:{key:"upstreams" value:"10.10.10.2:53"} 12:37:34.523 DEBG status.go:1093: event published: severity:WARNING category:DNS message:"All upstream servers failed (probe failed)" userMessage:"Unable to reach one or more DNS servers. This might affect your ability to connect to some services." metadata:{key:"upstreams" value:"10.10.10.2:53"} 12:37:34.531 DEBG status.go:1093: event published: severity:WARNING category:DNS message:"All upstream servers failed (probe failed)" userMessage:"Unable to reach one or more DNS servers. This might affect your ability to connect to some services." metadata:{key:"upstreams" value:"10.10.10.2:53"} ``` </details> **Screenshots** <!-- Attach screenshot of the macOS "Warning: DNS" notification here --> **Additional context** This is the **same class of bug** fixed for iOS in #5360 ("Ensured route settlement on iOS before handling DNS responses"). The `removing allowed IP` → `probing upstream` sequence in the logs proves the probe runs during the window where routes don't exist yet — it's a timing issue, not a network issue. Proposed fixes (any of these would resolve it): 1. **Port the #5360 fix to macOS**: wait for allowed IPs to be configured on `utun` before probing upstreams 2. **Grace period**: suppress WARNING events during the first N seconds after a network map update / reconnection 3. **Retry before warning**: if the first probe fails post-reconnection, retry after a short delay — only publish the WARNING if the retry also fails **Related issues and PRs:** - #5360 — **[iOS fix]** "Ensured route settlement on iOS before handling DNS responses" — same bug, fixed on iOS only. Requesting the same fix for macOS. - #5576 — **[Partial fix]** "Fix DNS probe thread safety" — improved probe concurrency but doesn't address the timing issue (probe before routes ready) - #3647 — **[Same bug]** "Incorrect error for DNS in netbird status debug output" — author identified the same root cause: "probe DNS before the communication between the peers is fully established" - #3367 — **[Same symptom]** "DNS Nameservers read udp timeout" — macOS, same `upstream.go` timeout pattern on every network map update - #4332 — **[Related]** "DNS resolution intermittent failure on macOS 15.5" — same intermittent behavior **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2026-08-05 01:31:38 -04:00
Author
Owner

@pierrechapuis commented on GitHub (Apr 10, 2026):

FYI: Same issue for me on 2 MacBooks.

<!-- gh-comment-id:4225785926 --> @pierrechapuis commented on GitHub (Apr 10, 2026): FYI: Same issue for me on 2 MacBooks.
Author
Owner

@SunDevil311 commented on GitHub (May 1, 2026):

I appear to be having the same issue as well, but I'm on Windows 11. This is the error message I see:
NetBird Warning

<!-- gh-comment-id:4360242058 --> @SunDevil311 commented on GitHub (May 1, 2026): I appear to be having the same issue as well, but I'm on Windows 11. This is the error message I see: [![NetBird Warning](https://i.ibb.co/YTKPXbVd/netbird-warning.png "NetBird Warning")](https://ibb.co/YTKPXbVd)
Author
Owner

@gilfreund commented on GitHub (May 7, 2026):

Same race condition, the failed probe permanently deactivates the upstream nameserver group for the entire session. DNS never recovers, even though the tunnel and routes are fully established seconds later.

From the log:

12:23:10.770 registering original nameservers [192.168.120.2]
12:23:11.271 probing upstream nameserver 192.168.120.2:53: i/o timeout  ← probe fires, tunnel not up
12:23:11.271 Temporarily deactivating nameservers group due to timeout
12:23:41.931 first wg handshake detected                                ← tunnel up 30s later

After the handshake, dig @192.168.120.2 -b 100.90.245.3 works. The route exists, the DNS server responds. But the NetBird DNS proxy at 100.90.255.254 returns NXDOMAIN for all queries because the upstream was deactivated and is never re-probed.

My temporary workaround: Custom Zones hostnames. But this doesn't really scale, and is a pain to manage.
The iOS fix in #5360 (wait for route settlement before probing) would probably resolve this on macOS as well.

<!-- gh-comment-id:4396090895 --> @gilfreund commented on GitHub (May 7, 2026): Same race condition, the failed probe **permanently deactivates** the upstream nameserver group for the entire session. DNS never recovers, even though the tunnel and routes are fully established seconds later. From the log: ``` 12:23:10.770 registering original nameservers [192.168.120.2] 12:23:11.271 probing upstream nameserver 192.168.120.2:53: i/o timeout ← probe fires, tunnel not up 12:23:11.271 Temporarily deactivating nameservers group due to timeout 12:23:41.931 first wg handshake detected ← tunnel up 30s later ``` After the handshake, `dig @192.168.120.2 -b 100.90.245.3` works. The route exists, the DNS server responds. But the NetBird DNS proxy at `100.90.255.254` returns NXDOMAIN for all queries because the upstream was deactivated and is never re-probed. My temporary workaround: Custom Zones hostnames. But this doesn't really scale, and is a pain to manage. The iOS fix in #5360 (wait for route settlement before probing) would probably resolve this on macOS as well.
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11918