[GH-ISSUE #6202] macOS: domain-based network resources fail until DNS is queried directly against NetBird DNS — scoped resolvers registered as Supplemental race with system DNS #11569

Open
opened 2026-08-05 01:30:03 -04:00 by saavagebueno · 0 comments
Owner

Originally created by @dmdhrumilmistry on GitHub (May 19, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/6202

Summary

On macOS, domain-based network resources only become reachable after an explicit dig @<netbird-dns-ip> <managed-domain> is run. Normal application traffic (curl, browsers, etc.) bypasses the tunnel because NetBird's DNS is never actually consulted — its scoped resolvers are registered with the Supplemental flag and race against the system's default resolver, which usually wins. Since NetBird's DNS never sees the query, it never resolves the upstream IPs, and no route is installed on utun*. Traffic then exits the default interface (publicly resolvable A records still resolve via system DNS), which for an internally-restricted ingress yields HTTP 403 / connection refused / etc.

Environment

  • macOS 15 (Darwin 25.3.0), Apple Silicon
  • NetBird CLI + Daemon: 0.71.2
  • NetBird UI: NetBird.app installed
  • Interface: utun100 (Userspace WireGuard)
  • Self-hosted management

Setup

  1. A publicly-resolvable hostname app.example.com (CNAME → ALB) is exposed only to allow-listed source IPs; reaching it from outside the VPN yields HTTP 403 from nginx.
  2. app.example.com is configured as a NetBird Network Resource of type "Domain". The resource is selected on the peer:
    netbird networks ls
    # …
    #  - ID: app.example.com
    #    Domains: app.example.com
    #    Status: Selected
    
  3. NetBird DNS group is configured. example.com is added to the group's Match Domains list.

Expected

curl https://app.example.com/ from the peer with NetBird up routes through utun100, reaching the resource over the tunnel. The route is installed on first resolution and stays correct as long as the peer is connected.

Actual

curl resolves the hostname via the system resolver (Cloudflare in our case), gets the public CNAME chain + A record, opens a direct TCP connection to the public LB IP outside the tunnel, and gets 403. No route to the resolved IP is added on utun100:

$ netstat -nr -f inet | grep <resolved-ip>
(no route)

Only after explicitly forcing the query at NetBird's DNS does the route appear and the tunnel work:

$ dig +short app.example.com @100.X.X.254       # NetBird DNS
<resolved-ip>
$ netstat -nr -f inet | grep <resolved-ip>
<resolved-ip>   utun100   UH1   utun100          # route now exists
$ curl https://app.example.com/                  # works (HTTP 200, multi-second latency = tunnel)

The route then persists for roughly the upstream TTL (≈60s here); after that, if nothing has re-queried NetBird's DNS for the name, the route eventually goes away and the failure mode returns. This means the manual warmup has to be repeated periodically.

Root cause

scutil --dns shows every NetBird-managed domain registered with the Supplemental flag:

resolver #4
  domain   : app.example.com
  nameserver[0] : 100.X.X.254
  port     : 53
  flags    : Supplemental, Request A records, Request AAAA records

On macOS, Supplemental resolvers do not become authoritative for their domain. mDNSResponder queries them in parallel with the primary resolver chain. The primary chain in our case is:

resolver #1
  nameserver[0] : 100.X.X.254     # NetBird
  nameserver[1] : 1.1.1.1         # Cloudflare
  nameserver[2] : 1.0.0.1

1.1.1.1 consistently returns an answer for a publicly-resolvable internal hostname faster than NetBird's DNS (which traverses the WireGuard tunnel). mDNSResponder takes Cloudflare's answer and discards NetBird's late reply (or never waits for it). The query never reaches NetBird's DNS server in any way the daemon's resolver/route-manager observes, so no route is installed.

Adding example.com to Match Domains in the dashboard registers an additional scope, but that scope is also Supplemental and at a less-specific suffix than the per-resource scopes, so it doesn't change the outcome for the resource subdomain. There is no dashboard toggle (at least not exposed in our version) to mark the scope as non-supplemental / primary.

Reproduction

  1. macOS peer running 0.71.2.
  2. Configure any Network Resource with a Domain that is also publicly resolvable (so the system resolver returns an A record).
  3. From a fresh state (sudo killall -HUP mDNSResponder && sudo dscacheutil -flushcache), curl the hostname.
  4. Observe: query is answered by the default system DNS (scutil --dns shows it), no route on utun*, traffic exits via the default interface.

Compare against the working case:

  1. dig @<netbird-dns-ip> <hostname>
  2. curl the hostname → route now exists, traffic flows over the tunnel.
  3. Wait past the upstream TTL without re-querying NetBird DNS; the route is removed and step 3 reproduces again.

Confirmed workaround

Manually pre-query NetBird's DNS before any application traffic, and re-query periodically:

dig @100.X.X.254 app.example.com

This is the only workaround I've verified end-to-end. It is obviously not viable as a real solution — it has to be repeated whenever the route expires.

Hypothesized fix to test (not yet verified)

Based on the diagnosis, creating same-specificity /etc/resolver/<domain> files for every NetBird-managed domain should work — macOS treats files under /etc/resolver/ as non-Supplemental primary scopes for their domain, which would beat NetBird's Supplemental scope at the same specificity and force mDNSResponder to send the query to NetBird's DNS:

sudo install -d -m 755 /etc/resolver
for d in $(netbird networks ls | awk '/Domains:/ { n=split($2,p,","); for(i=1;i<=n;i++){gsub(/[ \t]/,"",p[i]); if(p[i]!="") print p[i]} }' | sort -u); do
  echo "nameserver 100.X.X.254" | sudo tee "/etc/resolver/$d" >/dev/null
done
sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder

I have not actually validated this end-to-end yet (couldn't run sudo in my debug session), but it follows directly from how mDNSResponder documents scope resolution. If anyone reproducing this issue tries it, please report whether it works — and if it does, that's strong evidence the underlying fix is simply dropping the Supplemental flag when the daemon programs scoped resolvers via scutil.

Suggested fix

Allow the macOS daemon to register its scoped resolvers as non-Supplemental (primary for their domain), either:

  1. Automatically when a NetBird-managed domain is the configured "Match Domain" / authoritative for the peer; or
  2. Via an explicit per-resource or per-nameserver-group toggle ("Primary domain" / "Authoritative") in the management dashboard that is honored by the macOS daemon to drop the Supplemental flag when programming scutil.

For reference, files under /etc/resolver/<domain> produce exactly this flag set (no Supplemental), so the system resolver behavior to target is well-defined.

The race-loss is sensitive to upstream DNS latency. Hosts using a fast public resolver as nameserver[1+] (Cloudflare, Google, etc.) will hit this consistently. Hosts where NetBird DNS happens to be faster than the configured upstream may mask the issue.

Originally created by @dmdhrumilmistry on GitHub (May 19, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/6202 ## Summary On macOS, domain-based network resources only become reachable after an explicit `dig @<netbird-dns-ip> <managed-domain>` is run. Normal application traffic (curl, browsers, etc.) bypasses the tunnel because NetBird's DNS is never actually consulted — its scoped resolvers are registered with the `Supplemental` flag and race against the system's default resolver, which usually wins. Since NetBird's DNS never sees the query, it never resolves the upstream IPs, and no route is installed on `utun*`. Traffic then exits the default interface (publicly resolvable A records still resolve via system DNS), which for an internally-restricted ingress yields HTTP 403 / connection refused / etc. ## Environment - macOS 15 (Darwin 25.3.0), Apple Silicon - NetBird CLI + Daemon: `0.71.2` - NetBird UI: NetBird.app installed - Interface: `utun100` (Userspace WireGuard) - Self-hosted management ## Setup 1. A publicly-resolvable hostname `app.example.com` (CNAME → ALB) is exposed only to allow-listed source IPs; reaching it from outside the VPN yields HTTP 403 from nginx. 2. `app.example.com` is configured as a NetBird **Network Resource** of type "Domain". The resource is selected on the peer: ``` netbird networks ls # … # - ID: app.example.com # Domains: app.example.com # Status: Selected ``` 3. NetBird DNS group is configured. `example.com` is added to the group's **Match Domains** list. ## Expected `curl https://app.example.com/` from the peer with NetBird up routes through `utun100`, reaching the resource over the tunnel. The route is installed on first resolution and stays correct as long as the peer is connected. ## Actual `curl` resolves the hostname via the **system** resolver (Cloudflare in our case), gets the public CNAME chain + A record, opens a direct TCP connection to the public LB IP outside the tunnel, and gets 403. No route to the resolved IP is added on `utun100`: ``` $ netstat -nr -f inet | grep <resolved-ip> (no route) ``` Only after explicitly forcing the query at NetBird's DNS does the route appear and the tunnel work: ``` $ dig +short app.example.com @100.X.X.254 # NetBird DNS <resolved-ip> $ netstat -nr -f inet | grep <resolved-ip> <resolved-ip> utun100 UH1 utun100 # route now exists $ curl https://app.example.com/ # works (HTTP 200, multi-second latency = tunnel) ``` The route then persists for roughly the upstream TTL (≈60s here); after that, if nothing has re-queried NetBird's DNS for the name, the route eventually goes away and the failure mode returns. This means the manual warmup has to be repeated periodically. ## Root cause `scutil --dns` shows every NetBird-managed domain registered with the `Supplemental` flag: ``` resolver #4 domain : app.example.com nameserver[0] : 100.X.X.254 port : 53 flags : Supplemental, Request A records, Request AAAA records ``` On macOS, `Supplemental` resolvers do **not** become authoritative for their domain. `mDNSResponder` queries them **in parallel** with the primary resolver chain. The primary chain in our case is: ``` resolver #1 nameserver[0] : 100.X.X.254 # NetBird nameserver[1] : 1.1.1.1 # Cloudflare nameserver[2] : 1.0.0.1 ``` `1.1.1.1` consistently returns an answer for a publicly-resolvable internal hostname faster than NetBird's DNS (which traverses the WireGuard tunnel). `mDNSResponder` takes Cloudflare's answer and discards NetBird's late reply (or never waits for it). The query never reaches NetBird's DNS server in any way the daemon's resolver/route-manager observes, so no route is installed. Adding `example.com` to **Match Domains** in the dashboard registers an additional scope, but that scope is also `Supplemental` and at a less-specific suffix than the per-resource scopes, so it doesn't change the outcome for the resource subdomain. There is no dashboard toggle (at least not exposed in our version) to mark the scope as non-supplemental / primary. ## Reproduction 1. macOS peer running 0.71.2. 2. Configure any Network Resource with a Domain that is also publicly resolvable (so the system resolver returns an A record). 3. From a fresh state (`sudo killall -HUP mDNSResponder && sudo dscacheutil -flushcache`), `curl` the hostname. 4. Observe: query is answered by the default system DNS (`scutil --dns` shows it), no route on `utun*`, traffic exits via the default interface. Compare against the working case: 5. `dig @<netbird-dns-ip> <hostname>` 6. `curl` the hostname → route now exists, traffic flows over the tunnel. 7. Wait past the upstream TTL without re-querying NetBird DNS; the route is removed and step 3 reproduces again. ## Confirmed workaround Manually pre-query NetBird's DNS before any application traffic, and re-query periodically: ``` dig @100.X.X.254 app.example.com ``` This is the only workaround I've verified end-to-end. It is obviously not viable as a real solution — it has to be repeated whenever the route expires. ## Hypothesized fix to test (not yet verified) Based on the diagnosis, creating same-specificity `/etc/resolver/<domain>` files for every NetBird-managed domain *should* work — macOS treats files under `/etc/resolver/` as non-`Supplemental` primary scopes for their domain, which would beat NetBird's `Supplemental` scope at the same specificity and force `mDNSResponder` to send the query to NetBird's DNS: ``` sudo install -d -m 755 /etc/resolver for d in $(netbird networks ls | awk '/Domains:/ { n=split($2,p,","); for(i=1;i<=n;i++){gsub(/[ \t]/,"",p[i]); if(p[i]!="") print p[i]} }' | sort -u); do echo "nameserver 100.X.X.254" | sudo tee "/etc/resolver/$d" >/dev/null done sudo dscacheutil -flushcache && sudo killall -HUP mDNSResponder ``` I have not actually validated this end-to-end yet (couldn't run sudo in my debug session), but it follows directly from how `mDNSResponder` documents scope resolution. If anyone reproducing this issue tries it, please report whether it works — and if it does, that's strong evidence the underlying fix is simply dropping the `Supplemental` flag when the daemon programs scoped resolvers via `scutil`. ## Suggested fix Allow the macOS daemon to register its scoped resolvers as **non-Supplemental** (primary for their domain), either: 1. Automatically when a NetBird-managed domain is the configured "Match Domain" / authoritative for the peer; or 2. Via an explicit per-resource or per-nameserver-group toggle ("Primary domain" / "Authoritative") in the management dashboard that is honored by the macOS daemon to drop the `Supplemental` flag when programming `scutil`. For reference, files under `/etc/resolver/<domain>` produce exactly this flag set (no `Supplemental`), so the system resolver behavior to target is well-defined. ## Possibly related The race-loss is sensitive to upstream DNS latency. Hosts using a fast public resolver as `nameserver[1+]` (Cloudflare, Google, etc.) will hit this consistently. Hosts where NetBird DNS happens to be faster than the configured upstream may mask the issue.
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11569