[PR #5987] [CLOSED] [client] Fix mgmt cache bypass overlay #28850

Closed
opened 2026-08-05 08:07:03 -04:00 by saavagebueno · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/netbirdio/netbird/pull/5987
Author: @pappz
Created: 4/24/2026
Status: Closed

Base: mainHead: fix-mgmt-cache-bypass-overlay


📝 Commits (6)

  • 77ec257 client/dns/mgmt: bypass overlay for control-plane FQDN resolution
  • 14be474 client/dns: register pool-root domains as host-manager match domains
  • c89c30b client/dns: canonicalize pool-root membership check with toZone
  • b0b52b6 client/dns/mgmt: singleflight on-demand resolves
  • 12d0eda client/dns: split UpdateServerConfig helpers to cut cognitive complexity
  • 3cdfa11 client/dns/mgmt: strip wildcard prefix in pool-root membership check

📊 Changes

3 files changed (+373 additions, -23 deletions)

View changed files

client/internal/dns/mgmt/bypass_resolver.go (+55 -0)
📝 client/internal/dns/mgmt/mgmt.go (+196 -11)
📝 client/internal/dns/server.go (+122 -12)

📄 Description

Describe your changes

Exit-Node Peer Routing Issue and Fix

Problem

When an exit-node peer installs a 0.0.0.0/0 default route on wt0 before its WireGuard peer entry is active, any socket dialing an off-link address via that route returns kernel ENOKEY.

This deadlocks a specific path: when a remote peer is homed on a different relay instance than the local daemon, the daemon must dial a foreign relay FQDN (e.g. streamline-de-fra1-0.relay.netbird.io). That FQDN is not in the mgmt cache, so it falls through to the normal upstream handler — which takes the overlay-routed dial and fails. No DNS → no relay → no peer handshake → no DNS.

Deterministic, but only triggers when the relay load-balancer puts peers on different instances. Caused the intermittent exit-node e2e failures on rocky9 / debian-12.

Fix

  • mgmt.Resolver bypass mechanism: mgmt.Resolver gets a dedicated *net.Resolver whose Dial uses nbnet.NewDialer(). On Linux the socket gets SO_MARK=ControlPlaneMark, on darwin IP_BOUND_IF, on Windows IP_UNICAST_IF — the same bypass mechanism gRPC/STUN/TURN/relay-websocket already use. Policy routes keep it on the underlay interface.

  • Original nameservers: Dialer targets the original pre-NetBird system nameservers, captured from the registerFallback flow. No hardcoded fallback IPs; if the host had no original resolver, the bypass resolver stays nil and the stale-while-revalidate cache serves.

  • Pool-root domains: Pool-root domains (ServerDomains.Relay[], e.g. relay.netbird.io) are registered through a thin subdomainMatchHandler wrapper, so instance subdomains (streamline-*.relay.netbird.io) route to the mgmt cache.

  • Cache miss resolution: On cache miss under a pool root, ServeDNS resolves on demand via the bypass resolver, caches the result, and returns it.

  • OS DNS manager integration: Pool-roots are also pushed into extraDomains so applyHostConfig advertises them as match domains to systemd-resolved / NetworkManager / scutil. Without this, those OS DNS managers would answer the query from the host's global upstream, skipping the handler chain entirely — which is exactly what happened on ubuntu-20.04.

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

  • I added/updated documentation for this change
  • Documentation is not needed for this change (explain why)

Docs PR URL (required if "docs added" is checked)

Paste the PR link from https://github.com/netbirdio/docs here:

https://github.com/netbirdio/docs/pull/__

Summary by CodeRabbit

  • New Features

    • On-demand resolution for instance subdomains under relay domains with deduplicated lookups and cached A/AAAA answers for faster responses.
    • Configurable bypass resolver to route DNS queries directly to specified upstream nameservers with robust dialing and failover behavior.
  • Bug Fixes

    • Improved handling and tracking of relay “pool-root” domains during config updates to avoid stale registrations.
    • More reliable nameserver fallback, tracking, and failover when upstreams change or are absent.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/netbirdio/netbird/pull/5987 **Author:** [@pappz](https://github.com/pappz) **Created:** 4/24/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `fix-mgmt-cache-bypass-overlay` --- ### 📝 Commits (6) - [`77ec257`](https://github.com/netbirdio/netbird/commit/77ec25796e6436da4574b0b87d5b4e5b6c4bdf6a) client/dns/mgmt: bypass overlay for control-plane FQDN resolution - [`14be474`](https://github.com/netbirdio/netbird/commit/14be474e3d728898639e5bd27cf9b6c0cdafcd3d) client/dns: register pool-root domains as host-manager match domains - [`c89c30b`](https://github.com/netbirdio/netbird/commit/c89c30bb2866e2a7c7770389a483e7615f55fd82) client/dns: canonicalize pool-root membership check with toZone - [`b0b52b6`](https://github.com/netbirdio/netbird/commit/b0b52b677429b2c11115a9eab812ad5b3f3116f9) client/dns/mgmt: singleflight on-demand resolves - [`12d0eda`](https://github.com/netbirdio/netbird/commit/12d0edabc00123a5e3bf6afb259e1be9cf665646) client/dns: split UpdateServerConfig helpers to cut cognitive complexity - [`3cdfa11`](https://github.com/netbirdio/netbird/commit/3cdfa11cb83565567163e03074647c852e84d42c) client/dns/mgmt: strip wildcard prefix in pool-root membership check ### 📊 Changes **3 files changed** (+373 additions, -23 deletions) <details> <summary>View changed files</summary> ➕ `client/internal/dns/mgmt/bypass_resolver.go` (+55 -0) 📝 `client/internal/dns/mgmt/mgmt.go` (+196 -11) 📝 `client/internal/dns/server.go` (+122 -12) </details> ### 📄 Description ## Describe your changes # Exit-Node Peer Routing Issue and Fix ## Problem When an exit-node peer installs a 0.0.0.0/0 default route on wt0 before its WireGuard peer entry is active, any socket dialing an off-link address via that route returns kernel ENOKEY. This deadlocks a specific path: when a remote peer is homed on a different relay instance than the local daemon, the daemon must dial a foreign relay FQDN (e.g. streamline-de-fra1-0.relay.netbird.io). That FQDN is not in the mgmt cache, so it falls through to the normal upstream handler — which takes the overlay-routed dial and fails. No DNS → no relay → no peer handshake → no DNS. Deterministic, but only triggers when the relay load-balancer puts peers on different instances. Caused the intermittent exit-node e2e failures on rocky9 / debian-12. ## Fix - **mgmt.Resolver bypass mechanism**: mgmt.Resolver gets a dedicated *net.Resolver whose Dial uses nbnet.NewDialer(). On Linux the socket gets SO_MARK=ControlPlaneMark, on darwin IP_BOUND_IF, on Windows IP_UNICAST_IF — the same bypass mechanism gRPC/STUN/TURN/relay-websocket already use. Policy routes keep it on the underlay interface. - **Original nameservers**: Dialer targets the original pre-NetBird system nameservers, captured from the registerFallback flow. No hardcoded fallback IPs; if the host had no original resolver, the bypass resolver stays nil and the stale-while-revalidate cache serves. - **Pool-root domains**: Pool-root domains (ServerDomains.Relay[], e.g. relay.netbird.io) are registered through a thin subdomainMatchHandler wrapper, so instance subdomains (streamline-*.relay.netbird.io) route to the mgmt cache. - **Cache miss resolution**: On cache miss under a pool root, ServeDNS resolves on demand via the bypass resolver, caches the result, and returns it. - **OS DNS manager integration**: Pool-roots are also pushed into extraDomains so applyHostConfig advertises them as match domains to systemd-resolved / NetworkManager / scutil. Without this, those OS DNS managers would answer the query from the host's global upstream, skipping the handler chain entirely — which is exactly what happened on ubuntu-20.04. ## Issue ticket number and link ## Stack <!-- branch-stack --> ### Checklist - [x] Is it a bug fix - [ ] Is a typo/documentation fix - [ ] Is a feature enhancement - [ ] It is a refactor - [ ] Created tests that fail without the change (if possible) > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [ ] I added/updated documentation for this change - [x] Documentation is **not needed** for this change (explain why) ### Docs PR URL (required if "docs added" is checked) Paste the PR link from https://github.com/netbirdio/docs here: https://github.com/netbirdio/docs/pull/__ <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * On-demand resolution for instance subdomains under relay domains with deduplicated lookups and cached A/AAAA answers for faster responses. * Configurable bypass resolver to route DNS queries directly to specified upstream nameservers with robust dialing and failover behavior. * **Bug Fixes** * Improved handling and tracking of relay “pool-root” domains during config updates to avoid stale registrations. * More reliable nameserver fallback, tracking, and failover when upstreams change or are absent. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
saavagebueno added the pull-request label 2026-08-05 08:07:03 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#28850