SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-04 19:55:09 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Linux client DNS issue #2222

New Issue
Closed
opened 2025-11-20 07:06:05 -05:00 by saavagebueno · 5 comments
saavagebueno commented 2025-11-20 07:06:05 -05:00
Owner
Copy Link

Originally created by @EkiciLP on GitHub (Aug 23, 2025).

Describe the problem

So I have a custom DNS Server in my netbird net which is configured as the main DNS for the network in the UI.
On Linux clients it fails to add this DNS Server using systemd-resolved or resolvconf. When Connecting to the Net it throws this error:

Image

But I can confirm that the DNS Server is reachable because mobile devices work fine and a direct nslookup local.domain <DNS-IP> also works perfectly.

To Reproduce

Steps to reproduce the behavior:

  1. Have a custom DNS configured and added to netbird
  2. Install the Linux client using the official docs
  3. Connect using UI instructions
  4. Try to reach a local domain

Expected behavior

Resolving local domains correctly.

Are you using NetBird Cloud?
Self-Hosted

NetBird version

0.55.1

**Is any other VPN software installed?**t

Tailscale is installed but not configured.

Debug output

To help us resolve the problem, please attach the following anonymized status output

Peers detail:
 notebook-tim.anon-hVzdp.domain:
  NetBird IP: 100.125.6.124
  Public key: UOecXL5x4tIPm7k5VDeEC8dHncJtNNWdaj5foin64RU=
  Status: Connecting
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 6 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

 vpn-server.anon-hVzdp.domain:
  NetBird IP: 100.125.75.34
  Public key: 2+b9o09DLUjH5hopP5RSraYB3JtR/PBbTGGuAzlUmBw=
  Status: Connecting
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 6 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

 opnsense.anon-hVzdp.domain:
  NetBird IP: 100.125.152.145
  Public key: pFVvqWabLQvsqWQC/lihUBcJaQ4nmJi5kpo0pqFfxHU=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:57935/10.0.0.1:51820
  Relay server address: 
  Last connection update: 6 seconds ago
  Last WireGuard handshake: 6 seconds ago
  Transfer status (received/sent) 284 B/276 B
  Quantum resistance: false
  Networks: 10.0.0.0/24, 10.0.1.0/24, 10.0.2.0/24
  Latency: 354.416µs

 phone-tim.anon-hVzdp.domain:
  NetBird IP: 100.125.229.157
  Public key: +gJhLUkm9IlMkaJS4Q76Ht6RU7VVNIKb8w7OrVbERzg=
  Status: Connecting
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 6 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

Events:
  [INFO] SYSTEM (796e2dad-32d5-4ee2-b396-c243923f9017)
    Message: Network deselection changed
    Time: 28 minutes, 15 seconds ago
    Metadata: all: false, append: false, networks: Exit Node (vpn-server)
  [WARNING] DNS (878f18f1-622a-4676-abe3-1a9e4642beef)
    Message: All upstream servers failed (probe failed)
    Time: 27 minutes, 21 seconds ago
    Metadata: upstreams: 100.125.152.145:53
  [INFO] SYSTEM (dd43b0d8-d430-4960-b296-6ee72425914e)
    Message: Network map updated
    Time: 27 minutes, 21 seconds ago
  [WARNING] DNS (26da8caa-bfe3-452d-aef8-aa3b01c96554)
    Message: All upstream servers failed (probe failed)
    Time: 23 minutes, 22 seconds ago
    Metadata: upstreams: 100.125.152.145:53
  [INFO] SYSTEM (823510c5-327f-46de-9f2b-67d778ce7620)
    Message: Network map updated
    Time: 23 minutes, 22 seconds ago
  [WARNING] DNS (8c969815-415c-428c-9fe6-5b85e162ae19)
    Message: All upstream servers failed (probe failed)
    Time: 22 minutes, 56 seconds ago
    Metadata: upstreams: 100.125.152.145:53
  [INFO] SYSTEM (54101547-c220-4d14-8ff2-fa0688aab236)
    Message: Network map updated
    Time: 22 minutes, 56 seconds ago
  [INFO] SYSTEM (f4042941-e7e0-4702-9c80-8faf86551354)
    Message: Network map updated
    Time: 17 minutes, 56 seconds ago
  [WARNING] DNS (2218aa6c-65c0-4dc0-8e19-0aa3c60db706)
    Message: All upstream servers failed (probe failed)
    Time: 6 seconds ago
    Metadata: upstreams: 100.125.152.145:53
  [INFO] SYSTEM (1669dca1-965f-4445-8f96-a61e49739fa2)
    Message: Network map updated
    Time: 6 seconds ago
OS: linux/amd64
Daemon version: 0.55.1
CLI version: 0.55.1
Profile: default
Management: Connected to https://vpn.anon-eywAf.domain:443
Signal: Connected to https://vpn.anon-eywAf.domain:443
Relays: 
  [stun:vpn.anon-eywAf.domain:3478] is Unavailable, reason: stun request: context deadline exceeded
  [turn:vpn.anon-eywAf.domain:3478?transport=udp] is Available
  [rels://vpn.anon-eywAf.domain:33080/relay] is Unavailable, reason: relay client not connected
Nameservers: 
  [1.1.1.1:53, 1.0.0.1:53] for [.] is Available
  [100.125.152.145:53] for [.] is Available
FQDN: haudisaudi.anon-hVzdp.domain
NetBird IP: 100.125.220.6/16
Interface type: Kernel
Quantum resistance: false
Lazy connection: false
Networks: -
Forwarding rules: 0
Peers count: 1/4 Connected

Create and upload a debug bundle, and share the returned file key:

6635ce03d98a358c955fce55a9f07c0c156288359b833d61a81e8abf6485ee14/4ad07634-30b6-4eb8-8ed7-b76f60a350fe

Additional Context

The following 2 commands from the troubleshooting page did not return anything so maybe the dns proxy did not start properly?

sudo ss -nlptu 'sport = 53' | grep netbird
sudo netstat -ltnup | grep ':53' | grep netbird

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @EkiciLP on GitHub (Aug 23, 2025). **Describe the problem** So I have a custom DNS Server in my netbird net which is configured as the main DNS for the network in the UI. On Linux clients it fails to add this DNS Server using systemd-resolved or resolvconf. When Connecting to the Net it throws this error: <img width="328" height="108" alt="Image" src="https://github.com/user-attachments/assets/2ded4e34-98e6-456e-913d-0b8bd41fb841" /> But I can confirm that the DNS Server is reachable because mobile devices work fine and a direct `nslookup local.domain <DNS-IP>` also works perfectly. **To Reproduce** Steps to reproduce the behavior: 1. Have a custom DNS configured and added to netbird 2. Install the Linux client using the official docs 3. Connect using UI instructions 4. Try to reach a local domain **Expected behavior** Resolving local domains correctly. **Are you using NetBird Cloud?** Self-Hosted **NetBird version** `0.55.1` **Is any other VPN software installed?**t Tailscale is installed but not configured. **Debug output** To help us resolve the problem, please attach the following anonymized status output ``` Peers detail: notebook-tim.anon-hVzdp.domain: NetBird IP: 100.125.6.124 Public key: UOecXL5x4tIPm7k5VDeEC8dHncJtNNWdaj5foin64RU= Status: Connecting -- detail -- Connection type: P2P ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 6 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Networks: - Latency: 0s vpn-server.anon-hVzdp.domain: NetBird IP: 100.125.75.34 Public key: 2+b9o09DLUjH5hopP5RSraYB3JtR/PBbTGGuAzlUmBw= Status: Connecting -- detail -- Connection type: P2P ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 6 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Networks: - Latency: 0s opnsense.anon-hVzdp.domain: NetBird IP: 100.125.152.145 Public key: pFVvqWabLQvsqWQC/lihUBcJaQ4nmJi5kpo0pqFfxHU= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/prflx ICE candidate endpoints (Local/Remote): 198.51.100.0:57935/10.0.0.1:51820 Relay server address: Last connection update: 6 seconds ago Last WireGuard handshake: 6 seconds ago Transfer status (received/sent) 284 B/276 B Quantum resistance: false Networks: 10.0.0.0/24, 10.0.1.0/24, 10.0.2.0/24 Latency: 354.416µs phone-tim.anon-hVzdp.domain: NetBird IP: 100.125.229.157 Public key: +gJhLUkm9IlMkaJS4Q76Ht6RU7VVNIKb8w7OrVbERzg= Status: Connecting -- detail -- Connection type: P2P ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 6 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Networks: - Latency: 0s Events: [INFO] SYSTEM (796e2dad-32d5-4ee2-b396-c243923f9017) Message: Network deselection changed Time: 28 minutes, 15 seconds ago Metadata: all: false, append: false, networks: Exit Node (vpn-server) [WARNING] DNS (878f18f1-622a-4676-abe3-1a9e4642beef) Message: All upstream servers failed (probe failed) Time: 27 minutes, 21 seconds ago Metadata: upstreams: 100.125.152.145:53 [INFO] SYSTEM (dd43b0d8-d430-4960-b296-6ee72425914e) Message: Network map updated Time: 27 minutes, 21 seconds ago [WARNING] DNS (26da8caa-bfe3-452d-aef8-aa3b01c96554) Message: All upstream servers failed (probe failed) Time: 23 minutes, 22 seconds ago Metadata: upstreams: 100.125.152.145:53 [INFO] SYSTEM (823510c5-327f-46de-9f2b-67d778ce7620) Message: Network map updated Time: 23 minutes, 22 seconds ago [WARNING] DNS (8c969815-415c-428c-9fe6-5b85e162ae19) Message: All upstream servers failed (probe failed) Time: 22 minutes, 56 seconds ago Metadata: upstreams: 100.125.152.145:53 [INFO] SYSTEM (54101547-c220-4d14-8ff2-fa0688aab236) Message: Network map updated Time: 22 minutes, 56 seconds ago [INFO] SYSTEM (f4042941-e7e0-4702-9c80-8faf86551354) Message: Network map updated Time: 17 minutes, 56 seconds ago [WARNING] DNS (2218aa6c-65c0-4dc0-8e19-0aa3c60db706) Message: All upstream servers failed (probe failed) Time: 6 seconds ago Metadata: upstreams: 100.125.152.145:53 [INFO] SYSTEM (1669dca1-965f-4445-8f96-a61e49739fa2) Message: Network map updated Time: 6 seconds ago OS: linux/amd64 Daemon version: 0.55.1 CLI version: 0.55.1 Profile: default Management: Connected to https://vpn.anon-eywAf.domain:443 Signal: Connected to https://vpn.anon-eywAf.domain:443 Relays: [stun:vpn.anon-eywAf.domain:3478] is Unavailable, reason: stun request: context deadline exceeded [turn:vpn.anon-eywAf.domain:3478?transport=udp] is Available [rels://vpn.anon-eywAf.domain:33080/relay] is Unavailable, reason: relay client not connected Nameservers: [1.1.1.1:53, 1.0.0.1:53] for [.] is Available [100.125.152.145:53] for [.] is Available FQDN: haudisaudi.anon-hVzdp.domain NetBird IP: 100.125.220.6/16 Interface type: Kernel Quantum resistance: false Lazy connection: false Networks: - Forwarding rules: 0 Peers count: 1/4 Connected ``` Create and upload a debug bundle, and share the returned file key: `6635ce03d98a358c955fce55a9f07c0c156288359b833d61a81e8abf6485ee14/4ad07634-30b6-4eb8-8ed7-b76f60a350fe` **Additional Context** The following 2 commands from the troubleshooting page did not return anything so maybe the dns proxy did not start properly? ```bash sudo ss -nlptu 'sport = 53' | grep netbird sudo netstat -ltnup | grep ':53' | grep netbird ``` **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2025-11-20 07:06:05 -05:00
saavagebueno commented 2025-11-20 07:06:06 -05:00
Author
Owner
Copy Link

@sparkycz1 commented on GitHub (Sep 1, 2025):

Hello, I can confirm this issue. Netbird with its own DNS server is unusable on Linux with systemd-resolved. Please increase the priority. Thank you.

@sparkycz1 commented on GitHub (Sep 1, 2025): Hello, I can confirm this issue. Netbird with its own DNS server is unusable on Linux with systemd-resolved. Please increase the priority. Thank you.
saavagebueno commented 2025-11-20 07:06:06 -05:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Sep 1, 2025):

  1. It is normal for a Nameserver to fail briefly for the internal IPs before their routing is established. Nameservers are applied to the system before the connectivity with Peers is established and the resulting Network Resources/Routes are added to the system.
  2. I am not sure why are you trying to:
    1. register 2 global nameservers: they will clash with each other
    2. register NetBird's internal resolver as a Nameserver as a global nameserver. This won't be able to answer any more queries than the client-side's internal resolver
  3. I see some binding dns on 100.125.220.6:53 is not available, error: listen udp 100.125.220.6:53: bind: address already in use errors suggesting something else is already running on the port 53 on all interfaces. You can manually tell NetBird client daemon where to listen with NB_DNS_RESOLVER_ADDRESS envvar or --dns-resolver-address argument on netbird up
@nazarewk commented on GitHub (Sep 1, 2025): 1. It is normal for a Nameserver to fail briefly for the internal IPs before their routing is established. Nameservers are applied to the system before the connectivity with Peers is established and the resulting Network Resources/Routes are added to the system. 2. I am not sure why are you trying to: 1. register 2 global nameservers: they will clash with each other 2. register NetBird's internal resolver as a Nameserver as a global nameserver. This won't be able to answer any more queries than the client-side's internal resolver 3. I see some `binding dns on 100.125.220.6:53 is not available, error: listen udp 100.125.220.6:53: bind: address already in use` errors suggesting something else is already running on the port `53` on all interfaces. You can manually tell NetBird client daemon where to listen with `NB_DNS_RESOLVER_ADDRESS` envvar or `--dns-resolver-address` argument on `netbird up`
saavagebueno commented 2025-11-20 07:06:06 -05:00
Author
Owner
Copy Link

@ednxzu commented on GitHub (Sep 14, 2025):

Hello,

you mentioned

Nameservers are applied to the system before the connectivity with Peers is established and the resulting Network Resources/Routes are added to the system.

This however causes issues when using split horizon DNS.

If for example, netbird management node is registered as vpn.domain.tld in some public dns server upstream, but I have configured my in-vpn dns servers to server requests for domain.tld (which is also an internal domain, with different records)

Netbird will choke on itself and complain it cannot resolve the master name when trying to connect, and fail.

Example:

vpn.domain.tldf registered in 1.1.1.1.

I connect to my vpn mesh -> dns gets updated to include the netbird internal ones.

My internal netbird dns is set to resolve domain.tld,

because it is a split horizon DNS and the internal servers resolve many more records than the public servers, including the vpn.domain.tld name.

Since I am not connected yet, I can't resolve vpn.domain.tld anymore 

because of the DNS injection from netbird, so I can't connect.
@ednxzu commented on GitHub (Sep 14, 2025): Hello, you mentioned > Nameservers are applied to the system before the connectivity with Peers is established and the resulting Network Resources/Routes are added to the system. This however causes issues when using split horizon DNS. If for example, netbird management node is registered as vpn.domain.tld in some public dns server upstream, but I have configured my in-vpn dns servers to server requests for domain.tld (which is also an internal domain, with different records) Netbird will choke on itself and complain it cannot resolve the master name when trying to connect, and fail. Example: ``` vpn.domain.tldf registered in 1.1.1.1. I connect to my vpn mesh -> dns gets updated to include the netbird internal ones. My internal netbird dns is set to resolve domain.tld, because it is a split horizon DNS and the internal servers resolve many more records than the public servers, including the vpn.domain.tld name. Since I am not connected yet, I can't resolve vpn.domain.tld anymore because of the DNS injection from netbird, so I can't connect. ```
saavagebueno commented 2025-11-20 07:06:07 -05:00
Author
Owner
Copy Link

@tkloda commented on GitHub (Sep 23, 2025):

I can confirm this warning appears consistently across Linux, macOS, and Windows clients. It's generated upon every successful connection to the VPN.

The primary issue is the user confusion this causes. Although the warning is displayed, DNS resolution works perfectly, suggesting the warning is a false positive.

Could this be a race condition? It seems the warning may be triggered prematurely, before the DNS subsystem has fully initialized.

@tkloda commented on GitHub (Sep 23, 2025): I can confirm this warning appears consistently across Linux, macOS, and Windows clients. It's generated upon every successful connection to the VPN. The primary issue is the user confusion this causes. Although the warning is displayed, DNS resolution works perfectly, suggesting the warning is a false positive. Could this be a race condition? It seems the warning may be triggered prematurely, before the DNS subsystem has fully initialized.
saavagebueno commented 2025-11-20 07:06:07 -05:00
Author
Owner
Copy Link

@EkiciLP commented on GitHub (Oct 1, 2025):

Coming back to this: for me the issue was fixed because DNS just does not look up .lan Domains, which I used previously, correctly. I found this out after randomly using a normal domain on my local dns. This was a bad practice anyways so I set all my services to a subdomain of an actual domain.
I will close this issue because it was effectively fixed, but the errors in the logs still exist.

@EkiciLP commented on GitHub (Oct 1, 2025): Coming back to this: for me the issue was fixed because DNS just does not look up .lan Domains, which I used previously, correctly. I found this out after randomly using a normal domain on my local dns. This was a bad practice anyways so I set all my services to a subdomain of an actual domain. I will close this issue because it was effectively fixed, but the errors in the logs still exist.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#2222
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?