SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-06-10 17:31:53 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

DNS Nameservers read udp timeout #1641

New Issue
Open
opened 2025-11-20 06:03:57 -05:00 by saavagebueno · 11 comments
saavagebueno commented 2025-11-20 06:03:57 -05:00
Owner
Copy Link

Originally created by @nattapong-atk on GitHub (Feb 21, 2025).

Describe the problem

I have an issue with the DNS nameserver override. Although the internal nameserver mapping for gitlab.mwbkk.com is configured in the NetBird Dashboard (pointing to 100.83.38.245), my Mac client fails to use this override. Instead, it times out when trying to query the internal DNS server, resulting in DNS resolution errors and the message “All upstream servers failed (probe failed)” in the logs.

To Reproduce

  1. Go to the NetBird Dashboard and navigate to the DNS > Nameservers section.
  2. Configure a nameserver mapping for the domain gitlab.example.com with the internal DNS IP set to 100.83.38.245 and set the Match Domain to gitlab.example.com.
  3. Ensure that the BIND DNS server on your GitLab server is properly configured with a zone file like:
$TTL    604800
@       IN      SOA     ns1.example.com. admin.example.com. (
                              2025022201
                              604800
                              86400
                              2419200
                              604800 )
;
@       IN      NS      ns1.example.com.
ns1     IN      A       100.83.38.245
gitlab  IN      A       100.83.38.245
  1. Connect your Mac client to the NetBird VPN.
  2. Run netbird status -d on the Mac and observe DNS events showing the error:
read udp 100.83.178.108:51646->100.83.38.245:53: i/o timeout
  1. Attempt to access gitlab.example.com via a browser or command-line tools (e.g., curl), and note that the DNS query fails due to the timeout.

Expected behavior

The Mac client should successfully query the internal DNS server at 100.83.38.245 without any UDP read timeouts. The netbird status -d output should indicate that the nameserver mapping for gitlab.example.com is available (e.g., “Nameservers: 1/1 Available”), allowing DNS queries for gitlab.example.com to resolve correctly.

Are you using NetBird Cloud?

I am using a self-hosted NetBird control plane.

NetBird version

Daemon version: 0.37.0
CLI version: 0.37.0

NetBird status -d output:

Peers detail:
Events:
  [WARNING] DNS (a09e64d0-5dfe-4f81-a9b8-581add06b2a5)
    Message: All upstream servers failed (probe failed)
    Time: 14 minutes, 56 seconds ago
    Metadata: upstreams: 100.83.38.245:53
  [INFO] SYSTEM (5756441c-4d1c-454e-8b10-e1d249e76095)
    Message: Network map updated
    Time: 14 minutes, 56 seconds ago
  [WARNING] DNS (ecf4cda7-ec46-4f5a-87e5-b610b1c4c293)
    Message: All upstream servers failed (probe failed)
    Time: 14 minutes, 47 seconds ago
    Metadata: upstreams: 100.83.38.245:53
  [INFO] SYSTEM (38328189-566b-458a-807a-e766b804095f)
    Message: Network map updated
    Time: 14 minutes, 47 seconds ago
  [WARNING] DNS (800d36ee-8e8b-44ac-8b5a-134303498230)
    Message: All upstream servers failed (probe failed)
    Time: 9 minutes, 40 seconds ago
    Metadata: upstreams: 100.83.38.245:53
  [INFO] SYSTEM (a11118b2-3d9a-419b-8311-8c24b4af02de)
    Message: Network map updated
    Time: 9 minutes, 40 seconds ago
  [WARNING] DNS (5a06d02f-a65a-4ab0-a998-f8fdb6e496ff)
    Message: All upstream servers failed (probe failed)
    Time: 8 minutes, 35 seconds ago
    Metadata: upstreams: 100.83.38.245:53
  [INFO] SYSTEM (d6623f55-e8d3-4a72-89f3-25f5dd4fd30c)
    Message: Network map updated
    Time: 8 minutes, 35 seconds ago
  [WARNING] DNS (d634501c-a938-4c17-9022-b035aaed396e)
    Message: All upstream servers failed (probe failed)
    Time: 7 minutes, 41 seconds ago
    Metadata: upstreams: 100.83.38.245:53
  [INFO] SYSTEM (f6018af1-27e8-4467-bc04-d12bb7171a8e)
    Message: Network map updated
    Time: 7 minutes, 41 seconds ago
OS: darwin/arm64
Daemon version: 0.37.0
CLI version: 0.37.0
Management: Connected to https://netbird.example.com:443
Signal: Connected to https://netbird.example.com:443
Relays:
  [stun:netbird.example.com:3478] is Available
  [turn:netbird.example.com:3478?transport=udp] is Available
  [rels://netbird.example.com:443] is Available
Nameservers:
  [100.83.38.245:53] for [gitlab.example.com] is Unavailable, reason: 1 error occurred:
	* read udp 100.83.178.108:51646->100.83.38.245:53: i/o timeout
FQDN: macbook-pro-s-nattapong.netbird.selfhosted
NetBird IP: 100.83.178.108/16
Interface type: Userspace
Quantum resistance: false
Networks: -
Peers count: 0/0 Connected

Screenshots

Image

Image

Additional context

  • Direct queries to the BIND DNS server (e.g., using dig gitlab.example.com @100.83.38.245) return the correct A record, confirming that the internal DNS server is functioning.
  • The error messages indicate that the client is unable to reach the internal nameserver (100.83.38.245) due to UDP timeouts, suggesting a potential connectivity issue or misconfiguration in the DNS override settings.
  • Possible causes might include a distribution group mismatch or a network connectivity issue between the client and the internal DNS server.
  • Restarting the VPN client or reconnecting does not resolve the issue.

I hope this report provides the necessary details to help troubleshoot the "Nameservers read udp timeout" issue with the DNS override in NetBird.

Originally created by @nattapong-atk on GitHub (Feb 21, 2025). **Describe the problem** I have an issue with the DNS nameserver override. Although the internal nameserver mapping for gitlab.mwbkk.com is configured in the NetBird Dashboard (pointing to 100.83.38.245), my Mac client fails to use this override. Instead, it times out when trying to query the internal DNS server, resulting in DNS resolution errors and the message “All upstream servers failed (probe failed)” in the logs. **To Reproduce** 1. Go to the NetBird Dashboard and navigate to the DNS > Nameservers section. 2. Configure a nameserver mapping for the domain gitlab.example.com with the internal DNS IP set to 100.83.38.245 and set the Match Domain to gitlab.example.com. 3. Ensure that the BIND DNS server on your GitLab server is properly configured with a zone file like: ``` $TTL 604800 @ IN SOA ns1.example.com. admin.example.com. ( 2025022201 604800 86400 2419200 604800 ) ; @ IN NS ns1.example.com. ns1 IN A 100.83.38.245 gitlab IN A 100.83.38.245 ``` 4. Connect your Mac client to the NetBird VPN. 5. Run netbird status -d on the Mac and observe DNS events showing the error: ``` read udp 100.83.178.108:51646->100.83.38.245:53: i/o timeout ``` 6. Attempt to access gitlab.example.com via a browser or command-line tools (e.g., curl), and note that the DNS query fails due to the timeout. **Expected behavior** The Mac client should successfully query the internal DNS server at 100.83.38.245 without any UDP read timeouts. The netbird status -d output should indicate that the nameserver mapping for gitlab.example.com is available (e.g., “Nameservers: 1/1 Available”), allowing DNS queries for gitlab.example.com to resolve correctly. **Are you using NetBird Cloud?** I am using a self-hosted NetBird control plane. **NetBird version** Daemon version: 0.37.0 CLI version: 0.37.0 **NetBird status -d output:** ``` Peers detail: Events: [WARNING] DNS (a09e64d0-5dfe-4f81-a9b8-581add06b2a5) Message: All upstream servers failed (probe failed) Time: 14 minutes, 56 seconds ago Metadata: upstreams: 100.83.38.245:53 [INFO] SYSTEM (5756441c-4d1c-454e-8b10-e1d249e76095) Message: Network map updated Time: 14 minutes, 56 seconds ago [WARNING] DNS (ecf4cda7-ec46-4f5a-87e5-b610b1c4c293) Message: All upstream servers failed (probe failed) Time: 14 minutes, 47 seconds ago Metadata: upstreams: 100.83.38.245:53 [INFO] SYSTEM (38328189-566b-458a-807a-e766b804095f) Message: Network map updated Time: 14 minutes, 47 seconds ago [WARNING] DNS (800d36ee-8e8b-44ac-8b5a-134303498230) Message: All upstream servers failed (probe failed) Time: 9 minutes, 40 seconds ago Metadata: upstreams: 100.83.38.245:53 [INFO] SYSTEM (a11118b2-3d9a-419b-8311-8c24b4af02de) Message: Network map updated Time: 9 minutes, 40 seconds ago [WARNING] DNS (5a06d02f-a65a-4ab0-a998-f8fdb6e496ff) Message: All upstream servers failed (probe failed) Time: 8 minutes, 35 seconds ago Metadata: upstreams: 100.83.38.245:53 [INFO] SYSTEM (d6623f55-e8d3-4a72-89f3-25f5dd4fd30c) Message: Network map updated Time: 8 minutes, 35 seconds ago [WARNING] DNS (d634501c-a938-4c17-9022-b035aaed396e) Message: All upstream servers failed (probe failed) Time: 7 minutes, 41 seconds ago Metadata: upstreams: 100.83.38.245:53 [INFO] SYSTEM (f6018af1-27e8-4467-bc04-d12bb7171a8e) Message: Network map updated Time: 7 minutes, 41 seconds ago OS: darwin/arm64 Daemon version: 0.37.0 CLI version: 0.37.0 Management: Connected to https://netbird.example.com:443 Signal: Connected to https://netbird.example.com:443 Relays: [stun:netbird.example.com:3478] is Available [turn:netbird.example.com:3478?transport=udp] is Available [rels://netbird.example.com:443] is Available Nameservers: [100.83.38.245:53] for [gitlab.example.com] is Unavailable, reason: 1 error occurred: * read udp 100.83.178.108:51646->100.83.38.245:53: i/o timeout FQDN: macbook-pro-s-nattapong.netbird.selfhosted NetBird IP: 100.83.178.108/16 Interface type: Userspace Quantum resistance: false Networks: - Peers count: 0/0 Connected ``` **Screenshots** ![Image](https://github.com/user-attachments/assets/8e2f093f-7c31-4db5-b551-0ab87284a779) ![Image](https://github.com/user-attachments/assets/f51853c1-d66d-4e5f-997f-d14dd3e31a8f) **Additional context** - Direct queries to the BIND DNS server (e.g., using dig gitlab.example.com @100.83.38.245) return the correct A record, confirming that the internal DNS server is functioning. - The error messages indicate that the client is unable to reach the internal nameserver (100.83.38.245) due to UDP timeouts, suggesting a potential connectivity issue or misconfiguration in the DNS override settings. - Possible causes might include a distribution group mismatch or a network connectivity issue between the client and the internal DNS server. - Restarting the VPN client or reconnecting does not resolve the issue. I hope this report provides the necessary details to help troubleshoot the "Nameservers read udp timeout" issue with the DNS override in NetBird.
saavagebueno added the question label 2025-11-20 06:03:57 -05:00
saavagebueno commented 2025-11-20 06:03:58 -05:00
Author
Owner
Copy Link

@nattapong-atk commented on GitHub (Feb 21, 2025):

Do you face any (non-mobile) client issues?

Yes.

This is the result of debugging netbird for 1m -AS.
netbird.debug.1062627925.zip

And This is the result of debugging netbird debug bundle -AS.
netbird.debug.1119080183.zip

@nattapong-atk commented on GitHub (Feb 21, 2025): Do you face any (non-mobile) client issues? Yes. This is the result of debugging `netbird for 1m -AS`. [netbird.debug.1062627925.zip](https://github.com/user-attachments/files/18915104/netbird.debug.1062627925.zip) And This is the result of debugging `netbird debug bundle -AS`. [netbird.debug.1119080183.zip](https://github.com/user-attachments/files/18915118/netbird.debug.1119080183.zip)
saavagebueno commented 2025-11-20 06:03:58 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Feb 22, 2025):

@nattapong-atk can you please share the bundle from the following command without the -A flag:

netbird for 1m -S
@mlsmaycon commented on GitHub (Feb 22, 2025): @nattapong-atk can you please share the bundle from the following command without the -A flag: ``` netbird for 1m -S ```
saavagebueno commented 2025-11-20 06:03:58 -05:00
Author
Owner
Copy Link

@nattapong-atk commented on GitHub (Feb 22, 2025):

@mlsmaycon
This is the result of debugging netbird debug for 1m -S

netbird.debug.1369215731.zip

@nattapong-atk commented on GitHub (Feb 22, 2025): @mlsmaycon This is the result of debugging `netbird debug for 1m -S` [netbird.debug.1369215731.zip](https://github.com/user-attachments/files/18921979/netbird.debug.1369215731.zip)
saavagebueno commented 2025-11-20 06:03:58 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Feb 22, 2025):

Hi @nattapong-atk,

can you confirm that you have an access policy in place to allow access to the peer? If not, you will have to add it: https://docs.netbird.io/how-to/manage-network-access

Currently you have no accessible peers:

Peers count: 0/0 Connected
@lixmal commented on GitHub (Feb 22, 2025): Hi @nattapong-atk, can you confirm that you have an access policy in place to allow access to the peer? If not, you will have to add it: https://docs.netbird.io/how-to/manage-network-access Currently you have no accessible peers: ``` Peers count: 0/0 Connected ```
saavagebueno commented 2025-11-20 06:03:58 -05:00
Author
Owner
Copy Link

@nattapong-atk commented on GitHub (Feb 22, 2025):

@lixmal
I confirm that I haven't set up any Access Policies yet.

Image

Image

Image

Image

Image

Image

Image

Image

@nattapong-atk commented on GitHub (Feb 22, 2025): @lixmal I confirm that I haven't set up any Access Policies yet. ![Image](https://github.com/user-attachments/assets/08dd29f1-e302-4474-a5b3-b71a9addd798) ![Image](https://github.com/user-attachments/assets/e2dcc5bb-974e-48bf-a839-30477bbda631) ![Image](https://github.com/user-attachments/assets/53ddb85a-c6a9-4924-956b-55b1689ab54e) ![Image](https://github.com/user-attachments/assets/403684be-b436-493e-a8e8-0522f29fdc97) ![Image](https://github.com/user-attachments/assets/c6b142e6-16cf-4fdf-9a77-f22252811d0f) ![Image](https://github.com/user-attachments/assets/c6454752-096c-47c1-908e-f5bcd8579b69) ![Image](https://github.com/user-attachments/assets/fcddabbc-1696-414b-9614-b06c4c370db3) ![Image](https://github.com/user-attachments/assets/47e793d3-864a-4b65-91c6-664191293871)
saavagebueno commented 2025-11-20 06:03:59 -05:00
Author
Owner
Copy Link

@nattapong-atk commented on GitHub (Feb 22, 2025):

@lixmal
I have now set up the Access Policies as suggested (Source Group ALL to Destination Group ALL), and I'm back. However, my peers count still shows 2/19 Connected.

Image

and I ran the command netbird debug for 1m -S again after setting the Access Policies (Source Group ALL to Destination Group ALL). Here is the output I received:

netbird.debug.3732902210.zip

@nattapong-atk commented on GitHub (Feb 22, 2025): @lixmal I have now set up the Access Policies as suggested (Source Group ALL to Destination Group ALL), and I'm back. However, my peers count still shows 2/19 Connected. ![Image](https://github.com/user-attachments/assets/675cb57f-56cd-4365-bb1c-cf697288d708) and I ran the command `netbird debug for 1m -S` again after setting the Access Policies (Source Group ALL to Destination Group ALL). Here is the output I received: [netbird.debug.3732902210.zip](https://github.com/user-attachments/files/18922493/netbird.debug.3732902210.zip)
saavagebueno commented 2025-11-20 06:03:59 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Feb 22, 2025):

The configuration looks correct. Can you verify the addresses or interfaces your dns server binds to?
Netbird currently tries resolving com. to verify that a dns server is reachable, if your dns server doesn't resolve it, that might be the reason it is marked unavailable.

@lixmal commented on GitHub (Feb 22, 2025): The configuration looks correct. Can you verify the addresses or interfaces your dns server binds to? Netbird currently tries resolving `com.` to verify that a dns server is reachable, if your dns server doesn't resolve it, that might be the reason it is marked unavailable.
saavagebueno commented 2025-11-20 06:03:59 -05:00
Author
Owner
Copy Link

@nattapong-atk commented on GitHub (Feb 22, 2025):

this my DNS setting

Image

in netbird status -d result

Image

Nameservers:
  [100.83.38.245:53] for [gitlab.mwbkk.com] is Unavailable, reason: 1 error occurred:
	* read udp 100.83.178.108:52348->100.83.38.245:53: i/o timeout
@nattapong-atk commented on GitHub (Feb 22, 2025): this my DNS setting ![Image](https://github.com/user-attachments/assets/d12f0e81-889d-4101-a6af-1b647a7253ae) in `netbird status -d` result ![Image](https://github.com/user-attachments/assets/98901822-7c90-4f4c-aa96-5e47a7bb90b4) ``` Nameservers: [100.83.38.245:53] for [gitlab.mwbkk.com] is Unavailable, reason: 1 error occurred: * read udp 100.83.178.108:52348->100.83.38.245:53: i/o timeout ```
saavagebueno commented 2025-11-20 06:03:59 -05:00
Author
Owner
Copy Link

@nattapong-atk commented on GitHub (Feb 22, 2025):

@lixmal
i run command dig gitlab.mwbkk.com @100.83.38.245 for check DNS Server
this result:

; <<>> DiG 9.10.6 <<>> gitlab.mwbkk.com @100.83.38.245
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45739
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;gitlab.mwbkk.com.		IN	A

;; ANSWER SECTION:
gitlab.mwbkk.com.	604800	IN	A	100.83.38.245

;; Query time: 1 msec
;; SERVER: 100.83.38.245#53(100.83.38.245)
;; WHEN: Sat Feb 22 19:51:52 +07 2025
;; MSG SIZE  rcvd: 61
@nattapong-atk commented on GitHub (Feb 22, 2025): @lixmal i run command `dig gitlab.mwbkk.com @100.83.38.245` for check DNS Server this result: ``` ; <<>> DiG 9.10.6 <<>> gitlab.mwbkk.com @100.83.38.245 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45739 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ;; QUESTION SECTION: ;gitlab.mwbkk.com. IN A ;; ANSWER SECTION: gitlab.mwbkk.com. 604800 IN A 100.83.38.245 ;; Query time: 1 msec ;; SERVER: 100.83.38.245#53(100.83.38.245) ;; WHEN: Sat Feb 22 19:51:52 +07 2025 ;; MSG SIZE rcvd: 61 ```
saavagebueno commented 2025-11-20 06:03:59 -05:00
Author
Owner
Copy Link

@interface-fluidics commented on GitHub (Feb 26, 2025):

I have the same issue.
netbird adding . to my domains.
on top of this whole PC performance is degraded while resolving dns
netbird.debug.376836878.zip

@interface-fluidics commented on GitHub (Feb 26, 2025): I have the same issue. netbird adding . to my domains. on top of this whole PC performance is degraded while resolving dns [netbird.debug.376836878.zip](https://github.com/user-attachments/files/18989149/netbird.debug.376836878.zip)
saavagebueno commented 2025-11-20 06:03:59 -05:00
Author
Owner
Copy Link

@hti-un commented on GitHub (Apr 2, 2025):

DNS problem with netbird on Mac OS solved for us!

In our case it was due to the search domain distributed via dhcp in the network.
Mac OS makes a separate resolver entry for this. This can be displayed using scutil --dns. The moment you don't roll out a search domain via dhcp, it works.

Our configuration:
Mac OS clients in an isolated VLAN with dhcp (without search domain) and public DNS, e.g. 1.1.1.1
In the same network, a routing peer that has access to the internal networks via firewall rules.
Via networks and policies in netbird, individual resources (IP/port) from the internal network can be used by the clients in the isolated VLAN.
The speed is very good because the clients establish a P2P connection to the routing peer.
Since netbird also rolls out the search domain, we have lost nothing ;-)

Works great!

@hti-un commented on GitHub (Apr 2, 2025): DNS problem with netbird on Mac OS solved for us! In our case it was due to the search domain distributed via dhcp in the network. Mac OS makes a separate resolver entry for this. This can be displayed using scutil --dns. The moment you don't roll out a search domain via dhcp, it works. Our configuration: Mac OS clients in an isolated VLAN with dhcp (without search domain) and public DNS, e.g. 1.1.1.1 In the same network, a routing peer that has access to the internal networks via firewall rules. Via networks and policies in netbird, individual resources (IP/port) from the internal network can be used by the clients in the isolated VLAN. The speed is very good because the clients establish a P2P connection to the routing peer. Since netbird also rolls out the search domain, we have lost nothing ;-) Works great!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1641
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?