[GH-ISSUE #2157] DNS resolution fails when running a local resolver #4431

Open
opened 2026-08-05 00:57:41 -04:00 by saavagebueno · 26 comments
Owner

Originally created by @MDMeridio001 on GitHub (Jun 19, 2024).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/2157

Describe the problem

I run a local DNS resolver (unbound) on a netbird peer running Ubuntu Server 22.04.4 LTS. DNS management is disabled for the peer in question via the netbird dashboard. Everything worked fine in version 0.27.10, but after updating to 0.28.0 the peer is not able to reach the unbound server running on localhost and every DNS query results in a timeout.

To Reproduce

Steps to reproduce the behavior:

  1. Run "netbird up" on the peer
  2. Run "dig google.com @127.0.0.1" on the peer
  3. dig command results in ";; communications error to 127.0.0.1#53: timed out"
  4. Run "netbird down" on the peer
  5. DNS resolution is back to normal

Expected behavior

I expect the DNS resolution to not be impacted whether or not the peer is connected to netbird, especially because DNS management is disabled in the dashboard for the said peer.

Are you using NetBird Cloud?

No, I'm self-hosting it.

NetBird version

0.28.0

Originally created by @MDMeridio001 on GitHub (Jun 19, 2024). Original GitHub issue: https://github.com/netbirdio/netbird/issues/2157 **Describe the problem** I run a local DNS resolver (unbound) on a netbird peer running Ubuntu Server 22.04.4 LTS. DNS management is disabled for the peer in question via the netbird dashboard. Everything worked fine in version 0.27.10, but after updating to 0.28.0 the peer is not able to reach the unbound server running on localhost and every DNS query results in a timeout. **To Reproduce** Steps to reproduce the behavior: 1. Run "netbird up" on the peer 2. Run "dig google.com @127.0.0.1" on the peer 3. dig command results in ";; communications error to 127.0.0.1#53: timed out" 4. Run "netbird down" on the peer 5. DNS resolution is back to normal **Expected behavior** I expect the DNS resolution to not be impacted whether or not the peer is connected to netbird, especially because DNS management is disabled in the dashboard for the said peer. **Are you using NetBird Cloud?** No, I'm self-hosting it. **NetBird version** 0.28.0
saavagebueno added the bugclientdns labels 2026-08-05 00:57:41 -04:00
Author
Owner

@mrota83 commented on GitHub (Jun 19, 2024):

same thing here worked fine before 0.28.0

<!-- gh-comment-id:2179641639 --> @mrota83 commented on GitHub (Jun 19, 2024): same thing here worked fine before 0.28.0
Author
Owner

@nikolicjakov commented on GitHub (Jun 20, 2024):

I am having same problem running new 0.28.0 version. And i am also self hosting.

WARN [error: read udp 192.168.25.40:56297->192.168.40.168:53: i/o timeout, upstream: 192.168.40.168:53] client/internal/dns/upstream.go:101: got an error while connecting to upstream

DNS_PROBE_FINISHED_NXDOMAIN

I have 2 DNS servers running in private network and i have created network route to this subnet so that anyone who is connected to vpn can access these DNS servers (one of them 192.168.40.168). But for some reason after 0.28.0 update i am only experiencing issues with internal domain resolutions and i am seeing file getting created every 10 seconds.

And if i am trying to open internal domain client connection breaks since i can see my peer disappearing from dashboard and following files/logs are written...

unclean_shutdown_dns.txt with some strange id inside..

2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.168:53: read udp 192.168.25.40:59355->192.168.40.168:53: i/o timeout
2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.68:53: read udp 192.168.25.40:59354->192.168.40.68:53: i/o timeout
2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:262: Upstream resolving is Disabled for 30s

Something is definitely wrong with communication after 0.28.0 update please take look at this issue... 😢

<!-- gh-comment-id:2179905146 --> @nikolicjakov commented on GitHub (Jun 20, 2024): I am having same problem running new 0.28.0 version. And i am also self hosting. `WARN [error: read udp 192.168.25.40:56297->192.168.40.168:53: i/o timeout, upstream: 192.168.40.168:53] client/internal/dns/upstream.go:101: got an error while connecting to upstream` DNS_PROBE_FINISHED_NXDOMAIN I have 2 DNS servers running in private network and i have created network route to this subnet so that anyone who is connected to vpn can access these DNS servers (one of them 192.168.40.168). But for some reason after 0.28.0 update i am only experiencing issues with internal domain resolutions and i am seeing file getting created every 10 seconds. And if i am trying to open internal domain client connection breaks since i can see my peer disappearing from dashboard and following files/logs are written... `unclean_shutdown_dns.txt` with some strange id inside.. ``` 2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.168:53: read udp 192.168.25.40:59355->192.168.40.168:53: i/o timeout 2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.68:53: read udp 192.168.25.40:59354->192.168.40.68:53: i/o timeout 2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:262: Upstream resolving is Disabled for 30s ``` Something is definitely wrong with communication after 0.28.0 update please take look at this issue... 😢
Author
Owner

@lixmal commented on GitHub (Jun 20, 2024):

Can you provide a netbird status -dA please? Even better a complete debug bundle netbird debug for 1m -A

<!-- gh-comment-id:2179963612 --> @lixmal commented on GitHub (Jun 20, 2024): Can you provide a `netbird status -dA` please? Even better a complete debug bundle `netbird debug for 1m -A`
Author
Owner

@nikolicjakov commented on GitHub (Jun 20, 2024):

netbird.debug.3041785520.zip

@lixmal Here you go...

<!-- gh-comment-id:2179970296 --> @nikolicjakov commented on GitHub (Jun 20, 2024): [netbird.debug.3041785520.zip](https://github.com/user-attachments/files/15910150/netbird.debug.3041785520.zip) @lixmal Here you go...
Author
Owner

@yaneony commented on GitHub (Jun 20, 2024):

Having same issue here.

<!-- gh-comment-id:2180428032 --> @yaneony commented on GitHub (Jun 20, 2024): Having same issue here.
Author
Owner

@mlsmaycon commented on GitHub (Jun 20, 2024):

@yaneony @mrota83 @MDMeridio001 are you running on windows? can you share the output requested here: https://github.com/netbirdio/netbird/issues/2157#issuecomment-2179963612 ?

<!-- gh-comment-id:2180454861 --> @mlsmaycon commented on GitHub (Jun 20, 2024): @yaneony @mrota83 @MDMeridio001 are you running on windows? can you share the output requested here: https://github.com/netbirdio/netbird/issues/2157#issuecomment-2179963612 ?
Author
Owner

@nikolicjakov commented on GitHub (Jun 20, 2024):

Thank you guys for new release everything looks to be working fine in v0.28.1

<!-- gh-comment-id:2180587428 --> @nikolicjakov commented on GitHub (Jun 20, 2024): Thank you guys for new release everything looks to be working fine in v0.28.1
Author
Owner

@MDMeridio001 commented on GitHub (Jun 20, 2024):

@mlsmaycon No, the client is an Ubuntu Server machine. I have updated to 0.28.1 but I'm still facing the same issue.
Here is the output of netbird debug for 1m -A: netbird.debug.80462564.zip

<!-- gh-comment-id:2180610962 --> @MDMeridio001 commented on GitHub (Jun 20, 2024): @mlsmaycon No, the client is an Ubuntu Server machine. I have updated to 0.28.1 but I'm still facing the same issue. Here is the output of `netbird debug for 1m -A`: [netbird.debug.80462564.zip](https://github.com/user-attachments/files/15914108/netbird.debug.80462564.zip)
Author
Owner

@nikolicjakov commented on GitHub (Jun 20, 2024):

What did you update client or server components to new v0.28.1 version?

<!-- gh-comment-id:2180805574 --> @nikolicjakov commented on GitHub (Jun 20, 2024): What did you update client or server components to new v0.28.1 version?
Author
Owner

@MDMeridio001 commented on GitHub (Jun 20, 2024):

@nikolicjakov both of them.

<!-- gh-comment-id:2180822841 --> @MDMeridio001 commented on GitHub (Jun 20, 2024): @nikolicjakov both of them.
Author
Owner

@MDMeridio001 commented on GitHub (Jun 21, 2024):

Just an update, I have also tried updating both the server components and the client to version 0.28.2 and the issue persists.

<!-- gh-comment-id:2182385945 --> @MDMeridio001 commented on GitHub (Jun 21, 2024): Just an update, I have also tried updating both the server components and the client to version 0.28.2 and the issue persists.
Author
Owner

@Ryan00793 commented on GitHub (Jul 3, 2024):

This is not fixed in 0.28.3 hosts that previously had the issue however, on a fresh install of ubuntu 22.04 it appears to be working. Luckily for me this is being used in a VM that is only used for netbird so wasn't worried about other things.

<!-- gh-comment-id:2206047236 --> @Ryan00793 commented on GitHub (Jul 3, 2024): This is not fixed in 0.28.3 hosts that previously had the issue however, on a fresh install of ubuntu 22.04 it appears to be working. Luckily for me this is being used in a VM that is only used for netbird so wasn't worried about other things.
Author
Owner

@ykorzikowski commented on GitHub (Jul 3, 2024):

Maybe related to this https://github.com/netbirdio/netbird/issues/2219

I start noticing a really shaky server environment after I updated my server-netbird-clients (dns issues, packet loss).

Did not do any further debugging and downgraded all clients to 0.27.10 instead, looks stable again.

Edit:

Some minds: I notice i can dig&nslookup my custom dns server and the local netbird dns. But a curl results in a can not resolve error. This issue was not persistant (see my macbook dns firewall issue). Maybe its a iptables related issue? Some packages got filtered by a wrongly configured rule?

<!-- gh-comment-id:2206745335 --> @ykorzikowski commented on GitHub (Jul 3, 2024): Maybe related to this https://github.com/netbirdio/netbird/issues/2219 I start noticing a really shaky server environment after I updated my server-netbird-clients (dns issues, packet loss). Did not do any further debugging and downgraded all clients to 0.27.10 instead, looks stable again. Edit: Some minds: I notice i can dig&nslookup my custom dns server and the local netbird dns. But a curl results in a can not resolve error. This issue was not persistant (see my macbook dns firewall issue). Maybe its a iptables related issue? Some packages got filtered by a wrongly configured rule?
Author
Owner

@Riza-Aslan commented on GitHub (Jan 27, 2025):

I am having same problem running new 0.28.0 version. And i am also self hosting.

WARN [error: read udp 192.168.25.40:56297->192.168.40.168:53: i/o timeout, upstream: 192.168.40.168:53] client/internal/dns/upstream.go:101: got an error while connecting to upstream

DNS_PROBE_FINISHED_NXDOMAIN

I have 2 DNS servers running in private network and i have created network route to this subnet so that anyone who is connected to vpn can access these DNS servers (one of them 192.168.40.168). But for some reason after 0.28.0 update i am only experiencing issues with internal domain resolutions and i am seeing file getting created every 10 seconds.

And if i am trying to open internal domain client connection breaks since i can see my peer disappearing from dashboard and following files/logs are written...

unclean_shutdown_dns.txt with some strange id inside..

2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.168:53: read udp 192.168.25.40:59355->192.168.40.168:53: i/o timeout
2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.68:53: read udp 192.168.25.40:59354->192.168.40.68:53: i/o timeout
2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:262: Upstream resolving is Disabled for 30s

Something is definitely wrong with communication after 0.28.0 update please take look at this issue... 😢

I have exactly the same issue here. The strange thing is, that even in our internal network the clients are not able to resolve our internal domain anymore. My only solution is to reinstall the client and hope that it does not happen again...but it happens again.

Does anyone know how to fix the dns resolver when it has crashed?

Im using v.028.9 on Windows 11 (selfhosted)

<!-- gh-comment-id:2615086746 --> @Riza-Aslan commented on GitHub (Jan 27, 2025): > I am having same problem running new 0.28.0 version. And i am also self hosting. > > `WARN [error: read udp 192.168.25.40:56297->192.168.40.168:53: i/o timeout, upstream: 192.168.40.168:53] client/internal/dns/upstream.go:101: got an error while connecting to upstream` > > DNS_PROBE_FINISHED_NXDOMAIN > > I have 2 DNS servers running in private network and i have created network route to this subnet so that anyone who is connected to vpn can access these DNS servers (one of them 192.168.40.168). But for some reason after 0.28.0 update i am only experiencing issues with internal domain resolutions and i am seeing file getting created every 10 seconds. > > And if i am trying to open internal domain client connection breaks since i can see my peer disappearing from dashboard and following files/logs are written... > > `unclean_shutdown_dns.txt` with some strange id inside.. > > ``` > 2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.168:53: read udp 192.168.25.40:59355->192.168.40.168:53: i/o timeout > 2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:184: probing upstream nameserver 192.168.40.68:53: read udp 192.168.25.40:59354->192.168.40.68:53: i/o timeout > 2024-06-20T08:18:33+02:00 WARN client/internal/dns/upstream.go:262: Upstream resolving is Disabled for 30s > ``` > > Something is definitely wrong with communication after 0.28.0 update please take look at this issue... 😢 I have exactly the same issue here. The strange thing is, that even in our internal network the clients are not able to resolve our internal domain anymore. My only solution is to reinstall the client and hope that it does not happen again...but it happens again. Does anyone know how to fix the dns resolver when it has crashed? Im using v.028.9 on Windows 11 (selfhosted)
Author
Owner

@ptpu commented on GitHub (Mar 18, 2025):

Same issue here. I'm running 0.38.2 on all connected devices/server. At irregular intervals the DNS resolutions probe fails and the DNS server is marked as offline (timeout).

I have the local dns configured in the NetBird UI. Also I have configured an access policy allowing clients to connect to the server which is running dns on port UDP 53. When deactivating this rule and immediately activating it again, the dns server is marked as available on all the clients. Could this be a ACL bug?

Image

<!-- gh-comment-id:2734272451 --> @ptpu commented on GitHub (Mar 18, 2025): Same issue here. I'm running 0.38.2 on all connected devices/server. At irregular intervals the DNS resolutions probe fails and the DNS server is marked as offline (timeout). I have the local dns configured in the NetBird UI. Also I have configured an access policy allowing clients to connect to the server which is running dns on port UDP 53. When deactivating this rule and immediately activating it again, the dns server is marked as available on all the clients. Could this be a ACL bug? ![Image](https://github.com/user-attachments/assets/369416d1-0e5e-4987-a73a-61d8b87fa6ea)
Author
Owner

@scroguard commented on GitHub (Mar 20, 2025):

i too have this issue, for me it started in 0.38.2. however it's happening on more than just my dns access policies. i have had to toggle all my various policies off and then back on again to restore connectivity. at this point i have not been able to pinpoint anything that could be the root cause of the issue, thus i too wonder if it's an ACL bug.

Same issue here. I'm running 0.38.2 on all connected devices/server. At irregular intervals the DNS resolutions probe fails and the DNS server is marked as offline (timeout).

I have the local dns configured in the NetBird UI. Also I have configured an access policy allowing clients to connect to the server which is running dns on port UDP 53. When deactivating this rule and immediately activating it again, the dns server is marked as available on all the clients. Could this be a ACL bug?

Image

<!-- gh-comment-id:2740898291 --> @scroguard commented on GitHub (Mar 20, 2025): i too have this issue, for me it started in 0.38.2. however it's happening on more than just my dns access policies. i have had to toggle all my various policies off and then back on again to restore connectivity. at this point i have not been able to pinpoint anything that could be the root cause of the issue, thus i too wonder if it's an ACL bug. > Same issue here. I'm running 0.38.2 on all connected devices/server. At irregular intervals the DNS resolutions probe fails and the DNS server is marked as offline (timeout). > > I have the local dns configured in the NetBird UI. Also I have configured an access policy allowing clients to connect to the server which is running dns on port UDP 53. When deactivating this rule and immediately activating it again, the dns server is marked as available on all the clients. Could this be a ACL bug? > > ![Image](https://github.com/user-attachments/assets/369416d1-0e5e-4987-a73a-61d8b87fa6ea)
Author
Owner

@William-BCLabs commented on GitHub (Apr 7, 2025):

Having this issue with random peers on 39.2 as well. Have not worked on pinpointing the root cause just that it seems to mostly affect m1+ macbooks (but not all)

<!-- gh-comment-id:2783055210 --> @William-BCLabs commented on GitHub (Apr 7, 2025): Having this issue with random peers on 39.2 as well. Have not worked on pinpointing the root cause just that it seems to mostly affect m1+ macbooks (but not all)
Author
Owner

@nazarewk commented on GitHub (Apr 7, 2025):

@William-BCLabs since a lot of time has passed since the previous reports, could you enable trace logs and share a debug bundle after you observe the issue? You can send the logs to support at netbird.io or to kdn on the Slack.

see https://docs.netbird.io/how-to/troubleshooting-client#enabling-debug-logs-on-agent

<!-- gh-comment-id:2784059945 --> @nazarewk commented on GitHub (Apr 7, 2025): @William-BCLabs since a lot of time has passed since the previous reports, could you enable trace logs and share a debug bundle after you observe the issue? You can send the logs to support at netbird.io or to `kdn` on the Slack. see https://docs.netbird.io/how-to/troubleshooting-client#enabling-debug-logs-on-agent
Author
Owner

@lepazca commented on GitHub (Jul 30, 2025):

While doing some tests, I ran into DNS resolution issues for resources associated with a network, similar to what's described in this issue.

After enabling debug mode, I noticed that DNS queries were timing out when trying to reach the IP of the routing peer. It turned out that the firewall was enabled and blocking the traffic.

Once I added rules to allow traffic on ports 53 and 5353, DNS resolution started working as expected.

Hope this helps others running into the same problem.

<!-- gh-comment-id:3134782549 --> @lepazca commented on GitHub (Jul 30, 2025): While doing some tests, I ran into DNS resolution issues for resources associated with a network, similar to what's described in this issue. After enabling debug mode, I noticed that DNS queries were timing out when trying to reach the IP of the routing peer. It turned out that the firewall was enabled and blocking the traffic. Once I added rules to allow traffic on ports 53 and 5353, DNS resolution started working as expected. Hope this helps others running into the same problem.
Author
Owner

@cmonty14 commented on GitHub (Oct 3, 2025):

Hi,
I'm facing the same issue after upgrading to netbird 0.59.2.
I'm running a mailserver on Debian 12 with unbound DNS resolver.

tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      19996/unbound       
tcp        0      0 127.0.0.1:5953          0.0.0.0:*               LISTEN      1/init              
tcp6       0      0 ::1:5953                :::*                    LISTEN      1/init              
tcp6       0      0 ::1:53                  :::*                    LISTEN      19996/unbound       
udp        0      0 127.0.0.1:53            0.0.0.0:*                           19996/unbound       
udp6       0      0 ::1:53                  :::*                                19996/unbound

Creating debug bundle fails with this error:
Error: failed to restore log level: connect client not initialized

When netbird is up, I get this /etc/resolv.conf:

# Generated by resolvconf
search netbird.cloud
nameserver ::1
nameserver 127.0.0.1
options trust-ad
options trust-ad

Please advise how to fix this issue.

<!-- gh-comment-id:3366049237 --> @cmonty14 commented on GitHub (Oct 3, 2025): Hi, I'm facing the same issue after upgrading to _netbird 0.59.2_. I'm running a mailserver on Debian 12 with [unbound DNS resolver](https://github.com/NLnetLabs/unbound). ``` tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 19996/unbound tcp 0 0 127.0.0.1:5953 0.0.0.0:* LISTEN 1/init tcp6 0 0 ::1:5953 :::* LISTEN 1/init tcp6 0 0 ::1:53 :::* LISTEN 19996/unbound udp 0 0 127.0.0.1:53 0.0.0.0:* 19996/unbound udp6 0 0 ::1:53 :::* 19996/unbound ``` Creating debug bundle fails with this error: `Error: failed to restore log level: connect client not initialized` When netbird is up, I get this _/etc/resolv.conf_: ``` # Generated by resolvconf search netbird.cloud nameserver ::1 nameserver 127.0.0.1 options trust-ad options trust-ad ``` Please advise how to fix this issue.
Author
Owner

@mlsmaycon commented on GitHub (Oct 3, 2025):

Hello Thomas,

Can you please ensure netbird up was executed?

If it still doesn't work, please share the log files below:

C:\ProgramData\NetBird\client.log

C:\ProgramData\NetBird\netbird.err

Best,
Maycon
CTO & Co-Founder

On Fri, Oct 3, 2025, 4:56 PM Thomas @.***> wrote:

cmonty14 left a comment (netbirdio/netbird#2157)
https://github.com/netbirdio/netbird/issues/2157#issuecomment-3366049237

Hi,
I'm facing the same issue after upgrading to netbird 0.59.2.
I'm running a mailserver on Debian 12 with unbound DNS resolver
https://github.com/NLnetLabs/unbound.

Creating debug bundle fails with this error:
Error: failed to restore log level: connect client not initialized

Please advise how to fix this issue.


Reply to this email directly, view it on GitHub
https://github.com/netbirdio/netbird/issues/2157#issuecomment-3366049237,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3DRIEW3QWX76IJDY3AEYT3V2FCDAVCNFSM6AAAAABV5PEDUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNRWGA2DSMRTG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

<!-- gh-comment-id:3366132403 --> @mlsmaycon commented on GitHub (Oct 3, 2025): Hello Thomas, Can you please ensure netbird up was executed? If it still doesn't work, please share the log files below: C:\ProgramData\NetBird\client.log C:\ProgramData\NetBird\netbird.err Best, Maycon CTO & Co-Founder On Fri, Oct 3, 2025, 4:56 PM Thomas ***@***.***> wrote: > *cmonty14* left a comment (netbirdio/netbird#2157) > <https://github.com/netbirdio/netbird/issues/2157#issuecomment-3366049237> > > Hi, > I'm facing the same issue after upgrading to *netbird 0.59.2*. > I'm running a mailserver on Debian 12 with unbound DNS resolver > <https://github.com/NLnetLabs/unbound>. > > Creating debug bundle fails with this error: > Error: failed to restore log level: connect client not initialized > > Please advise how to fix this issue. > > — > Reply to this email directly, view it on GitHub > <https://github.com/netbirdio/netbird/issues/2157#issuecomment-3366049237>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AB3DRIEW3QWX76IJDY3AEYT3V2FCDAVCNFSM6AAAAABV5PEDUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTGNRWGA2DSMRTG4> > . > You are receiving this because you were mentioned.Message ID: > ***@***.***> >
Author
Owner

@cmonty14 commented on GitHub (Oct 7, 2025):

Hello Thomas,

Can you please ensure netbird up was executed?

If it still doesn't work, please share the log files below:

C:\ProgramData\NetBird\client.log

C:\ProgramData\NetBird\netbird.err

Best,
Maycon
CTO & Co-Founder

I reproduced the issue on 2025-10-07T07:42, created debug bundle and attached it here.
Imo the relevant error is:

2025-10-07T07:42:14+02:00 ERRO [request_id: 48a8b80f] client/internal/dns/upstream.go:214: all queries to the Upstream [[::1]:
53 127.0.0.1:53] failed for question domain=94.107.30.81.in-addr.arpa.
2025-10-07T07:42:14+02:00 WARN [request_id: ac91bb5a] client/internal/dns/upstream.go:200: upstream 127.0.0.1:53 timed out for
 question domain=94.107.30.81.in-addr.arpa. after 2.5s (timeout=2.5s) - error: with udp: read udp 127.0.0.1:47998->127.0.0.1:53: i/o timeout

While netbird debug was running (for 30s) I executed nslookup that returned this error:

# nslookup oracle.com
;; communications error to ::1#53: timed out
;; communications error to ::1#53: timed out
;; communications error to ::1#53: timed out
;; communications error to 127.0.0.1#53: timed out
;; no servers could be reached

Note: Peer is running on Ubuntu (not Windows).

netbird.debug.1881625687.zip

<!-- gh-comment-id:3375298203 --> @cmonty14 commented on GitHub (Oct 7, 2025): > Hello Thomas, > > Can you please ensure netbird up was executed? > > If it still doesn't work, please share the log files below: > > C:\ProgramData\NetBird\client.log > > C:\ProgramData\NetBird\netbird.err > > > Best, > Maycon > CTO & Co-Founder > […](#) I reproduced the issue on _2025-10-07T07:42_, created debug bundle and attached it here. Imo the relevant error is: ``` 2025-10-07T07:42:14+02:00 ERRO [request_id: 48a8b80f] client/internal/dns/upstream.go:214: all queries to the Upstream [[::1]: 53 127.0.0.1:53] failed for question domain=94.107.30.81.in-addr.arpa. 2025-10-07T07:42:14+02:00 WARN [request_id: ac91bb5a] client/internal/dns/upstream.go:200: upstream 127.0.0.1:53 timed out for question domain=94.107.30.81.in-addr.arpa. after 2.5s (timeout=2.5s) - error: with udp: read udp 127.0.0.1:47998->127.0.0.1:53: i/o timeout ``` While netbird debug was running (for 30s) I executed `nslookup` that returned this error: ``` # nslookup oracle.com ;; communications error to ::1#53: timed out ;; communications error to ::1#53: timed out ;; communications error to ::1#53: timed out ;; communications error to 127.0.0.1#53: timed out ;; no servers could be reached ``` Note: Peer is running on Ubuntu (not Windows). [netbird.debug.1881625687.zip](https://github.com/user-attachments/files/22735614/netbird.debug.1881625687.zip)
Author
Owner

@TuXbOy06 commented on GitHub (Dec 22, 2025):

Hello everyone,

Any updates about this issue ? I am having the same problem here...

Thanks

<!-- gh-comment-id:3684424859 --> @TuXbOy06 commented on GitHub (Dec 22, 2025): Hello everyone, Any updates about this issue ? I am having the same problem here... Thanks
Author
Owner

@openapphub commented on GitHub (Jan 15, 2026):

Any updates about this issue ? I am having the same problem here...

 [WARNING] DNS (0116e0d7-922b-4802-b5ba-7d38ef1cfae5)
    Message: All upstream servers failed (probe failed)
    Time: 3 minutes, 3 seconds ago
    Metadata: upstreams: 1.1.1.1:53, 1.0.0.1:53
  [WARNING] DNS (bc9564c6-6fb5-4753-9dea-de0299dd7fd4)
    Message: All upstream servers failed (probe failed)
    Time: 3 minutes, 3 seconds ago
    Metadata: upstreams: 8.8.8.8:53, 8.8.4.4:53
  [INFO] SYSTEM (681e18cc-833e-471b-92c7-8365fdaee554)
    Message: Network map updated
    Time: 3 minutes, 3 seconds ago
  [WARNING] DNS (1873be64-da6c-400d-98f2-5c21565b3028)
    Message: All upstream servers failed (probe failed)
    Time: 21 seconds ago
    Metadata: upstreams: 1.1.1.1:53, 1.0.0.1:53
  [WARNING] DNS (f2fa2c3d-f3d7-4f91-8e45-71ca1ba8d652)
    Message: All upstream servers failed (probe failed)
    Time: 21 seconds ago
    Metadata: upstreams: 8.8.8.8:53, 8.8.4.4:53
  [INFO] SYSTEM (37b22f26-aa15-4d05-a8f3-99ebf7a7d386)
    Message: Network map updated
    Time: 21 seconds ago
Nameservers: 
  [1.1.1.1:53, 1.0.0.1:53] for [.] is Available
  [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
  [223.5.5.5:53, 223.6.6.6:53] for [.] is Available
  [127.0.0.1:53] for [.] is Available
root@ImmortalWrt:~# nslookup www.google.com 100.64.37.8
nslookup: write to '100.64.37.8': Required key not available
;; connection timed out; no servers could be reached

root@ImmortalWrt:~# nslookup www.google.com 127.0.0.1
Server:         127.0.0.1
Address:        127.0.0.1:53

Non-authoritative answer:
Name:   www.google.com
Address: 142.251.42.164

Non-authoritative answer:
<!-- gh-comment-id:3753273533 --> @openapphub commented on GitHub (Jan 15, 2026): Any updates about this issue ? I am having the same problem here... ``` [WARNING] DNS (0116e0d7-922b-4802-b5ba-7d38ef1cfae5) Message: All upstream servers failed (probe failed) Time: 3 minutes, 3 seconds ago Metadata: upstreams: 1.1.1.1:53, 1.0.0.1:53 [WARNING] DNS (bc9564c6-6fb5-4753-9dea-de0299dd7fd4) Message: All upstream servers failed (probe failed) Time: 3 minutes, 3 seconds ago Metadata: upstreams: 8.8.8.8:53, 8.8.4.4:53 [INFO] SYSTEM (681e18cc-833e-471b-92c7-8365fdaee554) Message: Network map updated Time: 3 minutes, 3 seconds ago [WARNING] DNS (1873be64-da6c-400d-98f2-5c21565b3028) Message: All upstream servers failed (probe failed) Time: 21 seconds ago Metadata: upstreams: 1.1.1.1:53, 1.0.0.1:53 [WARNING] DNS (f2fa2c3d-f3d7-4f91-8e45-71ca1ba8d652) Message: All upstream servers failed (probe failed) Time: 21 seconds ago Metadata: upstreams: 8.8.8.8:53, 8.8.4.4:53 [INFO] SYSTEM (37b22f26-aa15-4d05-a8f3-99ebf7a7d386) Message: Network map updated Time: 21 seconds ago Nameservers: [1.1.1.1:53, 1.0.0.1:53] for [.] is Available [8.8.8.8:53, 8.8.4.4:53] for [.] is Available [223.5.5.5:53, 223.6.6.6:53] for [.] is Available [127.0.0.1:53] for [.] is Available ``` ``` root@ImmortalWrt:~# nslookup www.google.com 100.64.37.8 nslookup: write to '100.64.37.8': Required key not available ;; connection timed out; no servers could be reached root@ImmortalWrt:~# nslookup www.google.com 127.0.0.1 Server: 127.0.0.1 Address: 127.0.0.1:53 Non-authoritative answer: Name: www.google.com Address: 142.251.42.164 Non-authoritative answer: ```
Author
Owner

@sandroshu commented on GitHub (Mar 25, 2026):

I also get notifications about DNS issues.
The NS servers for specific domains provided in NetBird are available over Peers. I read in the forum that Netbird tries DNS first, then connects to network then retry DNS. Is it possible to change this order to connect to network first and then try DNS first if this is the case?

forum

First is a peer directly, then the other 2 are routed by another peer.

Events:
  [INFO] SYSTEM (56670c61-a332-4ee6-8816-36ddbb05a3d7)
    Message: Network map updated
    Time: 7 minutes, 55 seconds ago
  [WARNING] DNS (e6bc1ca3-f7d6-45f9-bc3c-fbefe511ef92)
    Message: All upstream servers failed (probe failed)
    Time: 7 minutes, 55 seconds ago
    Metadata: upstreams: 100.104.146.233:53
  [WARNING] DNS (30ef77d2-0cda-4544-9d86-fe64a085ee78)
    Message: All upstream servers failed (probe failed)
    Time: 7 minutes, 55 seconds ago
    Metadata: upstreams: 192.168.10.254:53
  [WARNING] DNS (d2f5e540-8ac0-4b7f-bffe-13ef676a776b)
    Message: All upstream servers failed (probe failed)
    Time: 7 minutes, 55 seconds ago
    Metadata: upstreams: 192.168.10.254:53
<!-- gh-comment-id:4129240498 --> @sandroshu commented on GitHub (Mar 25, 2026): I also get notifications about DNS issues. The NS servers for specific domains provided in NetBird are available over Peers. I read in the forum that Netbird tries DNS first, then connects to network then retry DNS. Is it possible to change this order to connect to network first and then try DNS first if this is the case? [forum](https://forum.netbird.io/t/unable-to-reach-one-or-more-dns-servers/204/4) First is a peer directly, then the other 2 are routed by another peer. ``` Events: [INFO] SYSTEM (56670c61-a332-4ee6-8816-36ddbb05a3d7) Message: Network map updated Time: 7 minutes, 55 seconds ago [WARNING] DNS (e6bc1ca3-f7d6-45f9-bc3c-fbefe511ef92) Message: All upstream servers failed (probe failed) Time: 7 minutes, 55 seconds ago Metadata: upstreams: 100.104.146.233:53 [WARNING] DNS (30ef77d2-0cda-4544-9d86-fe64a085ee78) Message: All upstream servers failed (probe failed) Time: 7 minutes, 55 seconds ago Metadata: upstreams: 192.168.10.254:53 [WARNING] DNS (d2f5e540-8ac0-4b7f-bffe-13ef676a776b) Message: All upstream servers failed (probe failed) Time: 7 minutes, 55 seconds ago Metadata: upstreams: 192.168.10.254:53 ```
Author
Owner

@Crumar commented on GitHub (Apr 2, 2026):

I have started with a fresh install and face the same problem, but only in the docker container that is also running the management docker container (self hosted).

So I have public server running netbird-server and netbird-client in separate compose stacks.
The netbird-server is behind a separate traefik instance on the same host.

I also have a private server in my home network, running netbird-client as docker container.

local private server:

private-srv:/# netbird status -dA
Peers detail:
 netcup.netbird.selfhosted:
  NetBird IP: 100.114.93.45
  Public key: R/XdAJiLII7AHBqCR+rmcfXxD4IEhjr/GJajnR7VikQ=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/host
  ICE candidate endpoints (Local/Remote): 172.24.0.1:33128/198.51.100.0:51820
  Relay server address: rels://netbird.anon-SEzhk.domain:443
  Last connection update: 34 seconds ago
  Last WireGuard handshake: 34 seconds ago
  Transfer status (received/sent) 124 B/212 B
  Quantum resistance: false
  Networks: -
  Latency: 12.798539ms

Events:
  [INFO] SYSTEM (f7713dda-4d07-4e68-b2bc-f81d121b9188)
    Message: Network map updated
    Time: 8 hours, 37 minutes ago
  [INFO] SYSTEM (546f9f6b-2239-41ba-8c04-fd5b94979e21)
    Message: Network map updated
    Time: 2 minutes, 50 seconds ago
  [INFO] SYSTEM (cff94515-056d-48e0-9210-b3a0818a3682)
    Message: Network map updated
    Time: 2 minutes, 13 seconds ago
  [INFO] SYSTEM (278a4d3e-cf15-4c1e-9b47-1e0914e1d73f)
    Message: Network map updated
    Time: 34 seconds ago
OS: linux/amd64
Daemon version: 0.67.1
CLI version: 0.67.1
Profile: default
Management: Connected to https://netbird.anon-SEzhk.domain:443
Signal: Connected to https://netbird.anon-SEzhk.domain:443
Relays: 
  [stun:netbird.anon-SEzhk.domain:3478] is Available
  [rels://netbird.anon-SEzhk.domain:443] is Available
Nameservers: 
FQDN: private-srv.netbird.selfhosted
NetBird IP: 100.114.82.206/16
Interface type: Kernel
Quantum resistance: false
Lazy connection: false
SSH Server: Disabled
Networks: -
Peers count: 1/1 Connected
private-srv:/# nslookup google.com
Server:         100.114.82.206
Address:        100.114.82.206:53

Non-authoritative answer:
Name:   google.com
Address: 142.250.186.78

Non-authoritative answer:
Name:   google.com
Address: 2a00:1450:4001:c0f::8a
Name:   google.com
Address: 2a00:1450:4001:c0f::65
Name:   google.com
Address: 2a00:1450:4001:c0f::64
Name:   google.com
Address: 2a00:1450:4001:c0f::66

Public server, client container

public-srv:/# netbird status -dA
Peers detail:
 drdocker.netbird.selfhosted:
  NetBird IP: 100.114.82.206
  Public key: nICaz3KMcZAnptZ92HcPOAvW4Lx+CNisKC5TqxmVFiY=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:51820
  Relay server address: rels://netbird.anon-z53bP.domain:443
  Last connection update: 27 seconds ago
  Last WireGuard handshake: 27 seconds ago
  Transfer status (received/sent) 212 B/124 B
  Quantum resistance: false
  Networks: -
  Latency: 13.070808ms

Events:
  [INFO] SYSTEM (e14e6eb9-f310-4c16-84af-67b6d6fab39a)
    Message: Network map updated
    Time: 27 seconds ago
OS: linux/amd64
Daemon version: 0.67.1
CLI version: 0.67.1
Profile: default
Management: Connected to https://netbird.anon-z53bP.domain:443
Signal: Connected to https://netbird.anon-z53bP.domain:443
Relays: 
  [stun:netbird.anon-z53bP.domain:3478] is Unavailable, reason: dial: failed to listen: dial udp: lookup netbird.anon-z53bP.domain on 100.114.93.45:53: server misbehaving
  [rels://netbird.anon-z53bP.domain:443] is Available
Nameservers: 
FQDN: public-srv.netbird.selfhosted
NetBird IP: 100.114.93.45/16
Interface type: Kernel
Quantum resistance: false
Lazy connection: false
SSH Server: Disabled
Networks: -
Peers count: 1/1 Connected
public-srv:/# nslookup google.com
Server:         100.114.93.45
Address:        100.114.93.45:53

** server can't find google.com: REFUSED

** server can't find google.com: REFUSED

public-srv:/# netbird down
Disconnected
public-srv:/# nslookup google.com
Server:         9.9.9.9
Address:        9.9.9.9:53

Non-authoritative answer:
Name:   google.com
Address: 2a00:1450:4001:c1f::71
Name:   google.com
Address: 2a00:1450:4001:c1f::65
Name:   google.com
Address: 2a00:1450:4001:c1f::8a
Name:   google.com
Address: 2a00:1450:4001:c1f::66

Non-authoritative answer:
Name:   google.com
Address: 142.251.143.110

Servers are not able to ping each other.
Both containers use the same setup, its just that one is running on the server that also hosts the netbird-server container

<!-- gh-comment-id:4174674867 --> @Crumar commented on GitHub (Apr 2, 2026): I have started with a fresh install and face the same problem, but only in the docker container that is also running the management docker container (self hosted). So I have public server running netbird-server and netbird-client in separate compose stacks. The netbird-server is behind a separate traefik instance on the same host. I also have a private server in my home network, running netbird-client as docker container. # local private server: ``` private-srv:/# netbird status -dA Peers detail: netcup.netbird.selfhosted: NetBird IP: 100.114.93.45 Public key: R/XdAJiLII7AHBqCR+rmcfXxD4IEhjr/GJajnR7VikQ= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/host ICE candidate endpoints (Local/Remote): 172.24.0.1:33128/198.51.100.0:51820 Relay server address: rels://netbird.anon-SEzhk.domain:443 Last connection update: 34 seconds ago Last WireGuard handshake: 34 seconds ago Transfer status (received/sent) 124 B/212 B Quantum resistance: false Networks: - Latency: 12.798539ms Events: [INFO] SYSTEM (f7713dda-4d07-4e68-b2bc-f81d121b9188) Message: Network map updated Time: 8 hours, 37 minutes ago [INFO] SYSTEM (546f9f6b-2239-41ba-8c04-fd5b94979e21) Message: Network map updated Time: 2 minutes, 50 seconds ago [INFO] SYSTEM (cff94515-056d-48e0-9210-b3a0818a3682) Message: Network map updated Time: 2 minutes, 13 seconds ago [INFO] SYSTEM (278a4d3e-cf15-4c1e-9b47-1e0914e1d73f) Message: Network map updated Time: 34 seconds ago OS: linux/amd64 Daemon version: 0.67.1 CLI version: 0.67.1 Profile: default Management: Connected to https://netbird.anon-SEzhk.domain:443 Signal: Connected to https://netbird.anon-SEzhk.domain:443 Relays: [stun:netbird.anon-SEzhk.domain:3478] is Available [rels://netbird.anon-SEzhk.domain:443] is Available Nameservers: FQDN: private-srv.netbird.selfhosted NetBird IP: 100.114.82.206/16 Interface type: Kernel Quantum resistance: false Lazy connection: false SSH Server: Disabled Networks: - Peers count: 1/1 Connected private-srv:/# nslookup google.com Server: 100.114.82.206 Address: 100.114.82.206:53 Non-authoritative answer: Name: google.com Address: 142.250.186.78 Non-authoritative answer: Name: google.com Address: 2a00:1450:4001:c0f::8a Name: google.com Address: 2a00:1450:4001:c0f::65 Name: google.com Address: 2a00:1450:4001:c0f::64 Name: google.com Address: 2a00:1450:4001:c0f::66 ``` # Public server, client container ``` public-srv:/# netbird status -dA Peers detail: drdocker.netbird.selfhosted: NetBird IP: 100.114.82.206 Public key: nICaz3KMcZAnptZ92HcPOAvW4Lx+CNisKC5TqxmVFiY= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:51820 Relay server address: rels://netbird.anon-z53bP.domain:443 Last connection update: 27 seconds ago Last WireGuard handshake: 27 seconds ago Transfer status (received/sent) 212 B/124 B Quantum resistance: false Networks: - Latency: 13.070808ms Events: [INFO] SYSTEM (e14e6eb9-f310-4c16-84af-67b6d6fab39a) Message: Network map updated Time: 27 seconds ago OS: linux/amd64 Daemon version: 0.67.1 CLI version: 0.67.1 Profile: default Management: Connected to https://netbird.anon-z53bP.domain:443 Signal: Connected to https://netbird.anon-z53bP.domain:443 Relays: [stun:netbird.anon-z53bP.domain:3478] is Unavailable, reason: dial: failed to listen: dial udp: lookup netbird.anon-z53bP.domain on 100.114.93.45:53: server misbehaving [rels://netbird.anon-z53bP.domain:443] is Available Nameservers: FQDN: public-srv.netbird.selfhosted NetBird IP: 100.114.93.45/16 Interface type: Kernel Quantum resistance: false Lazy connection: false SSH Server: Disabled Networks: - Peers count: 1/1 Connected public-srv:/# nslookup google.com Server: 100.114.93.45 Address: 100.114.93.45:53 ** server can't find google.com: REFUSED ** server can't find google.com: REFUSED public-srv:/# netbird down Disconnected public-srv:/# nslookup google.com Server: 9.9.9.9 Address: 9.9.9.9:53 Non-authoritative answer: Name: google.com Address: 2a00:1450:4001:c1f::71 Name: google.com Address: 2a00:1450:4001:c1f::65 Name: google.com Address: 2a00:1450:4001:c1f::8a Name: google.com Address: 2a00:1450:4001:c1f::66 Non-authoritative answer: Name: google.com Address: 142.251.143.110 ``` Servers are not able to ping each other. Both containers use the same setup, its just that one is running on the server that also hosts the netbird-server container
Sign in to join this conversation.
No Label bug client dns
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#4431