DNS settings break DNS resolution in k3s/rke2 clusters. #1957

Open
opened 2025-11-20 06:10:14 -05:00 by saavagebueno · 9 comments
Owner

Originally created by @theoriginalgri on GitHub (Jun 11, 2025).

Describe the problem

We are running netbird on Ubuntu 24.04 host systems (hetzner cloud + dedicated) which also run k3s / rke2.

Up until this Tuesday, this was working flawlessly (even though we have not installed any updates of netbird/ubuntu/k3s/rke2 within the days of breaking).

Since then, pods have been behaving erratic and failed installing alpine linux packages. We have now pinned the issue down to having DNS settings active in netbird. Once the machine is excluded using "Disable DNS management", alpine linux packages can be installed.

To Reproduce

Steps to reproduce the behavior:

  1. Setup a new server (e.g. on Hetzner Cloud)
  2. Install netbird:
curl -fsSL https://pkgs.netbird.io/install.sh | sh
netbird up
  1. Install k3s:
curl -sfL https://get.k3s.io | sh -
  1. Launch a bash session for alpine linux:
k3s kubectl run test-shell --rm -i --tty --image alpine:latest

Within that shell try to ping dl-cdn.alpinelinux.org:

ping dl-cdn.alpinelinux.org
ping: bad address 'dl-cdn.alpinelinux.org'

If either netbird service is stopped or the machine is excluded from DNS settings, pinging starts working again once the pod is restarted. Interestingly pinging ping dl-cdn.alpinelinux.org. (with dot suffix) works.

Expected behavior

Pinging to dl-cdn.alpinelinux.org succeeds.

Are you using NetBird Cloud?

Cloud

NetBird version

Tried multiple:

  • 0.46.0
  • 0.45.3
  • 0.44.0

Is any other VPN software installed?

No

Debug output

To help us resolve the problem, please attach the following anonymized status output

Peers detail:
 gitlab.netbird.cloud:
  NetBird IP: 100.92.104.216
  Public key: 5JcftOw3hmB/rW7xULTpX49hbEGi/SSYP6bWNxMiJ0k=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 10.42.0.0:51820/198.51.100.0:51820
  Relay server address: rels://streamline-de-fra1-0.relay.netbird.io:443
  Last connection update: 19 minutes, 58 seconds ago
  Last WireGuard handshake: 57 seconds ago
  Transfer status (received/sent) 1.5 KiB/4.4 KiB
  Quantum resistance: false
  Networks: -
  Latency: 3.360698ms

Events:
  [WARNING] DNS (eb819e1f-0e71-4bfa-aecf-6c98b0980181)
    Message: All upstream servers failed (probe failed)
    Time: 20 minutes, 2 seconds ago
    Metadata: upstreams: 100.92.104.216:5353
  [WARNING] DNS (ce264190-443d-4984-bea2-96d5d5f05748)
    Message: All upstream servers failed (probe failed)
    Time: 20 minutes, 2 seconds ago
    Metadata: upstreams: 100.92.104.216:5353
  [INFO] SYSTEM (0fed29a3-2ca0-4044-86c4-7d0ac9b2d606)
    Message: Network map updated
    Time: 20 minutes, 2 seconds ago
OS: linux/amd64
Daemon version: 0.44.0
CLI version: 0.44.0
Management: Connected to https://api.netbird.io:443
Signal: Connected to https://signal.netbird.io:443
Relays:
  [stun:stun.netbird.io:5555] is Available
  [turns:turn.netbird.io:443?transport=tcp] is Available
  [rels://streamline-de-fra1-0.relay.netbird.io:443] is Available
Nameservers:
  [100.92.104.216:5353] for [gitlab.anon-ZDWPs.domain, gitlab-ssh.anon-ZDWPs.domain] is Available
FQDN: ubuntu-16gb-fsn1-1.netbird.cloud
NetBird IP: 100.92.239.76/16
Interface type: Kernel
Quantum resistance: false
Networks: -
Forwarding rules: 0
Peers count: 1/1 Connected

Create and upload a debug bundle, and share the returned file key:

f79e391890ab27fb37c88b3b4be7011e22aa2e5ca6f38ffa9c4481884941f726/34c03ba3-60c9-44dc-8aec-f6f42fd25f8e

Screenshots

Additional context

Add any other context about the problem here:

Running tcpdump port 53 on the host machine reveals:

12:57:02.174698 IP static.179.xx.xxx.xxx.clients.your-server.de.datametrics > ns2.recursivedns.hetzner.com.domain: 57579+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54)
12:57:02.174733 IP static.179.xx.xxx.xxx.clients.your-server.de.12678 > ns2.recursivedns.hetzner.com.domain: 1475+ A? dl-cdn.alpinelinux.org.netbird.cloud. (54)
12:57:02.244589 IP static.179.xx.xxx.xxx.clients.your-server.de.59458 > ns1.recursivedns.hetzner.com.domain: 20223+ [1au] PTR? 2.64.12.185.in-addr.arpa. (53)
12:57:02.245085 IP ns1.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.59458: 20223 1/3/1 PTR ns2.recursivedns.hetzner.com. (182)
12:57:02.335256 IP ns2.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.12678: 1475 ServFail 0/0/0 (54)
12:57:02.335923 IP ns2.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.datametrics: 57579 ServFail 0/0/0 (54)

It looks like it's trying to lookup dl-cdn.alpinelinux.org.netbird.cloud., so it seems like it's adding the search domain to every lookup 🤷

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • [ ] Disabled other VPN software
  • Checked firewall settings
Originally created by @theoriginalgri on GitHub (Jun 11, 2025). **Describe the problem** We are running netbird on Ubuntu 24.04 host systems (hetzner cloud + dedicated) which also run k3s / rke2. Up until this Tuesday, this was working flawlessly (even though we have not installed any updates of netbird/ubuntu/k3s/rke2 within the days of breaking). Since then, pods have been behaving erratic and failed installing alpine linux packages. We have now pinned the issue down to having DNS settings active in netbird. Once the machine is excluded using "Disable DNS management", alpine linux packages can be installed. **To Reproduce** Steps to reproduce the behavior: 1. Setup a new server (e.g. on Hetzner Cloud) 2. Install netbird: ```shell curl -fsSL https://pkgs.netbird.io/install.sh | sh netbird up ``` 3. Install k3s: ```shell curl -sfL https://get.k3s.io | sh - ``` 4. Launch a bash session for alpine linux: ```shell k3s kubectl run test-shell --rm -i --tty --image alpine:latest ``` Within that shell try to ping `dl-cdn.alpinelinux.org`: ```shell ping dl-cdn.alpinelinux.org ping: bad address 'dl-cdn.alpinelinux.org' ``` If either netbird service is stopped or the machine is excluded from DNS settings, pinging starts working again once the pod is restarted. Interestingly pinging `ping dl-cdn.alpinelinux.org.` (with dot suffix) works. **Expected behavior** Pinging to `dl-cdn.alpinelinux.org` succeeds. **Are you using NetBird Cloud?** Cloud **NetBird version** Tried multiple: - 0.46.0 - 0.45.3 - 0.44.0 **Is any other VPN software installed?** No **Debug output** To help us resolve the problem, please attach the following anonymized status output ``` Peers detail: gitlab.netbird.cloud: NetBird IP: 100.92.104.216 Public key: 5JcftOw3hmB/rW7xULTpX49hbEGi/SSYP6bWNxMiJ0k= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/host ICE candidate endpoints (Local/Remote): 10.42.0.0:51820/198.51.100.0:51820 Relay server address: rels://streamline-de-fra1-0.relay.netbird.io:443 Last connection update: 19 minutes, 58 seconds ago Last WireGuard handshake: 57 seconds ago Transfer status (received/sent) 1.5 KiB/4.4 KiB Quantum resistance: false Networks: - Latency: 3.360698ms Events: [WARNING] DNS (eb819e1f-0e71-4bfa-aecf-6c98b0980181) Message: All upstream servers failed (probe failed) Time: 20 minutes, 2 seconds ago Metadata: upstreams: 100.92.104.216:5353 [WARNING] DNS (ce264190-443d-4984-bea2-96d5d5f05748) Message: All upstream servers failed (probe failed) Time: 20 minutes, 2 seconds ago Metadata: upstreams: 100.92.104.216:5353 [INFO] SYSTEM (0fed29a3-2ca0-4044-86c4-7d0ac9b2d606) Message: Network map updated Time: 20 minutes, 2 seconds ago OS: linux/amd64 Daemon version: 0.44.0 CLI version: 0.44.0 Management: Connected to https://api.netbird.io:443 Signal: Connected to https://signal.netbird.io:443 Relays: [stun:stun.netbird.io:5555] is Available [turns:turn.netbird.io:443?transport=tcp] is Available [rels://streamline-de-fra1-0.relay.netbird.io:443] is Available Nameservers: [100.92.104.216:5353] for [gitlab.anon-ZDWPs.domain, gitlab-ssh.anon-ZDWPs.domain] is Available FQDN: ubuntu-16gb-fsn1-1.netbird.cloud NetBird IP: 100.92.239.76/16 Interface type: Kernel Quantum resistance: false Networks: - Forwarding rules: 0 Peers count: 1/1 Connected ``` Create and upload a debug bundle, and share the returned file key: `f79e391890ab27fb37c88b3b4be7011e22aa2e5ca6f38ffa9c4481884941f726/34c03ba3-60c9-44dc-8aec-f6f42fd25f8e` **Screenshots** - **Additional context** Add any other context about the problem here: Running `tcpdump port 53` on the host machine reveals: ``` 12:57:02.174698 IP static.179.xx.xxx.xxx.clients.your-server.de.datametrics > ns2.recursivedns.hetzner.com.domain: 57579+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54) 12:57:02.174733 IP static.179.xx.xxx.xxx.clients.your-server.de.12678 > ns2.recursivedns.hetzner.com.domain: 1475+ A? dl-cdn.alpinelinux.org.netbird.cloud. (54) 12:57:02.244589 IP static.179.xx.xxx.xxx.clients.your-server.de.59458 > ns1.recursivedns.hetzner.com.domain: 20223+ [1au] PTR? 2.64.12.185.in-addr.arpa. (53) 12:57:02.245085 IP ns1.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.59458: 20223 1/3/1 PTR ns2.recursivedns.hetzner.com. (182) 12:57:02.335256 IP ns2.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.12678: 1475 ServFail 0/0/0 (54) 12:57:02.335923 IP ns2.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.datametrics: 57579 ServFail 0/0/0 (54) ``` It looks like it's trying to lookup `dl-cdn.alpinelinux.org.netbird.cloud.`, so it seems like it's adding the search domain to every lookup 🤷 **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - ~~[ ] Disabled other VPN software~~ - [x] Checked firewall settings
saavagebueno added the triage-needed label 2025-11-20 06:10:14 -05:00
Author
Owner

@lixmal commented on GitHub (Jun 11, 2025):

Can you share resolvectl status and cat /etc/resolv.conf? It's unclear why the machine appends search domains to unrelated queries (not directed to NetBird's resolver).

Besides, there seems to be something wrong with the upstream server:

A/AAAA queries:

12:57:02.174698 IP static.179.xx.xxx.xxx.clients.your-server.de.datametrics > ns2.recursivedns.hetzner.com.domain: 57579+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54)
12:57:02.174733 IP static.179.xx.xxx.xxx.clients.your-server.de.12678 > ns2.recursivedns.hetzner.com.domain: 1475+ A? dl-cdn.alpinelinux.org.netbird.cloud. (54)

responses:

12:57:02.335256 IP ns2.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.12678: 1475 ServFail 0/0/0 (54)
12:57:02.335923 IP ns2.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.datametrics: 57579 ServFail 0/0/0 (54)

Upstream should respond with either of these:

NXDOMAIN - if the fully qualified domain name doesn't exist
NOERROR with no records - if the domain exists but has no records for the requested type

Since we don't see any other traffic, I assume the alpine/musl resolver stops searching after encountering ServFail resulting in your issue.

One suggestion to address the issue is to assign an upstream resolver that covers all domains in the NetBird dashboard. One that doesn't point to these hetzner resolvers.

@lixmal commented on GitHub (Jun 11, 2025): Can you share `resolvectl status` and `cat /etc/resolv.conf`? It's unclear why the machine appends search domains to unrelated queries (not directed to NetBird's resolver). Besides, there seems to be something wrong with the upstream server: A/AAAA queries: ``` 12:57:02.174698 IP static.179.xx.xxx.xxx.clients.your-server.de.datametrics > ns2.recursivedns.hetzner.com.domain: 57579+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54) 12:57:02.174733 IP static.179.xx.xxx.xxx.clients.your-server.de.12678 > ns2.recursivedns.hetzner.com.domain: 1475+ A? dl-cdn.alpinelinux.org.netbird.cloud. (54) ``` responses: ``` 12:57:02.335256 IP ns2.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.12678: 1475 ServFail 0/0/0 (54) 12:57:02.335923 IP ns2.recursivedns.hetzner.com.domain > static.179.xx.xxx.xxx.clients.your-server.de.datametrics: 57579 ServFail 0/0/0 (54) ``` Upstream should respond with either of these: ``` NXDOMAIN - if the fully qualified domain name doesn't exist NOERROR with no records - if the domain exists but has no records for the requested type ``` Since we don't see any other traffic, I assume the alpine/musl resolver stops searching after encountering ServFail resulting in your issue. One suggestion to address the issue is to assign an upstream resolver that covers all domains in the NetBird dashboard. One that doesn't point to these hetzner resolvers.
Author
Owner

@theoriginalgri commented on GitHub (Jun 11, 2025):

Thanks for your response. I just created a fresh machine to answer your questions.

Activating Google DNS for all machines as a resolver for all domains on the netbird dashboard sadly did not change anything.

# resolvectl status (host machine)

Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 185.12.64.2
       DNS Servers: 185.12.64.2 185.12.64.1

Link 3 (wt0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 100.92.171.22
       DNS Servers: 100.92.171.22
        DNS Domain: ~gitlab.mycompany.com ~gitlab-ssh.mycompany.com netbird.cloud
                    ~92.100.in-addr.arpa ~.

Link 4 (flannel.1)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (cni0)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 8 (veth980ed412)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 9 (veth133bc637)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 10 (veth59030e25)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 11 (veth6c72b1a5)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 12 (veth2b2236f6)
    Current Scopes: none
         Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
# cat /etc/resolv.conf

# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0 trust-ad
search netbird.cloud

After overriding the netplan nameservers with the ones from google, resolvectl status shows:

root@ubuntu-2gb-fsn1-1:~# resolvectl status
Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 8.8.8.8
       DNS Servers: 8.8.8.8 8.8.4.4

Link 3 (wt0)
    Current Scopes: DNS
         Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 100.92.171.22
       DNS Servers: 100.92.171.22
        DNS Domain: ~gitlab.mycompany.com ~gitlab-ssh.mycompany.com netbird.cloud ~92.100.in-addr.arpa ~.

Now my host machine has 8.8.8.8 as nameserver set and netbird dns resolution has google dns active as well. tcpdump port 53 shows:

18:35:09.156693 IP static.179.58.119.168.clients.your-server.de.55711 > dns.google.domain: 6829+ A? dl-cdn.alpinelinux.org.netbird.cloud. (54)
18:35:09.156736 IP static.179.58.119.168.clients.your-server.de.32085 > dns.google.domain: 60076+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54)
18:35:09.175927 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.32085: 60076 ServFail 0/0/0 (54)
18:35:09.179519 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.55711: 6829 ServFail 0/0/0 (54)
18:35:09.197524 IP static.179.58.119.168.clients.your-server.de.49718 > dns.google.domain: 39735+ [1au] PTR? 4.4.8.8.in-addr.arpa. (49)
18:35:09.203232 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.49718: 39735 1/0/1 PTR dns.google. (73)
@theoriginalgri commented on GitHub (Jun 11, 2025): Thanks for your response. I just created a fresh machine to answer your questions. Activating Google DNS for all machines as a resolver for all domains on the netbird dashboard sadly did not change anything. ```shell # resolvectl status (host machine) Global Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported resolv.conf mode: stub Link 2 (eth0) Current Scopes: DNS Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Current DNS Server: 185.12.64.2 DNS Servers: 185.12.64.2 185.12.64.1 Link 3 (wt0) Current Scopes: DNS Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Current DNS Server: 100.92.171.22 DNS Servers: 100.92.171.22 DNS Domain: ~gitlab.mycompany.com ~gitlab-ssh.mycompany.com netbird.cloud ~92.100.in-addr.arpa ~. Link 4 (flannel.1) Current Scopes: none Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Link 5 (cni0) Current Scopes: none Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Link 8 (veth980ed412) Current Scopes: none Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Link 9 (veth133bc637) Current Scopes: none Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Link 10 (veth59030e25) Current Scopes: none Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Link 11 (veth6c72b1a5) Current Scopes: none Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Link 12 (veth2b2236f6) Current Scopes: none Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported ``` ```shell # cat /etc/resolv.conf # This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8). # Do not edit. # # This file might be symlinked as /etc/resolv.conf. If you're looking at # /etc/resolv.conf and seeing this text, you have followed the symlink. # # This is a dynamic resolv.conf file for connecting local clients to the # internal DNS stub resolver of systemd-resolved. This file lists all # configured search domains. # # Run "resolvectl status" to see details about the uplink DNS servers # currently in use. # # Third party programs should typically not access this file directly, but only # through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a # different way, replace this symlink by a static file or a different symlink. # # See man:systemd-resolved.service(8) for details about the supported modes of # operation for /etc/resolv.conf. nameserver 127.0.0.53 options edns0 trust-ad search netbird.cloud ``` After overriding the netplan nameservers with the ones from google, `resolvectl status` shows: ``` root@ubuntu-2gb-fsn1-1:~# resolvectl status Global Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported resolv.conf mode: stub Link 2 (eth0) Current Scopes: DNS Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Current DNS Server: 8.8.8.8 DNS Servers: 8.8.8.8 8.8.4.4 Link 3 (wt0) Current Scopes: DNS Protocols: +DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Current DNS Server: 100.92.171.22 DNS Servers: 100.92.171.22 DNS Domain: ~gitlab.mycompany.com ~gitlab-ssh.mycompany.com netbird.cloud ~92.100.in-addr.arpa ~. ``` Now my host machine has 8.8.8.8 as nameserver set and netbird dns resolution has google dns active as well. `tcpdump port 53` shows: ``` 18:35:09.156693 IP static.179.58.119.168.clients.your-server.de.55711 > dns.google.domain: 6829+ A? dl-cdn.alpinelinux.org.netbird.cloud. (54) 18:35:09.156736 IP static.179.58.119.168.clients.your-server.de.32085 > dns.google.domain: 60076+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54) 18:35:09.175927 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.32085: 60076 ServFail 0/0/0 (54) 18:35:09.179519 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.55711: 6829 ServFail 0/0/0 (54) 18:35:09.197524 IP static.179.58.119.168.clients.your-server.de.49718 > dns.google.domain: 39735+ [1au] PTR? 4.4.8.8.in-addr.arpa. (49) 18:35:09.203232 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.49718: 39735 1/0/1 PTR dns.google. (73) ```
Author
Owner

@lixmal commented on GitHub (Jun 11, 2025):

Ah, now it's clear.
Yes, systemd appends this to /etc/resolv.conf and musl doesn't like ServFail responses.

netbird.cloud stops doing that now, can you check if it works for you?

@lixmal commented on GitHub (Jun 11, 2025): Ah, now it's clear. Yes, systemd appends this to /etc/resolv.conf and musl doesn't like `ServFail` responses. `netbird.cloud` stops doing that now, can you check if it works for you?
Author
Owner

@theoriginalgri commented on GitHub (Jun 11, 2025):

Yeah, it seems to be working now as long as the Google DNS Server is active in the netbird dashboard 👍 . Once deactivated, it still fails.

Output with Google DNS Server activated:

19:37:23.665493 IP static.179.58.119.168.clients.your-server.de.55102 > dns.google.domain: 62454+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54)
19:37:23.671514 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.55102: 62454 NXDomain 0/1/0 (141)
19:37:23.671822 IP static.179.58.119.168.clients.your-server.de.55102 > dns.google.domain: 12043+ A? dl-cdn.alpinelinux.org. (40)
19:37:23.679492 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.55102: 12043 5/0/0 CNAME dualstack.j.sni.global.fastly.net., A 151.101.130.132, A 151.101.2.132, A 151.101.66.132, A 151.101.194.132 (151)
@theoriginalgri commented on GitHub (Jun 11, 2025): Yeah, it seems to be working now as long as the Google DNS Server is active in the netbird dashboard 👍 . Once deactivated, it still fails. Output with Google DNS Server activated: ``` 19:37:23.665493 IP static.179.58.119.168.clients.your-server.de.55102 > dns.google.domain: 62454+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54) 19:37:23.671514 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.55102: 62454 NXDomain 0/1/0 (141) 19:37:23.671822 IP static.179.58.119.168.clients.your-server.de.55102 > dns.google.domain: 12043+ A? dl-cdn.alpinelinux.org. (40) 19:37:23.679492 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.55102: 12043 5/0/0 CNAME dualstack.j.sni.global.fastly.net., A 151.101.130.132, A 151.101.2.132, A 151.101.66.132, A 151.101.194.132 (151) ```
Author
Owner

@lixmal commented on GitHub (Jun 12, 2025):

Does it still fail with the ISP nameservers today?
If yes, can you run these on the host (with google dns or netbird off):

dig dl-cdn.alpinelinux.org.netbird.cloud.
dig netbird.cloud.
@lixmal commented on GitHub (Jun 12, 2025): Does it still fail with the ISP nameservers today? If yes, can you run these on the host (with google dns or netbird off): ``` dig dl-cdn.alpinelinux.org.netbird.cloud. dig netbird.cloud. ```
Author
Owner

@theoriginalgri commented on GitHub (Jun 13, 2025):

Yes, it still fails if I use the host system's upstream dns and don't enable Google DNS for all lookups in the netbird dashboard.

With default settings (hetzner dns) and netbird off on the host system:

root@ubuntu-2gb-fsn1-1:~# dig dl-cdn.alpinelinux.org.netbird.cloud.

; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> dl-cdn.alpinelinux.org.netbird.cloud.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 28900
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;dl-cdn.alpinelinux.org.netbird.cloud. IN A

;; AUTHORITY SECTION:
netbird.cloud.		747	IN	SOA	ns-1769.awsdns-29.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 1 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Fri Jun 13 12:56:33 UTC 2025
;; MSG SIZE  rcvd: 152
root@ubuntu-2gb-fsn1-1:~# dig netbird.cloud.

; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> netbird.cloud.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45695
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;netbird.cloud.			IN	A

;; AUTHORITY SECTION:
netbird.cloud.		890	IN	SOA	ns-1769.awsdns-29.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400

;; Query time: 23 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Fri Jun 13 12:57:23 UTC 2025
;; MSG SIZE  rcvd: 129
@theoriginalgri commented on GitHub (Jun 13, 2025): Yes, it still fails if I use the host system's upstream dns and don't enable Google DNS for all lookups in the netbird dashboard. With default settings (hetzner dns) and netbird off on the host system: ``` root@ubuntu-2gb-fsn1-1:~# dig dl-cdn.alpinelinux.org.netbird.cloud. ; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> dl-cdn.alpinelinux.org.netbird.cloud. ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 28900 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;dl-cdn.alpinelinux.org.netbird.cloud. IN A ;; AUTHORITY SECTION: netbird.cloud. 747 IN SOA ns-1769.awsdns-29.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 1 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Fri Jun 13 12:56:33 UTC 2025 ;; MSG SIZE rcvd: 152 ``` ``` root@ubuntu-2gb-fsn1-1:~# dig netbird.cloud. ; <<>> DiG 9.18.30-0ubuntu0.24.04.2-Ubuntu <<>> netbird.cloud. ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45695 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 65494 ;; QUESTION SECTION: ;netbird.cloud. IN A ;; AUTHORITY SECTION: netbird.cloud. 890 IN SOA ns-1769.awsdns-29.co.uk. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 ;; Query time: 23 msec ;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP) ;; WHEN: Fri Jun 13 12:57:23 UTC 2025 ;; MSG SIZE rcvd: 129 ```
Author
Owner

@lixmal commented on GitHub (Jun 14, 2025):

Can you repeat the earlier tcpdumps, once with netbird upstream and one without + ping from the pod

@lixmal commented on GitHub (Jun 14, 2025): Can you repeat the earlier `tcpdump`s, once with netbird upstream and one without + ping from the pod
Author
Owner

@theoriginalgri commented on GitHub (Jun 15, 2025):

tcpdump results for pinging from pod:

netbird with google dns resolver for all domains enabled (pinging works):

17:38:04.086413 IP static.179.58.119.168.clients.your-server.de.30380 > ns1.recursivedns.hetzner.com.domain: 32495+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54)
17:38:04.116832 IP static.179.58.119.168.clients.your-server.de.42272 > dns.google.domain: 40084+ PTR? 1.64.12.185.in-addr.arpa. (42)
17:38:04.119494 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.30380: 32495 NXDomain 0/1/0 (141)
17:38:04.119935 IP static.179.58.119.168.clients.your-server.de.30380 > ns1.recursivedns.hetzner.com.domain: 46295+ AAAA? dl-cdn.alpinelinux.org. (40)
17:38:04.119999 IP static.179.58.119.168.clients.your-server.de.15659 > ns2.recursivedns.hetzner.com.domain: 31142+ A? dl-cdn.alpinelinux.org. (40)
17:38:04.120212 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.30380: 46295 2/4/0 CNAME dualstack.j.sni.global.fastly.net., AAAA 2a04:4e42:8e::644 (187)
17:38:04.120268 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.15659: 31142 2/4/0 CNAME dualstack.j.sni.global.fastly.net., A 146.75.122.132 (175)
17:38:04.122959 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.42272: 40084 1/0/0 PTR ns1.recursivedns.hetzner.com. (84)
17:38:08.293600 IP static.179.58.119.168.clients.your-server.de.42806 > dns.google.domain: 12655+ A? stun.netbird.io. (33)
17:38:08.293688 IP static.179.58.119.168.clients.your-server.de.34694 > dns.google.domain: 18939+ A? turn.netbird.io. (33)
17:38:08.293702 IP static.179.58.119.168.clients.your-server.de.49789 > dns.google.domain: 36504+ AAAA? stun.netbird.io. (33)
17:38:08.293809 IP static.179.58.119.168.clients.your-server.de.36763 > dns.google.domain: 10195+ AAAA? turn.netbird.io. (33)
17:38:08.299074 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.36763: 10195 0/1/0 (120)
17:38:08.299354 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.49789: 36504 0/1/0 (120)
17:38:08.301379 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.42806: 12655 1/0/0 A 94.237.91.111 (49)
17:38:08.301594 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.34694: 18939 1/0/0 A 94.237.85.254 (49)

netbird with default dns (pinging fails):

17:41:30.095938 IP static.179.58.119.168.clients.your-server.de.40414 > ns1.recursivedns.hetzner.com.domain: 23802+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54)
17:41:30.124519 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.40414: 23802 NXDomain 0/1/0 (141)
17:41:30.125499 IP static.179.58.119.168.clients.your-server.de.9752 > ns2.recursivedns.hetzner.com.domain: 40582+ A? dl-cdn.alpinelinux.org. (40)
17:41:30.125900 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.9752: 40582 2/4/0 CNAME dualstack.j.sni.global.fastly.net., A 146.75.122.132 (175)

netbird service stopped (pinging works):

17:45:07.857640 IP static.179.58.119.168.clients.your-server.de.15366 > ns2.recursivedns.hetzner.com.domain: 25366+ A? dl-cdn.alpinelinux.org. (40)
17:45:07.857686 IP static.179.58.119.168.clients.your-server.de.46908 > ns2.recursivedns.hetzner.com.domain: 12765+ AAAA? dl-cdn.alpinelinux.org. (40)
17:45:07.858261 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.15366: 25366 2/4/0 CNAME dualstack.j.sni.global.fastly.net., A 146.75.122.132 (175)
17:45:07.858307 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.46908: 12765 2/4/0 CNAME dualstack.j.sni.global.fastly.net., AAAA 2a04:4e42:8e::644 (187)
@theoriginalgri commented on GitHub (Jun 15, 2025): tcpdump results for pinging from pod: netbird with google dns resolver for all domains enabled (pinging works): ``` 17:38:04.086413 IP static.179.58.119.168.clients.your-server.de.30380 > ns1.recursivedns.hetzner.com.domain: 32495+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54) 17:38:04.116832 IP static.179.58.119.168.clients.your-server.de.42272 > dns.google.domain: 40084+ PTR? 1.64.12.185.in-addr.arpa. (42) 17:38:04.119494 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.30380: 32495 NXDomain 0/1/0 (141) 17:38:04.119935 IP static.179.58.119.168.clients.your-server.de.30380 > ns1.recursivedns.hetzner.com.domain: 46295+ AAAA? dl-cdn.alpinelinux.org. (40) 17:38:04.119999 IP static.179.58.119.168.clients.your-server.de.15659 > ns2.recursivedns.hetzner.com.domain: 31142+ A? dl-cdn.alpinelinux.org. (40) 17:38:04.120212 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.30380: 46295 2/4/0 CNAME dualstack.j.sni.global.fastly.net., AAAA 2a04:4e42:8e::644 (187) 17:38:04.120268 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.15659: 31142 2/4/0 CNAME dualstack.j.sni.global.fastly.net., A 146.75.122.132 (175) 17:38:04.122959 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.42272: 40084 1/0/0 PTR ns1.recursivedns.hetzner.com. (84) 17:38:08.293600 IP static.179.58.119.168.clients.your-server.de.42806 > dns.google.domain: 12655+ A? stun.netbird.io. (33) 17:38:08.293688 IP static.179.58.119.168.clients.your-server.de.34694 > dns.google.domain: 18939+ A? turn.netbird.io. (33) 17:38:08.293702 IP static.179.58.119.168.clients.your-server.de.49789 > dns.google.domain: 36504+ AAAA? stun.netbird.io. (33) 17:38:08.293809 IP static.179.58.119.168.clients.your-server.de.36763 > dns.google.domain: 10195+ AAAA? turn.netbird.io. (33) 17:38:08.299074 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.36763: 10195 0/1/0 (120) 17:38:08.299354 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.49789: 36504 0/1/0 (120) 17:38:08.301379 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.42806: 12655 1/0/0 A 94.237.91.111 (49) 17:38:08.301594 IP dns.google.domain > static.179.58.119.168.clients.your-server.de.34694: 18939 1/0/0 A 94.237.85.254 (49) ``` netbird with default dns (pinging fails): ``` 17:41:30.095938 IP static.179.58.119.168.clients.your-server.de.40414 > ns1.recursivedns.hetzner.com.domain: 23802+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54) 17:41:30.124519 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.40414: 23802 NXDomain 0/1/0 (141) 17:41:30.125499 IP static.179.58.119.168.clients.your-server.de.9752 > ns2.recursivedns.hetzner.com.domain: 40582+ A? dl-cdn.alpinelinux.org. (40) 17:41:30.125900 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.9752: 40582 2/4/0 CNAME dualstack.j.sni.global.fastly.net., A 146.75.122.132 (175) ``` netbird service stopped (pinging works): ``` 17:45:07.857640 IP static.179.58.119.168.clients.your-server.de.15366 > ns2.recursivedns.hetzner.com.domain: 25366+ A? dl-cdn.alpinelinux.org. (40) 17:45:07.857686 IP static.179.58.119.168.clients.your-server.de.46908 > ns2.recursivedns.hetzner.com.domain: 12765+ AAAA? dl-cdn.alpinelinux.org. (40) 17:45:07.858261 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.15366: 25366 2/4/0 CNAME dualstack.j.sni.global.fastly.net., A 146.75.122.132 (175) 17:45:07.858307 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.46908: 12765 2/4/0 CNAME dualstack.j.sni.global.fastly.net., AAAA 2a04:4e42:8e::644 (187) ```
Author
Owner

@theoriginalgri commented on GitHub (Jun 17, 2025):

Looks like you've fixed it with 0.47.2 🎉

tcpdump with netbird running but google dns not active:

17:22:35.863094 IP static.179.58.119.168.clients.your-server.de.28172 > ns2.recursivedns.hetzner.com.domain: 38453+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54)
17:22:35.863114 IP static.179.58.119.168.clients.your-server.de.7296 > ns1.recursivedns.hetzner.com.domain: 22921+ A? dl-cdn.alpinelinux.org.netbird.cloud. (54)
17:22:35.863644 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.7296: 22921 NXDomain 0/1/0 (141)
17:22:35.892848 IP static.179.58.119.168.clients.your-server.de.57979 > ns2.recursivedns.hetzner.com.domain: 56528+ [1au] PTR? 1.64.12.185.in-addr.arpa. (53)
17:22:35.893275 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.57979: 56528 1/3/1 PTR ns1.recursivedns.hetzner.com. (182)
17:22:35.906693 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.28172: 38453 NXDomain 0/1/0 (141)
17:22:35.907191 IP static.179.58.119.168.clients.your-server.de.7296 > ns1.recursivedns.hetzner.com.domain: 56645+ AAAA? dl-cdn.alpinelinux.org. (40)
17:22:35.907330 IP static.179.58.119.168.clients.your-server.de.56307 > ns1.recursivedns.hetzner.com.domain: 31523+ A? dl-cdn.alpinelinux.org. (40)
17:22:35.907406 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.7296: 56645 2/4/0 CNAME dualstack.j.sni.global.fastly.net., AAAA 2a04:4e42:8e::644 (187)
17:22:35.907567 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.56307: 31523 2/4/0 CNAME dualstack.j.sni.global.fastly.net., A 146.75.122.132 (175)
@theoriginalgri commented on GitHub (Jun 17, 2025): Looks like you've fixed it with 0.47.2 🎉 tcpdump with netbird running but google dns not active: ``` 17:22:35.863094 IP static.179.58.119.168.clients.your-server.de.28172 > ns2.recursivedns.hetzner.com.domain: 38453+ AAAA? dl-cdn.alpinelinux.org.netbird.cloud. (54) 17:22:35.863114 IP static.179.58.119.168.clients.your-server.de.7296 > ns1.recursivedns.hetzner.com.domain: 22921+ A? dl-cdn.alpinelinux.org.netbird.cloud. (54) 17:22:35.863644 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.7296: 22921 NXDomain 0/1/0 (141) 17:22:35.892848 IP static.179.58.119.168.clients.your-server.de.57979 > ns2.recursivedns.hetzner.com.domain: 56528+ [1au] PTR? 1.64.12.185.in-addr.arpa. (53) 17:22:35.893275 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.57979: 56528 1/3/1 PTR ns1.recursivedns.hetzner.com. (182) 17:22:35.906693 IP ns2.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.28172: 38453 NXDomain 0/1/0 (141) 17:22:35.907191 IP static.179.58.119.168.clients.your-server.de.7296 > ns1.recursivedns.hetzner.com.domain: 56645+ AAAA? dl-cdn.alpinelinux.org. (40) 17:22:35.907330 IP static.179.58.119.168.clients.your-server.de.56307 > ns1.recursivedns.hetzner.com.domain: 31523+ A? dl-cdn.alpinelinux.org. (40) 17:22:35.907406 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.7296: 56645 2/4/0 CNAME dualstack.j.sni.global.fastly.net., AAAA 2a04:4e42:8e::644 (187) 17:22:35.907567 IP ns1.recursivedns.hetzner.com.domain > static.179.58.119.168.clients.your-server.de.56307: 31523 2/4/0 CNAME dualstack.j.sni.global.fastly.net., A 146.75.122.132 (175) ```
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1957