Docker container or K8s pod do not send request to host, but correct resolve host to ip. #932

Open
opened 2025-11-20 05:20:08 -05:00 by saavagebueno · 12 comments
Owner

Originally created by @gawsoftpl on GitHub (May 27, 2024).

Describe the problem

Netbird in docker container or in kubernetes pod for self hosted netbird.
Dns resolve correct ip (nslookup works correct) but curl, ping, wget return error: bad address
For normal ubuntu instance without docker or kubernetes evereythings works correct

How I run docker

docker run --rm -d 
 --cap-add=NET_ADMIN  
 -e NB_SETUP_KEY=<REDACTED>
 -v netbird-client:/etc/netbird  
 -e NB_MANAGEMENT_URL=<REDACTED>
 netbirdio/netbird:latest

How I run k8s pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: netbird
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: netbird
  template:
    metadata:
      labels:
        app: netbird
    spec:
      containers:
        - name: netbird
          image: netbirdio/netbird:latest
          env:
            - name: NB_MANAGEMENT_URL
              value: "<REDACTED>"
            - name: NB_SETUP_KEY
              value: "<REDACTED>" # replace with your setup key
          securityContext:
            capabilities:
              add:
                - NET_ADMIN

Example commands executed in docker container or in k8s pods.
Nslookup return correct ip 100.87.97.114

nslookup netbird-client-1-2.netbird.selfhosted
Server:		100.87.150.177
Address:	100.87.150.177:53

** server can't find netbird-client-1-2.netbird.selfhosted: NXDOMAIN

Non-authoritative answer:
Name:	netbird-client-1-2.netbird.selfhosted
Address: 100.87.97.114

Ping with ip works correct

/ # ping 100.87.97.114
PING 100.87.97.114 (100.87.97.114): 56 data bytes
64 bytes from 100.87.97.114: seq=0 ttl=64 time=1.503 ms
64 bytes from 100.87.97.114: seq=1 ttl=64 time=1.032 ms

Wget with ip works correct

wget 100.87.97.114:5555
Connecting to 100.87.97.114:5555 (100.87.97.114:5555)
saving to 'index.html'
index.html           100% |******************************************************|     5  0:00:00 ETA
'index.html' saved

Error here

But when I execute client http request or ping for host I received dns resolve error:

ping netbird-client-1-2.netbird.selfhosted
ping: bad address 'netbird-client-1-2.netbird.selfhosted'
wget netbird-client-1-2.netbird.selfhosted
wget: bad address 'netbird-client-1-2.netbird.selfhosted'
wget netbird-client-1-2:5555
wget: bad address 'netbird-client-1-2:5555'

Expected behavior

Ping, curl or wget should send request

Are you using NetBird Cloud?

Netbird self hosted

NetBird version

0.27.10

NetBird status -d output:

I run netbird in container, I cant execute status output

Originally created by @gawsoftpl on GitHub (May 27, 2024). **Describe the problem** Netbird in docker container or in kubernetes pod for self hosted netbird. Dns resolve correct ip (nslookup works correct) but curl, ping, wget return error: bad address For normal ubuntu instance without docker or kubernetes evereythings works correct How I run docker ```sh docker run --rm -d --cap-add=NET_ADMIN -e NB_SETUP_KEY=<REDACTED> -v netbird-client:/etc/netbird -e NB_MANAGEMENT_URL=<REDACTED> netbirdio/netbird:latest ``` How I run k8s pod ```sh apiVersion: apps/v1 kind: Deployment metadata: name: netbird namespace: default spec: replicas: 1 selector: matchLabels: app: netbird template: metadata: labels: app: netbird spec: containers: - name: netbird image: netbirdio/netbird:latest env: - name: NB_MANAGEMENT_URL value: "<REDACTED>" - name: NB_SETUP_KEY value: "<REDACTED>" # replace with your setup key securityContext: capabilities: add: - NET_ADMIN ``` Example commands executed in docker container or in k8s pods. Nslookup return correct ip 100.87.97.114 ```sh nslookup netbird-client-1-2.netbird.selfhosted Server: 100.87.150.177 Address: 100.87.150.177:53 ** server can't find netbird-client-1-2.netbird.selfhosted: NXDOMAIN Non-authoritative answer: Name: netbird-client-1-2.netbird.selfhosted Address: 100.87.97.114 ``` Ping with ip works correct ```sh / # ping 100.87.97.114 PING 100.87.97.114 (100.87.97.114): 56 data bytes 64 bytes from 100.87.97.114: seq=0 ttl=64 time=1.503 ms 64 bytes from 100.87.97.114: seq=1 ttl=64 time=1.032 ms ``` Wget with ip works correct ```sh wget 100.87.97.114:5555 Connecting to 100.87.97.114:5555 (100.87.97.114:5555) saving to 'index.html' index.html 100% |******************************************************| 5 0:00:00 ETA 'index.html' saved ``` **Error here** But when I execute client http request or ping for host I received dns resolve error: ```sh ping netbird-client-1-2.netbird.selfhosted ping: bad address 'netbird-client-1-2.netbird.selfhosted' ``` ```sh wget netbird-client-1-2.netbird.selfhosted wget: bad address 'netbird-client-1-2.netbird.selfhosted' ``` ```sh wget netbird-client-1-2:5555 wget: bad address 'netbird-client-1-2:5555' ``` **Expected behavior** Ping, curl or wget should send request **Are you using NetBird Cloud?** Netbird self hosted **NetBird version** 0.27.10 **NetBird status -d output:** I run netbird in container, I cant execute status output
saavagebueno added the clientwaiting-feedbackdockerself-hosting labels 2025-11-20 05:20:08 -05:00
Author
Owner

@bcmmbaga commented on GitHub (May 28, 2024):

Hello @gawsoftpl, can you try testing again running client with these additional capabilities: --cap-add=SYS_ADMIN and --cap-add=SYS_RESOURCE ?

@bcmmbaga commented on GitHub (May 28, 2024): Hello @gawsoftpl, can you try testing again running client with these additional capabilities: `--cap-add=SYS_ADMIN` and `--cap-add=SYS_RESOURCE` ?
Author
Owner

@gawsoftpl commented on GitHub (May 28, 2024):

Hello @gawsoftpl, can you try testing again running client with these additional capabilities: --cap-add=SYS_ADMIN and --cap-add=SYS_RESOURCE ?

This not help in docker and k8s same issue.

I past resolv conf

cat /etc/resolv.conf
# Generated by NetBird
# If needed you can restore the original file by copying back /etc/resolv.conf.original.netbird

options timeout:4 attempts:1
search netbird.selfhosted
nameserver 100.87.0.130
nameserver 100.87.231.247
nameserver 185.12.64.1
nameserver 185.12.64.2
cat /etc/nsswitch.conf 
# musl itself does not support NSS, however some third-party DNS
# implementations use the nsswitch.conf file to determine what
# policy to follow.
# Editing this file is not recommended.
hosts: files dns
cat /etc/resolv.conf.original.netbird
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 100.87.231.247
nameserver 185.12.64.1
nameserver 185.12.64.2
search netbird.selfhosted

# Based on host file: '/run/systemd/resolve/resolv.conf' (legacy)
# Overrides: []

And ifconfig from docker container

eth0      Link encap:Ethernet  HWaddr 02:42:AC:11:00:02  
          inet addr:172.17.0.2  Bcast:172.17.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:948 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1107 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:135010 (131.8 KiB)  TX bytes:137313 (134.0 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:39 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3898 (3.8 KiB)  TX bytes:3898 (3.8 KiB)

wt0       Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:100.87.0.130  P-t-P:100.87.0.130  Mask:255.255.0.0
          UP POINTOPOINT RUNNING NOARP  MTU:1280  Metric:1
          RX packets:14 errors:0 dropped:0 overruns:0 frame:0
          TX packets:18 errors:5 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1364 (1.3 KiB)  TX bytes:1928 (1.8 KiB)
@gawsoftpl commented on GitHub (May 28, 2024): > Hello @gawsoftpl, can you try testing again running client with these additional capabilities: `--cap-add=SYS_ADMIN` and `--cap-add=SYS_RESOURCE` ? This not help in docker and k8s same issue. I past resolv conf ```sh cat /etc/resolv.conf # Generated by NetBird # If needed you can restore the original file by copying back /etc/resolv.conf.original.netbird options timeout:4 attempts:1 search netbird.selfhosted nameserver 100.87.0.130 nameserver 100.87.231.247 nameserver 185.12.64.1 nameserver 185.12.64.2 ``` ```sh cat /etc/nsswitch.conf # musl itself does not support NSS, however some third-party DNS # implementations use the nsswitch.conf file to determine what # policy to follow. # Editing this file is not recommended. hosts: files dns ``` ```sh cat /etc/resolv.conf.original.netbird # Generated by Docker Engine. # This file can be edited; Docker Engine will not make further changes once it # has been modified. nameserver 100.87.231.247 nameserver 185.12.64.1 nameserver 185.12.64.2 search netbird.selfhosted # Based on host file: '/run/systemd/resolve/resolv.conf' (legacy) # Overrides: [] ``` And ifconfig from docker container ```sh eth0 Link encap:Ethernet HWaddr 02:42:AC:11:00:02 inet addr:172.17.0.2 Bcast:172.17.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:948 errors:0 dropped:0 overruns:0 frame:0 TX packets:1107 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:135010 (131.8 KiB) TX bytes:137313 (134.0 KiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:39 errors:0 dropped:0 overruns:0 frame:0 TX packets:39 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3898 (3.8 KiB) TX bytes:3898 (3.8 KiB) wt0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:100.87.0.130 P-t-P:100.87.0.130 Mask:255.255.0.0 UP POINTOPOINT RUNNING NOARP MTU:1280 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:18 errors:5 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1364 (1.3 KiB) TX bytes:1928 (1.8 KiB) ```
Author
Owner

@bcmmbaga commented on GitHub (May 28, 2024):

@gawsoftpl you can enable the debug logs with -e NB_LOG_LEVEL=debug and share the client logs

@bcmmbaga commented on GitHub (May 28, 2024): @gawsoftpl you can enable the debug logs with `-e NB_LOG_LEVEL=debug` and share the client logs
Author
Owner

@gawsoftpl commented on GitHub (May 28, 2024):

logs from docker container:
logs.txt

@gawsoftpl commented on GitHub (May 28, 2024): logs from docker container: [logs.txt](https://github.com/netbirdio/netbird/files/15470483/logs.txt)
Author
Owner

@Kidswiss commented on GitHub (Jan 10, 2025):

I have the same issue in Kubernetes.

The DNS resolution isn't consistent. There are tools that work, and there are tools that don't.

For example, working for me are:

  • dig
  • curl

Not working are:

  • wget
  • curl

nslookup is somewhere in the middle, it does actually return the right IP, but it also throws an NXDOMAIN.

nslookup -debug myhost.netbird.selfhosted
Server:         10.43.0.10
Address:        10.43.0.10#53

------------
    QUESTIONS:
        myhost.netbird.selfhosted.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
        origin = ns.dns.cluster.local
        mail addr = hostmaster.cluster.local
        serial = 1736502945
        refresh = 7200
        retry = 1800
        expire = 86400
        minimum = 5
        ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find myhost.netbird.selfhosted.default.svc.cluster.local: NXDOMAIN
Server:         10.43.0.10
Address:        10.43.0.10#53

------------
    QUESTIONS:
        myhost.netbird.selfhosted.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
        origin = ns.dns.cluster.local
        mail addr = hostmaster.cluster.local
        serial = 1736502945
        refresh = 7200
        retry = 1800
        expire = 86400
        minimum = 5
        ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find myhost.netbird.selfhosted.svc.cluster.local: NXDOMAIN
Server:         10.43.0.10
Address:        10.43.0.10#53

------------
    QUESTIONS:
        myhost.netbird.selfhosted.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
        origin = ns.dns.cluster.local
        mail addr = hostmaster.cluster.local
        serial = 1736502945
        refresh = 7200
        retry = 1800
        expire = 86400
        minimum = 5
        ttl = 5
    ADDITIONAL RECORDS:
------------
** server can't find myhost.netbird.selfhosted.cluster.local: NXDOMAIN
Server:         10.43.0.10
Address:        10.43.0.10#53

------------
    QUESTIONS:
        myhost.netbird.selfhosted, type = A, class = IN
    ANSWERS:
    ->  myhost.netbird.selfhosted
        internet address = 100.69.86.108
        ttl = 300
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name:   myhost.netbird.selfhosted
Address: 100.69.86.108
------------
    QUESTIONS:
        myhost.netbird.selfhosted, type = AAAA, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
** server can't find myhost.netbird.selfhosted: NXDOMAIN

An here dig:

dig myhost.netbird.selfhosted

; <<>> DiG 9.18.25 <<>> myhost.netbird.selfhosted
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13706
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 710e26c6b25cfdd2 (echoed)
;; QUESTION SECTION:
;myhost.netbird.selfhosted.     IN      A

;; ANSWER SECTION:
myhost.netbird.selfhosted. 300  IN      A       100.69.86.108

;; Query time: 0 msec
;; SERVER: 10.43.0.10#53(10.43.0.10) (UDP)
;; WHEN: Fri Jan 10 10:14:02 UTC 2025
;; MSG SIZE  rcvd: 107

So there seems to be something wonky if the DNS lookups happen within a container for some reason.

@Kidswiss commented on GitHub (Jan 10, 2025): I have the same issue in Kubernetes. The DNS resolution isn't consistent. There are tools that work, and there are tools that don't. For example, working for me are: * dig * curl Not working are: * wget * curl nslookup is somewhere in the middle, it does actually return the right IP, but it also throws an NXDOMAIN. ``` nslookup -debug myhost.netbird.selfhosted Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: myhost.netbird.selfhosted.default.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1736502945 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find myhost.netbird.selfhosted.default.svc.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: myhost.netbird.selfhosted.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1736502945 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find myhost.netbird.selfhosted.svc.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: myhost.netbird.selfhosted.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> cluster.local origin = ns.dns.cluster.local mail addr = hostmaster.cluster.local serial = 1736502945 refresh = 7200 retry = 1800 expire = 86400 minimum = 5 ttl = 5 ADDITIONAL RECORDS: ------------ ** server can't find myhost.netbird.selfhosted.cluster.local: NXDOMAIN Server: 10.43.0.10 Address: 10.43.0.10#53 ------------ QUESTIONS: myhost.netbird.selfhosted, type = A, class = IN ANSWERS: -> myhost.netbird.selfhosted internet address = 100.69.86.108 ttl = 300 AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ Non-authoritative answer: Name: myhost.netbird.selfhosted Address: 100.69.86.108 ------------ QUESTIONS: myhost.netbird.selfhosted, type = AAAA, class = IN ANSWERS: AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ ** server can't find myhost.netbird.selfhosted: NXDOMAIN ``` An here dig: ``` dig myhost.netbird.selfhosted ; <<>> DiG 9.18.25 <<>> myhost.netbird.selfhosted ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13706 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1232 ; COOKIE: 710e26c6b25cfdd2 (echoed) ;; QUESTION SECTION: ;myhost.netbird.selfhosted. IN A ;; ANSWER SECTION: myhost.netbird.selfhosted. 300 IN A 100.69.86.108 ;; Query time: 0 msec ;; SERVER: 10.43.0.10#53(10.43.0.10) (UDP) ;; WHEN: Fri Jan 10 10:14:02 UTC 2025 ;; MSG SIZE rcvd: 107 ``` So there seems to be something wonky if the DNS lookups happen within a container for some reason.
Author
Owner

@nazarewk commented on GitHub (Apr 28, 2025):

Hello @gawsoftpl,

We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.

Could you please confirm if the issue is still there?

We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.

Thanks for your contribution to improving the project!

@nazarewk commented on GitHub (Apr 28, 2025): Hello @gawsoftpl, We're currently reviewing our open issues and would like to verify if this problem still exists in the [latest NetBird version](https://github.com/netbirdio/netbird/releases). Could you please confirm if the issue is still there? We may close this issue temporarily if we don't hear back from you within **2 weeks**, but feel free to reopen it with updated information. Thanks for your contribution to improving the project!
Author
Owner

@Kidswiss commented on GitHub (Apr 28, 2025):

@nazarewk

Can config, still the same behaviour as in my comment above. Just tested with Netbird 0.43.0.

@Kidswiss commented on GitHub (Apr 28, 2025): @nazarewk Can config, still the same behaviour as in my comment above. Just tested with Netbird 0.43.0.
Author
Owner

@Kidswiss commented on GitHub (Jul 11, 2025):

@nazarewk

I think I found the root cause of this:

Kubernetes injects resolv.conf into each, they look like this:

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5

The problem is the ndots:5 option. Netbird's DNS resolver seems to get confused by that. Removing that option fixes the issue (but will cause a lot more other issues with Kubernetes name resolution).

Here is the difference in DNS answers between tailscale DNS and netbird DNS:

# tailscale
nslookup -debug highcharity.ts.selfhosted. 100.100.100.100
Server:         100.100.100.100
Address:        100.100.100.100#53

------------
    QUESTIONS:
        highcharity.ts.selfhosted, type = A, class = IN
    ANSWERS:
    ->  highcharity.ts.selfhosted
        internet address = 100.64.0.2
        ttl = 600
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   highcharity.ts.selfhosted
Address: 100.64.0.2
------------
    QUESTIONS:
        highcharity.ts.selfhosted, type = AAAA, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------

#netbird
nslookup -debug highcharity.netbird.selfhosted 100.69.190.139
Server:         100.69.190.139
Address:        100.69.190.139#53

------------
    QUESTIONS:
        highcharity.netbird.selfhosted.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  highcharity.netbird.selfhosted.default.svc.cluster.local
        origin = highcharity.netbird.selfhosted.default.svc.cluster.local
        mail addr = nobody.invalid
        serial = 1
        refresh = 3600
        retry = 1200
        expire = 604800
        minimum = 10800
        ttl = 10800
    ADDITIONAL RECORDS:
    ->  explanation.invalid
        text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml"
        ttl = 10800
------------
** server can't find highcharity.netbird.selfhosted.default.svc.cluster.local: NXDOMAIN
Server:         100.69.190.139
Address:        100.69.190.139#53

------------
    QUESTIONS:
        highcharity.netbird.selfhosted.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  highcharity.netbird.selfhosted.svc.cluster.local
        origin = highcharity.netbird.selfhosted.svc.cluster.local
        mail addr = nobody.invalid
        serial = 1
        refresh = 3600
        retry = 1200
        expire = 604800
        minimum = 10800
        ttl = 10800
    ADDITIONAL RECORDS:
    ->  explanation.invalid
        text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml"
        ttl = 10800
------------
** server can't find highcharity.netbird.selfhosted.svc.cluster.local: NXDOMAIN
Server:         100.69.190.139
Address:        100.69.190.139#53

------------
    QUESTIONS:
        highcharity.netbird.selfhosted.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  highcharity.netbird.selfhosted.cluster.local
        origin = highcharity.netbird.selfhosted.cluster.local
        mail addr = nobody.invalid
        serial = 1
        refresh = 3600
        retry = 1200
        expire = 604800
        minimum = 10800
        ttl = 10800
    ADDITIONAL RECORDS:
    ->  explanation.invalid
        text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml"
        ttl = 10800
------------
** server can't find highcharity.netbird.selfhosted.cluster.local: NXDOMAIN
Server:         100.69.190.139
Address:        100.69.190.139#53

------------
    QUESTIONS:
        highcharity.netbird.selfhosted, type = A, class = IN
    ANSWERS:
    ->  highcharity.netbird.selfhosted
        internet address = 100.69.65.6
        ttl = 300
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name:   highcharity.netbird.selfhosted
Address: 100.69.65.6
------------
    QUESTIONS:
        highcharity.netbird.selfhosted, type = AAAA, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------

Netbird returns server can't find highcharity.netbird.selfhosted.cluster.local: NXDOMAIN for all tries that have the search domains appended, while Tailscale seemingly just skips them.

@Kidswiss commented on GitHub (Jul 11, 2025): @nazarewk I think I found the root cause of this: Kubernetes injects `resolv.conf` into each, they look like this: ``` search default.svc.cluster.local svc.cluster.local cluster.local nameserver 10.43.0.10 options ndots:5 ``` The problem is the `ndots:5` option. Netbird's DNS resolver seems to get confused by that. Removing that option fixes the issue (but will cause a lot more other issues with Kubernetes name resolution). Here is the difference in DNS answers between tailscale DNS and netbird DNS: ``` # tailscale nslookup -debug highcharity.ts.selfhosted. 100.100.100.100 Server: 100.100.100.100 Address: 100.100.100.100#53 ------------ QUESTIONS: highcharity.ts.selfhosted, type = A, class = IN ANSWERS: -> highcharity.ts.selfhosted internet address = 100.64.0.2 ttl = 600 AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ Name: highcharity.ts.selfhosted Address: 100.64.0.2 ------------ QUESTIONS: highcharity.ts.selfhosted, type = AAAA, class = IN ANSWERS: AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ #netbird nslookup -debug highcharity.netbird.selfhosted 100.69.190.139 Server: 100.69.190.139 Address: 100.69.190.139#53 ------------ QUESTIONS: highcharity.netbird.selfhosted.default.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> highcharity.netbird.selfhosted.default.svc.cluster.local origin = highcharity.netbird.selfhosted.default.svc.cluster.local mail addr = nobody.invalid serial = 1 refresh = 3600 retry = 1200 expire = 604800 minimum = 10800 ttl = 10800 ADDITIONAL RECORDS: -> explanation.invalid text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml" ttl = 10800 ------------ ** server can't find highcharity.netbird.selfhosted.default.svc.cluster.local: NXDOMAIN Server: 100.69.190.139 Address: 100.69.190.139#53 ------------ QUESTIONS: highcharity.netbird.selfhosted.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> highcharity.netbird.selfhosted.svc.cluster.local origin = highcharity.netbird.selfhosted.svc.cluster.local mail addr = nobody.invalid serial = 1 refresh = 3600 retry = 1200 expire = 604800 minimum = 10800 ttl = 10800 ADDITIONAL RECORDS: -> explanation.invalid text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml" ttl = 10800 ------------ ** server can't find highcharity.netbird.selfhosted.svc.cluster.local: NXDOMAIN Server: 100.69.190.139 Address: 100.69.190.139#53 ------------ QUESTIONS: highcharity.netbird.selfhosted.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> highcharity.netbird.selfhosted.cluster.local origin = highcharity.netbird.selfhosted.cluster.local mail addr = nobody.invalid serial = 1 refresh = 3600 retry = 1200 expire = 604800 minimum = 10800 ttl = 10800 ADDITIONAL RECORDS: -> explanation.invalid text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml" ttl = 10800 ------------ ** server can't find highcharity.netbird.selfhosted.cluster.local: NXDOMAIN Server: 100.69.190.139 Address: 100.69.190.139#53 ------------ QUESTIONS: highcharity.netbird.selfhosted, type = A, class = IN ANSWERS: -> highcharity.netbird.selfhosted internet address = 100.69.65.6 ttl = 300 AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ Non-authoritative answer: Name: highcharity.netbird.selfhosted Address: 100.69.65.6 ------------ QUESTIONS: highcharity.netbird.selfhosted, type = AAAA, class = IN ANSWERS: AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ ``` Netbird returns `server can't find highcharity.netbird.selfhosted.cluster.local: NXDOMAIN` for all tries that have the search domains appended, while Tailscale seemingly just skips them.
Author
Owner

@nazarewk commented on GitHub (Jul 11, 2025):

@Kidswiss Thanks for sharing this with us, let's dig slightly more into it before contacting developer team.

You have included a . at the end of the domain (making it FQDN == skipping search lookups) in Tailscale example, but not the NetBird example;

  1. this means only the last Question-Response is relevant.
  2. the last Question is for AAAA record, while we don't support IPv6: so lack of the Answer is expected
  3. second to last Question is for A record and does return a correct answer

I would expect tailscale to return exactly the same set of queries as soon as you skip the trailing dot ., because it's an operating-system level mechanism independent of the VPN used. It is simply what the OS/networking stack is told to do.

I'm seeing kubernetes operators are well aware of this default and sometimes suggest reducing ndots to 1 https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html .


Could you revise your answer to refine it with more information or confirm that it's indeed the case?

@nazarewk commented on GitHub (Jul 11, 2025): @Kidswiss Thanks for sharing this with us, let's dig slightly more into it before contacting developer team. You have included a `.` at the end of the domain (making it FQDN == skipping search lookups) in Tailscale example, but not the NetBird example; 1. this means only the last Question-Response is relevant. 2. the last Question is for `AAAA` record, while we don't support IPv6: so lack of the Answer is expected 3. second to last Question is for `A` record and does return a correct answer I would expect `tailscale` to return exactly the same set of queries as soon as you skip the trailing dot `.`, because it's an operating-system level mechanism independent of the VPN used. It is simply what the OS/networking stack is told to do. I'm seeing kubernetes operators are well aware of this default and sometimes suggest reducing `ndots` to `1` https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html . --- Could you revise your answer to refine it with more information or confirm that it's indeed the case?
Author
Owner

@Kidswiss commented on GitHub (Jul 11, 2025):

@nazarewk

Good catch! I did a new test:

# Tailscale
nslookup -debug highcharity.ts.selfhosted 100.100.100.100
Server:         100.100.100.100
Address:        100.100.100.100#53

------------
    QUESTIONS:
        highcharity.ts.selfhosted.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  .
        origin = a.root-servers.net
        mail addr = nstld.verisign-grs.com
        serial = 2025071101
        refresh = 1800
        retry = 900
        expire = 604800
        minimum = 86400
        ttl = 86400
    ADDITIONAL RECORDS:
------------
** server can't find highcharity.ts.selfhosted.default.svc.cluster.local: NXDOMAIN
Server:         100.100.100.100
Address:        100.100.100.100#53

------------
    QUESTIONS:
        highcharity.ts.selfhosted.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  .
        origin = a.root-servers.net
        mail addr = nstld.verisign-grs.com
        serial = 2025071101
        refresh = 1800
        retry = 900
        expire = 604800
        minimum = 86400
        ttl = 86400
    ADDITIONAL RECORDS:
------------
** server can't find highcharity.ts.selfhosted.svc.cluster.local: NXDOMAIN
Server:         100.100.100.100
Address:        100.100.100.100#53

------------
    QUESTIONS:
        highcharity.ts.selfhosted.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  .
        origin = a.root-servers.net
        mail addr = nstld.verisign-grs.com
        serial = 2025071101
        refresh = 1800
        retry = 900
        expire = 604800
        minimum = 86400
        ttl = 86400
    ADDITIONAL RECORDS:
------------
** server can't find highcharity.ts.selfhosted.cluster.local: NXDOMAIN
Server:         100.100.100.100
Address:        100.100.100.100#53

------------
    QUESTIONS:
        highcharity.ts.selfhosted, type = A, class = IN
    ANSWERS:
    ->  highcharity.ts.selfhosted
        internet address = 100.64.0.2
        ttl = 600
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   highcharity.ts.selfhosted
Address: 100.64.0.2
------------
    QUESTIONS:
        highcharity.ts.selfhosted, type = AAAA, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------

nslookup  highcharity.ts.selfhosted 100.100.100.100
Server:         100.100.100.100
Address:        100.100.100.100#53

Name:   highcharity.ts.selfhosted
Address: 100.64.0.2

# And netbird again
nslookup -debug highcharity.netbird.selfhosted 100.69.190.139
Server:         100.69.190.139
Address:        100.69.190.139#53

------------
    QUESTIONS:
        highcharity.netbird.selfhosted.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  highcharity.netbird.selfhosted.default.svc.cluster.local
        origin = highcharity.netbird.selfhosted.default.svc.cluster.local
        mail addr = nobody.invalid
        serial = 1
        refresh = 3600
        retry = 1200
        expire = 604800
        minimum = 10800
        ttl = 10800
    ADDITIONAL RECORDS:
    ->  explanation.invalid
        text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml"
        ttl = 10800
------------
** server can't find highcharity.netbird.selfhosted.default.svc.cluster.local: NXDOMAIN
Server:         100.69.190.139
Address:        100.69.190.139#53

------------
    QUESTIONS:
        highcharity.netbird.selfhosted.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  highcharity.netbird.selfhosted.svc.cluster.local
        origin = highcharity.netbird.selfhosted.svc.cluster.local
        mail addr = nobody.invalid
        serial = 1
        refresh = 3600
        retry = 1200
        expire = 604800
        minimum = 10800
        ttl = 10800
    ADDITIONAL RECORDS:
    ->  explanation.invalid
        text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml"
        ttl = 10800
------------
** server can't find highcharity.netbird.selfhosted.svc.cluster.local: NXDOMAIN
Server:         100.69.190.139
Address:        100.69.190.139#53

------------
    QUESTIONS:
        highcharity.netbird.selfhosted.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  highcharity.netbird.selfhosted.cluster.local
        origin = highcharity.netbird.selfhosted.cluster.local
        mail addr = nobody.invalid
        serial = 1
        refresh = 3600
        retry = 1200
        expire = 604800
        minimum = 10800
        ttl = 10800
    ADDITIONAL RECORDS:
    ->  explanation.invalid
        text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml"
        ttl = 10800
------------
** server can't find highcharity.netbird.selfhosted.cluster.local: NXDOMAIN
Server:         100.69.190.139
Address:        100.69.190.139#53

------------
    QUESTIONS:
        highcharity.netbird.selfhosted, type = A, class = IN
    ANSWERS:
    ->  highcharity.netbird.selfhosted
        internet address = 100.69.65.6
        ttl = 300
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
Name:   highcharity.netbird.selfhosted
Address: 100.69.65.6
------------
    QUESTIONS:
        highcharity.netbird.selfhosted, type = AAAA, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------

nslookup  highcharity.netbird.selfhosted 100.69.190.139
Server:         100.69.190.139
Address:        100.69.190.139#53

*** Can't find highcharity.netbird.selfhosted.default.svc.cluster.local: No answer

Their answers sections look almost identical, the only difference is, that Netbird's dinal answer is Non-authoritative answer while Tailscale's is. Also the main reason I'm using Tailscale is because the DNS resolution works reliable via K8s.

@Kidswiss commented on GitHub (Jul 11, 2025): @nazarewk Good catch! I did a new test: ``` # Tailscale nslookup -debug highcharity.ts.selfhosted 100.100.100.100 Server: 100.100.100.100 Address: 100.100.100.100#53 ------------ QUESTIONS: highcharity.ts.selfhosted.default.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> . origin = a.root-servers.net mail addr = nstld.verisign-grs.com serial = 2025071101 refresh = 1800 retry = 900 expire = 604800 minimum = 86400 ttl = 86400 ADDITIONAL RECORDS: ------------ ** server can't find highcharity.ts.selfhosted.default.svc.cluster.local: NXDOMAIN Server: 100.100.100.100 Address: 100.100.100.100#53 ------------ QUESTIONS: highcharity.ts.selfhosted.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> . origin = a.root-servers.net mail addr = nstld.verisign-grs.com serial = 2025071101 refresh = 1800 retry = 900 expire = 604800 minimum = 86400 ttl = 86400 ADDITIONAL RECORDS: ------------ ** server can't find highcharity.ts.selfhosted.svc.cluster.local: NXDOMAIN Server: 100.100.100.100 Address: 100.100.100.100#53 ------------ QUESTIONS: highcharity.ts.selfhosted.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> . origin = a.root-servers.net mail addr = nstld.verisign-grs.com serial = 2025071101 refresh = 1800 retry = 900 expire = 604800 minimum = 86400 ttl = 86400 ADDITIONAL RECORDS: ------------ ** server can't find highcharity.ts.selfhosted.cluster.local: NXDOMAIN Server: 100.100.100.100 Address: 100.100.100.100#53 ------------ QUESTIONS: highcharity.ts.selfhosted, type = A, class = IN ANSWERS: -> highcharity.ts.selfhosted internet address = 100.64.0.2 ttl = 600 AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ Name: highcharity.ts.selfhosted Address: 100.64.0.2 ------------ QUESTIONS: highcharity.ts.selfhosted, type = AAAA, class = IN ANSWERS: AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ nslookup highcharity.ts.selfhosted 100.100.100.100 Server: 100.100.100.100 Address: 100.100.100.100#53 Name: highcharity.ts.selfhosted Address: 100.64.0.2 # And netbird again nslookup -debug highcharity.netbird.selfhosted 100.69.190.139 Server: 100.69.190.139 Address: 100.69.190.139#53 ------------ QUESTIONS: highcharity.netbird.selfhosted.default.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> highcharity.netbird.selfhosted.default.svc.cluster.local origin = highcharity.netbird.selfhosted.default.svc.cluster.local mail addr = nobody.invalid serial = 1 refresh = 3600 retry = 1200 expire = 604800 minimum = 10800 ttl = 10800 ADDITIONAL RECORDS: -> explanation.invalid text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml" ttl = 10800 ------------ ** server can't find highcharity.netbird.selfhosted.default.svc.cluster.local: NXDOMAIN Server: 100.69.190.139 Address: 100.69.190.139#53 ------------ QUESTIONS: highcharity.netbird.selfhosted.svc.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> highcharity.netbird.selfhosted.svc.cluster.local origin = highcharity.netbird.selfhosted.svc.cluster.local mail addr = nobody.invalid serial = 1 refresh = 3600 retry = 1200 expire = 604800 minimum = 10800 ttl = 10800 ADDITIONAL RECORDS: -> explanation.invalid text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml" ttl = 10800 ------------ ** server can't find highcharity.netbird.selfhosted.svc.cluster.local: NXDOMAIN Server: 100.69.190.139 Address: 100.69.190.139#53 ------------ QUESTIONS: highcharity.netbird.selfhosted.cluster.local, type = A, class = IN ANSWERS: AUTHORITY RECORDS: -> highcharity.netbird.selfhosted.cluster.local origin = highcharity.netbird.selfhosted.cluster.local mail addr = nobody.invalid serial = 1 refresh = 3600 retry = 1200 expire = 604800 minimum = 10800 ttl = 10800 ADDITIONAL RECORDS: -> explanation.invalid text = "Blocking is mandated by standards, see references on https://www.iana.org/assignments/special-use-domain-names/special-use-domain-names.xhtml" ttl = 10800 ------------ ** server can't find highcharity.netbird.selfhosted.cluster.local: NXDOMAIN Server: 100.69.190.139 Address: 100.69.190.139#53 ------------ QUESTIONS: highcharity.netbird.selfhosted, type = A, class = IN ANSWERS: -> highcharity.netbird.selfhosted internet address = 100.69.65.6 ttl = 300 AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ Non-authoritative answer: Name: highcharity.netbird.selfhosted Address: 100.69.65.6 ------------ QUESTIONS: highcharity.netbird.selfhosted, type = AAAA, class = IN ANSWERS: AUTHORITY RECORDS: ADDITIONAL RECORDS: ------------ nslookup highcharity.netbird.selfhosted 100.69.190.139 Server: 100.69.190.139 Address: 100.69.190.139#53 *** Can't find highcharity.netbird.selfhosted.default.svc.cluster.local: No answer ``` Their answers sections look almost identical, the only difference is, that Netbird's dinal answer is `Non-authoritative answer` while Tailscale's is. Also the main reason I'm using Tailscale is because the DNS resolution works reliable via K8s.
Author
Owner

@nazarewk commented on GitHub (Jul 14, 2025):

I'm not sure how tailscale does it not to report Non-authorative answer, but it should not affect anything.
It just means the answer did not arrive directly from the nameserver advertising the domain, you will notice this whenever you put any kind of DNS resolver, proxy or cache in the middle of resolving the request.

I can ask the team about making this an authoritative answer, but I don't think it would get much priority as this is mostly a cosmetic change.


As per the previous comments in this Issue about the bad address errors. Do you have a minimal example that would reproduce this?

@nazarewk commented on GitHub (Jul 14, 2025): I'm not sure how `tailscale` does it not to report `Non-authorative answer`, but it should not affect anything. It just means the answer did not arrive **directly** from the nameserver advertising the domain, you will notice this whenever you put any kind of DNS resolver, proxy or cache in the middle of resolving the request. I can ask the team about making this an authoritative answer, but I don't think it would get much priority as this is mostly a cosmetic change. --- As per the previous comments in this Issue about the `bad address` errors. Do you have a minimal example that would reproduce this?
Author
Owner

@Kidswiss commented on GitHub (Jul 14, 2025):

It just means the answer did not arrive directly from the nameserver advertising the domain, you will notice this whenever you put any kind of DNS resolver, proxy or cache in the middle of resolving the request.

I agree, but that's the only difference I can make out in those two calls. It's very possible that the issue is something else and buried somewhere in the wireprotocol answer.

How to reproduce:

  1. install netbird on a linux host, I'm using Almalinux
  2. Make sure it has access to some other nodes and that the local DNS resolution works
  3. Install k3s
  4. Use this command to create an interactive pod to test out DNS
kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -n default
@Kidswiss commented on GitHub (Jul 14, 2025): > It just means the answer did not arrive directly from the nameserver advertising the domain, you will notice this whenever you put any kind of DNS resolver, proxy or cache in the middle of resolving the request. I agree, but that's the only difference I can make out in those two calls. It's very possible that the issue is something else and buried somewhere in the wireprotocol answer. How to reproduce: 1. install netbird on a linux host, I'm using Almalinux 2. Make sure it has access to some other nodes and that the local DNS resolution works 3. Install k3s 4. Use this command to create an interactive pod to test out DNS ``` kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot -n default ```
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#932