SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-04 19:55:09 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

DNS resolution error on the client when receiving a large response #1671

New Issue
Closed
opened 2025-11-20 06:04:30 -05:00 by saavagebueno · 4 comments
saavagebueno commented 2025-11-20 06:04:30 -05:00
Owner
Copy Link

Originally created by @strmaks on GitHub (Mar 2, 2025).

I have several domain name based routes to provide our CI system access to external resources.
As part of migrating from Network Routes to Networks I detected at least two domain names which break resolving flow: registry.terraform.io and registry.yarnpkg.com .

When trying to access them nothing happens, after a while the error that the domain name can not be resolved is returned.
I enabled debug log level on NB client and found the next lines:

command
curl -v https://registry.yarnpkg.com

produces the next logs

2025-03-02T19:46:54+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: overflowing header size
2025-03-02T19:46:55+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: overflowing header size
2025-03-02T19:46:57+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: overflowing header size
2025-03-02T19:47:01+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: overflowing header size

command
curl -v https://registry.terraform.io
produces

2025-03-02T20:07:39+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: buffer size too small
2025-03-02T20:07:40+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: buffer size too small
2025-03-02T20:07:42+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: buffer size too small
2025-03-02T20:07:47+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: buffer size too small

But for example
curl -v https://terraform.io
works fine without any problems as well as "OLD" flow using 0.34.1 client.

After some tcpdumping on client host i detect that DNS responces for "bad" (non resolvable) names is bigger than for "good" (resolvable) names.
607 bytes for registry.terraform.io and 1074 for registry.yarnpkg.com
"good" responces is 80-200 bytes.

googling 'dns: buffer size too small' error addressed me to https://github.com/miekg/dns/issues/214 issue and suggest to increase UDP buffer size.

Currently i'm reverted back the old client to keep our system working.

To Reproduce

Create "Network Routes" for both domains
Try to access to this resources on client

Expected behavior

Domain names are successfully resolved and passed through “exit nodes”.

Are you using NetBird Cloud?

Self-Hosted NetBird

NetBird version

0.37.1
on client and "exit nodes"

Client: MacOS 15.3.1 arm64
Exit Nodes: Ubuntu Linux x86 24.04.2

Originally created by @strmaks on GitHub (Mar 2, 2025). I have several domain name based routes to provide our CI system access to external resources. As part of migrating from Network Routes to Networks I detected at least two domain names which break resolving flow: `registry.terraform.io` and `registry.yarnpkg.com` . When trying to access them nothing happens, after a while the error that the domain name can not be resolved is returned. I enabled debug log level on NB client and found the next lines: command `curl -v https://registry.yarnpkg.com` produces the next logs ``` 2025-03-02T19:46:54+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: overflowing header size 2025-03-02T19:46:55+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: overflowing header size 2025-03-02T19:46:57+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: overflowing header size 2025-03-02T19:47:01+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: overflowing header size ``` command `curl -v https://registry.terraform.io` produces ``` 2025-03-02T20:07:39+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: buffer size too small 2025-03-02T20:07:40+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: buffer size too small 2025-03-02T20:07:42+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: buffer size too small 2025-03-02T20:07:47+01:00 ERRO client/internal/routemanager/dnsinterceptor/handler.go:178: failed to exchange DNS request with 100.x.x.x:5353: dns: buffer size too small ``` But for example `curl -v https://terraform.io` works fine without any problems as well as "OLD" flow using 0.34.1 client. After some tcpdumping on client host i detect that DNS responces for "bad" (non resolvable) names is bigger than for "good" (resolvable) names. 607 bytes for registry.terraform.io and 1074 for registry.yarnpkg.com "good" responces is 80-200 bytes. googling 'dns: buffer size too small' error addressed me to https://github.com/miekg/dns/issues/214 issue and suggest to increase UDP buffer size. Currently i'm reverted back the old client to keep our system working. **To Reproduce** Create "Network Routes" for both domains Try to access to this resources on client **Expected behavior** Domain names are successfully resolved and passed through “exit nodes”. **Are you using NetBird Cloud?** Self-Hosted NetBird **NetBird version** `0.37.1` on client and "exit nodes" Client: MacOS 15.3.1 arm64 Exit Nodes: Ubuntu Linux x86 24.04.2
saavagebueno added the bugdns labels 2025-11-20 06:04:30 -05:00
saavagebueno commented 2025-11-20 06:04:31 -05:00
Author
Owner
Copy Link

@kislerdm commented on GitHub (Mar 4, 2025):

Hey @strmaks!

Given that the client was not modified between 0.34.1 and 0.37.1, the root cause is likely to be in some other component, i.e., not in the client which reports the error.

Regarding your comment:

suggest to increase UDP buffer size.

It might seem pragmatic solution, however, it does not solve the root cause. Moreover, it poses a reasonable question - what is the "right size"? The answer is not quite trivial because of the packet fragmentation and other factors - find more here.

@kislerdm commented on GitHub (Mar 4, 2025): Hey @strmaks! Given that the client was not modified between 0.34.1 and 0.37.1, the root cause is likely to be in some other component, i.e., not in the client which reports the error. Regarding your comment: > suggest to increase UDP buffer size. It might seem pragmatic solution, however, it does not solve the root cause. Moreover, it poses a reasonable question - what is the "right size"? The answer is not quite trivial because of the packet fragmentation and other factors - find more [here](https://www.netmeister.org/blog/dns-size.html).
saavagebueno commented 2025-11-20 06:04:31 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Mar 5, 2025):

@strmaks Can you check if the linked PR fixes your issue?

You can grab the binaries here once the build has finished: https://github.com/netbirdio/netbird/actions/runs/13675762717
Only the routing client needs the update.

@lixmal commented on GitHub (Mar 5, 2025): @strmaks Can you check if the linked PR fixes your issue? You can grab the binaries here once the build has finished: https://github.com/netbirdio/netbird/actions/runs/13675762717 Only the routing client needs the update.
saavagebueno commented 2025-11-20 06:04:31 -05:00
Author
Owner
Copy Link

@strmaks commented on GitHub (Mar 19, 2025):

Hello @lixmal thank you very much for your work!
Accidentally i'm fully lost all notifications for this issue.

I'm tested using the client 0.37.2 version included this fix.
Results is very ambiguous.

On linux and windows clients it works at first glance.
But on mac domain registry.yarnpkg.com still not worked.

And i found that with netbird some DNS results is totally wrong.

without netbird

[user@localhost ~]$ host google.com
google.com has address 142.250.186.206
google.com has IPv6 address 2a00:1450:401b:80d::200e
google.com mail is handled by 10 smtp.google.com.

with netbird

[user@localhost ~]$ host google.com
google.com has IPv6 address 2a00:1450:4001:81d::200e
google.com has address 142.250.186.142
google.com has IPv6 address 2a00:1450:4001:81d::200e
google.com has address 142.250.186.142
google.com has IPv6 address 2a00:1450:4001:81d::200e
google.com has address 142.250.186.142

It seems like the results are repeated 3 times
But after a while, I realized the problem was much bigger and deeper.

Netbird resolver completely ignores (or forgets) the requested record type.

without netbird

[user@localhost ~]$ host -t mx google.com
google.com mail is handled by 10 smtp.google.com.
[user@localhost ~]$ host -t a google.com
google.com has address 172.217.16.14
[user@localhost ~]$ host -t aaaa google.com
google.com has IPv6 address 2a00:1450:401b:80d::200e

with netbird

[user@localhost ~]$ host -t mx google.com
google.com has IPv6 address 2a00:1450:4001:81d::200e
google.com has address 142.250.186.142
[user@localhost ~]$ host -t a google.com
google.com has IPv6 address 2a00:1450:4001:81d::200e
google.com has address 142.250.186.142
[user@localhost ~]$ host -t aaaa google.com
google.com has IPv6 address 2a00:1450:4001:81d::200e
google.com has address 142.250.186.174

I.e., the resolver responds to any request of any type with the same response containing A and AAAA records.

I'm will create a separate issue because from my understanding it's a new not a part of current issue related to UDP/DNS nuances

@strmaks commented on GitHub (Mar 19, 2025): Hello @lixmal thank you very much for your work! Accidentally i'm fully lost all notifications for this issue. I'm tested using the client 0.37.2 version included this fix. Results is very ambiguous. On linux and windows clients it works at first glance. But on mac domain registry.yarnpkg.com still not worked. And i found that with netbird some DNS results is totally wrong. without netbird > [user@localhost ~]$ host google.com > google.com has address 142.250.186.206 > google.com has IPv6 address 2a00:1450:401b:80d::200e > google.com mail is handled by 10 smtp.google.com. with netbird > [user@localhost ~]$ host google.com > google.com has IPv6 address 2a00:1450:4001:81d::200e > google.com has address 142.250.186.142 > google.com has IPv6 address 2a00:1450:4001:81d::200e > google.com has address 142.250.186.142 > google.com has IPv6 address 2a00:1450:4001:81d::200e > google.com has address 142.250.186.142 It seems like the results are repeated 3 times But after a while, I realized the problem was much bigger and deeper. Netbird resolver completely ignores (or forgets) the requested record type. without netbird > [user@localhost ~]$ host -t mx google.com > google.com mail is handled by 10 smtp.google.com. > [user@localhost ~]$ host -t a google.com > google.com has address 172.217.16.14 > [user@localhost ~]$ host -t aaaa google.com > google.com has IPv6 address 2a00:1450:401b:80d::200e with netbird > [user@localhost ~]$ host -t mx google.com > google.com has IPv6 address 2a00:1450:4001:81d::200e > google.com has address 142.250.186.142 > [user@localhost ~]$ host -t a google.com > google.com has IPv6 address 2a00:1450:4001:81d::200e > google.com has address 142.250.186.142 > [user@localhost ~]$ host -t aaaa google.com > google.com has IPv6 address 2a00:1450:4001:81d::200e > google.com has address 142.250.186.174 I.e., the resolver responds to any request of any type with the same response containing A and AAAA records. I'm will create a separate issue because from my understanding it's a new not a part of current issue related to UDP/DNS nuances
saavagebueno commented 2025-11-20 06:04:31 -05:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Mar 19, 2025):

FYI: we have identified an issue with Network Resource of the "domain" type at https://github.com/netbirdio/netbird/issues/3544

@nazarewk commented on GitHub (Mar 19, 2025): FYI: we have identified an issue with Network Resource of the "domain" type at https://github.com/netbirdio/netbird/issues/3544
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label bug dns
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1671
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?