SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-06-10 09:22:09 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Routing peer broken from 0.40.0 onwards #1899

New Issue
Closed
opened 2025-11-20 06:08:48 -05:00 by saavagebueno · 20 comments
saavagebueno commented 2025-11-20 06:08:48 -05:00
Owner
Copy Link

Originally created by @drewhemm on GitHub (May 19, 2025).

Describe the problem

Using any version higher than 0.40.0 for a routing peer prevents access to any networks routed via that peer. The software version for 'client' peers is not a factor. I am running 0.44.0 on the client peer.

Networks are completely inaccessible by all protocols.

To Reproduce

Steps to reproduce the behavior:

  1. Create a routing peer with Netbird 0.39.2
  2. Connect via a client peer
  3. Start a ping from the client to an IP routed via the routing peer
  4. Upgrade the routing peer to 0.40.0 (or any newer version)
  5. Ping will start to time out
  6. Downgrade the routing peer to 0.39.2 (or any earlier version)
  7. Ping will resume

Expected behavior

Network connectivity should not be broken, as there are no breaking changes announced in 0.40.0

Are you using NetBird Cloud?

No, I am using self-hosted Netbird.

NetBird version

As described above, 0.39.2 is okay, 0.40.0 and newer are broken.

Is any other VPN software installed?

Yes, but not active during testing. Wireguard client. Draytek Smart VPN.

Debug output

I will attach the output of netbird status -dA and netbird debug for 1m -AS shortly.

Additional context

Have you tried these troubleshooting steps?

  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @drewhemm on GitHub (May 19, 2025). **Describe the problem** Using any version higher than 0.40.0 for a routing peer prevents access to any networks routed via that peer. The software version for 'client' peers is not a factor. I am running 0.44.0 on the client peer. Networks are completely inaccessible by all protocols. **To Reproduce** Steps to reproduce the behavior: 1. Create a routing peer with Netbird 0.39.2 2. Connect via a client peer 3. Start a ping from the client to an IP routed via the routing peer 4. Upgrade the routing peer to 0.40.0 (or any newer version) 5. Ping will start to time out 6. Downgrade the routing peer to 0.39.2 (or any earlier version) 7. Ping will resume **Expected behavior** Network connectivity should not be broken, as there are no breaking changes announced in [0.40.0](https://github.com/netbirdio/netbird/releases/tag/v0.40.0) **Are you using NetBird Cloud?** No, I am using self-hosted Netbird. **NetBird version** As described above, [0.39.2](https://github.com/netbirdio/netbird/releases/tag/v0.39.2) is okay, [0.40.0](https://github.com/netbirdio/netbird/releases/tag/v0.40.0) and newer are broken. **Is any other VPN software installed?** Yes, but not active during testing. Wireguard client. Draytek Smart VPN. **Debug output** I will attach the output of `netbird status -dA` and `netbird debug for 1m -AS` shortly. **Additional context** **Have you tried these troubleshooting steps?** - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2025-11-20 06:08:48 -05:00
saavagebueno commented 2025-11-20 06:08:49 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 19, 2025):

trouble started after upgrade from 0.39.2.

symptoms both for iptables and nftables: routes missing in mangle forward tables after upgrade

nftables:
chain netbird-rt-fwd {
        ct state established,related counter packets 2173 bytes 501199 accept
}

iptables:

*filter
:INPUT ACCEPT [68048:11434405]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [59536:9294961]
:NETBIRD-ACL-INPUT - [0:0]
:NETBIRD-RT-FWD-IN - [0:0]
:NETBIRD-RT-FWD-OUT - [0:0]
-A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wt0 -j NETBIRD-ACL-INPUT
-A INPUT -i wt0 -j DROP
-A FORWARD -i wt0 -j NETBIRD-RT-FWD-IN
-A FORWARD -m mark --mark 0x1bd20 -j ACCEPT
-A FORWARD -o wt0 -j NETBIRD-RT-FWD-OUT
-A FORWARD -i wt0 -j DROP
-A NETBIRD-ACL-INPUT -j ACCEPT
-A NETBIRD-RT-FWD-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A NETBIRD-RT-FWD-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT

up to 0.39.2 everything is good

iptables:

# Completed on Mon May 19 19:42:35 2025
# Generated by iptables-save v1.8.10 on Mon May 19 19:42:35 2025
*filter
:INPUT ACCEPT [184193:53467170]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [160221:41665270]
:NETBIRD-ACL-INPUT - [0:0]
:NETBIRD-RT-FWD-IN - [0:0]
:NETBIRD-RT-FWD-OUT - [0:0]
-A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wt0 -j NETBIRD-ACL-INPUT
-A INPUT -i wt0 -j DROP
-A FORWARD -i wt0 -j NETBIRD-RT-FWD-IN
-A FORWARD -m mark --mark 0x1bd01 -j ACCEPT
-A FORWARD -o wt0 -j NETBIRD-RT-FWD-OUT
-A FORWARD -i wt0 -j DROP
-A NETBIRD-ACL-INPUT -j ACCEPT
-A NETBIRD-RT-FWD-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A NETBIRD-RT-FWD-IN -d 1.2.3.4/24 -j ACCEPT
-A NETBIRD-RT-FWD-IN -d 5.6.7.8/24 -j ACCEPT
...
-A NETBIRD-RT-FWD-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT

nftables:

chain netbird-rt-fwd {
        ct state established,related counter packets 28739 bytes 96
        ip daddr 1.2.3.0/24 counter packets 0 bytes 0 accept
        ip daddr 4.5.6.0/24 counter packets 0 bytes 0 accept
        ....
}
@bravosierrasierra commented on GitHub (May 19, 2025): trouble started after upgrade from 0.39.2. symptoms both for iptables and nftables: routes missing in mangle forward tables after upgrade ``` nftables: chain netbird-rt-fwd { ct state established,related counter packets 2173 bytes 501199 accept } ``` iptables: ``` *filter :INPUT ACCEPT [68048:11434405] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [59536:9294961] :NETBIRD-ACL-INPUT - [0:0] :NETBIRD-RT-FWD-IN - [0:0] :NETBIRD-RT-FWD-OUT - [0:0] -A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A INPUT -i wt0 -j NETBIRD-ACL-INPUT -A INPUT -i wt0 -j DROP -A FORWARD -i wt0 -j NETBIRD-RT-FWD-IN -A FORWARD -m mark --mark 0x1bd20 -j ACCEPT -A FORWARD -o wt0 -j NETBIRD-RT-FWD-OUT -A FORWARD -i wt0 -j DROP -A NETBIRD-ACL-INPUT -j ACCEPT -A NETBIRD-RT-FWD-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A NETBIRD-RT-FWD-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT COMMIT ``` up to 0.39.2 everything is good iptables: ``` # Completed on Mon May 19 19:42:35 2025 # Generated by iptables-save v1.8.10 on Mon May 19 19:42:35 2025 *filter :INPUT ACCEPT [184193:53467170] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [160221:41665270] :NETBIRD-ACL-INPUT - [0:0] :NETBIRD-RT-FWD-IN - [0:0] :NETBIRD-RT-FWD-OUT - [0:0] -A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A INPUT -i wt0 -j NETBIRD-ACL-INPUT -A INPUT -i wt0 -j DROP -A FORWARD -i wt0 -j NETBIRD-RT-FWD-IN -A FORWARD -m mark --mark 0x1bd01 -j ACCEPT -A FORWARD -o wt0 -j NETBIRD-RT-FWD-OUT -A FORWARD -i wt0 -j DROP -A NETBIRD-ACL-INPUT -j ACCEPT -A NETBIRD-RT-FWD-IN -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A NETBIRD-RT-FWD-IN -d 1.2.3.4/24 -j ACCEPT -A NETBIRD-RT-FWD-IN -d 5.6.7.8/24 -j ACCEPT ... -A NETBIRD-RT-FWD-OUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT COMMIT ``` nftables: ``` chain netbird-rt-fwd { ct state established,related counter packets 28739 bytes 96 ip daddr 1.2.3.0/24 counter packets 0 bytes 0 accept ip daddr 4.5.6.0/24 counter packets 0 bytes 0 accept .... } ```
saavagebueno commented 2025-11-20 06:08:49 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (May 19, 2025):

@bravosierrasierra, can you share your logs, the distribution and kernel version please?

@lixmal commented on GitHub (May 19, 2025): @bravosierrasierra, can you [share your logs](https://docs.netbird.io/how-to/troubleshooting-client#debug-bundle), the distribution and kernel version please?
saavagebueno commented 2025-11-20 06:08:49 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 20, 2025):

Ubuntu 24.04.2 LTS

# uname -a
Linux netbird-platform-gw-dev-a-01 6.8.0-57-generic #59-Ubuntu SMP PREEMPT_DYNAMIC Sat Mar 15 17:40:59 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
# cat /etc/issue
Ubuntu 24.04.2 LTS \n \l

anonymous logs contains too many sensitive information about our network on 10/8 address space to place it to public space.

@bravosierrasierra commented on GitHub (May 20, 2025): Ubuntu 24.04.2 LTS ``` # uname -a Linux netbird-platform-gw-dev-a-01 6.8.0-57-generic #59-Ubuntu SMP PREEMPT_DYNAMIC Sat Mar 15 17:40:59 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux # cat /etc/issue Ubuntu 24.04.2 LTS \n \l ``` anonymous logs contains too many sensitive information about our network on 10/8 address space to place it to public space.
saavagebueno commented 2025-11-20 06:08:49 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (May 20, 2025):

@bravosierrasierra
You can upload it directly instead: https://docs.netbird.io/how-to/troubleshooting-client#debug-bundle-uploads.
Best to use netbird debug for 1m -AUS, so we get trace level logs.

@lixmal commented on GitHub (May 20, 2025): @bravosierrasierra You can upload it directly instead: https://docs.netbird.io/how-to/troubleshooting-client#debug-bundle-uploads. Best to use `netbird debug for 1m -AUS`, so we get trace level logs.
saavagebueno commented 2025-11-20 06:08:49 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 20, 2025):

Upload file key:
ee8a9fe4673b82224a70a42cda2fd951120f718ac9b2e3554107e0a19973daf3/b7ac9adf-58ab-4103-bc89-dadcd259d6d8

@bravosierrasierra commented on GitHub (May 20, 2025): Upload file key: ee8a9fe4673b82224a70a42cda2fd951120f718ac9b2e3554107e0a19973daf3/b7ac9adf-58ab-4103-bc89-dadcd259d6d8
saavagebueno commented 2025-11-20 06:08:49 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (May 20, 2025):

Thanks @bravosierrasierra,

the received network map doesn't show any routed rules at all, so the peer doesn't set them up:

  "routesFirewallRules":  [],
  "routesFirewallRulesIsEmpty":  false,

Can you confirm the version on the management server?
Have you changed the version on the management at all, or is 0.39.2 -> 0.40.0 on the client all it takes to break?

@lixmal commented on GitHub (May 20, 2025): Thanks @bravosierrasierra, the received network map doesn't show any routed rules at all, so the peer doesn't set them up: ``` "routesFirewallRules": [], "routesFirewallRulesIsEmpty": false, ``` Can you confirm the version on the management server? Have you changed the version on the management at all, or is `0.39.2` -> `0.40.0` on the client all it takes to break?
saavagebueno commented 2025-11-20 06:08:50 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 20, 2025):

Management: netbirdio/management:0.29.2
Upgrade to any version after 0.39.2 breaks all on this client.

@bravosierrasierra commented on GitHub (May 20, 2025): Management: netbirdio/management:0.29.2 Upgrade to any version after 0.39.2 breaks all on this client.
saavagebueno commented 2025-11-20 06:08:50 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 20, 2025):

@bravosierrasierra to confirm you and @drewhemm have the same deployment or he have a different case?

@mlsmaycon commented on GitHub (May 20, 2025): @bravosierrasierra to confirm you and @drewhemm have the same deployment or he have a different case?
saavagebueno commented 2025-11-20 06:08:50 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 20, 2025):

@mlsmaycon No, we have different installations.

@bravosierrasierra commented on GitHub (May 20, 2025): @mlsmaycon No, we have different installations.
saavagebueno commented 2025-11-20 06:08:50 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 20, 2025):

Got it.

Could you please downgraded the client to 0.39.2 and run:

netbird debug bundle -S

Then share the file with us? This version doesn't have upload.

We want to confirm something.

@mlsmaycon commented on GitHub (May 20, 2025): Got it. Could you please downgraded the client to 0.39.2 and run: netbird debug bundle -S Then share the file with us? This version doesn't have upload. We want to confirm something.
saavagebueno commented 2025-11-20 06:08:50 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 20, 2025):

@drewhemm please go over the steps from previous messages. Sharing debug bundle from both versions will be very helpful

@mlsmaycon commented on GitHub (May 20, 2025): @drewhemm please go over the steps from previous messages. Sharing debug bundle from both versions will be very helpful
saavagebueno commented 2025-11-20 06:08:50 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 20, 2025):

@mlsmaycon "Error: unknown shorthand flag: 'U' in -US" when call "netbird debug for 1m -AUS"
:(

@bravosierrasierra commented on GitHub (May 20, 2025): @mlsmaycon "Error: unknown shorthand flag: 'U' in -US" when call "netbird debug for 1m -AUS" :(
saavagebueno commented 2025-11-20 06:08:51 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 20, 2025):

You can without it and only the bundle command:

netbird debug bundle -S

Feel free to share the file via slack. Just look for mlsmaycon there

@mlsmaycon commented on GitHub (May 20, 2025): You can without it and only the bundle command: `netbird debug bundle -S` Feel free to share the file via slack. Just look for mlsmaycon there
saavagebueno commented 2025-11-20 06:08:51 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 20, 2025):

@mlsmaycon but where i should place all sensitive data about our network in 10/8 address space? Github issue is not good place for it

@bravosierrasierra commented on GitHub (May 20, 2025): @mlsmaycon but where i should place all sensitive data about our network in 10/8 address space? Github issue is not good place for it
saavagebueno commented 2025-11-20 06:08:51 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 20, 2025):

@mlsmaycon but where i should place all sensitive data about our network in 10/8 address space? Github issue is not good place for it

You can share with us via slack, look for mlsmaycon or via support@netbird.io

@mlsmaycon commented on GitHub (May 20, 2025): > @mlsmaycon but where i should place all sensitive data about our network in 10/8 address space? Github issue is not good place for it You can share with us via slack, look for mlsmaycon or via support@netbird.io
saavagebueno commented 2025-11-20 06:08:51 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 20, 2025):

@mlsmaycon succesfully sent via slack, thank you

@bravosierrasierra commented on GitHub (May 20, 2025): @mlsmaycon succesfully sent via slack, thank you
saavagebueno commented 2025-11-20 06:08:51 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 20, 2025):

@mlsmaycon give a solution: upgrade management version to 0.30.3 and dashboard to v2.6.2. Problem seems to be gone. We plan full upgrade to fresh versions

@mlsmaycon Thank you for your excellent product

@bravosierrasierra commented on GitHub (May 20, 2025): @mlsmaycon give a solution: upgrade management version to 0.30.3 and dashboard to v2.6.2. Problem seems to be gone. We plan full upgrade to fresh versions @mlsmaycon Thank you for your excellent product
saavagebueno commented 2025-11-20 06:08:51 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 20, 2025):

That's great @bravosierrasierra . Please check the change log of the newer versions since a few things will change, like, setup keys store and some of the dashboard versions will change too.

check both:

https://github.com/netbirdio/netbird/releases
https://github.com/netbirdio/dashboard/releases

@mlsmaycon commented on GitHub (May 20, 2025): That's great @bravosierrasierra . Please check the change log of the newer versions since a few things will change, like, setup keys store and some of the dashboard versions will change too. check both: https://github.com/netbirdio/netbird/releases https://github.com/netbirdio/dashboard/releases
saavagebueno commented 2025-11-20 06:08:52 -05:00
Author
Owner
Copy Link

@bravosierrasierra commented on GitHub (May 20, 2025):

Updates: users found troubles with login on 0.30.3/v2.6.2. Management upgraded to latest versions and problems is gone. Checking...

@bravosierrasierra commented on GitHub (May 20, 2025): Updates: users found troubles with login on 0.30.3/v2.6.2. Management upgraded to latest versions and problems is gone. Checking...
saavagebueno commented 2025-11-20 06:08:52 -05:00
Author
Owner
Copy Link

@drewhemm commented on GitHub (May 27, 2025):

I thought I had upgraded my Management instance as I had pulled the latest images and ran docker compose restart, but I then realised that a restart is not sufficient to replace the containers. I am now running the newer version. I have also moved to Networks instead of Network Routes and can confirm the issues have all gone away with the latest peer version.

@drewhemm commented on GitHub (May 27, 2025): I thought I had upgraded my Management instance as I had pulled the latest images and ran `docker compose restart`, but I then realised that a restart is not sufficient to replace the containers. I am now running the newer version. I have also moved to Networks instead of Network Routes and can confirm the issues have all gone away with the latest peer version.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1899
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?