SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-01 02:18:39 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Routes issue after updating to 0.29.0 #1228

New Issue
Closed
opened 2025-11-20 05:26:34 -05:00 by saavagebueno · 4 comments
saavagebueno commented 2025-11-20 05:26:34 -05:00
Owner
Copy Link

Originally created by @MDMeridio001 on GitHub (Sep 10, 2024).

Describe the problem

After updating to version 0.29.0 I noticed an issue with some of the routes. If I disconnect and then reconnect a client, some of the routes stop working. This happens for all my clients, the route is added to the OS routing table but the networks remain unreachable. Oddly enough, it doesn't seem to happen whenever the connection to a routing peer is relayed instead of direct. The only workaround I found is disabling and re-enabling the affected routes either from the dashboard or the clients themselves.
I have also noticed that whenever the issue presents itself, the route for the network has a score of 0 as seen in the "client.log" file:

New chosen route is crg8ap00o87s73dticd0 with peer aRANLusWN6RNPjrWWXCDEe7F70IElX6/lDSCuC4rIwQ= with score 0.000000 for network [10.6.0.0/24]

To Reproduce

Steps to reproduce the behavior:

  1. Disconnect and reconnect a client
  2. Some of the routes stop working

Expected behavior

Network routes shoud be reachable.

Are you using NetBird Cloud?

No, self-hosted

NetBird version

0.29.0

NetBird status -dA output:

Output of one of the clients:

Peers detail:
 client2.netbird.selfhosted:
  NetBird IP: 100.66.0.102
  Public key: Yp3lfeq4HN9mGjI/I7SOf6Bpiyv+mwArcY38fI7fQWk=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 client3.netbird.selfhosted:
  NetBird IP: 100.66.5.21
  Public key: aRANLusWN6RNPjrWWXCDEe7F70IElX6/lDSCuC4rIwQ=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 127.0.0.1:51820/192.168.1.175:51821
  Relay server address: rels://netbird.anon-cU1qT.domain:443
  Last connection update: 13 minutes, 42 seconds ago
  Last WireGuard handshake: 1 minute, 12 seconds ago
  Transfer status (received/sent) 644 B/2.2 KiB
  Quantum resistance: true
  Routes: 10.6.0.0/24
  Latency: 1.7191ms

 client4.netbird.selfhosted:
  NetBird IP: 100.66.41.179
  Public key: 7qIse3SEhrJRl3wpS7X9+woWM4/lM2rD00VAmEasCyM=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 client5.netbird.selfhosted:
  NetBird IP: 100.66.86.62
  Public key: 6y0DD+Ff3E1Op0FtfWvvX3t3JztaQgsu+T7PxHvkln8=
  Status: Disconnected
  -- detail --
  Connection type:
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 client6.netbird.selfhosted:
  NetBird IP: 100.66.157.250
  Public key: h77qUpO0bhXG6GAzuQ3i43PgLVAP5Wa3Y9Sv7gV37WY=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.117.2:51820/198.51.100.0:51820
  Relay server address: rels://netbird.anon-cU1qT.domain:443
  Last connection update: 12 minutes, 26 seconds ago
  Last WireGuard handshake: 32 seconds ago
  Transfer status (received/sent) 26.9 KiB/18.5 KiB
  Quantum resistance: true
  Routes: 192.168.2.0/24
  Latency: 13.0028ms

OS: windows/amd64
Daemon version: 0.29.0
CLI version: 0.29.0
Management: Connected to https://netbird.anon-cU1qT.domain:443
Signal: Connected to https://netbird.anon-cU1qT.domain:443
Relays:
  [stun:netbird.anon-cU1qT.domain:3478] is Available
  [turn:netbird.anon-cU1qT.domain:3478?transport=udp] is Available
  [rels://netbird.anon-cU1qT.domain:443] is Available
Nameservers:
  [100.66.157.250:53] for [.] is Available
FQDN: client1.netbird.selfhosted
NetBird IP: 100.66.30.225/16
Interface type: Userspace
Quantum resistance: true (permissive)
Routes: -
Peers count: 2/5 Connected

Do you face any (non-mobile) client issues?

Yes, here is the output of a Windows client but the same thing also happens on Linux:
netbird.debug.2921127547.zip

Originally created by @MDMeridio001 on GitHub (Sep 10, 2024). **Describe the problem** After updating to version 0.29.0 I noticed an issue with some of the routes. If I disconnect and then reconnect a client, some of the routes stop working. This happens for all my clients, the route is added to the OS routing table but the networks remain unreachable. Oddly enough, it doesn't seem to happen whenever the connection to a routing peer is relayed instead of direct. The only workaround I found is disabling and re-enabling the affected routes either from the dashboard or the clients themselves. I have also noticed that whenever the issue presents itself, the route for the network has a score of 0 as seen in the "client.log" file: ``` New chosen route is crg8ap00o87s73dticd0 with peer aRANLusWN6RNPjrWWXCDEe7F70IElX6/lDSCuC4rIwQ= with score 0.000000 for network [10.6.0.0/24] ``` **To Reproduce** Steps to reproduce the behavior: 1. Disconnect and reconnect a client 2. Some of the routes stop working **Expected behavior** Network routes shoud be reachable. **Are you using NetBird Cloud?** No, self-hosted **NetBird version** 0.29.0 **NetBird status -dA output:** Output of one of the clients: ``` Peers detail: client2.netbird.selfhosted: NetBird IP: 100.66.0.102 Public key: Yp3lfeq4HN9mGjI/I7SOf6Bpiyv+mwArcY38fI7fQWk= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s client3.netbird.selfhosted: NetBird IP: 100.66.5.21 Public key: aRANLusWN6RNPjrWWXCDEe7F70IElX6/lDSCuC4rIwQ= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/host ICE candidate endpoints (Local/Remote): 127.0.0.1:51820/192.168.1.175:51821 Relay server address: rels://netbird.anon-cU1qT.domain:443 Last connection update: 13 minutes, 42 seconds ago Last WireGuard handshake: 1 minute, 12 seconds ago Transfer status (received/sent) 644 B/2.2 KiB Quantum resistance: true Routes: 10.6.0.0/24 Latency: 1.7191ms client4.netbird.selfhosted: NetBird IP: 100.66.41.179 Public key: 7qIse3SEhrJRl3wpS7X9+woWM4/lM2rD00VAmEasCyM= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s client5.netbird.selfhosted: NetBird IP: 100.66.86.62 Public key: 6y0DD+Ff3E1Op0FtfWvvX3t3JztaQgsu+T7PxHvkln8= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s client6.netbird.selfhosted: NetBird IP: 100.66.157.250 Public key: h77qUpO0bhXG6GAzuQ3i43PgLVAP5Wa3Y9Sv7gV37WY= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.117.2:51820/198.51.100.0:51820 Relay server address: rels://netbird.anon-cU1qT.domain:443 Last connection update: 12 minutes, 26 seconds ago Last WireGuard handshake: 32 seconds ago Transfer status (received/sent) 26.9 KiB/18.5 KiB Quantum resistance: true Routes: 192.168.2.0/24 Latency: 13.0028ms OS: windows/amd64 Daemon version: 0.29.0 CLI version: 0.29.0 Management: Connected to https://netbird.anon-cU1qT.domain:443 Signal: Connected to https://netbird.anon-cU1qT.domain:443 Relays: [stun:netbird.anon-cU1qT.domain:3478] is Available [turn:netbird.anon-cU1qT.domain:3478?transport=udp] is Available [rels://netbird.anon-cU1qT.domain:443] is Available Nameservers: [100.66.157.250:53] for [.] is Available FQDN: client1.netbird.selfhosted NetBird IP: 100.66.30.225/16 Interface type: Userspace Quantum resistance: true (permissive) Routes: - Peers count: 2/5 Connected ``` **Do you face any (non-mobile) client issues?** Yes, here is the output of a Windows client but the same thing also happens on Linux: [netbird.debug.2921127547.zip](https://github.com/user-attachments/files/16950018/netbird.debug.2921127547.zip)
saavagebueno added the bugclientroutes labels 2025-11-20 05:26:34 -05:00
saavagebueno commented 2025-11-20 05:26:34 -05:00
Author
Owner
Copy Link

@moserpjm commented on GitHub (Sep 10, 2024):

Same problem here. Router and "client" running Linux.
netbird down, netbird up -> The routes stop working.
Disabling and enabling them again fixes the issue. Also restarting the service helps.
After reading the logs I suspected it has something to do with the new relay feature.

2024-09-10T22:12:55+02:00 DEBG [peer: ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw=] client/internal/peer/worker_relay.go:103: enable WireGuard watcher
2024-09-10T22:12:55+02:00 INFO [peer: ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw=] client/internal/peer/conn.go:573: start to communicate with peer via relay
2024-09-10T22:12:55+02:00 DEBG client/internal/routemanager/client.go:190: triggered route state update for Peer ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw=, state: Connected
2024-09-10T22:12:55+02:00 WARN client/internal/routemanager/client.go:130: peer ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw= has 0 latency
2024-09-10T22:12:55+02:00 INFO client/internal/routemanager/client.go:171: New chosen route is co2sqbl5c20s73a1g6kg:cqapjel5c20s738j02tg with peer ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw= with score 0.000000 for network [10.244.0.0/16]

Turns out I was right. After removing the relay from management.json everything started working normally again. I also noticed that the clients only forget the relay if I restart the service. IDK if this is by design.

@moserpjm commented on GitHub (Sep 10, 2024): Same problem here. Router and "client" running Linux. netbird down, netbird up -> The routes stop working. Disabling and enabling them again fixes the issue. Also restarting the service helps. After reading the logs I suspected it has something to do with the new relay feature. ``` 2024-09-10T22:12:55+02:00 DEBG [peer: ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw=] client/internal/peer/worker_relay.go:103: enable WireGuard watcher 2024-09-10T22:12:55+02:00 INFO [peer: ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw=] client/internal/peer/conn.go:573: start to communicate with peer via relay 2024-09-10T22:12:55+02:00 DEBG client/internal/routemanager/client.go:190: triggered route state update for Peer ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw=, state: Connected 2024-09-10T22:12:55+02:00 WARN client/internal/routemanager/client.go:130: peer ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw= has 0 latency 2024-09-10T22:12:55+02:00 INFO client/internal/routemanager/client.go:171: New chosen route is co2sqbl5c20s73a1g6kg:cqapjel5c20s738j02tg with peer ADVjeYa6yMPg5Lg/Sz+R0Zkh8UoIetI0gtBuRx5YGnw= with score 0.000000 for network [10.244.0.0/16] ``` Turns out I was right. After removing the relay from management.json everything started working normally again. I also noticed that the clients only forget the relay if I restart the service. IDK if this is by design.
saavagebueno commented 2025-11-20 05:26:34 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Sep 11, 2024):

Hey @MDMeridio001 , please test the new 0.29.1 release. You must upgrade your relay and management servers before upgrading the clients if running self-hosted.

@mlsmaycon commented on GitHub (Sep 11, 2024): Hey @MDMeridio001 , please test the new 0.29.1 release. You must upgrade your relay and management servers before upgrading the clients if running self-hosted.
saavagebueno commented 2025-11-20 05:26:35 -05:00
Author
Owner
Copy Link

@MDMeridio001 commented on GitHub (Sep 11, 2024):

@mlsmaycon The issue seems to be resolved completely with version 0.29.1. Thanks!

@MDMeridio001 commented on GitHub (Sep 11, 2024): @mlsmaycon The issue seems to be resolved completely with version 0.29.1. Thanks!
saavagebueno commented 2025-11-20 05:26:35 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Sep 12, 2024):

Great, thanks for the feedback

@mlsmaycon commented on GitHub (Sep 12, 2024): Great, thanks for the feedback
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label bug client routes
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1228
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?