SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-05-27 10:32:35 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Routes disappear after short time #1342

New Issue
Closed
opened 2025-11-20 05:28:43 -05:00 by saavagebueno · 6 comments
saavagebueno commented 2025-11-20 05:28:43 -05:00
Owner
Copy Link

Originally created by @xan-it on GitHub (Oct 15, 2024).

Describe the problem

We use selfhosted netbird installation. We have one PC that is behaving strangely:
After connecting to netbird, all the routes (or the one we select with the UI) are available and listed by the command "route print". We can ping private IPs from the remote site. After a while (<5 min) the routes disappear in the "route print" command and we can't ping the IPs anymore. After a restart of the netbird service the routes reappear for the next short period. A "netbird status -d" shows all remote peers as connected.

To Reproduce

Steps to reproduce the behavior:

  1. connecting to netbird
  2. use command "route print" to see network routes
  3. do a "ping -t" to an IP on the remote site
  4. after a while the ping does not work anymore
  5. check with "route print". The routes are disappeared

Expected behavior

the routes should not disappear after a while.

Are you using NetBird Cloud?

selfhosted version

NetBird version

server 0.29.3
client: we have testet it with 0.29.3 and 0.30.1

NetBird status -dA output:

Peers detail:
dxx-pve1.netbird.selfhosted:
NetBird IP: 100.97.19.116
Public key: Y5PDJ/o0hVszK+fjm9d6P8mhvYz/jTMTn7s6KFtfrnM=
Status: Connected
-- detail --
Connection type: P2P
ICE candidate (Local/Remote): host/srflx
ICE candidate endpoints (Local/Remote): 192.168.21.1:51820/198.51.100.0:51820
Relay server address: rels://netbird.anon-k3E3o.domain:443
Last connection update: 6 minutes, 17 seconds ago
Last WireGuard handshake: 10 seconds ago
Transfer status (received/sent) 17.2 KiB/17.8 KiB
Quantum resistance: false
Routes: 192.168.100.0/24
Latency: 41.0731ms

OS: windows/amd64
Daemon version: 0.30.1
CLI version: 0.30.1
Management: Connected to https://netbird.anon-k3E3o.domain:443
Signal: Connected to https://netbird.anon-k3E3o.domain:443
Relays:
[stun:netbird.anon-k3E3o.domain:3478] is Available
[turn:netbird.anon-k3E3o.domain:3478?transport=udp] is Available
[rels://netbird.anon-k3E3o.domain:443] is Available
Nameservers:
[192.168.100.102:53] for [anon-2wYrW.domain, domapor.k.anon-2wYrW.domain] is Available
FQDN: wh-pc633.netbird.selfhosted
NetBird IP: 100.97.138.222/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 1/1 Connected

Do you face any (non-mobile) client issues?

here are the relevant part of the debug log:
2024-10-16T12:04:49+02:00 INFO client/internal/dns/host_windows.go:149: added 3 match domains to the state. Domain list: [.anon-Ws4WY.domain .domapor.k.anon-Ws4WY.domain .netbird.selfhosted]
2024-10-16T12:04:49+02:00 INFO client/internal/dns/host_windows.go:176: updated the search domains in the registry with 1 domains. Domain list: [netbird.selfhosted]
2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface
2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface
2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface
2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface
2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface
2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface
2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface
2024-10-16T12:09:47+02:00 DEBG client/internal/peer/conn_monitor.go:131: Gathering ICE candidates
2024-10-16T12:09:47+02:00 DEBG client/internal/engine.go:1308: received management probe request, healthy: true

Screenshots

Additional context

Originally created by @xan-it on GitHub (Oct 15, 2024). **Describe the problem** We use selfhosted netbird installation. We have one PC that is behaving strangely: After connecting to netbird, all the routes (or the one we select with the UI) are available and listed by the command "route print". We can ping private IPs from the remote site. After a while (<5 min) the routes disappear in the "route print" command and we can't ping the IPs anymore. After a restart of the netbird service the routes reappear for the next short period. A "netbird status -d" shows all remote peers as connected. **To Reproduce** Steps to reproduce the behavior: 1. connecting to netbird 2. use command "route print" to see network routes 3. do a "ping -t" to an IP on the remote site 4. after a while the ping does not work anymore 5. check with "route print". The routes are disappeared **Expected behavior** the routes should not disappear after a while. **Are you using NetBird Cloud?** selfhosted version **NetBird version** server `0.29.3` client: we have testet it with 0.29.3 and 0.30.1 **NetBird status -dA output:** Peers detail: dxx-pve1.netbird.selfhosted: NetBird IP: 100.97.19.116 Public key: Y5PDJ/o0hVszK+fjm9d6P8mhvYz/jTMTn7s6KFtfrnM= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.21.1:51820/198.51.100.0:51820 Relay server address: rels://netbird.anon-k3E3o.domain:443 Last connection update: 6 minutes, 17 seconds ago Last WireGuard handshake: 10 seconds ago Transfer status (received/sent) 17.2 KiB/17.8 KiB Quantum resistance: false Routes: 192.168.100.0/24 Latency: 41.0731ms OS: windows/amd64 Daemon version: 0.30.1 CLI version: 0.30.1 Management: Connected to https://netbird.anon-k3E3o.domain:443 Signal: Connected to https://netbird.anon-k3E3o.domain:443 Relays: [stun:netbird.anon-k3E3o.domain:3478] is Available [turn:netbird.anon-k3E3o.domain:3478?transport=udp] is Available [rels://netbird.anon-k3E3o.domain:443] is Available Nameservers: [192.168.100.102:53] for [anon-2wYrW.domain, domapor.k.anon-2wYrW.domain] is Available FQDN: wh-pc633.netbird.selfhosted NetBird IP: 100.97.138.222/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 1/1 Connected **Do you face any (non-mobile) client issues?** here are the relevant part of the debug log: 2024-10-16T12:04:49+02:00 INFO client/internal/dns/host_windows.go:149: added 3 match domains to the state. Domain list: [.anon-Ws4WY.domain .domapor.k.anon-Ws4WY.domain .netbird.selfhosted] 2024-10-16T12:04:49+02:00 INFO client/internal/dns/host_windows.go:176: updated the search domains in the registry with 1 domains. Domain list: [netbird.selfhosted] 2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface 2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface 2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface 2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface 2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface 2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface 2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface 2024-10-16T12:09:47+02:00 DEBG client/internal/peer/conn_monitor.go:131: Gathering ICE candidates 2024-10-16T12:09:47+02:00 DEBG client/internal/engine.go:1308: received management probe request, healthy: true **Screenshots** **Additional context**
saavagebueno added the waiting-feedbacktriage-needed labels 2025-11-20 05:28:43 -05:00
saavagebueno commented 2025-11-20 05:28:43 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Nov 5, 2024):

@xan-it can you test with 0.28.9 version? Here is the link for the installer: https://github.com/netbirdio/netbird/releases/download/v0.28.8/netbird_installer_0.28.8_windows_amd64.exe

@mlsmaycon commented on GitHub (Nov 5, 2024): @xan-it can you test with 0.28.9 version? Here is the link for the installer: https://github.com/netbirdio/netbird/releases/download/v0.28.8/netbird_installer_0.28.8_windows_amd64.exe
saavagebueno commented 2025-11-20 05:28:44 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Nov 11, 2024):

HI @xan-it,

could you test the latest release https://github.com/netbirdio/netbird/releases/tag/v0.31.1 please?
If the issue still occurs:

  • please provide the log file, e.g. through netbird debug bundle -A (the attached logs are irrelevant to the observed issue).
  • is there anything on the system that could have deleted the routes?
@lixmal commented on GitHub (Nov 11, 2024): HI @xan-it, could you test the latest release https://github.com/netbirdio/netbird/releases/tag/v0.31.1 please? If the issue still occurs: - please provide the log file, e.g. through `netbird debug bundle -A` (the attached logs are irrelevant to the observed issue). - is there anything on the system that could have deleted the routes?
saavagebueno commented 2025-11-20 05:28:44 -05:00
Author
Owner
Copy Link

@xan-it commented on GitHub (Nov 21, 2024):

Hi @mlsmaycon, hi @lixmal,
we have tried the versions 0.28.9, 0.31.0, 0.31.1 and 0.32.0. These versions also let routes disappearing after a while. We have manually added routes to other network adapters in the system. This routes were not automatically removed after the netbird routes are gone. So we don't think there is a process which kills routes.
I have added a new logfile bundle.
netbird.debug.3741012979.zip

@xan-it commented on GitHub (Nov 21, 2024): Hi @mlsmaycon, hi @lixmal, we have tried the versions 0.28.9, 0.31.0, 0.31.1 and 0.32.0. These versions also let routes disappearing after a while. We have manually added routes to other network adapters in the system. This routes were not automatically removed after the netbird routes are gone. So we don't think there is a process which kills routes. I have added a new logfile bundle. [netbird.debug.3741012979.zip](https://github.com/user-attachments/files/17846844/netbird.debug.3741012979.zip)
saavagebueno commented 2025-11-20 05:28:44 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Nov 21, 2024):

Hi @xan-it,

thanks for the insight. I've taken a look at the log, there's nothing out of the ordinary.
However, we can increase the log level, this will show us if any route removal function was called.

Can you please run the following in an elevated shell:

netbird service stop
netbird service uninstall
netbird service install --log-level trace
netbird service start

and then share netbird bundle -AS when the issue appears.

If you are certain you can reproduce the issue within a given time frame (you mentioned 5 minutes) you can alternatively run

netbird debug for 5m -AS

Edit:

I've noticed our wireguard interface is not in the list.

For context:

These log messages:

2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface

indicate that we received a route change from windows, but the message included an interface that doesn't exist anymore. If that's our wireguard interface that was somehow removed then it would also explain the automatic disappearance of routes pointing to it.

There's a Talk2m-eCatcher interface in the list, which appears to be another VPN client. Could it be responsible?

@lixmal commented on GitHub (Nov 21, 2024): Hi @xan-it, thanks for the insight. I've taken a look at the log, there's nothing out of the ordinary. However, we can increase the log level, this will show us if any route removal function was called. Can you please run the following in an elevated shell: ``` netbird service stop netbird service uninstall netbird service install --log-level trace netbird service start ``` and then share `netbird bundle -AS` when the issue appears. If you are certain you can reproduce the issue within a given time frame (you mentioned 5 minutes) you can alternatively run ``` netbird debug for 5m -AS ``` Edit: I've noticed our wireguard interface is not in the list. For context: These log messages: > 2024-10-16T12:08:15+02:00 ERRO client/internal/routemanager/systemops/systemops_windows.go:198: Failed to parse route update: get interface name: route ip+net: no such network interface indicate that we received a route change from windows, but the message included an interface that doesn't exist anymore. If that's our wireguard interface that was somehow removed then it would also explain the automatic disappearance of routes pointing to it. There's a `Talk2m-eCatcher` interface in the list, which appears to be another VPN client. Could it be responsible?
saavagebueno commented 2025-11-20 05:28:44 -05:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Apr 28, 2025):

Hello @xan-it,

We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.

Could you please confirm if the issue is still there?

We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.

Thanks for your contribution to improving the project!

@nazarewk commented on GitHub (Apr 28, 2025): Hello @xan-it, We're currently reviewing our open issues and would like to verify if this problem still exists in the [latest NetBird version](https://github.com/netbirdio/netbird/releases). Could you please confirm if the issue is still there? We may close this issue temporarily if we don't hear back from you within **2 weeks**, but feel free to reopen it with updated information. Thanks for your contribution to improving the project!
saavagebueno commented 2025-11-20 05:28:44 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Jun 1, 2025):

closing issue due to no recent feedback. Feel free to open a new one if the issue persist or reopen if this was a feature request.

@mlsmaycon commented on GitHub (Jun 1, 2025): closing issue due to no recent feedback. Feel free to open a new one if the issue persist or reopen if this was a feature request.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label triage-needed waiting-feedback
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1342
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?