DYNR/netbird
2
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-07 02:29:06 -04:00
Code Issues 23.5k Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #3602] Newer NetBird causes traffic to fail when Linux system has VRFs configured #7211

New Issue
Open
opened 2026-08-05 01:12:02 -04:00 by saavagebueno · 10 comments
saavagebueno commented 2026-08-05 01:12:02 -04:00
Owner
Copy Link

Originally created by @l0crian1 on GitHub (Mar 28, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/3602

Describe the problem

I previously created this solution: https://lev-0.com/2024/03/11/scalable-and-secure-vxlan-multisite-using-netbird-part-1initial-config-l3vpn/

The solution does not work with newer NetBird versions.

I am using NetBird as a container on VyOS. When configuring a VRF, NetBird causes all traffic to cease. You're not even able to ping local IPs on the host itself. I checked previous versions of NetBird, and it works on 0.25.9, but seems to fail beyond that.

To recover the system, I can either disable the NetBird container, or delete the VRF. I'm unsure of what NetBird is doing that would impact that behavior.

To Reproduce

Install latest NetBird on VyOS, and configure a VRF:

set vrf name test table '5001'

Expected behavior

NetBird should not break a host's routing when VRFs are configured.

Are you using NetBird Cloud?

No

NetBird version

Breaking: 0.39.2
Working: 0.25.9

Is any other VPN software installed?

No

Debug output

NOTE: Both debug commands fail whether in the working state or the broken state.

To help us resolve the problem, please attach the following debug output

/ # netbird status -dA
Error: failed to connect to daemon error: context deadline exceeded
If the daemon is not running please run:
netbird service install
netbird service start

As well as the file created by

/ # netbird debug for 1m -AS
Error: failed to connect to daemon error: context deadline exceeded
If the daemon is not running please run:
netbird service install
netbird service start

Have you tried these troubleshooting steps?

  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
    I disabled the firewall entirely, including fully flushing the nft ruleset
Originally created by @l0crian1 on GitHub (Mar 28, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/3602 **Describe the problem** I previously created this solution: https://lev-0.com/2024/03/11/scalable-and-secure-vxlan-multisite-using-netbird-part-1initial-config-l3vpn/ The solution does not work with newer NetBird versions. I am using NetBird as a container on VyOS. When configuring a VRF, NetBird causes all traffic to cease. You're not even able to ping local IPs on the host itself. I checked previous versions of NetBird, and it works on 0.25.9, but seems to fail beyond that. To recover the system, I can either disable the NetBird container, or delete the VRF. I'm unsure of what NetBird is doing that would impact that behavior. **To Reproduce** Install latest NetBird on VyOS, and configure a VRF: ``` set vrf name test table '5001' ``` **Expected behavior** NetBird should not break a host's routing when VRFs are configured. **Are you using NetBird Cloud?** No **NetBird version** `Breaking: 0.39.2` `Working: 0.25.9` **Is any other VPN software installed?** No **Debug output** NOTE: Both debug commands fail whether in the working state or the broken state. To help us resolve the problem, please attach the following debug output ``` / # netbird status -dA Error: failed to connect to daemon error: context deadline exceeded If the daemon is not running please run: netbird service install netbird service start ``` As well as the file created by ``` / # netbird debug for 1m -AS Error: failed to connect to daemon error: context deadline exceeded If the daemon is not running please run: netbird service install netbird service start ``` **Have you tried these troubleshooting steps?** - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [ ] Disabled other VPN software - [x] Checked firewall settings I disabled the firewall entirely, including fully flushing the nft ruleset
saavagebueno added the triage-needed label 2026-08-05 01:12:02 -04:00
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Mar 29, 2025):

Hello @l0crian1 it seems like the daemon is not running. Does VyOS has systemD? if so, can you share the logs from:

/var/log/netbird/client.log
/var/log/netbird/netbird.err

<!-- gh-comment-id:2763275604 --> @mlsmaycon commented on GitHub (Mar 29, 2025): Hello @l0crian1 it seems like the daemon is not running. Does VyOS has systemD? if so, can you share the logs from: /var/log/netbird/client.log /var/log/netbird/netbird.err
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@l0crian1 commented on GitHub (Mar 29, 2025):

Hello @l0crian1 it seems like the daemon is not running. Does VyOS has systemD? if so, can you share the logs from:

/var/log/netbird/client.log /var/log/netbird/netbird.err

I can't pull those directly. Here is the log from the container. One thing to note, is even when it's working, that debug says the daemon is down, so that seems to be a red herring:

# None Working State (with VRF configured)

2025-03-29T15:42:54Z INFO client/internal/connect.go:122: starting NetBird client version 0.39.2 on linux/amd64
2025-03-29T15:42:54Z INFO util/net/env_linux.go:70: system supports advanced routing
2025-03-29T15:42:54Z INFO client/internal/connect.go:255: connecting to the Relay service(s): rels://relay.netbird.io:443
2025-03-29T15:42:54Z INFO relay/client/picker.go:72: try to connecting to relay server: rels://relay.netbird.io:443
2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:164: create new relay connection: local peerID: <redacted>=, local peer hashedID: sha-<redacted>=
2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:170: connecting to relay server
2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via quic
2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via WS
2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:89: successfully dialed via: WS
2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:75: connection attempt aborted via: quic
2025-03-29T15:42:54Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:186: relay connection established
2025-03-29T15:42:54Z INFO relay/client/picker.go:90: connected to Relay server: rels://relay.netbird.io:443
2025-03-29T15:42:54Z INFO relay/client/picker.go:64: chosen home Relay server: rels://relay.netbird.io:443
2025-03-29T15:42:54Z INFO client/iface/wgproxy/factory_kernel.go:25: WireGuard Proxy Factory will produce UDP proxy
2025-03-29T15:42:54Z WARN client/iface/wgproxy/factory_kernel.go:26: failed to initialize ebpf proxy, fallback to user space proxy: failed to set memlock rlimit: operation not permitted
2025-03-29T15:42:54Z ERRO client/internal/routemanager/systemops/systemops_linux.go:102: Error setting up sysctl: 1 error occurred:
        * write sysctl net.ipv4.conf.all.src_valid_mark: open /proc/sys/net/ipv4/conf/all/src_valid_mark: read-only file system
2025-03-29T15:42:54Z INFO client/internal/routemanager/manager.go:193: Routing setup complete
2025-03-29T15:42:54Z INFO client/internal/engine.go:459: firewall is disabled
2025-03-29T15:42:54Z INFO client/internal/dns/host_unix.go:54: System DNS manager discovered: file
2025-03-29T15:43:29Z ERRO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:474: health check timeout
2025-03-29T15:43:29Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:336: start to Relay read loop exit
2025-03-29T15:43:29Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:530: closing all peer connections
2025-03-29T15:43:29Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:538: waiting for read loop to close
2025-03-29T15:43:29Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:540: relay connection closed
2025-03-29T15:43:31Z INFO relay/client/guard.go:81: try to reconnect to Relay server: rels://relay.netbird.io:443
2025-03-29T15:43:31Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:170: connecting to relay server
2025-03-29T15:43:31Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via quic
2025-03-29T15:43:31Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via WS
2025-03-29T15:43:34Z WARN signal/client/grpc.go:143: disconnected from the Signal Exchange due to an error: didn't receive a registration header from the Signal server whille connecting to the streams
2025-03-29T15:43:41Z ERRO relay/client/dialer/ws/ws.go:50: failed to dial to Relay server 'wss://relay.netbird.io:443': failed to WebSocket dial: failed to send handshake request: Get "https://relay.netbird.io:443/relay": d.Dialer.DialContext: dial tcp: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:33119->4.2.2.2:53: i/o timeout
2025-03-29T15:43:41Z ERRO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/dialer/race_dialer.go:77: failed to dial via WS: failed to WebSocket dial: failed to send handshake request: Get "https://relay.netbird.io:443/relay": d.Dialer.DialContext: dial tcp: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:33119->4.2.2.2:53: i/o timeout
2025-03-29T15:43:41Z ERRO relay/client/dialer/quic/quic.go:46: failed to resolve UDP address: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:39559->4.2.2.2:53: i/o timeout
2025-03-29T15:43:41Z ERRO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/dialer/race_dialer.go:77: failed to dial via quic: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:39559->4.2.2.2:53: i/o timeout
2025-03-29T15:43:41Z ERRO relay/client/guard.go:84: failed to reconnect to relay server: failed to dial to Relay server on any protocol
2025-03-29T15:43:41Z INFO relay/client/guard.go:91: try to pick up a new Relay server
2025-03-29T15:43:41Z INFO relay/client/picker.go:72: try to connecting to relay server: rels://relay.netbird.io:443
2025-03-29T15:43:41Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:164: create new relay connection: local peerID: <redacted>=, local peer hashedID: sha-<redacted>=
2025-03-29T15:43:41Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:170: connecting to relay server
2025-03-29T15:43:41Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via quic
2025-03-29T15:43:41Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via WS
2025-03-29T15:43:45Z ERRO util/grpc/dialer.go:45: Failed to dial: d.Dialer.DialContext: dial tcp: lookup signal.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:49247->4.2.2.2:53: i/o timeout
2025-03-29T15:43:51Z ERRO relay/client/dialer/quic/quic.go:46: failed to resolve UDP address: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:43302->4.2.2.2:53: i/o timeout
2025-03-29T15:43:51Z ERRO relay/client/dialer/ws/ws.go:50: failed to dial to Relay server 'wss://relay.netbird.io:443': failed to WebSocket dial: failed to send handshake request: Get "https://relay.netbird.io:443/relay": d.Dialer.DialContext: dial tcp: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:34450->4.2.2.2:53: i/o timeout
2025-03-29T15:43:51Z ERRO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:77: failed to dial via quic: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:43302->4.2.2.2:53: i/o timeout
2025-03-29T15:43:51Z ERRO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:77: failed to dial via WS: failed to WebSocket dial: failed to send handshake request: Get "https://relay.netbird.io:443/relay": d.Dialer.DialContext: dial tcp: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:34450->4.2.2.2:53: i/o timeout
2025-03-29T15:43:51Z ERRO relay/client/guard.go:58: failed to pick new Relay server: failed to connect to any relay server: all attempts failed

# Working State (with VRF removed)

2025-03-29T15:51:59Z INFO client/internal/connect.go:122: starting NetBird client version 0.39.2 on linux/amd64
2025-03-29T15:51:59Z INFO util/net/env_linux.go:70: system supports advanced routing
2025-03-29T15:51:59Z INFO client/internal/connect.go:255: connecting to the Relay service(s): rels://relay.netbird.io:443
2025-03-29T15:51:59Z INFO relay/client/picker.go:72: try to connecting to relay server: rels://relay.netbird.io:443
2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:164: create new relay connection: local peerID: <redacted>=, local peer hashedID: sha-<redacted>=
2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:170: connecting to relay server
2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via quic
2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via WS
2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:89: successfully dialed via: WS
2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:75: connection attempt aborted via: quic
2025-03-29T15:51:59Z INFO [relay: rels://streamline-us-nyc1-0.relay.netbird.io:443] relay/client/client.go:186: relay connection established
2025-03-29T15:51:59Z INFO relay/client/picker.go:90: connected to Relay server: rels://relay.netbird.io:443
2025-03-29T15:51:59Z INFO relay/client/picker.go:64: chosen home Relay server: rels://relay.netbird.io:443
2025-03-29T15:51:59Z INFO client/iface/wgproxy/factory_kernel.go:25: WireGuard Proxy Factory will produce UDP proxy
2025-03-29T15:51:59Z WARN client/iface/wgproxy/factory_kernel.go:26: failed to initialize ebpf proxy, fallback to user space proxy: failed to set memlock rlimit: operation not permitted
2025-03-29T15:51:59Z ERRO client/internal/routemanager/systemops/systemops_linux.go:102: Error setting up sysctl: 1 error occurred:
        * write sysctl net.ipv4.conf.all.src_valid_mark: open /proc/sys/net/ipv4/conf/all/src_valid_mark: read-only file system
2025-03-29T15:51:59Z INFO client/internal/routemanager/manager.go:193: Routing setup complete
2025-03-29T15:51:59Z INFO client/internal/engine.go:459: firewall is disabled
2025-03-29T15:51:59Z INFO client/internal/dns/host_unix.go:54: System DNS manager discovered: file
2025-03-29T15:52:00Z INFO client/internal/peer/guard/sr_watcher.go:106: reconnected to Signal or Relay server
2025-03-29T15:52:00Z INFO signal/client/grpc.go:149: connected to the Signal Service stream
2025-03-29T15:52:00Z INFO client/internal/engine.go:1664: Network monitor is disabled, not starting
2025-03-29T15:52:00Z INFO client/internal/connect.go:281: Netbird engine started, the IP is: 100.90.0.233/16
2025-03-29T15:52:00Z INFO management/client/grpc.go:156: connected to the Management Service stream
2025-03-29T15:52:00Z INFO relay/client/manager.go:223: update relay server URLs: [rels://relay.netbird.io:443]
2025-03-29T15:52:00Z WARN client/internal/engine.go:785: running SSH server is not permitted
2025-03-29T15:52:00Z WARN client/internal/engine.go:1829: firewall is disabled, not updating forwarding rules
2025-03-29T15:52:00Z INFO [peer: <redacted> client/internal/peer/handshaker.go:79: wait for remote offer confirmation
2025-03-29T15:52:00Z INFO [peer: <redacted> client/internal/peer/handshaker.go:79: wait for remote offer confirmation
2025-03-29T15:52:00Z INFO [peer: <redacted> client/internal/peer/handshaker.go:79: wait for remote offer confirmation
2025-03-29T15:52:00Z ERRO client/internal/dns/server.go:424: unable to configure DNS for this peer using file manager without a nameserver group with all domains configured
2025-03-29T15:52:03Z INFO [peer: <redacted> client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-03-29T15:52:03Z INFO [peer: <redacted> client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-03-29T15:52:03Z INFO [peer: <redacted> client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-03-29T15:52:04Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
2025-03-29T15:52:04Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
2025-03-29T15:52:04Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
2025-03-29T15:52:05Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
2025-03-29T15:52:06Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
2025-03-29T15:52:06Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
2025-03-29T15:52:08Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
2025-03-29T15:52:09Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
2025-03-29T15:52:09Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
2025-03-29T15:52:14Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer
<!-- gh-comment-id:2763587377 --> @l0crian1 commented on GitHub (Mar 29, 2025): > Hello [@l0crian1](https://github.com/l0crian1) it seems like the daemon is not running. Does VyOS has systemD? if so, can you share the logs from: > > /var/log/netbird/client.log /var/log/netbird/netbird.err I can't pull those directly. Here is the log from the container. One thing to note, is even when it's working, that debug says the daemon is down, so that seems to be a red herring: ## \# None Working State (with VRF configured) ``` 2025-03-29T15:42:54Z INFO client/internal/connect.go:122: starting NetBird client version 0.39.2 on linux/amd64 2025-03-29T15:42:54Z INFO util/net/env_linux.go:70: system supports advanced routing 2025-03-29T15:42:54Z INFO client/internal/connect.go:255: connecting to the Relay service(s): rels://relay.netbird.io:443 2025-03-29T15:42:54Z INFO relay/client/picker.go:72: try to connecting to relay server: rels://relay.netbird.io:443 2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:164: create new relay connection: local peerID: <redacted>=, local peer hashedID: sha-<redacted>= 2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:170: connecting to relay server 2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via quic 2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via WS 2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:89: successfully dialed via: WS 2025-03-29T15:42:54Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:75: connection attempt aborted via: quic 2025-03-29T15:42:54Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:186: relay connection established 2025-03-29T15:42:54Z INFO relay/client/picker.go:90: connected to Relay server: rels://relay.netbird.io:443 2025-03-29T15:42:54Z INFO relay/client/picker.go:64: chosen home Relay server: rels://relay.netbird.io:443 2025-03-29T15:42:54Z INFO client/iface/wgproxy/factory_kernel.go:25: WireGuard Proxy Factory will produce UDP proxy 2025-03-29T15:42:54Z WARN client/iface/wgproxy/factory_kernel.go:26: failed to initialize ebpf proxy, fallback to user space proxy: failed to set memlock rlimit: operation not permitted 2025-03-29T15:42:54Z ERRO client/internal/routemanager/systemops/systemops_linux.go:102: Error setting up sysctl: 1 error occurred: * write sysctl net.ipv4.conf.all.src_valid_mark: open /proc/sys/net/ipv4/conf/all/src_valid_mark: read-only file system 2025-03-29T15:42:54Z INFO client/internal/routemanager/manager.go:193: Routing setup complete 2025-03-29T15:42:54Z INFO client/internal/engine.go:459: firewall is disabled 2025-03-29T15:42:54Z INFO client/internal/dns/host_unix.go:54: System DNS manager discovered: file 2025-03-29T15:43:29Z ERRO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:474: health check timeout 2025-03-29T15:43:29Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:336: start to Relay read loop exit 2025-03-29T15:43:29Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:530: closing all peer connections 2025-03-29T15:43:29Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:538: waiting for read loop to close 2025-03-29T15:43:29Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:540: relay connection closed 2025-03-29T15:43:31Z INFO relay/client/guard.go:81: try to reconnect to Relay server: rels://relay.netbird.io:443 2025-03-29T15:43:31Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/client.go:170: connecting to relay server 2025-03-29T15:43:31Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via quic 2025-03-29T15:43:31Z INFO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via WS 2025-03-29T15:43:34Z WARN signal/client/grpc.go:143: disconnected from the Signal Exchange due to an error: didn't receive a registration header from the Signal server whille connecting to the streams 2025-03-29T15:43:41Z ERRO relay/client/dialer/ws/ws.go:50: failed to dial to Relay server 'wss://relay.netbird.io:443': failed to WebSocket dial: failed to send handshake request: Get "https://relay.netbird.io:443/relay": d.Dialer.DialContext: dial tcp: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:33119->4.2.2.2:53: i/o timeout 2025-03-29T15:43:41Z ERRO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/dialer/race_dialer.go:77: failed to dial via WS: failed to WebSocket dial: failed to send handshake request: Get "https://relay.netbird.io:443/relay": d.Dialer.DialContext: dial tcp: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:33119->4.2.2.2:53: i/o timeout 2025-03-29T15:43:41Z ERRO relay/client/dialer/quic/quic.go:46: failed to resolve UDP address: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:39559->4.2.2.2:53: i/o timeout 2025-03-29T15:43:41Z ERRO [relay: rels://streamline-us-nyc1-1.relay.netbird.io:443] relay/client/dialer/race_dialer.go:77: failed to dial via quic: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:39559->4.2.2.2:53: i/o timeout 2025-03-29T15:43:41Z ERRO relay/client/guard.go:84: failed to reconnect to relay server: failed to dial to Relay server on any protocol 2025-03-29T15:43:41Z INFO relay/client/guard.go:91: try to pick up a new Relay server 2025-03-29T15:43:41Z INFO relay/client/picker.go:72: try to connecting to relay server: rels://relay.netbird.io:443 2025-03-29T15:43:41Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:164: create new relay connection: local peerID: <redacted>=, local peer hashedID: sha-<redacted>= 2025-03-29T15:43:41Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:170: connecting to relay server 2025-03-29T15:43:41Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via quic 2025-03-29T15:43:41Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via WS 2025-03-29T15:43:45Z ERRO util/grpc/dialer.go:45: Failed to dial: d.Dialer.DialContext: dial tcp: lookup signal.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:49247->4.2.2.2:53: i/o timeout 2025-03-29T15:43:51Z ERRO relay/client/dialer/quic/quic.go:46: failed to resolve UDP address: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:43302->4.2.2.2:53: i/o timeout 2025-03-29T15:43:51Z ERRO relay/client/dialer/ws/ws.go:50: failed to dial to Relay server 'wss://relay.netbird.io:443': failed to WebSocket dial: failed to send handshake request: Get "https://relay.netbird.io:443/relay": d.Dialer.DialContext: dial tcp: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:34450->4.2.2.2:53: i/o timeout 2025-03-29T15:43:51Z ERRO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:77: failed to dial via quic: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:43302->4.2.2.2:53: i/o timeout 2025-03-29T15:43:51Z ERRO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:77: failed to dial via WS: failed to WebSocket dial: failed to send handshake request: Get "https://relay.netbird.io:443/relay": d.Dialer.DialContext: dial tcp: lookup relay.netbird.io on 4.2.2.2:53: read udp 10.0.101.245:34450->4.2.2.2:53: i/o timeout 2025-03-29T15:43:51Z ERRO relay/client/guard.go:58: failed to pick new Relay server: failed to connect to any relay server: all attempts failed ``` ## \# Working State (with VRF removed) ``` 2025-03-29T15:51:59Z INFO client/internal/connect.go:122: starting NetBird client version 0.39.2 on linux/amd64 2025-03-29T15:51:59Z INFO util/net/env_linux.go:70: system supports advanced routing 2025-03-29T15:51:59Z INFO client/internal/connect.go:255: connecting to the Relay service(s): rels://relay.netbird.io:443 2025-03-29T15:51:59Z INFO relay/client/picker.go:72: try to connecting to relay server: rels://relay.netbird.io:443 2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:164: create new relay connection: local peerID: <redacted>=, local peer hashedID: sha-<redacted>= 2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/client.go:170: connecting to relay server 2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via quic 2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:64: dialing Relay server via WS 2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:89: successfully dialed via: WS 2025-03-29T15:51:59Z INFO [relay: rels://relay.netbird.io:443] relay/client/dialer/race_dialer.go:75: connection attempt aborted via: quic 2025-03-29T15:51:59Z INFO [relay: rels://streamline-us-nyc1-0.relay.netbird.io:443] relay/client/client.go:186: relay connection established 2025-03-29T15:51:59Z INFO relay/client/picker.go:90: connected to Relay server: rels://relay.netbird.io:443 2025-03-29T15:51:59Z INFO relay/client/picker.go:64: chosen home Relay server: rels://relay.netbird.io:443 2025-03-29T15:51:59Z INFO client/iface/wgproxy/factory_kernel.go:25: WireGuard Proxy Factory will produce UDP proxy 2025-03-29T15:51:59Z WARN client/iface/wgproxy/factory_kernel.go:26: failed to initialize ebpf proxy, fallback to user space proxy: failed to set memlock rlimit: operation not permitted 2025-03-29T15:51:59Z ERRO client/internal/routemanager/systemops/systemops_linux.go:102: Error setting up sysctl: 1 error occurred: * write sysctl net.ipv4.conf.all.src_valid_mark: open /proc/sys/net/ipv4/conf/all/src_valid_mark: read-only file system 2025-03-29T15:51:59Z INFO client/internal/routemanager/manager.go:193: Routing setup complete 2025-03-29T15:51:59Z INFO client/internal/engine.go:459: firewall is disabled 2025-03-29T15:51:59Z INFO client/internal/dns/host_unix.go:54: System DNS manager discovered: file 2025-03-29T15:52:00Z INFO client/internal/peer/guard/sr_watcher.go:106: reconnected to Signal or Relay server 2025-03-29T15:52:00Z INFO signal/client/grpc.go:149: connected to the Signal Service stream 2025-03-29T15:52:00Z INFO client/internal/engine.go:1664: Network monitor is disabled, not starting 2025-03-29T15:52:00Z INFO client/internal/connect.go:281: Netbird engine started, the IP is: 100.90.0.233/16 2025-03-29T15:52:00Z INFO management/client/grpc.go:156: connected to the Management Service stream 2025-03-29T15:52:00Z INFO relay/client/manager.go:223: update relay server URLs: [rels://relay.netbird.io:443] 2025-03-29T15:52:00Z WARN client/internal/engine.go:785: running SSH server is not permitted 2025-03-29T15:52:00Z WARN client/internal/engine.go:1829: firewall is disabled, not updating forwarding rules 2025-03-29T15:52:00Z INFO [peer: <redacted> client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-03-29T15:52:00Z INFO [peer: <redacted> client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-03-29T15:52:00Z INFO [peer: <redacted> client/internal/peer/handshaker.go:79: wait for remote offer confirmation 2025-03-29T15:52:00Z ERRO client/internal/dns/server.go:424: unable to configure DNS for this peer using file manager without a nameserver group with all domains configured 2025-03-29T15:52:03Z INFO [peer: <redacted> client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-03-29T15:52:03Z INFO [peer: <redacted> client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-03-29T15:52:03Z INFO [peer: <redacted> client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-03-29T15:52:04Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer 2025-03-29T15:52:04Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer 2025-03-29T15:52:04Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer 2025-03-29T15:52:05Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer 2025-03-29T15:52:06Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer 2025-03-29T15:52:06Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer 2025-03-29T15:52:08Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer 2025-03-29T15:52:09Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer 2025-03-29T15:52:09Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer 2025-03-29T15:52:14Z INFO [peer: <redacted> client/internal/peer/conn.go:548: send offer to peer ```
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Mar 29, 2025):

Got it.

The container runs in the foreground and doesn't support status or other commands.

Since 0.25.9 we've introduced many changes to the client and how firewall and routes are managed. One of these changes might be causing you issues.

Can you try exporting the following variables to the container and see if they help?

NB_DISABLE_CUSTOM_ROUTING=true
NB_USE_LEGACY_ROUTING=true

If they work, try without NB_DISABLE_CUSTOM_ROUTING=true.

<!-- gh-comment-id:2763625944 --> @mlsmaycon commented on GitHub (Mar 29, 2025): Got it. The container runs in the foreground and doesn't support status or other commands. Since 0.25.9 we've introduced many changes to the client and how firewall and routes are managed. One of these changes might be causing you issues. Can you try exporting the following variables to the container and see if they help? ``` NB_DISABLE_CUSTOM_ROUTING=true NB_USE_LEGACY_ROUTING=true ``` If they work, try without `NB_DISABLE_CUSTOM_ROUTING=true`.
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@l0crian1 commented on GitHub (Mar 29, 2025):

That works thanks! I tried it with both, as well as each of them one at a time. Either option corrected the issue.

<!-- gh-comment-id:2763650146 --> @l0crian1 commented on GitHub (Mar 29, 2025): That works thanks! I tried it with both, as well as each of them one at a time. Either option corrected the issue.
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Mar 29, 2025):

@lixmal which flag would be the correct one?

<!-- gh-comment-id:2763655962 --> @mlsmaycon commented on GitHub (Mar 29, 2025): @lixmal which flag would be the correct one?
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Apr 2, 2025):

NB_USE_LEGACY_ROUTING is preferable; with NB_DISABLE_CUSTOM_ROUTING, you run the risk of routing loops.

I'm curious about the output of ip rule when running without those env vars, I assume there must be an issue with the order of the rules.

<!-- gh-comment-id:2771886331 --> @lixmal commented on GitHub (Apr 2, 2025): `NB_USE_LEGACY_ROUTING` is preferable; with `NB_DISABLE_CUSTOM_ROUTING`, you run the risk of routing loops. I'm curious about the output of `ip rule` when running without those env vars, I assume there must be an issue with the order of the rules.
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@l0crian1 commented on GitHub (Apr 3, 2025):

I'm curious about the output of ip rule when running without those env vars, I assume there must be an issue with the order of the rules.

Here are the ouputs for that:

Before creation of VRFs:

0:      from all lookup local
100:    from all lookup main suppress_prefixlength 0
110:    not from all fwmark 0x1bd00 lookup 7120
32766:  from all lookup main
32767:  from all lookup default

After VRF creation:

100:    from all lookup main suppress_prefixlength 0
110:    not from all fwmark 0x1bd00 lookup 7120
1000:   from all lookup [l3mdev-table]
2000:   from all lookup [l3mdev-table] unreachable
32765:  from all lookup local
32766:  from all lookup main
32767:  from all lookup default
<!-- gh-comment-id:2776418132 --> @l0crian1 commented on GitHub (Apr 3, 2025): > I'm curious about the output of `ip rule` when running without those env vars, I assume there must be an issue with the order of the rules. Here are the ouputs for that: ### Before creation of VRFs: ``` 0: from all lookup local 100: from all lookup main suppress_prefixlength 0 110: not from all fwmark 0x1bd00 lookup 7120 32766: from all lookup main 32767: from all lookup default ``` ### After VRF creation: ``` 100: from all lookup main suppress_prefixlength 0 110: not from all fwmark 0x1bd00 lookup 7120 1000: from all lookup [l3mdev-table] 2000: from all lookup [l3mdev-table] unreachable 32765: from all lookup local 32766: from all lookup main 32767: from all lookup default ```
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Apr 26, 2025):

I see, I'm not sure changing the priority will help, but have you tried changing the 1000/2000 rules to prio <100 or between 100 and 110? If that's configurable at all

<!-- gh-comment-id:2831992183 --> @lixmal commented on GitHub (Apr 26, 2025): I see, I'm not sure changing the priority will help, but have you tried changing the `1000`/`2000` rules to prio <100 or between 100 and 110? If that's configurable at all
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@bl0way commented on GitHub (Aug 13, 2025):

I ran into the same issue today with the version 0.54.1 and setting NB_USE_LEGACY_ROUTING=true solved the issue (no ip rules anymore). Thanks for that 😃

The rule 1000 is somehow the default for linux when you create one or more VRF. However, the 2000 rule and the re-ordering of the local table lookup are not done by default on Linux. This link is a good explanation for the reason behind those modifications.

It would be great to have the possibility to change the netbird rule numbers. Or use VRF directly instead of firewall marks + rules. Maybe, It could simplify some code?

PS: As VyOS is debian based, the order can be changed manually (some ip rule commands) but this is breaking the VyOS way of doing things and would affect further vyos commands.

<!-- gh-comment-id:3186061293 --> @bl0way commented on GitHub (Aug 13, 2025): I ran into the same issue today with the version `0.54.1` and setting `NB_USE_LEGACY_ROUTING=true` solved the issue (no ip rules anymore). Thanks for that 😃 The rule `1000` is somehow the default for linux when you create one or more VRF. However, the `2000` rule and the re-ordering of the `local` table lookup are not done by default on Linux. This [link](https://stbuehler.de/blog/article/2020/02/29/using_vrf__virtual_routing_and_forwarding__on_linux.html) is a good explanation for the reason behind those modifications. It would be great to have the possibility to change the netbird rule numbers. Or use VRF directly instead of firewall marks + rules. Maybe, It could simplify some code? PS: As VyOS is debian based, the order can be changed manually (some ip rule commands) but this is breaking the VyOS way of doing things and would affect further vyos commands.
saavagebueno commented 2026-08-05 01:12:03 -04:00
Author
Owner
Copy Link

@WuuBoLin commented on GitHub (Apr 14, 2026):

A year later and this issue persists. Setting NB_USE_LEGACY_ROUTING helps.

I stumbled upon this recently and spent half an hour debugging before finding this thread, which describes the exact same scenario. Very grateful. This saved me a lot of time!

That said, there is a minor issue with NB_USE_LEGACY_ROUTING.
In my topology, I have two WAN (PPPoE) interfaces, one of which is inside a VRF (pppoe1). When the NetBird container starts up and configures routes, it occasionally selects the wrong next-hop device.

vyos@router:~$ ip route list table main
default nhid 17 dev pppoe0 proto static metric 20
10.10.0.1 via <redacted> dev pppoe0
10.10.20.80 via <redacted> dev pppoe0
100.100.0.0/14 dev wg0 proto kernel scope link src 100.100.2.86
172.16.6.0/24 dev eth1.6 proto kernel scope link src 172.16.6.254
172.16.16.0/24 nhid 10 dev eth1.16 proto static metric 20
172.16.246.1 via <redacted> dev pppoe0
192.168.1.86 via <redacted> dev pppoe1 # <- Sets to the wrong interface
192.168.214.1 via <redacted> dev pppoe1 # <- Sets to the wrong interface
<redacted> via <redacted> dev pppoe0

I'm not sure whether this is a NetBird specific issue, but the only workaround is setting NB_DISABLE_CUSTOM_ROUTING.
This works fine with static routing, but may break things if I ever want to use this node as an exit node.

NetBird version: 0.68.2
VyOS version: 2026.04.01-0034-rolling

<!-- gh-comment-id:4244061520 --> @WuuBoLin commented on GitHub (Apr 14, 2026): A year later and this issue persists. Setting `NB_USE_LEGACY_ROUTING` helps. I stumbled upon this recently and spent half an hour debugging before finding this thread, which describes the **exact** same scenario. Very grateful. This saved me a lot of time! That said, there is a minor issue with `NB_USE_LEGACY_ROUTING`. In my topology, I have two WAN (PPPoE) interfaces, one of which is inside a VRF (`pppoe1`). When the NetBird container starts up and configures routes, it occasionally selects the wrong next-hop device. ```shell vyos@router:~$ ip route list table main default nhid 17 dev pppoe0 proto static metric 20 10.10.0.1 via <redacted> dev pppoe0 10.10.20.80 via <redacted> dev pppoe0 100.100.0.0/14 dev wg0 proto kernel scope link src 100.100.2.86 172.16.6.0/24 dev eth1.6 proto kernel scope link src 172.16.6.254 172.16.16.0/24 nhid 10 dev eth1.16 proto static metric 20 172.16.246.1 via <redacted> dev pppoe0 192.168.1.86 via <redacted> dev pppoe1 # <- Sets to the wrong interface 192.168.214.1 via <redacted> dev pppoe1 # <- Sets to the wrong interface <redacted> via <redacted> dev pppoe0 ``` I'm not sure whether this is a NetBird specific issue, but the only workaround is setting `NB_DISABLE_CUSTOM_ROUTING`. This works fine with static routing, but may break things if I ever want to use this node as an exit node. --- **NetBird** version: `0.68.2` **VyOS** version: `2026.04.01-0034-rolling`
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2021 Q4
2021 Q4
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
accessibility
accessibility
accessibility
acl
acl
acl
agent
agent
agent
agent
agent
agent
Android
Android
Android
Android
Android
Android
api
api
api
authentik
authentik
authentik
automation
automation
automation
azure
azure
azure
battery-usage
battery-usage
battery-usage
bug
cache
cache
cache
client
client
client
client-ui
client-ui
client-ui
cloud
cloud
cloud
cloud-only
cloud-only
cloud-only
cloudflare
cloudflare
cloudflare
community
community
community
compatibility
compatibility
compatibility
config-idp
config-idp
config-idp
config-issue
config-issue
config-issue
connection
connection
connection
contribution
contribution
contribution
coturn
coturn
coturn
cross-vpn
cross-vpn
cross-vpn
dashboard
dashboard
dashboard
data-usage
data-usage
data-usage
distribution
distribution
distribution
dns
dns
dns
docker
docker
docker
documentation
documentation
documentation
duplicate
duplicate
duplicate
enhancement
enhancement
event-stream
event-stream
event-stream
feature-request
feature-request
feature-request
freebsd
freebsd
freebsd
getting-started
getting-started
getting-started
go
go
go
good first issue
good first issue
good first issue
gui
gui
gui
help wanted
help wanted
help wanted
home-assistant
home-assistant
home-assistant
idp
idp
idp
inconsistency
inconsistency
inconsistency
integration
integration
integration
integrations
integrations
integrations
ios
ios
ios
ipv6
ipv6
ipv6
jwt
jwt
jwt
k8s
k8s
k8s
keycloak
keycloak
keycloak
linux
linux
linux
login
login
login
macos
macos
macos
management-service
management-service
management-service
Medium
Medium
Medium
missing-docs
missing-docs
missing-docs
mobile
mobile
mobile
moved-internal
moved-internal
moved-internal
needs-review
needs-review
needs-review
netbird-ui
netbird-ui
netbird-ui
networking
networking
networking
new-platform
new-platform
new-platform
nginx
nginx
nginx
notification
notification
notification
okta
okta
okta
openwrt
openwrt
openwrt
P2
P2
P2
packaging
packaging
packaging
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
performance
performance
performance
postgres
postgres
postgres
posture-checks
posture-checks
posture-checks
psk
psk
psk
pull-request
Mirrored from GitHub Pull Request
 question
question
question
refactor
refactor
refactor
relay
relay
relay
release
release
release
rfc
rfc
rfc
routes
routes
routes
security
security
security
security-improvement
security-improvement
security-improvement
security-related
security-related
security-related
self-hosting
self-hosting
self-hosting
server
server
server
signal
signal
signal
sleep-issue
sleep-issue
sleep-issue
ssh
ssh
ssh
ssl
ssl
ssl
status
status
status
store
store
store
synology
synology
synology
system-compatibility-issue
system-compatibility-issue
system-compatibility-issue
test-suite
test-suite
test-suite
third-party-integration
third-party-integration
third-party-integration
triage
triage
triage
triage
triage
triage
triage-needed
triage-needed
triage-needed
troubleshooting
troubleshooting
troubleshooting
UX
UX
UX
waiting-feedback
waiting-feedback
waiting-feedback
windows
windows
windows
wontfix
wontfix
wontfix
zitadel
zitadel
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dynr-servers saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#7211
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?