DYNR/netbird
2
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-05 00:15:26 -04:00
Code Issues 23.5k Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #2421] Client does not create host route to the exit nodes / peers #5108

New Issue
Closed
opened 2026-08-05 01:01:24 -04:00 by saavagebueno · 10 comments
saavagebueno commented 2026-08-05 01:01:24 -04:00
Owner
Copy Link

Originally created by @Takalele on GitHub (Aug 11, 2024).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/2421

Describe the problem

When a Exit Node is defined and enabled, a client with that "default route" enabled will send ALL traffic through the tunnel creating a blackhole/loop.

adding the hostroute manually fixed solves the problem.

To Reproduce

Steps to reproduce the behavior:

  1. Add Exit Node
  2. Enabled it on the client
  3. backhole created
  4. See error

Expected behavior

the client creates a hostroute (/32) to each peer based of the original default route.

Are you using NetBird Cloud?

Please specify whether you use NetBird Cloud or self-host NetBird's control plane.

NetBird selfhosted

netbird 0.28.7

NetBird status -dA output:

C:\Windows\System32>netbird status -dA
Peers detail:
 exit-node-vpn.netbird.selfhosted:
  NetBird IP: 100.105.9.17
  Public key: SuE8Hl0ChqsnMfR2/TrxFBtJrJV5ZbOV2dcnPitnBkA=
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.48.1:51820/198.51.100.0:51820
  Last connection update: 23 minutes, 45 seconds ago
  Last WireGuard handshake: 1 minute, 42 seconds ago
  Transfer status (received/sent) 12.3 MiB/8.1 MiB
  Quantum resistance: false
  Routes: 0.0.0.0/0
  Latency: 20.6718ms

 handy.netbird.selfhosted:
  NetBird IP: 100.105.33.250
  Public key: KqK8OQ6jSinnSU/yBYNCS35GKRj5FLHWlffDkeXcfT8=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:62997/198.51.100.1:25439
  Last connection update: 8 minutes, 30 seconds ago
  Last WireGuard handshake: 2 seconds ago
  Transfer status (received/sent) 1.6 KiB/1.3 KiB
  Quantum resistance: false
  Routes: -
  Latency: 171.03ms

 dsda-1.netbird.selfhosted:
  NetBird IP: 100.105.191.186
  Public key: w7Bd10GqBG98bM+2oqJmNlDvXRzjfmFHiv5AGhfm6CU=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/srflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:58185/198.51.100.2:51820
  Last connection update: 23 minutes, 40 seconds ago
  Last WireGuard handshake: 44 seconds ago
  Transfer status (received/sent) 3.8 KiB/2.8 KiB
  Quantum resistance: false
  Routes: -
  Latency: 37.3066ms

 is2vbb5.netbird.selfhosted:
  NetBird IP: 100.105.230.218
  Public key: KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=
  Status: Disconnected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:62997/198.51.100.1:25439
  Last connection update: -
  Last WireGuard handshake: 2 seconds ago
  Transfer status (received/sent) 1.6 KiB/1.3 KiB
  Quantum resistance: false
  Routes: -
  Latency: 0s

OS: windows/amd64
Daemon version: 0.28.7
CLI version: 0.28.7
Management: Connected to https://vpn.anon-nDrWh.domain:443
Signal: Connected to https://vpn.anon-nDrWh.domain:443
Relays:
  [stun:turn.anon-nDrWh.domain:3478] is Available
  [turn:turn.anon-nDrWh.domain:3478?transport=udp] is Available
Nameservers:
FQDN: voyager-1.netbird.selfhosted
NetBird IP: 100.105.209.49/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 3/4 Connected
Originally created by @Takalele on GitHub (Aug 11, 2024). Original GitHub issue: https://github.com/netbirdio/netbird/issues/2421 **Describe the problem** When a Exit Node is defined and enabled, a client with that "default route" enabled will send ALL traffic through the tunnel creating a blackhole/loop. adding the hostroute manually fixed solves the problem. **To Reproduce** Steps to reproduce the behavior: 1. Add Exit Node 2. Enabled it on the client 3. backhole created 4. See error **Expected behavior** the client creates a hostroute (/32) to each peer based of the original default route. **Are you using NetBird Cloud?** Please specify whether you use NetBird Cloud or self-host NetBird's control plane. **NetBird selfhosted** `netbird 0.28.7` **NetBird status -dA output:** ``` C:\Windows\System32>netbird status -dA Peers detail: exit-node-vpn.netbird.selfhosted: NetBird IP: 100.105.9.17 Public key: SuE8Hl0ChqsnMfR2/TrxFBtJrJV5ZbOV2dcnPitnBkA= Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): host/host ICE candidate endpoints (Local/Remote): 192.168.48.1:51820/198.51.100.0:51820 Last connection update: 23 minutes, 45 seconds ago Last WireGuard handshake: 1 minute, 42 seconds ago Transfer status (received/sent) 12.3 MiB/8.1 MiB Quantum resistance: false Routes: 0.0.0.0/0 Latency: 20.6718ms handy.netbird.selfhosted: NetBird IP: 100.105.33.250 Public key: KqK8OQ6jSinnSU/yBYNCS35GKRj5FLHWlffDkeXcfT8= Status: Connected -- detail -- Connection type: Relayed Direct: false ICE candidate (Local/Remote): relay/prflx ICE candidate endpoints (Local/Remote): 198.51.100.0:62997/198.51.100.1:25439 Last connection update: 8 minutes, 30 seconds ago Last WireGuard handshake: 2 seconds ago Transfer status (received/sent) 1.6 KiB/1.3 KiB Quantum resistance: false Routes: - Latency: 171.03ms dsda-1.netbird.selfhosted: NetBird IP: 100.105.191.186 Public key: w7Bd10GqBG98bM+2oqJmNlDvXRzjfmFHiv5AGhfm6CU= Status: Connected -- detail -- Connection type: Relayed Direct: false ICE candidate (Local/Remote): relay/srflx ICE candidate endpoints (Local/Remote): 198.51.100.0:58185/198.51.100.2:51820 Last connection update: 23 minutes, 40 seconds ago Last WireGuard handshake: 44 seconds ago Transfer status (received/sent) 3.8 KiB/2.8 KiB Quantum resistance: false Routes: - Latency: 37.3066ms is2vbb5.netbird.selfhosted: NetBird IP: 100.105.230.218 Public key: KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs= Status: Disconnected -- detail -- Connection type: Relayed Direct: false ICE candidate (Local/Remote): relay/prflx ICE candidate endpoints (Local/Remote): 198.51.100.0:62997/198.51.100.1:25439 Last connection update: - Last WireGuard handshake: 2 seconds ago Transfer status (received/sent) 1.6 KiB/1.3 KiB Quantum resistance: false Routes: - Latency: 0s OS: windows/amd64 Daemon version: 0.28.7 CLI version: 0.28.7 Management: Connected to https://vpn.anon-nDrWh.domain:443 Signal: Connected to https://vpn.anon-nDrWh.domain:443 Relays: [stun:turn.anon-nDrWh.domain:3478] is Available [turn:turn.anon-nDrWh.domain:3478?transport=udp] is Available Nameservers: FQDN: voyager-1.netbird.selfhosted NetBird IP: 100.105.209.49/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 3/4 Connected ```
saavagebueno added the clientroutes labels 2026-08-05 01:01:24 -04:00
saavagebueno commented 2026-08-05 01:01:24 -04:00
Author
Owner
Copy Link

@pascal-fischer commented on GitHub (Aug 12, 2024):

Hi @Takalele, I have some trouble understanding what could be going wrong here. Can you share some logs before adding any routes to the peer (following https://docs.netbird.io/how-to/troubleshooting-client#debug-bundle).
And a snapshot of the important parts of the routing table before, with netbird and after you added the route? (maybe in some redacted format)

<!-- gh-comment-id:2283598530 --> @pascal-fischer commented on GitHub (Aug 12, 2024): Hi @Takalele, I have some trouble understanding what could be going wrong here. Can you share some logs before adding any routes to the peer (following https://docs.netbird.io/how-to/troubleshooting-client#debug-bundle). And a snapshot of the important parts of the routing table before, with netbird and after you added the route? (maybe in some redacted format)
saavagebueno commented 2026-08-05 01:01:24 -04:00
Author
Owner
Copy Link

@Takalele commented on GitHub (Aug 13, 2024):

hi, @pascal-fischer

Unfortunately, I’m unable to address that at the moment as I’m encountering another issue. While the login is successful, the client is throwing the following error:
image
mgmt logs:

WARN [context: GRPC, requestID: 6f5d8624-f48f-4a80-9118-70e385cdbd9b, accountID: UNKNOWN, peerID: xyz] management/server/grpcserver.go:456: failed validating JWT token sent
from peer xyz with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: Token used before issued. Trying again as it may be due to the IdP cache issue

The same issue also occurs when I log into the admin portal, but it resolves itself after a couple of seconds

2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued
2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued
2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/http/util/util.go:81: got a handler error: token invalid
2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/telemetry/http_api_metrics.go:191: HTTP response 3ac89db3-ea21-4957-afa9-104c0a390af7: GET /api/users status 401
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/http/util/util.go:81: got a handler error: token invalid
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/telemetry/http_api_metrics.go:191: HTTP response f4b5bd0b-ed84-400f-9581-853ab5f1efb4: GET /api/users status 401
2024-08-13T17:07:54Z ERRO [requestID: c7244da2-af01-4289-9c40-301a95847b23, context: HTTP] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: c7244da2-af01-4289-9c40-301a95847b23] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: c7244da2-af01-4289-9c40-301a95847b23] management/server/http/util/util.go:81: got a handler error: token invalid
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: c7244da2-af01-4289-9c40-301a95847b23] management/server/telemetry/http_api_metrics.go:191: HTTP response c7244da2-af01-4289-9c40-301a95847b23: GET /api/users status 401
2024-08-13T17:07:55Z ERRO [context: HTTP, requestID: 93721015-4a45-4b48-96e2-425308036673] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued
2024-08-13T17:07:55Z ERRO [requestID: 93721015-4a45-4b48-96e2-425308036673, context: HTTP] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued
2024-08-13T17:07:55Z ERRO [context: HTTP, requestID: 93721015-4a45-4b48-96e2-425308036673] management/server/http/util/util.go:81: got a handler error: token invalid
2024-08-13T17:07:55Z ERRO [context: HTTP, requestID: 93721015-4a45-4b48-96e2-425308036673] management/server/telemetry/http_api_metrics.go:191: HTTP response 93721015-4a45-4b48-96e2-425308036673: GET /api/users status 401

I suspected a clock issue, but both systems and all the containers have the correct time.

any ideas?

<!-- gh-comment-id:2286719595 --> @Takalele commented on GitHub (Aug 13, 2024): hi, @pascal-fischer Unfortunately, I’m unable to address that at the moment as I’m encountering another issue. While the login is successful, the client is throwing the following error: ![image](https://github.com/user-attachments/assets/8d207e68-abf4-429e-8ec1-9d12b057d13c) mgmt logs: ``` WARN [context: GRPC, requestID: 6f5d8624-f48f-4a80-9118-70e385cdbd9b, accountID: UNKNOWN, peerID: xyz] management/server/grpcserver.go:456: failed validating JWT token sent from peer xyz with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: Token used before issued. Trying again as it may be due to the IdP cache issue ``` The same issue also occurs when I log into the admin portal, but it resolves itself after a couple of seconds ``` 2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued 2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued 2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/http/util/util.go:81: got a handler error: token invalid 2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/telemetry/http_api_metrics.go:191: HTTP response 3ac89db3-ea21-4957-afa9-104c0a390af7: GET /api/users status 401 2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued 2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued 2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/http/util/util.go:81: got a handler error: token invalid 2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/telemetry/http_api_metrics.go:191: HTTP response f4b5bd0b-ed84-400f-9581-853ab5f1efb4: GET /api/users status 401 2024-08-13T17:07:54Z ERRO [requestID: c7244da2-af01-4289-9c40-301a95847b23, context: HTTP] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued 2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: c7244da2-af01-4289-9c40-301a95847b23] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued 2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: c7244da2-af01-4289-9c40-301a95847b23] management/server/http/util/util.go:81: got a handler error: token invalid 2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: c7244da2-af01-4289-9c40-301a95847b23] management/server/telemetry/http_api_metrics.go:191: HTTP response c7244da2-af01-4289-9c40-301a95847b23: GET /api/users status 401 2024-08-13T17:07:55Z ERRO [context: HTTP, requestID: 93721015-4a45-4b48-96e2-425308036673] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued 2024-08-13T17:07:55Z ERRO [requestID: 93721015-4a45-4b48-96e2-425308036673, context: HTTP] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued 2024-08-13T17:07:55Z ERRO [context: HTTP, requestID: 93721015-4a45-4b48-96e2-425308036673] management/server/http/util/util.go:81: got a handler error: token invalid 2024-08-13T17:07:55Z ERRO [context: HTTP, requestID: 93721015-4a45-4b48-96e2-425308036673] management/server/telemetry/http_api_metrics.go:191: HTTP response 93721015-4a45-4b48-96e2-425308036673: GET /api/users status 401 ``` I suspected a clock issue, but both systems and all the containers have the correct time. any ideas?
saavagebueno commented 2026-08-05 01:01:25 -04:00
Author
Owner
Copy Link

@pascal-fischer commented on GitHub (Aug 13, 2024):

What is the IDP that you are using?
You need to make sure the time used for the container running your IDP and the time used where the management container is running are in sync, down to the second.
You can use sudo ntpdate pool.ntp.org to sync the time of those machines. Afterwards you need to restart the services.

<!-- gh-comment-id:2286754758 --> @pascal-fischer commented on GitHub (Aug 13, 2024): What is the IDP that you are using? You need to make sure the time used for the container running your IDP and the time used where the management container is running are in sync, down to the second. You can use `sudo ntpdate pool.ntp.org` to sync the time of those machines. Afterwards you need to restart the services.
saavagebueno commented 2026-08-05 01:01:25 -04:00
Author
Owner
Copy Link

@Takalele commented on GitHub (Aug 13, 2024):

@pascal-fischer it was definitely a time synchronization issue. The host running Netbird was delayed by 2 seconds. Unfortunately, I don't have control over that host, so I had to migrate the entire setup to another cluster.

However, I'm now ready for debugging. The first thing I had to do was disable the network monitor due to Issue 2395

Client nat IP: eg .:10.8.2.36
Client Real-IP: eg.: 212.1.1.2

Peer/Exit Node Real-IP: 198.51.100.1

Client routing table:
routing table before the connection:

Aktive Routen:
     Netzwerkziel    Netzwerkmaske          Gateway    Schnittstelle Metrik
          0.0.0.0          0.0.0.0         10.8.2.1        10.8.2.36     40
         10.8.2.0    255.255.255.0   Auf Verbindung         10.8.2.36    296
        10.8.2.36  255.255.255.255   Auf Verbindung         10.8.2.36    296
       10.8.2.255  255.255.255.255   Auf Verbindung         10.8.2.36    296
        127.0.0.0        255.0.0.0   Auf Verbindung         127.0.0.1    331
        127.0.0.1  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
     172.22.240.0    255.255.240.0   Auf Verbindung      172.22.240.1   5256
     172.22.240.1  255.255.255.255   Auf Verbindung      172.22.240.1   5256
   172.22.255.255  255.255.255.255   Auf Verbindung      172.22.240.1   5256
     172.29.160.0    255.255.240.0   Auf Verbindung      172.29.160.1   5256
     172.29.160.1  255.255.255.255   Auf Verbindung      172.29.160.1   5256
   172.29.175.255  255.255.255.255   Auf Verbindung      172.29.160.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung         127.0.0.1    331
        224.0.0.0        240.0.0.0   Auf Verbindung         10.8.2.36    296
        224.0.0.0        240.0.0.0   Auf Verbindung      172.29.160.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung      172.22.240.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  255.255.255.255  255.255.255.255   Auf Verbindung         10.8.2.36    296
  255.255.255.255  255.255.255.255   Auf Verbindung      172.29.160.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung      172.22.240.1   5256

routing table during the connection:

Aktive Routen:
     Netzwerkziel    Netzwerkmaske          Gateway    Schnittstelle Metrik
          0.0.0.0          0.0.0.0         10.8.2.1        10.8.2.36     40
          0.0.0.0        128.0.0.0   Auf Verbindung   100.105.102.132      6
         10.8.2.0    255.255.255.0   Auf Verbindung         10.8.2.36    296
        10.8.2.36  255.255.255.255   Auf Verbindung         10.8.2.36    296
       10.8.2.255  255.255.255.255   Auf Verbindung         10.8.2.36    296
      100.105.0.0      255.255.0.0   Auf Verbindung   100.105.102.132    261
  100.105.102.132  255.255.255.255   Auf Verbindung   100.105.102.132    261
  100.105.255.255  255.255.255.255   Auf Verbindung   100.105.102.132    261
        127.0.0.0        255.0.0.0   Auf Verbindung         127.0.0.1    331
        127.0.0.1  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung   100.105.102.132    261
        128.0.0.0        128.0.0.0   Auf Verbindung   100.105.102.132      6
     172.22.240.0    255.255.240.0   Auf Verbindung      172.22.240.1   5256
     172.22.240.1  255.255.255.255   Auf Verbindung      172.22.240.1   5256
   172.22.255.255  255.255.255.255   Auf Verbindung      172.22.240.1   5256
     172.29.160.0    255.255.240.0   Auf Verbindung      172.29.160.1   5256
     172.29.160.1  255.255.255.255   Auf Verbindung      172.29.160.1   5256
   172.29.175.255  255.255.255.255   Auf Verbindung      172.29.160.1   5256
     192.168.48.0    255.255.240.0   Auf Verbindung      192.168.48.1   5256
     192.168.48.1  255.255.255.255   Auf Verbindung      192.168.48.1   5256
   192.168.63.255  255.255.255.255   Auf Verbindung      192.168.48.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung         127.0.0.1    331
        224.0.0.0        240.0.0.0   Auf Verbindung         10.8.2.36    296
        224.0.0.0        240.0.0.0   Auf Verbindung      172.29.160.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung      172.22.240.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung      192.168.48.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung   100.105.102.132    261
  255.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  255.255.255.255  255.255.255.255   Auf Verbindung         10.8.2.36    296
  255.255.255.255  255.255.255.255   Auf Verbindung      172.29.160.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung      172.22.240.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung      192.168.48.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung   100.105.102.132    261

Peer/Exit Node routing table:
routing table before the connection and during the connection:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 venet0
10.255.254.0    0.0.0.0         255.255.255.0   U     0      0        0 tap_soft-vpn
100.105.0.0     0.0.0.0         255.255.0.0     U     0      0        0 wt0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.255.0   U     0      0        0 proxy
198.51.100.0    0.0.0.0         255.255.255.0   U     0      0        0 venet0

During testing, I noticed that when the connection was terminated, the manually added route was automatically deleted.

BR
Takalele

netbird.debug.1315957774.zip

<!-- gh-comment-id:2287292797 --> @Takalele commented on GitHub (Aug 13, 2024): @pascal-fischer it was definitely a time synchronization issue. The host running Netbird was delayed by 2 seconds. Unfortunately, I don't have control over that host, so I had to migrate the entire setup to another cluster. However, I'm now ready for debugging. The first thing I had to do was disable the network monitor due to [Issue 2395](https://github.com/netbirdio/netbird/issues/2395) Client nat IP: eg .:10.8.2.36 Client Real-IP: eg.: 212.1.1.2 Peer/Exit Node Real-IP: 198.51.100.1 Client routing table: routing table before the connection: ``` Aktive Routen: Netzwerkziel Netzwerkmaske Gateway Schnittstelle Metrik 0.0.0.0 0.0.0.0 10.8.2.1 10.8.2.36 40 10.8.2.0 255.255.255.0 Auf Verbindung 10.8.2.36 296 10.8.2.36 255.255.255.255 Auf Verbindung 10.8.2.36 296 10.8.2.255 255.255.255.255 Auf Verbindung 10.8.2.36 296 127.0.0.0 255.0.0.0 Auf Verbindung 127.0.0.1 331 127.0.0.1 255.255.255.255 Auf Verbindung 127.0.0.1 331 127.255.255.255 255.255.255.255 Auf Verbindung 127.0.0.1 331 172.22.240.0 255.255.240.0 Auf Verbindung 172.22.240.1 5256 172.22.240.1 255.255.255.255 Auf Verbindung 172.22.240.1 5256 172.22.255.255 255.255.255.255 Auf Verbindung 172.22.240.1 5256 172.29.160.0 255.255.240.0 Auf Verbindung 172.29.160.1 5256 172.29.160.1 255.255.255.255 Auf Verbindung 172.29.160.1 5256 172.29.175.255 255.255.255.255 Auf Verbindung 172.29.160.1 5256 224.0.0.0 240.0.0.0 Auf Verbindung 127.0.0.1 331 224.0.0.0 240.0.0.0 Auf Verbindung 10.8.2.36 296 224.0.0.0 240.0.0.0 Auf Verbindung 172.29.160.1 5256 224.0.0.0 240.0.0.0 Auf Verbindung 172.22.240.1 5256 255.255.255.255 255.255.255.255 Auf Verbindung 127.0.0.1 331 255.255.255.255 255.255.255.255 Auf Verbindung 10.8.2.36 296 255.255.255.255 255.255.255.255 Auf Verbindung 172.29.160.1 5256 255.255.255.255 255.255.255.255 Auf Verbindung 172.22.240.1 5256 ``` routing table during the connection: ``` Aktive Routen: Netzwerkziel Netzwerkmaske Gateway Schnittstelle Metrik 0.0.0.0 0.0.0.0 10.8.2.1 10.8.2.36 40 0.0.0.0 128.0.0.0 Auf Verbindung 100.105.102.132 6 10.8.2.0 255.255.255.0 Auf Verbindung 10.8.2.36 296 10.8.2.36 255.255.255.255 Auf Verbindung 10.8.2.36 296 10.8.2.255 255.255.255.255 Auf Verbindung 10.8.2.36 296 100.105.0.0 255.255.0.0 Auf Verbindung 100.105.102.132 261 100.105.102.132 255.255.255.255 Auf Verbindung 100.105.102.132 261 100.105.255.255 255.255.255.255 Auf Verbindung 100.105.102.132 261 127.0.0.0 255.0.0.0 Auf Verbindung 127.0.0.1 331 127.0.0.1 255.255.255.255 Auf Verbindung 127.0.0.1 331 127.255.255.255 255.255.255.255 Auf Verbindung 127.0.0.1 331 127.255.255.255 255.255.255.255 Auf Verbindung 100.105.102.132 261 128.0.0.0 128.0.0.0 Auf Verbindung 100.105.102.132 6 172.22.240.0 255.255.240.0 Auf Verbindung 172.22.240.1 5256 172.22.240.1 255.255.255.255 Auf Verbindung 172.22.240.1 5256 172.22.255.255 255.255.255.255 Auf Verbindung 172.22.240.1 5256 172.29.160.0 255.255.240.0 Auf Verbindung 172.29.160.1 5256 172.29.160.1 255.255.255.255 Auf Verbindung 172.29.160.1 5256 172.29.175.255 255.255.255.255 Auf Verbindung 172.29.160.1 5256 192.168.48.0 255.255.240.0 Auf Verbindung 192.168.48.1 5256 192.168.48.1 255.255.255.255 Auf Verbindung 192.168.48.1 5256 192.168.63.255 255.255.255.255 Auf Verbindung 192.168.48.1 5256 224.0.0.0 240.0.0.0 Auf Verbindung 127.0.0.1 331 224.0.0.0 240.0.0.0 Auf Verbindung 10.8.2.36 296 224.0.0.0 240.0.0.0 Auf Verbindung 172.29.160.1 5256 224.0.0.0 240.0.0.0 Auf Verbindung 172.22.240.1 5256 224.0.0.0 240.0.0.0 Auf Verbindung 192.168.48.1 5256 224.0.0.0 240.0.0.0 Auf Verbindung 100.105.102.132 261 255.255.255.255 255.255.255.255 Auf Verbindung 127.0.0.1 331 255.255.255.255 255.255.255.255 Auf Verbindung 10.8.2.36 296 255.255.255.255 255.255.255.255 Auf Verbindung 172.29.160.1 5256 255.255.255.255 255.255.255.255 Auf Verbindung 172.22.240.1 5256 255.255.255.255 255.255.255.255 Auf Verbindung 192.168.48.1 5256 255.255.255.255 255.255.255.255 Auf Verbindung 100.105.102.132 261 ``` Peer/Exit Node routing table: routing table before the connection and during the connection: ``` Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 0.0.0.0 0.0.0.0 U 0 0 0 venet0 10.255.254.0 0.0.0.0 255.255.255.0 U 0 0 0 tap_soft-vpn 100.105.0.0 0.0.0.0 255.255.0.0 U 0 0 0 wt0 172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 172.18.0.0 0.0.0.0 255.255.255.0 U 0 0 0 proxy 198.51.100.0 0.0.0.0 255.255.255.0 U 0 0 0 venet0 ``` During testing, I noticed that when the connection was terminated, the manually added route was automatically deleted. BR Takalele [netbird.debug.1315957774.zip](https://github.com/user-attachments/files/16605985/netbird.debug.1315957774.zip)
saavagebueno commented 2026-08-05 01:01:25 -04:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Aug 14, 2024):

Hi @Takalele, for some reason your debug bundle does not contain trace/debug level logs. Could you run netbird debug log level trace and then run netbird debug for 2m -AS again, please?

Additionally, to help resolve the network monitor issue, could you provide Get-NetAdapter (from powershell), please?
We're interested in whether the virtual interface(s) share mac addresses with other interfaces so we can exclude them from the availability check.

<!-- gh-comment-id:2288478831 --> @lixmal commented on GitHub (Aug 14, 2024): Hi @Takalele, for some reason your debug bundle does not contain trace/debug level logs. Could you run ` netbird debug log level trace` and then run `netbird debug for 2m -AS` again, please? Additionally, to help resolve the network monitor issue, could you provide `Get-NetAdapter` (from powershell), please? We're interested in whether the virtual interface(s) share mac addresses with other interfaces so we can exclude them from the availability check.
saavagebueno commented 2026-08-05 01:01:25 -04:00
Author
Owner
Copy Link

@Takalele commented on GitHub (Aug 15, 2024):

Hi @lixmal, sure:

Microsoft Windows [Version 10.0.22631.3958]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\Windows\System32>netbird debug log level trace
Log level set successfully to trace

C:\Windows\System32>netbird up --network-monitor=false
Connected

C:\Windows\System32>netbird debug for 2m -AS
Netbird down
Netbird up
Remaining time: 00:00:01
Duration completed
Creating debug bundle...
C:\WINDOWS\SystemTemp\netbird.debug.3155538032.zip



C:\Windows\System32>netbird up --network-monitor=true
Connected

C:\Windows\System32>netbird debug for 2m -AS
Netbird down
Netbird up
Remaining time: 00:00:01
Duration completed
Creating debug bundle...
C:\WINDOWS\SystemTemp\netbird.debug.1750120769.zip

Get-NetAdapter

PS C:\WINDOWS\system32> Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
Ethernet 2                Surface Ethernet Adapter                     30 Disconnected 1C-1A-DF-BF-73-7A          0 bps
WLAN                      Intel(R) Wi-Fi 6 AX200 160MHz                22 Up           6C-A1-00-53-DA-B2       300 Mbps
Bluetooth-Netzwerkverb... Bluetooth Device (Personal Area Netw...       9 Disconnected 6C-A1-00-53-DA-B6         3 Mbps
vEthernet (WLAN)          Hyper-V Virtual Ethernet Adapter #3           5 Disconnected 6C-A1-00-53-DA-B2        10 Gbps
VPN - VPN Client          VPN Client Adapter - VPN                      4 Disconnected 5E-87-00-E0-98-40       100 Mbps

logs:

netbird.debug.1750120769_network-monitor=true.zip
netbird.debug.3155538032_network-monitor=false.zip

<!-- gh-comment-id:2291179397 --> @Takalele commented on GitHub (Aug 15, 2024): Hi @lixmal, sure: ``` Microsoft Windows [Version 10.0.22631.3958] (c) Microsoft Corporation. Alle Rechte vorbehalten. C:\Windows\System32>netbird debug log level trace Log level set successfully to trace C:\Windows\System32>netbird up --network-monitor=false Connected C:\Windows\System32>netbird debug for 2m -AS Netbird down Netbird up Remaining time: 00:00:01 Duration completed Creating debug bundle... C:\WINDOWS\SystemTemp\netbird.debug.3155538032.zip C:\Windows\System32>netbird up --network-monitor=true Connected C:\Windows\System32>netbird debug for 2m -AS Netbird down Netbird up Remaining time: 00:00:01 Duration completed Creating debug bundle... C:\WINDOWS\SystemTemp\netbird.debug.1750120769.zip ``` Get-NetAdapter ``` PS C:\WINDOWS\system32> Get-NetAdapter Name InterfaceDescription ifIndex Status MacAddress LinkSpeed ---- -------------------- ------- ------ ---------- --------- Ethernet 2 Surface Ethernet Adapter 30 Disconnected 1C-1A-DF-BF-73-7A 0 bps WLAN Intel(R) Wi-Fi 6 AX200 160MHz 22 Up 6C-A1-00-53-DA-B2 300 Mbps Bluetooth-Netzwerkverb... Bluetooth Device (Personal Area Netw... 9 Disconnected 6C-A1-00-53-DA-B6 3 Mbps vEthernet (WLAN) Hyper-V Virtual Ethernet Adapter #3 5 Disconnected 6C-A1-00-53-DA-B2 10 Gbps VPN - VPN Client VPN Client Adapter - VPN 4 Disconnected 5E-87-00-E0-98-40 100 Mbps ``` logs: [netbird.debug.1750120769_network-monitor=true.zip](https://github.com/user-attachments/files/16625119/netbird.debug.1750120769_network-monitor.true.zip) [netbird.debug.3155538032_network-monitor=false.zip](https://github.com/user-attachments/files/16625120/netbird.debug.3155538032_network-monitor.false.zip)
saavagebueno commented 2026-08-05 01:01:25 -04:00
Author
Owner
Copy Link

@Takalele commented on GitHub (Aug 15, 2024):

@lixmal @pascal-fischer, The client must also add a host route for the TURN server; otherwise, it won't function properly if the TURN server is needed. This is particularly crucial in the case of an exit node or default route, as in such scenarios, the TURN server traffic will also be routed through the tunnel. To ensure that TURN works correctly, its traffic must not pass through the tunnel.

<!-- gh-comment-id:2291238397 --> @Takalele commented on GitHub (Aug 15, 2024): @lixmal @pascal-fischer, The client must also add a host route for the TURN server; otherwise, it won't function properly if the TURN server is needed. This is particularly crucial in the case of an exit node or default route, as in such scenarios, the TURN server traffic will also be routed through the tunnel. To ensure that TURN works correctly, its traffic must not pass through the tunnel.
saavagebueno commented 2026-08-05 01:01:25 -04:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Aug 16, 2024):

Hi @Takalele,

thanks for the interfaces, that's helpful.
Regarding the routes, there are the following logs

2024-08-15T14:07:43+02:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 198.51.100.1:51820 for ddytZXiJzeKPzAhi
                                                                           ^-- our peer's IP
                                                                           
2024-08-15T14:07:43+02:00 DEBG util/net/listener_nonios.go:120: Listener resolved IP for 198.51.100.1:3478: 198.51.100.1
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 0 for prefix 198.51.100.1/32 with [{invalid IP <nil>}]
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:58: Adding for prefix 198.51.100.1/32 with [{invalid IP <nil>}]
2024-08-15T14:07:43+02:00 DEBG client/internal/routemanager/systemops/systemops_generic.go:167: Adding a new route for prefix 198.51.100.1/32 with next hop 10.8.2.1
2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:603: discovered local candidate udp4 srflx 198.51.100.2:18212 related 0.0.0.0:51820
2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:633: selected candidate pair [local <-> remote] -> [udp4 host 127.0.0.1:51820 <-> udp4 host 198.51.100.1:51820], peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=
2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:639: peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs= ICE ConnectionState has changed to Connected
2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:412: Conn resolved IP for 198.51.100.1:51820: 198.51.100.1
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 198.51.100.1/32 10.8.2.1 if 5:  OK!
                                                                                                                                          ^-- here the peer IP route was added, pointing to the default gw

2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 1 for prefix 198.51.100.1/32 with [{10.8.2.1 0xc000740540}]
2024-08-15T14:07:43+02:00 DEBG iface/iface.go:92: updating interface wt0 peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=, endpoint 198.51.100.1:51820
2024-08-15T14:07:43+02:00 INFO client/internal/peer/conn.go:362: connected to peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=, endpoint address: 198.51.100.1:51820

2024-08-15T14:07:43+02:00 INFO client/internal/routemanager/client.go:177: New chosen route is cqtt443iojjd2cor2ocg with peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs= with score 2.000000 for network [0.0.0.0/0]
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 0 for prefix 0.0.0.0/0 with [<nil>]
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:58: Adding for prefix 0.0.0.0/0 with [<nil>]
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 0.0.0.0/1 0.0.0.0 if 83:  OK!
2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 198.51.100.3/1 0.0.0.0 if 83:  OK!
2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add ::/1 :: if 83:  OK!
2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 8000::/1 :: if 83:  OK!
                                                                                                                              ^-- here the default routes were added to the vpn interface

2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 0 for prefix 0.0.0.0/0 with []
2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:58: Adding for prefix 0.0.0.0/0 with []

So everything seems in order and I don't see any errors related to that.
Is it possible something else keeps deleting these routes while netbird is active?

<!-- gh-comment-id:2294412056 --> @lixmal commented on GitHub (Aug 16, 2024): Hi @Takalele, thanks for the interfaces, that's helpful. Regarding the routes, there are the following logs ``` 2024-08-15T14:07:43+02:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 198.51.100.1:51820 for ddytZXiJzeKPzAhi ^-- our peer's IP 2024-08-15T14:07:43+02:00 DEBG util/net/listener_nonios.go:120: Listener resolved IP for 198.51.100.1:3478: 198.51.100.1 2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 0 for prefix 198.51.100.1/32 with [{invalid IP <nil>}] 2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:58: Adding for prefix 198.51.100.1/32 with [{invalid IP <nil>}] 2024-08-15T14:07:43+02:00 DEBG client/internal/routemanager/systemops/systemops_generic.go:167: Adding a new route for prefix 198.51.100.1/32 with next hop 10.8.2.1 2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:603: discovered local candidate udp4 srflx 198.51.100.2:18212 related 0.0.0.0:51820 2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:633: selected candidate pair [local <-> remote] -> [udp4 host 127.0.0.1:51820 <-> udp4 host 198.51.100.1:51820], peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs= 2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:639: peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs= ICE ConnectionState has changed to Connected 2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:412: Conn resolved IP for 198.51.100.1:51820: 198.51.100.1 2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 198.51.100.1/32 10.8.2.1 if 5: OK! ^-- here the peer IP route was added, pointing to the default gw 2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 1 for prefix 198.51.100.1/32 with [{10.8.2.1 0xc000740540}] 2024-08-15T14:07:43+02:00 DEBG iface/iface.go:92: updating interface wt0 peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=, endpoint 198.51.100.1:51820 2024-08-15T14:07:43+02:00 INFO client/internal/peer/conn.go:362: connected to peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=, endpoint address: 198.51.100.1:51820 2024-08-15T14:07:43+02:00 INFO client/internal/routemanager/client.go:177: New chosen route is cqtt443iojjd2cor2ocg with peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs= with score 2.000000 for network [0.0.0.0/0] 2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 0 for prefix 0.0.0.0/0 with [<nil>] 2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:58: Adding for prefix 0.0.0.0/0 with [<nil>] 2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 0.0.0.0/1 0.0.0.0 if 83: OK! 2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 198.51.100.3/1 0.0.0.0 if 83: OK! 2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add ::/1 :: if 83: OK! 2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 8000::/1 :: if 83: OK! ^-- here the default routes were added to the vpn interface 2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 0 for prefix 0.0.0.0/0 with [] 2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:58: Adding for prefix 0.0.0.0/0 with [] ``` So everything seems in order and I don't see any errors related to that. Is it possible something else keeps deleting these routes while netbird is active?
saavagebueno commented 2026-08-05 01:01:25 -04:00
Author
Owner
Copy Link

@Takalele commented on GitHub (Aug 17, 2024):

Hi @lixmal,

I created a small PowerShell script to monitor the network routes (see below) in order to debug this issue further. You were right — there are routes being added, but they're being created on the wrong interface (ifIndex 80). In my case, the correct index would be 16. From the debugging, I discovered that this happens because, for some unknown Hyper-V reason, both interfaces have the same MAC address, even though no bridging is currently configured. When bridging is configured, there are three network interfaces with the same MAC address. In my tests, with bridging enabled, the routes were created correctly, but to prevent future issues, it could be good to use the ifIndex when creating the routes on windows systems.

After playing around with the Hyper-V virtual switch settings, i can no longer reproduce the issue, so it seems like this was a Hyper-V bug that has already been fixed.

Thanks for your help!

BR
Takalele

Bad ifindex routes:

[2024-08-17 18:14:10] Route added: DestinationPrefix: 255.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 224.0.0.0/4, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 100.105.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 100.105.102.132/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 100.105.0.0/16, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 198.51.100.3/32, NextHop: 10.8.2.1, InterfaceIndex: 5, InterfaceAlias: vEthernet (WLAN)
[2024-08-17 18:14:10] Route added: DestinationPrefix: ff00::/8, NextHop: ::, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:16] Route added: DestinationPrefix: 198.51.100.1/32, NextHop: 10.8.2.1, InterfaceIndex: 5, InterfaceAlias: vEthernet (WLAN)
[2024-08-17 18:14:16] Route added: DestinationPrefix: 128.0.0.0/1, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:16] Route added: DestinationPrefix: 127.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0

Good ifindex routes:

[2024-08-17 18:12:17] Route added: DestinationPrefix: 255.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 224.0.0.0/4, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 100.105.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 100.105.102.132/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 100.105.0.0/16, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 198.51.100.3/32, NextHop: 10.8.2.1, InterfaceIndex: 16, InterfaceAlias: WLAN
[2024-08-17 18:12:17] Route added: DestinationPrefix: ff00::/8, NextHop: ::, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:22] Route added: DestinationPrefix: 198.51.100.1/32, NextHop: 10.8.2.1, InterfaceIndex: 16, InterfaceAlias: WLAN
[2024-08-17 18:12:23] Route added: DestinationPrefix: 128.0.0.0/1, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:23] Route added: DestinationPrefix: 127.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:23] Route added: DestinationPrefix: 8000::/1, NextHop: ::, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:23] Route added: DestinationPrefix: ::/1, NextHop: ::, InterfaceIndex: 37, InterfaceAlias: wt0

Get-NetAdapter with the issue

PS C:\WINDOWS\system32> Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
WLAN                      Intel(R) Wi-Fi 6 AX200 160MHz                16 Up           6C-A1-00-53-DA-B2       300 Mbps
vEthernet (WLAN)          Hyper-V Virtual Ethernet Adapter #3           80 Disconnected 6C-A1-00-53-DA-B2        10 Gbps

Get-NetAdapter with Bridging Enabled

PS C:\WINDOWS\system32> Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
vEthernet (WLAN)          Hyper-V Virtual Ethernet Adapter #3          44 Up           6C-A1-00-53-DA-B2       300 Mbps
WLAN                      Intel(R) Wi-Fi 6 AX200 160MHz                16 Up           6C-A1-00-53-DA-B2       300 Mbps
Netzwerkbrücke            Microsoft Network Adapter Multiplexo...      38 Up           6C-A1-00-53-DA-B2       300 Mbps

MonitorRouteChanges.ps1:

$logFile = "C:\route_changes.txt"
$previousRoutes = Get-NetRoute

while ($true) {
    $currentRoutes = Get-NetRoute
    $addedRoutes = Compare-Object $previousRoutes $currentRoutes -Property DestinationPrefix,NextHop,InterfaceIndex,InterfaceAlias -PassThru | Where-Object { $_.SideIndicator -eq '=>' }
    $removedRoutes = Compare-Object $previousRoutes $currentRoutes -Property DestinationPrefix,NextHop,InterfaceIndex,InterfaceAlias -PassThru | Where-Object { $_.SideIndicator -eq '<=' }

    $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
    
    if ($addedRoutes) {
        $addedRoutes | ForEach-Object {
            $routeInfo = "[$timestamp] Route added: DestinationPrefix: $($_.DestinationPrefix), NextHop: $($_.NextHop), InterfaceIndex: $($_.InterfaceIndex), InterfaceAlias: $($_.InterfaceAlias)"
            Add-Content -Path $logFile -Value $routeInfo
        }
    }

    if ($removedRoutes) {
        $removedRoutes | ForEach-Object {
            $routeInfo = "[$timestamp] Route removed: DestinationPrefix: $($_.DestinationPrefix), NextHop: $($_.NextHop), InterfaceIndex: $($_.InterfaceIndex), InterfaceAlias: $($_.InterfaceAlias)"
            Add-Content -Path $logFile -Value $routeInfo
        }
    }

    $previousRoutes = $currentRoutes
    Start-Sleep -Seconds 1
}
<!-- gh-comment-id:2294925706 --> @Takalele commented on GitHub (Aug 17, 2024): Hi @lixmal, I created a small PowerShell script to monitor the network routes (see below) in order to debug this issue further. You were right — there are routes being added, but they're being created on the wrong interface (ifIndex 80). In my case, the correct index would be 16. From the debugging, I discovered that this happens because, for some unknown Hyper-V reason, both interfaces have the same MAC address, even though no bridging is currently configured. When bridging is configured, there are three network interfaces with the same MAC address. In my tests, with bridging enabled, the routes were created correctly, but to prevent future issues, it could be good to use the ifIndex when creating the routes on windows systems. After playing around with the Hyper-V virtual switch settings, i can no longer reproduce the issue, so it seems like this was a Hyper-V bug that has already been fixed. Thanks for your help! BR Takalele Bad ifindex routes: ``` [2024-08-17 18:14:10] Route added: DestinationPrefix: 255.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0 [2024-08-17 18:14:10] Route added: DestinationPrefix: 224.0.0.0/4, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0 [2024-08-17 18:14:10] Route added: DestinationPrefix: 100.105.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0 [2024-08-17 18:14:10] Route added: DestinationPrefix: 100.105.102.132/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0 [2024-08-17 18:14:10] Route added: DestinationPrefix: 100.105.0.0/16, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0 [2024-08-17 18:14:10] Route added: DestinationPrefix: 198.51.100.3/32, NextHop: 10.8.2.1, InterfaceIndex: 5, InterfaceAlias: vEthernet (WLAN) [2024-08-17 18:14:10] Route added: DestinationPrefix: ff00::/8, NextHop: ::, InterfaceIndex: 80, InterfaceAlias: wt0 [2024-08-17 18:14:16] Route added: DestinationPrefix: 198.51.100.1/32, NextHop: 10.8.2.1, InterfaceIndex: 5, InterfaceAlias: vEthernet (WLAN) [2024-08-17 18:14:16] Route added: DestinationPrefix: 128.0.0.0/1, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0 [2024-08-17 18:14:16] Route added: DestinationPrefix: 127.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0 ``` Good ifindex routes: ``` [2024-08-17 18:12:17] Route added: DestinationPrefix: 255.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0 [2024-08-17 18:12:17] Route added: DestinationPrefix: 224.0.0.0/4, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0 [2024-08-17 18:12:17] Route added: DestinationPrefix: 100.105.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0 [2024-08-17 18:12:17] Route added: DestinationPrefix: 100.105.102.132/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0 [2024-08-17 18:12:17] Route added: DestinationPrefix: 100.105.0.0/16, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0 [2024-08-17 18:12:17] Route added: DestinationPrefix: 198.51.100.3/32, NextHop: 10.8.2.1, InterfaceIndex: 16, InterfaceAlias: WLAN [2024-08-17 18:12:17] Route added: DestinationPrefix: ff00::/8, NextHop: ::, InterfaceIndex: 37, InterfaceAlias: wt0 [2024-08-17 18:12:22] Route added: DestinationPrefix: 198.51.100.1/32, NextHop: 10.8.2.1, InterfaceIndex: 16, InterfaceAlias: WLAN [2024-08-17 18:12:23] Route added: DestinationPrefix: 128.0.0.0/1, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0 [2024-08-17 18:12:23] Route added: DestinationPrefix: 127.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0 [2024-08-17 18:12:23] Route added: DestinationPrefix: 8000::/1, NextHop: ::, InterfaceIndex: 37, InterfaceAlias: wt0 [2024-08-17 18:12:23] Route added: DestinationPrefix: ::/1, NextHop: ::, InterfaceIndex: 37, InterfaceAlias: wt0 ``` Get-NetAdapter with the issue ``` PS C:\WINDOWS\system32> Get-NetAdapter Name InterfaceDescription ifIndex Status MacAddress LinkSpeed ---- -------------------- ------- ------ ---------- --------- WLAN Intel(R) Wi-Fi 6 AX200 160MHz 16 Up 6C-A1-00-53-DA-B2 300 Mbps vEthernet (WLAN) Hyper-V Virtual Ethernet Adapter #3 80 Disconnected 6C-A1-00-53-DA-B2 10 Gbps ``` Get-NetAdapter with Bridging Enabled ``` PS C:\WINDOWS\system32> Get-NetAdapter Name InterfaceDescription ifIndex Status MacAddress LinkSpeed ---- -------------------- ------- ------ ---------- --------- vEthernet (WLAN) Hyper-V Virtual Ethernet Adapter #3 44 Up 6C-A1-00-53-DA-B2 300 Mbps WLAN Intel(R) Wi-Fi 6 AX200 160MHz 16 Up 6C-A1-00-53-DA-B2 300 Mbps Netzwerkbrücke Microsoft Network Adapter Multiplexo... 38 Up 6C-A1-00-53-DA-B2 300 Mbps ``` MonitorRouteChanges.ps1: ``` $logFile = "C:\route_changes.txt" $previousRoutes = Get-NetRoute while ($true) { $currentRoutes = Get-NetRoute $addedRoutes = Compare-Object $previousRoutes $currentRoutes -Property DestinationPrefix,NextHop,InterfaceIndex,InterfaceAlias -PassThru | Where-Object { $_.SideIndicator -eq '=>' } $removedRoutes = Compare-Object $previousRoutes $currentRoutes -Property DestinationPrefix,NextHop,InterfaceIndex,InterfaceAlias -PassThru | Where-Object { $_.SideIndicator -eq '<=' } $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss" if ($addedRoutes) { $addedRoutes | ForEach-Object { $routeInfo = "[$timestamp] Route added: DestinationPrefix: $($_.DestinationPrefix), NextHop: $($_.NextHop), InterfaceIndex: $($_.InterfaceIndex), InterfaceAlias: $($_.InterfaceAlias)" Add-Content -Path $logFile -Value $routeInfo } } if ($removedRoutes) { $removedRoutes | ForEach-Object { $routeInfo = "[$timestamp] Route removed: DestinationPrefix: $($_.DestinationPrefix), NextHop: $($_.NextHop), InterfaceIndex: $($_.InterfaceIndex), InterfaceAlias: $($_.InterfaceAlias)" Add-Content -Path $logFile -Value $routeInfo } } $previousRoutes = $currentRoutes Start-Sleep -Seconds 1 } ```
saavagebueno commented 2026-08-05 01:01:25 -04:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Aug 17, 2024):

Thanks for digging into that. We're using GetBestRoute2 to figure out the default route. So if windows reports the wrong thing that might be a bug there indeed.
Let us know if you're able to reproduce it.

<!-- gh-comment-id:2294928956 --> @lixmal commented on GitHub (Aug 17, 2024): Thanks for digging into that. We're using [GetBestRoute2](https://learn.microsoft.com/en-us/windows/win32/api/netioapi/nf-netioapi-getbestroute2) to figure out the default route. So if windows reports the wrong thing that might be a bug there indeed. Let us know if you're able to reproduce it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2021 Q4
2021 Q4
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
accessibility
accessibility
accessibility
acl
acl
acl
agent
agent
agent
agent
agent
agent
Android
Android
Android
Android
Android
Android
api
api
api
authentik
authentik
authentik
automation
automation
automation
azure
azure
azure
battery-usage
battery-usage
battery-usage
bug
cache
cache
cache
client
client
client
client-ui
client-ui
client-ui
cloud
cloud
cloud
cloud-only
cloud-only
cloud-only
cloudflare
cloudflare
cloudflare
community
community
community
compatibility
compatibility
compatibility
config-idp
config-idp
config-idp
config-issue
config-issue
config-issue
connection
connection
connection
contribution
contribution
contribution
coturn
coturn
coturn
cross-vpn
cross-vpn
cross-vpn
dashboard
dashboard
dashboard
data-usage
data-usage
data-usage
distribution
distribution
distribution
dns
dns
dns
docker
docker
docker
documentation
documentation
documentation
duplicate
duplicate
duplicate
enhancement
enhancement
event-stream
event-stream
event-stream
feature-request
feature-request
feature-request
freebsd
freebsd
freebsd
getting-started
getting-started
getting-started
go
go
go
good first issue
good first issue
good first issue
gui
gui
gui
help wanted
help wanted
help wanted
home-assistant
home-assistant
home-assistant
idp
idp
idp
inconsistency
inconsistency
inconsistency
integration
integration
integration
integrations
integrations
integrations
ios
ios
ios
ipv6
ipv6
ipv6
jwt
jwt
jwt
k8s
k8s
k8s
keycloak
keycloak
keycloak
linux
linux
linux
login
login
login
macos
macos
macos
management-service
management-service
management-service
Medium
Medium
Medium
missing-docs
missing-docs
missing-docs
mobile
mobile
mobile
moved-internal
moved-internal
moved-internal
needs-review
needs-review
needs-review
netbird-ui
netbird-ui
netbird-ui
networking
networking
networking
new-platform
new-platform
new-platform
nginx
nginx
nginx
notification
notification
notification
okta
okta
okta
openwrt
openwrt
openwrt
P2
P2
P2
packaging
packaging
packaging
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
performance
performance
performance
postgres
postgres
postgres
posture-checks
posture-checks
posture-checks
psk
psk
psk
pull-request
Mirrored from GitHub Pull Request
 question
question
question
refactor
refactor
refactor
relay
relay
relay
release
release
release
rfc
rfc
rfc
routes
routes
routes
security
security
security
security-improvement
security-improvement
security-improvement
security-related
security-related
security-related
self-hosting
self-hosting
self-hosting
server
server
server
signal
signal
signal
sleep-issue
sleep-issue
sleep-issue
ssh
ssh
ssh
ssl
ssl
ssl
status
status
status
store
store
store
synology
synology
synology
system-compatibility-issue
system-compatibility-issue
system-compatibility-issue
test-suite
test-suite
test-suite
third-party-integration
third-party-integration
third-party-integration
triage
triage
triage
triage
triage
triage
triage-needed
triage-needed
triage-needed
troubleshooting
troubleshooting
troubleshooting
UX
UX
UX
waiting-feedback
waiting-feedback
waiting-feedback
windows
windows
windows
wontfix
wontfix
wontfix
zitadel
zitadel
zitadel
No Label client routes
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dynr-servers saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#5108
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?