[GH-ISSUE #4476] Traffic diverting to Relay and intermittent service #8730

Open
opened 2026-08-05 01:19:19 -04:00 by saavagebueno · 0 comments
Owner

Originally created by @Josefellype on GitHub (Sep 10, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/4476

Scenario

  • BRIDGE-TO-MGMT
    • Router whose backbone routes are behind it, with Netbird running
    • Netbird IP: 10.255.255.32
    • Agent version: 0.56.1
  • Zabbix Proxy
    • Container with Netbird running
    • Netbird IP: 10.255.255.151
    • Agent version: 0.55.1
  • Backbone networks
    • 100.76.0.0/16
    • 172.16.0.0/12
  • Zabbix Server address: 100.76.180.207

Objective

Make the host 10.255.255.151 (Zabbix proxy) communicate with all backbone addresses, not just 100.76.0.0/16, including 172.16.0.0/12 and other networks reachable via routing performed by 10.255.255.32.

Example of expected behavior (direct traceroute via BRIDGE-TO-MGMT):

Source: 10.255.255.151

root@b27850a60f04:/var/lib/zabbix# traceroute 100.76.180.207
traceroute to 100.76.180.207 (100.76.180.207), 30 hops max, 60 byte packets
1  [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32)  0.621 ms  0.607 ms  0.596 ms
2  100.76.180.207 (100.76.180.207)  4.323 ms  4.331 ms  4.334 ms

The goal is for the traffic to go from 10.255.255.151 to 10.255.255.32 over the point-to-point WireGuard connection and then be routed to 100.76.180.207, which is behind 10.255.255.32. As shown in the documentation:

Image

Observed issue

When using the network route 100.76.0.0/16 on the BRIDGE-TO-MGMT peer, the expected behavior described in the objective does not hold; at several moments the traffic goes through the Relay Server.

Creating the 100.76.0.0/16 network route

Image

Traceroute after the change:

root@b27850a60f04:/var/lib/zabbix# traceroute 100.76.180.207
traceroute to 100.76.180.207 (100.76.180.207), 30 hops max, 60 byte packets
1  [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32)  87.933 ms  89.622 ms  89.954 ms
2  100.76.180.207 (100.76.180.207)  94.145 ms  94.165 ms  94.150 ms

Note: there is no Relay Server address that appears in the traceroutes; latency is the evidence that makes it clear the traffic is going through the Relay Server. The source and destination hosts are within the same ISP, which is why in the earlier traceroute (from the objective) the latency to the destination is 4 ms.

Test with a more specific route for 100.76.180.207

  • With 100.76.180.207/32: objective achieved
  • Smallest prefix that worked stably: 100.76.180.192/27
  • With /26 the problem occurs again
Image

Traceroute after using 100.76.180.192/27:

root@b27850a60f04:/var/lib/zabbix# traceroute 100.76.180.207
traceroute to 100.76.180.207 (100.76.180.207), 30 hops max, 60 byte packets
1  [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32)  0.626 ms  0.617 ms  0.596 ms
2  100.76.180.207 (100.76.180.207)  4.215 ms  4.200 ms  4.113 ms

Access to other backbone networks (172.16.0.0/12)

The Zabbix proxy also needs to access networks in the 172.16.0.0/12 block. The same problem occurs when a broad route is used.

Test with 172.16.0.0/12

Applied configuration: network 172.16.0.0/12

Image

Traceroute after the configuration:

root@b27850a60f04:/var/lib/zabbix# traceroute 172.16.243.254
traceroute to 172.16.243.254 (172.16.243.254), 30 hops max, 60 byte packets
1  [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32)  81.935 ms  81.946 ms  81.958 ms
2  100.76.180.193 (100.76.180.193)  82.288 ms  82.286 ms  82.275 ms
3  100.76.1.5 (100.76.1.5)  83.032 ms  83.415 ms  103.106 ms
4  100.76.4.6 (100.76.4.6)  154.128 ms  154.134 ms  154.122 ms
5  172.16.4.30 (172.16.4.30)  124.695 ms 100.76.4.14 (100.76.4.14)  127.622 ms  127.751 ms
6  172.17.14.10 (172.17.14.10)  127.130 ms  85.272 ms  86.105 ms
7  172.17.24.10 (172.17.24.10)  87.614 ms  87.822 ms  87.415 ms
8  100.75.24.49 (100.75.24.49)  88.615 ms  88.013 ms  88.537 ms
9  100.75.24.78 (100.75.24.78)  88.613 ms  89.099 ms  89.211 ms
10  100.75.89.6 (100.75.89.6)  90.430 ms  91.390 ms  90.981 ms
11  100.75.90.14 (100.75.90.14)  96.894 ms  96.888 ms  96.863 ms
12  100.75.90.6 (100.75.90.6)  92.232 ms  92.015 ms  92.197 ms
13  * * *
14  172.16.243.254 (172.16.243.254)  93.251 ms  93.119 ms  92.606 ms

Attempts with prefixes for 172.16.243.254 (unsuccessful)

  • 172.16.243.0/24
    Image

  • 172.16.243.192/26
    Image

  • 172.16.243.224/27
    Image

  • 172.16.243.254/32
    Image


Target change: 172.16.9.78

A few minutes earlier, a network for 172.16.9.78/32 had been created and the traceroute came out as expected. The rule was then disabled to test a less specific prefix.

Traceroute with 172.16.9.78/32 (OK result):

root@b27850a60f04:/var/lib/zabbix# traceroute 172.16.9.78
traceroute to 172.16.9.78 (172.16.9.78), 30 hops max, 60 byte packets
1  [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32)  0.525 ms  0.494 ms  0.504 ms
2  100.76.180.193 (100.76.180.193)  0.660 ms  0.634 ms  0.710 ms
3  172.16.9.78 (172.16.9.78)  0.895 ms  0.951 ms  0.965 ms 

Attempt with 172.16.9.0/24:

root@b27850a60f04:/var/lib/zabbix# traceroute 172.16.9.78
traceroute to 172.16.9.78 (172.16.9.78), 30 hops max, 60 byte packets
1  [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32)  81.460 ms  83.221 ms  83.236 ms
2  100.76.180.193 (100.76.180.193)  93.814 ms  93.817 ms  93.809 ms
3  172.16.9.78 (172.16.9.78)  93.838 ms  93.834 ms  93.826 ms 

Upon reverting to 172.16.9.78/32 and restarting the Netbird service on the Zabbix proxy, the traffic started going through the Relay Server:
Image

The same happens with 100.76.180.207 even with 100.76.180.207/32, the packets go through the Relay Server:
Image

Another demonstration of intermittency:
Image

The only change made was the addition of the 100.76.0.0/17 network right before bringing the Netbird service down and up again, which would explain the change in traceroute behavior for 100.76.4.255, but not for the remaining addresses.


Summary of behavior

  • Very specific prefixes (/32 and, in one case, /27) tend to work initially.
  • With less specific prefixes (/26, /16, /12), traffic often detours via the Relay Server.
  • There is intermittency: the same specific prefix can change behavior after route changes, even if the configuration returns to the initial pre-change state.

Originally created by @Josefellype on GitHub (Sep 10, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/4476 ## Scenario - BRIDGE-TO-MGMT - Router whose backbone routes are behind it, with Netbird running - Netbird IP: 10.255.255.32 - Agent version: 0.56.1 - Zabbix Proxy - Container with Netbird running - Netbird IP: 10.255.255.151 - Agent version: 0.55.1 - Backbone networks - 100.76.0.0/16 - 172.16.0.0/12 - Zabbix Server address: 100.76.180.207 ## Objective Make the host 10.255.255.151 (Zabbix proxy) communicate with all backbone addresses, not just 100.76.0.0/16, including 172.16.0.0/12 and other networks reachable via routing performed by 10.255.255.32. Example of expected behavior (direct traceroute via BRIDGE-TO-MGMT): Source: 10.255.255.151 ```bash root@b27850a60f04:/var/lib/zabbix# traceroute 100.76.180.207 traceroute to 100.76.180.207 (100.76.180.207), 30 hops max, 60 byte packets 1 [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32) 0.621 ms 0.607 ms 0.596 ms 2 100.76.180.207 (100.76.180.207) 4.323 ms 4.331 ms 4.334 ms ``` The goal is for the traffic to go from 10.255.255.151 to 10.255.255.32 over the point-to-point WireGuard connection and then be routed to 100.76.180.207, which is behind 10.255.255.32. As shown in the documentation: <img width="775" height="453" alt="Image" src="https://github.com/user-attachments/assets/38561d4f-7664-463a-8431-50fe31c436ff" /> --- ## Observed issue When using the network route 100.76.0.0/16 on the BRIDGE-TO-MGMT peer, the expected behavior described in the objective does not hold; at several moments the traffic goes through the Relay Server. ### Creating the 100.76.0.0/16 network route <img width="1191" height="1043" alt="Image" src="https://github.com/user-attachments/assets/10e8a81a-f206-4474-b58b-5df74b5ee2a6" /> Traceroute after the change: ```bash root@b27850a60f04:/var/lib/zabbix# traceroute 100.76.180.207 traceroute to 100.76.180.207 (100.76.180.207), 30 hops max, 60 byte packets 1 [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32) 87.933 ms 89.622 ms 89.954 ms 2 100.76.180.207 (100.76.180.207) 94.145 ms 94.165 ms 94.150 ms ``` Note: there is no Relay Server address that appears in the traceroutes; latency is the evidence that makes it clear the traffic is going through the Relay Server. The source and destination hosts are within the same ISP, which is why in the earlier traceroute (from the objective) the latency to the destination is 4 ms. ### Test with a more specific route for 100.76.180.207 - With 100.76.180.207/32: objective achieved - Smallest prefix that worked stably: 100.76.180.192/27 - With /26 the problem occurs again <img width="1271" height="907" alt="Image" src="https://github.com/user-attachments/assets/e4e7f6db-1d7a-44a0-af9f-c908fc43204d" /> Traceroute after using 100.76.180.192/27: ```bash root@b27850a60f04:/var/lib/zabbix# traceroute 100.76.180.207 traceroute to 100.76.180.207 (100.76.180.207), 30 hops max, 60 byte packets 1 [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32) 0.626 ms 0.617 ms 0.596 ms 2 100.76.180.207 (100.76.180.207) 4.215 ms 4.200 ms 4.113 ms ``` --- ## Access to other backbone networks (172.16.0.0/12) The Zabbix proxy also needs to access networks in the 172.16.0.0/12 block. The same problem occurs when a broad route is used. ### Test with 172.16.0.0/12 Applied configuration: network 172.16.0.0/12 <img width="1188" height="1083" alt="Image" src="https://github.com/user-attachments/assets/d8cfa3dc-586d-4d4a-8220-64e3e3dc49a9" /> Traceroute after the configuration: ```bash root@b27850a60f04:/var/lib/zabbix# traceroute 172.16.243.254 traceroute to 172.16.243.254 (172.16.243.254), 30 hops max, 60 byte packets 1 [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32) 81.935 ms 81.946 ms 81.958 ms 2 100.76.180.193 (100.76.180.193) 82.288 ms 82.286 ms 82.275 ms 3 100.76.1.5 (100.76.1.5) 83.032 ms 83.415 ms 103.106 ms 4 100.76.4.6 (100.76.4.6) 154.128 ms 154.134 ms 154.122 ms 5 172.16.4.30 (172.16.4.30) 124.695 ms 100.76.4.14 (100.76.4.14) 127.622 ms 127.751 ms 6 172.17.14.10 (172.17.14.10) 127.130 ms 85.272 ms 86.105 ms 7 172.17.24.10 (172.17.24.10) 87.614 ms 87.822 ms 87.415 ms 8 100.75.24.49 (100.75.24.49) 88.615 ms 88.013 ms 88.537 ms 9 100.75.24.78 (100.75.24.78) 88.613 ms 89.099 ms 89.211 ms 10 100.75.89.6 (100.75.89.6) 90.430 ms 91.390 ms 90.981 ms 11 100.75.90.14 (100.75.90.14) 96.894 ms 96.888 ms 96.863 ms 12 100.75.90.6 (100.75.90.6) 92.232 ms 92.015 ms 92.197 ms 13 * * * 14 172.16.243.254 (172.16.243.254) 93.251 ms 93.119 ms 92.606 ms ``` ### Attempts with prefixes for 172.16.243.254 (unsuccessful) - 172.16.243.0/24 <img width="853" height="717" alt="Image" src="https://github.com/user-attachments/assets/42507ddb-86a6-45c4-adce-f939c42dbbb8" /> - 172.16.243.192/26 <img width="851" height="717" alt="Image" src="https://github.com/user-attachments/assets/2bd102d2-fab9-4643-824c-180fe0f0b1ba" /> - 172.16.243.224/27 <img width="854" height="715" alt="Image" src="https://github.com/user-attachments/assets/6b960076-8cee-47c7-8fe7-9d58c658bf06" /> - 172.16.243.254/32 <img width="853" height="716" alt="Image" src="https://github.com/user-attachments/assets/d2f6bc36-a8ee-4f8e-b0a2-01f7620231b2" /> --- ## Target change: 172.16.9.78 A few minutes earlier, a network for 172.16.9.78/32 had been created and the traceroute came out as expected. The rule was then disabled to test a less specific prefix. Traceroute with 172.16.9.78/32 (OK result): ```bash root@b27850a60f04:/var/lib/zabbix# traceroute 172.16.9.78 traceroute to 172.16.9.78 (172.16.9.78), 30 hops max, 60 byte packets 1 [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32) 0.525 ms 0.494 ms 0.504 ms 2 100.76.180.193 (100.76.180.193) 0.660 ms 0.634 ms 0.710 ms 3 172.16.9.78 (172.16.9.78) 0.895 ms 0.951 ms 0.965 ms ``` Attempt with 172.16.9.0/24: ```bash root@b27850a60f04:/var/lib/zabbix# traceroute 172.16.9.78 traceroute to 172.16.9.78 (172.16.9.78), 30 hops max, 60 byte packets 1 [bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br](http://bridge-to-mgmt-cli-stda-via-linux.vpnnetbird.nortenettelecom.net.br) (10.255.255.32) 81.460 ms 83.221 ms 83.236 ms 2 100.76.180.193 (100.76.180.193) 93.814 ms 93.817 ms 93.809 ms 3 172.16.9.78 (172.16.9.78) 93.838 ms 93.834 ms 93.826 ms ``` Upon reverting to 172.16.9.78/32 and restarting the Netbird service on the Zabbix proxy, the traffic started going through the Relay Server: <img width="849" height="459" alt="Image" src="https://github.com/user-attachments/assets/9951817a-559a-4451-b53f-bebf9b86edc6" /> The same happens with 100.76.180.207 even with 100.76.180.207/32, the packets go through the Relay Server: <img width="849" height="445" alt="Image" src="https://github.com/user-attachments/assets/b1c166e9-4656-490b-a28c-8a8178432584" /> Another demonstration of intermittency: <img width="758" height="866" alt="Image" src="https://github.com/user-attachments/assets/586db098-4548-42c7-8394-d5da5c08081b" /> The only change made was the addition of the 100.76.0.0/17 network right before bringing the Netbird service down and up again, which would explain the change in traceroute behavior for 100.76.4.255, but not for the remaining addresses. --- ## Summary of behavior - Very specific prefixes (/32 and, in one case, /27) tend to work initially. - With less specific prefixes (/26, /16, /12), traffic often detours via the Relay Server. - There is intermittency: the same specific prefix can change behavior after route changes, even if the configuration returns to the initial pre-change state. ---
saavagebueno added the triage-needed label 2026-08-05 01:19:19 -04:00
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#8730