[GH-ISSUE #4047] Unable to P2P between peer and exit node #8592

New Issue

@Silex commented on GitHub (Jun 25, 2025):

Have one of the peer allow inbound UDP port 51820

@Silex commented on GitHub (Jun 25, 2025): Have one of the peer allow inbound UDP port 51820

saavagebueno commented

@eveyraud commented on GitHub (Jun 25, 2025):

Have one of the peer allow inbound UDP port 51820

Thanks for your reply,
Both of my peers allows UDP/51820 inbound and outbound. As I said in my ticket, I even tried to disable the firewall of my Windows and the EDR but the problem keeps unchanged. Regarding my Linux peer, it has been able to P2P with another Linux and an Android, only Windows isn't working.

@eveyraud commented on GitHub (Jun 25, 2025): > Have one of the peer allow inbound UDP port 51820 Thanks for your reply, Both of my peers allows UDP/51820 inbound and outbound. As I said in my ticket, I even tried to disable the firewall of my Windows and the EDR but the problem keeps unchanged. Regarding my Linux peer, it has been able to P2P with another Linux and an Android, only Windows isn't working.

saavagebueno commented

@eveyraud commented on GitHub (Jun 25, 2025):

UPDATE
When I try to connect both of them on a cloud netbird, they can both connect in P2P. This issue seems to only concern self-hosted instances

@eveyraud commented on GitHub (Jun 25, 2025): *UPDATE* When I try to connect both of them on a cloud netbird, they can both connect in P2P. This issue seems to only concern self-hosted instances

saavagebueno commented

@SasSam commented on GitHub (Jun 25, 2025):

It's maybe related to the issue that I've raised today: https://github.com/netbirdio/netbird/issues/4045
Try to downgrade the Windows client to 0.48.0

@SasSam commented on GitHub (Jun 25, 2025): It's maybe related to the issue that I've raised today: https://github.com/netbirdio/netbird/issues/4045 Try to downgrade the Windows client to 0.48.0

saavagebueno commented

@eveyraud commented on GitHub (Jun 26, 2025):

It's maybe related to the issue that I've raised today: https://github.com/netbirdio/netbird/issues/4045
Try to downgrade the Windows client to 0.48.0

Unfortunatly, no, I had the issue on 0.47.2, I then upgraded to 0.48.0, still had it and then finally upgraded to 0.49.0

@eveyraud commented on GitHub (Jun 26, 2025): > It's maybe related to the issue that I've raised today: https://github.com/netbirdio/netbird/issues/4045 > Try to downgrade the Windows client to 0.48.0 > Unfortunatly, no, I had the issue on 0.47.2, I then upgraded to 0.48.0, still had it and then finally upgraded to 0.49.0

saavagebueno commented

@eveyraud commented on GitHub (Jun 26, 2025):

Have one of the peer allow inbound UDP port 51820

After reading some more docs I found this:
https://netbird.io/knowledge-hub/enhancing-network-visibility-with-traffic-events-logging

I then tried hopelessly to remove the Linux his role of exit node for the Windows peer, now I'm able to P2P between the two machines.
Now my questions are:
- Is that a normal behavior ?
- Is it possible to use UDP in such a context ?

@eveyraud commented on GitHub (Jun 26, 2025): > Have one of the peer allow inbound UDP port 51820 After reading some more docs I found this: https://netbird.io/knowledge-hub/enhancing-network-visibility-with-traffic-events-logging I then tried hopelessly to remove the Linux his role of exit node for the Windows peer, now I'm able to P2P between the two machines. Now my questions are: - Is that a normal behavior ? - Is it possible to use UDP in such a context ?

saavagebueno commented

@lepazca commented on GitHub (Jun 28, 2025):

I did some tests to compare how traffic behaves when using a traditional WireGuard client versus the NetBird client while routing all traffic through an exit node:

🔹 Test 1: Traditional WireGuard client

When connecting to a WireGuard server:

Destination       Gateway         Genmask         Flags Metric Ref Use Iface
0.0.0.0           10.46.27.186    0.0.0.0         UG    0      0     0 wlp42s0
10.46.27.0        0.0.0.0         255.255.255.0   U     0      0     0 wlp42s0
55.44.33.22       10.46.27.186    255.255.255.255 UGH   0      0     0 wlp42s0  <- WG server public IP

The default route (0.0.0.0/0) is set to go through the tunnel, and an explicit route is added for the WireGuard server’s public IP via the real interface, ensuring it is reachable outside the tunnel (avoiding loops or fallback to relayed mode).

🔹 Test 2: NetBird client using an exit node

With NetBird connected to an exit node:

Destination       Gateway         Genmask         Flags Metric Ref Use Iface
0.0.0.0           10.46.27.186    0.0.0.0         UG    0      0     0 wlp42s0
10.46.27.0        0.0.0.0         255.255.255.0   U     0      0     0 wlp42s0
100.88.0.0        0.0.0.0         255.255.0.0     U     0      0     0 wt0

The default route (0.0.0.0/0) is pushed through the NetBird tunnel (wt0), but no dedicated route is added for the exit node’s public IP. As a result, traffic to the exit node itself also goes through the tunnel interface, which leads to:

A P2P connection being established temporarily.
Then falling back to relayed mode a few seconds later.

This behavior is consistent and happens only when using the peer as an exit node.

I only run these tests on Linux; both peers are using NetBird v0.49.0.

@lepazca commented on GitHub (Jun 28, 2025): I did some tests to compare how traffic behaves when using a traditional WireGuard client versus the NetBird client while routing all traffic through an exit node: 🔹 Test 1: Traditional WireGuard client When connecting to a WireGuard server: ``` Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.46.27.186 0.0.0.0 UG 0 0 0 wlp42s0 10.46.27.0 0.0.0.0 255.255.255.0 U 0 0 0 wlp42s0 55.44.33.22 10.46.27.186 255.255.255.255 UGH 0 0 0 wlp42s0 <- WG server public IP ``` The default route (0.0.0.0/0) is set to go through the tunnel, and an explicit route is added for the WireGuard server’s public IP via the real interface, ensuring it is reachable outside the tunnel (avoiding loops or fallback to relayed mode). 🔹 Test 2: NetBird client using an exit node With NetBird connected to an exit node: ``` Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 10.46.27.186 0.0.0.0 UG 0 0 0 wlp42s0 10.46.27.0 0.0.0.0 255.255.255.0 U 0 0 0 wlp42s0 100.88.0.0 0.0.0.0 255.255.0.0 U 0 0 0 wt0 ``` The default route (0.0.0.0/0) is pushed through the NetBird tunnel (wt0), but no dedicated route is added for the exit node’s public IP. As a result, traffic to the exit node itself also goes through the tunnel interface, which leads to: - A P2P connection being established temporarily. - Then falling back to relayed mode a few seconds later. This behavior is consistent and happens only when using the peer as an exit node. I only run these tests on Linux; both peers are using NetBird v0.49.0.

saavagebueno commented

@lepazca commented on GitHub (Jun 28, 2025):

I just tested it on Windows and it behaved in the same way. The Netbird client does not add a static route for the public IP address of the exit node. I assume this is the only way to avoid interfering with the P2P connection between the client and the exit node. Please correct me if I am wrong.

@lepazca commented on GitHub (Jun 28, 2025): I just tested it on Windows and it behaved in the same way. The Netbird client does not add a static route for the public IP address of the exit node. I assume this is the only way to avoid interfering with the P2P connection between the client and the exit node. Please correct me if I am wrong.

saavagebueno commented

@eveyraud commented on GitHub (Jun 30, 2025):

I just tested it on Windows and it behaved in the same way. The Netbird client does not add a static route for the public IP address of the exit node. I assume this is the only way to avoid interfering with the P2P connection between the client and the exit node. Please correct me if I am wrong.

I thought about it but tbh I didn't search further. You are absolutely right, that's what I notice too. I'm not sure this behavior is wanted by the devs but just in case, I created a feature request: #4069

@eveyraud commented on GitHub (Jun 30, 2025): > I just tested it on Windows and it behaved in the same way. The Netbird client does not add a static route for the public IP address of the exit node. I assume this is the only way to avoid interfering with the P2P connection between the client and the exit node. Please correct me if I am wrong. I thought about it but tbh I didn't search further. You are absolutely right, that's what I notice too. I'm not sure this behavior is wanted by the devs but just in case, I created a feature request: #4069

saavagebueno commented

@nazarewk commented on GitHub (Jun 30, 2025):

@lepazca thanks for the analysis! Turns out the dev team was disussing this some time in the past, but didn't pursue the implementation, we will definitely look at it sooner than later

@nazarewk commented on GitHub (Jun 30, 2025): @lepazca thanks for the analysis! Turns out the dev team was disussing this some time in the past, but didn't pursue the implementation, we will definitely look at it sooner than later

saavagebueno commented

@nazarewk commented on GitHub (Jun 30, 2025):

@lepazca actually after further clarification this (routing of IPs) is something we are doing either in a separate routing table (linux) or on-demand upon detecting outbound connection (most other systems) so it should not be an issue.

Would it be possible for you to provide a netbird debug for 1m -SU for the affected devices? We would like to investigate this further

@nazarewk commented on GitHub (Jun 30, 2025): @lepazca actually after further clarification this (routing of IPs) is something we are doing either in a separate routing table (linux) or on-demand upon detecting outbound connection (most other systems) so it should not be an issue. Would it be possible for you to provide a `netbird debug for 1m -SU` for the affected devices? We would like to investigate this further

saavagebueno commented

@nazarewk commented on GitHub (Jun 30, 2025):

@eveyraud Could you also send us a debug bundle from netbird-gateway.netbird.selfhosted Peer? We have noticed a suspicious candidate from within NetBird networking range which should not be there and want to learn more about it:

2025-06-25T14:02:36+02:00 WARN client/iface/bind/udp_mux_universal.go:168: Address 100.71.125.247:51820 is part of the NetBird network 100.71.104.14/16, refusing to write

@nazarewk commented on GitHub (Jun 30, 2025): @eveyraud Could you also send us a debug bundle from `netbird-gateway.netbird.selfhosted` Peer? We have noticed a suspicious candidate from within NetBird networking range which should not be there and want to learn more about it: ``` 2025-06-25T14:02:36+02:00 WARN client/iface/bind/udp_mux_universal.go:168: Address 100.71.125.247:51820 is part of the NetBird network 100.71.104.14/16, refusing to write ```

saavagebueno commented

@lepazca commented on GitHub (Jun 30, 2025):

@nazarewk Thanks for the clarification and for looking into this.

I've performed the tests as requested and uploaded the debug data.

Here is the upload file key:

1234567890ab27fb37c88b3b4be7011e22aa2e5ca6f38ffa9c4481884941f726/12345678-90ab-cdef-1234-567890abcdef

Let me know if you need anything else from my side.

@lepazca commented on GitHub (Jun 30, 2025): @nazarewk Thanks for the clarification and for looking into this. I've performed the tests as requested and uploaded the debug data. Here is the upload file key: ``` 1234567890ab27fb37c88b3b4be7011e22aa2e5ca6f38ffa9c4481884941f726/12345678-90ab-cdef-1234-567890abcdef ``` Let me know if you need anything else from my side.

saavagebueno commented

@eveyraud commented on GitHub (Jun 30, 2025):

@eveyraud Could you also send us a debug bundle from netbird-gateway.netbird.selfhosted Peer? We have noticed a suspicious candidate from within NetBird networking range which should not be there and want to learn more about it:
2025-06-25T14:02:36+02:00 WARN client/iface/bind/udp_mux_universal.go:168: Address 100.71.125.247:51820 is part of the NetBird network 100.71.104.14/16, refusing to write

IP addresses changed as I removed and added back some machines, but the IP your pointing was the peer "netbird-gateway". Just in case it can help you, here are the files from both peers:

Upload file key from NetBird-Gateway:
0107b729c324c4562e1e10d3b5be55567aa55c08302d40e822c7a4ca939cd561/2c406be5-8009-4208-9db0-19c88ccf3c9d

Upload file key from Windows peer:
0107b729c324c4562e1e10d3b5be55567aa55c08302d40e822c7a4ca939cd561/0eb3d2b5-c8ed-418b-a468-8f8d92d46c91

file keys updated I gave wrong ones

@eveyraud commented on GitHub (Jun 30, 2025): > [@eveyraud](https://github.com/eveyraud) Could you also send us a debug bundle from `netbird-gateway.netbird.selfhosted` Peer? We have noticed a suspicious candidate from within NetBird networking range which should not be there and want to learn more about it: > > ``` > 2025-06-25T14:02:36+02:00 WARN client/iface/bind/udp_mux_universal.go:168: Address 100.71.125.247:51820 is part of the NetBird network 100.71.104.14/16, refusing to write > ``` IP addresses changed as I removed and added back some machines, but the IP your pointing was the peer "netbird-gateway". Just in case it can help you, here are the files from both peers: Upload file key from NetBird-Gateway: `0107b729c324c4562e1e10d3b5be55567aa55c08302d40e822c7a4ca939cd561/2c406be5-8009-4208-9db0-19c88ccf3c9d` Upload file key from Windows peer: `0107b729c324c4562e1e10d3b5be55567aa55c08302d40e822c7a4ca939cd561/0eb3d2b5-c8ed-418b-a468-8f8d92d46c91` _file keys updated I gave wrong ones_

saavagebueno commented

@eveyraud commented on GitHub (Jul 15, 2025):

@eveyraud Could you also send us a debug bundle from netbird-gateway.netbird.selfhosted Peer? We have noticed a suspicious candidate from within NetBird networking range which should not be there and want to learn more about it:
2025-06-25T14:02:36+02:00 WARN client/iface/bind/udp_mux_universal.go:168: Address 100.71.125.247:51820 is part of the NetBird network 100.71.104.14/16, refusing to write

Could you please let me know if there have been any updates on this topic?
Regarding the warning you mentioned, do you think it's something concerning, or is it simply log noise?

Thanks for your help and answers

@eveyraud commented on GitHub (Jul 15, 2025): > [@eveyraud](https://github.com/eveyraud) Could you also send us a debug bundle from `netbird-gateway.netbird.selfhosted` Peer? We have noticed a suspicious candidate from within NetBird networking range which should not be there and want to learn more about it: > > ``` > 2025-06-25T14:02:36+02:00 WARN client/iface/bind/udp_mux_universal.go:168: Address 100.71.125.247:51820 is part of the NetBird network 100.71.104.14/16, refusing to write > ``` Could you please let me know if there have been any updates on this topic? Regarding the warning you mentioned, do you think it's something concerning, or is it simply log noise? Thanks for your help and answers

saavagebueno commented

@stevo11811 commented on GitHub (Jul 15, 2025):

Ah, thanks for the detailed post, I just ran into the same issue during testing and was beating my head against a wall.

@stevo11811 commented on GitHub (Jul 15, 2025): Ah, thanks for the detailed post, I just ran into the same issue during testing and was beating my head against a wall.

saavagebueno commented

@nazarewk commented on GitHub (Jul 15, 2025):

@eveyraud The part about trying to use NetBird IP as an ICE candidate is suspicious. We have found some additional information there, but we will need additional ICE logs (enabled separately) to get to the bottom of this issue.
Could you enable PION logs on your Windows client and gather a debug bundle for 1 minute again? https://docs.netbird.io/how-to/troubleshooting-client#debugging-ice-connections

PS: It would help if the debug bundle wasn't anonymized for more easily determining the exact IP address <> domain correlations. The logs are only accessible by developers through the internal storage system and are deleted after 30 days if you have any concerns about it.

@nazarewk commented on GitHub (Jul 15, 2025): @eveyraud The part about trying to use NetBird IP as an ICE candidate is suspicious. We have found some additional information there, but we will need additional ICE logs (enabled separately) to get to the bottom of this issue. Could you enable PION logs on your Windows client and gather a debug bundle for 1 minute again? https://docs.netbird.io/how-to/troubleshooting-client#debugging-ice-connections PS: It would help if the debug bundle wasn't anonymized for more easily determining the exact IP address <> domain correlations. The logs are only accessible by developers through the internal storage system and are deleted after 30 days if you have any concerns about it.

saavagebueno commented

@eveyraud commented on GitHub (Jul 16, 2025):

@eveyraud The part about trying to use NetBird IP as an ICE candidate is suspicious. We have found some additional information there, but we will need additional ICE logs (enabled separately) to get to the bottom of this issue. Could you enable PION logs on your Windows client and gather a debug bundle for 1 minute again? https://docs.netbird.io/how-to/troubleshooting-client#debugging-ice-connections

PS: It would help if the debug bundle wasn't anonymized for more easily determining the exact IP address <> domain correlations. The logs are only accessible by developers through the internal storage system and are deleted after 30 days if you have any concerns about it.

Of course, I can give you the logs. However due to cybersecurity policy I am only able to provide you the anonymized version. Giving you non-anonymized IPs or domains just isn't something we can do.

Also, the doc you gave me concerns linux commands, I tried to manually set the environment variable: [Environment]::SetEnvironmentVariable("PIONS_LOG_DEBUG", "all", "Machine")
[Environment]::SetEnvironmentVariable("NB_LOG_LEVEL", "debug", "Machine")
netbird up -F --log-level debug > C:\Temp\netbird.log 2>&1

But the given logs seems not to be very different from those I gave you. Am I wrong somewhere between the commands used and the returned logs ?

@eveyraud commented on GitHub (Jul 16, 2025): > [@eveyraud](https://github.com/eveyraud) The part about trying to use NetBird IP as an ICE candidate is suspicious. We have found some additional information there, but we will need additional ICE logs (enabled separately) to get to the bottom of this issue. Could you enable PION logs on your Windows client and gather a debug bundle for 1 minute again? https://docs.netbird.io/how-to/troubleshooting-client#debugging-ice-connections > > PS: It would help if the debug bundle wasn't anonymized for more easily determining the exact IP address <> domain correlations. The logs are only accessible by developers through the internal storage system and are deleted after 30 days if you have any concerns about it. Of course, I can give you the logs. However due to cybersecurity policy I am only able to provide you the anonymized version. Giving you non-anonymized IPs or domains just isn't something we can do. Also, the doc you gave me concerns linux commands, I tried to manually set the environment variable: `[Environment]::SetEnvironmentVariable("PIONS_LOG_DEBUG", "all", "Machine")` `[Environment]::SetEnvironmentVariable("NB_LOG_LEVEL", "debug", "Machine")` `netbird up -F --log-level debug > C:\Temp\netbird.log 2>&1` But the given logs seems not to be very different from those I gave you. Am I wrong somewhere between the commands used and the returned logs ?

saavagebueno commented

@nazarewk commented on GitHub (Jul 17, 2025):

@eveyraud I have checked this myself and indeed it looks like the logs aren't redirected to the expected places (the ICE library uses different logger implementation), on a fresh Ubuntu instance I found the PION logs at /var/log/netbird/netbird.out. I'll try to find out how to access those in other environments and get back to you.

For the future, you don't really need to run NetBird in the foreground; just restarting a service should suffice.

@nazarewk commented on GitHub (Jul 17, 2025): @eveyraud I have checked this myself and indeed it looks like the logs aren't redirected to the expected places (the ICE library uses different logger implementation), on a fresh Ubuntu instance I found the PION logs at `/var/log/netbird/netbird.out`. I'll try to find out how to access those in other environments and get back to you. For the future, you don't really need to run NetBird in the foreground; just restarting a service should suffice.

saavagebueno commented

@lixmal commented on GitHub (Jul 22, 2025):

Hey folks, can you please test https://github.com/netbirdio/netbird/releases/tag/v0.51.2 and report whether the problem has been fixed for you.

@lixmal commented on GitHub (Jul 22, 2025): Hey folks, can you please test https://github.com/netbirdio/netbird/releases/tag/v0.51.2 and report whether the problem has been fixed for you.

saavagebueno commented

@eveyraud commented on GitHub (Jul 22, 2025):

Hey folks, can you please test https://github.com/netbirdio/netbird/releases/tag/v0.51.2 and report whether the problem has been fixed for you.

Issue fixed on my side. Still have the log spam @nazarewk mentionned though.
Thank you a lot for the fix !

@eveyraud commented on GitHub (Jul 22, 2025): > Hey folks, can you please test https://github.com/netbirdio/netbird/releases/tag/v0.51.2 and report whether the problem has been fixed for you. Issue fixed on my side. Still have the log spam @nazarewk mentionned though. Thank you a lot for the fix !

saavagebueno commented

@stevo11811 commented on GitHub (Jul 22, 2025):

This is working for me! Thanks everyone.

@stevo11811 commented on GitHub (Jul 22, 2025): This is working for me! Thanks everyone.

saavagebueno commented

@maxideus85 commented on GitHub (Jul 24, 2025):

I'm not sure if this is the correct place to ask this, but I am having this issue even on 0.51.2. Please let me know if I need to spin up my own thread. I have some details below in case I can write my issue here.

I have a test environment with an Ubuntu VM running Docker CE and the Netbird containers deployed following the advanced guide. I also deployed a container for the Netbird agent on the same host to act as an exit node that's using host networking.

In the prior version of 0.51.1 for all containers, I was able to get P2P connections between my Windows endpoint and the exit node. However, in the latest release of 0.51.2, I am now only able to get relayed connections. I have ensured all ports are accessible so I don't think it's a firewall or NAT issue.

Can I get some help?

@maxideus85 commented on GitHub (Jul 24, 2025): I'm not sure if this is the correct place to ask this, but I am having this issue even on 0.51.2. Please let me know if I need to spin up my own thread. I have some details below in case I can write my issue here. I have a test environment with an Ubuntu VM running Docker CE and the Netbird containers deployed following the advanced guide. I also deployed a container for the Netbird agent on the same host to act as an exit node that's using host networking. In the prior version of 0.51.1 for all containers, I was able to get P2P connections between my Windows endpoint and the exit node. However, in the latest release of 0.51.2, I am now only able to get relayed connections. I have ensured all ports are accessible so I don't think it's a firewall or NAT issue. Can I get some help?

saavagebueno commented

@lixmal commented on GitHub (Jul 25, 2025):

@maxideus85

I'm not sure if this is the correct place to ask this, but I am having this issue even on 0.51.2. Please let me know if I need to spin up my own thread. I have some details below in case I can write my issue here

please do

@lixmal commented on GitHub (Jul 25, 2025): @maxideus85 >I'm not sure if this is the correct place to ask this, but I am having this issue even on 0.51.2. Please let me know if I need to spin up my own thread. I have some details below in case I can write my issue here please do

saavagebueno commented

@lepazca commented on GitHub (Jul 27, 2025):

Hi, sorry for the delay in providing feedback.

I'm still experiencing the same issue even after updating to version 0.51.2. When I enable the exit node, the connection switches to relayed. I understand some users have reported this issue as resolved, but perhaps the root cause in my case is different.

As before I captured the logs in case they help identify the issue.

c04854b95ffd40870e1dff93227236c1cce3c359fdeb83899a046975941e66e7/fab17525-9dba-463d-988d-7e94f1ae548a

Thank you

@lepazca commented on GitHub (Jul 27, 2025): Hi, sorry for the delay in providing feedback. I'm still experiencing the same issue even after updating to version 0.51.2. When I enable the exit node, the connection switches to relayed. I understand some users have reported this issue as resolved, but perhaps the root cause in my case is different. As before I captured the logs in case they help identify the issue. `c04854b95ffd40870e1dff93227236c1cce3c359fdeb83899a046975941e66e7/fab17525-9dba-463d-988d-7e94f1ae548a ` Thank you

saavagebueno commented

@lixmal commented on GitHub (Jul 27, 2025):

@lepazca, this is a Linux machine; Linux is not affected by this bug.

You have a block rule in the OUTPUT chain. Could that be the issue?

-A OUTPUT -j evpn.OUTPUT
[...]
-A evpn.OUTPUT -j evpn.a.100.blockAll
-A evpn.r.100.blockAll -j REJECT --reject-with icmp-port-unreachable

@lixmal commented on GitHub (Jul 27, 2025): @lepazca, this is a Linux machine; Linux is not affected by this bug. You have a block rule in the `OUTPUT` chain. Could that be the issue? ``` -A OUTPUT -j evpn.OUTPUT [...] -A evpn.OUTPUT -j evpn.a.100.blockAll -A evpn.r.100.blockAll -j REJECT --reject-with icmp-port-unreachable ```

saavagebueno commented