high packet loss from continuous peer reconnections in some cases with windows to windows connections #465

@mlsmaycon commented on GitHub (Oct 5, 2023):

Hello @Fantu can you share more details about your NetBird configuration? Do you have network routes involved and what are the networks the two peers are part of?

@mlsmaycon commented on GitHub (Oct 5, 2023): Hello @Fantu can you share more details about your NetBird configuration? Do you have network routes involved and what are the networks the two peers are part of?

saavagebueno commented

@Fantu commented on GitHub (Oct 5, 2023):

@mlsmaycon thanks for reply, netbird server is selfhosted, there are no network routes on netbird configuration and the devices of the tests have 2 groups ("All" and "M2R")
on access control page I see: "M2R"<->"M2R" protocol ALL ports ALL
the Default related to "All"<->"All" is disabled instead
sorry if I have almost no knowledge of netbird and have only done a few quick things when my boss told me to install and then 2 times (including this one) where he reported problems to fix
tell me if the info provided is wrong or incomplete

@Fantu commented on GitHub (Oct 5, 2023): @mlsmaycon thanks for reply, netbird server is selfhosted, there are no network routes on netbird configuration and the devices of the tests have 2 groups ("All" and "M2R") on access control page I see: "M2R"<->"M2R" protocol ALL ports ALL the Default related to "All"<->"All" is disabled instead sorry if I have almost no knowledge of netbird and have only done a few quick things when my boss told me to install and then 2 times (including this one) where he reported problems to fix tell me if the info provided is wrong or incomplete

saavagebueno commented

@mlsmaycon commented on GitHub (Oct 5, 2023):

Thanks @Fantu , we see a lot of reconnection in your logs. Can you share more details about the network of both machines? Are they in the same office, data center, or cloud?

Regarding the connection, they also seem to be using relay, can you ensure that the server where you are running NetBird services, have the following ports accessible from the internet:

Open TCP ports 80, 443, 33073, 10000 (Dashboard HTTP & HTTPS, Management gRCP & HTTP APIs, Signal gRPC API respectively) on your server.
Coturn is used for relay using the STUN/TURN protocols. It requires a listening port, UDP 3478, and range of ports, UDP 49152-65535, for dynamic relay connections. These are set as defaults in setup file, but can be configured to your requirements.

@mlsmaycon commented on GitHub (Oct 5, 2023): Thanks @Fantu , we see a lot of reconnection in your logs. Can you share more details about the network of both machines? Are they in the same office, data center, or cloud? Regarding the connection, they also seem to be using relay, can you ensure that the server where you are running NetBird services, have the following ports accessible from the internet: - Open TCP ports 80, 443, 33073, 10000 (Dashboard HTTP & HTTPS, Management gRCP & HTTP APIs, Signal gRPC API respectively) on your server. - Coturn is used for relay using the STUN/TURN protocols. It requires a listening port, UDP 3478, and range of ports, UDP 49152-65535, for dynamic relay connections. These are set as defaults in setup file, but can be configured to your requirements.

saavagebueno commented

@Fantu commented on GitHub (Oct 5, 2023):

netbird server and destination devices are in the same LAN on office
and here my first question: why these reconnections on LAN connections which in theory should have no problems?
source device is in different network on my home and use internet, anyway both internet connection works correctly and using a normal wireguard vpn to office network instead of netbird I can work on same devices without any issue and 0 packet lost

netbird server seems have all ports ok FWIK and with port forwarding on NAT of gateway. there was an issue on custom signal port on start but was solved with 8d18190c94
about coturn range is custom but range in coturn configuration is correct and udp ports list viewed with "ss" command are all in the correct range
I don't know a way to test and check are really working correctly, is there?

anyway I have another question: connection with coturn should be only the relay one when direct connection is not possible or on start when direct connection are not started and after should use direct connections between devices or am I wrong?
at least take a quick look at the documentation
but from the logs I don't seem to see direct connections between the devices and I also don't understand why in the source device logs there is always the loopback IP as an endpoint unlike the devices in the LAN which have the IP of the netbird server or of other devices. shouldn't there be the internet IP of the netbird server instead on the source device?
is there a cli command to view all devices, the active connections and their details (or if are missed connection) or some other useful command for debugging? for example of have a small experience with zerotier and on it there is zerotier-cli peers useful for similar thing

thanks for any reply and sorry for my bad english

@Fantu commented on GitHub (Oct 5, 2023): netbird server and destination devices are in the same LAN on office and here my first question: why these reconnections on LAN connections which in theory should have no problems? source device is in different network on my home and use internet, anyway both internet connection works correctly and using a normal wireguard vpn to office network instead of netbird I can work on same devices without any issue and 0 packet lost netbird server seems have all ports ok FWIK and with port forwarding on NAT of gateway. there was an issue on custom signal port on start but was solved with https://github.com/netbirdio/netbird/commit/8d18190c9412268aa0279a4e89d4e02cbd1bdb46 about coturn range is custom but range in coturn configuration is correct and udp ports list viewed with "ss" command are all in the correct range I don't know a way to test and check are really working correctly, is there? anyway I have another question: connection with coturn should be only the relay one when direct connection is not possible or on start when direct connection are not started and after should use direct connections between devices or am I wrong? at least take a quick look at the documentation but from the logs I don't seem to see direct connections between the devices and I also don't understand why in the source device logs there is always the loopback IP as an endpoint unlike the devices in the LAN which have the IP of the netbird server or of other devices. shouldn't there be the internet IP of the netbird server instead on the source device? is there a cli command to view all devices, the active connections and their details (or if are missed connection) or some other useful command for debugging? for example of have a small experience with zerotier and on it there is `zerotier-cli peers` useful for similar thing thanks for any reply and sorry for my bad english

saavagebueno commented

@Fantu commented on GitHub (Oct 9, 2023):

I found the cli command to check peers connection status netbird status -d --filter-by-ips ... and I did some tests
with both devices (source and destination) on LAN (or connected with other stable VPN) there is P2P connetion stable

Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: 100.117.66.12
  Public key: xxx
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): srflx/host
  Last connection update: 2023-10-09 12:22:54

connection from remote instead have a relay connection that continue to disconnect/reconnect:

Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: 100.117.66.12
  Public key: xxx
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  Last connection update: 2023-10-09 14:38:31

...

Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: 100.117.66.12
  Public key: xxx
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  Last connection update: 2023-10-09 14:39:06

...

Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: 100.117.66.12
  Public key: xxx
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  Last connection update: 2023-10-09 14:41:19

for the brief moments it remains connected it seems works well and packets seems are lost only during disconnections
Is it perhaps that it keeps trying to connect in P2P but fails and goes back to relay or are there other problems?
If you need further information/tests let me know

EDIT:
from a fast look netbird use ice for manage peers connections, update ice from 2.3.1 to 2.3.11 which include many bugfixes can be useful to fix something and probably also this?

@Fantu commented on GitHub (Oct 9, 2023): I found the cli command to check peers connection status `netbird status -d --filter-by-ips ...` and I did some tests with both devices (source and destination) on LAN (or connected with other stable VPN) there is P2P connetion stable ``` Peers detail: administration.netbird.selfhosted: NetBird IP: 100.117.66.12 Public key: xxx Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/host Last connection update: 2023-10-09 12:22:54 ``` connection from remote instead have a relay connection that continue to disconnect/reconnect: ``` Peers detail: administration.netbird.selfhosted: NetBird IP: 100.117.66.12 Public key: xxx Status: Connected -- detail -- Connection type: Relayed Direct: false ICE candidate (Local/Remote): relay/host Last connection update: 2023-10-09 14:38:31 ... Peers detail: administration.netbird.selfhosted: NetBird IP: 100.117.66.12 Public key: xxx Status: Connected -- detail -- Connection type: Relayed Direct: false ICE candidate (Local/Remote): relay/host Last connection update: 2023-10-09 14:39:06 ... Peers detail: administration.netbird.selfhosted: NetBird IP: 100.117.66.12 Public key: xxx Status: Connected -- detail -- Connection type: Relayed Direct: false ICE candidate (Local/Remote): relay/host Last connection update: 2023-10-09 14:41:19 ``` for the brief moments it remains connected it seems works well and packets seems are lost only during disconnections Is it perhaps that it keeps trying to connect in P2P but fails and goes back to relay or are there other problems? If you need further information/tests let me know EDIT: from a fast look netbird use ice for manage peers connections, update ice from 2.3.1 to 2.3.11 which include many bugfixes can be useful to fix something and probably also this?

saavagebueno commented

@Fantu commented on GitHub (Oct 17, 2023):

I did some tests with custom build with updated ICE (3b00ee9a42) but issue persist :(
issue seems happen only when one or both peer is inside a network with symmetrical nat (pfsense, freebsd don't support fullcone for now https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219803) and double nat (the router is fritzbox that have fullcone nat)

I suppose other users have symmetrical nat, nobody saw this issue before?

on major of cases netbird is more stable that zerotier and faster to connect/reconnect between peers except this case where with netbird is unusable (for reconnect "loop") and with zerotier even if with packet loss, and occasional rdp disconnect is enough usable

at the moment I don't know what else to do and my boss seems willing to abandon the use of netbird
any advice is appreciated
thanks for any reply and sorry for my bad english

@Fantu commented on GitHub (Oct 17, 2023): I did some tests with custom build with updated ICE (https://github.com/M2Rbiz/netbird/commit/3b00ee9a4241b26f313a6cdfcef8d7891a4a7809) but issue persist :( issue seems happen only when one or both peer is inside a network with symmetrical nat (pfsense, freebsd don't support fullcone for now https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219803) and double nat (the router is fritzbox that have fullcone nat) I suppose other users have symmetrical nat, nobody saw this issue before? on major of cases netbird is more stable that zerotier and faster to connect/reconnect between peers except this case where with netbird is unusable (for reconnect "loop") and with zerotier even if with packet loss, and occasional rdp disconnect is enough usable at the moment I don't know what else to do and my boss seems willing to abandon the use of netbird any advice is appreciated thanks for any reply and sorry for my bad english

saavagebueno commented

@Fantu commented on GitHub (Oct 18, 2023):

here a client log debug on a peer with the issue:
https://paste.debian.net/1295455/

netbird status -d --filter-by-ips 100.117.66.12
Peers detail:
 administration.netbird.selfhosted:
  NetBird IP: 100.117.66.12
  Public key: dbrzwmy0grg6A18e4h4QPG32b/g2B68boB5SZ1lwPEo=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  Last connection update: 2023-10-18 18:44:52

peer dbrzwmy0grg6A18e4h4QPG32b/g2B68boB5SZ1lwPEo= is one with reconnect loop issue

I don't understand what the problem could be looking the logs, maybe I don't have enough experience, does anyone have any ideas/advice please?

@Fantu commented on GitHub (Oct 18, 2023): here a client log debug on a peer with the issue: https://paste.debian.net/1295455/ ``` netbird status -d --filter-by-ips 100.117.66.12 Peers detail: administration.netbird.selfhosted: NetBird IP: 100.117.66.12 Public key: dbrzwmy0grg6A18e4h4QPG32b/g2B68boB5SZ1lwPEo= Status: Connected -- detail -- Connection type: Relayed Direct: false ICE candidate (Local/Remote): relay/host Last connection update: 2023-10-18 18:44:52 ``` peer `dbrzwmy0grg6A18e4h4QPG32b/g2B68boB5SZ1lwPEo=` is one with reconnect loop issue I don't understand what the problem could be looking the logs, maybe I don't have enough experience, does anyone have any ideas/advice please?

saavagebueno commented

@Fantu commented on GitHub (Oct 21, 2023):

From a fast test with managed netbird service on same device the issue don't happen
So seems happen only when one windows peer in lan with netbird server connect to external peer.
I also saw the other cases, using the same selfhosted netbird server, where windows peer of one external network connect to peers of other external network don't have the issue and even if stay in relay is stable.
I'll probably do more tests next week to be sure.

small question OT:
I tried managed netbird for this fast test and I saw only peers list and is only possible add peers with login,
I didn't saw these feature listed in free plan: Access controls, Private DNS, Network routes, Management activity logging
I suppose the cause is my user result with role "user" and not possible to change to admin
should not be set "admin", as the first user of the selfhosted?

@Fantu commented on GitHub (Oct 21, 2023): From a fast test with managed netbird service on same device the issue don't happen So seems happen only when one windows peer in lan with netbird server connect to external peer. I also saw the other cases, using the same selfhosted netbird server, where windows peer of one external network connect to peers of other external network don't have the issue and even if stay in relay is stable. I'll probably do more tests next week to be sure. small question OT: I tried managed netbird for this fast test and I saw only peers list and is only possible add peers with login, I didn't saw these feature listed in free plan: Access controls, Private DNS, Network routes, Management activity logging I suppose the cause is my user result with role "user" and not possible to change to admin should not be set "admin", as the first user of the selfhosted?

saavagebueno commented

@mlsmaycon commented on GitHub (Oct 23, 2023):

Hello @Fantu it should be set as admin, but is very likely that there is an admin account for your domain already.

Can you reach out on slack so we can discuss your case in detail?

@mlsmaycon commented on GitHub (Oct 23, 2023): Hello @Fantu it should be set as admin, but is very likely that there is an admin account for your domain already. Can you reach out on slack so we can discuss your case in detail?

saavagebueno commented

@Fantu commented on GitHub (Oct 23, 2023):

thanks for the reply, there is another user with the same domain who had signed up before and there were no peers because he only had that signup/login by mistake. I would recommend adding something in the managed service interface to make sure that cases like these are seen/understood

returning in topic:
from the first tests this morning with the same selfhost server adding peers from another external network with pfsense (therefore nat symmetrical and double like the office) and was in relay without problems it seemed confirmed that the problem was with windows peer in lan with netbird server.
instead then trying from other peers from 2 other networks (both always w10) towards an office peer (with which we had problems) in those cases there were no problems and one also has a "worse" wifi connection instead of vdsl or FTTH but he had no disconnections, no packet loss but only some rare packets with latency of about 1 second
so at the moment I have no idea what other tests to do and under what conditions this problem occurs

@Fantu commented on GitHub (Oct 23, 2023): thanks for the reply, there is another user with the same domain who had signed up before and there were no peers because he only had that signup/login by mistake. I would recommend adding something in the managed service interface to make sure that cases like these are seen/understood returning in topic: from the first tests this morning with the same selfhost server adding peers from another external network with pfsense (therefore nat symmetrical and double like the office) and was in relay without problems it seemed confirmed that the problem was with windows peer in lan with netbird server. instead then trying from other peers from 2 other networks (both always w10) towards an office peer (with which we had problems) in those cases there were no problems and one also has a "worse" wifi connection instead of vdsl or FTTH but he had no disconnections, no packet loss but only some rare packets with latency of about 1 second so at the moment I have no idea what other tests to do and under what conditions this problem occurs

saavagebueno commented

@teoder commented on GitHub (Nov 15, 2023):

Having the exact same issue.
A bunch of Windows PCs scattered behind various Firewalls and they are disconnecting/reconnecting frequently.
We basically only use one Peer as exit node for a bunch of services and have a self-hosted Management/Admin server with a Public IP and the required port-openings.

If I take my laptop and use LTE-connection without any NAT and get status "Direct: true" it works without any issues.
I even get P2P/True to some devices, so the issues seems to be related to having NAT on both source and destination.

I'll keep troubleshooting and update if I have any findings that might help you out.

@teoder commented on GitHub (Nov 15, 2023): Having the exact same issue. A bunch of Windows PCs scattered behind various Firewalls and they are disconnecting/reconnecting frequently. We basically only use one Peer as exit node for a bunch of services and have a self-hosted Management/Admin server with a Public IP and the required port-openings. If I take my laptop and use LTE-connection without any NAT and get status "Direct: true" it works without any issues. I even get P2P/True to some devices, so the issues seems to be related to having NAT on both source and destination. I'll keep troubleshooting and update if I have any findings that might help you out.

saavagebueno commented

@Fantu commented on GitHub (Dec 22, 2023):

About the issue in my office it seems to be related to the netbird selfhost server (more exactly issue seems related to coturn) in a network with pfsense behind another nat and without access to a public ip. using an external coturn with single nat with public IP and the managed service there were no problems.

After my employer has started to put netbird using the managed service on some devices of some customers (I think he will want to completely replace zerotier with that) and problems of frequent disconnections have started on some windows devices even using the managed service.

After some debugging attempts without being able to find the cause and in which the netbird debug logs did not give any details regarding the disconnection or possible problems to work to try to resolve or improve yesterday thanks to the help of Pascal Fisher I finally managed to find something which could be useful. Initially try to enable ICE debug logs was difficult because trying $env:NB_LOG_LEVEL="debug"; $env:PIONS_LOG_DEBUG="all"; netbird service run | Tee-Object -FilePath "disconnect.logs" was not working, as seems don't possible run netbird in foreground like with linux. I tried to add env. variable also in the windows service with regedit and in other way but netbird client.log was always without ICE debug.
Pascal Fisher found a solution hardcoding it in custom build I after used for debug on peers with disconnect issue.

Here a cut from one ICE debug log with strange thing spotted after disconnect:

ice WARNING: 2023/12/21 15:54:07 Discard message from (udp4 relay 35.234.101.101:58033 related 192.168.168.110:54256), unknown TransactionID 0x8f16f56aba969acc059bffeb
turnc DEBUG: 15:54:07.929646 allocation.go:132: Refresh timer 1 expired
turnc DEBUG: 15:54:07.945809 allocation.go:173: Set new nonce with 16 bytes
turnc DEBUG: 15:54:07.945809 allocation.go:56: Refresh allocation: 438, got new nonce.
turnc DEBUG: 15:54:07.971915 allocation.go:127: Refresh permissions successful
ice WARNING: 2023/12/21 15:54:11 Discard message from (udp4 relay 35.234.101.101:58033 related 192.168.168.110:54256), unknown TransactionID 0x13a8d2430fae1880ea81004d
ice INFO: 2023/12/21 15:54:17 Setting new connection state: Disconnected
turnc DEBUG: 15:54:17.797803 allocation.go:77: Send refresh request (dontWait=true)
ice WARNING: 2023/12/21 15:54:17 Failed to read from candidate udp4 relay 35.234.101.101:37089 related 192.168.178.27:64981: read udp 35.234.101.101:37089: use of closed network connection
turnc DEBUG: 15:54:17.798374 allocation.go:84: Refresh request sent
turnc DEBUG: 15:54:17.798374 client.go:177: Failed to read: read tcp4 192.168.178.27:64981->35.234.101.101:443: use of closed network connection. Exiting loop
ice INFO: 2023/12/21 15:54:17 Setting new connection state: Closed
ice DEBUG: 15:54:25.123955 agent.go:397: Started agent: isControlling? true, remoteUfrag: "OicdTxJdOSmLLGcq", remotePwd: "mwlAbQuRQYPTcrZReOiJwwAWuPJZXLzw"
ice INFO: 2023/12/21 15:54:25 Setting new connection state: Checking
ice WARNING: 2023/12/21 15:54:25 Failed to ping without candidate pairs. Connection is not possible yet.
ice DEBUG: 15:54:25.147639 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.147639 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.180758 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.180758 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.182850 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.182850 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.221630 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.221630 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: turn.netbird.io:443: lookup turn.netbird.io: no such host
ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: stun.netbird.io:5555: lookup stun.netbird.io: no such host
ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true
ice DEBUG: 15:54:25.315599 agent.go:817: Ignore duplicate candidate: udp4 srflx <hiddenip>:51820 related 0.0.0.0:51820
turnc DEBUG: 15:54:25.364673 client.go:119: Resolved TURN server turn.netbird.io:443 to 35.198.143.10:443
turnc DEBUG: 15:54:25.411742 udp_conn.go:66: Initial lifetime: 600 seconds
turnc DEBUG: 15:54:25.411742 udp_conn.go:81: Started refresh allocation timer
turnc DEBUG: 15:54:25.411742 udp_conn.go:84: Started refresh permission timer
turnc DEBUG: 15:54:25.462244 udp_conn.go:438: Channel binding successful: 192.168.168.115:51820 16384
ice INFO: 2023/12/21 15:54:25 Failed to send packet: CreatePermission error response (error 403: Forbidden IP)
turnc DEBUG: 15:54:25.509529 udp_conn.go:438: Channel binding successful: 192.168.168.110:51820 16385
turnc DEBUG: 15:54:25.533151 udp_conn.go:438: Channel binding successful: <hiddenip>:61335 16386
ice DEBUG: 15:54:25.533698 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true

the strange thing found is:

ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: turn.netbird.io:443: lookup turn.netbird.io: no such host
ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: stun.netbird.io:5555: lookup stun.netbird.io: no such host

resolution on windows works correctly, I also tried to change windows dns server and restart netbird but the error still happen.
I tried for example to keep only 8.8.8.8 (google) as dns server and keep active ping on it and turn.netbird.io and was both without packet lost when netbird peer disconnect and have same error in ICE debug log.
On linux peers disconnect issue is rare, but I tried to debug also it with:
sudo bash -c "NB_LOG_LEVEL=debug PIONS_LOG_DEBUG=all netbird service run" | tee disconnect.logs
and on one ubuntu 20.04 after a disconnect I had the same resolve error.

I don't know if this problem is linked to that of disconnections, which are a significant problem on some Windows peers, or just causes a possible delay in the connection/reconnection between the peers.
Is there an issue with netbird internal dns resolution?

Any answer and advice are welcome.
If there are other debugging possibilities that might be useful that I don't know about let me know please.

@Fantu commented on GitHub (Dec 22, 2023): About the issue in my office it seems to be related to the netbird selfhost server (more exactly issue seems related to coturn) in a network with pfsense behind another nat and without access to a public ip. using an external coturn with single nat with public IP and the managed service there were no problems. After my employer has started to put netbird using the managed service on some devices of some customers (I think he will want to completely replace zerotier with that) and problems of frequent disconnections have started on some windows devices even using the managed service. After some debugging attempts without being able to find the cause and in which the netbird debug logs did not give any details regarding the disconnection or possible problems to work to try to resolve or improve yesterday thanks to the help of Pascal Fisher I finally managed to find something which could be useful. Initially try to enable ICE debug logs was difficult because trying `$env:NB_LOG_LEVEL="debug"; $env:PIONS_LOG_DEBUG="all"; netbird service run | Tee-Object -FilePath "disconnect.logs"` was not working, as seems don't possible run netbird in foreground like with linux. I tried to add env. variable also in the windows service with regedit and in other way but netbird client.log was always without ICE debug. Pascal Fisher found a solution hardcoding it in custom build I after used for debug on peers with disconnect issue. Here a cut from one ICE debug log with strange thing spotted after disconnect: ``` ice WARNING: 2023/12/21 15:54:07 Discard message from (udp4 relay 35.234.101.101:58033 related 192.168.168.110:54256), unknown TransactionID 0x8f16f56aba969acc059bffeb turnc DEBUG: 15:54:07.929646 allocation.go:132: Refresh timer 1 expired turnc DEBUG: 15:54:07.945809 allocation.go:173: Set new nonce with 16 bytes turnc DEBUG: 15:54:07.945809 allocation.go:56: Refresh allocation: 438, got new nonce. turnc DEBUG: 15:54:07.971915 allocation.go:127: Refresh permissions successful ice WARNING: 2023/12/21 15:54:11 Discard message from (udp4 relay 35.234.101.101:58033 related 192.168.168.110:54256), unknown TransactionID 0x13a8d2430fae1880ea81004d ice INFO: 2023/12/21 15:54:17 Setting new connection state: Disconnected turnc DEBUG: 15:54:17.797803 allocation.go:77: Send refresh request (dontWait=true) ice WARNING: 2023/12/21 15:54:17 Failed to read from candidate udp4 relay 35.234.101.101:37089 related 192.168.178.27:64981: read udp 35.234.101.101:37089: use of closed network connection turnc DEBUG: 15:54:17.798374 allocation.go:84: Refresh request sent turnc DEBUG: 15:54:17.798374 client.go:177: Failed to read: read tcp4 192.168.178.27:64981->35.234.101.101:443: use of closed network connection. Exiting loop ice INFO: 2023/12/21 15:54:17 Setting new connection state: Closed ice DEBUG: 15:54:25.123955 agent.go:397: Started agent: isControlling? true, remoteUfrag: "OicdTxJdOSmLLGcq", remotePwd: "mwlAbQuRQYPTcrZReOiJwwAWuPJZXLzw" ice INFO: 2023/12/21 15:54:25 Setting new connection state: Checking ice WARNING: 2023/12/21 15:54:25 Failed to ping without candidate pairs. Connection is not possible yet. ice DEBUG: 15:54:25.147639 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.147639 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.180758 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.180758 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.182850 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.182850 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.221630 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.221630 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: turn.netbird.io:443: lookup turn.netbird.io: no such host ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: stun.netbird.io:5555: lookup stun.netbird.io: no such host ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.284471 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ice DEBUG: 15:54:25.315599 agent.go:817: Ignore duplicate candidate: udp4 srflx <hiddenip>:51820 related 0.0.0.0:51820 turnc DEBUG: 15:54:25.364673 client.go:119: Resolved TURN server turn.netbird.io:443 to 35.198.143.10:443 turnc DEBUG: 15:54:25.411742 udp_conn.go:66: Initial lifetime: 600 seconds turnc DEBUG: 15:54:25.411742 udp_conn.go:81: Started refresh allocation timer turnc DEBUG: 15:54:25.411742 udp_conn.go:84: Started refresh permission timer turnc DEBUG: 15:54:25.462244 udp_conn.go:438: Channel binding successful: 192.168.168.115:51820 16384 ice INFO: 2023/12/21 15:54:25 Failed to send packet: CreatePermission error response (error 403: Forbidden IP) turnc DEBUG: 15:54:25.509529 udp_conn.go:438: Channel binding successful: 192.168.168.110:51820 16385 turnc DEBUG: 15:54:25.533151 udp_conn.go:438: Channel binding successful: <hiddenip>:61335 16386 ice DEBUG: 15:54:25.533698 agent.go:1072: Inbound STUN message: isControlling && a.isControlling == true ``` the strange thing found is: ``` ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: turn.netbird.io:443: lookup turn.netbird.io: no such host ice DEBUG: 15:54:25.262682 gather.go:403: Failed to resolve STUN host: stun.netbird.io:5555: lookup stun.netbird.io: no such host ``` resolution on windows works correctly, I also tried to change windows dns server and restart netbird but the error still happen. I tried for example to keep only 8.8.8.8 (google) as dns server and keep active ping on it and turn.netbird.io and was both without packet lost when netbird peer disconnect and have same error in ICE debug log. On linux peers disconnect issue is rare, but I tried to debug also it with: `sudo bash -c "NB_LOG_LEVEL=debug PIONS_LOG_DEBUG=all netbird service run" | tee disconnect.logs` and on one ubuntu 20.04 after a disconnect I had the same resolve error. I don't know if this problem is linked to that of disconnections, which are a significant problem on some Windows peers, or just causes a possible delay in the connection/reconnection between the peers. Is there an issue with netbird internal dns resolution? Any answer and advice are welcome. If there are other debugging possibilities that might be useful that I don't know about let me know please.

saavagebueno commented

@Fantu commented on GitHub (Jan 3, 2024):

Working ICE debug method on Windows, thanks to @mlsmaycon :

download psexec: https://learn.microsoft.com/en-us/sysinternals/downloads/psexec
add PION_LOG_TRACE=all as system environment variables
stop netbird: netbird service stop
run the following in a privileged PowerShell session: .\PsExec64.exe -s cmd.exe /c "netbird service run --log-level debug --log-file console > c:\windows\temp\netbird.out.log 2>&1"

So the netbird.out.log will contain netbird debug log and ICE trace logs

@Fantu commented on GitHub (Jan 3, 2024): Working ICE debug method on Windows, thanks to @mlsmaycon : - download psexec: https://learn.microsoft.com/en-us/sysinternals/downloads/psexec - add PION_LOG_TRACE=all as system environment variables - stop netbird: `netbird service stop` - run the following in a privileged PowerShell session: `.\PsExec64.exe -s cmd.exe /c "netbird service run --log-level debug --log-file console > c:\windows\temp\netbird.out.log 2>&1"` So the netbird.out.log will contain netbird debug log and ICE trace logs

saavagebueno commented

https://github.com/netbirdio/netbird/blob/main/client/internal/peer/env_config.go#L14

@shyer commented on GitHub (Feb 1, 2024):

I have the same problem, I change NB_ICE_DISCONNECTED_TIMEOUT_SEC to 13，solved this problem.
on centos: vi /etc/sysconfig/netbird

# NB_LOG_LEVEL=debug
# PION_LOG_DEBUG=all
NB_ICE_DISCONNECTED_TIMEOUT_SEC=13

i think iceDisconnectedTimeoutDefault should be more than iceKeepAliveDefault * 2：

if iceDisconnectedTimeoutDefault < iceKeepAliveDefault * 2

ping -> lost, -> disconnected -> netbird clean peer and reconnect

if iceDisconnectedTimeoutDefault > iceKeepAliveDefault * 2:

ping -> lost, ping -> lost, -> disconnected -> netbird reconnect
ping -> lost, ping -> pong: no disconnected, netbird ok <--

	envICEKeepAliveIntervalSec   = "NB_ICE_KEEP_ALIVE_INTERVAL_SEC"
	envICEDisconnectedTimeoutSec = "NB_ICE_DISCONNECTED_TIMEOUT_SEC"

https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L27

	iceKeepAliveDefault           = 4 * time.Second
	iceDisconnectedTimeoutDefault = 6 * time.Second # <-- may be 10  * time.Second ( 2.5 * iceKeepAliveDefault)

https://github.com/pion/ice/blob/master/agent_config.go#L21

	// keepaliveInterval used to keep candidates alive
	defaultKeepaliveInterval = 2 * time.Second

	// defaultDisconnectedTimeout is the default time till an Agent transitions disconnected
	defaultDisconnectedTimeout = 5 * time.Second

https://github.com/pion/ice/blob/master/agent.go#L631

	case a.disconnectedTimeout != 0 && disconnectedTime > a.disconnectedTimeout:
		a.updateConnectionState(ConnectionStateDisconnected)

@shyer commented on GitHub (Feb 1, 2024): I have the same problem, I change `NB_ICE_DISCONNECTED_TIMEOUT_SEC` to 13，solved this problem. on centos: vi /etc/sysconfig/netbird ``` # NB_LOG_LEVEL=debug # PION_LOG_DEBUG=all NB_ICE_DISCONNECTED_TIMEOUT_SEC=13 ``` i think `iceDisconnectedTimeoutDefault` should be more than `iceKeepAliveDefault` * 2： if iceDisconnectedTimeoutDefault < `iceKeepAliveDefault` * 2 > ping -> lost, -> disconnected -> netbird clean peer and reconnect if iceDisconnectedTimeoutDefault > `iceKeepAliveDefault` * 2: > ping -> lost, ping -> lost, -> disconnected -> netbird reconnect > ping -> lost, ping -> pong: no disconnected, netbird ok <-- https://github.com/netbirdio/netbird/blob/main/client/internal/peer/env_config.go#L14 ``` envICEKeepAliveIntervalSec = "NB_ICE_KEEP_ALIVE_INTERVAL_SEC" envICEDisconnectedTimeoutSec = "NB_ICE_DISCONNECTED_TIMEOUT_SEC" ``` https://github.com/netbirdio/netbird/blob/main/client/internal/peer/conn.go#L27 ``` iceKeepAliveDefault = 4 * time.Second iceDisconnectedTimeoutDefault = 6 * time.Second # <-- may be 10 * time.Second ( 2.5 * iceKeepAliveDefault) ``` https://github.com/pion/ice/blob/master/agent_config.go#L21 ``` // keepaliveInterval used to keep candidates alive defaultKeepaliveInterval = 2 * time.Second // defaultDisconnectedTimeout is the default time till an Agent transitions disconnected defaultDisconnectedTimeout = 5 * time.Second ``` https://github.com/pion/ice/blob/master/agent.go#L631 ``` case a.disconnectedTimeout != 0 && disconnectedTime > a.disconnectedTimeout: a.updateConnectionState(ConnectionStateDisconnected) ```

saavagebueno commented

@Fantu commented on GitHub (Feb 7, 2024):

Thanks for the information, I did some tests with keepalive lower without result, but I hadn't thought carefully about the disconnect timeout.
I tried today, starting with NB_ICE_DISCONNECTED_TIMEOUT_SEC to 30 on one case with continuous nearby disconnections, setting it on both "source" and "destination" peer and solved.
After I tried NB_ICE_DISCONNECTED_TIMEOUT_SEC to 10 which seems more reasonable to not increase the reconnection time too much in case the connection actually needs to be re-established and also with that 0 disconnection and 0 packet lost in for the test period.
Unfortunately, however, I have not yet been able to test with the ongoing problem on some networks which are occasionally very problematic and I have some doubts whether increasing the disconnection timeout will resolve.

@Fantu commented on GitHub (Feb 7, 2024): Thanks for the information, I did some tests with keepalive lower without result, but I hadn't thought carefully about the disconnect timeout. I tried today, starting with NB_ICE_DISCONNECTED_TIMEOUT_SEC to 30 on one case with continuous nearby disconnections, setting it on both "source" and "destination" peer and solved. After I tried NB_ICE_DISCONNECTED_TIMEOUT_SEC to 10 which seems more reasonable to not increase the reconnection time too much in case the connection actually needs to be re-established and also with that 0 disconnection and 0 packet lost in for the test period. Unfortunately, however, I have not yet been able to test with the ongoing problem on some networks which are occasionally very problematic and I have some doubts whether increasing the disconnection timeout will resolve.

saavagebueno commented

@Grimbart commented on GitHub (Feb 23, 2024):

i got the same problem... any news how to fix it?

or can anyone discribe me how to add the "NB_ICE_DISCONNECTED_TIMEOUT_SEC" Var?

@Grimbart commented on GitHub (Feb 23, 2024): i got the same problem... any news how to fix it? or can anyone discribe me how to add the "NB_ICE_DISCONNECTED_TIMEOUT_SEC" Var?

saavagebueno commented

@Fantu commented on GitHub (Feb 24, 2024):

For example, if you have frequent disconnections between 2 peer windows, on both peers open PowerShell as administrator and do:
[Environment]::SetEnvironmentVariable("NB_ICE_DISCONNECTED_TIMEOUT_SEC", "10", "Machine")
After restart netbird service:

netbird service stop
netbird service start

After you will try, can you write if increase disconnect timeout to 10 seconds has solved your problem, please?

@Fantu commented on GitHub (Feb 24, 2024): For example, if you have frequent disconnections between 2 peer windows, on both peers open PowerShell as administrator and do: `[Environment]::SetEnvironmentVariable("NB_ICE_DISCONNECTED_TIMEOUT_SEC", "10", "Machine")` After restart netbird service: ``` netbird service stop netbird service start ``` After you will try, can you write if increase disconnect timeout to 10 seconds has solved your problem, please?

saavagebueno commented

@Grimbart commented on GitHub (Feb 26, 2024):

Only the Period between the interrupts increase, but its not completely gone. I tried multiple Values on both Side from 5 to 30. Also I tried a Debian 12 that connect to the Windows Server and there is no Interruption.

My current Setup: I use Windows 11 (22H2) that connects to a Windows Server 2202 (21H2) over RDP. The Netbird Instance is Self-Hosted on an ESXi-Hosts that got an OPNSense as a Firewall to NAT the Inbound needed Ports.

@Grimbart commented on GitHub (Feb 26, 2024): Only the Period between the interrupts increase, but its not completely gone. I tried multiple Values on both Side from 5 to 30. Also I tried a Debian 12 that connect to the Windows Server and there is no Interruption. My current Setup: I use Windows 11 (22H2) that connects to a Windows Server 2202 (21H2) over RDP. The Netbird Instance is Self-Hosted on an ESXi-Hosts that got an OPNSense as a Firewall to NAT the Inbound needed Ports.

saavagebueno commented

@Fantu commented on GitHub (Feb 26, 2024):

Probably is the same other issue I have on my office using netbird selfhosted server in LAN behind pfsense, the problem seems double nat and with only 1 public ipv4 I can't assign one on pfsense.
Increase timeout decreased the issue, specify external-ip in the turnserver.conf seems also helped.

@Fantu commented on GitHub (Feb 26, 2024): Probably is the same other issue I have on my office using netbird selfhosted server in LAN behind pfsense, the problem seems double nat and with only 1 public ipv4 I can't assign one on pfsense. Increase timeout decreased the issue, specify external-ip in the turnserver.conf seems also helped.

saavagebueno commented

@Grimbart commented on GitHub (Feb 26, 2024):

its not only 1 public IP-Address, but in the same subnet range...

the office network send the information e.g. 12.23.34.55 (my windows 11 machine + Netbird Instance) over a Palo Alto Firewall to the e.g. 12.23.34.56 (windows server) that has the OPNSense... i will try later in the evening to modify the turnserver.conf.

@Grimbart commented on GitHub (Feb 26, 2024): its not only 1 public IP-Address, but in the same subnet range... the office network send the information e.g. 12.23.34.55 (my windows 11 machine + Netbird Instance) over a Palo Alto Firewall to the e.g. 12.23.34.56 (windows server) that has the OPNSense... i will try later in the evening to modify the turnserver.conf.

saavagebueno commented

@webash commented on GitHub (Jul 15, 2024):

@Fantu did you make any progress here? I'm having some terribly annoying inconsistencies trying to stay connected to a remote Linux box behind a NAT'd 4G connection from my Windows 11 PC. Are we saying that the connection lifetime is too short on Windows to properly handle navigating NAT delays?

@webash commented on GitHub (Jul 15, 2024): @Fantu did you make any progress here? I'm having some terribly annoying inconsistencies trying to stay connected to a remote Linux box behind a NAT'd 4G connection from my Windows 11 PC. Are we saying that the connection lifetime is too short on Windows to properly handle navigating NAT delays?

saavagebueno commented

@kgiannandrea commented on GitHub (Jul 15, 2024):

@Fantu did you make any progress here? I'm having some terribly annoying inconsistencies trying to stay connected to a remote Linux box behind a NAT'd 4G connection from my Windows 11 PC. Are we saying that the connection lifetime is too short on Windows to properly handle navigating NAT delays?

Hey @webash , we have had good results setting the following environment variable on the client machines

Variable: NB_ICE_DISCONNECTED_TIMEOUT_SEC
Value: 10

This can be set in powershell using
[Environment]::SetEnvironmentVariable("NB_ICE_DISCONNECTED_TIMEOUT_SEC", "10", "Machine")

@kgiannandrea commented on GitHub (Jul 15, 2024): > @Fantu did you make any progress here? I'm having some terribly annoying inconsistencies trying to stay connected to a remote Linux box behind a NAT'd 4G connection from my Windows 11 PC. Are we saying that the connection lifetime is too short on Windows to properly handle navigating NAT delays? Hey @webash , we have had good results setting the following environment variable on the client machines Variable: NB_ICE_DISCONNECTED_TIMEOUT_SEC Value: 10 This can be set in powershell using [Environment]::SetEnvironmentVariable("NB_ICE_DISCONNECTED_TIMEOUT_SEC", "10", "Machine")

saavagebueno commented

@webash commented on GitHub (Jul 15, 2024):

Do you know how to set this on Linux, too? I think its the sporadic nature of the 4G connection, but being the target that is the issue; so I think its disconnecting too often when actually it should just be a bit more patient. In some cases I seem to be able to keep an SSH session open to do a bunch of stuff, other times it drops rapidly.

@webash commented on GitHub (Jul 15, 2024): Do you know how to set this on Linux, too? I think its the sporadic nature of the 4G connection, but being the target that is the issue; so I think its disconnecting too often when actually it should just be a bit more patient. In some cases I seem to be able to keep an SSH session open to do a bunch of stuff, other times it drops rapidly.

saavagebueno commented