[GH-ISSUE #4769] Intermittent connectivity loss to routing peers requiring manual restart #9802

Open
opened 2026-08-05 01:23:31 -04:00 by saavagebueno · 10 comments
Owner

Originally created by @theophileds on GitHub (Nov 10, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/4769

Describe the problem

Users intermittently lose connectivity to NetBird routing peers (VPC gateway instances), making infrastructure resources unreachable. The issue affects both redundant routing peers simultaneously, despite
each being on separate EC2 instances with independent network paths.

Currently, the only reliable recovery method is manually executing netbird down && netbird up on the affected routing peer(s). We have implemented automated daily restarts via cron (with retry logic and
randomized delays), but connectivity issues still occur randomly between scheduled restarts, requiring manual intervention from operations team.

To Reproduce

Steps to reproduce the behavior:

  1. Set up self-hosted NetBird with routing peers configured as VPC gateways
  2. Configure redundant routing peers (2 per VPC) for high availability
  3. Connect client machines to access resources through routing peers
  4. Normal operation works initially
  5. After random intervals (hours to days), clients lose connectivity to resources behind routing peers
  6. Routing peer shows Status: Idle or Status: Connecting for affected client peers with "Last WireGuard handshake: -"
  7. Client-side netbird status shows routing peer in "Connecting" state
  8. Only resolution: SSH to routing peer and run netbird down && netbird up

Expected behavior

  • Routing peers should maintain stable connectivity to client peers

Are you using NetBird Cloud?

Self-hosted NetBird control plane (on-premise deployment)

NetBird version

Routing peers: 0.59.12 (Amazon Linux 2023)
Clients: Mixed versions 0.59.10 - 0.59.12 (all affected randomly regardless of version)

Is any other VPN software installed?

No other VPN software installed on routing peers or affected clients.

Debug output

ERRO shared/signal/client/grpc.go:417: Stream receive error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
WARN shared/signal/client/grpc.go:177: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR

Additional Errors Found:

  1. Management service keepalive failures:
WARN shared/management/client/grpc.go:172: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout
  1. Relay connection issues:
WARN client/internal/peer/worker_relay.go:124: failed to close relay connection: use of closed network connection
  1. TURN connection failures:
ERRO client/iface/wgproxy/ebpf/wrapper.go:159: failed to read from turn conn: use of closed network connection

Environment:

  • Deployment: AWS EC2 routing peers (Amazon Linux 2023.9.20251014, kernel 6.12.46)
  • Architecture: 2 routing peers per VPC (dev + production VPCs)
  • Use case: VPC gateway routing to access RDS, EC2, ...
  • Network: Behind AWS NLB with HTTP2Optional ALPN policy for gRPC support

Mitigation attempts:

  • Implemented automated daily reconnection via cron with 10 retry attempts
  • Deployed redundant routing peers for failover
  • Neither prevents the issue - both peers can become unreachable simultaneously for some users.

Pattern observations:

  • Issue affects users randomly regardless of client version (0.59.10-0.59.12)
  • Both routing peers can fail simultaneously despite being on separate instances
  • Manual netbird down && netbird up on routing peer immediately resolves connectivity

Question: Have you seen this pattern before in other deployments? It seems like a connection state management issue where NetBird doesn't detect or recover from stale peer connections automatically.

Have you tried these troubleshooting steps?

  • Reviewed https://docs.netbird.io/how-to/troubleshooting-client
  • Checked for newer NetBird versions (running latest 0.59.x)
  • Searched for similar issues on GitHub
  • Restarted the NetBird client (this is the only working fix)
  • Disabled other VPN software (none installed)
  • Checked firewall settings (NLB ALPN policy configured correctly)
Originally created by @theophileds on GitHub (Nov 10, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/4769 **Describe the problem** Users intermittently lose connectivity to NetBird routing peers (VPC gateway instances), making infrastructure resources unreachable. The issue affects both redundant routing peers simultaneously, despite each being on separate EC2 instances with independent network paths. Currently, the only reliable recovery method is manually executing `netbird down && netbird up` on the affected routing peer(s). We have implemented automated daily restarts via cron (with retry logic and randomized delays), but connectivity issues still occur randomly between scheduled restarts, requiring manual intervention from operations team. **To Reproduce** Steps to reproduce the behavior: 1. Set up self-hosted NetBird with routing peers configured as VPC gateways 2. Configure redundant routing peers (2 per VPC) for high availability 3. Connect client machines to access resources through routing peers 4. Normal operation works initially 5. After random intervals (hours to days), clients lose connectivity to resources behind routing peers 6. Routing peer shows Status: Idle or Status: Connecting for affected client peers with "Last WireGuard handshake: -" 7. Client-side netbird status shows routing peer in "Connecting" state 8. Only resolution: SSH to routing peer and run `netbird down && netbird up` **Expected behavior** - Routing peers should maintain stable connectivity to client peers **Are you using NetBird Cloud?** Self-hosted NetBird control plane (on-premise deployment) **NetBird version** Routing peers: 0.59.12 (Amazon Linux 2023) Clients: Mixed versions 0.59.10 - 0.59.12 (all affected randomly regardless of version) **Is any other VPN software installed?** No other VPN software installed on routing peers or affected clients. **Debug output** ``` ERRO shared/signal/client/grpc.go:417: Stream receive error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR WARN shared/signal/client/grpc.go:177: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR ``` Additional Errors Found: 1. Management service keepalive failures: ``` WARN shared/management/client/grpc.go:172: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within timeout ``` 2. Relay connection issues: ``` WARN client/internal/peer/worker_relay.go:124: failed to close relay connection: use of closed network connection ```` 3. TURN connection failures: ``` ERRO client/iface/wgproxy/ebpf/wrapper.go:159: failed to read from turn conn: use of closed network connection ``` Environment: - Deployment: AWS EC2 routing peers (Amazon Linux 2023.9.20251014, kernel 6.12.46) - Architecture: 2 routing peers per VPC (dev + production VPCs) - Use case: VPC gateway routing to access RDS, EC2, ... - Network: Behind AWS NLB with HTTP2Optional ALPN policy for gRPC support Mitigation attempts: - Implemented automated daily reconnection via cron with 10 retry attempts - Deployed redundant routing peers for failover - Neither prevents the issue - both peers can become unreachable simultaneously for some users. Pattern observations: - Issue affects users randomly regardless of client version (0.59.10-0.59.12) - Both routing peers can fail simultaneously despite being on separate instances - Manual netbird down && netbird up on routing peer immediately resolves connectivity Question: Have you seen this pattern before in other deployments? It seems like a connection state management issue where NetBird doesn't detect or recover from stale peer connections automatically. Have you tried these troubleshooting steps? - Reviewed https://docs.netbird.io/how-to/troubleshooting-client - Checked for newer NetBird versions (running latest 0.59.x) - Searched for similar issues on GitHub - Restarted the NetBird client (this is the only working fix) - Disabled other VPN software (none installed) - Checked firewall settings (NLB ALPN policy configured correctly)
saavagebueno added the triage-needed label 2026-08-05 01:23:31 -04:00
Author
Owner

@pkarc commented on GitHub (Nov 13, 2025):

I have exactly the same problem. The only thing that's different from the self-hosted guide is that I'm using Haproxy instead of nginx. But the behavior it's the same I have to restart the router peer client to make the routing work again, the clients see them connected, but no routing.

<!-- gh-comment-id:3530693555 --> @pkarc commented on GitHub (Nov 13, 2025): I have exactly the same problem. The only thing that's different from the self-hosted guide is that I'm using Haproxy instead of nginx. But the behavior it's the same I have to restart the router peer client to make the routing work again, the clients see them connected, but no routing.
Author
Owner

@theophileds commented on GitHub (Nov 14, 2025):

I was experiencing similar timeout errors:

2025/11/09 23:00:40 [error] 29#29: *12284319 upstream timed out (110: Operation timed out) while reading upstream, client: x.x.x.x, server: example.domain.com, request: "POST /management.ManagementService/Sync HTTP/2.0", upstream: "grpc://192.168.x.x:80", host: "example.domain.com:443"

Fixed it with these ingress settings:

ingress:
  enabled: true
  className: nginx-nlb
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: 'GRPC'
    nginx.ingress.kubernetes.io/proxy-read-timeout: '604800'
    nginx.ingress.kubernetes.io/proxy-send-timeout: '604800'
    nginx.ingress.kubernetes.io/server-snippet: |
      grpc_socket_keepalive on;
      client_header_timeout 7d;
      client_body_timeout 7d;
<!-- gh-comment-id:3531889615 --> @theophileds commented on GitHub (Nov 14, 2025): I was experiencing similar timeout errors: ``` 2025/11/09 23:00:40 [error] 29#29: *12284319 upstream timed out (110: Operation timed out) while reading upstream, client: x.x.x.x, server: example.domain.com, request: "POST /management.ManagementService/Sync HTTP/2.0", upstream: "grpc://192.168.x.x:80", host: "example.domain.com:443" ``` Fixed it with these ingress settings: ```yaml ingress: enabled: true className: nginx-nlb annotations: nginx.ingress.kubernetes.io/backend-protocol: 'GRPC' nginx.ingress.kubernetes.io/proxy-read-timeout: '604800' nginx.ingress.kubernetes.io/proxy-send-timeout: '604800' nginx.ingress.kubernetes.io/server-snippet: | grpc_socket_keepalive on; client_header_timeout 7d; client_body_timeout 7d; ```
Author
Owner

@Krishnacore commented on GitHub (Nov 25, 2025):

Same problem here - only router restarts help. Are any devs actively investigating this?

<!-- gh-comment-id:3577373500 --> @Krishnacore commented on GitHub (Nov 25, 2025): Same problem here - only router restarts help. Are any devs actively investigating this?
Author
Owner

@Silex commented on GitHub (Jan 13, 2026):

From my finding this comes from wireguard connexion not being properly detected as "dead".

Their latest handshake is -:

foo.bar.baz:
  NetBird IP: 100.70.205.198
  Public key: oWxONSNjVcitGblH6DA5OGDTh6PEIM6m4UwU37CODUE=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 192.168.168.106:51820/116.202.18.146:58980
  Relay server address:
  Last connection update: 3 hours, 36 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/354.8 KiB
  Quantum resistance: false
  Networks: -
  Latency: 51.9013ms
<!-- gh-comment-id:3743889428 --> @Silex commented on GitHub (Jan 13, 2026): From my finding this comes from wireguard connexion not being properly detected as "dead". Their latest handshake is `-`: ``` foo.bar.baz: NetBird IP: 100.70.205.198 Public key: oWxONSNjVcitGblH6DA5OGDTh6PEIM6m4UwU37CODUE= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/prflx ICE candidate endpoints (Local/Remote): 192.168.168.106:51820/116.202.18.146:58980 Relay server address: Last connection update: 3 hours, 36 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/354.8 KiB Quantum resistance: false Networks: - Latency: 51.9013ms ```
Author
Owner

@pappz commented on GitHub (Jan 13, 2026):

From my finding this comes from wireguard connexion not being properly detected as "dead".

Their latest handshake is -:

foo.bar.baz:
  NetBird IP: 100.70.205.198
  Public key: oWxONSNjVcitGblH6DA5OGDTh6PEIM6m4UwU37CODUE=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 192.168.168.106:51820/116.202.18.146:58980
  Relay server address:
  Last connection update: 3 hours, 36 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/354.8 KiB
  Quantum resistance: false
  Networks: -
  Latency: 51.9013ms

Could you generate a debug bundle in case of this situation? "netbird debug bundle -S -U"

<!-- gh-comment-id:3747351245 --> @pappz commented on GitHub (Jan 13, 2026): > From my finding this comes from wireguard connexion not being properly detected as "dead". > > Their latest handshake is `-`: > > ``` > foo.bar.baz: > NetBird IP: 100.70.205.198 > Public key: oWxONSNjVcitGblH6DA5OGDTh6PEIM6m4UwU37CODUE= > Status: Connected > -- detail -- > Connection type: P2P > ICE candidate (Local/Remote): host/prflx > ICE candidate endpoints (Local/Remote): 192.168.168.106:51820/116.202.18.146:58980 > Relay server address: > Last connection update: 3 hours, 36 minutes ago > Last WireGuard handshake: - > Transfer status (received/sent) 0 B/354.8 KiB > Quantum resistance: false > Networks: - > Latency: 51.9013ms > ``` Could you generate a debug bundle in case of this situation? "netbird debug bundle -S -U"
Author
Owner

@Silex commented on GitHub (Jan 14, 2026):

@pappz: will do that in the PR but unfortunately I have to wait until it happens again

<!-- gh-comment-id:3748457506 --> @Silex commented on GitHub (Jan 14, 2026): @pappz: will do that in the PR but unfortunately I have to wait until it happens again
Author
Owner

@weskoerber commented on GitHub (Feb 10, 2026):

We're experiencing this issue too, and it is happening on several different client on a daily basis. We have 73 peers (a few of them routing peers) on Windows, MacOS, and Windows. Our management version is v0.65.4. The clients we've been having the most issues with are also v0.65.4. It seems our Windows and MacOS peers are most problematic.

I have debug bundles for both the routing peer and a client experiencing the issues, as well as netbird status -dA output from both:

unzip -l netbird-debug-bundles.zip
Archive:  netbird-debug-bundles.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2026-02-10 20:00   netbird-debug-bundles/
   574580  2026-02-10 19:43   netbird-debug-bundles/netbird.debug.1187686631.zip
  1680231  2026-02-10 19:38   netbird-debug-bundles/netbird.debug.1662341671.zip
    30370  2026-02-10 19:58   netbird-debug-bundles/netbird_status_client
    30222  2026-02-10 20:00   netbird-debug-bundles/netbird_status_routing_peer
      221  2026-02-10 20:00   netbird-debug-bundles/files.txt
---------                     -------
  2315624                     6 files

Within this zip, files.txt has a very brief description of each filename.

Please let me know if I should create a new issue for this, or if here is an appropriate place. Also, let me know if I need to provide anything else!

Edit: forgot to attach the zip.

netbird-debug-bundles.zip

<!-- gh-comment-id:3881573464 --> @weskoerber commented on GitHub (Feb 10, 2026): We're experiencing this issue too, and it is happening on several different client on a daily basis. We have 73 peers (a few of them routing peers) on Windows, MacOS, and Windows. Our management version is `v0.65.4`. The clients we've been having the most issues with are also `v0.65.4`. It seems our Windows and MacOS peers are most problematic. I have debug bundles for both the routing peer and a client experiencing the issues, as well as `netbird status -dA` output from both: ```shell unzip -l netbird-debug-bundles.zip Archive: netbird-debug-bundles.zip Length Date Time Name --------- ---------- ----- ---- 0 2026-02-10 20:00 netbird-debug-bundles/ 574580 2026-02-10 19:43 netbird-debug-bundles/netbird.debug.1187686631.zip 1680231 2026-02-10 19:38 netbird-debug-bundles/netbird.debug.1662341671.zip 30370 2026-02-10 19:58 netbird-debug-bundles/netbird_status_client 30222 2026-02-10 20:00 netbird-debug-bundles/netbird_status_routing_peer 221 2026-02-10 20:00 netbird-debug-bundles/files.txt --------- ------- 2315624 6 files ``` Within this zip, `files.txt` has a very brief description of each filename. Please let me know if I should create a new issue for this, or if here is an appropriate place. Also, let me know if I need to provide anything else! Edit: forgot to attach the zip. [netbird-debug-bundles.zip](https://github.com/user-attachments/files/25224408/netbird-debug-bundles.zip)
Author
Owner

@Silex commented on GitHub (Feb 11, 2026):

@pappz 👀

<!-- gh-comment-id:3882687322 --> @Silex commented on GitHub (Feb 11, 2026): @pappz 👀
Author
Owner

@weskoerber commented on GitHub (Feb 11, 2026):

Status after doing netbird down and netbird up:

PS C:\Users\acs> netbird status -dA --filter-by-names rpi4-wg-sales
Peers detail:
 rpi4-wg-sales.netbird.selfhosted:
  NetBird IP: 100.110.72.80
  Public key: NbkWiDrul7SrkpS+sJALmghK1tk3nxG1fCxCN8falzA=
  Status: Idle
  -- detail --
  Connection type: -
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address:
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: 192.168.2.0/24
  Latency: 0s

After doing down & up, the status still showed the same problematic-looking status. I tried to ping a host on the 192.168.2.0/24 subnet, and the first time it failed. However, after retrying the ping, it succeeded:

Pinging 192.168.2.97 with 32 bytes of data:
Request timed out.

Ping statistics for 192.168.2.97:
    Packets: Sent = 1, Received = 0, Lost = 1 (100% loss),
Control-C
PS C:\Users\acs> ping 192.168.2.97

Pinging 192.168.2.97 with 32 bytes of data:
Reply from 192.168.2.97: bytes=32 time=103ms TTL=127
Reply from 192.168.2.97: bytes=32 time=103ms TTL=127

Ping statistics for 192.168.2.97:
    Packets: Sent = 2, Received = 2, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 103ms, Maximum = 103ms, Average = 103ms
Control-C

I suspect that during the first ping, the lazy connection was being initialized. It must have succeeded, because the second ping reached the host.

Now, after the successful ping, here's the status output (filtered on the routing peer) on the client:

PS C:\Users\acs> netbird status -dA --filter-by-names rpi4-wg-sales
Peers detail:
 rpi4-wg-sales.netbird.selfhosted:
  NetBird IP: 100.110.72.80
  Public key: NbkWiDrul7SrkpS+sJALmghK1tk3nxG1fCxCN8falzA=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://netbird.anon-ctbtg.domain:443
  Last connection update: 14 seconds ago
  Last WireGuard handshake: 9 seconds ago
  Transfer status (received/sent) 412 B/468 B
  Quantum resistance: false
  Networks: 192.168.2.0/24
  Latency: 0s

Here's the debug bundle for this client:

vm11-netbird-debug.zip

<!-- gh-comment-id:3887100680 --> @weskoerber commented on GitHub (Feb 11, 2026): Status after doing `netbird down` and `netbird up`: ``` PS C:\Users\acs> netbird status -dA --filter-by-names rpi4-wg-sales Peers detail: rpi4-wg-sales.netbird.selfhosted: NetBird IP: 100.110.72.80 Public key: NbkWiDrul7SrkpS+sJALmghK1tk3nxG1fCxCN8falzA= Status: Idle -- detail -- Connection type: - ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Networks: 192.168.2.0/24 Latency: 0s ``` After doing down & up, the status still showed the same problematic-*looking* status. I tried to ping a host on the 192.168.2.0/24 subnet, and the first time it failed. However, after retrying the ping, it succeeded: ``` Pinging 192.168.2.97 with 32 bytes of data: Request timed out. Ping statistics for 192.168.2.97: Packets: Sent = 1, Received = 0, Lost = 1 (100% loss), Control-C PS C:\Users\acs> ping 192.168.2.97 Pinging 192.168.2.97 with 32 bytes of data: Reply from 192.168.2.97: bytes=32 time=103ms TTL=127 Reply from 192.168.2.97: bytes=32 time=103ms TTL=127 Ping statistics for 192.168.2.97: Packets: Sent = 2, Received = 2, Lost = 0 (0% loss), Approximate round trip times in milli-seconds: Minimum = 103ms, Maximum = 103ms, Average = 103ms Control-C ``` I suspect that during the first ping, the lazy connection was being initialized. It must have succeeded, because the second ping reached the host. Now, after the successful ping, here's the status output (filtered on the routing peer) on the client: ``` PS C:\Users\acs> netbird status -dA --filter-by-names rpi4-wg-sales Peers detail: rpi4-wg-sales.netbird.selfhosted: NetBird IP: 100.110.72.80 Public key: NbkWiDrul7SrkpS+sJALmghK1tk3nxG1fCxCN8falzA= Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://netbird.anon-ctbtg.domain:443 Last connection update: 14 seconds ago Last WireGuard handshake: 9 seconds ago Transfer status (received/sent) 412 B/468 B Quantum resistance: false Networks: 192.168.2.0/24 Latency: 0s ``` Here's the debug bundle for this client: [vm11-netbird-debug.zip](https://github.com/user-attachments/files/25244008/vm11-netbird-debug.zip)
Author
Owner

@weskoerber commented on GitHub (Feb 27, 2026):

After disabling lazy connections on the server and migrating from coturn to the embedded STUN server (docs, our intermittent connectivity issues seem to have gone away.

That's not to say the issue is solved, just that we aren't hitting the conditions that cause it in the first place.

<!-- gh-comment-id:3975386343 --> @weskoerber commented on GitHub (Feb 27, 2026): After disabling lazy connections on the server and migrating from coturn to the embedded STUN server ([docs](https://docs.netbird.io/selfhosted/migration/coturn-to-stun-migration), our intermittent connectivity issues seem to have gone away. That's not to say the issue is solved, just that we aren't hitting the conditions that cause it in the first place.
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#9802