[GH-ISSUE #5602] netbird-server: relayed connection not working with traefik reverse proxy #10832

Closed
opened 2026-08-05 01:27:26 -04:00 by saavagebueno · 2 comments
Owner

Originally created by @Rudi3 on GitHub (Mar 16, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5602

Describe the problem

A clear and concise description of what the problem is.

To Reproduce

Have netbirdio/netbird-server and netbirdio/dashboard running in a kubernetes cluster (https://github.com/cclloyd/helm-netbird with manual ingress) and configure ingress with Traefik CRDs. Set idle timeout to 0 for entrypoint websecure. Create UDP ingressroute for 3478 on netbird-server (builtin STUN).

Connect 2 clients behind a NAT. Management connects, but Wireguard doesn't. There seems to be an issue with relaying or similar. Tries to reconnect every 30-60s.

(P2P connections seem to be working.)

Using the suggested ICE Agent test, I get a srflx response, but no relay. Is that to be expected with the builtin setup? (I've also tried increasing the UDP timeout, but I'm not sure if this is relevant; RFC 3489 suggests that 30s are fine)

Are there any ways I can debug this better? E.g. see if rels://my.domain:443 is working?

When I try to reconnect the clients, I'm seeing this in the sever logs:

PEER_B has Client Version 0.66.2
PEER_A has Client Version 0.66.4

INFO [peer_id: sha-PEER_B] relay/server/peer.go:120: peer exited gracefully
INFO [peer_id: sha-PEER_B] relay/server/relay.go:139: peer connected from: [...]
INFO [peer_id: sha-PEER_A] relay/server/peer.go:120: peer exited gracefully
WARN [context: GRPC, requestID: [...], accountID: [...]] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
INFO [peer_id: sha-PEER_A] relay/server/relay.go:139: peer connected from: [...]

In the traefik logs, I can see that these paths are accessed, but seemingly nothing else:

/management.ManagementService/Login
/management.ManagementService/GetServerKey
/signalexchange.SignalExchange/Send (repeated)

Are you using NetBird Cloud?

No

NetBird version

0.66.4

Is any other VPN software installed?

No

Debug output

Peers detail:
 peer.netbird:
  NetBird IP: [...]
  Public key: [...]
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://my.domain:443
  Last connection update: 2 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/148 B
  Quantum resistance: false
  Networks: [...]
  Latency: 0s

Events:
  [INFO] SYSTEM (...)
    Message: Network map updated
    Time: 16 hours, 23 minutes ago
  [INFO] SYSTEM (...)
    Message: Network map updated
    Time: 13 hours, 51 minutes ago
  [INFO] SYSTEM (...)
    Message: Network map updated
    Time: 13 hours, 45 minutes ago
  [INFO] SYSTEM (...)
    Message: Network map updated
    Time: 13 hours, 32 minutes ago
  [INFO] SYSTEM (...)
    Message: Network map updated
    Time: 10 hours, 37 minutes ago
  [INFO] SYSTEM (...)
    Message: Network map updated
    Time: 10 hours, 12 minutes ago
  [INFO] SYSTEM (...)
    Message: Network map updated
    Time: 10 hours, 10 minutes ago
  [INFO] SYSTEM (...)
    Message: Network map updated
    Time: 9 minutes, 51 seconds ago
OS: linux/amd64
Daemon version: 0.66.4
CLI version: 0.66.4
Profile: default
Management: Connected to https://my.domain:443
Signal: Connected to https://my.domain:443
Relays:
  [stun:my.domain:3478] is Available
  [rels://my.domain:443] is Available
Nameservers:
FQDN: local.netbird
NetBird IP: [...]
Interface type: Kernel
Quantum resistance: false
Lazy connection: false
SSH Server: Disabled
Networks: -
Peers count: 1/1 Connected

This repeats in the logs:

2026-03-15T14:13:48.275+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:159: sending offer with serial: 07f3219d4e
2026-03-15T14:13:48.282+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:294: OnRemoteOffer, on status ICE: Disconnected, status Relay: Disconnected
2026-03-15T14:13:48.282+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:91: received offer, running version 0.59.13, remote WireGuard listen port 51820, session id: 3d7fbe977a
2026-03-15T14:13:48.282+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:166: sending answer with serial: 07f3219d4e
2026-03-15T14:13:48.282+01:00 INFO [relay: rels://my.domain:443] shared/relay/client/client.go:255: prepare the relayed connection, waiting for remote peer: sha-[...]
2026-03-15T14:13:48.318+01:00 INFO [relay: rels://my.domain:443] shared/relay/client/client.go:293: remote peer is available: sha-[...]
2026-03-15T14:13:48.318+01:00 INFO client/iface/wgproxy/ebpf/proxy.go:120: turn conn added to wg proxy store: rels://my.domain:443, endpoint port: :2
2026-03-15T14:13:48.318+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:481: created new wgProxy for relay connection: 127.0.0.1:2
2026-03-15T14:13:48.318+01:00 INFO [peer: [PEER]] client/internal/peer/wg_watcher.go:93: WireGuard watcher started
2026-03-15T14:13:48.318+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:512: start to communicate with peer via relay
2026-03-15T14:13:48.490+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:265: OnRemoteAnswer, priority: PriorityRelay, status ICE: Disconnected, status relay: Connected
2026-03-15T14:13:48.490+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:105: received answer, running version 0.59.13, remote WireGuard listen port 51820, session id: 3d7fbe977a
2026-03-15T14:13:49.751+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:159: sending offer with serial: 42066decd8
2026-03-15T14:13:50.023+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:265: OnRemoteAnswer, priority: PriorityRelay, status ICE: Disconnected, status relay: Connected
2026-03-15T14:13:50.023+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:105: received answer, running version 0.59.13, remote WireGuard listen port 51820, session id: a97543aaf1
2026-03-15T14:14:10.928+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:294: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected
2026-03-15T14:14:10.928+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:91: received offer, running version 0.59.13, remote WireGuard listen port 51820, session id: a68d8e3866
2026-03-15T14:14:10.929+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:166: sending answer with serial: 6aaacb453a
2026-03-15T14:14:18.318+01:00 WARN [peer: [PEER]] client/internal/peer/wg_watcher.go:145: WireGuard handshake timed out: 0001-01-01 00:00:00 +0000 UTC
2026-03-15T14:14:18.318+01:00 WARN [peer: [PEER]] client/internal/peer/conn.go:577: WireGuard handshake timeout detected, closing current connection
2026-03-15T14:14:18.318+01:00 INFO [relay: rels://my.domain:443] shared/relay/client/client.go:623: free up connection to peer: sha-[...]
2026-03-15T14:14:18.319+01:00 ERRO client/iface/wgproxy/ebpf/wrapper.go:278: failed to read from turn conn (endpoint: :2): use of closed network connection

Additional context

Add any other context about the problem here.

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @Rudi3 on GitHub (Mar 16, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5602 **Describe the problem** A clear and concise description of what the problem is. **To Reproduce** Have `netbirdio/netbird-server` and `netbirdio/dashboard` running in a kubernetes cluster (https://github.com/cclloyd/helm-netbird with manual ingress) and configure ingress with Traefik CRDs. Set idle timeout to 0 for entrypoint websecure. Create UDP ingressroute for `3478` on `netbird-server` (builtin STUN). Connect 2 clients behind a NAT. Management connects, but Wireguard doesn't. There seems to be an issue with relaying or similar. Tries to reconnect every 30-60s. (P2P connections seem to be working.) Using the suggested ICE Agent test, I get a `srflx` response, but no `relay`. Is that to be expected with the builtin setup? (I've also tried increasing the UDP timeout, but I'm not sure if this is relevant; RFC 3489 suggests that 30s are fine) Are there any ways I can debug this better? E.g. see if rels://my.domain:443 is working? When I try to reconnect the clients, I'm seeing this in the sever logs: `PEER_B` has Client Version `0.66.2` `PEER_A` has Client Version `0.66.4` ``` INFO [peer_id: sha-PEER_B] relay/server/peer.go:120: peer exited gracefully INFO [peer_id: sha-PEER_B] relay/server/relay.go:139: peer connected from: [...] INFO [peer_id: sha-PEER_A] relay/server/peer.go:120: peer exited gracefully WARN [context: GRPC, requestID: [...], accountID: [...]] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled INFO [peer_id: sha-PEER_A] relay/server/relay.go:139: peer connected from: [...] ``` In the traefik logs, I can see that these paths are accessed, but seemingly nothing else: ``` /management.ManagementService/Login /management.ManagementService/GetServerKey /signalexchange.SignalExchange/Send (repeated) ``` **Are you using NetBird Cloud?** No **NetBird version** `0.66.4` **Is any other VPN software installed?** No **Debug output** ``` Peers detail: peer.netbird: NetBird IP: [...] Public key: [...] Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://my.domain:443 Last connection update: 2 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/148 B Quantum resistance: false Networks: [...] Latency: 0s Events: [INFO] SYSTEM (...) Message: Network map updated Time: 16 hours, 23 minutes ago [INFO] SYSTEM (...) Message: Network map updated Time: 13 hours, 51 minutes ago [INFO] SYSTEM (...) Message: Network map updated Time: 13 hours, 45 minutes ago [INFO] SYSTEM (...) Message: Network map updated Time: 13 hours, 32 minutes ago [INFO] SYSTEM (...) Message: Network map updated Time: 10 hours, 37 minutes ago [INFO] SYSTEM (...) Message: Network map updated Time: 10 hours, 12 minutes ago [INFO] SYSTEM (...) Message: Network map updated Time: 10 hours, 10 minutes ago [INFO] SYSTEM (...) Message: Network map updated Time: 9 minutes, 51 seconds ago OS: linux/amd64 Daemon version: 0.66.4 CLI version: 0.66.4 Profile: default Management: Connected to https://my.domain:443 Signal: Connected to https://my.domain:443 Relays: [stun:my.domain:3478] is Available [rels://my.domain:443] is Available Nameservers: FQDN: local.netbird NetBird IP: [...] Interface type: Kernel Quantum resistance: false Lazy connection: false SSH Server: Disabled Networks: - Peers count: 1/1 Connected ``` This repeats in the logs: ``` 2026-03-15T14:13:48.275+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:159: sending offer with serial: 07f3219d4e 2026-03-15T14:13:48.282+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:294: OnRemoteOffer, on status ICE: Disconnected, status Relay: Disconnected 2026-03-15T14:13:48.282+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:91: received offer, running version 0.59.13, remote WireGuard listen port 51820, session id: 3d7fbe977a 2026-03-15T14:13:48.282+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:166: sending answer with serial: 07f3219d4e 2026-03-15T14:13:48.282+01:00 INFO [relay: rels://my.domain:443] shared/relay/client/client.go:255: prepare the relayed connection, waiting for remote peer: sha-[...] 2026-03-15T14:13:48.318+01:00 INFO [relay: rels://my.domain:443] shared/relay/client/client.go:293: remote peer is available: sha-[...] 2026-03-15T14:13:48.318+01:00 INFO client/iface/wgproxy/ebpf/proxy.go:120: turn conn added to wg proxy store: rels://my.domain:443, endpoint port: :2 2026-03-15T14:13:48.318+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:481: created new wgProxy for relay connection: 127.0.0.1:2 2026-03-15T14:13:48.318+01:00 INFO [peer: [PEER]] client/internal/peer/wg_watcher.go:93: WireGuard watcher started 2026-03-15T14:13:48.318+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:512: start to communicate with peer via relay 2026-03-15T14:13:48.490+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:265: OnRemoteAnswer, priority: PriorityRelay, status ICE: Disconnected, status relay: Connected 2026-03-15T14:13:48.490+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:105: received answer, running version 0.59.13, remote WireGuard listen port 51820, session id: 3d7fbe977a 2026-03-15T14:13:49.751+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:159: sending offer with serial: 42066decd8 2026-03-15T14:13:50.023+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:265: OnRemoteAnswer, priority: PriorityRelay, status ICE: Disconnected, status relay: Connected 2026-03-15T14:13:50.023+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:105: received answer, running version 0.59.13, remote WireGuard listen port 51820, session id: a97543aaf1 2026-03-15T14:14:10.928+01:00 INFO [peer: [PEER]] client/internal/peer/conn.go:294: OnRemoteOffer, on status ICE: Disconnected, status Relay: Connected 2026-03-15T14:14:10.928+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:91: received offer, running version 0.59.13, remote WireGuard listen port 51820, session id: a68d8e3866 2026-03-15T14:14:10.929+01:00 INFO [peer: [PEER]] client/internal/peer/handshaker.go:166: sending answer with serial: 6aaacb453a 2026-03-15T14:14:18.318+01:00 WARN [peer: [PEER]] client/internal/peer/wg_watcher.go:145: WireGuard handshake timed out: 0001-01-01 00:00:00 +0000 UTC 2026-03-15T14:14:18.318+01:00 WARN [peer: [PEER]] client/internal/peer/conn.go:577: WireGuard handshake timeout detected, closing current connection 2026-03-15T14:14:18.318+01:00 INFO [relay: rels://my.domain:443] shared/relay/client/client.go:623: free up connection to peer: sha-[...] 2026-03-15T14:14:18.319+01:00 ERRO client/iface/wgproxy/ebpf/wrapper.go:278: failed to read from turn conn (endpoint: :2): use of closed network connection ``` **Additional context** Add any other context about the problem here. **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2026-08-05 01:27:26 -04:00
Author
Owner

@Rudi3 commented on GitHub (Mar 16, 2026):

Actually, it looks like it's working just fine. PEER_B seems to have a problem. Relayed connections work fine with different peer.
I'll recheck and probably close this issue afterwards.

<!-- gh-comment-id:4071556037 --> @Rudi3 commented on GitHub (Mar 16, 2026): Actually, it looks like it's working just fine. `PEER_B` seems to have a problem. Relayed connections work fine with different peer. I'll recheck and probably close this issue afterwards.
Author
Owner

@Rudi3 commented on GitHub (Mar 23, 2026):

Yes, it was an just issue with the client on openwrt. (Changing to user-space wireguard procd_append_param env NB_WG_KERNEL_DISABLED=true made it work there as well - but there may better ways to fix that.)

<!-- gh-comment-id:4114310097 --> @Rudi3 commented on GitHub (Mar 23, 2026): Yes, it was an just issue with the client on openwrt. (Changing to user-space wireguard `procd_append_param env NB_WG_KERNEL_DISABLED=true` made it work there as well - but there may better ways to fix that.)
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#10832