[GH-ISSUE #5492] 💥 Reverse Proxy Fatal: Embedded NetBird Clients Can't Authenticate - All Forwarding Fails #11302

Open
opened 2026-08-05 01:29:13 -04:00 by saavagebueno · 0 comments
Owner

Originally created by @Nexx4 on GitHub (Mar 3, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5492

Describe the problem

The NetBird reverse proxy successfully connects to the management service and receives service mappings, but fails to forward ANY traffic due to embedded NetBird client authentication timeouts. The proxy accepts incoming HTTPS requests and correctly identifies target backends, but returns 504 Gateway Timeout after ~30 seconds because it cannot create authenticated NetBird client connections for peer tunneling.

Error: client start failed: create auth client: create connection: dial context: context deadline exceeded

To Reproduce

Steps to reproduce the behavior:

  1. Deploy self-hosted NetBird with built-in Traefik using getting-started.sh (v0.66.0)
  2. Enable reverse proxy feature during setup
  3. Create a reverse proxy service in the NetBird dashboard:
    • Choose any NetBird peer as target (local or remote)
    • Set target port (e.g., 3000, 8888)
    • Assign a subdomain (e.g., test.proxy.xxx.cloud)
  4. Access the service via browser: https://test.proxy.xxx.cloud
  5. Observe 504 Gateway Timeout after ~30 seconds

Expected behavior

The proxy should:

  1. Create an embedded NetBird client for the target peer
  2. Authenticate the client with the management service
  3. Establish a WireGuard tunnel to the peer
  4. Forward HTTP traffic through the tunnel to the backend service
  5. Return the backend response to the client

Actual behavior:

  • Proxy receives the request
  • Proxy matches the service configuration
  • Proxy attempts to create NetBird auth client
  • Auth client connection times out after ~30s
  • Returns 504 Gateway Timeout to client

Are you using NetBird Cloud?

No, using self-hosted NetBird control plane (management + signal + relay in combined mode).

NetBird version

netbird version: 0.66.0

Proxy container: netbirdio/reverse-proxy:latest
Server container: netbirdio/netbird-server:latest

Is any other VPN software installed?

Yes, Tailscale is installed on the VPS host (used for management access).

  • Tailscale does NOT interfere with NetBird
  • VPS NetBird peer works correctly when enabled (can ping other peers, 2/2 connected)
  • Issue persists with Tailscale running, stopped, or with VPS NetBird peer disabled

Debug output

Proxy Container Logs (Debug Mode)

Successful Initial Connection:

2026-03-03T18:26:37.428Z DEBG [gRPC_address: netbird-server:80, TLS_enabled: false] proxy/server.go:383: starting management gRPC client
2026-03-03T18:26:37.429Z INFO [account_id: d6je6q5pk78c73abqer0, service_id: d6jeam5pk78c7396htp0, public_key: lJweckS9ZbrqgtBqGef0y0QMbMpBNUbSgkyxFNXgCEM=] proxy/internal/roundtrip/netbird.go:226: proxy peer authenticated successfully with management
2026-03-03T18:26:37.431Z INFO proxy/server.go:614: Initial mapping sync complete

Failed Forwarding Request:

2026-03-03T18:37:04.770Z DEBG proxy/internal/accesslog/middleware.go:27: request: request_id=d6jik55pk78c73ctkm6g method=GET host=local-test.proxy.xxx.cloud path=/
2026-03-03T18:37:04.770Z DEBG proxy/internal/auth/middleware.go:95: checking authentication for host: local-test.proxy.xxx.cloud, exists: true
2026-03-03T18:37:04.770Z DEBG proxy/internal/proxy/servicemapping.go:59: matched host: local-test.proxy.xxx.cloud, path: / -> http://100.66.129.92:8888/

[30 seconds of timeout...]

2026-03-03T18:37:24.943Z ERRO shared/management/client/grpc.go:82: failed creating connection to Management Service: create connection: dial context: context deadline exceeded
2026-03-03T18:37:24.943Z ERRO client/internal/auth/auth.go:51: failed connecting to Management Service http://netbird-server:80: create connection: dial context: context deadline exceeded
2026-03-03T18:37:24.943Z WARN proxy/internal/proxy/reverseproxy.go:298: proxy error: request_id=d6jik55pk78c73ctkm6g client_ip=<CLIENT_IP> method=GET host=100.66.129.92:8888 path=/ status=504 title="Request Timeout" err=client start failed: create auth client: create connection: dial context: context deadline exceeded
2026-03-03T18:37:24.943Z DEBG proxy/internal/accesslog/middleware.go:70: response: request_id=d6jik55pk78c73ctkm6g method=GET host=local-test.proxy.xxx.cloud path=/ status=504 duration=80171ms source=<CLIENT_IP> origin=proxy_error service=d6jijodpk78c73apr4hg account=d6je6q5pk78c73abqer0

Management Server Logs

Proxy Registration (Successful):

2026-03-03T18:26:37.313Z INFO management/internals/server/boot.go:163: ProxyService registered on gRPC server
2026-03-03T18:26:37.313Z INFO management/internals/shared/grpc/proxy.go:159: New proxy connection from 172.30.0.10:58244
2026-03-03T18:26:37.313Z INFO management/internals/shared/grpc/proxy.go:188: Proxy registered in cluster [proxy_id: netbird-proxy-20260303182636, total_proxies: 1]
2026-03-03T18:26:37.317Z INFO management/internals/shared/grpc/onetime_token.go:126: Token validated and consumed for proxy

Note: No auth client connection attempts appear in management server logs - the embedded NetBird clients never reach the management service.

Network Connectivity Tests (All Successful)

# From proxy container:
$ docker exec netbird-proxy ping -c 2 100.66.129.92
64 bytes from 100.66.129.92: seq=0 ttl=63 time=18.149 ms  ✅

$ docker exec netbird-proxy ping -c 2 100.66.181.135
64 bytes from 100.66.181.135: seq=0 ttl=63 time=18.079 ms  ✅

$ docker exec netbird-proxy wget -qO- http://172.30.0.5
<h1>Hello from VPS Local Nginx!</h1>  ✅

# Proxy container CAN reach all targets via standard networking

Additional context

Key Discovery: Two Separate Connection Types

The proxy makes TWO distinct types of connections to the management service:

  1. Mapping Stream (proxy's own gRPC connection)

    • Status: Works perfectly every time
    • Purpose: Get service configurations from management.ProxyService/GetMappingUpdate
    • Result: "proxy peer authenticated successfully", "Initial mapping sync complete"
  2. Auth Clients (embedded NetBird clients for peer connections)

    • Status: Always timeout after ~30 seconds
    • Purpose: Authenticate and establish WireGuard tunnels to target peers
    • Result: "create auth client: create connection: dial context: context deadline exceeded"

Testing Matrix

Target Type Target Details Network Test Proxy Forward Error
Remote peer Desktop 100.66.181.135:3000 Ping 18ms 504 Auth client timeout
Local peer VPS 100.66.129.92:8888 Direct access 504 Auth client timeout
Docker container nginx 172.30.0.5:80 (same network) Direct wget 504 Auth client timeout

Conclusion: The issue is NOT network/routing related - it's a fundamental problem with embedded NetBird client authentication.

Configuration Attempted

Management Address:

  • http://netbird-server:80 (official config per getting-started.sh)
  • http://172.30.0.10:80 (Traefik internal IP)
  • http://<container-ip>:80 (direct container IP)
  • All result in the same auth client timeout

Traefik Routing:
Tried 10+ routing configurations including:

  • Path-based matching: PathPrefix(/management.) || PathPrefix(/signalexchange.)
  • Host-based matching: Host(172.30.0.10)
  • Header-based matching: HeadersRegexp(Content-Type, ^application/grpc)
  • Catch-all routing
  • Separate HTTP/HTTPS routers
  • None resolved the auth client timeout

Network Mode:

  • Bridge network (default)
  • Host network mode
  • Both exhibit the same issue

Environment Details

Deployment:

  • Docker Compose on Debian 12
  • Traefik v3.6 (built-in configuration from getting-started.sh)
  • Docker network: 172.30.0.0/24
  • No global HTTP→HTTPS redirect (removed to support internal gRPC on port 80)

Containers:

proxy:
  image: netbirdio/reverse-proxy:latest
  networks: [netbird]
  cap_add: [NET_ADMIN, NET_RAW]
  env:
    NB_PROXY_MANAGEMENT_ADDRESS: http://netbird-server:80
    NB_PROXY_ALLOW_INSECURE: true
    NB_PROXY_DOMAIN: proxy.xxx.cloud

VPS NetBird Peer Status (when enabled):

Management: Connected
Signal: Connected
Peers count: 2/2 Connected
NetBird IP: 100.66.129.92/16
  1. Protocol Mismatch: netbird-server speaks HTTP/1.1 on port 80, requiring Traefik's h2c service to translate to HTTP/2 for gRPC clients

  2. Official Script: The getting-started.sh script routing rules do NOT include /management.ProxyService/ - only /management.ManagementService/ and /signalexchange.SignalExchange/

  3. Management Server Behavior: Successfully processes proxy's mapping stream connection but never receives auth client connection attempts in logs

Request for Help

  1. Is there additional configuration required for embedded NetBird client authentication?
  2. Should embedded clients use a different management endpoint than the proxy's mapping stream?
  3. Are there Docker networking requirements we're missing?

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting
  • Checked for newer NetBird versions (using latest 0.66.0)
  • Searched for similar issues on GitHub (found #5349, #2934, #1048)
  • Restarted the NetBird client (multiple times, full docker compose down/up)
  • Disabled other VPN software (Tailscale - issue persists)
  • Checked firewall settings (UFW disabled, iptables rules verified, Docker networking functional)

Thank you for any guidance!

Originally created by @Nexx4 on GitHub (Mar 3, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5492 **Describe the problem** The NetBird reverse proxy successfully connects to the management service and receives service mappings, but fails to forward ANY traffic due to embedded NetBird client authentication timeouts. The proxy accepts incoming HTTPS requests and correctly identifies target backends, but returns 504 Gateway Timeout after ~30 seconds because it cannot create authenticated NetBird client connections for peer tunneling. Error: `client start failed: create auth client: create connection: dial context: context deadline exceeded` **To Reproduce** Steps to reproduce the behavior: 1. Deploy self-hosted NetBird with built-in Traefik using `getting-started.sh` (v0.66.0) 2. Enable reverse proxy feature during setup 3. Create a reverse proxy service in the NetBird dashboard: - Choose any NetBird peer as target (local or remote) - Set target port (e.g., 3000, 8888) - Assign a subdomain (e.g., `test.proxy.xxx.cloud`) 4. Access the service via browser: `https://test.proxy.xxx.cloud` 5. Observe 504 Gateway Timeout after ~30 seconds **Expected behavior** The proxy should: 1. Create an embedded NetBird client for the target peer 2. Authenticate the client with the management service 3. Establish a WireGuard tunnel to the peer 4. Forward HTTP traffic through the tunnel to the backend service 5. Return the backend response to the client **Actual behavior:** - Proxy receives the request ✅ - Proxy matches the service configuration ✅ - Proxy attempts to create NetBird auth client ❌ - Auth client connection times out after ~30s ❌ - Returns 504 Gateway Timeout to client ❌ **Are you using NetBird Cloud?** No, using **self-hosted** NetBird control plane (management + signal + relay in combined mode). **NetBird version** ``` netbird version: 0.66.0 ``` Proxy container: `netbirdio/reverse-proxy:latest` Server container: `netbirdio/netbird-server:latest` **Is any other VPN software installed?** Yes, Tailscale is installed on the VPS host (used for management access). - Tailscale does NOT interfere with NetBird - VPS NetBird peer works correctly when enabled (can ping other peers, 2/2 connected) - Issue persists with Tailscale running, stopped, or with VPS NetBird peer disabled **Debug output** ### Proxy Container Logs (Debug Mode) **Successful Initial Connection:** ``` 2026-03-03T18:26:37.428Z DEBG [gRPC_address: netbird-server:80, TLS_enabled: false] proxy/server.go:383: starting management gRPC client 2026-03-03T18:26:37.429Z INFO [account_id: d6je6q5pk78c73abqer0, service_id: d6jeam5pk78c7396htp0, public_key: lJweckS9ZbrqgtBqGef0y0QMbMpBNUbSgkyxFNXgCEM=] proxy/internal/roundtrip/netbird.go:226: proxy peer authenticated successfully with management 2026-03-03T18:26:37.431Z INFO proxy/server.go:614: Initial mapping sync complete ``` **Failed Forwarding Request:** ``` 2026-03-03T18:37:04.770Z DEBG proxy/internal/accesslog/middleware.go:27: request: request_id=d6jik55pk78c73ctkm6g method=GET host=local-test.proxy.xxx.cloud path=/ 2026-03-03T18:37:04.770Z DEBG proxy/internal/auth/middleware.go:95: checking authentication for host: local-test.proxy.xxx.cloud, exists: true 2026-03-03T18:37:04.770Z DEBG proxy/internal/proxy/servicemapping.go:59: matched host: local-test.proxy.xxx.cloud, path: / -> http://100.66.129.92:8888/ [30 seconds of timeout...] 2026-03-03T18:37:24.943Z ERRO shared/management/client/grpc.go:82: failed creating connection to Management Service: create connection: dial context: context deadline exceeded 2026-03-03T18:37:24.943Z ERRO client/internal/auth/auth.go:51: failed connecting to Management Service http://netbird-server:80: create connection: dial context: context deadline exceeded 2026-03-03T18:37:24.943Z WARN proxy/internal/proxy/reverseproxy.go:298: proxy error: request_id=d6jik55pk78c73ctkm6g client_ip=<CLIENT_IP> method=GET host=100.66.129.92:8888 path=/ status=504 title="Request Timeout" err=client start failed: create auth client: create connection: dial context: context deadline exceeded 2026-03-03T18:37:24.943Z DEBG proxy/internal/accesslog/middleware.go:70: response: request_id=d6jik55pk78c73ctkm6g method=GET host=local-test.proxy.xxx.cloud path=/ status=504 duration=80171ms source=<CLIENT_IP> origin=proxy_error service=d6jijodpk78c73apr4hg account=d6je6q5pk78c73abqer0 ``` ### Management Server Logs **Proxy Registration (Successful):** ``` 2026-03-03T18:26:37.313Z INFO management/internals/server/boot.go:163: ProxyService registered on gRPC server 2026-03-03T18:26:37.313Z INFO management/internals/shared/grpc/proxy.go:159: New proxy connection from 172.30.0.10:58244 2026-03-03T18:26:37.313Z INFO management/internals/shared/grpc/proxy.go:188: Proxy registered in cluster [proxy_id: netbird-proxy-20260303182636, total_proxies: 1] 2026-03-03T18:26:37.317Z INFO management/internals/shared/grpc/onetime_token.go:126: Token validated and consumed for proxy ``` **Note:** No auth client connection attempts appear in management server logs - the embedded NetBird clients never reach the management service. ### Network Connectivity Tests (All Successful) ```bash # From proxy container: $ docker exec netbird-proxy ping -c 2 100.66.129.92 64 bytes from 100.66.129.92: seq=0 ttl=63 time=18.149 ms ✅ $ docker exec netbird-proxy ping -c 2 100.66.181.135 64 bytes from 100.66.181.135: seq=0 ttl=63 time=18.079 ms ✅ $ docker exec netbird-proxy wget -qO- http://172.30.0.5 <h1>Hello from VPS Local Nginx!</h1> ✅ # Proxy container CAN reach all targets via standard networking ``` **Additional context** ### Key Discovery: Two Separate Connection Types The proxy makes TWO distinct types of connections to the management service: 1. **Mapping Stream** (proxy's own gRPC connection) - Status: ✅ **Works perfectly every time** - Purpose: Get service configurations from `management.ProxyService/GetMappingUpdate` - Result: "proxy peer authenticated successfully", "Initial mapping sync complete" 2. **Auth Clients** (embedded NetBird clients for peer connections) - Status: ❌ **Always timeout after ~30 seconds** - Purpose: Authenticate and establish WireGuard tunnels to target peers - Result: "create auth client: create connection: dial context: context deadline exceeded" ### Testing Matrix | Target Type | Target Details | Network Test | Proxy Forward | Error | |-------------|----------------|--------------|---------------|-------| | Remote peer | Desktop 100.66.181.135:3000 | ✅ Ping 18ms | ❌ 504 | Auth client timeout | | Local peer | VPS 100.66.129.92:8888 | ✅ Direct access | ❌ 504 | Auth client timeout | | Docker container | nginx 172.30.0.5:80 (same network) | ✅ Direct wget | ❌ 504 | Auth client timeout | **Conclusion:** The issue is NOT network/routing related - it's a fundamental problem with embedded NetBird client authentication. ### Configuration Attempted **Management Address:** - `http://netbird-server:80` (official config per getting-started.sh) - `http://172.30.0.10:80` (Traefik internal IP) - `http://<container-ip>:80` (direct container IP) - All result in the same auth client timeout **Traefik Routing:** Tried 10+ routing configurations including: - Path-based matching: `PathPrefix(/management.) || PathPrefix(/signalexchange.)` - Host-based matching: `Host(172.30.0.10)` - Header-based matching: `HeadersRegexp(Content-Type, ^application/grpc)` - Catch-all routing - Separate HTTP/HTTPS routers - None resolved the auth client timeout **Network Mode:** - Bridge network (default) - Host network mode - Both exhibit the same issue ### Environment Details **Deployment:** - Docker Compose on Debian 12 - Traefik v3.6 (built-in configuration from getting-started.sh) - Docker network: 172.30.0.0/24 - No global HTTP→HTTPS redirect (removed to support internal gRPC on port 80) **Containers:** ```yaml proxy: image: netbirdio/reverse-proxy:latest networks: [netbird] cap_add: [NET_ADMIN, NET_RAW] env: NB_PROXY_MANAGEMENT_ADDRESS: http://netbird-server:80 NB_PROXY_ALLOW_INSECURE: true NB_PROXY_DOMAIN: proxy.xxx.cloud ``` **VPS NetBird Peer Status (when enabled):** ``` Management: Connected Signal: Connected Peers count: 2/2 Connected NetBird IP: 100.66.129.92/16 ``` ### Related Observations 1. **Protocol Mismatch**: netbird-server speaks HTTP/1.1 on port 80, requiring Traefik's h2c service to translate to HTTP/2 for gRPC clients 2. **Official Script**: The `getting-started.sh` script routing rules do NOT include `/management.ProxyService/` - only `/management.ManagementService/` and `/signalexchange.SignalExchange/` 3. **Management Server Behavior**: Successfully processes proxy's mapping stream connection but never receives auth client connection attempts in logs ### Request for Help 1. Is there additional configuration required for embedded NetBird client authentication? 2. Should embedded clients use a different management endpoint than the proxy's mapping stream? 3. Are there Docker networking requirements we're missing? **Have you tried these troubleshooting steps?** - [x] Reviewed client troubleshooting - [x] Checked for newer NetBird versions (using latest 0.66.0) - [x] Searched for similar issues on GitHub (found #5349, #2934, #1048) - [x] Restarted the NetBird client (multiple times, full docker compose down/up) - [x] Disabled other VPN software (Tailscale - issue persists) - [x] Checked firewall settings (UFW disabled, iptables rules verified, Docker networking functional) Thank you for any guidance!
saavagebueno added the triage-needed label 2026-08-05 01:29:13 -04:00
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11302