DYNR/netbird
2
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-05 00:15:26 -04:00
Code Issues 23.5k Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5492] 💥 Reverse Proxy Fatal: Embedded NetBird Clients Can't Authenticate - All Forwarding Fails #11581

New Issue
Closed
opened 2026-08-05 01:30:05 -04:00 by saavagebueno · 3 comments
saavagebueno commented 2026-08-05 01:30:05 -04:00
Owner
Copy Link

Originally created by @Nexx4 on GitHub (Mar 3, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5492

Describe the problem

The NetBird reverse proxy successfully connects to the management service and receives service mappings, but fails to forward ANY traffic due to embedded NetBird client authentication timeouts. The proxy accepts incoming HTTPS requests and correctly identifies target backends, but returns 504 Gateway Timeout after ~30 seconds because it cannot create authenticated NetBird client connections for peer tunneling.

Error: client start failed: create auth client: create connection: dial context: context deadline exceeded

To Reproduce

Steps to reproduce the behavior:

  1. Deploy self-hosted NetBird with built-in Traefik using getting-started.sh (v0.66.0)
  2. Enable reverse proxy feature during setup
  3. Create a reverse proxy service in the NetBird dashboard:
    • Choose any NetBird peer as target (local or remote)
    • Set target port (e.g., 3000, 8888)
    • Assign a subdomain (e.g., test.proxy.xxx.cloud)
  4. Access the service via browser: https://test.proxy.xxx.cloud
  5. Observe 504 Gateway Timeout after ~30 seconds

Expected behavior

The proxy should:

  1. Create an embedded NetBird client for the target peer
  2. Authenticate the client with the management service
  3. Establish a WireGuard tunnel to the peer
  4. Forward HTTP traffic through the tunnel to the backend service
  5. Return the backend response to the client

Actual behavior:

  • Proxy receives the request ✅
  • Proxy matches the service configuration ✅
  • Proxy attempts to create NetBird auth client ❌
  • Auth client connection times out after ~30s ❌
  • Returns 504 Gateway Timeout to client ❌

Are you using NetBird Cloud?

No, using self-hosted NetBird control plane (management + signal + relay in combined mode).

NetBird version

netbird version: 0.66.0

Proxy container: netbirdio/reverse-proxy:latest
Server container: netbirdio/netbird-server:latest

Is any other VPN software installed?

Yes, Tailscale is installed on the VPS host (used for management access).

  • Tailscale does NOT interfere with NetBird
  • VPS NetBird peer works correctly when enabled (can ping other peers, 2/2 connected)
  • Issue persists with Tailscale running, stopped, or with VPS NetBird peer disabled

Debug output

Proxy Container Logs (Debug Mode)

Successful Initial Connection:

2026-03-03T18:26:37.428Z DEBG [gRPC_address: netbird-server:80, TLS_enabled: false] proxy/server.go:383: starting management gRPC client
2026-03-03T18:26:37.429Z INFO [account_id: d6je6q5pk78c73abqer0, service_id: d6jeam5pk78c7396htp0, public_key: lJweckS9ZbrqgtBqGef0y0QMbMpBNUbSgkyxFNXgCEM=] proxy/internal/roundtrip/netbird.go:226: proxy peer authenticated successfully with management
2026-03-03T18:26:37.431Z INFO proxy/server.go:614: Initial mapping sync complete

Failed Forwarding Request:

2026-03-03T18:37:04.770Z DEBG proxy/internal/accesslog/middleware.go:27: request: request_id=d6jik55pk78c73ctkm6g method=GET host=local-test.proxy.xxx.cloud path=/
2026-03-03T18:37:04.770Z DEBG proxy/internal/auth/middleware.go:95: checking authentication for host: local-test.proxy.xxx.cloud, exists: true
2026-03-03T18:37:04.770Z DEBG proxy/internal/proxy/servicemapping.go:59: matched host: local-test.proxy.xxx.cloud, path: / -> http://100.66.129.92:8888/

[30 seconds of timeout...]

2026-03-03T18:37:24.943Z ERRO shared/management/client/grpc.go:82: failed creating connection to Management Service: create connection: dial context: context deadline exceeded
2026-03-03T18:37:24.943Z ERRO client/internal/auth/auth.go:51: failed connecting to Management Service http://netbird-server:80: create connection: dial context: context deadline exceeded
2026-03-03T18:37:24.943Z WARN proxy/internal/proxy/reverseproxy.go:298: proxy error: request_id=d6jik55pk78c73ctkm6g client_ip=<CLIENT_IP> method=GET host=100.66.129.92:8888 path=/ status=504 title="Request Timeout" err=client start failed: create auth client: create connection: dial context: context deadline exceeded
2026-03-03T18:37:24.943Z DEBG proxy/internal/accesslog/middleware.go:70: response: request_id=d6jik55pk78c73ctkm6g method=GET host=local-test.proxy.xxx.cloud path=/ status=504 duration=80171ms source=<CLIENT_IP> origin=proxy_error service=d6jijodpk78c73apr4hg account=d6je6q5pk78c73abqer0

Management Server Logs

Proxy Registration (Successful):

2026-03-03T18:26:37.313Z INFO management/internals/server/boot.go:163: ProxyService registered on gRPC server
2026-03-03T18:26:37.313Z INFO management/internals/shared/grpc/proxy.go:159: New proxy connection from 172.30.0.10:58244
2026-03-03T18:26:37.313Z INFO management/internals/shared/grpc/proxy.go:188: Proxy registered in cluster [proxy_id: netbird-proxy-20260303182636, total_proxies: 1]
2026-03-03T18:26:37.317Z INFO management/internals/shared/grpc/onetime_token.go:126: Token validated and consumed for proxy

Note: No auth client connection attempts appear in management server logs - the embedded NetBird clients never reach the management service.

Network Connectivity Tests (All Successful)

# From proxy container:
$ docker exec netbird-proxy ping -c 2 100.66.129.92
64 bytes from 100.66.129.92: seq=0 ttl=63 time=18.149 ms  ✅

$ docker exec netbird-proxy ping -c 2 100.66.181.135
64 bytes from 100.66.181.135: seq=0 ttl=63 time=18.079 ms  ✅

$ docker exec netbird-proxy wget -qO- http://172.30.0.5
<h1>Hello from VPS Local Nginx!</h1>  ✅

# Proxy container CAN reach all targets via standard networking

Additional context

Key Discovery: Two Separate Connection Types

The proxy makes TWO distinct types of connections to the management service:

  1. Mapping Stream (proxy's own gRPC connection)

    • Status: ✅ Works perfectly every time
    • Purpose: Get service configurations from management.ProxyService/GetMappingUpdate
    • Result: "proxy peer authenticated successfully", "Initial mapping sync complete"

  2. Auth Clients (embedded NetBird clients for peer connections)

    • Status: ❌ Always timeout after ~30 seconds
    • Purpose: Authenticate and establish WireGuard tunnels to target peers
    • Result: "create auth client: create connection: dial context: context deadline exceeded"

Testing Matrix

Target Type Target Details Network Test Proxy Forward Error
Remote peer Desktop 100.66.181.135:3000 ✅ Ping 18ms ❌ 504 Auth client timeout
Local peer VPS 100.66.129.92:8888 ✅ Direct access ❌ 504 Auth client timeout
Docker container nginx 172.30.0.5:80 (same network) ✅ Direct wget ❌ 504 Auth client timeout

Conclusion: The issue is NOT network/routing related - it's a fundamental problem with embedded NetBird client authentication.

Configuration Attempted

Management Address:

  • http://netbird-server:80 (official config per getting-started.sh)
  • http://172.30.0.10:80 (Traefik internal IP)
  • http://<container-ip>:80 (direct container IP)
  • All result in the same auth client timeout

Traefik Routing:
Tried 10+ routing configurations including:

  • Path-based matching: PathPrefix(/management.) || PathPrefix(/signalexchange.)
  • Host-based matching: Host(172.30.0.10)
  • Header-based matching: HeadersRegexp(Content-Type, ^application/grpc)
  • Catch-all routing
  • Separate HTTP/HTTPS routers
  • None resolved the auth client timeout

Network Mode:

  • Bridge network (default)
  • Host network mode
  • Both exhibit the same issue

Environment Details

Deployment:

  • Docker Compose on Debian 12
  • Traefik v3.6 (built-in configuration from getting-started.sh)
  • Docker network: 172.30.0.0/24
  • No global HTTP→HTTPS redirect (removed to support internal gRPC on port 80)

Containers:

proxy:
  image: netbirdio/reverse-proxy:latest
  networks: [netbird]
  cap_add: [NET_ADMIN, NET_RAW]
  env:
    NB_PROXY_MANAGEMENT_ADDRESS: http://netbird-server:80
    NB_PROXY_ALLOW_INSECURE: true
    NB_PROXY_DOMAIN: proxy.xxx.cloud

VPS NetBird Peer Status (when enabled):

Management: Connected
Signal: Connected
Peers count: 2/2 Connected
NetBird IP: 100.66.129.92/16

Related Observations

  1. Protocol Mismatch: netbird-server speaks HTTP/1.1 on port 80, requiring Traefik's h2c service to translate to HTTP/2 for gRPC clients

  2. Official Script: The getting-started.sh script routing rules do NOT include /management.ProxyService/ - only /management.ManagementService/ and /signalexchange.SignalExchange/

  3. Management Server Behavior: Successfully processes proxy's mapping stream connection but never receives auth client connection attempts in logs

Request for Help

  1. Is there additional configuration required for embedded NetBird client authentication?
  2. Should embedded clients use a different management endpoint than the proxy's mapping stream?
  3. Are there Docker networking requirements we're missing?

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting
  • Checked for newer NetBird versions (using latest 0.66.0)
  • Searched for similar issues on GitHub (found #5349, #2934, #1048)
  • Restarted the NetBird client (multiple times, full docker compose down/up)
  • Disabled other VPN software (Tailscale - issue persists)
  • Checked firewall settings (UFW disabled, iptables rules verified, Docker networking functional)

Thank you for any guidance!

Originally created by @Nexx4 on GitHub (Mar 3, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5492 **Describe the problem** The NetBird reverse proxy successfully connects to the management service and receives service mappings, but fails to forward ANY traffic due to embedded NetBird client authentication timeouts. The proxy accepts incoming HTTPS requests and correctly identifies target backends, but returns 504 Gateway Timeout after ~30 seconds because it cannot create authenticated NetBird client connections for peer tunneling. Error: `client start failed: create auth client: create connection: dial context: context deadline exceeded` **To Reproduce** Steps to reproduce the behavior: 1. Deploy self-hosted NetBird with built-in Traefik using `getting-started.sh` (v0.66.0) 2. Enable reverse proxy feature during setup 3. Create a reverse proxy service in the NetBird dashboard: - Choose any NetBird peer as target (local or remote) - Set target port (e.g., 3000, 8888) - Assign a subdomain (e.g., `test.proxy.xxx.cloud`) 4. Access the service via browser: `https://test.proxy.xxx.cloud` 5. Observe 504 Gateway Timeout after ~30 seconds **Expected behavior** The proxy should: 1. Create an embedded NetBird client for the target peer 2. Authenticate the client with the management service 3. Establish a WireGuard tunnel to the peer 4. Forward HTTP traffic through the tunnel to the backend service 5. Return the backend response to the client **Actual behavior:** - Proxy receives the request ✅ - Proxy matches the service configuration ✅ - Proxy attempts to create NetBird auth client ❌ - Auth client connection times out after ~30s ❌ - Returns 504 Gateway Timeout to client ❌ **Are you using NetBird Cloud?** No, using **self-hosted** NetBird control plane (management + signal + relay in combined mode). **NetBird version** ``` netbird version: 0.66.0 ``` Proxy container: `netbirdio/reverse-proxy:latest` Server container: `netbirdio/netbird-server:latest` **Is any other VPN software installed?** Yes, Tailscale is installed on the VPS host (used for management access). - Tailscale does NOT interfere with NetBird - VPS NetBird peer works correctly when enabled (can ping other peers, 2/2 connected) - Issue persists with Tailscale running, stopped, or with VPS NetBird peer disabled **Debug output** ### Proxy Container Logs (Debug Mode) **Successful Initial Connection:** ``` 2026-03-03T18:26:37.428Z DEBG [gRPC_address: netbird-server:80, TLS_enabled: false] proxy/server.go:383: starting management gRPC client 2026-03-03T18:26:37.429Z INFO [account_id: d6je6q5pk78c73abqer0, service_id: d6jeam5pk78c7396htp0, public_key: lJweckS9ZbrqgtBqGef0y0QMbMpBNUbSgkyxFNXgCEM=] proxy/internal/roundtrip/netbird.go:226: proxy peer authenticated successfully with management 2026-03-03T18:26:37.431Z INFO proxy/server.go:614: Initial mapping sync complete ``` **Failed Forwarding Request:** ``` 2026-03-03T18:37:04.770Z DEBG proxy/internal/accesslog/middleware.go:27: request: request_id=d6jik55pk78c73ctkm6g method=GET host=local-test.proxy.xxx.cloud path=/ 2026-03-03T18:37:04.770Z DEBG proxy/internal/auth/middleware.go:95: checking authentication for host: local-test.proxy.xxx.cloud, exists: true 2026-03-03T18:37:04.770Z DEBG proxy/internal/proxy/servicemapping.go:59: matched host: local-test.proxy.xxx.cloud, path: / -> http://100.66.129.92:8888/ [30 seconds of timeout...] 2026-03-03T18:37:24.943Z ERRO shared/management/client/grpc.go:82: failed creating connection to Management Service: create connection: dial context: context deadline exceeded 2026-03-03T18:37:24.943Z ERRO client/internal/auth/auth.go:51: failed connecting to Management Service http://netbird-server:80: create connection: dial context: context deadline exceeded 2026-03-03T18:37:24.943Z WARN proxy/internal/proxy/reverseproxy.go:298: proxy error: request_id=d6jik55pk78c73ctkm6g client_ip=<CLIENT_IP> method=GET host=100.66.129.92:8888 path=/ status=504 title="Request Timeout" err=client start failed: create auth client: create connection: dial context: context deadline exceeded 2026-03-03T18:37:24.943Z DEBG proxy/internal/accesslog/middleware.go:70: response: request_id=d6jik55pk78c73ctkm6g method=GET host=local-test.proxy.xxx.cloud path=/ status=504 duration=80171ms source=<CLIENT_IP> origin=proxy_error service=d6jijodpk78c73apr4hg account=d6je6q5pk78c73abqer0 ``` ### Management Server Logs **Proxy Registration (Successful):** ``` 2026-03-03T18:26:37.313Z INFO management/internals/server/boot.go:163: ProxyService registered on gRPC server 2026-03-03T18:26:37.313Z INFO management/internals/shared/grpc/proxy.go:159: New proxy connection from 172.30.0.10:58244 2026-03-03T18:26:37.313Z INFO management/internals/shared/grpc/proxy.go:188: Proxy registered in cluster [proxy_id: netbird-proxy-20260303182636, total_proxies: 1] 2026-03-03T18:26:37.317Z INFO management/internals/shared/grpc/onetime_token.go:126: Token validated and consumed for proxy ``` **Note:** No auth client connection attempts appear in management server logs - the embedded NetBird clients never reach the management service. ### Network Connectivity Tests (All Successful) ```bash # From proxy container: $ docker exec netbird-proxy ping -c 2 100.66.129.92 64 bytes from 100.66.129.92: seq=0 ttl=63 time=18.149 ms ✅ $ docker exec netbird-proxy ping -c 2 100.66.181.135 64 bytes from 100.66.181.135: seq=0 ttl=63 time=18.079 ms ✅ $ docker exec netbird-proxy wget -qO- http://172.30.0.5 <h1>Hello from VPS Local Nginx!</h1> ✅ # Proxy container CAN reach all targets via standard networking ``` **Additional context** ### Key Discovery: Two Separate Connection Types The proxy makes TWO distinct types of connections to the management service: 1. **Mapping Stream** (proxy's own gRPC connection) - Status: ✅ **Works perfectly every time** - Purpose: Get service configurations from `management.ProxyService/GetMappingUpdate` - Result: "proxy peer authenticated successfully", "Initial mapping sync complete" 2. **Auth Clients** (embedded NetBird clients for peer connections) - Status: ❌ **Always timeout after ~30 seconds** - Purpose: Authenticate and establish WireGuard tunnels to target peers - Result: "create auth client: create connection: dial context: context deadline exceeded" ### Testing Matrix | Target Type | Target Details | Network Test | Proxy Forward | Error | |-------------|----------------|--------------|---------------|-------| | Remote peer | Desktop 100.66.181.135:3000 | ✅ Ping 18ms | ❌ 504 | Auth client timeout | | Local peer | VPS 100.66.129.92:8888 | ✅ Direct access | ❌ 504 | Auth client timeout | | Docker container | nginx 172.30.0.5:80 (same network) | ✅ Direct wget | ❌ 504 | Auth client timeout | **Conclusion:** The issue is NOT network/routing related - it's a fundamental problem with embedded NetBird client authentication. ### Configuration Attempted **Management Address:** - `http://netbird-server:80` (official config per getting-started.sh) - `http://172.30.0.10:80` (Traefik internal IP) - `http://<container-ip>:80` (direct container IP) - All result in the same auth client timeout **Traefik Routing:** Tried 10+ routing configurations including: - Path-based matching: `PathPrefix(/management.) || PathPrefix(/signalexchange.)` - Host-based matching: `Host(172.30.0.10)` - Header-based matching: `HeadersRegexp(Content-Type, ^application/grpc)` - Catch-all routing - Separate HTTP/HTTPS routers - None resolved the auth client timeout **Network Mode:** - Bridge network (default) - Host network mode - Both exhibit the same issue ### Environment Details **Deployment:** - Docker Compose on Debian 12 - Traefik v3.6 (built-in configuration from getting-started.sh) - Docker network: 172.30.0.0/24 - No global HTTP→HTTPS redirect (removed to support internal gRPC on port 80) **Containers:** ```yaml proxy: image: netbirdio/reverse-proxy:latest networks: [netbird] cap_add: [NET_ADMIN, NET_RAW] env: NB_PROXY_MANAGEMENT_ADDRESS: http://netbird-server:80 NB_PROXY_ALLOW_INSECURE: true NB_PROXY_DOMAIN: proxy.xxx.cloud ``` **VPS NetBird Peer Status (when enabled):** ``` Management: Connected Signal: Connected Peers count: 2/2 Connected NetBird IP: 100.66.129.92/16 ``` ### Related Observations 1. **Protocol Mismatch**: netbird-server speaks HTTP/1.1 on port 80, requiring Traefik's h2c service to translate to HTTP/2 for gRPC clients 2. **Official Script**: The `getting-started.sh` script routing rules do NOT include `/management.ProxyService/` - only `/management.ManagementService/` and `/signalexchange.SignalExchange/` 3. **Management Server Behavior**: Successfully processes proxy's mapping stream connection but never receives auth client connection attempts in logs ### Request for Help 1. Is there additional configuration required for embedded NetBird client authentication? 2. Should embedded clients use a different management endpoint than the proxy's mapping stream? 3. Are there Docker networking requirements we're missing? **Have you tried these troubleshooting steps?** - [x] Reviewed client troubleshooting - [x] Checked for newer NetBird versions (using latest 0.66.0) - [x] Searched for similar issues on GitHub (found #5349, #2934, #1048) - [x] Restarted the NetBird client (multiple times, full docker compose down/up) - [x] Disabled other VPN software (Tailscale - issue persists) - [x] Checked firewall settings (UFW disabled, iptables rules verified, Docker networking functional) Thank you for any guidance!
saavagebueno added the triage-needed label 2026-08-05 01:30:05 -04:00
saavagebueno commented 2026-08-05 01:30:06 -04:00
Author
Owner
Copy Link

@zgv163 commented on GitHub (Mar 4, 2026):

I ran into the same issue deploying the reverse proxy behind Traefik on a self-hosted setup and tracked it down.

The problem: The proxy makes two separate types of connections to management:

  1. Mapping stream: uses NB_PROXY_MANAGEMENT_ADDRESS (your internal http://netbird-server:80) → this works fine
  2. Embedded NetBird clients: these are full NetBird clients that resolve and connect to the public management URL (e.g. https://xxx.cloud:443) independently → these timeout

From inside the Docker network, your public domain either doesn't resolve to the host or there's no hairpin NAT path back to Traefik. That's why management server logs show zero auth client connection attempts, the traffic never arrives.

Fix: add extra_hosts to the proxy service so the management domain resolves to somewhere reachable from inside Docker:

proxy:
  image: netbirdio/reverse-proxy:latest
  extra_hosts:
    - "your-management-domain.cloud:host-gateway"
  # ... rest of your config

host-gateway resolves to the Docker host IP, which has Traefik listening on port 443, keeping traffic local and avoiding the hairpin NAT problem entirely.

If your Traefik is on an overlay network (not bridge), use its container IP instead:

docker inspect <traefik-container> \
  --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}}: {{$v.IPAddress}}{{"\n"}}{{end}}'

Also worth checking: The getting-started.sh Traefik routing rules only include /management.ManagementService/ and /signalexchange.SignalExchange/. Make sure these gRPC paths are routed to the server with HTTP/2 cleartext (h2c) backend. Without the h2c scheme on the Traefik service, gRPC connections will fail silently or timeout.

<!-- gh-comment-id:3996677040 --> @zgv163 commented on GitHub (Mar 4, 2026): I ran into the same issue deploying the reverse proxy behind Traefik on a self-hosted setup and tracked it down. **The problem:** The proxy makes two separate types of connections to management: 1. **Mapping stream:** uses `NB_PROXY_MANAGEMENT_ADDRESS` (your internal `http://netbird-server:80`) → this works fine 2. **Embedded NetBird clients:** these are full NetBird clients that resolve and connect to the **public management URL** (e.g. `https://xxx.cloud:443`) independently → these timeout From inside the Docker network, your public domain either doesn't resolve to the host or there's no hairpin NAT path back to Traefik. That's why management server logs show zero auth client connection attempts, the traffic never arrives. **Fix: add `extra_hosts` to the proxy service** so the management domain resolves to somewhere reachable from inside Docker: ```yaml proxy: image: netbirdio/reverse-proxy:latest extra_hosts: - "your-management-domain.cloud:host-gateway" # ... rest of your config ``` `host-gateway` resolves to the Docker host IP, which has Traefik listening on port 443, keeping traffic local and avoiding the hairpin NAT problem entirely. If your Traefik is on an overlay network (not bridge), use its container IP instead: ```bash docker inspect <traefik-container> \ --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}}: {{$v.IPAddress}}{{"\n"}}{{end}}' ``` **Also worth checking:** The `getting-started.sh` Traefik routing rules only include `/management.ManagementService/` and `/signalexchange.SignalExchange/`. Make sure these gRPC paths are routed to the server with HTTP/2 cleartext (h2c) backend. Without the h2c scheme on the Traefik service, gRPC connections will fail silently or timeout.
saavagebueno commented 2026-08-05 01:30:06 -04:00
Author
Owner
Copy Link

@Nexx4 commented on GitHub (Mar 5, 2026):

Update: extra_hosts approach tested extensively - still fails

Thanks @zgv163 for the suggestion. I've spent significant time testing this approach with multiple networking configurations. Unfortunately, the embedded client timeout persists in all cases.

Environment

  • NetBird Server: netbirdio/netbird-server:latest
  • Reverse Proxy: netbirdio/reverse-proxy:latest (pulled fresh 2026-03-05)
  • Traefik: v3.6 (TLS termination + h2c backend to netbird-server)
  • Docker network: bridge, 172.30.0.0/24, Traefik at static 172.30.0.10
  • Host OS: Debian 12 on OVH VPS

Configuration

proxy.env:

NB_PROXY_DEBUG_LOGS=true
NB_PROXY_MANAGEMENT_ADDRESS=https://netbird.xyz.cloud
NB_PROXY_DOMAIN=proxy.xyz.cloud
NB_PROXY_ADDRESS=:8443
NB_PROXY_TOKEN=nbx_xxxxx
NB_PROXY_CERTIFICATE_DIRECTORY=/certs
NB_PROXY_ACME_CERTIFICATES=true
NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01
NB_PROXY_FORWARDED_PROTO=https
NB_PROXY_PROXY_PROTOCOL=true
NB_PROXY_TRUSTED_PROXIES=172.30.0.10
NB_MANAGEMENT_URL=https://netbird.xyz.cloud
NB_LOG_LEVEL=debug
NB_ADMIN_URL=https://netbird.xyz.cloud

docker-compose.yml (proxy section):

proxy:
  image: netbirdio/reverse-proxy:latest
  container_name: netbird-proxy
  restart: unless-stopped
  networks: [netbird]
  user: "0:0"
  cap_add:
    - NET_ADMIN
    - NET_RAW
  depends_on:
    - netbird-server
  env_file:
    - ./proxy.env
  extra_hosts:
    - "netbird.xyz.cloud:172.30.0.10"  # Traefik static IP
    - "host.docker.internal:host-gateway"
  volumes:
    - netbird_proxy_state:/var/lib/netbird
    - netbird_proxy_certs:/certs

config.yaml (server):

server:
  listenAddress: ":80"
  exposedAddress: "https://netbird.xyz.cloud:443"

Traefik gRPC routing:

# TLS termination + h2c backend
- traefik.http.routers.netbird-grpc-external.rule=Host(`netbird.xyz.cloud`) && (PathPrefix(`/management.`) || PathPrefix(`/signalexchange.`))
- traefik.http.routers.netbird-grpc-external.entrypoints=websecure
- traefik.http.routers.netbird-grpc-external.tls=true
- traefik.http.routers.netbird-grpc-external.service=netbird-server-h2c
- traefik.http.services.netbird-server-h2c.loadbalancer.server.port=80
- traefik.http.services.netbird-server-h2c.loadbalancer.server.scheme=h2c

What was tested

# Approach extra_hosts / networking Result
1 Traefik static IP netbird.xyz.cloud:172.30.0.10 30s timeout
2 netbird-server direct IP netbird.xyz.cloud:172.30.0.3 30s timeout
3 host-gateway netbird.xyz.cloud:host-gateway (-> 172.17.0.1) 30s timeout
4 Host networking network_mode: host (host's /etc/hosts -> 127.0.0.1) 30s timeout
5 Host net + no NetBird network_mode: host + stopped host's NetBird client 30s timeout
6 Fresh image docker pull latest image + recreate 30s timeout
7 HTTP internal NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80 30s timeout

What DOES work (same address, same container)

  1. Proxy mapping stream connects in ~45ms:

    DEBG [gRPC_address: netbird.xyz.cloud, TLS_enabled: true] starting management gRPC client
DEBG management mapping stream established

  2. Proxy peer authentication succeeds in ~110ms:

    INFO proxy peer authenticated successfully with management
INFO created new client for account
INFO Initial mapping sync complete

  3. HTTP/HTTPS connectivity from inside the container works:

    # wget to Traefik via extra_hosts -> returns dashboard HTML instantly
docker exec netbird-proxy wget --no-check-certificate -qO- https://netbird.xyz.cloud/

# curl HTTP/2 from host to Traefik internal IP -> works
curl --http2 -k --resolve netbird.xyz.cloud:443:172.30.0.10 https://netbird.xyz.cloud/management.ManagementService/GetServerKey
# Returns 404 (expected, wrong method for gRPC) but connection succeeds

  4. Traefik access logs show ProxyService gRPC calls arriving fine from the proxy container:

    172.30.0.1 - "POST /management.ProxyService/SendAccessLog HTTP/2.0" 200 - "netbird-grpc-external@docker" "h2c://172.30.0.3:80" 4ms

The actual failure

Exactly 30 seconds after "created new client for account", the embedded client times out:

08:52:20.602 INFO  proxy peer authenticated successfully with management
08:52:20.605 INFO  created new client for account
08:52:20.606 INFO  Initial mapping sync complete
08:52:50.606 ERRO  failed creating connection to Management Service: create connection: dial context: context deadline exceeded
08:52:50.606 ERRO  failed connecting to Management Service https://netbird.xyz.cloud:443: create connection: dial context: context deadline exceeded
08:52:50.606 WARN  netbird client start timed out, will retry on first request

The embedded client's grpc.DialContext in shared/management/client/grpc.go:82 hangs for 30 seconds on the exact same address that the proxy's own gRPC mapping stream connects to in <100ms. No connection attempts appear in Traefik's access logs from the embedded client during those 30 seconds.

Conclusion

This does not appear to be a DNS, Docker networking, or hairpin NAT issue. The proxy's own gRPC client and the embedded NetBird client use the same management address, but only the embedded client fails. The embedded client's TCP dial never reaches Traefik at all (no access log entries), suggesting the connection attempt is blocked or deadlocked at the application level before any TCP SYN is sent.

<!-- gh-comment-id:4003470271 --> @Nexx4 commented on GitHub (Mar 5, 2026): **Update: `extra_hosts` approach tested extensively - still fails** Thanks @zgv163 for the suggestion. I've spent significant time testing this approach with multiple networking configurations. Unfortunately, the embedded client timeout persists in all cases. ## Environment - **NetBird Server**: `netbirdio/netbird-server:latest` - **Reverse Proxy**: `netbirdio/reverse-proxy:latest` (pulled fresh 2026-03-05) - **Traefik**: v3.6 (TLS termination + h2c backend to netbird-server) - **Docker network**: bridge, `172.30.0.0/24`, Traefik at static `172.30.0.10` - **Host OS**: Debian 12 on OVH VPS ## Configuration **proxy.env:** ```env NB_PROXY_DEBUG_LOGS=true NB_PROXY_MANAGEMENT_ADDRESS=https://netbird.xyz.cloud NB_PROXY_DOMAIN=proxy.xyz.cloud NB_PROXY_ADDRESS=:8443 NB_PROXY_TOKEN=nbx_xxxxx NB_PROXY_CERTIFICATE_DIRECTORY=/certs NB_PROXY_ACME_CERTIFICATES=true NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01 NB_PROXY_FORWARDED_PROTO=https NB_PROXY_PROXY_PROTOCOL=true NB_PROXY_TRUSTED_PROXIES=172.30.0.10 NB_MANAGEMENT_URL=https://netbird.xyz.cloud NB_LOG_LEVEL=debug NB_ADMIN_URL=https://netbird.xyz.cloud ``` **docker-compose.yml (proxy section):** ```yaml proxy: image: netbirdio/reverse-proxy:latest container_name: netbird-proxy restart: unless-stopped networks: [netbird] user: "0:0" cap_add: - NET_ADMIN - NET_RAW depends_on: - netbird-server env_file: - ./proxy.env extra_hosts: - "netbird.xyz.cloud:172.30.0.10" # Traefik static IP - "host.docker.internal:host-gateway" volumes: - netbird_proxy_state:/var/lib/netbird - netbird_proxy_certs:/certs ``` **config.yaml (server):** ```yaml server: listenAddress: ":80" exposedAddress: "https://netbird.xyz.cloud:443" ``` **Traefik gRPC routing:** ```yaml # TLS termination + h2c backend - traefik.http.routers.netbird-grpc-external.rule=Host(`netbird.xyz.cloud`) && (PathPrefix(`/management.`) || PathPrefix(`/signalexchange.`)) - traefik.http.routers.netbird-grpc-external.entrypoints=websecure - traefik.http.routers.netbird-grpc-external.tls=true - traefik.http.routers.netbird-grpc-external.service=netbird-server-h2c - traefik.http.services.netbird-server-h2c.loadbalancer.server.port=80 - traefik.http.services.netbird-server-h2c.loadbalancer.server.scheme=h2c ``` ## What was tested | # | Approach | `extra_hosts` / networking | Result | |---|---|---|---| | 1 | Traefik static IP | `netbird.xyz.cloud:172.30.0.10` | 30s timeout | | 2 | netbird-server direct IP | `netbird.xyz.cloud:172.30.0.3` | 30s timeout | | 3 | host-gateway | `netbird.xyz.cloud:host-gateway` (-> 172.17.0.1) | 30s timeout | | 4 | Host networking | `network_mode: host` (host's `/etc/hosts` -> 127.0.0.1) | 30s timeout | | 5 | Host net + no NetBird | `network_mode: host` + stopped host's NetBird client | 30s timeout | | 6 | Fresh image | `docker pull` latest image + recreate | 30s timeout | | 7 | HTTP internal | `NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80` | 30s timeout | ## What DOES work (same address, same container) 1. **Proxy mapping stream** connects in ~45ms: ``` DEBG [gRPC_address: netbird.xyz.cloud, TLS_enabled: true] starting management gRPC client DEBG management mapping stream established ``` 2. **Proxy peer authentication** succeeds in ~110ms: ``` INFO proxy peer authenticated successfully with management INFO created new client for account INFO Initial mapping sync complete ``` 3. **HTTP/HTTPS connectivity** from inside the container works: ```bash # wget to Traefik via extra_hosts -> returns dashboard HTML instantly docker exec netbird-proxy wget --no-check-certificate -qO- https://netbird.xyz.cloud/ # curl HTTP/2 from host to Traefik internal IP -> works curl --http2 -k --resolve netbird.xyz.cloud:443:172.30.0.10 https://netbird.xyz.cloud/management.ManagementService/GetServerKey # Returns 404 (expected, wrong method for gRPC) but connection succeeds ``` 4. **Traefik access logs** show ProxyService gRPC calls arriving fine from the proxy container: ``` 172.30.0.1 - "POST /management.ProxyService/SendAccessLog HTTP/2.0" 200 - "netbird-grpc-external@docker" "h2c://172.30.0.3:80" 4ms ``` ## The actual failure Exactly 30 seconds after "created new client for account", the embedded client times out: ``` 08:52:20.602 INFO proxy peer authenticated successfully with management 08:52:20.605 INFO created new client for account 08:52:20.606 INFO Initial mapping sync complete 08:52:50.606 ERRO failed creating connection to Management Service: create connection: dial context: context deadline exceeded 08:52:50.606 ERRO failed connecting to Management Service https://netbird.xyz.cloud:443: create connection: dial context: context deadline exceeded 08:52:50.606 WARN netbird client start timed out, will retry on first request ``` The embedded client's `grpc.DialContext` in `shared/management/client/grpc.go:82` hangs for 30 seconds on the **exact same address** that the proxy's own gRPC mapping stream connects to in <100ms. No connection attempts appear in Traefik's access logs from the embedded client during those 30 seconds. ## Conclusion This does not appear to be a DNS, Docker networking, or hairpin NAT issue. The proxy's own gRPC client and the embedded NetBird client use the same management address, but only the embedded client fails. The embedded client's TCP dial never reaches Traefik at all (no access log entries), suggesting the connection attempt is blocked or deadlocked at the application level before any TCP SYN is sent.
saavagebueno commented 2026-08-05 01:30:06 -04:00
Author
Owner
Copy Link

@Nexx4 commented on GitHub (Mar 5, 2026):

Update: Root cause found and fix confirmed - proxy is now fully working!

After extensive source code analysis of the NetBird codebase, I tracked down the exact cause of the 30-second timeout.

Root Cause

The issue turned out to be self-inflicted: during an earlier debugging session I had added user: "0:0" to the proxy service in docker-compose.yml, overriding the image's default user (uid=1000(netbird)) to run as root. This triggered a completely different code path in the gRPC dialer that causes the embedded client to hang.

Here's what happens in client/grpc/dialer_generic.go:

func WithCustomDialer(_ bool, _ string) grpc.DialOption {
    return grpc.WithContextDialer(func(ctx context.Context, addr string) (net.Conn, error) {
        if runtime.GOOS == "linux" {
            currentUser, err := user.Current()
            // ...
            if currentUser.Uid != "0" {
                log.Debug("Not running as root, using standard dialer")
                dialer := &net.Dialer{}
                return dialer.DialContext(ctx, "tcp", addr)
            }
        }
        // Root path: uses custom dialer with fwmark Control callback
        conn, err := nbnet.NewDialer().DialContext(ctx, "tcp", addr)
        return conn, nil
    })
}
  • Non-root (uid=1000, image default): Uses a plain net.Dialer{} - connects instantly.
  • Root (uid=0): Uses nbnet.NewDialer(), which attaches a setRawSocketMark() Control callback to the socket (via client/net/dialer_init_linux.go). Even though this callback is effectively a no-op in the proxy context (because AdvancedRouting() returns false - the embedded client enables netstack mode in embed.go:127-134 and nbnet.Init() is never called in the embed path), the presence of the syscall.RawConn.Control() callback somehow causes grpc.DialContext with grpc.WithBlock() to hang for the full 30-second timeout.

This explains why the proxy's own mapping stream worked fine - it uses grpc.NewClient() (non-blocking, no custom dialer), while the embedded client uses the deprecated grpc.DialContext() + grpc.WithBlock() path that goes through WithCustomDialer().

The Fix

Removed user: "0:0" from docker-compose.yml so the container runs as the image's default user (uid=1000(netbird)), which takes the plain net.Dialer{} code path.

Since the volumes were previously created as root, I also had to fix ownership:

sudo chown -R 1000:1000 /var/lib/docker/volumes/<project>_netbird_proxy_certs/_data
sudo chown -R 1000:1000 /var/lib/docker/volumes/<project>_netbird_proxy_state/_data

Result

The embedded client now connects instantly instead of timing out:

09:48:25.596 INFO  [Channel #1] Channel Connectivity change to READY
09:48:25.711 INFO  proxy peer authenticated successfully with management
09:48:25.713 INFO  created new client for account
09:48:26.112 INFO  notified management about tunnel connection  (test2.proxy)
09:48:26.118 INFO  notified management about tunnel connection  (local-test.proxy)
09:48:26.132 INFO  notified management about tunnel connection  (ngdrr.proxy)
09:48:26.136 INFO  notified management about tunnel connection  (test.proxy)

All proxied services return HTTP 200 through the WireGuard tunnel:

Domain Target Status Latency
local-test.proxy.xyz.cloud 100.66.x.x:8888 (self) 200 16ms
test.proxy.xyz.cloud 100.66.x.x:3088 (remote peer) 200 48ms
ngdrr.proxy.xyz.cloud 100.66.x.x:3000 (remote peer) 200 168ms

Takeaway

The standard installation is not affected by this. The image defaults to uid=1000 which uses the correct dialer. You would only hit this issue if you install/run the NetBird server directly on the host as root without the Docker image or explicitly override the container user to root (e.g. user: "0:0" in docker-compose). In either case, the process runs as uid=0, which triggers the nbnet.NewDialer() code path with the fwmark socket Control callback that causes the hang.

That said, it might still be worth noting in client/grpc/dialer_generic.go that the root code path can cause issues for the embedded client in the reverse proxy context, since nbnet.Init() is never called from the client/embed startup path and the fwmark Control callback - while technically a no-op - still interferes with the blocking gRPC dial. A potential fix could be to also check for netstack mode (which the embedded client always enables) and use the plain dialer in that case, regardless of uid.

<!-- gh-comment-id:4003874654 --> @Nexx4 commented on GitHub (Mar 5, 2026): **Update: Root cause found and fix confirmed - proxy is now fully working!** After extensive source code analysis of the NetBird codebase, I tracked down the exact cause of the 30-second timeout. ## Root Cause The issue turned out to be self-inflicted: during an earlier debugging session I had added `user: "0:0"` to the proxy service in docker-compose.yml, overriding the image's default user (`uid=1000(netbird)`) to run as root. This triggered a completely different code path in the gRPC dialer that causes the embedded client to hang. Here's what happens in `client/grpc/dialer_generic.go`: ```go func WithCustomDialer(_ bool, _ string) grpc.DialOption { return grpc.WithContextDialer(func(ctx context.Context, addr string) (net.Conn, error) { if runtime.GOOS == "linux" { currentUser, err := user.Current() // ... if currentUser.Uid != "0" { log.Debug("Not running as root, using standard dialer") dialer := &net.Dialer{} return dialer.DialContext(ctx, "tcp", addr) } } // Root path: uses custom dialer with fwmark Control callback conn, err := nbnet.NewDialer().DialContext(ctx, "tcp", addr) return conn, nil }) } ``` - **Non-root (uid=1000, image default)**: Uses a plain `net.Dialer{}` - connects instantly. - **Root (uid=0)**: Uses `nbnet.NewDialer()`, which attaches a `setRawSocketMark()` Control callback to the socket (via `client/net/dialer_init_linux.go`). Even though this callback is effectively a no-op in the proxy context (because `AdvancedRouting()` returns `false` - the embedded client enables netstack mode in `embed.go:127-134` and `nbnet.Init()` is never called in the embed path), the presence of the `syscall.RawConn.Control()` callback somehow causes `grpc.DialContext` with `grpc.WithBlock()` to hang for the full 30-second timeout. This explains why the proxy's **own** mapping stream worked fine - it uses `grpc.NewClient()` (non-blocking, no custom dialer), while the embedded client uses the deprecated `grpc.DialContext()` + `grpc.WithBlock()` path that goes through `WithCustomDialer()`. ## The Fix Removed `user: "0:0"` from docker-compose.yml so the container runs as the image's default user (`uid=1000(netbird)`), which takes the plain `net.Dialer{}` code path. Since the volumes were previously created as root, I also had to fix ownership: ```bash sudo chown -R 1000:1000 /var/lib/docker/volumes/<project>_netbird_proxy_certs/_data sudo chown -R 1000:1000 /var/lib/docker/volumes/<project>_netbird_proxy_state/_data ``` ## Result The embedded client now connects instantly instead of timing out: ``` 09:48:25.596 INFO [Channel #1] Channel Connectivity change to READY 09:48:25.711 INFO proxy peer authenticated successfully with management 09:48:25.713 INFO created new client for account 09:48:26.112 INFO notified management about tunnel connection (test2.proxy) 09:48:26.118 INFO notified management about tunnel connection (local-test.proxy) 09:48:26.132 INFO notified management about tunnel connection (ngdrr.proxy) 09:48:26.136 INFO notified management about tunnel connection (test.proxy) ``` All proxied services return HTTP 200 through the WireGuard tunnel: | Domain | Target | Status | Latency | |--------|--------|--------|---------| | local-test.proxy.xyz.cloud | 100.66.x.x:8888 (self) | 200 | 16ms | | test.proxy.xyz.cloud | 100.66.x.x:3088 (remote peer) | 200 | 48ms | | ngdrr.proxy.xyz.cloud | 100.66.x.x:3000 (remote peer) | 200 | 168ms | ## Takeaway **The standard installation is not affected by this.** The image defaults to `uid=1000` which uses the correct dialer. You would only hit this issue if you install/run the NetBird server directly on the host as root without the Docker image or explicitly override the container user to root (e.g. `user: "0:0"` in docker-compose). In either case, the process runs as uid=0, which triggers the `nbnet.NewDialer()` code path with the fwmark socket Control callback that causes the hang. That said, it might still be worth noting in `client/grpc/dialer_generic.go` that the root code path can cause issues for the embedded client in the reverse proxy context, since `nbnet.Init()` is never called from the `client/embed` startup path and the fwmark Control callback - while technically a no-op - still interferes with the blocking gRPC dial. A potential fix could be to also check for netstack mode (which the embedded client always enables) and use the plain dialer in that case, regardless of uid.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2021 Q4
2021 Q4
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
accessibility
accessibility
accessibility
acl
acl
acl
agent
agent
agent
agent
agent
agent
Android
Android
Android
Android
Android
Android
api
api
api
authentik
authentik
authentik
automation
automation
automation
azure
azure
azure
battery-usage
battery-usage
battery-usage
bug
cache
cache
cache
client
client
client
client-ui
client-ui
client-ui
cloud
cloud
cloud
cloud-only
cloud-only
cloud-only
cloudflare
cloudflare
cloudflare
community
community
community
compatibility
compatibility
compatibility
config-idp
config-idp
config-idp
config-issue
config-issue
config-issue
connection
connection
connection
contribution
contribution
contribution
coturn
coturn
coturn
cross-vpn
cross-vpn
cross-vpn
dashboard
dashboard
dashboard
data-usage
data-usage
data-usage
distribution
distribution
distribution
dns
dns
dns
docker
docker
docker
documentation
documentation
documentation
duplicate
duplicate
duplicate
enhancement
enhancement
event-stream
event-stream
event-stream
feature-request
feature-request
feature-request
freebsd
freebsd
freebsd
getting-started
getting-started
getting-started
go
go
go
good first issue
good first issue
good first issue
gui
gui
gui
help wanted
help wanted
help wanted
home-assistant
home-assistant
home-assistant
idp
idp
idp
inconsistency
inconsistency
inconsistency
integration
integration
integration
integrations
integrations
integrations
ios
ios
ios
ipv6
ipv6
ipv6
jwt
jwt
jwt
k8s
k8s
k8s
keycloak
keycloak
keycloak
linux
linux
linux
login
login
login
macos
macos
macos
management-service
management-service
management-service
Medium
Medium
Medium
missing-docs
missing-docs
missing-docs
mobile
mobile
mobile
moved-internal
moved-internal
moved-internal
needs-review
needs-review
needs-review
netbird-ui
netbird-ui
netbird-ui
networking
networking
networking
new-platform
new-platform
new-platform
nginx
nginx
nginx
notification
notification
notification
okta
okta
okta
openwrt
openwrt
openwrt
P2
P2
P2
packaging
packaging
packaging
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
performance
performance
performance
postgres
postgres
postgres
posture-checks
posture-checks
posture-checks
psk
psk
psk
pull-request
Mirrored from GitHub Pull Request
 question
question
question
refactor
refactor
refactor
relay
relay
relay
release
release
release
rfc
rfc
rfc
routes
routes
routes
security
security
security
security-improvement
security-improvement
security-improvement
security-related
security-related
security-related
self-hosting
self-hosting
self-hosting
server
server
server
signal
signal
signal
sleep-issue
sleep-issue
sleep-issue
ssh
ssh
ssh
ssl
ssl
ssl
status
status
status
store
store
store
synology
synology
synology
system-compatibility-issue
system-compatibility-issue
system-compatibility-issue
test-suite
test-suite
test-suite
third-party-integration
third-party-integration
third-party-integration
triage
triage
triage
triage
triage
triage
triage-needed
triage-needed
triage-needed
troubleshooting
troubleshooting
troubleshooting
UX
UX
UX
waiting-feedback
waiting-feedback
waiting-feedback
windows
windows
windows
wontfix
wontfix
wontfix
zitadel
zitadel
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dynr-servers saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11581
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?