[GH-ISSUE #5485] Reverse proxy unable to reach destination on a network resource #11300

Closed
opened 2026-08-05 01:29:13 -04:00 by saavagebueno · 1 comment
Owner

Originally created by @jowsef-git on GitHub (Mar 2, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5485

Describe the problem
When using the reverse proxy on a fairly new instance of netbird (combined container) the proxy is unable to reach the destination. The destination is on a network that has a peer, the destination service is accessible directly when connected to netbird with the client.

When adding a pin or password to the service the website will ask for that pin/pass and then eventually give up connecting to the destination

Image

To Reproduce

Steps to reproduce the behavior:

  1. Create a VPS
  2. Install Docker Components
  3. Run the Netbird selfhost install script
  4. Register Peer on local network
  5. Point DNS records and verify in proxy settings on web portal
  6. Connect

Expected behavior

Reverse proxy expected to show destination website

Are you using NetBird Cloud?

Self-Hosted

NetBird version

0.66.0

Is any other VPN software installed?

no

Debug output

docker logs netbird-proxy -f
2026-03-02T15:34:14.869Z WARN [peer: waf6v3fBW/NPrOKyL9KATdoQoUIHeKriFCra6I/UGWQ=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet
2026-03-02T15:35:48.374Z WARN [peer: waf6v3fBW/NPrOKyL9KATdoQoUIHeKriFCra6I/UGWQ=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet
2026-03-02T15:35:48.376Z WARN [peer: waf6v3fBW/NPrOKyL9KATdoQoUIHeKriFCra6I/UGWQ=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet

cat proxy.env
# NetBird Proxy Configuration
NB_PROXY_DEBUG_LOGS=true# Use internal Docker network to connect to management (avoids hairpin NAT issues)
NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80
# Allow insecure gRPC connection to management (required for internal Docker network)
NB_PROXY_ALLOW_INSECURE=true
# Public URL where this proxy is reachable (used for cluster registration)
NB_PROXY_DOMAIN=proxy.jtlab.uk
NB_PROXY_ADDRESS=:8443
NB_PROXY_TOKEN=[removed]
NB_PROXY_CERTIFICATE_DIRECTORY=/certs
NB_PROXY_ACME_CERTIFICATES=true
NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01
NB_PROXY_FORWARDED_PROTO=https
# Enable PROXY protocol to preserve client IPs through L4 proxies (Traefik TCP passthrough)
NB_PROXY_PROXY_PROTOCOL=true
# Trust Traefik's IP for PROXY protocol headers
NB_PROXY_TRUSTED_PROXIES=172.30.0.10
Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • [n/a] Restarted the NetBird client
  • [n/a] Disabled other VPN software
  • Checked firewall settings
Originally created by @jowsef-git on GitHub (Mar 2, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5485 **Describe the problem** When using the reverse proxy on a fairly new instance of netbird (combined container) the proxy is unable to reach the destination. The destination is on a network that has a peer, the destination service is accessible directly when connected to netbird with the client. When adding a pin or password to the service the website will ask for that pin/pass and then eventually give up connecting to the destination <img width="944" height="625" alt="Image" src="https://github.com/user-attachments/assets/80724957-3177-4fb4-be22-656abba278fb" /> **To Reproduce** Steps to reproduce the behavior: 1. Create a VPS 2. Install Docker Components 3. Run the Netbird selfhost install script 4. Register Peer on local network 5. Point DNS records and verify in proxy settings on web portal 6. Connect **Expected behavior** Reverse proxy expected to show destination website **Are you using NetBird Cloud?** Self-Hosted **NetBird version** 0.66.0 **Is any other VPN software installed?** no **Debug output** docker logs netbird-proxy -f 2026-03-02T15:34:14.869Z WARN [peer: waf6v3fBW/NPrOKyL9KATdoQoUIHeKriFCra6I/UGWQ=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet 2026-03-02T15:35:48.374Z WARN [peer: waf6v3fBW/NPrOKyL9KATdoQoUIHeKriFCra6I/UGWQ=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet 2026-03-02T15:35:48.376Z WARN [peer: waf6v3fBW/NPrOKyL9KATdoQoUIHeKriFCra6I/UGWQ=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet cat proxy.env \# NetBird Proxy Configuration NB_PROXY_DEBUG_LOGS=true# Use internal Docker network to connect to management (avoids hairpin NAT issues) NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80 \# Allow insecure gRPC connection to management (required for internal Docker network) NB_PROXY_ALLOW_INSECURE=true \# Public URL where this proxy is reachable (used for cluster registration) NB_PROXY_DOMAIN=proxy.jtlab.uk NB_PROXY_ADDRESS=:8443 NB_PROXY_TOKEN=[removed] NB_PROXY_CERTIFICATE_DIRECTORY=/certs NB_PROXY_ACME_CERTIFICATES=true NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01 NB_PROXY_FORWARDED_PROTO=https \# Enable PROXY protocol to preserve client IPs through L4 proxies (Traefik TCP passthrough) NB_PROXY_PROXY_PROTOCOL=true \# Trust Traefik's IP for PROXY protocol headers NB_PROXY_TRUSTED_PROXIES=172.30.0.10 **Screenshots** If applicable, add screenshots to help explain your problem. **Additional context** Add any other context about the problem here. **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [n/a] Restarted the NetBird client - [n/a] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2026-08-05 01:29:13 -04:00
Author
Owner

@jowsef-git commented on GitHub (Mar 2, 2026):

Resolved - Token Invalid:

docker logs netbird-server -f

2026-03-02T15:13:15.530Z INFO [peer_id: sha-RWTSDB/w6jA1ZZ6rgUDt1uZdPXvXLkI2RtJESVHVWPg=] relay/server/peer.go:120: peer exited gracefully
2026-03-02T15:13:15.533Z WARN [context: GRPC, requestID: d6iqc17pp18c73e7vafg, accountID: d6dm9mnpp18c73df6b10] management/internals/shared/grpc/server.go:370: recv job response error: rpc error: code = Canceled desc = context canceled
2026-03-02T15:44:32.717Z INFO [requestID: d6ir0o7pp18c73e810gg, context: HTTP] management/server/http/handlers/instance/instance_handler.go:49: instance setup status: false
2026-03-02T15:44:39.167Z INFO [context: HTTP, requestID: d6ir0pvpp18c73e810m0] management/server/http/handlers/instance/instance_handler.go:49: instance setup status: false
2026-03-02T15:44:39.766Z INFO [context: HTTP, requestID: d6ir0pvpp18c73e810mg] management/server/http/handlers/instance/instance_handler.go:49: instance setup status: false
2026-03-02T17:27:16.523Z ERRO relay/server/relay.go:130: failed to handshake: validate sha-EyP8gPE3cPX5XchK7T4uv+JP7cr+YaCUv+ttFfpQxKw= (172.30.0.10:51750): expired token
2026-03-02T17:27:16.705Z ERRO relay/server/relay.go:130: failed to handshake: validate sha-EyP8gPE3cPX5XchK7T4uv+JP7cr+YaCUv+ttFfpQxKw= (172.30.0.10:51762): expired token
2026-03-02T17:27:18.694Z ERRO relay/server/relay.go:130: failed to handshake: validate sha-EyP8gPE3cPX5XchK7T4uv+JP7cr+YaCUv+ttFfpQxKw= (172.30.0.10:51776): expired token

Changed proxy token following - https://docs.netbird.io/selfhosted/migration/enable-reverse-proxy#step-3-add-the-proxy-service-to-docker-compose-yml

docker exec -it netbird-server /go/bin/netbird-server token create --name "my-proxy" --config /etc/netbird/config.yaml

Changed token in proxy.env

docker compose down

docker compose up -d

<!-- gh-comment-id:3986263966 --> @jowsef-git commented on GitHub (Mar 2, 2026): Resolved - Token Invalid: docker logs netbird-server -f 2026-03-02T15:13:15.530Z INFO [peer_id: sha-RWTSDB/w6jA1ZZ6rgUDt1uZdPXvXLkI2RtJESVHVWPg=] relay/server/peer.go:120: peer exited gracefully 2026-03-02T15:13:15.533Z WARN [context: GRPC, requestID: d6iqc17pp18c73e7vafg, accountID: d6dm9mnpp18c73df6b10] management/internals/shared/grpc/server.go:370: recv job response error: rpc error: code = Canceled desc = context canceled 2026-03-02T15:44:32.717Z INFO [requestID: d6ir0o7pp18c73e810gg, context: HTTP] management/server/http/handlers/instance/instance_handler.go:49: instance setup status: false 2026-03-02T15:44:39.167Z INFO [context: HTTP, requestID: d6ir0pvpp18c73e810m0] management/server/http/handlers/instance/instance_handler.go:49: instance setup status: false 2026-03-02T15:44:39.766Z INFO [context: HTTP, requestID: d6ir0pvpp18c73e810mg] management/server/http/handlers/instance/instance_handler.go:49: instance setup status: false 2026-03-02T17:27:16.523Z ERRO relay/server/relay.go:130: failed to handshake: validate sha-EyP8gPE3cPX5XchK7T4uv+JP7cr+YaCUv+ttFfpQxKw= (172.30.0.10:51750): expired token 2026-03-02T17:27:16.705Z ERRO relay/server/relay.go:130: failed to handshake: validate sha-EyP8gPE3cPX5XchK7T4uv+JP7cr+YaCUv+ttFfpQxKw= (172.30.0.10:51762): expired token 2026-03-02T17:27:18.694Z ERRO relay/server/relay.go:130: failed to handshake: validate sha-EyP8gPE3cPX5XchK7T4uv+JP7cr+YaCUv+ttFfpQxKw= (172.30.0.10:51776): expired token Changed proxy token following - https://docs.netbird.io/selfhosted/migration/enable-reverse-proxy#step-3-add-the-proxy-service-to-docker-compose-yml docker exec -it netbird-server /go/bin/netbird-server token create --name "my-proxy" --config /etc/netbird/config.yaml Changed token in proxy.env docker compose down docker compose up -d
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11300