[GH-ISSUE #6400] Reverse Proxy 0.72.x regression: mode=tcp services no longer bind their custom ports (works in 0.71.4) #11604

Closed
opened 2026-08-05 01:30:08 -04:00 by saavagebueno · 3 comments
Owner

Originally created by @wgetnz on GitHub (Jun 10, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/6400

Description

After upgrading netbirdio/reverse-proxy from 0.71.4 to 0.72.x, all mode=tcp services became unreachable from outside. The proxy process no longer binds the configured custom ports — inside the container only :8443 is listening. Downgrading only the proxy container back to 0.71.4 (management/server kept at 0.72.3) immediately restores correct behavior.

Environment

  • Self-hosted, single-account
  • netbird-server (management/signal/relay): 0.72.3
  • netbird-proxy: broken on 0.72.0–0.72.3, works on 0.71.4
  • Several mode=tcp services (SSH→22, RDP→3389, …) with custom listen ports (10001–10004, 3389)

Symptom

  • External TCP connect to any mode=tcp service's listen port → connection refused/reset
  • mode=http services (443) work fine
  • docker exec netbird-proxy netstat -tlnp on 0.72.3 shows only :::8443 — none of the configured custom ports
  • Startup log shows a single proxy main listener bound :8443; no per-port listeners

Differential evidence (same environment, only the proxy image changed)

  • 0.72.3: netstat → only :::8443. TCP services unreachable.
  • 0.71.4: netstat:::8443 :::10001 :::10002 :::10003 :::10004 :::3389. After Initial mapping sync complete the proxy binds each custom port; external SSH works (received a real SSH banner through the tunnel).

In both versions the proxy registers the services correctly (debug clients lists the l4:<id> keys, mapping sync completes, the WireGuard tunnel to the target peer is healthy). The only difference is that 0.72.x never opens the per-port TCP listeners.

Steps to reproduce

  1. Self-host netbird with the reverse-proxy
  2. Create a mode=tcp service with a custom listen_port (e.g. SSH, listen 10001 → target peer :22)
  3. On proxy 0.72.x: external connect to :10001 fails; netstat inside the container shows no :10001 listener
  4. Downgrade only the proxy container to 0.71.4 → :10001 starts listening, connection works

Notes

  • Not the same as #6068 (that one is mode=tls / SNI passthrough / autocert). This is mode=tcp raw per-port forwarding.
  • Side note: 0.71.4 force-downloads the GeoLite2 DB at startup and hard-blocks if it can't reach pkgs.netbird.io (0.72.3 only warns) — relevant for restricted/air-gapped networks.
Originally created by @wgetnz on GitHub (Jun 10, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/6400 ## Description After upgrading `netbirdio/reverse-proxy` from 0.71.4 to 0.72.x, all `mode=tcp` services became unreachable from outside. The proxy process no longer binds the configured custom ports — inside the container only `:8443` is listening. Downgrading **only** the proxy container back to 0.71.4 (management/server kept at 0.72.3) immediately restores correct behavior. ## Environment - Self-hosted, single-account - netbird-server (management/signal/relay): **0.72.3** - netbird-proxy: broken on **0.72.0–0.72.3**, works on **0.71.4** - Several `mode=tcp` services (SSH→22, RDP→3389, …) with custom listen ports (10001–10004, 3389) ## Symptom - External TCP connect to any `mode=tcp` service's listen port → connection refused/reset - `mode=http` services (443) work fine - `docker exec netbird-proxy netstat -tlnp` on 0.72.3 shows only `:::8443` — none of the configured custom ports - Startup log shows a single `proxy main listener bound :8443`; no per-port listeners ## Differential evidence (same environment, only the proxy image changed) - **0.72.3:** `netstat` → only `:::8443`. TCP services unreachable. - **0.71.4:** `netstat` → `:::8443 :::10001 :::10002 :::10003 :::10004 :::3389`. After `Initial mapping sync complete` the proxy binds each custom port; external SSH works (received a real SSH banner through the tunnel). In both versions the proxy registers the services correctly (`debug clients` lists the `l4:<id>` keys, mapping sync completes, the WireGuard tunnel to the target peer is healthy). The only difference is that 0.72.x never opens the per-port TCP listeners. ## Steps to reproduce 1. Self-host netbird with the reverse-proxy 2. Create a `mode=tcp` service with a custom `listen_port` (e.g. SSH, listen 10001 → target peer :22) 3. On proxy 0.72.x: external connect to :10001 fails; `netstat` inside the container shows no :10001 listener 4. Downgrade only the proxy container to 0.71.4 → :10001 starts listening, connection works ## Notes - Not the same as #6068 (that one is `mode=tls` / SNI passthrough / autocert). This is `mode=tcp` raw per-port forwarding. - Side note: 0.71.4 force-downloads the GeoLite2 DB at startup and hard-blocks if it can't reach `pkgs.netbird.io` (0.72.3 only warns) — relevant for restricted/air-gapped networks.
Author
Owner

@linear-code[bot] commented on GitHub (Jun 10, 2026):

NET-1259

<!-- gh-comment-id:4676915942 --> @linear-code[bot] commented on GitHub (Jun 10, 2026): <!-- linear-linkback --> <p><a href="https://linear.app/netbird/issue/NET-1259">NET-1259</a></p>
Author
Owner

@erdelmaero commented on GitHub (Jun 11, 2026):

Additional datapoint that narrows the regression window: for us the break is specifically 0.72.2 → 0.72.3, not all of 0.72.x.

Same symptom as described — mode=tcp services register fine (mapping sync completes, l4:<id> keys present, WireGuard tunnel to the target peer healthy), but the per-port TCP listeners never bind; inside the container only the UDP L4 listeners + :8443 are present. mode=http (443) and mode=udp services are unaffected.

Environment (self-hosted, single account):

  • management/signal/relay: 0.72.3
  • Services in question are LiveKit media: two mode=tcp services with custom listen ports 7881 and 7883 → target peers on 192.168.x.x:7881/7883 (plus matching mode=udp 7882/7884, which keep working).

Differential evidence — only the proxy image changed, same env:

proxy image netstat -tlnp inside container external TCP connect
0.72.2 (org.opencontainers.image.version=0.72.2, digest sha256:64c8170d…) :::7881 :::7883 LISTEN present Open
0.72.3 (digest sha256:1fc638d4…) only :::8443 + the UDP ports; no 7881/7883 Closed

So unlike the report of "0.72.0–0.72.3 broken / 0.71.4 good", 0.72.2 binds the custom TCP ports correctly here — whatever broke it landed between the 0.72.2 and 0.72.3 proxy builds. Likely suspect in the 0.72.3 proxy changelog is the non-blocking mapping updates change (#6369), given the symptom is "mapping registered but listener never opened".

Workaround confirmed: pin only the proxy container to netbirdio/reverse-proxy@sha256:64c8170d… (0.72.2) and keep management/signal/relay on 0.72.3 — TCP listeners come back immediately.

<!-- gh-comment-id:4682638793 --> @erdelmaero commented on GitHub (Jun 11, 2026): Additional datapoint that narrows the regression window: for us the break is specifically **0.72.2 → 0.72.3**, not all of 0.72.x. Same symptom as described — `mode=tcp` services register fine (mapping sync completes, `l4:<id>` keys present, WireGuard tunnel to the target peer healthy), but the per-port TCP listeners never bind; inside the container only the UDP L4 listeners + `:8443` are present. `mode=http` (443) and `mode=udp` services are unaffected. **Environment (self-hosted, single account):** - management/signal/relay: **0.72.3** - Services in question are LiveKit media: two `mode=tcp` services with custom listen ports `7881` and `7883` → target peers on `192.168.x.x:7881/7883` (plus matching `mode=udp` 7882/7884, which keep working). **Differential evidence — only the proxy image changed, same env:** | proxy image | `netstat -tlnp` inside container | external TCP connect | |---|---|---| | **0.72.2** (`org.opencontainers.image.version=0.72.2`, digest `sha256:64c8170d…`) | `:::7881 :::7883` **LISTEN** present | **Open** | | **0.72.3** (digest `sha256:1fc638d4…`) | only `:::8443` + the UDP ports; **no 7881/7883** | **Closed** | So unlike the report of "0.72.0–0.72.3 broken / 0.71.4 good", **0.72.2 binds the custom TCP ports correctly here** — whatever broke it landed between the 0.72.2 and 0.72.3 proxy builds. Likely suspect in the 0.72.3 proxy changelog is the *non-blocking mapping updates* change (#6369), given the symptom is "mapping registered but listener never opened". Workaround confirmed: pin **only** the proxy container to `netbirdio/reverse-proxy@sha256:64c8170d…` (0.72.2) and keep management/signal/relay on 0.72.3 — TCP listeners come back immediately.
Author
Owner

@Kariton commented on GitHub (Jun 14, 2026):

rel: https://github.com/netbirdio/netbird/discussions/6399

<!-- gh-comment-id:4702053642 --> @Kariton commented on GitHub (Jun 14, 2026): rel: https://github.com/netbirdio/netbird/discussions/6399
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11604