DYNR/netbird
2
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-05 00:15:26 -04:00
Code Issues 23.5k Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #5948] Built-in OIDC discovery shim in NetBird Reverse Proxy to break embedded-Dex startup deadlock #11372

New Issue
Open
opened 2026-08-05 01:29:29 -04:00 by saavagebueno · 0 comments
saavagebueno commented 2026-08-05 01:29:29 -04:00
Owner
Copy Link

Originally created by @renne on GitHub (Apr 21, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5948

Is your feature request related to a problem? Please describe.

When the NetBird Reverse Proxy is used to expose an internal service that also
serves as an OIDC IdP consumed by the embedded Dex connector (e.g., Nextcloud
behind the reverse proxy, configured as a Dex OIDC connector for SSO into the
NetBird dashboard), a hard circular dependency emerges between the management
server and the reverse proxy:

  1. netbird-server starts → Dex initialises OIDC connectors → Dex fetches
    https://<idp-host>/.well-known/openid-configuration.
  2. <idp-host> resolves (via container DNS / split-horizon / public DNS) back
    into the same box, hits Traefik → reverse-proxy TCP SNI passthrough →
    netbird-reverse-proxy:8443.
  3. netbird-reverse-proxy dials the upstream via the NetBird overlay (e.g.,
    nextcloud.docker1), which requires a connected peer key obtained from…
    netbird-server — which is still blocked on Dex init.
  4. Dex hangs indefinitely. netbird-server never reaches "running gRPC server".
    No peer ever reconnects. System is wedged.

This is reproducible on any combined+embedded-Dex deployment where:

  • a Dex connector's issuer is hosted on a domain whose HTTPS is served through
    the NetBird Reverse Proxy, and
  • the overlay path to that upstream depends on the management server being up.

Once the cycle is broken (e.g., by temporarily disabling the Dex connector,
letting management boot, letting the overlay recover), the problem reappears on
every subsequent netbird-server restart.

Describe the solution you'd like

Add a built-in OIDC discovery shim to netbirdio/reverse-proxy:

  1. Ingest: a small option on a reverse-proxy service definition, e.g.
    oidc_idp_cache: true or a dedicated service type, declaring that the
    upstream is an OIDC IdP whose discovery documents must be cached locally.
  2. Prefetch + refresh: parse responses for
    /.well-known/openid-configuration and follow jwks_uri to cache the JSON
    in memory (and optionally on-disk for restart survival). Refresh on a
    configurable interval, keeping the latest known-good doc.
  3. Fast path: when a request arrives for
    /.well-known/openid-configuration or the configured jwks_uri path on that
    host, serve the cached document directly if the upstream is unreachable (or
    always, gated by config). Aggressive short timeouts so we never block
    management startup.
  4. Failure mode: if no cached document is available yet and the upstream is
    unreachable, return 503 quickly (not 502 after long timeouts), so Dex's
    connector init fails fast and netbird-server keeps the rest of its startup
    progressing. Alternatively return the last-known-good doc with a stale
    indicator.

This makes the management server's Dex init no longer dependent on the
overlay being up — the exact dependency that creates the deadlock.

For the specific case where netbird-server itself is the IdP host (embedded
Dex), no shim is needed — Dex serves its own metadata locally. The shim matters
for third-party OIDC IdPs reverse-proxied through NetBird (Nextcloud,
GitLab, Keycloak-behind-proxy, etc.) where the proxy is on the same host as the
management server.

Describe alternatives you've considered

  1. External nginx sidecar — the workaround deployed: a nginx:alpine
    container joined to the netbird Docker network with a network alias
    matching the IdP hostname, serving prefetched
    /.well-known/openid-configuration and jwks_uri locally and
    reverse-proxying everything else to Traefik → netbird-reverse-proxy →
    overlay. Combined with network aliases on netbird-server so Dex resolves
    the IdP host to the shim. Works, but:

    • adds a whole extra service operators must configure, run, and secure
    • duplicates a certificate or requires insecureSkipVerify on the Dex
      connector
    • needs manual refresh of the cached JSON when the upstream rotates keys
    • operators who don't understand the deadlock will not discover this
      pattern on their own

  2. extra_hosts pointing to Traefik's IPv4 — does not break the loop;
    it's the same chain, just a different entry point.

  3. Start Dex lazily / add connector-init retries that don't block server
    boot     — would help, but doesn't solve the problem: clients still can't
    authenticate while the overlay is broken, and restarts of netbird-server
    during overlay incidents will still hang until the overlay comes back.

  4. Disable connectors in idp.db temporarily during restarts —
    operationally unacceptable; loses all inbound SSO logins during that window.

Additional context

Minimum viable design sketch for the reverse-proxy config change:

services:
  - id: nextcloud
    host: idp.example.com
    upstream:
      peer_key: "…"
      domain: nextcloud-aio-apache.docker1
      port: 443
    oidc_cache:
      enabled: true
      paths:
        - "/.well-known/openid-configuration"
        # jwks_uri discovered automatically from the document above
      refresh_interval: 1h
      stale_ttl: 24h
      serve_stale_on_upstream_error: true

Behaviour at startup: reverse-proxy loads the cached JSON from its persistent
volume if present, then refreshes in the background. Serves cached immediately
on request regardless of overlay state. A small feature surface that solves a
sharp real-world deployment deadlock.

Companion bug: #5947 (missing DeviceAuthorizationFlow/PKCEAuthorizationFlow in
combined+embedded-Dex mode).

Originally created by @renne on GitHub (Apr 21, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5948 **Is your feature request related to a problem? Please describe.** When the NetBird Reverse Proxy is used to expose an internal service that **also serves as an OIDC IdP consumed by the embedded Dex connector** (e.g., Nextcloud behind the reverse proxy, configured as a Dex OIDC connector for SSO into the NetBird dashboard), a hard circular dependency emerges between the management server and the reverse proxy: 1. `netbird-server` starts → Dex initialises OIDC connectors → Dex fetches `https://<idp-host>/.well-known/openid-configuration`. 2. `<idp-host>` resolves (via container DNS / split-horizon / public DNS) back into the same box, hits Traefik → reverse-proxy TCP SNI passthrough → `netbird-reverse-proxy:8443`. 3. `netbird-reverse-proxy` dials the upstream via the NetBird overlay (e.g., `nextcloud.docker1`), which requires a connected peer key obtained from… `netbird-server` — which is still blocked on Dex init. 4. Dex hangs indefinitely. `netbird-server` never reaches "running gRPC server". No peer ever reconnects. System is wedged. This is reproducible on any combined+embedded-Dex deployment where: - a Dex connector's `issuer` is hosted on a domain whose HTTPS is served through the NetBird Reverse Proxy, and - the overlay path to that upstream depends on the management server being up. Once the cycle is broken (e.g., by temporarily disabling the Dex connector, letting management boot, letting the overlay recover), the problem reappears on every subsequent `netbird-server` restart. **Describe the solution you'd like** Add a built-in **OIDC discovery shim** to `netbirdio/reverse-proxy`: 1. **Ingest**: a small option on a reverse-proxy service definition, e.g. `oidc_idp_cache: true` or a dedicated service type, declaring that the upstream is an OIDC IdP whose discovery documents must be cached locally. 2. **Prefetch + refresh**: parse responses for `/.well-known/openid-configuration` and follow `jwks_uri` to cache the JSON in memory (and optionally on-disk for restart survival). Refresh on a configurable interval, keeping the latest known-good doc. 3. **Fast path**: when a request arrives for `/.well-known/openid-configuration` or the configured `jwks_uri` path on that host, serve the cached document directly if the upstream is unreachable (or always, gated by config). Aggressive short timeouts so we never block management startup. 4. **Failure mode**: if no cached document is available yet and the upstream is unreachable, return 503 quickly (not 502 after long timeouts), so Dex's connector init fails fast and netbird-server keeps the rest of its startup progressing. Alternatively return the last-known-good doc with a stale indicator. This makes the management server's Dex init **no longer dependent** on the overlay being up — the exact dependency that creates the deadlock. For the specific case where netbird-server itself is the IdP host (embedded Dex), no shim is needed — Dex serves its own metadata locally. The shim matters for **third-party OIDC IdPs reverse-proxied through NetBird** (Nextcloud, GitLab, Keycloak-behind-proxy, etc.) where the proxy is on the same host as the management server. **Describe alternatives you've considered** 1. **External nginx sidecar** — the workaround deployed: a `nginx:alpine` container joined to the `netbird` Docker network with a network alias matching the IdP hostname, serving prefetched `/.well-known/openid-configuration` and `jwks_uri` locally and reverse-proxying everything else to Traefik → netbird-reverse-proxy → overlay. Combined with network aliases on `netbird-server` so Dex resolves the IdP host to the shim. Works, but: - adds a whole extra service operators must configure, run, and secure - duplicates a certificate or requires `insecureSkipVerify` on the Dex connector - needs manual refresh of the cached JSON when the upstream rotates keys - operators who don't understand the deadlock will not discover this pattern on their own 2. **`extra_hosts` pointing to Traefik's IPv4** — does not break the loop; it's the same chain, just a different entry point. 3. **Start Dex lazily / add connector-init retries that don't block server boot** — would help, but doesn't solve the problem: clients still can't authenticate while the overlay is broken, and restarts of `netbird-server` during overlay incidents will still hang until the overlay comes back. 4. **Disable connectors in `idp.db` temporarily during restarts** — operationally unacceptable; loses all inbound SSO logins during that window. **Additional context** Minimum viable design sketch for the reverse-proxy config change: ```yaml services: - id: nextcloud host: idp.example.com upstream: peer_key: "…" domain: nextcloud-aio-apache.docker1 port: 443 oidc_cache: enabled: true paths: - "/.well-known/openid-configuration" # jwks_uri discovered automatically from the document above refresh_interval: 1h stale_ttl: 24h serve_stale_on_upstream_error: true ``` Behaviour at startup: reverse-proxy loads the cached JSON from its persistent volume if present, then refreshes in the background. Serves cached immediately on request regardless of overlay state. A small feature surface that solves a sharp real-world deployment deadlock. Companion bug: #5947 (missing DeviceAuthorizationFlow/PKCEAuthorizationFlow in combined+embedded-Dex mode).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2021 Q4
2021 Q4
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
accessibility
accessibility
accessibility
acl
acl
acl
agent
agent
agent
agent
agent
agent
Android
Android
Android
Android
Android
Android
api
api
api
authentik
authentik
authentik
automation
automation
automation
azure
azure
azure
battery-usage
battery-usage
battery-usage
bug
cache
cache
cache
client
client
client
client-ui
client-ui
client-ui
cloud
cloud
cloud
cloud-only
cloud-only
cloud-only
cloudflare
cloudflare
cloudflare
community
community
community
compatibility
compatibility
compatibility
config-idp
config-idp
config-idp
config-issue
config-issue
config-issue
connection
connection
connection
contribution
contribution
contribution
coturn
coturn
coturn
cross-vpn
cross-vpn
cross-vpn
dashboard
dashboard
dashboard
data-usage
data-usage
data-usage
distribution
distribution
distribution
dns
dns
dns
docker
docker
docker
documentation
documentation
documentation
duplicate
duplicate
duplicate
enhancement
enhancement
event-stream
event-stream
event-stream
feature-request
feature-request
feature-request
freebsd
freebsd
freebsd
getting-started
getting-started
getting-started
go
go
go
good first issue
good first issue
good first issue
gui
gui
gui
help wanted
help wanted
help wanted
home-assistant
home-assistant
home-assistant
idp
idp
idp
inconsistency
inconsistency
inconsistency
integration
integration
integration
integrations
integrations
integrations
ios
ios
ios
ipv6
ipv6
ipv6
jwt
jwt
jwt
k8s
k8s
k8s
keycloak
keycloak
keycloak
linux
linux
linux
login
login
login
macos
macos
macos
management-service
management-service
management-service
Medium
Medium
Medium
missing-docs
missing-docs
missing-docs
mobile
mobile
mobile
moved-internal
moved-internal
moved-internal
needs-review
needs-review
needs-review
netbird-ui
netbird-ui
netbird-ui
networking
networking
networking
new-platform
new-platform
new-platform
nginx
nginx
nginx
notification
notification
notification
okta
okta
okta
openwrt
openwrt
openwrt
P2
P2
P2
packaging
packaging
packaging
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
performance
performance
performance
postgres
postgres
postgres
posture-checks
posture-checks
posture-checks
psk
psk
psk
pull-request
Mirrored from GitHub Pull Request
 question
question
question
refactor
refactor
refactor
relay
relay
relay
release
release
release
rfc
rfc
rfc
routes
routes
routes
security
security
security
security-improvement
security-improvement
security-improvement
security-related
security-related
security-related
self-hosting
self-hosting
self-hosting
server
server
server
signal
signal
signal
sleep-issue
sleep-issue
sleep-issue
ssh
ssh
ssh
ssl
ssl
ssl
status
status
status
store
store
store
synology
synology
synology
system-compatibility-issue
system-compatibility-issue
system-compatibility-issue
test-suite
test-suite
test-suite
third-party-integration
third-party-integration
third-party-integration
triage
triage
triage
triage
triage
triage
triage-needed
triage-needed
triage-needed
troubleshooting
troubleshooting
troubleshooting
UX
UX
UX
waiting-feedback
waiting-feedback
waiting-feedback
windows
windows
windows
wontfix
wontfix
wontfix
zitadel
zitadel
zitadel
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dynr-servers saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11372
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?