[GH-ISSUE #5449] netbird ssh JWT auth fails - client discards fresh SSH token, sends stale token from netbird up #10588

Closed
opened 2026-08-05 01:26:32 -04:00 by saavagebueno · 2 comments
Owner

Originally created by @chris-prosser on GitHub (Feb 25, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5449

Describe the problem

After upgrading our self-hosted NetBird management server to v0.65.2, netbird ssh with JWT authentication always fails. The SSH server's JWT validator rejects the token with age=5m0.Xs, max=5m0s
every time, regardless of how quickly the SSH command is run after netbird up. The OIDC flow for SSH shows "Authentication successful!" but the token sent to the target peer is always the one
obtained during netbird up, not a fresh token. This was working before the management server upgrade.

To Reproduce

  1. Self-hosted NetBird with Entra ID (Azure AD) OIDC
  2. Run netbird up - OIDC completes successfully
  3. Run netbird ssh username@ 'hostname' - OIDC completes, shows "Authentication successful!"
  4. SSH handshake fails: ssh: unable to authenticate, attempted methods [none password], no supported methods remain
  5. Target peer log shows: JWT authentication failed: token expired for user=...: age=5m0.884820059s, max=5m0s
  6. Repeating immediately gives the same result - token age is always ~5m0s

Expected behavior

The client should obtain a fresh JWT token for the netbird ssh session rather than reusing the cached token from netbird up. Alternatively, the max age for JWT validation should be configurable on
the SSH server side. The current hardcoded max=5m0s in client/ssh/server/server.go makes JWT SSH auth unusable since the token from netbird up will always be expired by the time an SSH session is
attempted.

Are you using NetBird Cloud?

No. Self-hosted management server, signal, relay, and dashboard - all on v0.65.2. Identity provider is Microsoft Entra ID (Azure AD).

NetBird version

Tested with both client versions - identical behavior:

  • netbird version: 0.65.3
  • netbird version: 0.66.0

Is any other VPN software installed?

No

Debug output

Target peer SSH server log (/var/log/netbird/client.log):
INFO client/ssh/server/server.go:779: SSH connection from NetBird peer 100.x.x.x:51482 allowed
INFO client/ssh/server/server.go:463: JWT validator initialized successfully
WARN [session: hydra-admin@100.x.x.x:51482] client/ssh/server/server.go:610: JWT authentication failed: token expired for user=...: age=5m0.884820059s, max=5m0s

Client output:
SSH authentication required.
Please do the SSO login in your browser.
Waiting for authentication...
Authentication successful!
Failed to connect to hydra-admin@100.x.x.x:22

Error: dial 100..x.x.x:22: ssh handshake: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

  • All system clocks verified in sync via chrony (< 1 second drift between client and target)
  • --ssh-jwt-cache-ttl on the target peer only controls caching, does not change the max=5m0s validation
  • --disable-ssh-auth on the target peer works as a workaround but bypasses JWT auth entirely

Server-side debug:
398f1d2d0c02a12b5e7fb8a49d10b7a02cc8a90a9608f046f41f308b31d93fe1/be538b8b-a379-4e88-8b15-7f717df3a69f

Client-side debug:
398f1d2d0c02a12b5e7fb8a49d10b7a02cc8a90a9608f046f41f308b31d93fe1/08f4bd7a-7eb8-4276-ab75-59a59d49e345

Key finding from client log: JWT caching disabled, not storing token - the client obtains a fresh JWT via OIDC for the SSH session but discards it, then sends the stale token from netbird up which
is always ~5 minutes old by the time the SSH session is attempted.

Originally created by @chris-prosser on GitHub (Feb 25, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5449 **Describe the problem** After upgrading our self-hosted NetBird management server to v0.65.2, netbird ssh with JWT authentication always fails. The SSH server's JWT validator rejects the token with age=5m0.Xs, max=5m0s every time, regardless of how quickly the SSH command is run after netbird up. The OIDC flow for SSH shows "Authentication successful!" but the token sent to the target peer is always the one obtained during netbird up, not a fresh token. This was working before the management server upgrade. **To Reproduce** 1. Self-hosted NetBird with Entra ID (Azure AD) OIDC 2. Run netbird up - OIDC completes successfully 3. Run netbird ssh username@<peer-ip> 'hostname' - OIDC completes, shows "Authentication successful!" 4. SSH handshake fails: ssh: unable to authenticate, attempted methods [none password], no supported methods remain 5. Target peer log shows: JWT authentication failed: token expired for user=...: age=5m0.884820059s, max=5m0s 6. Repeating immediately gives the same result - token age is always ~5m0s **Expected behavior** The client should obtain a fresh JWT token for the netbird ssh session rather than reusing the cached token from netbird up. Alternatively, the max age for JWT validation should be configurable on the SSH server side. The current hardcoded max=5m0s in client/ssh/server/server.go makes JWT SSH auth unusable since the token from netbird up will always be expired by the time an SSH session is attempted. **Are you using NetBird Cloud?** No. Self-hosted management server, signal, relay, and dashboard - all on v0.65.2. Identity provider is Microsoft Entra ID (Azure AD). **NetBird version** Tested with both client versions - identical behavior: - netbird version: 0.65.3 - netbird version: 0.66.0 **Is any other VPN software installed?** No **Debug output** Target peer SSH server log (/var/log/netbird/client.log): INFO client/ssh/server/server.go:779: SSH connection from NetBird peer 100.x.x.x:51482 allowed INFO client/ssh/server/server.go:463: JWT validator initialized successfully WARN [session: hydra-admin@100.x.x.x:51482] client/ssh/server/server.go:610: JWT authentication failed: token expired for user=...: age=5m0.884820059s, max=5m0s Client output: SSH authentication required. Please do the SSO login in your browser. Waiting for authentication... Authentication successful! Failed to connect to hydra-admin@100.x.x.x:22 Error: dial 100..x.x.x:22: ssh handshake: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain - All system clocks verified in sync via chrony (< 1 second drift between client and target) - --ssh-jwt-cache-ttl on the target peer only controls caching, does not change the max=5m0s validation - --disable-ssh-auth on the target peer works as a workaround but bypasses JWT auth entirely Server-side debug: 398f1d2d0c02a12b5e7fb8a49d10b7a02cc8a90a9608f046f41f308b31d93fe1/be538b8b-a379-4e88-8b15-7f717df3a69f Client-side debug: 398f1d2d0c02a12b5e7fb8a49d10b7a02cc8a90a9608f046f41f308b31d93fe1/08f4bd7a-7eb8-4276-ab75-59a59d49e345 Key finding from client log: JWT caching disabled, not storing token - the client obtains a fresh JWT via OIDC for the SSH session but discards it, then sends the stale token from netbird up which is always ~5 minutes old by the time the SSH session is attempted.
saavagebueno added the triage-needed label 2026-08-05 01:26:32 -04:00
Author
Owner

@lixmal commented on GitHub (Feb 25, 2026):

netbird up doesn't cache any tokens, how did you come to this conclusion?

Can you check if your IdP happens to issue tokens with the wrong issue date?

<!-- gh-comment-id:3961737240 --> @lixmal commented on GitHub (Feb 25, 2026): `netbird up` doesn't cache any tokens, how did you come to this conclusion? Can you check if your IdP happens to issue tokens with the wrong issue date?
Author
Owner

@chris-prosser commented on GitHub (Feb 25, 2026):

You were right to ask about the IdP. We've confirmed that Azure Entra ID backdates the iat claim by exactly 300 seconds (5 minutes) on every token it issues, regardless of auth flow or parameters.

Test methodology:

We requested a fresh token via device code flow and compared timestamps:

iat (issued at): 1772052031 (20:40:31Z)
device code req: 1772052290 (20:44:50Z)
token exchange: 1772052331 (20:45:31Z)

iat = exchange_time - 300s (exactly)
token age at exchange: 300s

The iat is always exactly 5 minutes before the actual token exchange time. This isn't related to max_age=0 or prompt=login -- it appears to be a hardcoded behavior in Entra ID's token service.

Since the SSH server's DefaultJWTMaxTokenAge is also 300 seconds, the token is expired the instant it arrives.

Our setup:

  • Management server: v0.65.2, LoginFlag: 0 (no max_age=0 in auth URL)
  • Client: v0.66.0
  • IdP: Azure Entra ID (Microsoft identity platform v2.0)

Server-side log from the target peer confirms:
JWT authentication failed: token expired for user=...: age=5m1.046441158s, max=5m0s

Is there any work-around you can offer?

<!-- gh-comment-id:3961995073 --> @chris-prosser commented on GitHub (Feb 25, 2026): You were right to ask about the IdP. We've confirmed that Azure Entra ID backdates the iat claim by exactly 300 seconds (5 minutes) on every token it issues, regardless of auth flow or parameters. Test methodology: We requested a fresh token via device code flow and compared timestamps: iat (issued at): 1772052031 (20:40:31Z) device code req: 1772052290 (20:44:50Z) token exchange: 1772052331 (20:45:31Z) iat = exchange_time - 300s (exactly) token age at exchange: 300s The iat is always exactly 5 minutes before the actual token exchange time. This isn't related to max_age=0 or prompt=login -- it appears to be a hardcoded behavior in Entra ID's token service. Since the SSH server's DefaultJWTMaxTokenAge is also 300 seconds, the token is expired the instant it arrives. Our setup: - Management server: v0.65.2, LoginFlag: 0 (no max_age=0 in auth URL) - Client: v0.66.0 - IdP: Azure Entra ID (Microsoft identity platform v2.0) Server-side log from the target peer confirms: JWT authentication failed: token expired for user=...: age=5m1.046441158s, max=5m0s Is there any work-around you can offer?
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#10588