"Error parsing token: unable to find appropriate key" fixed with restarting management service #1061

Open
opened 2025-11-20 05:23:16 -05:00 by saavagebueno · 16 comments
Owner

Originally created by @mohamed-essam on GitHub (Jul 14, 2024).

Describe the problem

Randomly every few days, netbird up fails with Error: waiting sso login failed with: rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key, this is fixed by restarting management service

To Reproduce

Unknown

Expected behavior

Connection to be established normally

Are you using NetBird Cloud?

Self-hosted.

NetBird version

0.28.4

NetBird status -d output:

N/A

Screenshots

N/A

Additional context

Management server logs:

management-1  | 2024-07-14T07:18:21Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
management-1  | 2024-07-14T07:18:21Z WARN management/server/grpcserver.go:429: failed validating JWT token sent from peer <REDACTED> with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key. Trying again as it may be due to the IdP cache issue
management-1  | 2024-07-14T07:18:22Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
management-1  | 2024-07-14T07:18:22Z WARN management/server/grpcserver.go:429: failed validating JWT token sent from peer <REDACTED> with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key. Trying again as it may be due to the IdP cache issue
management-1  | 2024-07-14T07:18:22Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key
management-1  | 2024-07-14T07:18:22Z WARN management/server/grpcserver.go:429: failed validating JWT token sent from peer <REDACTED> with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key. Trying again as it may be due to the IdP cache issue

iDP Used: Google Workspace

netbird up -F -l debug output:

2024-07-14T10:26:02+03:00 DEBG client/internal/login.go:93: connecting to the Management service <REDACTED>
2024-07-14T10:26:02+03:00 DEBG client/internal/login.go:63: connected to the Management service <REDACTED>
2024-07-14T10:26:03+03:00 DEBG client/internal/login.go:72: peer registration required
2024-07-14T10:26:03+03:00 DEBG client/internal/login.go:122: sending peer registration request to Management Service
2024-07-14T10:26:04+03:00 ERRO client/internal/login.go:126: failed registering peer rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key,00000000-0000-0000-0000-000000000000
Error: foreground login failed: login failed: rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key
Originally created by @mohamed-essam on GitHub (Jul 14, 2024). **Describe the problem** Randomly every few days, `netbird up` fails with `Error: waiting sso login failed with: rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key`, this is fixed by restarting management service **To Reproduce** Unknown **Expected behavior** Connection to be established normally **Are you using NetBird Cloud?** Self-hosted. **NetBird version** 0.28.4 **NetBird status -d output:** N/A **Screenshots** N/A **Additional context** Management server logs: ``` management-1 | 2024-07-14T07:18:21Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key management-1 | 2024-07-14T07:18:21Z WARN management/server/grpcserver.go:429: failed validating JWT token sent from peer <REDACTED> with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key. Trying again as it may be due to the IdP cache issue management-1 | 2024-07-14T07:18:22Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key management-1 | 2024-07-14T07:18:22Z WARN management/server/grpcserver.go:429: failed validating JWT token sent from peer <REDACTED> with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key. Trying again as it may be due to the IdP cache issue management-1 | 2024-07-14T07:18:22Z ERRO management/server/jwtclaims/jwtValidator.go:160: error parsing token: unable to find appropriate key management-1 | 2024-07-14T07:18:22Z WARN management/server/grpcserver.go:429: failed validating JWT token sent from peer <REDACTED> with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key. Trying again as it may be due to the IdP cache issue ``` iDP Used: Google Workspace `netbird up -F -l debug` output: ``` 2024-07-14T10:26:02+03:00 DEBG client/internal/login.go:93: connecting to the Management service <REDACTED> 2024-07-14T10:26:02+03:00 DEBG client/internal/login.go:63: connected to the Management service <REDACTED> 2024-07-14T10:26:03+03:00 DEBG client/internal/login.go:72: peer registration required 2024-07-14T10:26:03+03:00 DEBG client/internal/login.go:122: sending peer registration request to Management Service 2024-07-14T10:26:04+03:00 ERRO client/internal/login.go:126: failed registering peer rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key,00000000-0000-0000-0000-000000000000 Error: foreground login failed: login failed: rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: unable to find appropriate key ```
saavagebueno added the triage-neededself-hosting labels 2025-11-20 05:23:16 -05:00
Author
Owner

@Zwordi commented on GitHub (Jul 14, 2024):

Hi,

I’ m having the same situation and same behavior with latest and G.Workspace.

Thanks for creating the issue.

@Zwordi commented on GitHub (Jul 14, 2024): Hi, I’ m having the same situation and same behavior with latest and G.Workspace. Thanks for creating the issue.
Author
Owner

@braginini commented on GitHub (Jul 15, 2024):

hey @mohamed-essam and @Zwordi
Does this happen on older NetBird versions? Could you please share the generated JWT token contents jwt.io through Slack?

@braginini commented on GitHub (Jul 15, 2024): hey @mohamed-essam and @Zwordi Does this happen on older NetBird versions? Could you please share the generated JWT token contents [jwt.io](jwt.io) through Slack?
Author
Owner

@mohamed-essam commented on GitHub (Jul 15, 2024):

Hello @braginini,

Do you mean client or management versions? As this installation is used by multiple other people within my organization I will be unable to downgrade the server version to test for extended amounts of time.

As for the generated JWT token contents, does that appear in debug logs? Or do I need to do something specific to get that once the error occurs?

I took a quick look in the code and I think the root cause may be a failure in updating the JSONWebKey in https://github.com/netbirdio/netbird/blob/main/management/server/jwtclaims/jwtValidator.go#L108 , I turned on debug logging yesterday and waiting for the issue to occur again to be able to share the debug logs around the time the issue starts.

On a separate topic I believe that line of logging should definitely be a Warn or Error not Debug.

@mohamed-essam commented on GitHub (Jul 15, 2024): Hello @braginini, Do you mean client or management versions? As this installation is used by multiple other people within my organization I will be unable to downgrade the server version to test for extended amounts of time. As for the generated JWT token contents, does that appear in debug logs? Or do I need to do something specific to get that once the error occurs? I took a quick look in the code and I think the root cause may be a failure in updating the JSONWebKey in https://github.com/netbirdio/netbird/blob/main/management/server/jwtclaims/jwtValidator.go#L108 , I turned on debug logging yesterday and waiting for the issue to occur again to be able to share the debug logs around the time the issue starts. On a separate topic I believe that line of logging should definitely be a Warn or Error not Debug.
Author
Owner

@mohamed-essam commented on GitHub (Jul 15, 2024):

Some extra information I forgot to include: This issue is most likely server-side as it caused all SSO clients to be unable to connect (my own client and 4 other personnel were unable to authenticate)

@mohamed-essam commented on GitHub (Jul 15, 2024): Some extra information I forgot to include: This issue is most likely server-side as it caused all SSO clients to be unable to connect (my own client and 4 other personnel were unable to authenticate)
Author
Owner

@mohamed-essam commented on GitHub (Jul 21, 2024):

I found that the config generated by the setup script has HttpConfig.IdpSignKeyRefreshEnabled set to false, changed it to true manually, and will check if it works and report back

Side note: the issue occurred again today, it seems to be occurring almost weekly

@mohamed-essam commented on GitHub (Jul 21, 2024): I found that the config generated by the setup script has `HttpConfig.IdpSignKeyRefreshEnabled` set to `false`, changed it to true manually, and will check if it works and report back Side note: the issue occurred again today, it seems to be occurring almost weekly
Author
Owner

@mohamed-essam commented on GitHub (Jul 29, 2024):

This week no issue occurred, the issue seems to be the setup script disables refreshing idp keys for Google workspace when it should be enabled

@mohamed-essam commented on GitHub (Jul 29, 2024): This week no issue occurred, the issue seems to be the setup script disables refreshing idp keys for Google workspace when it should be enabled
Author
Owner

@a8uhnf commented on GitHub (Sep 7, 2024):

I found that the config generated by the setup script has HttpConfig.IdpSignKeyRefreshEnabled set to false, changed it to true manually, and will check if it works and report back

Side note: the issue occurred again today, it seems to be occurring almost weekly

thanks a lot. seems working for me. IMO, for SSO it should be default behaviour

@a8uhnf commented on GitHub (Sep 7, 2024): > I found that the config generated by the setup script has `HttpConfig.IdpSignKeyRefreshEnabled` set to `false`, changed it to true manually, and will check if it works and report back > > Side note: the issue occurred again today, it seems to be occurring almost weekly thanks a lot. seems working for me. IMO, for SSO it should be default behaviour
Author
Owner

@sammyarmfield commented on GitHub (Sep 25, 2024):

I also was having this same issue on an end user's device. I fixed it by navigating to the management.json file located within the management container and changed IdpSignKeyRefreshEnabled: false to IdpSignKeyRefreshEnabled: true

We are using EntraAD.

@sammyarmfield commented on GitHub (Sep 25, 2024): I also was having this same issue on an end user's device. I fixed it by navigating to the management.json file located within the management container and changed IdpSignKeyRefreshEnabled: false to IdpSignKeyRefreshEnabled: true We are using EntraAD.
Author
Owner

@gecube commented on GitHub (Nov 18, 2024):

Hi!

The same. The issue is that I can't see what is missing in configuration. The error message is ambiguous. I would be happy if additional details would be provided.

2024-11-18T07:34:28Z ERRO [context: HTTP, requestID: f6809b65-b37e-48b9-930a-876464efed58] management/server/jwtclaims/jwtValidator.go:161: error parsing token: unable to find appropriate key
2024-11-18T07:34:28Z ERRO [context: HTTP, requestID: f6809b65-b37e-48b9-930a-876464efed58] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: unable to find appropriate key
2024-11-18T07:34:28Z ERRO [context: HTTP, requestID: f6809b65-b37e-48b9-930a-876464efed58] management/server/http/util/util.go:81: got a handler error: token invalid
2024-11-18T07:34:28Z ERRO [context: HTTP, requestID: f6809b65-b37e-48b9-930a-876464efed58] management/server/telemetry/http_api_metrics.go:191: HTTP response f6809b65-b37e-48b9-930a-876464efed58: GET /api/users status 401
2024-11-18T07:34:36Z ERRO [context: HTTP, requestID: 42c07c18-5b2b-4f8e-be89-75b0113cadab] management/server/jwtclaims/jwtValidator.go:161: error parsing token: unable to find appropriate key
2024-11-18T07:34:36Z ERRO [context: HTTP, requestID: 42c07c18-5b2b-4f8e-be89-75b0113cadab] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: unable to find appropriate key
2024-11-18T07:34:36Z ERRO [context: HTTP, requestID: 42c07c18-5b2b-4f8e-be89-75b0113cadab] management/server/http/util/util.go:81: got a handler error: token invalid
2024-11-18T07:34:36Z ERRO [context: HTTP, requestID: 42c07c18-5b2b-4f8e-be89-75b0113cadab] management/server/telemetry/http_api_metrics.go:191: HTTP response 42c07c18-5b2b-4f8e-be89-75b0113cadab: GET /api/users status 401
2024-11-18T07:34:38Z ERRO [context: HTTP, requestID: 2a36a45b-17ba-4d6c-886a-b7c8de4156fa] management/server/jwtclaims/jwtValidator.go:161: error parsing token: unable to find appropriate key
2024-11-18T07:34:38Z ERRO [context: HTTP, requestID: 2a36a45b-17ba-4d6c-886a-b7c8de4156fa] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: unable to find appropriate key
2024-11-18T07:34:38Z ERRO [context: HTTP, requestID: 2a36a45b-17ba-4d6c-886a-b7c8de4156fa] management/server/http/util/util.go:81: got a handler error: token invalid
@gecube commented on GitHub (Nov 18, 2024): Hi! The same. The issue is that I can't see what is missing in configuration. The error message is ambiguous. I would be happy if additional details would be provided. ``` 2024-11-18T07:34:28Z ERRO [context: HTTP, requestID: f6809b65-b37e-48b9-930a-876464efed58] management/server/jwtclaims/jwtValidator.go:161: error parsing token: unable to find appropriate key 2024-11-18T07:34:28Z ERRO [context: HTTP, requestID: f6809b65-b37e-48b9-930a-876464efed58] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: unable to find appropriate key 2024-11-18T07:34:28Z ERRO [context: HTTP, requestID: f6809b65-b37e-48b9-930a-876464efed58] management/server/http/util/util.go:81: got a handler error: token invalid 2024-11-18T07:34:28Z ERRO [context: HTTP, requestID: f6809b65-b37e-48b9-930a-876464efed58] management/server/telemetry/http_api_metrics.go:191: HTTP response f6809b65-b37e-48b9-930a-876464efed58: GET /api/users status 401 2024-11-18T07:34:36Z ERRO [context: HTTP, requestID: 42c07c18-5b2b-4f8e-be89-75b0113cadab] management/server/jwtclaims/jwtValidator.go:161: error parsing token: unable to find appropriate key 2024-11-18T07:34:36Z ERRO [context: HTTP, requestID: 42c07c18-5b2b-4f8e-be89-75b0113cadab] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: unable to find appropriate key 2024-11-18T07:34:36Z ERRO [context: HTTP, requestID: 42c07c18-5b2b-4f8e-be89-75b0113cadab] management/server/http/util/util.go:81: got a handler error: token invalid 2024-11-18T07:34:36Z ERRO [context: HTTP, requestID: 42c07c18-5b2b-4f8e-be89-75b0113cadab] management/server/telemetry/http_api_metrics.go:191: HTTP response 42c07c18-5b2b-4f8e-be89-75b0113cadab: GET /api/users status 401 2024-11-18T07:34:38Z ERRO [context: HTTP, requestID: 2a36a45b-17ba-4d6c-886a-b7c8de4156fa] management/server/jwtclaims/jwtValidator.go:161: error parsing token: unable to find appropriate key 2024-11-18T07:34:38Z ERRO [context: HTTP, requestID: 2a36a45b-17ba-4d6c-886a-b7c8de4156fa] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: unable to find appropriate key 2024-11-18T07:34:38Z ERRO [context: HTTP, requestID: 2a36a45b-17ba-4d6c-886a-b7c8de4156fa] management/server/http/util/util.go:81: got a handler error: token invalid ```
Author
Owner

@varrcan commented on GitHub (Nov 21, 2024):

A similar error occurs a couple of times a week. I set IdpSignKeyRefreshEnabled to true, but this did not solve the problem. SSO is integrated via Zitadel.
The problem is solved only by restarting.

2024-11-21T06:33:32Z ERRO [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:128: getPublicKey error: unable to find appropriate key
2024-11-21T06:33:32Z ERRO [context: HTTP, requestID: 9faba45a-85fc-49db-858c-74144870d756] management/server/jwtclaims/jwtValidator.go:168: error parsing token: unable to find appropriate key
2024-11-21T06:33:32Z ERRO [context: HTTP, requestID: 9faba45a-85fc-49db-858c-74144870d756] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: error parsing token: unable to find appropriate key
2024-11-21T06:33:32Z ERRO [context: HTTP, requestID: 9faba45a-85fc-49db-858c-74144870d756] management/server/http/util/util.go:81: got a handler error: token invalid
2024-11-21T06:33:32Z ERRO [context: HTTP, requestID: 9faba45a-85fc-49db-858c-74144870d756] management/server/telemetry/http_api_metrics.go:168: HTTP response 9faba45a-85fc-49db-858c-74144870d756: GET /api/users status 401
@varrcan commented on GitHub (Nov 21, 2024): A similar error occurs a couple of times a week. I set `IdpSignKeyRefreshEnabled` to `true`, but this did not solve the problem. SSO is integrated via Zitadel. The problem is solved only by restarting. ``` 2024-11-21T06:33:32Z ERRO [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:128: getPublicKey error: unable to find appropriate key 2024-11-21T06:33:32Z ERRO [context: HTTP, requestID: 9faba45a-85fc-49db-858c-74144870d756] management/server/jwtclaims/jwtValidator.go:168: error parsing token: unable to find appropriate key 2024-11-21T06:33:32Z ERRO [context: HTTP, requestID: 9faba45a-85fc-49db-858c-74144870d756] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: error parsing token: unable to find appropriate key 2024-11-21T06:33:32Z ERRO [context: HTTP, requestID: 9faba45a-85fc-49db-858c-74144870d756] management/server/http/util/util.go:81: got a handler error: token invalid 2024-11-21T06:33:32Z ERRO [context: HTTP, requestID: 9faba45a-85fc-49db-858c-74144870d756] management/server/telemetry/http_api_metrics.go:168: HTTP response 9faba45a-85fc-49db-858c-74144870d756: GET /api/users status 401 ```
Author
Owner

@gecube commented on GitHub (Nov 21, 2024):

It helps to restart management server, but only for some time. From the experiment I understood when I restart dex (which I use like a source of users), then this error occurs.

@gecube commented on GitHub (Nov 21, 2024): It helps to restart management server, but only for some time. From the experiment I understood when I restart dex (which I use like a source of users), then this error occurs.
Author
Owner

@nazarewk commented on GitHub (Apr 28, 2025):

Hello @mohamed-essam,

We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.

Could you please confirm if the issue is still there?

We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.

Thanks for your contribution to improving the project!

@nazarewk commented on GitHub (Apr 28, 2025): Hello @mohamed-essam, We're currently reviewing our open issues and would like to verify if this problem still exists in the [latest NetBird version](https://github.com/netbirdio/netbird/releases). Could you please confirm if the issue is still there? We may close this issue temporarily if we don't hear back from you within **2 weeks**, but feel free to reopen it with updated information. Thanks for your contribution to improving the project!
Author
Owner

@altyntsevlexus commented on GitHub (May 2, 2025):

@nazarewk
Faced this issue today, can't resolve it.

@altyntsevlexus commented on GitHub (May 2, 2025): @nazarewk Faced this issue today, can't resolve it.
Author
Owner

@Coler-e commented on GitHub (Jul 12, 2025):

Hi @nazarewk , bumping this up as I also faced this issue today and do semi frequently. So far I tinker with everything I can like restart the management server multiple times and complete uninstall and reinstall of my client and it seems to work again.

I haven't been able to narrow down what actually does something yet, would happily provide any information required to get rid of this pain point!

@Coler-e commented on GitHub (Jul 12, 2025): Hi @nazarewk , bumping this up as I also faced this issue today and do semi frequently. So far I tinker with everything I can like restart the management server multiple times and complete uninstall and reinstall of my client and it seems to work again. I haven't been able to narrow down what actually does something yet, would happily provide any information required to get rid of this pain point!
Author
Owner

@Coler-e commented on GitHub (Jul 12, 2025):

Hmmm my issue does seem different from the original however :

2025-07-12T09:14:16Z ERRO [context: GRPC, requestID: 22459455-28b5-4eec-b02a-1c265fecf5fd, accountID: d167g3d8plqs73em4fug, peerID: x+nxM7XDXuxmtUAvdl8+0wnNNuopsz2qZbkBjdN3Q24=] management/server/auth/jwt/validator.go:161: token could not be parsed: Token is expired
2025-07-12T09:14:16Z WARN [context: GRPC, requestID: 22459455-28b5-4eec-b02a-1c265fecf5fd, accountID: d167g3d8plqs73em4fug, peerID: x+nxM7XDXuxmtUAvdl8+0wnNNuopsz2qZbkBjdN3Q24=] management/server/grpcserver.go:553: failed validating JWT token sent from peer x+nxM7XDXuxmtUAvdl8+0wnNNuopsz2qZbkBjdN3Q24= with error rpc error: code = InvalidArgument desc = invalid jwt token, err: token could not be parsed: Token is expired. Trying again as it may be due to the IdP cache issue
@Coler-e commented on GitHub (Jul 12, 2025): Hmmm my issue does seem different from the original however : ``` 2025-07-12T09:14:16Z ERRO [context: GRPC, requestID: 22459455-28b5-4eec-b02a-1c265fecf5fd, accountID: d167g3d8plqs73em4fug, peerID: x+nxM7XDXuxmtUAvdl8+0wnNNuopsz2qZbkBjdN3Q24=] management/server/auth/jwt/validator.go:161: token could not be parsed: Token is expired 2025-07-12T09:14:16Z WARN [context: GRPC, requestID: 22459455-28b5-4eec-b02a-1c265fecf5fd, accountID: d167g3d8plqs73em4fug, peerID: x+nxM7XDXuxmtUAvdl8+0wnNNuopsz2qZbkBjdN3Q24=] management/server/grpcserver.go:553: failed validating JWT token sent from peer x+nxM7XDXuxmtUAvdl8+0wnNNuopsz2qZbkBjdN3Q24= with error rpc error: code = InvalidArgument desc = invalid jwt token, err: token could not be parsed: Token is expired. Trying again as it may be due to the IdP cache issue ```
Author
Owner

@nazarewk commented on GitHub (Jul 15, 2025):

@Coler-e are you on Entra ID? please see https://github.com/netbirdio/netbird/issues/4143#issuecomment-3073535425

@nazarewk commented on GitHub (Jul 15, 2025): @Coler-e are you on Entra ID? please see https://github.com/netbirdio/netbird/issues/4143#issuecomment-3073535425
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1061