Add support for rotating signing keys on expiry #313

Closed
opened 2025-11-20 05:09:28 -05:00 by saavagebueno · 3 comments
Owner

Originally created by @Thunderbottom on GitHub (Apr 13, 2023).

Describe the problem
Currently, signing keys are loaded on management startup, and are never refreshed on key expiry. IdPs often rotate these keys at a set interval for added security.

To Reproduce
Steps to reproduce the behavior:

  1. Set up an IdP that rotates signing key at set intervals, eg. dexIdP
  2. Integrate the IdP with Netbird and start the Netbird management server, this should load the signing key from the IdP on startup.
  3. Wait for the signing key to expire, in the case of dexIdP, the default expiry is set to 6h.
  4. Try to login to the management service and see:
    image

Additional management service logs:

infrastructure_files-management-1  | 2023-04-06T12:55:11Z DEBG management/server/jwtclaims/jwtValidator.go:135: error parsing token: unable to find appropriate key
infrastructure_files-management-1  | 2023-04-06T12:55:11Z DEBG management/server/http/middleware/auth_middleware.go:60: Error when validating JWT claims: Error parsing token: unable to find appropriate key
infrastructure_files-management-1  | 2023-04-06T12:55:11Z ERRO management/server/telemetry/http_api_metrics.go:150: HTTP response 4198917813: GET /api/users status 401
infrastructure_files-management-1  | 2023-04-06T12:55:11Z DEBG management/server/jwtclaims/jwtValidator.go:135: error parsing token: unable to find appropriate key
infrastructure_files-management-1  | 2023-04-06T12:55:11Z DEBG management/server/http/middleware/auth_middleware.go:60: Error when validating JWT claims: Error parsing token: unable to find appropriate key
infrastructure_files-management-1  | 2023-04-06T12:55:11Z ERRO management/server/telemetry/http_api_metrics.go:150: HTTP response 1761936853: GET /api/users status 401
infrastructure_files-management-1  | 2023-04-06T12:55:11Z DEBG management/server/jwtclaims/jwtValidator.go:135: error parsing token: unable to find appropriate key
infrastructure_files-management-1  | 2023-04-06T12:55:11Z DEBG management/server/http/middleware/auth_middleware.go:60: Error when validating JWT claims: Error parsing token: unable to find appropriate key
infrastructure_files-management-1  | 2023-04-06T12:55:11Z ERRO management/server/telemetry/http_api_metrics.go:150: HTTP response 1177296465: GET /api/groups status 401

Expected behavior
Netbird reloads the signing key after IdP signing key rotation.

Originally created by @Thunderbottom on GitHub (Apr 13, 2023). **Describe the problem** Currently, signing keys are loaded on management startup, and are never refreshed on key expiry. IdPs often rotate these keys at a set interval for added security. **To Reproduce** Steps to reproduce the behavior: 1. Set up an IdP that rotates signing key at set intervals, eg. [dexIdP](https://dexidp.io) 2. Integrate the IdP with Netbird and start the Netbird management server, this should load the signing key from the IdP on startup. 3. Wait for the signing key to expire, in the case of dexIdP, the default expiry is set to `6h`. 4. Try to login to the management service and see: ![image](https://user-images.githubusercontent.com/11243138/231723454-9e3fa3d3-ae2f-4a91-87b0-c47174b55c49.png) **Additional management service logs:** ``` infrastructure_files-management-1 | 2023-04-06T12:55:11Z DEBG management/server/jwtclaims/jwtValidator.go:135: error parsing token: unable to find appropriate key infrastructure_files-management-1 | 2023-04-06T12:55:11Z DEBG management/server/http/middleware/auth_middleware.go:60: Error when validating JWT claims: Error parsing token: unable to find appropriate key infrastructure_files-management-1 | 2023-04-06T12:55:11Z ERRO management/server/telemetry/http_api_metrics.go:150: HTTP response 4198917813: GET /api/users status 401 infrastructure_files-management-1 | 2023-04-06T12:55:11Z DEBG management/server/jwtclaims/jwtValidator.go:135: error parsing token: unable to find appropriate key infrastructure_files-management-1 | 2023-04-06T12:55:11Z DEBG management/server/http/middleware/auth_middleware.go:60: Error when validating JWT claims: Error parsing token: unable to find appropriate key infrastructure_files-management-1 | 2023-04-06T12:55:11Z ERRO management/server/telemetry/http_api_metrics.go:150: HTTP response 1761936853: GET /api/users status 401 infrastructure_files-management-1 | 2023-04-06T12:55:11Z DEBG management/server/jwtclaims/jwtValidator.go:135: error parsing token: unable to find appropriate key infrastructure_files-management-1 | 2023-04-06T12:55:11Z DEBG management/server/http/middleware/auth_middleware.go:60: Error when validating JWT claims: Error parsing token: unable to find appropriate key infrastructure_files-management-1 | 2023-04-06T12:55:11Z ERRO management/server/telemetry/http_api_metrics.go:150: HTTP response 1177296465: GET /api/groups status 401 ``` **Expected behavior** Netbird reloads the signing key after IdP signing key rotation.
Author
Owner

@Thunderbottom commented on GitHub (Apr 13, 2023):

A hacky workaround right now is to extend the signing key expiry period (if possible) at the IdP level to a large number, eg. 180d, or disable the expiry altogether.

@Thunderbottom commented on GitHub (Apr 13, 2023): A hacky workaround right now is to extend the signing key expiry period (if possible) at the IdP level to a large number, eg. `180d`, or disable the expiry altogether.
Author
Owner

@mlsmaycon commented on GitHub (Jun 16, 2023):

Support was added in PR #808; you can set it with the flag: --idp-sign-key-refresh-enabled

@mlsmaycon commented on GitHub (Jun 16, 2023): Support was added in PR #808; you can set it with the flag: `--idp-sign-key-refresh-enabled`
Author
Owner

@Fantu commented on GitHub (Oct 9, 2023):

hi, today is the second time (the previous was days ago) I have this issue (or atleast seems from same error logs) and I need to reboot the netbird server to "solve" because with page refresh don't solve and don't show the login page
after reboot return working and also don't require google auth again (I use google auth setted following https://docs.netbird.io/selfhosted/identity-providers#google-workspace)
this setting should solves?
if yes how to set it?
in setup.env I don't found it (I suppose not added looking the PR)
but I also not sure how to add it to management.json (after the configure), looking https://github.com/netbirdio/netbird/pull/808/files config.HttpConfig.IdpSignKeyRefreshEnabled = idpSignKeyRefreshEnabled I suppose i should add:

"HttpConfig": {
        ...
        "IdpSignKeyRefreshEnabled": true
    },

is it correct? I suppose add it in default management.json and probably a variable for it also in setup.env can be useful to make easier/faster change it

@Fantu commented on GitHub (Oct 9, 2023): hi, today is the second time (the previous was days ago) I have this issue (or atleast seems from same error logs) and I need to reboot the netbird server to "solve" because with page refresh don't solve and don't show the login page after reboot return working and also don't require google auth again (I use google auth setted following https://docs.netbird.io/selfhosted/identity-providers#google-workspace) this setting should solves? if yes how to set it? in setup.env I don't found it (I suppose not added looking the PR) but I also not sure how to add it to management.json (after the configure), looking https://github.com/netbirdio/netbird/pull/808/files `config.HttpConfig.IdpSignKeyRefreshEnabled = idpSignKeyRefreshEnabled` I suppose i should add: ``` "HttpConfig": { ... "IdpSignKeyRefreshEnabled": true }, ``` is it correct? I suppose add it in default management.json and probably a variable for it also in setup.env can be useful to make easier/faster change it
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#313