Error: token invalid 401 #388

Closed
opened 2025-11-20 05:10:38 -05:00 by saavagebueno · 16 comments
Owner

Originally created by @UncleJ4ck on GitHub (Jun 29, 2023).

Describe the problem

After upgrading from v0.14.4 to v0.21.3 I am unable to access the dashboard and get this error: image

I suspect this is related to the new IdP changes but I am not sure what.

To Reproduce Steps to reproduce the behavior:

1. Pull latest `https://github.com/netbirdio/netbird.git`

2. Copy `infrastructure_files/setup.env.template` to `infrastructure_files/setup.env` (overwrite existing file from old version)

3. Fill in correct values noting these changes:
   
   1. Generate a client secret for the `netbird-client` in Keycloak and `NETBIRD_AUTH_CLIENT_SECRET`
   2. Create a new client `netbird-backend` and follow the steps here: https://docs.netbird.io/selfhosted/identity-providers#step-8-create-a-net-bird-backend-client

4. Re-run `./configure.sh`

5. Run `docker compose pull && docker compose down && docker compose up -d`

Expected behavior

Things work as they used to

Additional context

I tried setting NETBIRD_MGMT_IDP="none" and the dashboard loads but no clients can connect with this error:

netbird up --management-url https://netbird.XXX:33073 --admin-url https://netbird.XXX:443
Error: login failed: rpc error: code = NotFound desc = no SSO provider returned from management. If you are using hosting Netbird see documentation at https://github.com/netbirdio/netbird/tree/main/management for details

i have the same error with a different return code:

2023-06-29-02 24 15-screenshot(1)

Request failed with status code 401. Please refresh the page if the issue continues.
token invalid

docker management logs:


2023-06-28T23:33:07Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
2023-06-28T23:33:07Z INFO management/server/account.go:638: single account mode enabled, accounts number 0
2023-06-28T23:33:07Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
2023-06-28T23:33:07Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443
2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401

that's my setup.env:

## example file, you can copy this file to setup.env and update its values
##
# Dashboard domain. e.g. app.mydomain.com
NETBIRD_DOMAIN="XXXXX"

# -------------------------------------------
# OIDC
#  e.g., https://example.eu.auth0.com/.well-known/openid-configuration
# ------------------------------------------
NETBIRD_AUTH_DEVICE_AUTH_PROVIDER=hosted
NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT="https://XXXXX/auth/realms/netbird/.well-known/openid-configuration"
NETBIRD_USE_AUTH0=false
NETBIRD_AUTH_CLIENT_ID="netbird-client"
NETBIRD_AUTH_AUDIENCE="netbird-client"
NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID="netbird-client"
NETBIRD_MGMT_IDP="keycloak"
NETBIRD_IDP_MGMT_CLIENT_ID="netbird-backend"
NETBIRD_IDP_MGMT_CLIENT_SECRET="XXXXX"
NETBIRD_IDP_MGMT_EXTRA_ADMIN_ENDPOINT="https://XXXXX/auth/realms/netbird"

# -------------------------------------------
# Letsencrypt
# -------------------------------------------
# Disable letsencrypt
#  if disabled, cannot use HTTPS anymore and requires setting up a reverse-proxy to do it instead
# e.g. hello@mydomain.com
NETBIRD_LETSENCRYPT_EMAIL="XXXXX"
# -------------------------------------------
# Extra settings
# -------------------------------------------
# Disable anonymous metrics collection, see more information at https://netbird.io/docs/FAQ/metrics-collection
NETBIRD_DISABLE_ANONYMOUS_METRICS=false
# DNS DOMAIN configures the domain name used for peer resolution. By default it is netbird.selfhosted
NETBIRD_MGMT_DNS_DOMAIN=netbird.selfhosted

when i try to connect in the client side:

sudo netbird up --management-url https://XXXXXXXX:33073
Error: login failed: rpc error: code = NotFound desc = no SSO provider returned from management. If you are using hosting Netbird see documentation at https://github.com/netbirdio/netbird/tree/main/management for details

for the netbird-backend, i didn't do the management steps because, the services accounts roles tab is missing
2023-06-29-01 09 26-screenshot

new management logs:

2023-06-28T23:33:07Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
2023-06-28T23:33:07Z INFO management/server/account.go:638: single account mode enabled, accounts number 0
2023-06-28T23:33:07Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
2023-06-28T23:33:07Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443
2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3101701186: GET /api/users status 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 335186381: GET /api/peers status 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 449313820: GET /api/groups status 401
2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 780097130: GET /api/users status 401
2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3241716118: GET /api/peers status 401
2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 376855912: GET /api/users status 401
2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3584425485: GET /api/groups status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 997841889: GET /api/users status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3647231711: GET /api/users status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2307183365: GET /api/peers status 401
2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 418149988: GET /api/groups status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3370837239: GET /api/groups status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2633305945: GET /api/users status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1883989961: GET /api/users?service_user=false status 401
2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1727792694: GET /api/groups status 401
2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 127726435: GET /api/users status 401
2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1972559671: GET /api/groups status 401
2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3935102461: GET /api/peers status 401
2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 549383702: GET /api/peers status 401
2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3104191757: GET /api/users status 401
2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3006261035: GET /api/groups status 401
2023-06-29T00:30:18Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w=
2023-06-29T00:30:21Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w=
2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2288542812: GET /api/users status 401
2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1587588152: GET /api/peers status 401
2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 584506530: GET /api/groups status 401
2023-06-29T00:33:55Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w=

Originally posted by @UncleJ4ck in https://github.com/netbirdio/netbird/issues/959#issuecomment-1612271029

Originally created by @UncleJ4ck on GitHub (Jun 29, 2023). > **Describe the problem** > > After upgrading from v0.14.4 to v0.21.3 I am unable to access the dashboard and get this error: <img alt="image" width="1487" src="https://user-images.githubusercontent.com/38473816/246527112-52759eae-c926-4b12-8172-a202c5a3af8a.png"> > > I suspect this is related to the new IdP changes but I am not sure what. > > **To Reproduce** Steps to reproduce the behavior: > > 1. Pull latest `https://github.com/netbirdio/netbird.git` > > 2. Copy `infrastructure_files/setup.env.template` to `infrastructure_files/setup.env` (overwrite existing file from old version) > > 3. Fill in correct values noting these changes: > > 1. Generate a client secret for the `netbird-client` in Keycloak and `NETBIRD_AUTH_CLIENT_SECRET` > 2. Create a new client `netbird-backend` and follow the steps here: https://docs.netbird.io/selfhosted/identity-providers#step-8-create-a-net-bird-backend-client > > 4. Re-run `./configure.sh` > > 5. Run `docker compose pull && docker compose down && docker compose up -d` > > > **Expected behavior** > > Things work as they used to > > **Additional context** > > I tried setting `NETBIRD_MGMT_IDP="none"` and the dashboard loads but no clients can connect with this error: > > ``` > netbird up --management-url https://netbird.XXX:33073 --admin-url https://netbird.XXX:443 > Error: login failed: rpc error: code = NotFound desc = no SSO provider returned from management. If you are using hosting Netbird see documentation at https://github.com/netbirdio/netbird/tree/main/management for details > ``` i have the same error with a different return code: ![2023-06-29-02 24 15-screenshot(1)](https://github.com/netbirdio/netbird/assets/52085661/fb8aefc4-eb04-4d1f-bbef-651e801388e5) ``` Request failed with status code 401. Please refresh the page if the issue continues. token invalid ``` docker management logs: ``` 2023-06-28T23:33:07Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081 2023-06-28T23:33:07Z INFO management/server/account.go:638: single account mode enabled, accounts number 0 2023-06-28T23:33:07Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073 2023-06-28T23:33:07Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443 2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401 ``` that's my setup.env: ``` ## example file, you can copy this file to setup.env and update its values ## # Dashboard domain. e.g. app.mydomain.com NETBIRD_DOMAIN="XXXXX" # ------------------------------------------- # OIDC # e.g., https://example.eu.auth0.com/.well-known/openid-configuration # ------------------------------------------ NETBIRD_AUTH_DEVICE_AUTH_PROVIDER=hosted NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT="https://XXXXX/auth/realms/netbird/.well-known/openid-configuration" NETBIRD_USE_AUTH0=false NETBIRD_AUTH_CLIENT_ID="netbird-client" NETBIRD_AUTH_AUDIENCE="netbird-client" NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID="netbird-client" NETBIRD_MGMT_IDP="keycloak" NETBIRD_IDP_MGMT_CLIENT_ID="netbird-backend" NETBIRD_IDP_MGMT_CLIENT_SECRET="XXXXX" NETBIRD_IDP_MGMT_EXTRA_ADMIN_ENDPOINT="https://XXXXX/auth/realms/netbird" # ------------------------------------------- # Letsencrypt # ------------------------------------------- # Disable letsencrypt # if disabled, cannot use HTTPS anymore and requires setting up a reverse-proxy to do it instead # e.g. hello@mydomain.com NETBIRD_LETSENCRYPT_EMAIL="XXXXX" # ------------------------------------------- # Extra settings # ------------------------------------------- # Disable anonymous metrics collection, see more information at https://netbird.io/docs/FAQ/metrics-collection NETBIRD_DISABLE_ANONYMOUS_METRICS=false # DNS DOMAIN configures the domain name used for peer resolution. By default it is netbird.selfhosted NETBIRD_MGMT_DNS_DOMAIN=netbird.selfhosted ``` when i try to connect in the client side: ``` sudo netbird up --management-url https://XXXXXXXX:33073 Error: login failed: rpc error: code = NotFound desc = no SSO provider returned from management. If you are using hosting Netbird see documentation at https://github.com/netbirdio/netbird/tree/main/management for details ``` for the netbird-backend, i didn't do the management steps because, the ```services accounts roles``` tab is missing ![2023-06-29-01 09 26-screenshot](https://github.com/netbirdio/netbird/assets/52085661/5e12e1b0-0711-4deb-a49b-21cc1c92c92f) new management logs: ``` 2023-06-28T23:33:07Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081 2023-06-28T23:33:07Z INFO management/server/account.go:638: single account mode enabled, accounts number 0 2023-06-28T23:33:07Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073 2023-06-28T23:33:07Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443 2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401 2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3101701186: GET /api/users status 401 2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 335186381: GET /api/peers status 401 2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 449313820: GET /api/groups status 401 2023-06-29T00:19:15Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 780097130: GET /api/users status 401 2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3241716118: GET /api/peers status 401 2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 376855912: GET /api/users status 401 2023-06-29T00:23:46Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3584425485: GET /api/groups status 401 2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 997841889: GET /api/users status 401 2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3647231711: GET /api/users status 401 2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2307183365: GET /api/peers status 401 2023-06-29T00:23:50Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 418149988: GET /api/groups status 401 2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3370837239: GET /api/groups status 401 2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2633305945: GET /api/users status 401 2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1883989961: GET /api/users?service_user=false status 401 2023-06-29T00:24:03Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1727792694: GET /api/groups status 401 2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 127726435: GET /api/users status 401 2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1972559671: GET /api/groups status 401 2023-06-29T00:24:04Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3935102461: GET /api/peers status 401 2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 549383702: GET /api/peers status 401 2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3104191757: GET /api/users status 401 2023-06-29T00:28:21Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 3006261035: GET /api/groups status 401 2023-06-29T00:30:18Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w= 2023-06-29T00:30:21Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w= 2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2288542812: GET /api/users status 401 2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1587588152: GET /api/peers status 401 2023-06-29T00:32:52Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 584506530: GET /api/groups status 401 2023-06-29T00:33:55Z WARN management/server/grpcserver.go:322: failed logging in peer sajM8Azu99EOT4XkyUrxnVzlk9xMn5pL1tkkNCgT72w= ```` _Originally posted by @UncleJ4ck in https://github.com/netbirdio/netbird/issues/959#issuecomment-1612271029_
Author
Owner

@bcmmbaga commented on GitHub (Jun 29, 2023):

@UncleJ4ck It seems that there was a warning in the logs regarding cache warming up failure, specifically due to an error with obtaining the Keycloak token. The error message indicated a status code of 401.

2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401

It appears that the configuration of your netbird-backend client may be incorrect. To address this issue, I recommend following the steps outlined in the documentation at https://docs.netbird.io/selfhosted/identity-providers#keycloak to ensure it is properly configured.

@bcmmbaga commented on GitHub (Jun 29, 2023): @UncleJ4ck It seems that there was a warning in the logs regarding cache warming up failure, specifically due to an error with obtaining the Keycloak token. The error message indicated a status code of 401. `2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401` It appears that the configuration of your `netbird-backend` client may be incorrect. To address this issue, I recommend following the steps outlined in the documentation at https://docs.netbird.io/selfhosted/identity-providers#keycloak to ensure it is properly configured.
Author
Owner

@UncleJ4ck commented on GitHub (Jun 29, 2023):

i fixed the error, because there was no service accounts roles but after that the error 2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401 disappeared as you can see here: https://github.com/netbirdio/netbird/issues/959#issuecomment-1612278033

i even restarted the keycloak server and i followed the steps word by word, im sure there's no error or any misconfiguration in keycloak because i reconfigured keycloak 2 times with netbird

@UncleJ4ck commented on GitHub (Jun 29, 2023): i fixed the error, because there was no service accounts roles but after that the error ```2023-06-28T23:33:11Z WARN management/server/account.go:674: failed warming up cache due to error: unable to get keycloak token, statusCode 401``` disappeared as you can see here: https://github.com/netbirdio/netbird/issues/959#issuecomment-1612278033 i even restarted the keycloak server and i followed the steps word by word, im sure there's no error or any misconfiguration in keycloak because i reconfigured keycloak 2 times with netbird
Author
Owner

@Akruidenberg commented on GitHub (Jul 2, 2023):

I've got the same error with Authentik. Running with a fresh install.

Schermafdruk van 2023-07-02 14-24-50

@Akruidenberg commented on GitHub (Jul 2, 2023): I've got the same error with Authentik. Running with a fresh install. ![Schermafdruk van 2023-07-02 14-24-50](https://github.com/netbirdio/netbird/assets/18529293/75e7f92e-597e-47cb-b3ec-d27c6aad41a5)
Author
Owner

@mlsmaycon commented on GitHub (Jul 3, 2023):

@Akruidenberg @UncleJ4ck can you confirm that you have added the Scopes variable to your setup.env files?

Keycloak

NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api"

Authentik

NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api"
@mlsmaycon commented on GitHub (Jul 3, 2023): @Akruidenberg @UncleJ4ck can you confirm that you have added the Scopes variable to your setup.env files? **Keycloak** ```shell NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api" ``` **Authentik** ```shell NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email offline_access api" ```
Author
Owner

@UncleJ4ck commented on GitHub (Jul 3, 2023):

Yes i confirm

@UncleJ4ck commented on GitHub (Jul 3, 2023): Yes i confirm
Author
Owner

@Akruidenberg commented on GitHub (Jul 3, 2023):

Yes.

@Akruidenberg commented on GitHub (Jul 3, 2023): Yes.
Author
Owner

@thijsa commented on GitHub (Jul 4, 2023):

I also encountered this issue using authentik on a fresh install. I fixed the issue by setting a JWKS signing key in authentik under providers -> netbird -> protocol settings -> signing key. You may need to reconfigure / restart netbird.

@thijsa commented on GitHub (Jul 4, 2023): I also encountered this issue using authentik on a fresh install. I fixed the issue by setting a JWKS signing key in authentik under providers -> netbird -> protocol settings -> signing key. You may need to reconfigure / restart netbird.
Author
Owner

@hadleyrich commented on GitHub (Jul 4, 2023):

I've been seeing this issue with Zitadel IDP behind Traefik since I set up the install a few versions ago and up until current release.

I do not have the cache warming up error from OP, management starts as so:

netbird-management | time="2023-07-04T00:39:15Z" level=info msg="loading OIDC configuration from the provided IDP configuration endpoint https://zitadel.DOMAIN.nz/.well-known/openid-configuration"
netbird-management | time="2023-07-04T00:39:16Z" level=info msg="loaded OIDC configuration from the provided IDP configuration endpoint: https://zitadel.DOMAIN.nz/.well-known/openid-configuration"
netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding HttpConfig.AuthIssuer with a new value https://zitadel.DOMAIN.nz, previously configured value: https://zitadel.DOMAIN.nz"
netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://zitadel.DOMAIN.nz/oauth/v2/keys, previously configured value: https://zitadel.DOMAIN.nz/oauth/v2/keys"
netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://zitadel.DOMAIN.nz/oauth/v2/token, previously configured value: https://zitadel.DOMAIN.nz/oauth/v2/token"
netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://zitadel.DOMAIN.nz/oauth/v2/device_authorization, previously configured value: https://zitadel.DOMAIN.nz/oauth/v2/device_authorization"
netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: zitadel.DOMAIN.nz, previously configured value: "
netbird-management | 2023-07-04T00:39:16Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
netbird-management | 2023-07-04T00:39:16Z INFO management/server/account.go:671: single account mode enabled, accounts number 1
netbird-management | 2023-07-04T00:39:16Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
netbird-management | 2023-07-04T00:39:16Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443
netbird-management | 2023-07-04T00:39:19Z INFO management/server/account.go:858: warmed up IDP cache with 1 entries
netbird-management | 2023-07-04T00:39:38Z INFO management/server/account.go:1256: overriding JWT Domain and DomainCategory claims since single account mode is enabled
netbird-management | 2023-07-04T00:39:38Z INFO management/server/account.go:1256: overriding JWT Domain and DomainCategory claims since single account mode is enabled

The dashboard will always log in correctly and management actually responds as you can see above for some hours after restarting management, I'm guessing until token expiration. After that happens management starts responding with 401 as so:

netbird-management | 2023-07-04T23:38:16Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 125147462: GET /api/peers status 401
netbird-management | 2023-07-04T23:38:16Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1167195797: GET /api/groups status 401
netbird-management | 2023-07-04T23:38:29Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2431638793: GET /api/users status 401

I have been messing with config files over several versions and weeks to try and debug this further, I had a feeling that it was something to do with the token being invalid due to a mismatch in URL vs. URL:443 or something but I can't seem to figure that out and I'm not sure if that's the issue at all.

EDIT: Actually I just came across #806 and #808 which I think is likely to be my issue. I've recreated my containers now and will test that.

@hadleyrich commented on GitHub (Jul 4, 2023): I've been seeing this issue with Zitadel IDP behind Traefik since I set up the install a few versions ago and up until current release. I do not have the cache warming up error from OP, management starts as so: > netbird-management | time="2023-07-04T00:39:15Z" level=info msg="loading OIDC configuration from the provided IDP configuration endpoint https://zitadel.DOMAIN.nz/.well-known/openid-configuration" > netbird-management | time="2023-07-04T00:39:16Z" level=info msg="loaded OIDC configuration from the provided IDP configuration endpoint: https://zitadel.DOMAIN.nz/.well-known/openid-configuration" > netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding HttpConfig.AuthIssuer with a new value https://zitadel.DOMAIN.nz, previously configured value: https://zitadel.DOMAIN.nz" > netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://zitadel.DOMAIN.nz/oauth/v2/keys, previously configured value: https://zitadel.DOMAIN.nz/oauth/v2/keys" > netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://zitadel.DOMAIN.nz/oauth/v2/token, previously configured value: https://zitadel.DOMAIN.nz/oauth/v2/token" > netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://zitadel.DOMAIN.nz/oauth/v2/device_authorization, previously configured value: https://zitadel.DOMAIN.nz/oauth/v2/device_authorization" > netbird-management | time="2023-07-04T00:39:16Z" level=info msg="overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: zitadel.DOMAIN.nz, previously configured value: " > netbird-management | 2023-07-04T00:39:16Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081 > netbird-management | 2023-07-04T00:39:16Z INFO management/server/account.go:671: single account mode enabled, accounts number 1 > netbird-management | 2023-07-04T00:39:16Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073 > netbird-management | 2023-07-04T00:39:16Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443 > netbird-management | 2023-07-04T00:39:19Z INFO management/server/account.go:858: warmed up IDP cache with 1 entries > netbird-management | 2023-07-04T00:39:38Z INFO management/server/account.go:1256: overriding JWT Domain and DomainCategory claims since single account mode is enabled > netbird-management | 2023-07-04T00:39:38Z INFO management/server/account.go:1256: overriding JWT Domain and DomainCategory claims since single account mode is enabled The dashboard will always log in correctly and management actually responds as you can see above for some hours after restarting management, I'm guessing until token expiration. After that happens management starts responding with 401 as so: > netbird-management | 2023-07-04T23:38:16Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 125147462: GET /api/peers status 401 > netbird-management | 2023-07-04T23:38:16Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 1167195797: GET /api/groups status 401 > netbird-management | 2023-07-04T23:38:29Z ERRO management/server/telemetry/http_api_metrics.go:181: HTTP response 2431638793: GET /api/users status 401 I have been messing with config files over several versions and weeks to try and debug this further, I had a feeling that it was something to do with the token being invalid due to a mismatch in URL vs. URL:443 or something but I can't seem to figure that out and I'm not sure if that's the issue at all. EDIT: Actually I just came across #806 and #808 which I think is likely to be my issue. I've recreated my containers now and will test that.
Author
Owner

@Akruidenberg commented on GitHub (Jul 5, 2023):

I also encountered this issue using authentik on a fresh install. I fixed the issue by setting a JWKS signing key in authentik under providers -> netbird -> protocol settings -> signing key. You may need to reconfigure / restart netbird.

Great! its working now in the web interface. Thanks for that. However, after login in with Authentik, I got a white page and the following error in the management container:

### 2023-07-05T16:22:51+02:00 WARN management/server/grpcserver.go:322: failed logging in peer R7Z6hJvUc6+dCLF3Y3U7rpfreXIwhPEqygmEPFrpEXs=

Becuase I see no other error in the logs, I dont know where to start.

@Akruidenberg commented on GitHub (Jul 5, 2023): > I also encountered this issue using authentik on a fresh install. I fixed the issue by setting a JWKS signing key in authentik under providers -> netbird -> protocol settings -> signing key. You may need to reconfigure / restart netbird. Great! its working now in the web interface. Thanks for that. However, after login in with Authentik, I got a white page and the following error in the management container: ### **### `2023-07-05T16:22:51+02:00 WARN management/server/grpcserver.go:322: failed logging in peer R7Z6hJvUc6+dCLF3Y3U7rpfreXIwhPEqygmEPFrpEXs=`** Becuase I see no other error in the logs, I dont know where to start.
Author
Owner

@hadleyrich commented on GitHub (Jul 5, 2023):

EDIT: Actually I just came across #806 and #808 which I think is likely to be my issue. I've recreated my containers now and will test that.

This has solved my issue and it appears to be unrelated to this issue. Apologies for the noise.

@hadleyrich commented on GitHub (Jul 5, 2023): > EDIT: Actually I just came across #806 and #808 which I think is likely to be my issue. I've recreated my containers now and will test that. This has solved my issue and it appears to be unrelated to this issue. Apologies for the noise.
Author
Owner

@UncleJ4ck commented on GitHub (Jul 22, 2023):

the issue still exists in the new version 0.21.9 with keycloak
2023-07-22-14 43 43-screenshot(1)

dashboard_1   | 2023/07/22 13:59:57 [crit] 35#35: *131 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 159.223.19.103, server: 0.0.0.0:443
dashboard_1   | X - - [22/Jul/2023:13:59:57 +0000] "GET / HTTP/1.1" 200 347 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
dashboard_1   | X - - [22/Jul/2023:13:59:58 +0000] "GET /private/api/v1/service/premaster HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
dashboard_1   | X - - [22/Jul/2023:13:59:58 +0000] "GET /favicon.ico HTTP/1.1" 200 11587 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-"
dashboard_1   | X  - - [22/Jul/2023:14:02:52 +0000] "GET / HTTP/1.1" 200 602 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-"
dashboard_1   | X - - [22/Jul/2023:14:02:52 +0000] "GET / HTTP/1.1" 200 602 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7" "-"
dashboard_1   | X - - [22/Jul/2023:14:14:53 +0000] "GET /.env HTTP/1.1" 400 650 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-"
dashboard_1   | X - - [22/Jul/2023:14:14:53 +0000] "GET /.env HTTP/1.1" 200 347 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-"
> some logs before they looks fine
management_1  | 2023-07-22T12:37:45Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
management_1  | 2023-07-22T12:37:45Z INFO management/server/account.go:672: single account mode enabled, accounts number 0
management_1  | 2023-07-22T12:37:46Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
management_1  | 2023-07-22T12:37:46Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443
@UncleJ4ck commented on GitHub (Jul 22, 2023): the issue still exists in the new version 0.21.9 with keycloak ![2023-07-22-14 43 43-screenshot(1)](https://github.com/netbirdio/netbird/assets/52085661/1ba71874-6131-49f6-a40a-3aef14825e24) ``` dashboard_1 | 2023/07/22 13:59:57 [crit] 35#35: *131 SSL_do_handshake() failed (SSL: error:141CF06C:SSL routines:tls_parse_ctos_key_share:bad key share) while SSL handshaking, client: 159.223.19.103, server: 0.0.0.0:443 dashboard_1 | X - - [22/Jul/2023:13:59:57 +0000] "GET / HTTP/1.1" 200 347 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-" dashboard_1 | X - - [22/Jul/2023:13:59:58 +0000] "GET /private/api/v1/service/premaster HTTP/1.1" 400 650 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-" dashboard_1 | X - - [22/Jul/2023:13:59:58 +0000] "GET /favicon.ico HTTP/1.1" 200 11587 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36" "-" dashboard_1 | X - - [22/Jul/2023:14:02:52 +0000] "GET / HTTP/1.1" 200 602 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" "-" dashboard_1 | X - - [22/Jul/2023:14:02:52 +0000] "GET / HTTP/1.1" 200 602 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/601.7.7 (KHTML, like Gecko) Version/9.1.2 Safari/601.7.7" "-" dashboard_1 | X - - [22/Jul/2023:14:14:53 +0000] "GET /.env HTTP/1.1" 400 650 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-" dashboard_1 | X - - [22/Jul/2023:14:14:53 +0000] "GET /.env HTTP/1.1" 200 347 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.129 Safari/537.36" "-" ``` ``` > some logs before they looks fine management_1 | 2023-07-22T12:37:45Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081 management_1 | 2023-07-22T12:37:45Z INFO management/server/account.go:672: single account mode enabled, accounts number 0 management_1 | 2023-07-22T12:37:46Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073 management_1 | 2023-07-22T12:37:46Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443 ```
Author
Owner

@mlsmaycon commented on GitHub (Jul 22, 2023):

Hello folks; sorry for the delay in replying.

The issue is probably connected to a wrong Audience configuration. We added logs in the management system, but unfortunately they were set to debug. Can you all edit your docker-compose.yml and add the following line to the management service command list:

"--log-level", "debug",

The whole command should look similar to this:

    command: [
      "--port", "443",
      "--log-file", "console",
      "--log-level", "debug",
      "--disable-anonymous-metrics=false",
      "--single-account-mode-domain=netbird.selfhosted",
      "--dns-domain=netbird.selfhosted"
      ]

After that run docker-compose up -d to recreate the container.

If you have an issue with the audience, you should see logs like these:

2023-07-22T16:56:48+02:00 DEBG management/server/jwtclaims/jwtValidator.go:158: error parsing token: invalid audience
2023-07-22T16:56:48+02:00 DEBG management/server/http/middleware/auth_middleware.go:60: Error when validating JWT claims: Error parsing token: invalid audience

If that is the case, please review the settings in the management.json if they match your configuration.

I've opened the PR #1026 to fix this log-level issue.

@mlsmaycon commented on GitHub (Jul 22, 2023): Hello folks; sorry for the delay in replying. The issue is probably connected to a wrong Audience configuration. We added logs in the management system, but unfortunately they were set to debug. Can you all edit your docker-compose.yml and add the following line to the management service command list: `"--log-level", "debug",` The whole command should look similar to this: ``` command: [ "--port", "443", "--log-file", "console", "--log-level", "debug", "--disable-anonymous-metrics=false", "--single-account-mode-domain=netbird.selfhosted", "--dns-domain=netbird.selfhosted" ] ``` After that run `docker-compose up -d` to recreate the container. If you have an issue with the audience, you should see logs like these: ```shell 2023-07-22T16:56:48+02:00 DEBG management/server/jwtclaims/jwtValidator.go:158: error parsing token: invalid audience 2023-07-22T16:56:48+02:00 DEBG management/server/http/middleware/auth_middleware.go:60: Error when validating JWT claims: Error parsing token: invalid audience ``` If that is the case, please review the settings in the management.json if they match your configuration. I've opened the PR #1026 to fix this log-level issue.
Author
Owner

@UncleJ4ck commented on GitHub (Jul 22, 2023):

i added the log-level debug into the docker-compose file and i recreated the container, but i don't have this type of error

Attaching to infrastructure_files_management_1
management_1  | time="2023-07-22T16:52:28Z" level=info msg="loading OIDC configuration from the provided IDP configuration endpoint https://domain/realms/netbird/.well-known/openid-configuration"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="loaded OIDC configuration from the provided IDP configuration endpoint: https://domain/realms/netbird/.well-known/openid-configuration"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding HttpConfig.AuthIssuer with a new value https://domain/realms/netbird, previously configured value: https://domain/realms/netbird"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://domain/realms/netbird/protocol/openid-connect/certs, previously configured value: https://domain/realms/netbird/protocol/openid-connect/certs"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://domain/realms/netbird/protocol/openid-connect/token, previously configured value: https://domain/realms/netbird/protocol/openid-connect/token"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://domain/realms/netbird/protocol/openid-connect/auth/device, previously configured value: https://domain/realms/netbird/protocol/openid-connect/auth/device"
management_1  | time="2023-07-22T16:52:28Z" level=info msg="overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: domain, previously configured value: "
management_1  | 2023-07-22T16:52:28Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081
management_1  | 2023-07-22T16:52:28Z DEBG management/server/file_store.go:248: took 0 ms to persist the FileStore
management_1  | 2023-07-22T16:52:28Z INFO management/server/account.go:672: single account mode enabled, accounts number 0
management_1  | 2023-07-22T16:52:28Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073
management_1  | 2023-07-22T16:52:28Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443

for the dashboard logs, i don't see any errors all requests have 200 as a response

@UncleJ4ck commented on GitHub (Jul 22, 2023): i added the log-level debug into the docker-compose file and i recreated the container, but i don't have this type of error ``` Attaching to infrastructure_files_management_1 management_1 | time="2023-07-22T16:52:28Z" level=info msg="loading OIDC configuration from the provided IDP configuration endpoint https://domain/realms/netbird/.well-known/openid-configuration" management_1 | time="2023-07-22T16:52:28Z" level=info msg="loaded OIDC configuration from the provided IDP configuration endpoint: https://domain/realms/netbird/.well-known/openid-configuration" management_1 | time="2023-07-22T16:52:28Z" level=info msg="overriding HttpConfig.AuthIssuer with a new value https://domain/realms/netbird, previously configured value: https://domain/realms/netbird" management_1 | time="2023-07-22T16:52:28Z" level=info msg="overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://domain/realms/netbird/protocol/openid-connect/certs, previously configured value: https://domain/realms/netbird/protocol/openid-connect/certs" management_1 | time="2023-07-22T16:52:28Z" level=info msg="overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://domain/realms/netbird/protocol/openid-connect/token, previously configured value: https://domain/realms/netbird/protocol/openid-connect/token" management_1 | time="2023-07-22T16:52:28Z" level=info msg="overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://domain/realms/netbird/protocol/openid-connect/auth/device, previously configured value: https://domain/realms/netbird/protocol/openid-connect/auth/device" management_1 | time="2023-07-22T16:52:28Z" level=info msg="overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: domain, previously configured value: " management_1 | 2023-07-22T16:52:28Z INFO management/server/telemetry/app_metrics.go:161: enabled application metrics and exposing on http://0.0.0.0:8081 management_1 | 2023-07-22T16:52:28Z DEBG management/server/file_store.go:248: took 0 ms to persist the FileStore management_1 | 2023-07-22T16:52:28Z INFO management/server/account.go:672: single account mode enabled, accounts number 0 management_1 | 2023-07-22T16:52:28Z INFO management/cmd/management.go:233: running gRPC backward compatibility server: [::]:33073 management_1 | 2023-07-22T16:52:28Z INFO management/cmd/management.go:265: running HTTP server and gRPC server on the same port: [::]:443 ``` for the dashboard logs, i don't see any errors all requests have 200 as a response
Author
Owner

@mlsmaycon commented on GitHub (Jul 22, 2023):

Hello @UncleJ4ck do you mind joining our Slack channel it may be faster for us to troubleshoot the issue.

@mlsmaycon commented on GitHub (Jul 22, 2023): Hello @UncleJ4ck do you mind joining our [Slack channel](https://join.slack.com/t/netbirdio/shared_invite/zt-vrahf41g-ik1v7fV8du6t0RwxSrJ96A) it may be faster for us to troubleshoot the issue.
Author
Owner

@UncleJ4ck commented on GitHub (Jul 22, 2023):

Sure

@UncleJ4ck commented on GitHub (Jul 22, 2023): Sure
Author
Owner

@UncleJ4ck commented on GitHub (Jul 22, 2023):

fixed, after fixing keycloak

@UncleJ4ck commented on GitHub (Jul 22, 2023): fixed, after fixing keycloak
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#388