SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-07-24 23:13:59 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

management api randomly dropping roles from accounts table locking me out (google workspace) #1599

New Issue
Closed
opened 2025-11-20 05:33:32 -05:00 by saavagebueno · 22 comments
saavagebueno commented 2025-11-20 05:33:32 -05:00
Owner
Copy Link

Originally created by @rgdev on GitHub (Feb 4, 2025).

Describe the problem

After updating from 0.34.1 to 0.36.5 the UI became unusable as I keep getting locked out. It works for a little while, I can move around and load pages and suddenly I'm no longer owner (or admin) & become user so I get 403 errors and am presented with the "Add new device to your network" page. Only way to fix it is to go into pg and manually change my role back.

Mangement is running with those args :

            - "--port"
            - "80"
            - "--log-file"
            - "console"
            - "--log-level"
            - "info"
            - "--disable-anonymous-metrics=true"
            - "--single-account-mode-domain=mydomain.com"
            - "--dns-domain=mydomain.com"
            - "--idp-sign-key-refresh-enabled"

Config looks like this :

{
  // relay, turn, signal, stun, reverse proxy config skipped for readability
  "HttpConfig": {
    "Address": "0.0.0.0:80",
    "AuthIssuer": "https://accounts.google.com",
    "AuthAudience": "xxxxxxxxxxxxxxx.apps.googleusercontent.com",
    "AuthKeysLocation": "https://www.googleapis.com/oauth2/v3/certs",
    "AuthUserIDClaim": "sub",
    "CertFile": "",
    "CertKey": "",
    "IdpSignKeyRefreshEnabled": false,
    "OIDCConfigEndpoint": "https://accounts.google.com/.well-known/openid-configuration"
  },
  "IdpManagerConfig": {
    "ManagerType": "none",
    "ClientConfig": {
      "Issuer": "https://accounts.google.com",
      "TokenEndpoint": "https://oauth2.googleapis.com/token",
      "ClientID": "xxxxxxxxxxxxxxx.apps.googleusercontent.com",
      "ClientSecret": "xxxxxxxxxxxxxxx",
      "GrantType": "client_credentials"
    },
    "ExtraConfig": {
      "CustomerId": "xxxxxxxxxxxxxxx",
      "ServiceAccountKey": "xxxxxxxxxxxxxxx"
    },
    "Auth0ClientCredentials": null,
    "AzureClientCredentials": null,
    "KeycloakClientCredentials": null,
    "ZitadelClientCredentials": null
  },
  "DeviceAuthorizationFlow": {
    "Provider": "none",
    "ProviderConfig": {
      "Audience": "xxxxxxxxxxxxxxx.apps.googleusercontent.com",
      "AuthorizationEndpoint": "",
      "Domain": "",
      "ClientID": "xxxxxxxxxxxxxxx.apps.googleusercontent.com",
      "ClientSecret": "",
      "TokenEndpoint": "https://oauth2.googleapis.com/token",
      "DeviceAuthEndpoint": "https://oauth2.googleapis.com/device/code",
      "Scope": "openid",
      "UseIDToken": true,
      "RedirectURLs": null
    }
  },
  "PKCEAuthorizationFlow": {
    "ProviderConfig": {
      "Audience": "xxxxxxxxxxxxxxx.apps.googleusercontent.com",
      "ClientID": "xxxxxxxxxxxxxxx.apps.googleusercontent.com",
      "ClientSecret": "xxxxxxxxxxxxxxx",
      "Domain": "",
      "AuthorizationEndpoint": "https://accounts.google.com/o/oauth2/v2/auth",
      "TokenEndpoint": "https://oauth2.googleapis.com/token",
      "Scope": "openid profile email",
      "RedirectURLs": [
        "http://localhost:53000/",
        "http://localhost:54000/",
        "https://mydomain.com/auth",
        "https://mydomain.com/silent-auth"
      ],
      "UseIDToken": true
    }
  }
}

To Reproduce

Steps to reproduce the behavior:

Deploy 0.36.5 with google workspace idp

Expected behavior

Mangement shouldn't turn admin and owner roles into user role.

Are you using NetBird Cloud?

Self Hosted

NetBird version

0.36.5

Screenshots

accounts table showing me as owner :

Image

Later on, my role dropped and i'm now a user :

Image

Additional context

Add any other context about the problem here.

Originally created by @rgdev on GitHub (Feb 4, 2025). **Describe the problem** After updating from 0.34.1 to 0.36.5 the UI became unusable as I keep getting locked out. It works for a little while, I can move around and load pages and suddenly I'm no longer owner (or admin) & become user so I get 403 errors and am presented with the "Add new device to your network" page. Only way to fix it is to go into pg and manually change my role back. Mangement is running with those args : ``` - "--port" - "80" - "--log-file" - "console" - "--log-level" - "info" - "--disable-anonymous-metrics=true" - "--single-account-mode-domain=mydomain.com" - "--dns-domain=mydomain.com" - "--idp-sign-key-refresh-enabled" ``` Config looks like this : ``` { // relay, turn, signal, stun, reverse proxy config skipped for readability "HttpConfig": { "Address": "0.0.0.0:80", "AuthIssuer": "https://accounts.google.com", "AuthAudience": "xxxxxxxxxxxxxxx.apps.googleusercontent.com", "AuthKeysLocation": "https://www.googleapis.com/oauth2/v3/certs", "AuthUserIDClaim": "sub", "CertFile": "", "CertKey": "", "IdpSignKeyRefreshEnabled": false, "OIDCConfigEndpoint": "https://accounts.google.com/.well-known/openid-configuration" }, "IdpManagerConfig": { "ManagerType": "none", "ClientConfig": { "Issuer": "https://accounts.google.com", "TokenEndpoint": "https://oauth2.googleapis.com/token", "ClientID": "xxxxxxxxxxxxxxx.apps.googleusercontent.com", "ClientSecret": "xxxxxxxxxxxxxxx", "GrantType": "client_credentials" }, "ExtraConfig": { "CustomerId": "xxxxxxxxxxxxxxx", "ServiceAccountKey": "xxxxxxxxxxxxxxx" }, "Auth0ClientCredentials": null, "AzureClientCredentials": null, "KeycloakClientCredentials": null, "ZitadelClientCredentials": null }, "DeviceAuthorizationFlow": { "Provider": "none", "ProviderConfig": { "Audience": "xxxxxxxxxxxxxxx.apps.googleusercontent.com", "AuthorizationEndpoint": "", "Domain": "", "ClientID": "xxxxxxxxxxxxxxx.apps.googleusercontent.com", "ClientSecret": "", "TokenEndpoint": "https://oauth2.googleapis.com/token", "DeviceAuthEndpoint": "https://oauth2.googleapis.com/device/code", "Scope": "openid", "UseIDToken": true, "RedirectURLs": null } }, "PKCEAuthorizationFlow": { "ProviderConfig": { "Audience": "xxxxxxxxxxxxxxx.apps.googleusercontent.com", "ClientID": "xxxxxxxxxxxxxxx.apps.googleusercontent.com", "ClientSecret": "xxxxxxxxxxxxxxx", "Domain": "", "AuthorizationEndpoint": "https://accounts.google.com/o/oauth2/v2/auth", "TokenEndpoint": "https://oauth2.googleapis.com/token", "Scope": "openid profile email", "RedirectURLs": [ "http://localhost:53000/", "http://localhost:54000/", "https://mydomain.com/auth", "https://mydomain.com/silent-auth" ], "UseIDToken": true } } } ``` **To Reproduce** Steps to reproduce the behavior: Deploy 0.36.5 with google workspace idp **Expected behavior** Mangement shouldn't turn admin and owner roles into user role. **Are you using NetBird Cloud?** Self Hosted **NetBird version** 0.36.5 **Screenshots** accounts table showing me as owner : <img width="1082" alt="Image" src="https://github.com/user-attachments/assets/04a40bd1-a548-4b3b-8f6f-37dee1afc819" /> Later on, my role dropped and i'm now a user : <img width="1081" alt="Image" src="https://github.com/user-attachments/assets/9e859417-d9b9-4726-93e7-9ec6935e5efd" /> **Additional context** Add any other context about the problem here.
saavagebueno added the bugmanagement-serviceself-hostingpostgres labels 2025-11-20 05:33:32 -05:00
saavagebueno commented 2025-11-20 05:33:33 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 4, 2025):

I just noticed in the Activity log there's a "User joined NetBird" message that appears around the same time my role is modified so I'm starting to think the user is somehow recreated entirely?

@rgdev commented on GitHub (Feb 4, 2025): I just noticed in the Activity log there's a "User joined NetBird" message that appears around the same time my role is modified so I'm starting to think the user is somehow recreated entirely?
saavagebueno commented 2025-11-20 05:33:33 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Feb 4, 2025):

Hello @rgdev ,

Regarding the locked out behavior, does this mean you are completely unable to log in or is your functionality limited?

Additionally, could you provide details about the user count in your account? Are other users experiencing similar issues? It would also be helpful to check the Activity Log to see if it explicitly shows that your role has been updated

Additionally, please check if your dashboard is updated to match the latest version. If not, it should be updated when updating the management.

@bcmmbaga commented on GitHub (Feb 4, 2025): Hello @rgdev , Regarding the locked out behavior, does this mean you are completely unable to log in or is your functionality limited? Additionally, could you provide details about the user count in your account? Are other users experiencing similar issues? It would also be helpful to check the Activity Log to see if it explicitly shows that your role has been updated Additionally, please check if your dashboard is updated to match the latest version. If not, it should be updated when updating the management.
saavagebueno commented 2025-11-20 05:33:33 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 4, 2025):

Hi @bcmmbaga

It just happened again and basically what I get is the "error 403" banner popping at the top, all the UI locks up and my role switches from "owner" to "user" so i'm unable to do anything because the "user" role has no dashboard privilege.

Image

I checked the devtools in chrome for additional context and this is what /api/users returns :

[
    {
        "auto_groups": [],
        "email": "",
        "id": "1XXXXXXXXXXXXXX7",
        "is_blocked": false,
        "is_current": true,
        "is_service_user": false,
        "issued": "api",
        "last_login": "0001-01-01T00:00:00Z",
        "name": "",
        "permissions": {
            "dashboard_view": "blocked"
        },
        "role": "user",
        "status": "active"
    }
]

As you can see the API now thinks i'm a regular user and I have the dashboard_view:blocked.

I checked the activity log and the only relevant message is the creation of my own user that.. already exist. Everything else is normal events (peer added), no traces of a role related event.

Image

We have maybe 5 users on this test instance so nothing crazy, other users with "admin" roles are also affected and can randomly be converted to regular users.

Dashboard is running v2.9.0.

@rgdev commented on GitHub (Feb 4, 2025): Hi @bcmmbaga It just happened again and basically what I get is the "error 403" banner popping at the top, all the UI locks up and my role switches from "owner" to "user" so i'm unable to do anything because the "user" role has no dashboard privilege. <img width="1641" alt="Image" src="https://github.com/user-attachments/assets/f4f1bdc9-1ea2-43b8-831e-741bdb540836" /> I checked the devtools in chrome for additional context and this is what /api/users returns : ``` [ { "auto_groups": [], "email": "", "id": "1XXXXXXXXXXXXXX7", "is_blocked": false, "is_current": true, "is_service_user": false, "issued": "api", "last_login": "0001-01-01T00:00:00Z", "name": "", "permissions": { "dashboard_view": "blocked" }, "role": "user", "status": "active" } ] ``` As you can see the API now thinks i'm a regular user and I have the `dashboard_view:blocked`. I checked the activity log and the only relevant message is the creation of my own user that.. already exist. Everything else is normal events (peer added), no traces of a role related event. <img width="782" alt="Image" src="https://github.com/user-attachments/assets/f24c5224-c338-4cd9-bcbb-2b4939aff7e2" /> We have maybe 5 users on this test instance so nothing crazy, other users with "admin" roles are also affected and can randomly be converted to regular users. Dashboard is running v2.9.0.
saavagebueno commented 2025-11-20 05:33:33 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 4, 2025):

To be clear : this is a dashboard and api issue, I can still connect to peers, login via the netbird cli, all the "vpn side" works fine, it's really about the using the dashboard.

@rgdev commented on GitHub (Feb 4, 2025): To be clear : this is a dashboard and api issue, I can still connect to peers, login via the netbird cli, all the "vpn side" works fine, it's really about the using the dashboard.
saavagebueno commented 2025-11-20 05:33:33 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Feb 4, 2025):

can you share the management logs? In the meantime, I will try to reproduce this on our end.

@bcmmbaga commented on GitHub (Feb 4, 2025): can you share the management logs? In the meantime, I will try to reproduce this on our end.
saavagebueno commented 2025-11-20 05:33:33 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 4, 2025):

I managed to grab some logs when it occurred and it's really odd. In the span of a second you can see I made multiple queries and got OK 200 and suddenly I get 503s on everything.

All those CENSORED_GOOGLE_ID are mine.

// looks good
2025-02-04T19:31:48Z DEBG [context: HTTP, requestID: 25d25a5a-9912-4b64-9aae-023e5ddd2d88] management/server/telemetry/http_api_metrics.go:202: request GET /api/posture-checks took 40 ms and finished with status 200
2025-02-04T19:31:48Z DEBG [context: HTTP, requestID: 9d61120b-9784-4afa-be20-8dca327475a9] management/server/telemetry/http_api_metrics.go:202: request GET /api/networks took 63 ms and finished with status 200
2025-02-04T19:31:49Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK
2025-02-04T19:31:49Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK
2025-02-04T19:31:49Z DEBG [requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b] management/server/account.go:1484: no primary account found for domain affluences.com, acquiring global lock
2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 9bebc5ee-f409-45fd-adcd-c31b53ca4db0] management/server/store/sql_store.go:211: took 130 ms to persist an account to the store
2025-02-04T19:31:49Z DEBG [requestID: 9bebc5ee-f409-45fd-adcd-c31b53ca4db0, accountID: crgn4rskjf2s73fb8bp0, userID: 2877aa18-2818-445e-a8a3-9139119a3bb9, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:49Z DEBG [requestID: 9bebc5ee-f409-45fd-adcd-c31b53ca4db0, accountID: crgn4rskjf2s73fb8bp0, userID: 2877aa18-2818-445e-a8a3-9139119a3bb9, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:49Z DEBG [accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 9bebc5ee-f409-45fd-adcd-c31b53ca4db0] management/server/telemetry/http_api_metrics.go:202: request GET /api/peers took 236 ms and finished with status 200
2025-02-04T19:31:49Z DEBG [accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5, accountID: , userID: CENSORED_GOOGLE_ID] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:49Z DEBG [userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b, accountID: ] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
// locked out
2025-02-04T19:31:49Z ERRO [accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5] management/server/http/util/util.go:85: got a handler error: only users with admin power can view Network Routes
2025-02-04T19:31:49Z ERRO [context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5] management/server/telemetry/http_api_metrics.go:189: HTTP response 028e7c3e-af92-4eb3-8844-0d869a228df5: GET /api/routes status 403
2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5] management/server/telemetry/http_api_metrics.go:202: request GET /api/routes took 163 ms and finished with status 403
2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b] management/server/telemetry/http_api_metrics.go:202: request GET /api/peers took 175 ms and finished with status 200
2025-02-04T19:31:50Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK
2025-02-04T19:31:50Z DEBG [context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:50Z DEBG [context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88, accountID: , userID: CENSORED_GOOGLE_ID] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:50Z DEBG [userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88, accountID: ] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:50Z ERRO [context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88, accountID: , userID: CENSORED_GOOGLE_ID] management/server/http/util/util.go:85: got a handler error: admin role required to perform this action
2025-02-04T19:31:50Z ERRO [context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88] management/server/telemetry/http_api_metrics.go:189: HTTP response 104c44ab-4bae-489f-8046-d6d4fa16ab88: GET /api/setup-keys status 403
2025-02-04T19:31:50Z DEBG [requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88, context: HTTP] management/server/telemetry/http_api_metrics.go:202: request GET /api/setup-keys took 26 ms and finished with status 403
2025-02-04T19:31:53Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK
2025-02-04T19:31:53Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK
2025-02-04T19:31:53Z DEBG [context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:53Z DEBG [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:53Z DEBG [accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:53Z DEBG [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:53Z DEBG [context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc, accountID: , userID: CENSORED_GOOGLE_ID] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:53Z DEBG [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2025-02-04T19:31:53Z ERRO [requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc, accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP] management/server/http/util/util.go:85: got a handler error: admin role required to perform this action
2025-02-04T19:31:53Z ERRO [context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc] management/server/telemetry/http_api_metrics.go:189: HTTP response e5ce6916-2e10-4cb4-a6d3-79c7fed234bc: GET /api/posture-checks status 403
2025-02-04T19:31:53Z DEBG [context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc] management/server/telemetry/http_api_metrics.go:202: request GET /api/posture-checks took 39 ms and finished with status 403
2025-02-04T19:31:53Z ERRO [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP] management/server/http/util/util.go:85: got a handler error: permission denied
2025-02-04T19:31:53Z ERRO [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, context: HTTP] management/server/telemetry/http_api_metrics.go:189: HTTP response cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a: GET /api/networks status 403
2025-02-04T19:31:53Z DEBG [context: HTTP, requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a] management/server/telemetry/http_api_metrics.go:202: request GET /api/networks took 43 ms and finished with status 403
@rgdev commented on GitHub (Feb 4, 2025): I managed to grab some logs when it occurred and it's really odd. In the span of a second you can see I made multiple queries and got OK 200 and suddenly I get 503s on everything. All those CENSORED_GOOGLE_ID are mine. ``` // looks good 2025-02-04T19:31:48Z DEBG [context: HTTP, requestID: 25d25a5a-9912-4b64-9aae-023e5ddd2d88] management/server/telemetry/http_api_metrics.go:202: request GET /api/posture-checks took 40 ms and finished with status 200 2025-02-04T19:31:48Z DEBG [context: HTTP, requestID: 9d61120b-9784-4afa-be20-8dca327475a9] management/server/telemetry/http_api_metrics.go:202: request GET /api/networks took 63 ms and finished with status 200 2025-02-04T19:31:49Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK 2025-02-04T19:31:49Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK 2025-02-04T19:31:49Z DEBG [requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b] management/server/account.go:1484: no primary account found for domain affluences.com, acquiring global lock 2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 9bebc5ee-f409-45fd-adcd-c31b53ca4db0] management/server/store/sql_store.go:211: took 130 ms to persist an account to the store 2025-02-04T19:31:49Z DEBG [requestID: 9bebc5ee-f409-45fd-adcd-c31b53ca4db0, accountID: crgn4rskjf2s73fb8bp0, userID: 2877aa18-2818-445e-a8a3-9139119a3bb9, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:49Z DEBG [requestID: 9bebc5ee-f409-45fd-adcd-c31b53ca4db0, accountID: crgn4rskjf2s73fb8bp0, userID: 2877aa18-2818-445e-a8a3-9139119a3bb9, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:49Z DEBG [accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 9bebc5ee-f409-45fd-adcd-c31b53ca4db0] management/server/telemetry/http_api_metrics.go:202: request GET /api/peers took 236 ms and finished with status 200 2025-02-04T19:31:49Z DEBG [accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5, accountID: , userID: CENSORED_GOOGLE_ID] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:49Z DEBG [userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b, accountID: ] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled // locked out 2025-02-04T19:31:49Z ERRO [accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5] management/server/http/util/util.go:85: got a handler error: only users with admin power can view Network Routes 2025-02-04T19:31:49Z ERRO [context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5] management/server/telemetry/http_api_metrics.go:189: HTTP response 028e7c3e-af92-4eb3-8844-0d869a228df5: GET /api/routes status 403 2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 028e7c3e-af92-4eb3-8844-0d869a228df5] management/server/telemetry/http_api_metrics.go:202: request GET /api/routes took 163 ms and finished with status 403 2025-02-04T19:31:49Z DEBG [context: HTTP, requestID: 311aba9f-9316-4063-9b30-37747762965b] management/server/telemetry/http_api_metrics.go:202: request GET /api/peers took 175 ms and finished with status 200 2025-02-04T19:31:50Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK 2025-02-04T19:31:50Z DEBG [context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:50Z DEBG [context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88, accountID: , userID: CENSORED_GOOGLE_ID] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:50Z DEBG [userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88, accountID: ] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:50Z ERRO [context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88, accountID: , userID: CENSORED_GOOGLE_ID] management/server/http/util/util.go:85: got a handler error: admin role required to perform this action 2025-02-04T19:31:50Z ERRO [context: HTTP, requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88] management/server/telemetry/http_api_metrics.go:189: HTTP response 104c44ab-4bae-489f-8046-d6d4fa16ab88: GET /api/setup-keys status 403 2025-02-04T19:31:50Z DEBG [requestID: 104c44ab-4bae-489f-8046-d6d4fa16ab88, context: HTTP] management/server/telemetry/http_api_metrics.go:202: request GET /api/setup-keys took 26 ms and finished with status 403 2025-02-04T19:31:53Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK 2025-02-04T19:31:53Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:233: generating PublicKey from RSA JWK 2025-02-04T19:31:53Z DEBG [context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:53Z DEBG [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:53Z DEBG [accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:53Z DEBG [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:53Z DEBG [context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc, accountID: , userID: CENSORED_GOOGLE_ID] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:53Z DEBG [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP] management/server/account.go:1218: overriding JWT Domain and DomainCategory claims since single account mode is enabled 2025-02-04T19:31:53Z ERRO [requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc, accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP] management/server/http/util/util.go:85: got a handler error: admin role required to perform this action 2025-02-04T19:31:53Z ERRO [context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc] management/server/telemetry/http_api_metrics.go:189: HTTP response e5ce6916-2e10-4cb4-a6d3-79c7fed234bc: GET /api/posture-checks status 403 2025-02-04T19:31:53Z DEBG [context: HTTP, requestID: e5ce6916-2e10-4cb4-a6d3-79c7fed234bc] management/server/telemetry/http_api_metrics.go:202: request GET /api/posture-checks took 39 ms and finished with status 403 2025-02-04T19:31:53Z ERRO [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, accountID: , userID: CENSORED_GOOGLE_ID, context: HTTP] management/server/http/util/util.go:85: got a handler error: permission denied 2025-02-04T19:31:53Z ERRO [requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a, context: HTTP] management/server/telemetry/http_api_metrics.go:189: HTTP response cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a: GET /api/networks status 403 2025-02-04T19:31:53Z DEBG [context: HTTP, requestID: cc7e9b3d-3d48-4d61-bb8d-3c90c71e439a] management/server/telemetry/http_api_metrics.go:202: request GET /api/networks took 43 ms and finished with status 403 ```
saavagebueno commented 2025-11-20 05:33:33 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 4, 2025):

And the "User joined" event's timestamp (UTC+1) matches perfectly with the timestamps in the logs (UTC)

Image
@rgdev commented on GitHub (Feb 4, 2025): And the "User joined" event's timestamp (UTC+1) matches perfectly with the timestamps in the logs (UTC) <img width="784" alt="Image" src="https://github.com/user-attachments/assets/7f0197f1-d90c-4aef-a452-d561f334b5e6" />
saavagebueno commented 2025-11-20 05:33:34 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Feb 4, 2025):

Can you share the full management logs? Please remember to mask any sensitive information

@bcmmbaga commented on GitHub (Feb 4, 2025): Can you share the full management logs? Please remember to mask any sensitive information
saavagebueno commented 2025-11-20 05:33:34 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 4, 2025):

Sent over email to the address registered on your Github profile. Thank you for the help.

@rgdev commented on GitHub (Feb 4, 2025): Sent over email to the address registered on your Github profile. Thank you for the help.
saavagebueno commented 2025-11-20 05:33:34 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 6, 2025):

Additional notes : there was an issue with workspace causing only the google user ID to appear as seen in my previous messages. I've made sure to set IdpManagerConfig.ManagerType to "google" and bound the service account to the proper role in the workspace admin console and now names and emails do appear in the UI and API calls.

Image

2025-02-06T15:41:40Z INFO [context: SYSTEM] management/server/account.go:564: 1 entries received from IdP management
2025-02-06T15:41:40Z INFO [context: SYSTEM] management/server/account.go:595: warmed up IDP cache with 5 entries for 1 accounts

However this has not resolved the issue with my role turning back to "user".

Additionally I noticed a few things when looking at the users table in pg :

  • auto_groups field is set to [] stripping me from my auto groups
  • last_login field is set to null
  • created_at points to a date in the past which suggests that in fact the row isn't recreated but updated

Image

@rgdev commented on GitHub (Feb 6, 2025): Additional notes : there was an issue with workspace causing only the google user ID to appear as seen in my previous messages. I've made sure to set `IdpManagerConfig.ManagerType` to "google" and bound the service account to the proper role in the workspace admin console and now names and emails do appear in the UI and API calls. ![Image](https://github.com/user-attachments/assets/4d44a49c-28e9-450c-8af2-f2152b830131) ``` 2025-02-06T15:41:40Z INFO [context: SYSTEM] management/server/account.go:564: 1 entries received from IdP management 2025-02-06T15:41:40Z INFO [context: SYSTEM] management/server/account.go:595: warmed up IDP cache with 5 entries for 1 accounts ``` However this has not resolved the issue with my role turning back to "user". Additionally I noticed a few things when looking at the `users` table in pg : - `auto_groups` field is set to `[]` stripping me from my auto groups - `last_login` field is set to `null` - `created_at` points to a date in the past which suggests that in fact the row isn't recreated but updated ![Image](https://github.com/user-attachments/assets/8491aa2f-b3ed-4165-9291-8da4c0e976bf)
saavagebueno commented 2025-11-20 05:33:34 -05:00
Author
Owner
Copy Link

@WGOS commented on GitHub (Feb 12, 2025):

I just stumbled upon the same behavior in our self-hosted deployment on 0.36.5 too. We're using Google Workspaces as IdP but we connect this issue with using API from a script.

Here's how I manage to reproduce this:

  1. Start a script (bash/python/whatever) that would make an API call to one of the methods: Get Users, Get Peers, Get Groups
  2. While script makes a call press F5 in the browser
  3. Now you stripped yourself of admin/owner permissions and also auto-groups

Netbird also shows that affected user "joined Netbird" just as same as in @rgdev case.

I am willing to provide you with any debug info you may need.

@WGOS commented on GitHub (Feb 12, 2025): I just stumbled upon the same behavior in our self-hosted deployment on 0.36.5 too. We're using Google Workspaces as IdP but we connect this issue with using API from a script. Here's how I manage to reproduce this: 1. Start a script (bash/python/whatever) that would make an API call to one of the methods: Get Users, Get Peers, Get Groups 2. While script makes a call press F5 in the browser 3. Now you stripped yourself of admin/owner permissions and also auto-groups Netbird also shows that affected user "joined Netbird" just as same as in @rgdev case. I am willing to provide you with any debug info you may need.
saavagebueno commented 2025-11-20 05:33:34 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Feb 12, 2025):

I'm still working on reproducing this issue and will provide more feedback as soon as we have more details.

In the meantime @WGOS, could you please share the debug logs? Also for the API calls, do you use the token from the account owner or from a different service user?

@bcmmbaga commented on GitHub (Feb 12, 2025): I'm still working on reproducing this issue and will provide more feedback as soon as we have more details. In the meantime @WGOS, could you please share the debug logs? Also for the API calls, do you use the token from the account owner or from a different service user?
saavagebueno commented 2025-11-20 05:33:34 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 12, 2025):

We also have scripts querying the API every few minutes to fetch the state of peers and the System Accounts involved all have their own access token created by my account (col created_by in table personal_access_token), this might be the link

@rgdev commented on GitHub (Feb 12, 2025): We also have scripts querying the API every few minutes to fetch the state of peers and the System Accounts involved all have their own access token created by my account (col `created_by` in table `personal_access_token`), this might be the link
saavagebueno commented 2025-11-20 05:33:35 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Feb 12, 2025):

We also have scripts querying the API every few minutes to fetch the state of peers and the System Accounts involved all have their own access token created by my account (col created_by in table personal_access_token), this might be the link

Thanks for clarifying. I’ll try to replicate the same scenario in our tests.

@bcmmbaga commented on GitHub (Feb 12, 2025): > We also have scripts querying the API every few minutes to fetch the state of peers and the System Accounts involved all have their own access token created by my account (col `created_by` in table `personal_access_token`), this might be the link Thanks for clarifying. I’ll try to replicate the same scenario in our tests.
saavagebueno commented 2025-11-20 05:33:35 -05:00
Author
Owner
Copy Link

@WGOS commented on GitHub (Feb 12, 2025):

I'm still working on reproducing this issue and will provide more feedback as soon as we have more details.

In the meantime @WGOS, could you please share the debug logs? Also for the API calls, do you use the token from the account owner or from a different service user?

Sure. I've sent it via email you provided in your Github profile.

We're using service account with its own token to access API.

It also should be noted that throttling connection to the DB increases chances to reproduce this issue.

P.S. We're using Postgres for our deployment

@WGOS commented on GitHub (Feb 12, 2025): > I'm still working on reproducing this issue and will provide more feedback as soon as we have more details. > > In the meantime [@WGOS](https://github.com/WGOS), could you please share the debug logs? Also for the API calls, do you use the token from the account owner or from a different service user? Sure. I've sent it via email you provided in your Github profile. We're using service account with its own token to access API. It also should be noted that throttling connection to the DB increases chances to reproduce this issue. P.S. We're using Postgres for our deployment
saavagebueno commented 2025-11-20 05:33:35 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Feb 14, 2025):

I was able to reproduce the issue and we've already fixed it. The fix should be included in the next week release.

@bcmmbaga commented on GitHub (Feb 14, 2025): I was able to reproduce the issue and we've already fixed it. The fix should be included in the next week release.
saavagebueno commented 2025-11-20 05:33:35 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 14, 2025):

Fantastic news, looking forward to the update 👍

@rgdev commented on GitHub (Feb 14, 2025): Fantastic news, looking forward to the update 👍
saavagebueno commented 2025-11-20 05:33:35 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Feb 26, 2025):

Hi @rgdev @WGOS can you update to v0.37.1, do the tests and let us know if works?

@bcmmbaga commented on GitHub (Feb 26, 2025): Hi @rgdev @WGOS can you update to v0.37.1, do the tests and let us know if works?
saavagebueno commented 2025-11-20 05:33:35 -05:00
Author
Owner
Copy Link

@WGOS commented on GitHub (Feb 26, 2025):

@bcmmbaga sure! We will update our instance by the end of this week

@WGOS commented on GitHub (Feb 26, 2025): @bcmmbaga sure! We will update our instance by the end of this week
saavagebueno commented 2025-11-20 05:33:36 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 26, 2025):

Hi @bcmmbaga, I've updated ours and I'll let you know if the issue occurs again in the next 48h.

@rgdev commented on GitHub (Feb 26, 2025): Hi @bcmmbaga, I've updated ours and I'll let you know if the issue occurs again in the next 48h.
saavagebueno commented 2025-11-20 05:33:36 -05:00
Author
Owner
Copy Link

@rgdev commented on GitHub (Feb 28, 2025):

@bcmmbaga No issues so far on my end 👍

@rgdev commented on GitHub (Feb 28, 2025): @bcmmbaga No issues so far on my end 👍
saavagebueno commented 2025-11-20 05:33:36 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Feb 28, 2025):

@bcmmbaga No issues so far on my end 👍

Thanks. I will go ahead and close this issue

@bcmmbaga commented on GitHub (Feb 28, 2025): > [@bcmmbaga](https://github.com/bcmmbaga) No issues so far on my end 👍 Thanks. I will go ahead and close this issue
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label bug management-service postgres self-hosting
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1599
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?