SELF HOST not success, request /api/users get 502 Bad Gateway #924

Open
opened 2025-11-20 05:19:58 -05:00 by saavagebueno · 12 comments
Owner

Originally created by @shadowofmoo on GitHub (May 24, 2024).

Describe the problem
use the quick start script export NETBIRD_DOMAIN=netbird.*****.com; curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started-with-zitadel.sh | bash success deploy.

and visit https://netbird.****.com/peers is always loading, use F12 look up, find request /api/users get 502 Bad Gateway

I'm deploy netbird in tencent cloud, so maybe is network problem, pls help me

To Reproduce

None

Expected behavior

Are you using NetBird Cloud?

NO, self host in tencent cloud

NetBird version

latest version

NetBird status -d output:

{"level":"error","ts":1716540731.158369,"logger":"http.log.error","msg":"dial tcp 172.30.0.6:80: connect: connection refused","request":{"remote_ip":"113.128.240.98","remote_port":"12824","client_ip":"113.128.240.98","proto":"HTTP/2.0","method":"GET","host":"netbird.****.com","uri":"/api/users","headers":{"Content-Type":["application/json"],"Accept":["application/json"],"Sec-Fetch-Site":["same-origin"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"Dnt":["1"],"Accept-Language":["zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,zh-TW;q=0.5,ca;q=0.4,da;q=0.3"],"Cookie":[],"Priority":["u=1, i"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0"],"Sec-Fetch-Mode":["cors"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua":["\"Microsoft Edge\";v=\"125\", \"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""],"Sec-Ch-Ua-Platform":["\"Windows\""],"Referer":["https://netbird.****.com/peers"],"Sec-Ch-Ua-Mobile":["?0"],"Authorization":[],"Sec-Fetch-Dest":["empty"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"netbird.****.com"}},"duration":0.000954148,"status":502,"err_id":"2fsfvx5p0","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"}

Screenshots

netbird management
image

zitadel
image

Additional context

Add any other context about the problem here.

Originally created by @shadowofmoo on GitHub (May 24, 2024). **Describe the problem** use the quick start script `export NETBIRD_DOMAIN=netbird.*****.com; curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started-with-zitadel.sh | bash` success deploy. and visit https://netbird.****.com/peers is always loading, use F12 look up, find request /api/users get 502 Bad Gateway I'm deploy netbird in tencent cloud, so maybe is network problem, pls help me **To Reproduce** None **Expected behavior** **Are you using NetBird Cloud?** NO, self host in tencent cloud **NetBird version** latest version **NetBird status -d output:** ``` {"level":"error","ts":1716540731.158369,"logger":"http.log.error","msg":"dial tcp 172.30.0.6:80: connect: connection refused","request":{"remote_ip":"113.128.240.98","remote_port":"12824","client_ip":"113.128.240.98","proto":"HTTP/2.0","method":"GET","host":"netbird.****.com","uri":"/api/users","headers":{"Content-Type":["application/json"],"Accept":["application/json"],"Sec-Fetch-Site":["same-origin"],"Pragma":["no-cache"],"Cache-Control":["no-cache"],"Dnt":["1"],"Accept-Language":["zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6,zh-TW;q=0.5,ca;q=0.4,da;q=0.3"],"Cookie":[],"Priority":["u=1, i"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 Edg/125.0.0.0"],"Sec-Fetch-Mode":["cors"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Ch-Ua":["\"Microsoft Edge\";v=\"125\", \"Chromium\";v=\"125\", \"Not.A/Brand\";v=\"24\""],"Sec-Ch-Ua-Platform":["\"Windows\""],"Referer":["https://netbird.****.com/peers"],"Sec-Ch-Ua-Mobile":["?0"],"Authorization":[],"Sec-Fetch-Dest":["empty"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"netbird.****.com"}},"duration":0.000954148,"status":502,"err_id":"2fsfvx5p0","err_trace":"reverseproxy.statusError (reverseproxy.go:1267)"} ``` **Screenshots** netbird management ![image](https://github.com/netbirdio/netbird/assets/124780081/2a99a408-51db-4da3-b9cb-838d5bc61bf1) zitadel ![image](https://github.com/netbirdio/netbird/assets/124780081/f0d03bbb-1d0f-4cc5-aad7-9383b77fb1ea) **Additional context** Add any other context about the problem here.
saavagebueno added the triage-needed label 2025-11-20 05:19:58 -05:00
Author
Owner

@shadowofmoo commented on GitHub (May 24, 2024):

I may found the problem, request management:80 refused
image

how solve this problem

@shadowofmoo commented on GitHub (May 24, 2024): I may found the problem, request management:80 refused ![image](https://github.com/netbirdio/netbird/assets/124780081/6923d4ee-6061-4ef1-b3ae-b8b27fdde428) how solve this problem
Author
Owner

@shadowofmoo commented on GitHub (May 24, 2024):

management log

2024-05-24T09:23:18Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://netbird.****.com/.well-known/openid-configuration Get "https://netbird.****.com/.well-known/openid-configuration": dial tcp 150.158.51.44:443: connect: connection refused
2024-05-24T09:23:18Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: 
2024-05-24T09:23:19Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: 
2024-05-24T09:23:20Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: 
2024-05-24T09:23:21Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: 
2024-05-24T09:23:23Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration
2024-05-24T09:23:23Z INFO management/cmd/management.go:460: loaded OIDC configuration from the provided IDP configuration endpoint: https://netbird.****.com/.well-known/openid-configuration
2024-05-24T09:23:23Z INFO management/cmd/management.go:462: overriding HttpConfig.AuthIssuer with a new value https://netbird.****.com, previously configured value: https://netbird.****.com
2024-05-24T09:23:23Z INFO management/cmd/management.go:466: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://netbird.****.com/oauth/v2/keys, previously configured value: https://netbird.****.com/oauth/v2/keys
2024-05-24T09:23:23Z INFO management/cmd/management.go:471: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://netbird.****.com/oauth/v2/token, previously configured value: https://netbird.****.com/oauth/v2/token
2024-05-24T09:23:23Z INFO management/cmd/management.go:474: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://netbird.****.com/oauth/v2/device_authorization, previously configured value: https://netbird.****.com/oauth/v2/device_authorization
2024-05-24T09:23:23Z INFO management/cmd/management.go:482: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: netbird.****.com, previously configured value: netbird.****.com
2024-05-24T09:23:23Z INFO management/cmd/management.go:492: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://netbird.****.com/oauth/v2/token, previously configured value: https://netbird.****.com/oauth/v2/token
2024-05-24T09:23:23Z INFO management/cmd/management.go:495: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://netbird.****.com/oauth/v2/authorize, previously configured value: https://netbird.****.com/oauth/v2/authorize
2024-05-24T09:23:23Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
2024-05-24T09:23:23Z INFO management/server/store.go:95: using SQLite store engine
2024-05-24T09:23:23Z DEBG management/server/migration/migration.go:41: No records in table accounts, no migration needed
2024-05-24T09:23:23Z DEBG management/server/migration/migration.go:41: No records in table routes, no migration needed
2024-05-24T09:23:23Z DEBG management/server/migration/migration.go:41: No records in table routes, no migration needed
2024-05-24T09:23:23Z INFO management/server/migration/migration.go:128: No records in table peers, no migration needed
2024-05-24T09:23:23Z INFO management/server/migration/migration.go:128: No records in table peers, no migration needed
2024-05-24T09:23:23Z DEBG management/server/activity/sqlite/sqlite.go:328: check deleted_users table version

@shadowofmoo commented on GitHub (May 24, 2024): management log ``` 2024-05-24T09:23:18Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://netbird.****.com/.well-known/openid-configuration Get "https://netbird.****.com/.well-known/openid-configuration": dial tcp 150.158.51.44:443: connect: connection refused 2024-05-24T09:23:18Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: 2024-05-24T09:23:19Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: 2024-05-24T09:23:20Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: 2024-05-24T09:23:21Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: OIDC configuration request returned status 502 with response: 2024-05-24T09:23:23Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://netbird.****.com/.well-known/openid-configuration 2024-05-24T09:23:23Z INFO management/cmd/management.go:460: loaded OIDC configuration from the provided IDP configuration endpoint: https://netbird.****.com/.well-known/openid-configuration 2024-05-24T09:23:23Z INFO management/cmd/management.go:462: overriding HttpConfig.AuthIssuer with a new value https://netbird.****.com, previously configured value: https://netbird.****.com 2024-05-24T09:23:23Z INFO management/cmd/management.go:466: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://netbird.****.com/oauth/v2/keys, previously configured value: https://netbird.****.com/oauth/v2/keys 2024-05-24T09:23:23Z INFO management/cmd/management.go:471: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://netbird.****.com/oauth/v2/token, previously configured value: https://netbird.****.com/oauth/v2/token 2024-05-24T09:23:23Z INFO management/cmd/management.go:474: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://netbird.****.com/oauth/v2/device_authorization, previously configured value: https://netbird.****.com/oauth/v2/device_authorization 2024-05-24T09:23:23Z INFO management/cmd/management.go:482: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: netbird.****.com, previously configured value: netbird.****.com 2024-05-24T09:23:23Z INFO management/cmd/management.go:492: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://netbird.****.com/oauth/v2/token, previously configured value: https://netbird.****.com/oauth/v2/token 2024-05-24T09:23:23Z INFO management/cmd/management.go:495: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://netbird.****.com/oauth/v2/authorize, previously configured value: https://netbird.****.com/oauth/v2/authorize 2024-05-24T09:23:23Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081 2024-05-24T09:23:23Z INFO management/server/store.go:95: using SQLite store engine 2024-05-24T09:23:23Z DEBG management/server/migration/migration.go:41: No records in table accounts, no migration needed 2024-05-24T09:23:23Z DEBG management/server/migration/migration.go:41: No records in table routes, no migration needed 2024-05-24T09:23:23Z DEBG management/server/migration/migration.go:41: No records in table routes, no migration needed 2024-05-24T09:23:23Z INFO management/server/migration/migration.go:128: No records in table peers, no migration needed 2024-05-24T09:23:23Z INFO management/server/migration/migration.go:128: No records in table peers, no migration needed 2024-05-24T09:23:23Z DEBG management/server/activity/sqlite/sqlite.go:328: check deleted_users table version ```
Author
Owner

@shadowofmoo commented on GitHub (May 24, 2024):

and i found maybe zitadel problem, this log

  _____  ___   _____      _      ____    _____   _
 |__  / |_ _| |_   _|    / \    |  _ \  | ____| | |
   / /   | |    | |     / _ \   | | | | |  _|   | |
  / /_   | |    | |    / ___ \  | |_| | | |___  | |___
 /____| |___|   |_|   /_/   \_\ |____/  |_____| |_____|

 ===============================================================

 Version          : v2.31.3
 TLS enabled      : false
 External Secure  : true
 Console URL      : https://netbird.****.com:443/ui/console
 Health Check URL : https://netbird.****.com:443/debug/healthz

 ===============================================================

time="2024-05-24T09:23:22Z" level=info msg="spooler started" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/spooler/spooler.go:73" lockerID=268513641651765254 workers=1
time="2024-05-24T09:23:22Z" level=info msg="spooler started" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/spooler/spooler.go:73" lockerID=268513641651830790 workers=1
time="2024-05-24T09:23:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:23:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:23:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:23:23Z" level=info msg="server is listening on [::]:8080" caller="/home/runner/work/zitadel/zitadel/cmd/start/start.go:427"
time="2024-05-24T09:23:23Z" level=warning msg="missing translation" args="map[ChangeDate:2024-05-24 08:48:09.94331 +0000 UTC CreationDate:2024-05-24 08:48:09.926895 +0000 UTC DisplayName:Zitadel Admin FirstName:Zitadel LastEmail:admin@netbird.****.com LastName:Admin LastPhone: LoginNames:[admin@netbird.****.com] NickName: PreferredLoginName:admin@netbird.****.com UserName:admin@netbird.****.com VerifiedEmail:admin@netbird.****.com VerifiedPhone:]" caller="/home/runner/work/zitadel/zitadel/internal/i18n/i18n.go:210" error="message \"PasswordChange.Footer\" not found in language \"en\"" id=PasswordChange.Footer
time="2024-05-24T09:23:23Z" level=debug msg="initializing SMTP channel failed" caller="/home/runner/work/zitadel/zitadel/internal/notification/senders/email.go:30" error="ID=QUERY-fwofw Message=Errors.SMTPConfig.NotFound Parent=(sql: no rows in result set)" instance=268510026279157764
time="2024-05-24T09:23:23Z" level=error msg="trigger failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:305" error="ID=MAIL-83nof Message=Errors.Notification.Channels.NotPresent" instanceIDs="[268510026279157764]" projection=projections.notifications
time="2024-05-24T09:23:23Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:23:24Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:24:00Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165
time="2024-05-24T09:24:01Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165
time="2024-05-24T09:24:01Z" level=debug msg="statement dropped" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:247" currentSequence="&{268510026279157764 167}" statement="&{user 166 165 268510026279157764 <nil>}"
time="2024-05-24T09:24:01Z" level=debug msg="statement dropped" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:247" currentSequence="&{268510026279157764 167}" statement="&{user 167 166 268510026279157764 <nil>}"
time="2024-05-24T09:24:02Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165
time="2024-05-24T09:24:03Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165
time="2024-05-24T09:24:04Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165
time="2024-05-24T09:24:05Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165
time="2024-05-24T09:24:06Z" level=warning msg="unable to process all events from subscription" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:229" error="some statements failed" projection=projections.notifications
time="2024-05-24T09:24:06Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166
time="2024-05-24T09:24:07Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166
time="2024-05-24T09:24:07Z" level=debug msg="statement dropped" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:247" currentSequence="&{268510026279157764 169}" statement="&{user 168 167 268510026279157764 <nil>}"
time="2024-05-24T09:24:07Z" level=debug msg="statement dropped" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:247" currentSequence="&{268510026279157764 169}" statement="&{user 169 168 268510026279157764 <nil>}"
time="2024-05-24T09:24:07Z" level=warning msg="unable to process all events from subscription" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:229" error="<nil>" projection=projections.users8
time="2024-05-24T09:24:08Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166
time="2024-05-24T09:24:09Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166
time="2024-05-24T09:24:10Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166
time="2024-05-24T09:24:11Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166
time="2024-05-24T09:24:12Z" level=warning msg="unable to process all events from subscription" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:229" error="some statements failed" projection=projections.notifications
time="2024-05-24T09:24:12Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168
time="2024-05-24T09:24:13Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168
time="2024-05-24T09:24:14Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168
time="2024-05-24T09:24:15Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168
time="2024-05-24T09:24:16Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168
time="2024-05-24T09:24:17Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168
time="2024-05-24T09:24:18Z" level=warning msg="unable to process all events from subscription" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:229" error="some statements failed" projection=projections.notifications
time="2024-05-24T09:24:23Z" level=warning msg="missing translation" args="map[ChangeDate:2024-05-24 08:48:09.94331 +0000 UTC CreationDate:2024-05-24 08:48:09.926895 +0000 UTC DisplayName:Zitadel Admin FirstName:Zitadel LastEmail:admin@netbird.****.com LastName:Admin LastPhone: LoginNames:[admin@netbird.****.com] NickName: PreferredLoginName:admin@netbird.****.com UserName:admin@netbird.****.com VerifiedEmail:admin@netbird.****.com VerifiedPhone:]" caller="/home/runner/work/zitadel/zitadel/internal/i18n/i18n.go:210" error="message \"PasswordChange.Footer\" not found in language \"en\"" id=PasswordChange.Footer
time="2024-05-24T09:24:23Z" level=debug msg="initializing SMTP channel failed" caller="/home/runner/work/zitadel/zitadel/internal/notification/senders/email.go:30" error="ID=QUERY-fwofw Message=Errors.SMTPConfig.NotFound Parent=(sql: no rows in result set)" instance=268510026279157764
time="2024-05-24T09:24:23Z" level=error msg="trigger failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:305" error="ID=MAIL-83nof Message=Errors.Notification.Channels.NotPresent" instanceIDs="[268510026279157764]" projection=projections.notifications
time="2024-05-24T09:25:23Z" level=warning msg="missing translation" args="map[ChangeDate:2024-05-24 08:48:09.94331 +0000 UTC CreationDate:2024-05-24 08:48:09.926895 +0000 UTC DisplayName:Zitadel Admin FirstName:Zitadel LastEmail:admin@netbird.****.com LastName:Admin LastPhone: LoginNames:[admin@netbird.****.com] NickName: PreferredLoginName:admin@netbird.****.com UserName:admin@netbird.****.com VerifiedEmail:admin@netbird.****.com VerifiedPhone:]" caller="/home/runner/work/zitadel/zitadel/internal/i18n/i18n.go:210" error="message \"PasswordChange.Footer\" not found in language \"en\"" id=PasswordChange.Footer
time="2024-05-24T09:25:23Z" level=debug msg="initializing SMTP channel failed" caller="/home/runner/work/zitadel/zitadel/internal/notification/senders/email.go:30" error="ID=QUERY-fwofw Message=Errors.SMTPConfig.NotFound Parent=(sql: no rows in result set)" instance=268510026279157764
time="2024-05-24T09:25:23Z" level=error msg="trigger failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:305" error="ID=MAIL-83nof Message=Errors.Notification.Channels.NotPresent" instanceIDs="[268510026279157764]" projection=projections.notifications
time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions
time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2
time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2
time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens
time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens
time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions

@shadowofmoo commented on GitHub (May 24, 2024): and i found maybe zitadel problem, this log ``` _____ ___ _____ _ ____ _____ _ |__ / |_ _| |_ _| / \ | _ \ | ____| | | / / | | | | / _ \ | | | | | _| | | / /_ | | | | / ___ \ | |_| | | |___ | |___ /____| |___| |_| /_/ \_\ |____/ |_____| |_____| =============================================================== Version : v2.31.3 TLS enabled : false External Secure : true Console URL : https://netbird.****.com:443/ui/console Health Check URL : https://netbird.****.com:443/debug/healthz =============================================================== time="2024-05-24T09:23:22Z" level=info msg="spooler started" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/spooler/spooler.go:73" lockerID=268513641651765254 workers=1 time="2024-05-24T09:23:22Z" level=info msg="spooler started" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/spooler/spooler.go:73" lockerID=268513641651830790 workers=1 time="2024-05-24T09:23:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:23:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:23:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:23:23Z" level=info msg="server is listening on [::]:8080" caller="/home/runner/work/zitadel/zitadel/cmd/start/start.go:427" time="2024-05-24T09:23:23Z" level=warning msg="missing translation" args="map[ChangeDate:2024-05-24 08:48:09.94331 +0000 UTC CreationDate:2024-05-24 08:48:09.926895 +0000 UTC DisplayName:Zitadel Admin FirstName:Zitadel LastEmail:admin@netbird.****.com LastName:Admin LastPhone: LoginNames:[admin@netbird.****.com] NickName: PreferredLoginName:admin@netbird.****.com UserName:admin@netbird.****.com VerifiedEmail:admin@netbird.****.com VerifiedPhone:]" caller="/home/runner/work/zitadel/zitadel/internal/i18n/i18n.go:210" error="message \"PasswordChange.Footer\" not found in language \"en\"" id=PasswordChange.Footer time="2024-05-24T09:23:23Z" level=debug msg="initializing SMTP channel failed" caller="/home/runner/work/zitadel/zitadel/internal/notification/senders/email.go:30" error="ID=QUERY-fwofw Message=Errors.SMTPConfig.NotFound Parent=(sql: no rows in result set)" instance=268510026279157764 time="2024-05-24T09:23:23Z" level=error msg="trigger failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:305" error="ID=MAIL-83nof Message=Errors.Notification.Channels.NotPresent" instanceIDs="[268510026279157764]" projection=projections.notifications time="2024-05-24T09:23:23Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:23:24Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:24:00Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165 time="2024-05-24T09:24:01Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165 time="2024-05-24T09:24:01Z" level=debug msg="statement dropped" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:247" currentSequence="&{268510026279157764 167}" statement="&{user 166 165 268510026279157764 <nil>}" time="2024-05-24T09:24:01Z" level=debug msg="statement dropped" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:247" currentSequence="&{268510026279157764 167}" statement="&{user 167 166 268510026279157764 <nil>}" time="2024-05-24T09:24:02Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165 time="2024-05-24T09:24:03Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165 time="2024-05-24T09:24:04Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165 time="2024-05-24T09:24:05Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=164 projection=projections.notifications sequence=165 time="2024-05-24T09:24:06Z" level=warning msg="unable to process all events from subscription" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:229" error="some statements failed" projection=projections.notifications time="2024-05-24T09:24:06Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166 time="2024-05-24T09:24:07Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166 time="2024-05-24T09:24:07Z" level=debug msg="statement dropped" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:247" currentSequence="&{268510026279157764 169}" statement="&{user 168 167 268510026279157764 <nil>}" time="2024-05-24T09:24:07Z" level=debug msg="statement dropped" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:247" currentSequence="&{268510026279157764 169}" statement="&{user 169 168 268510026279157764 <nil>}" time="2024-05-24T09:24:07Z" level=warning msg="unable to process all events from subscription" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:229" error="<nil>" projection=projections.users8 time="2024-05-24T09:24:08Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166 time="2024-05-24T09:24:09Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166 time="2024-05-24T09:24:10Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166 time="2024-05-24T09:24:11Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=165 projection=projections.notifications sequence=166 time="2024-05-24T09:24:12Z" level=warning msg="unable to process all events from subscription" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:229" error="some statements failed" projection=projections.notifications time="2024-05-24T09:24:12Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168 time="2024-05-24T09:24:13Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168 time="2024-05-24T09:24:14Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168 time="2024-05-24T09:24:15Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168 time="2024-05-24T09:24:16Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168 time="2024-05-24T09:24:17Z" level=warning msg="sequences do not match" aggregateType=user caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/crdb/handler_stmt.go:256" currentSeq=139 prevSeq=167 projection=projections.notifications sequence=168 time="2024-05-24T09:24:18Z" level=warning msg="unable to process all events from subscription" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:229" error="some statements failed" projection=projections.notifications time="2024-05-24T09:24:23Z" level=warning msg="missing translation" args="map[ChangeDate:2024-05-24 08:48:09.94331 +0000 UTC CreationDate:2024-05-24 08:48:09.926895 +0000 UTC DisplayName:Zitadel Admin FirstName:Zitadel LastEmail:admin@netbird.****.com LastName:Admin LastPhone: LoginNames:[admin@netbird.****.com] NickName: PreferredLoginName:admin@netbird.****.com UserName:admin@netbird.****.com VerifiedEmail:admin@netbird.****.com VerifiedPhone:]" caller="/home/runner/work/zitadel/zitadel/internal/i18n/i18n.go:210" error="message \"PasswordChange.Footer\" not found in language \"en\"" id=PasswordChange.Footer time="2024-05-24T09:24:23Z" level=debug msg="initializing SMTP channel failed" caller="/home/runner/work/zitadel/zitadel/internal/notification/senders/email.go:30" error="ID=QUERY-fwofw Message=Errors.SMTPConfig.NotFound Parent=(sql: no rows in result set)" instance=268510026279157764 time="2024-05-24T09:24:23Z" level=error msg="trigger failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:305" error="ID=MAIL-83nof Message=Errors.Notification.Channels.NotPresent" instanceIDs="[268510026279157764]" projection=projections.notifications time="2024-05-24T09:25:23Z" level=warning msg="missing translation" args="map[ChangeDate:2024-05-24 08:48:09.94331 +0000 UTC CreationDate:2024-05-24 08:48:09.926895 +0000 UTC DisplayName:Zitadel Admin FirstName:Zitadel LastEmail:admin@netbird.****.com LastName:Admin LastPhone: LoginNames:[admin@netbird.****.com] NickName: PreferredLoginName:admin@netbird.****.com UserName:admin@netbird.****.com VerifiedEmail:admin@netbird.****.com VerifiedPhone:]" caller="/home/runner/work/zitadel/zitadel/internal/i18n/i18n.go:210" error="message \"PasswordChange.Footer\" not found in language \"en\"" id=PasswordChange.Footer time="2024-05-24T09:25:23Z" level=debug msg="initializing SMTP channel failed" caller="/home/runner/work/zitadel/zitadel/internal/notification/senders/email.go:30" error="ID=QUERY-fwofw Message=Errors.SMTPConfig.NotFound Parent=(sql: no rows in result set)" instance=268510026279157764 time="2024-05-24T09:25:23Z" level=error msg="trigger failed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/handler_projection.go:305" error="ID=MAIL-83nof Message=Errors.Notification.Channels.NotPresent" instanceIDs="[268510026279157764]" projection=projections.notifications time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:26:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:29:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:32:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:35:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:38:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:41:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:44:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:47:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:50:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:53:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:56:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.users2 time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651830790--0 logID=LOCKE-lOgbg view=adminapi.styling2 time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.refresh_tokens time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.tokens time="2024-05-24T09:59:22Z" level=debug msg="locker changed" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/v1/locker/lock.go:44" locker=268513641651765254--0 logID=LOCKE-lOgbg view=auth.user_sessions ```
Author
Owner

@KeyRotate commented on GitHub (May 26, 2024):

I'm having the same problem, I've noticed that it occasionally doesn't get the 502 error and then I can get into the dashboard , but most of the time I get the 502 error.

It's very likely that it has something to do with the network, as deployment on Google Cloud won't have similar issues, but it will have this problem in the Chinese network environment. Maybe some workaround is needed

@KeyRotate commented on GitHub (May 26, 2024): I'm having the same problem, I've noticed that it occasionally doesn't get the 502 error and then I can get into the dashboard , but most of the time I get the 502 error. It's very likely that it has something to do with the network, as deployment on Google Cloud won't have similar issues, but it will have this problem in the Chinese network environment. Maybe some workaround is needed
Author
Owner

@Crownor commented on GitHub (May 30, 2024):

Seems management pod stuck in "check deleted_users table version", and not be ready for incoming requests

@Crownor commented on GitHub (May 30, 2024): Seems management pod stuck in "check deleted_users table version", and not be ready for incoming requests
Author
Owner

@keac commented on GitHub (Jun 1, 2024):

If you are using servers from Tencent Cloud, there is a high probability that you will encounter network issues.
https://github.com/netbirdio/netbird/blob/main/management/cmd/management.go#L169

geo, err := geolocation.NewGeolocation(config.Datadir)

https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L25
Ther need download

geoLiteCityTarGZURL     = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City/download?suffix=tar.gz"
geoLiteCityZipURL       = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City-CSV/download?suffix=zip"

However, this download process provides no logs or prompts, so it appears to be stuck or frozen!

Download file and make GeoLite2-City-Locations-en.csv to geonames.db
https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L55

And put into /var/lib/docker/volumes/artifacts_netbird-mgmt/_data
image

Afterwards, you should be able to start your service normally.

@keac commented on GitHub (Jun 1, 2024): If you are using servers from Tencent Cloud, there is a high probability that you will encounter network issues. https://github.com/netbirdio/netbird/blob/main/management/cmd/management.go#L169 ``` geo, err := geolocation.NewGeolocation(config.Datadir) ``` https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L25 Ther need download ``` geoLiteCityTarGZURL = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City/download?suffix=tar.gz" geoLiteCityZipURL = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City-CSV/download?suffix=zip" ``` However, this download process provides no logs or prompts, so it appears to be stuck or frozen! Download file and make GeoLite2-City-Locations-en.csv to geonames.db https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L55 And put into /var/lib/docker/volumes/artifacts_netbird-mgmt/_data ![image](https://github.com/netbirdio/netbird/assets/16091665/03cd1c4c-f7f4-4180-bdd2-e8c25992654e) Afterwards, you should be able to start your service normally.
Author
Owner

@SJFCS commented on GitHub (Jun 12, 2024):

如果你使用的是腾讯云的服务器,那么很有可能会遇到网络问题。https ://github.com/netbirdio/netbird/blob/main/management/cmd/management.go#L169

geo, err := geolocation.NewGeolocation(config.Datadir)

https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L25 需要下载

geoLiteCityTarGZURL     = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City/download?suffix=tar.gz"
geoLiteCityZipURL       = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City-CSV/download?suffix=zip"

但是,这个下载过程没有提供任何日志或提示,所以它似乎被卡住了或冻结了!

下载文件并将 GeoLite2-City-Locations-en.csv 制作为 geonames.db https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L55

并放入 /var/lib/docker/volumes/artifacts_netbird-mgmt/_data 图像

此后,您应该能够正常启动您的服务。

如果你使用的是腾讯云的服务器,那么很有可能会遇到网络问题。https ://github.com/netbirdio/netbird/blob/main/management/cmd/management.go#L169

geo, err := geolocation.NewGeolocation(config.Datadir)

https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L25 需要下载

geoLiteCityTarGZURL     = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City/download?suffix=tar.gz"
geoLiteCityZipURL       = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City-CSV/download?suffix=zip"

但是,这个下载过程没有提供任何日志或提示,所以它似乎被卡住了或冻结了!

下载文件并将 GeoLite2-City-Locations-en.csv 制作为 geonames.db https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L55

并放入 /var/lib/docker/volumes/artifacts_netbird-mgmt/_data 图像

此后,您应该能够正常启动您的服务。

You saved my life ! !
Thank you for helping me save a lot of time
I have a little bit to add:After installing with Getting-Started-With-Zitadel.sh, the Management service default mounting path is /var/lib/docker/volumes/netbird_netbird_management/_data

root@archlinux volumes/netbird_netbird_management/_data ➜  ls
events.db  GeoLite2-City.mmdb  geonames.db  store.db
@SJFCS commented on GitHub (Jun 12, 2024): > 如果你使用的是腾讯云的服务器,那么很有可能会遇到网络问题。https [://github.com/netbirdio/netbird/blob/main/management/cmd/management.go#L169](https://github.com/netbirdio/netbird/blob/main/management/cmd/management.go#L169) > > ``` > geo, err := geolocation.NewGeolocation(config.Datadir) > ``` > > https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L25 需要下载 > > ``` > geoLiteCityTarGZURL = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City/download?suffix=tar.gz" > geoLiteCityZipURL = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City-CSV/download?suffix=zip" > ``` > > 但是,这个下载过程没有提供任何日志或提示,所以它似乎被卡住了或冻结了! > > 下载文件并将 GeoLite2-City-Locations-en.csv 制作为 geonames.db https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L55 > > 并放入 /var/lib/docker/volumes/artifacts_netbird-mgmt/_data ![图像](https://private-user-images.githubusercontent.com/16091665/335785916-03cd1c4c-f7f4-4180-bdd2-e8c25992654e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTgyMDIzOTIsIm5iZiI6MTcxODIwMjA5MiwicGF0aCI6Ii8xNjA5MTY2NS8zMzU3ODU5MTYtMDNjZDFjNGMtZjdmNC00MTgwLWJkZDItZThjMjU5OTI2NTRlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjEyVDE0MjEzMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTAxMTk5ODE0OWY4OWUxMjFlMDgwNWUyYTViNmU1Y2JmNDk4ZjczZDk1N2QwMGVjNTA3M2Y4N2UxZDI1ZDRmYzMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.31ZDVOS3l4dxSPoPc6ewB2uTMU1REqfRG4jLDMqcNOU) > > 此后,您应该能够正常启动您的服务。 > 如果你使用的是腾讯云的服务器,那么很有可能会遇到网络问题。https [://github.com/netbirdio/netbird/blob/main/management/cmd/management.go#L169](https://github.com/netbirdio/netbird/blob/main/management/cmd/management.go#L169) > > ``` > geo, err := geolocation.NewGeolocation(config.Datadir) > ``` > > https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L25 需要下载 > > ``` > geoLiteCityTarGZURL = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City/download?suffix=tar.gz" > geoLiteCityZipURL = "https://pkgs.netbird.io/geolocation-dbs/GeoLite2-City-CSV/download?suffix=zip" > ``` > > 但是,这个下载过程没有提供任何日志或提示,所以它似乎被卡住了或冻结了! > > 下载文件并将 GeoLite2-City-Locations-en.csv 制作为 geonames.db https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L55 > > 并放入 /var/lib/docker/volumes/artifacts_netbird-mgmt/_data ![图像](https://private-user-images.githubusercontent.com/16091665/335785916-03cd1c4c-f7f4-4180-bdd2-e8c25992654e.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTgyMDIzOTIsIm5iZiI6MTcxODIwMjA5MiwicGF0aCI6Ii8xNjA5MTY2NS8zMzU3ODU5MTYtMDNjZDFjNGMtZjdmNC00MTgwLWJkZDItZThjMjU5OTI2NTRlLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjEyVDE0MjEzMlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTAxMTk5ODE0OWY4OWUxMjFlMDgwNWUyYTViNmU1Y2JmNDk4ZjczZDk1N2QwMGVjNTA3M2Y4N2UxZDI1ZDRmYzMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.31ZDVOS3l4dxSPoPc6ewB2uTMU1REqfRG4jLDMqcNOU) > > 此后,您应该能够正常启动您的服务。 You saved my life ! ! Thank you for helping me save a lot of time I have a little bit to add:After installing with Getting-Started-With-Zitadel.sh, the Management service default mounting path is `/var/lib/docker/volumes/netbird_netbird_management/_data` ```bash root@archlinux volumes/netbird_netbird_management/_data ➜ ls events.db GeoLite2-City.mmdb geonames.db store.db ```
Author
Owner

@danielalexis commented on GitHub (Mar 31, 2025):

I have the same problem, attached is the log for the management pod.

Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:12Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:13Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:13Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:15Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:16Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:20Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:26Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:39Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:07:05Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:07:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:08:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:09:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:01Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:02Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:03Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:03Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:04Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:05Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:08Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:15Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:28Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:54Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:12:45Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:13:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:14:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:15:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:16:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused```
@danielalexis commented on GitHub (Mar 31, 2025): I have the same problem, attached is the log for the management pod. ```2025-03-31T18:06:12Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:06:12Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:06:13Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:06:13Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:06:15Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:06:16Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:06:20Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:06:26Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:06:39Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:07:05Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:07:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:08:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:09:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:01Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:02Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:03Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:03Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:04Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:05Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:08Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:15Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:28Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:11:54Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:12:45Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:13:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:14:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:15:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused 2025-03-31T18:16:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused```
Author
Owner

@mitcharseneau commented on GitHub (Apr 10, 2025):

I have the same issue, tested with versions 0.40.1, 0.40.0, 0.39.1

@mitcharseneau commented on GitHub (Apr 10, 2025): I have the same issue, tested with versions 0.40.1, 0.40.0, 0.39.1
Author
Owner

@arkylin commented on GitHub (Apr 14, 2025):

The same issue.

@arkylin commented on GitHub (Apr 14, 2025): The same issue.
Author
Owner

@djex commented on GitHub (Oct 2, 2025):

failed reading provided config

I have the same problem, attached is the log for the management pod.

Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:12Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:13Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:13Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:15Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:16Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:20Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:26Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:06:39Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:07:05Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:07:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:08:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:09:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:01Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:02Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:03Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:03Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:04Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:05Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:08Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:15Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:28Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:11:54Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:12:45Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:13:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:14:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:15:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused
2025-03-31T18:16:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused```

I had this exact same issue. This fixed it: https://github.com/netbirdio/netbird/issues/2226#issuecomment-2230020459

@djex commented on GitHub (Oct 2, 2025): > failed reading provided config > I have the same problem, attached is the log for the management pod. > > ``` > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:06:12Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:06:13Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:06:13Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:06:15Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:06:16Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:06:20Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:06:26Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:06:39Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:07:05Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:07:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:08:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:09:57Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:01Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:02Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:03Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:03Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:04Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:05Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:08Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:15Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:28Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:11:54Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:12:45Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:13:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:14:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:15:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused > 2025-03-31T18:16:46Z INFO [context: SYSTEM] management/cmd/management.go:512: loading OIDC configuration from the provided IDP configuration endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration > Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://vpn.hestiatechnology.pt/.well-known/openid-configuration Get "https://vpn.hestiatechnology.pt/.well-known/openid-configuration": dial tcp 127.0.1.1:443: connect: connection refused``` > ``` I had this exact same issue. This fixed it: https://github.com/netbirdio/netbird/issues/2226#issuecomment-2230020459
Author
Owner

@freddong commented on GitHub (Oct 6, 2025):

  1. find the missing files
$ docker compose logs -n 10 -f management
Geolocation database file geonames_20250905.db not found, file will be downloaded
  1. download the files
  2. move to the docker volume
    find the path
docker volume ls
docker volume inspect

copy files
then docker compose restart management.

@freddong commented on GitHub (Oct 6, 2025): 1. find the missing files ``` $ docker compose logs -n 10 -f management Geolocation database file geonames_20250905.db not found, file will be downloaded ``` 2. download the files - download from [Management Geolocation - NetBird Docs](https://docs.netbird.io/selfhosted/geo-support#downloading-the-databases) - extract "GeoLite2-City.mmdb" from "GeoLite2 City Database", rename to "GeoLite2-City_20250905.mmdb" - extract "GeoLite2-City-Locations-en.csv" from "GeoLite2 City CSV Database", convert to sqlite db(you can ask ai to write a script like [netbird/management/server/geolocation/database.go at main · netbirdio/netbird](https://github.com/netbirdio/netbird/blob/main/management/server/geolocation/database.go#L55)), then rename to "geonames_20250905.db" 3. move to the docker volume find the path ``` docker volume ls docker volume inspect ``` copy files then `docker compose restart management`.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#924