SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-07-01 11:42:00 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Management container error after updating to 0.46.0 #1954

New Issue
Closed
opened 2025-11-20 06:10:11 -05:00 by saavagebueno · 10 comments
saavagebueno commented 2025-11-20 06:10:11 -05:00
Owner
Copy Link

Originally created by @proteus88 on GitHub (Jun 10, 2025).

Describe the problem

After updateing to 0.46.0 the management container is not coming up

Management logs:

2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/store/store.go:256: using SQLite store engine 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/store/sql_store.go:89: Set max open db connections to 1 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:257: No plain setup keys found in table setup_keys, no migration needed 2025-06-08T19:14:37Z INFO management/server/migration/migration.go:295: Migration of plain setup key to hashed setup key completed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty enabled found in table network_resources, no migration needed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty enabled to default value in table network_resources completed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty enabled found in table network_routers, no migration needed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty enabled to default value in table network_routers completed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/activity/store/sql_store.go:260: using sqlite as activity event store engine 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty name found in table deleted_users, no migration needed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty name to default value in table deleted_users completed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty enc_algo found in table deleted_users, no migration needed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty enc_algo to default value in table deleted_users completed Error: failed to initialize database: events database migration: failed to migrate deleted user b98f8cb97dcdcb688606c9079be333495810dcb4d38c16ce6f149e718bc82db5: failed to decrypt email: invalid padding size

A clear and concise description of what the problem is.

To Reproduce
Update to 0.46.0

Are you using NetBird Cloud?
No

Please specify whether you use NetBird Cloud or self-host NetBird's control plane.

NetBird version

  • self-host - 0.43.2
  • authentik
Originally created by @proteus88 on GitHub (Jun 10, 2025). **Describe the problem** After updateing to 0.46.0 the management container is not coming up Management logs: `2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/store/store.go:256: using SQLite store engine 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/store/sql_store.go:89: Set max open db connections to 1 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:257: No plain setup keys found in table setup_keys, no migration needed 2025-06-08T19:14:37Z INFO management/server/migration/migration.go:295: Migration of plain setup key to hashed setup key completed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty enabled found in table network_resources, no migration needed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty enabled to default value in table network_resources completed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty enabled found in table network_routers, no migration needed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty enabled to default value in table network_routers completed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/activity/store/sql_store.go:260: using sqlite as activity event store engine 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty name found in table deleted_users, no migration needed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty name to default value in table deleted_users completed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:338: No rows with empty enc_algo found in table deleted_users, no migration needed 2025-06-08T19:14:37Z INFO [context: SYSTEM] management/server/migration/migration.go:352: Migration of empty enc_algo to default value in table deleted_users completed Error: failed to initialize database: events database migration: failed to migrate deleted user b98f8cb97dcdcb688606c9079be333495810dcb4d38c16ce6f149e718bc82db5: failed to decrypt email: invalid padding size` A clear and concise description of what the problem is. **To Reproduce** Update to 0.46.0 **Are you using NetBird Cloud?** No Please specify whether you use NetBird Cloud or self-host NetBird's control plane. **NetBird version** - self-host - 0.43.2 - authentik
saavagebueno added the management-serviceself-hosting labels 2025-11-20 06:10:11 -05:00
saavagebueno commented 2025-11-20 06:10:12 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Jun 12, 2025):

Hello @proteus88, thanks for reporting the issue. This indicate a mismatch or corruption in the encryption key or encrypted data format.

To help debug this, could you please run the following commands and share the results?

  mkdir backup
  docker compose cp -a management:/var/lib/netbird/ backup/
  sqlite3 backup/events.db

Then inside the sqlite3 prompt run:

select count(*) from deleted_users where enc_algo = 'GCM';

and

select count(*) from deleted_users where enc_algo IS NULL OR enc_algo != 'GCM';

Also just to confirm, have you recently rotated or changed the DataStoreEncryptionKey value in your management.json config file?

@bcmmbaga commented on GitHub (Jun 12, 2025): Hello @proteus88, thanks for reporting the issue. This indicate a mismatch or corruption in the encryption key or encrypted data format. To help debug this, could you please run the following commands and share the results? ```sh mkdir backup docker compose cp -a management:/var/lib/netbird/ backup/ sqlite3 backup/events.db ``` Then inside the sqlite3 prompt run: ```sql select count(*) from deleted_users where enc_algo = 'GCM'; ``` and ```sql select count(*) from deleted_users where enc_algo IS NULL OR enc_algo != 'GCM'; ``` Also just to confirm, have you recently rotated or changed the `DataStoreEncryptionKey` value in your `management.json` config file?
saavagebueno commented 2025-11-20 06:10:12 -05:00
Author
Owner
Copy Link

@proteus88 commented on GitHub (Jun 12, 2025):

sqlite> select count(*) from deleted_users where enc_algo = 'GCM';
0

sqlite> select count(*) from deleted_users where enc_algo IS NULL OR enc_algo != 'GCM';
7

Recently I did not change "DataStoreEncryptionKey". Maybe when I installed it the first time last year and I don't remember...

@proteus88 commented on GitHub (Jun 12, 2025): ``` sqlite> select count(*) from deleted_users where enc_algo = 'GCM'; 0 ``` ``` sqlite> select count(*) from deleted_users where enc_algo IS NULL OR enc_algo != 'GCM'; 7 ``` Recently I did not change "DataStoreEncryptionKey". Maybe when I installed it the first time last year and I don't remember...
saavagebueno commented 2025-11-20 06:10:12 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Jun 12, 2025):

Since v0.29.3 all entries in the deleted_users table have been migrated to use GCM encryption. However based on your current db state, it appears that all data were not migrated even though you were previously running v0.43.2 before upgrading to v0.46.0. We would expect all entries to have been migrated while running v0.43.2.

Is there any chance that an old events.db backup was restored?

@bcmmbaga commented on GitHub (Jun 12, 2025): Since [v0.29.3](https://github.com/netbirdio/netbird/releases/tag/v0.29.3) all entries in the `deleted_users` table have been migrated to use GCM encryption. However based on your current db state, it appears that all data were not migrated even though you were previously running `v0.43.2` before upgrading to `v0.46.0`. We would expect all entries to have been migrated while running `v0.43.2`. Is there any chance that an old `events.db` backup was restored?
saavagebueno commented 2025-11-20 06:10:12 -05:00
Author
Owner
Copy Link

@proteus88 commented on GitHub (Jun 12, 2025):

Never restored anything related to NetBird.

EDIT:
I only restored the full VM when the updated failed to 0.46.0 but I don't think its relevant.

@proteus88 commented on GitHub (Jun 12, 2025): Never restored anything related to NetBird. EDIT: I only restored the full VM when the updated failed to 0.46.0 but I don't think its relevant.
saavagebueno commented 2025-11-20 06:10:12 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Jun 12, 2025):

If you're currently running management v0.43.2, can you share the logs?

docker compose logs management > mgmt.log 2>&1
@bcmmbaga commented on GitHub (Jun 12, 2025): If you're currently running management `v0.43.2`, can you share the logs? ```sh docker compose logs management > mgmt.log 2>&1 ```
saavagebueno commented 2025-11-20 06:10:13 -05:00
Author
Owner
Copy Link

@proteus88 commented on GitHub (Jun 12, 2025):

mgmt.log

@proteus88 commented on GitHub (Jun 12, 2025): [mgmt.log](https://github.com/user-attachments/files/20710130/mgmt.log)
saavagebueno commented 2025-11-20 06:10:13 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Jun 12, 2025):

mgmt.log

Thanks. The migration also failed on version v0.43.2 due to a change in the encryption key which made it impossible to decrypt the existing data. As a result the migration was skipped and not enforced.

Since the deleted_users table is only used for activity events, you can safely back up your current db and then clear the data from the deleted_users table. After that, you should be able to upgrade to the latest version without any issues.

DELETE FROM deleted_users;
@bcmmbaga commented on GitHub (Jun 12, 2025): > [mgmt.log](https://github.com/user-attachments/files/20710130/mgmt.log) Thanks. The migration also failed on version `v0.43.2` due to a change in the encryption key which made it impossible to decrypt the existing data. As a result the migration was skipped and not enforced. Since the `deleted_users` table is only used for activity events, you can safely back up your current db and then clear the data from the `deleted_users` table. After that, you should be able to upgrade to the latest version without any issues. ```sql DELETE FROM deleted_users; ```
saavagebueno commented 2025-11-20 06:10:13 -05:00
Author
Owner
Copy Link

@proteus88 commented on GitHub (Jun 12, 2025):

Thank you, updated to 0.46.0 without errors.

@proteus88 commented on GitHub (Jun 12, 2025): Thank you, updated to 0.46.0 without errors.
saavagebueno commented 2025-11-20 06:10:13 -05:00
Author
Owner
Copy Link

@bcmmbaga commented on GitHub (Jun 12, 2025):

Thank you, updated to 0.46.0 without errors.

Awesome. I’ll go ahead and close the issue now

@bcmmbaga commented on GitHub (Jun 12, 2025): > Thank you, updated to 0.46.0 without errors. Awesome. I’ll go ahead and close the issue now
saavagebueno commented 2025-11-20 06:10:13 -05:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Jun 25, 2025):

for future reference, it's best to run:

DELETE FROM deleted_users WHERE enc_algo IS NULL OR enc_algo != 'GCM';
@nazarewk commented on GitHub (Jun 25, 2025): for future reference, it's best to run: ```sql DELETE FROM deleted_users WHERE enc_algo IS NULL OR enc_algo != 'GCM'; ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label management-service self-hosting
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1954
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?