[GH-ISSUE #5681] Error accessing existing reverse proxy services #10922

Closed
opened 2026-08-05 01:27:43 -04:00 by saavagebueno · 7 comments
Owner

Originally created by @ghazyami on GitHub (Mar 24, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5681

Describe the problem

  • On the dashboard accessing reverse proxy services page, shows an empty list and an error notification is displayed Request failed with status code 500
  • This happened more than one time, the most recent is after update from 0.66.4  to 0.67.0
  • The proxy container shows this repeatedly
2026-03-24T03:50:56.513Z DEBG proxy/server.go:913: connecting to management mapping stream
2026-03-24T03:50:56.513Z DEBG proxy/server.go:936: management mapping stream established
2026-03-24T03:50:56.520Z WARN proxy/server.go:955: management connection failed, retrying in 294ms: mapping stream: receive msg: rpc error: code = Unknown desc = send snapshot to proxy netbird-proxy-20260323212726: get services from store: failed to get services: decrypt service data: decrypt: cipher: message authentication failed
  • Fixed by deleting all records in targets and services database tables (using postgres as DB in my setup)

To Reproduce

Steps to reproduce the behavior:
Not sure

Expected behavior

Reverse proxy services should remain accessible

Are you using NetBird Cloud?

Self-hosted

NetBird version

0.67.0

Is any other VPN software installed?

No

Debug output

N/A

Screenshots

Image

Additional context

Issues seems to be a decryption issue of existing data

Have you tried these troubleshooting steps?

  • Delete all records in targets and services database tables, and restart proxy container fixed it
Originally created by @ghazyami on GitHub (Mar 24, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5681 **Describe the problem** - On the dashboard accessing reverse proxy services page, shows an empty list and an error notification is displayed `Request failed with status code 500` - This happened more than one time, the most recent is after update from `0.66.4`  to `0.67.0` - The proxy container shows this repeatedly ``` 2026-03-24T03:50:56.513Z DEBG proxy/server.go:913: connecting to management mapping stream 2026-03-24T03:50:56.513Z DEBG proxy/server.go:936: management mapping stream established 2026-03-24T03:50:56.520Z WARN proxy/server.go:955: management connection failed, retrying in 294ms: mapping stream: receive msg: rpc error: code = Unknown desc = send snapshot to proxy netbird-proxy-20260323212726: get services from store: failed to get services: decrypt service data: decrypt: cipher: message authentication failed ``` - Fixed by deleting all records in `targets` and `services` database tables (using postgres as DB in my setup) **To Reproduce** Steps to reproduce the behavior: Not sure **Expected behavior** Reverse proxy services should remain accessible **Are you using NetBird Cloud?** Self-hosted **NetBird version** `0.67.0` **Is any other VPN software installed?** No **Debug output** N/A **Screenshots** ![Image](https://github.com/user-attachments/assets/cfefd158-7c36-4ec9-8c2e-6fd61a1bd313) **Additional context** Issues seems to be a decryption issue of existing data **Have you tried these troubleshooting steps?** - Delete all records in `targets` and `services` database tables, and restart proxy container fixed it
saavagebueno added the triage-needed label 2026-08-05 01:27:43 -04:00
Author
Owner

@ghazyami commented on GitHub (Mar 27, 2026):

This happened again on the same 0.67.0 version without update, there might have been a server reboot and container re-creation, but I am not sure. The only volume attached to the proxy container is /certs if the container needs anything after re-creation that is stored elsewhere or kept in-memory it will be lost and might explain the behavior

<!-- gh-comment-id:4145151172 --> @ghazyami commented on GitHub (Mar 27, 2026): This happened again on the same `0.67.0` version without update, there might have been a server reboot and container re-creation, but I am not sure. The only volume attached to the proxy container is `/certs` if the container needs anything after re-creation that is stored elsewhere or kept in-memory it will be lost and might explain the behavior
Author
Owner

@ghazyami commented on GitHub (Mar 28, 2026):

Now I am sure this happens on container delete and re-creation, which happens, on update or server reboot for me as I am using podman

<!-- gh-comment-id:4147928364 --> @ghazyami commented on GitHub (Mar 28, 2026): Now I am sure this happens on container delete and re-creation, which happens, on update or server reboot for me as I am using podman
Author
Owner

@NDebx commented on GitHub (Apr 30, 2026):

I have the same issue, my reverse proxy crashes if there's a new update of netbird and then when I manually update this and force recreate then it's fixed. But it's really annoying. @ghazyami

<!-- gh-comment-id:4352691928 --> @NDebx commented on GitHub (Apr 30, 2026): I have the same issue, my reverse proxy crashes if there's a new update of netbird and then when I manually update this and force recreate then it's fixed. But it's really annoying. @ghazyami
Author
Owner

@ghazyami commented on GitHub (May 14, 2026):

I isolated this to only the management container being deleted and re-created, that is when this happens for me.

<!-- gh-comment-id:4450703214 --> @ghazyami commented on GitHub (May 14, 2026): I isolated this to only the management container being deleted and re-created, that is when this happens for me.
Author
Owner

@supu2 commented on GitHub (May 18, 2026):

NetBird Management Configuration: Data Store Encryption
To ensure the security of sensitive data stored by NetBird's management service, you must explicitly configure a data store encryption key in your configuration file (typically management.json).

Required Configuration Line
Within the JSON configuration file for the NetBird management server, the following key-value pair must be present in the configuration object:

"DataStoreEncryptionKey": "${NETBIRD_DATASTORE_ENC_KEY}",
Purpose: This line tells the NetBird Management service to look for an encryption key. The value ${NETBIRD_DATASTORE_ENC_KEY} acts as a placeholder environment variable that will be evaluated at runtime, or it can be replaced directly with a hardcoded string.

Reference: This configuration mirrors the structure defined in the official NetBird template repository, specifically around line 28 of the template file (view source template).

Generating a Secure Encryption Key
NetBird requires a strong, cryptographically secure 256-bit key encoded in Base64. You can generate a compatible key using the OpenSSL command-line tool.

Run the following command in your terminal:

openssl rand -base64 32
How it works:

openssl rand: Calls OpenSSL's cryptographically secure pseudo-random number generator.

-base64: Encodes the output using standard Base64 encoding so it can be safely pasted into configuration files without breaking JSON formatting.

32: Requests 32 bytes of random data (equivalent to 256 bits, which is the standard size for AES-256 encryption).

Example Output:
Running this command will output a string similar to this:
dWp6NHJ0Z3JlZndzYWRmY3Z4emFzZGYxMjM0NTY3ODk=

Deployment Methods
Depending on how you run your NetBird Management service, you can apply this key in one of two ways:

Method A: Environment Variable (Recommended for Docker/Templates)
If you use the exact string "${NETBIRD_DATASTORE_ENC_KEY}" in your JSON, you must inject the key as an environment variable before starting the container or service.

Docker Compose Example:

services: netbird-management: image: netbirdio/management:latest environment: - NETBIRD_DATASTORE_ENC_KEY=YOUR_GENERATED_BASE64_KEY_HERE # ... rest of configuration

Method B: Direct Configuration Entry
Alternatively, you can replace the variable placeholder directly inside your management.json file with the generated key:

{ "ManagementVersion": "1.0", "DataStoreEncryptionKey": "YOUR_GENERATED_BASE64_KEY_HERE", "HttpConfig": { ... } }
⚠️ Important Security Note: Treat your generated encryption key with the same level of confidentiality as a root password. If this key is lost, the data store will become unreadable, and if it is compromised, unauthorized users could potentially decrypt your network's configuration data.

<!-- gh-comment-id:4478983474 --> @supu2 commented on GitHub (May 18, 2026): NetBird Management Configuration: Data Store Encryption To ensure the security of sensitive data stored by NetBird's management service, you must explicitly configure a data store encryption key in your configuration file (typically management.json). **Required Configuration Line** Within the JSON configuration file for the NetBird management server, the following key-value pair must be present in the configuration object: ` "DataStoreEncryptionKey": "${NETBIRD_DATASTORE_ENC_KEY}", ` Purpose: This line tells the NetBird Management service to look for an encryption key. The value ${NETBIRD_DATASTORE_ENC_KEY} acts as a placeholder environment variable that will be evaluated at runtime, or it can be replaced directly with a hardcoded string. Reference: This configuration mirrors the structure defined in the official NetBird template repository, specifically around line 28 of the template file ([view source template](https://github.com/supu2/netbird/blob/main/netbird/management.tmpl.json#L28)). **Generating a Secure Encryption Key** NetBird requires a strong, cryptographically secure 256-bit key encoded in Base64. You can generate a compatible key using the OpenSSL command-line tool. Run the following command in your terminal: ` openssl rand -base64 32 ` How it works: openssl rand: Calls OpenSSL's cryptographically secure pseudo-random number generator. -base64: Encodes the output using standard Base64 encoding so it can be safely pasted into configuration files without breaking JSON formatting. 32: Requests 32 bytes of random data (equivalent to 256 bits, which is the standard size for AES-256 encryption). Example Output: Running this command will output a string similar to this: dWp6NHJ0Z3JlZndzYWRmY3Z4emFzZGYxMjM0NTY3ODk= **Deployment Methods** Depending on how you run your NetBird Management service, you can apply this key in one of two ways: Method A: Environment Variable (Recommended for Docker/Templates) If you use the exact string "${NETBIRD_DATASTORE_ENC_KEY}" in your JSON, you must inject the key as an environment variable before starting the container or service. Docker Compose Example: ` services: netbird-management: image: netbirdio/management:latest environment: - NETBIRD_DATASTORE_ENC_KEY=YOUR_GENERATED_BASE64_KEY_HERE # ... rest of configuration ` Method B: Direct Configuration Entry Alternatively, you can replace the variable placeholder directly inside your management.json file with the generated key: ` { "ManagementVersion": "1.0", "DataStoreEncryptionKey": "YOUR_GENERATED_BASE64_KEY_HERE", "HttpConfig": { ... } } ` ⚠️ Important Security Note: Treat your generated encryption key with the same level of confidentiality as a root password. If this key is lost, the data store will become unreadable, and if it is compromised, unauthorized users could potentially decrypt your network's configuration data.
Author
Owner

@ghazyami commented on GitHub (May 18, 2026):

@supu2 missing DataStoreEncryptionKey value in management.json might be it, let me do some tests and confirm

<!-- gh-comment-id:4481821477 --> @ghazyami commented on GitHub (May 18, 2026): @supu2 missing `DataStoreEncryptionKey` value in `management.json` might be it, let me do some tests and confirm
Author
Owner

@ghazyami commented on GitHub (May 20, 2026):

I confirm this is fixed now, closing the issue

<!-- gh-comment-id:4496037475 --> @ghazyami commented on GitHub (May 20, 2026): I confirm this is fixed now, closing the issue
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#10922