[GH-ISSUE #5968] PUT /api/reverse-proxies/services/{id} always times out with 500 — SQLite self-deadlock in persistServiceUpdate #11969

Open
opened 2026-08-05 01:31:55 -04:00 by saavagebueno · 0 comments
Owner

Originally created by @kcwebby on GitHub (Apr 22, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5968

Describe the problem

Updating an existing reverse proxy service via the dashboard always fails with HTTP 500. The operation times out after exactly 5 minutes every time, regardless of server load or other activity. Creating a new service works correctly — only UPDATE is affected.

Root cause identified from stack trace: persistServiceUpdate calls ExecuteInTransaction, which acquires the single SQLite connection (max open db connections: 1), then attempts to call GetNetworkResourceByID from within that same transaction. This re-enters the store on the already-held connection, causing a self-deadlock. The transaction waits the full 5m0s timeout before failing.

To Reproduce

  1. Self-host the combined netbirdio/netbird-server container
  2. Create a reverse proxy service with a network resource (subnet type) as the target
  3. Confirm the service is created successfully
  4. Navigate to the reverse proxy service in the dashboard and make any change (e.g. update a field)
  5. Click Save/Update
  6. Observe HTTP 500 after exactly 5 minutes

Expected behavior

The reverse proxy service update should persist successfully and return HTTP 200.

Are you using NetBird Cloud?

No — self-hosted using the official netbirdio/netbird-server combined container, deployed via Docker Compose:

✔ Image netbirdio/netbird-server:latest Pulled   6.8s
 ✔ Image traefik:v2.11                   Pulled             6.3s
 ✔ Image netbirdio/dashboard:latest      Pulled     

Second (and third) node running only as relay and reverse proxy:

 ✔ Image netbirdio/relay:latest         Pulled          1.1s
 ✔ Image netbirdio/reverse-proxy:latest Pulled    1.5s
 ✔ Image traefik:v2.11                  Pulled  

NetBird version
Reproduced on both 0.66.4 and the latest image pulled on 2026-04-23 (which referenced golang.org/x/net@v0.52.0). The management server startup log reports:

management server version 0.66.4

Is any other VPN software installed?

No.

Debug output

This bug is on the server/management side — the netbird status and debug bundle commands apply to the client. The relevant diagnostic information is the server log stack trace, included below.

Screenshots

Dashboard error toast:

Code 500: Look up resource target "d7gkkk2m6v2c73clrgig": failed to get network resource from store

Image

Additional context

Server log stack trace (reproduced identically across two separate attempts, with proxies stable and connected throughout — not a timing/race issue):

ERRO management/server/store/sql_store.go:4287: failed to get network resource from store: context deadline exceeded
WARN management/server/store/sql_store.go:3430: transaction exceeded 5m0s timeout after 5m0.005638867s, stack:
github.com/netbirdio/netbird/management/server/store.(*SqlStore).ExecuteInTransaction
        management/server/store/sql_store.go:3430
github.com/netbirdio/netbird/management/internals/modules/reverseproxy/service/manager.(*Manager).persistServiceUpdate
        management/internals/modules/reverseproxy/service/manager/manager.go:541
github.com/netbirdio/netbird/management/internals/modules/reverseproxy/service/manager.(*Manager).UpdateService
        management/internals/modules/reverseproxy/service/manager/manager.go:506
github.com/netbirdio/netbird/management/internals/modules/reverseproxy/service/manager.(*handler).updateService
        management/internals/modules/reverseproxy/service/manager/api.go:152
ERRO shared/management/http/util/util.go:85: got a handler error: look up resource target "<resource_id>": failed to get network resource from store
ERRO management/server/telemetry/http_api_metrics.go:212: HTTP response: PUT /api/reverse-proxies/services/<service_id> status 500

Database state confirmed healthy — PRAGMA integrity_check returns ok, database size 1.43MB, no WAL files present, not in WAL mode. The network resource referenced by the target exists and is valid in the store. The issue is exclusively the re-entrant lock in persistServiceUpdate.

Startup config confirms single connection pool:

Set max open db connections to 1, max idle to 1

Workaround: Delete the service and recreate it with the desired settings. The create path does not reproduce the deadlock.

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (not applicable — server-side bug)
  • Checked for newer NetBird versions (reproduced on latest as of 2026-04-23)
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software (none installed)
  • Checked firewall settings
Originally created by @kcwebby on GitHub (Apr 22, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5968 **Describe the problem** Updating an existing reverse proxy service via the dashboard always fails with HTTP 500. The operation times out after exactly 5 minutes every time, regardless of server load or other activity. Creating a new service works correctly — only UPDATE is affected. Root cause identified from stack trace: `persistServiceUpdate` calls `ExecuteInTransaction`, which acquires the single SQLite connection (`max open db connections: 1`), then attempts to call `GetNetworkResourceByID` from within that same transaction. This re-enters the store on the already-held connection, causing a self-deadlock. The transaction waits the full 5m0s timeout before failing. **To Reproduce** 1. Self-host the combined `netbirdio/netbird-server` container 2. Create a reverse proxy service with a network resource (subnet type) as the target 3. Confirm the service is created successfully 4. Navigate to the reverse proxy service in the dashboard and make any change (e.g. update a field) 5. Click Save/Update 6. Observe HTTP 500 after exactly 5 minutes **Expected behavior** The reverse proxy service update should persist successfully and return HTTP 200. **Are you using NetBird Cloud?** No — self-hosted using the official `netbirdio/netbird-server` combined container, deployed via Docker Compose: ``` ✔ Image netbirdio/netbird-server:latest Pulled 6.8s ✔ Image traefik:v2.11 Pulled 6.3s ✔ Image netbirdio/dashboard:latest Pulled ``` Second (and third) node running only as relay and reverse proxy: ``` ✔ Image netbirdio/relay:latest Pulled 1.1s ✔ Image netbirdio/reverse-proxy:latest Pulled 1.5s ✔ Image traefik:v2.11 Pulled ``` **NetBird version** Reproduced on both `0.66.4` and the latest image pulled on 2026-04-23 (which referenced `golang.org/x/net@v0.52.0`). The management server startup log reports: ``` management server version 0.66.4 ``` **Is any other VPN software installed?** No. **Debug output** This bug is on the server/management side — the `netbird status` and debug bundle commands apply to the client. The relevant diagnostic information is the server log stack trace, included below. **Screenshots** Dashboard error toast: > `Code 500: Look up resource target "d7gkkk2m6v2c73clrgig": failed to get network resource from store` <img width="1011" height="420" alt="Image" src="https://github.com/user-attachments/assets/0ad3482f-123f-42f4-b8d7-6381dd0861a0" /> **Additional context** Server log stack trace (reproduced identically across two separate attempts, with proxies stable and connected throughout — not a timing/race issue): ``` ERRO management/server/store/sql_store.go:4287: failed to get network resource from store: context deadline exceeded WARN management/server/store/sql_store.go:3430: transaction exceeded 5m0s timeout after 5m0.005638867s, stack: github.com/netbirdio/netbird/management/server/store.(*SqlStore).ExecuteInTransaction management/server/store/sql_store.go:3430 github.com/netbirdio/netbird/management/internals/modules/reverseproxy/service/manager.(*Manager).persistServiceUpdate management/internals/modules/reverseproxy/service/manager/manager.go:541 github.com/netbirdio/netbird/management/internals/modules/reverseproxy/service/manager.(*Manager).UpdateService management/internals/modules/reverseproxy/service/manager/manager.go:506 github.com/netbirdio/netbird/management/internals/modules/reverseproxy/service/manager.(*handler).updateService management/internals/modules/reverseproxy/service/manager/api.go:152 ERRO shared/management/http/util/util.go:85: got a handler error: look up resource target "<resource_id>": failed to get network resource from store ERRO management/server/telemetry/http_api_metrics.go:212: HTTP response: PUT /api/reverse-proxies/services/<service_id> status 500 ``` Database state confirmed healthy — `PRAGMA integrity_check` returns `ok`, database size 1.43MB, no WAL files present, not in WAL mode. The network resource referenced by the target exists and is valid in the store. The issue is exclusively the re-entrant lock in `persistServiceUpdate`. Startup config confirms single connection pool: ``` Set max open db connections to 1, max idle to 1 ``` **Workaround:** Delete the service and recreate it with the desired settings. The create path does not reproduce the deadlock. **Have you tried these troubleshooting steps?** - [x] Reviewed client troubleshooting (not applicable — server-side bug) - [x] Checked for newer NetBird versions (reproduced on latest as of 2026-04-23) - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [ ] Disabled other VPN software (none installed) - [x] Checked firewall settings
saavagebueno added the triage-needed label 2026-08-05 01:31:55 -04:00
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11969