[GH-ISSUE #4212] Large Netbird Setup: Peers can't connect after netbird >=0.51.1, DeadlineExceeded desc = context deadline exceeded #8909

Open
opened 2026-08-05 01:20:00 -04:00 by saavagebueno · 7 comments
Owner

Originally created by @jr-timme on GitHub (Jul 24, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/4212

Describe the problem

After adding ~5k Nodes and updating to netbird >=0.51.1 and restarting the management node peers can't connect to the management node anymore, the connection always fails with: DeadlineExceeded desc = context deadline exceeded

To Reproduce

Steps to reproduce the behavior:

  1. add ~5k peers to the network
  2. update to >=0.51.1
  3. restart control-plane/management server

Expected behavior

all peers should be able to reconnect to the management Node/reestablish the connections to the other peers

Are you using NetBird Cloud?

self-hosted control plane

NetBird version

>=0.51.1

Is any other VPN software installed?

No

Additional context

As the previous Sections aren't really suited for an indepth explanation here is a longer version, which we previously posted to your Slack but haven't gotten any response to. All dates and things like "yesterday" and "today" are relative to 2025-07-16:


I'm not sure if this is the right place for this, but as we also don't know if this is a bug or a regression with the newer version I'll try it here anyway:

We're currently evaluating if we can switch our VPN Setup over to netbird. For this we have set up a self-hosted Instance for now, which we might switch to the cloud offering later on due to a requirement for a HA Management Setup and Multi Tenant functionality. We currently have ~5k Nodes connected to netbird, which we added in increasing batches and as long as we limited the parallel number of netbird up commands everything worked fine. Yesterday we updated the docker-compose stack which lead to a restart of almost all services, except coturn and dashboard, which then lead to all peers reconnecting to the management node which took a bit, but after a short while everything was working normally. Today we did the same and updated to the newly released 0.51.1 and now almost no nodes can connect anymore, and the Dashboard is not accessible. The docker logs are flooded with entries like these:

management-1  | 2025-07-16T11:16:53Z DEBG [context: GRPC, requestID: xxxxxxxxxx] management/server/grpcserver.go:441: Login request from peer [xxxxxxxxxx] [x.x.x.x]

which is expected since all peers need to reconnect. And in between there are messages like these:

management-1  | 2025-07-16T11:15:28Z DEBG [context: GRPC, requestID: xxxxxxxxxx, accountID: xxxxxxxxxx, peerID: xxxxxxxxxx] management/server/account.go:1623: SyncAndMarkPeer: took 8m23.407635647s

management-1  | 2025-07-16T11:15:28Z DEBG [context: GRPC, requestID: xxxxxxxxxx, accountID: xxxxxxxxxx, peerID: xxxxxxxxxx] management/server/grpcserver.go:200: Sync: took 8m30.352848964s

management-1  | 2025-07-16T11:15:28Z DEBG [peerID: xxxxxxxxxx, context: GRPC, requestID: xxxxxxxxxx, accountID: xxxxxxxxxx] management/server/peer.go:110: MarkPeerConnected: took 685.825795ms

which probably means after a very long time some peers are able to connect and sync.

The clients, user-facing ones which need to log in through our keycloak instance as well as headless servers which were added with setup-keys, mostly output:

Error: waiting sso login failed with: rpc error: code = DeadlineExceeded desc = context deadline exceeded

and

ERRO management/client/grpc.go:349: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Currently, we're still using the SQLite Backends as it worked fine until today, however switching to postgres or mysql (which according to the commits is also planned in the future?) would probably be a good idea.

To finally arrive at my question: Should netbird currently be able to handle a setup like this or is it more meant for setups with at most a few hundred Peers? And if it should be able to handle larger amounts of peers could this be a bug or regression with the current version?

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @jr-timme on GitHub (Jul 24, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/4212 **Describe the problem** After adding ~5k Nodes and updating to netbird `>=0.51.1` and restarting the management node peers can't connect to the management node anymore, the connection always fails with: `DeadlineExceeded desc = context deadline exceeded` **To Reproduce** Steps to reproduce the behavior: 1. add ~5k peers to the network 2. update to `>=0.51.1` 3. restart control-plane/management server **Expected behavior** all peers should be able to reconnect to the management Node/reestablish the connections to the other peers **Are you using NetBird Cloud?** self-hosted control plane **NetBird version** `>=0.51.1` **Is any other VPN software installed?** No **Additional context** As the previous Sections aren't really suited for an indepth explanation here is a longer version, which we previously posted to your Slack but haven't gotten any response to. All dates and things like "yesterday" and "today" are relative to 2025-07-16: -------- I'm not sure if this is the right place for this, but as we also don't know if this is a bug or a regression with the newer version I'll try it here anyway: We're currently evaluating if we can switch our VPN Setup over to netbird. For this we have set up a self-hosted Instance for now, which we might switch to the cloud offering later on due to a requirement for a HA Management Setup and Multi Tenant functionality. We currently have ~5k Nodes connected to netbird, which we added in increasing batches and as long as we limited the parallel number of `netbird up` commands everything worked fine. Yesterday we updated the docker-compose stack which lead to a restart of almost all services, except coturn and dashboard, which then lead to all peers reconnecting to the management node which took a bit, but after a short while everything was working normally. Today we did the same and updated to the newly released `0.51.1` and now almost no nodes can connect anymore, and the Dashboard is not accessible. The docker logs are flooded with entries like these: ```log management-1 | 2025-07-16T11:16:53Z DEBG [context: GRPC, requestID: xxxxxxxxxx] management/server/grpcserver.go:441: Login request from peer [xxxxxxxxxx] [x.x.x.x] ``` which is expected since all peers need to reconnect. And in between there are messages like these: ```log management-1 | 2025-07-16T11:15:28Z DEBG [context: GRPC, requestID: xxxxxxxxxx, accountID: xxxxxxxxxx, peerID: xxxxxxxxxx] management/server/account.go:1623: SyncAndMarkPeer: took 8m23.407635647s management-1 | 2025-07-16T11:15:28Z DEBG [context: GRPC, requestID: xxxxxxxxxx, accountID: xxxxxxxxxx, peerID: xxxxxxxxxx] management/server/grpcserver.go:200: Sync: took 8m30.352848964s management-1 | 2025-07-16T11:15:28Z DEBG [peerID: xxxxxxxxxx, context: GRPC, requestID: xxxxxxxxxx, accountID: xxxxxxxxxx] management/server/peer.go:110: MarkPeerConnected: took 685.825795ms ``` which probably means after a very long time some peers are able to connect and sync. The clients, user-facing ones which need to log in through our keycloak instance as well as headless servers which were added with setup-keys, mostly output: ```log Error: waiting sso login failed with: rpc error: code = DeadlineExceeded desc = context deadline exceeded ``` and ```log ERRO management/client/grpc.go:349: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded ``` Currently, we're still using the SQLite Backends as it worked fine until today, however switching to postgres or mysql (which according to the commits is also planned in the future?) would probably be a good idea. To finally arrive at my question: Should netbird currently be able to handle a setup like this or is it more meant for setups with at most a few hundred Peers? And if it should be able to handle larger amounts of peers could this be a bug or regression with the current version? **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-neededself-hosting labels 2026-08-05 01:20:00 -04:00
Author
Owner

@nazarewk commented on GitHub (Jul 24, 2025):

Did it happen when upgrading from 0.51.0 to 0.51.1 or from an earlier version? https://github.com/netbirdio/netbird/releases/tag/v0.51.0 notes the need to update Relays before things start working correctly

<!-- gh-comment-id:3113291103 --> @nazarewk commented on GitHub (Jul 24, 2025): Did it happen when upgrading from `0.51.0` to `0.51.1` or from an earlier version? https://github.com/netbirdio/netbird/releases/tag/v0.51.0 notes the need to update Relays before things start working correctly
Author
Owner

@jr-timme commented on GitHub (Jul 24, 2025):

We were using latest in our compose file at that point for all services so signal, relay and management were updated at the same time, so the update the day before (2025-06-15 ~4 pm) should have pulled the 2 hours earlier released 0.51.0. But I did not check the installed version afterwards.

I also can't say for sure that all peers were already on 0.51.0 as they pull their updates through unattended-upgrades (runs once a day) from your debian repo.

The next day we did the same docker compose pull, so signal, relay and management were updated to 0.51.1 at the same time.

iirc dashboard and coturn did not have an update on either of the two days


Also interesting would be if Netbird is meant for/able to handle deployments of this size and if there are any differences between the self-hosted control plane and the managed version in this regard.

<!-- gh-comment-id:3113737762 --> @jr-timme commented on GitHub (Jul 24, 2025): We were using `latest` in our compose file at that point for all services so `signal`, `relay` and `management` were updated at the same time, so the update the day before (2025-06-15 ~4 pm) should have pulled the 2 hours earlier released 0.51.0. But I did not check the installed version afterwards. I also can't say for sure that all peers were already on 0.51.0 as they pull their updates through unattended-upgrades (runs once a day) from your debian repo. The next day we did the same `docker compose pull`, so `signal`, `relay` and `management` were updated to 0.51.1 at the same time. iirc `dashboard` and `coturn` did not have an update on either of the two days ------ Also interesting would be if Netbird is meant for/able to handle deployments of this size and if there are any differences between the self-hosted control plane and the managed version in this regard.
Author
Owner

@nazarewk commented on GitHub (Jul 25, 2025):

What you're observing (context deadline exceeded) might have similar causes to our current performance issues. While probably not very practical, it would definitely help if you staggered the reconnection rates to spread those over time until the whole thing stabilizes.

<!-- gh-comment-id:3116808851 --> @nazarewk commented on GitHub (Jul 25, 2025): What you're observing (context deadline exceeded) might have similar causes to our current performance issues. While probably not very practical, it would definitely help if you staggered the reconnection rates to spread those over time until the whole thing stabilizes.
Author
Owner

@nazarewk commented on GitHub (Jul 25, 2025):

Did you have the same ballpark number of Peers while doing previous updates? While nothing might be failing on the backend, it might be simply too slow to respond to meet timeout requirements on the clients while a lot of those are reconnecting.

<!-- gh-comment-id:3116829071 --> @nazarewk commented on GitHub (Jul 25, 2025): Did you have the same ballpark number of Peers while doing previous updates? While nothing might be **failing** on the backend, it might be simply too slow to respond to meet timeout requirements on the clients while a lot of those are reconnecting.
Author
Owner

@jr-timme commented on GitHub (Jul 25, 2025):

Ok, good to know, that large scale Deployments are a use case you want to support and that it is on the road map/being worked on. We'll probably pause our tests for now and reevaluate netbird at a later time, maybe closer to the first stable release.

it would definitely help if you staggered the reconnection rates to spread those over time until the whole thing stabilizes.

I don't suppose there is an option for this? Off the top of my head I can only think of a setup, where a script monitors if the management service is reachable, but netbird is not connected and then stops the local netbird service and after a random delay starts it again. But that would be quite difficult to coordinate over such a large number of peers and would mean a lot of peers aren't reachable at all during that time. (Due to regulations we have to access all peers over a secured network, which would be netbird, once we switch over from our current solution)

Did you have the same ballpark number of Peers while doing previous updates? While nothing might be failing on the backend, it might be simply too slow to respond to meet timeout requirements on the clients while a lot of those are reconnecting.

Yes we had the same number of Peers, perhaps a couple less for the last update. We had already added all peers to the network and then did both updates. But as some peers get shutdown and deleted every day and because we don't automatically add newly created peers the number was probably a bit, low 2-digits at most, lower.

What I forgot to mention: The constant reconnects and timeouts lead to OOM events on our management server as the RAM usage keeps increasing over time and after a few hours all RAM (which isn't that much, currently "only" 32 GB, which was more than enough when everything was working) is consumed which then invokes the OOM-Reaper, so just waiting for some peers to connect, which should decrease the number of concurrent reconnect-requests, which then should lead to all peers being connected after a long while is also not an option

<!-- gh-comment-id:3116839221 --> @jr-timme commented on GitHub (Jul 25, 2025): Ok, good to know, that large scale Deployments are a use case you want to support and that it is on the road map/being worked on. We'll probably pause our tests for now and reevaluate netbird at a later time, maybe closer to the first stable release. > it would definitely help if you staggered the reconnection rates to spread those over time until the whole thing stabilizes. I don't suppose there is an option for this? Off the top of my head I can only think of a setup, where a script monitors if the management service is reachable, but netbird is not connected and then stops the local netbird service and after a random delay starts it again. But that would be quite difficult to coordinate over such a large number of peers and would mean a lot of peers aren't reachable at all during that time. (Due to regulations we have to access all peers over a secured network, which would be netbird, once we switch over from our current solution) > Did you have the same ballpark number of Peers while doing previous updates? While nothing might be failing on the backend, it might be simply too slow to respond to meet timeout requirements on the clients while a lot of those are reconnecting. Yes we had the same number of Peers, perhaps a couple less for the last update. We had already added all peers to the network and then did both updates. But as some peers get shutdown and deleted every day and because we don't automatically add newly created peers the number was probably a bit, low 2-digits at most, lower. What I forgot to mention: The constant reconnects and timeouts lead to OOM events on our management server as the RAM usage keeps increasing over time and after a few hours all RAM (which isn't that much, currently "only" 32 GB, which was more than enough when everything was working) is consumed which then invokes the OOM-Reaper, so just waiting for some peers to connect, which should decrease the number of concurrent reconnect-requests, which then should lead to all peers being connected after a long while is also not an option
Author
Owner

@mlsmaycon commented on GitHub (Jul 25, 2025):

@jr-timme we have accounts with the same number of peers and they run without issues. It could be a matter of different policies or a potential bug.

Would like to jump in a call for us to go over your setup in detail? You can send us an email to support@netbird.io and we will share a calendar link.

<!-- gh-comment-id:3117281678 --> @mlsmaycon commented on GitHub (Jul 25, 2025): @jr-timme we have accounts with the same number of peers and they run without issues. It could be a matter of different policies or a potential bug. Would like to jump in a call for us to go over your setup in detail? You can send us an email to support@netbird.io and we will share a calendar link.
Author
Owner

@jr-timme commented on GitHub (Aug 6, 2025):

As suggested by @mlsmaycon we scheduled a meeting to analyze our problems further and while during the meeting we were not able to find any problematic configuration or be able to fix it, I also left the meeting with a couple of additional To-Do's which weren't practical to try during our meeting. It took me a while to get to those but in case it is of interest to anyone else having similar problems here are my additional findings:

The setup before the additional tests was the simple docker compose stack using SQLite and Keycloak described in the official documentation.

tl;dr: don't use SQLite for large scale Deployments and switch to PostgreSQL instead. Also don't disable lazy connections if you use consumer grade hardware and have ACLs which allow some users to see/connect to a lot of peers.

  1. Increasing/Specifying NB_PEER_UPDATE_STARTUP_PERIOD_S=300 and NB_PEER_UPDATE_STARTUP_PERIOD_S=3
  • this did not improve the situation
  1. We provisioned most our peers with non-reusable setup_keys created through the API, but we did not remove the expired keys afterward, which meant there were still ~3000 setup_keys present, which are also loaded during update operations and could lead to issues
  • removing the expired setup_keys did not help
  1. Disabling lazy connections
  • as lazy connections are still an experimental feature it was suggested, that we try disabling it.
  • This did, at first, not change anything. However after we were able to fix the connection issues disabling lazy connections lead to massive problems as a "desktop peer" from a higher level sysadmin which has access to pretty much all peers lead to netbird establishing a connection to every other peer which overloaded the notebook as well as the consumer-grade router in use.
  1. Migrating from SQLite to PostgreSQL
  • while SQLite theoretically should also be able to handle this amount of peers a migration to MariaDB or PostgreSQL was already on our To-Do List and should improve Performance
  • the migration process does not work exactly as described in the docs (see also: https://github.com/netbirdio/docs/issues/402) but this did in fact solve all of our problems.
  1. Migrating to a different host system with faster storage (more in regard to IOPS than raw speed)
  • as the migration to PostgreSQL already fixed our issues and did only cause a short load spike during the initial start we did not test this. We were also able to temporarily scale down our Management node from 16 Cores / 32 GB RAM to 4 Cores / 8 GB RAM while using the same storage backend and everything still worked, albeit the load spike after a restart of all services took quite a bit longer.

For us the issue is resolved, but perhaps the documentation could be expanded to include a notice/warning that the default SQLite backend should not be used for larger setups.

<!-- gh-comment-id:3158731327 --> @jr-timme commented on GitHub (Aug 6, 2025): As suggested by @mlsmaycon we scheduled a meeting to analyze our problems further and while during the meeting we were not able to find any problematic configuration or be able to fix it, I also left the meeting with a couple of additional To-Do's which weren't practical to try during our meeting. It took me a while to get to those but in case it is of interest to anyone else having similar problems here are my additional findings: The setup before the additional tests was the simple docker compose stack using SQLite and Keycloak described in the official documentation. tl;dr: don't use SQLite for large scale Deployments and switch to PostgreSQL instead. Also don't disable lazy connections if you use consumer grade hardware and have ACLs which allow some users to see/connect to a lot of peers. 1. Increasing/Specifying `NB_PEER_UPDATE_STARTUP_PERIOD_S=300` and `NB_PEER_UPDATE_STARTUP_PERIOD_S=3` - this did not improve the situation 2. We provisioned most our peers with non-reusable setup_keys created through the API, but we did not remove the expired keys afterward, which meant there were still ~3000 setup_keys present, which are also loaded during update operations and could lead to issues - removing the expired setup_keys did not help 3. Disabling lazy connections - as lazy connections are still an experimental feature it was suggested, that we try disabling it. - This did, at first, not change anything. However after we were able to fix the connection issues disabling lazy connections lead to massive problems as a "desktop peer" from a higher level sysadmin which has access to pretty much all peers lead to netbird establishing a connection to every other peer which overloaded the notebook as well as the consumer-grade router in use. 4. Migrating from SQLite to PostgreSQL - while SQLite theoretically should also be able to handle this amount of peers a migration to MariaDB or PostgreSQL was already on our To-Do List and should improve Performance - the migration process does not work exactly as described in the docs (see also: [https://github.com/netbirdio/docs/issues/402](https://github.com/netbirdio/docs/issues/402)) but this did in fact solve all of our problems. 6. Migrating to a different host system with faster storage (more in regard to IOPS than raw speed) - as the migration to PostgreSQL already fixed our issues and did only cause a short load spike during the initial start we did not test this. We were also able to temporarily scale down our Management node from 16 Cores / 32 GB RAM to 4 Cores / 8 GB RAM while using the same storage backend and everything still worked, albeit the load spike after a restart of all services took quite a bit longer. For us the issue is resolved, but perhaps the documentation could be expanded to include a notice/warning that the default SQLite backend should not be used for larger setups.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#8909