[GH-ISSUE #3678] Memory leak after 0.40.0 update #7721

Open
opened 2026-08-05 01:14:14 -04:00 by saavagebueno · 24 comments
Owner

Originally created by @EmbeddedAndMore on GitHub (Apr 15, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/3678

Originally assigned to: @nazarewk on GitHub.

Describe the problem

After updating Windows nodes' clients to 0.40.0, then 0.40.1, and finally to 0.41.0, we received many alerts from multiple nodes about excessive Mem usage. It turned out that NetBird is using a lot of memory. in some cases as much as 2GB after running for a few days.

Is it normal? some of our nodes come with limited Memory like 4GB and there we have Memory usage of up to 600MB.

We didn't notice that in our laptops since they usually come with 32GB RAM.
To Reproduce

Steps to reproduce the behavior:
install the client version 0.39.0 and then update it to 0.41.0 with ".msi" installer or just install the 0.41.0 with MSI installer. I didn't install it with ".exe" artifact.

Question is, is it normal behavior? and if yes why it didn't happen before

Expected behavior

previously we had less than 1MB ram usage.

Are you using NetBird Cloud?

No. Self hosted.

NetBird version

netbird client version: 0.41.0
netbird management, relay, signal: 0.40.0

Is any other VPN software installed?

No

Debug output

To help us resolve the problem, please attach the following debug output

status-dA.txt

As well as the file created by

netbird.debug.650734575.zip

Debug outputs are from the node that currently using 1.9GB of RAM.

Screenshots

If applicable, add screenshots to help explain your problem.
Image

Update:
The trend of the RAM usage in the exact machine: started at 60MB and even reached 1.9GB at some point.

Image

Originally created by @EmbeddedAndMore on GitHub (Apr 15, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/3678 Originally assigned to: @nazarewk on GitHub. **Describe the problem** After updating Windows nodes' clients to 0.40.0, then 0.40.1, and finally to 0.41.0, we received many alerts from multiple nodes about excessive Mem usage. It turned out that NetBird is using a lot of memory. in some cases as much as 2GB after running for a few days. Is it normal? some of our nodes come with limited Memory like 4GB and there we have Memory usage of up to 600MB. We didn't notice that in our laptops since they usually come with 32GB RAM. **To Reproduce** Steps to reproduce the behavior: install the client version 0.39.0 and then update it to 0.41.0 with ".msi" installer or just install the 0.41.0 with MSI installer. I didn't install it with ".exe" artifact. Question is, is it normal behavior? and if yes why it didn't happen before **Expected behavior** previously we had less than 1MB ram usage. **Are you using NetBird Cloud?** No. Self hosted. **NetBird version** netbird client version: `0.41.0` netbird management, relay, signal: `0.40.0` **Is any other VPN software installed?** No **Debug output** To help us resolve the problem, please attach the following debug output [status-dA.txt](https://github.com/user-attachments/files/19753736/status-dA.txt) As well as the file created by [netbird.debug.650734575.zip](https://github.com/user-attachments/files/19753740/netbird.debug.650734575.zip) Debug outputs are from the node that currently using 1.9GB of RAM. **Screenshots** If applicable, add screenshots to help explain your problem. ![Image](https://github.com/user-attachments/assets/69a7d0b0-1284-4895-a688-4f9d1a79b47f) Update: The trend of the RAM usage in the exact machine: started at 60MB and even reached 1.9GB at some point. ![Image](https://github.com/user-attachments/assets/56ac4678-0b1f-4bf7-83b8-c2a1f1a6612b)
saavagebueno added the bug label 2026-08-05 01:14:14 -04:00
Author
Owner

@FusselTV commented on GitHub (Apr 15, 2025):

I'm experiencing the same issue while using a Debian LXC. It's only running NetBird and nothing else and the ram usage is slowly creeping up. My monitoring system reported that some resources went down... (Running 0.40.0)

Image

Image

<!-- gh-comment-id:2804292364 --> @FusselTV commented on GitHub (Apr 15, 2025): I'm experiencing the same issue while using a Debian LXC. It's only running NetBird and nothing else and the ram usage is slowly creeping up. My monitoring system reported that some resources went down... (Running 0.40.0) ![Image](https://github.com/user-attachments/assets/3e0faaf3-bb06-4c24-926b-c359ba15ec91) ![Image](https://github.com/user-attachments/assets/e219ce7e-0377-4241-879b-adf03561617d)
Author
Owner

@EmbeddedAndMore commented on GitHub (Apr 15, 2025):

This is also the RAM usage from 4GB node. we are constantly getting Alerts from all of our nodes with 4GB RAMs. all of them are using 0.41.0. it was also the same for 0.40.0 and 0.40.1.

Image

<!-- gh-comment-id:2804557221 --> @EmbeddedAndMore commented on GitHub (Apr 15, 2025): This is also the RAM usage from 4GB node. we are constantly getting Alerts from all of our nodes with 4GB RAMs. all of them are using 0.41.0. it was also the same for 0.40.0 and 0.40.1. ![Image](https://github.com/user-attachments/assets/efcb24b2-da98-445b-8ee2-d2986be2e5ad)
Author
Owner

@nazarewk commented on GitHub (Apr 15, 2025):

Just to give you an update:

We are working on it internally. While we have ideas of potential culprits, we have also added additional memory profiling reports to the debug bundle and might be releasing a new client version to be sure to debug the issue further in future.

<!-- gh-comment-id:2804619297 --> @nazarewk commented on GitHub (Apr 15, 2025): Just to give you an update: We are working on it internally. While we have ideas of potential culprits, we have also added additional memory profiling reports to the debug bundle and might be releasing a new client version to be sure to debug the issue further in future.
Author
Owner

@nazarewk commented on GitHub (Apr 15, 2025):

@EmbeddedAndMore @FusselTV could you install 0.41.1 and send us debug bundles when memory usage grows?

<!-- gh-comment-id:2804825845 --> @nazarewk commented on GitHub (Apr 15, 2025): @EmbeddedAndMore @FusselTV could you install [`0.41.1`](https://github.com/netbirdio/netbird/releases/tag/v0.41.1) and send us debug bundles when memory usage grows?
Author
Owner

@EmbeddedAndMore commented on GitHub (Apr 15, 2025):

@nazarewk i installed it in 2 nodes. and will send the profile output shortly

<!-- gh-comment-id:2804833062 --> @EmbeddedAndMore commented on GitHub (Apr 15, 2025): @nazarewk i installed it in 2 nodes. and will send the profile output shortly
Author
Owner

@EmbeddedAndMore commented on GitHub (Apr 15, 2025):

@nazarewk

in both of my nodes, I can see again RAM ramping up.

Here is 5 minutes of running in debug mode with the 0.41.1 version.

netbird.debug.608146089.zip

<!-- gh-comment-id:2805381381 --> @EmbeddedAndMore commented on GitHub (Apr 15, 2025): @nazarewk in both of my nodes, I can see again RAM ramping up. Here is 5 minutes of running in debug mode with the 0.41.1 version. [netbird.debug.608146089.zip](https://github.com/user-attachments/files/19759896/netbird.debug.608146089.zip)
Author
Owner

@Maescool commented on GitHub (Apr 15, 2025):

I've noticed the same on the Linux build, 0.37.1 stable memory use, higher up started leaking

<!-- gh-comment-id:2806414057 --> @Maescool commented on GitHub (Apr 15, 2025): I've noticed the same on the Linux build, 0.37.1 stable memory use, higher up started leaking
Author
Owner

@nazarewk commented on GitHub (Apr 15, 2025):

We think we have fixed the leak at v0.41.2 / https://github.com/netbirdio/netbird/pull/3684 , but I'll keep the issue open for a while just in case somebody finds another one.

<!-- gh-comment-id:2806646747 --> @nazarewk commented on GitHub (Apr 15, 2025): We think we have fixed the leak at [`v0.41.2`](https://github.com/netbirdio/netbird/releases/tag/v0.41.2) / https://github.com/netbirdio/netbird/pull/3684 , but I'll keep the issue open for a while just in case somebody finds another one.
Author
Owner

@nazarewk commented on GitHub (Apr 16, 2025):

@EmbeddedAndMore @Maescool @FusselTV
Can you confirm the memory issue got solved for you on v0.41.2?

<!-- gh-comment-id:2809711368 --> @nazarewk commented on GitHub (Apr 16, 2025): @EmbeddedAndMore @Maescool @FusselTV Can you confirm the memory issue got solved for you on v0.41.2?
Author
Owner

@FusselTV commented on GitHub (Apr 16, 2025):

@nazarewk I updated right away, and I think it's slowly creeping up again. I'm experiencing the same problem on two hosts—one is running Docker with NetBird version 0.41.2, and the other is my LXC using the Debian package (same version).

Image

Image

netbird.debug.2533783209.zip

<!-- gh-comment-id:2809912342 --> @FusselTV commented on GitHub (Apr 16, 2025): @nazarewk I updated right away, and I think it's slowly creeping up again. I'm experiencing the same problem on two hosts—one is running Docker with NetBird version 0.41.2, and the other is my LXC using the Debian package (same version). ![Image](https://github.com/user-attachments/assets/a0c18fd3-ed48-43e3-bcba-a5ec7b89c8fa) ![Image](https://github.com/user-attachments/assets/a8fb1c1e-1fde-4b49-bba4-08a7cf6701ca) [netbird.debug.2533783209.zip](https://github.com/user-attachments/files/19779524/netbird.debug.2533783209.zip)
Author
Owner

@Maescool commented on GitHub (Apr 17, 2025):

It looks stable to me on my end (0.41.2)

Image

What is was before (when leaking, forgot to check exact version, but was definitely later than 0.37.1):

Image

What it is on 0.37.1

Image

<!-- gh-comment-id:2812090053 --> @Maescool commented on GitHub (Apr 17, 2025): It looks stable to me on my end (0.41.2) ![Image](https://github.com/user-attachments/assets/2188826a-d140-4cd1-9fc8-57be321fadb6) What is was before (when leaking, forgot to check exact version, but was definitely later than 0.37.1): ![Image](https://github.com/user-attachments/assets/9a89ba84-0b16-4912-9bbd-5b4bb76eb26e) What it is on 0.37.1 ![Image](https://github.com/user-attachments/assets/1f6740a8-bea8-44b8-b09e-eb2cee04bab2)
Author
Owner

@mlsmaycon commented on GitHub (Apr 17, 2025):

Thanks @Maescool

@FusselTV can you share new metrics? I am interested in the growth progression since yesterday

<!-- gh-comment-id:2812100214 --> @mlsmaycon commented on GitHub (Apr 17, 2025): Thanks @Maescool @FusselTV can you share new metrics? I am interested in the growth progression since yesterday
Author
Owner

@EmbeddedAndMore commented on GitHub (Apr 17, 2025):

From what I can see, 0.41.2 and 0.41.3 are working without problem. I will monitor all nodes and let you know the final result in a few days.

<!-- gh-comment-id:2812195345 --> @EmbeddedAndMore commented on GitHub (Apr 17, 2025): From what I can see, 0.41.2 and 0.41.3 are working without problem. I will monitor all nodes and let you know the final result in a few days.
Author
Owner

@FusselTV commented on GitHub (Apr 17, 2025):

@mlsmaycon On the Docker host, usage climbed to 60 MB, and then the container was automatically updated to version 0.41.3. It's now climbing to 47.60 MB. The LXC looks fine, though....

Image

Image

Container Logs:
netbird.csv

Unfortunately, I cannot create a debug bundle in the container:

Image

<!-- gh-comment-id:2812429179 --> @FusselTV commented on GitHub (Apr 17, 2025): @mlsmaycon On the Docker host, usage climbed to 60 MB, and then the container was automatically updated to version 0.41.3. It's now climbing to 47.60 MB. The LXC looks fine, though.... ![Image](https://github.com/user-attachments/assets/de07e420-9e5e-4581-aede-af84e8e2de85) ![Image](https://github.com/user-attachments/assets/c058e997-b893-4254-8aaf-f11163f41ff5) Container Logs: [netbird.csv](https://github.com/user-attachments/files/19792567/netbird.csv) Unfortunately, I cannot create a debug bundle in the container: ![Image](https://github.com/user-attachments/assets/f4842043-8642-479b-b972-8230ce0a8153)
Author
Owner

@FusselTV commented on GitHub (Apr 20, 2025):

Just to give an update: all my LXCs are still fine after 0.41.3, and the Docker container also seems fine now.
Image

<!-- gh-comment-id:2817109129 --> @FusselTV commented on GitHub (Apr 20, 2025): Just to give an update: all my LXCs are still fine after 0.41.3, and the Docker container also seems fine now. ![Image](https://github.com/user-attachments/assets/d4c73961-357d-4612-957f-eefc6d6a0960)
Author
Owner

@TheOverpassArsonist commented on GitHub (Mar 8, 2026):

Strange, I'm running a fairly recent version right now and having this issue, even though it looks like it should've already been fixed.

output of 'netbird status'

OS: linux/amd64
Daemon version: 0.65.3
CLI version: 0.66.2

meanwhile the netbird process is currently taking 10GB of ram. (Slightly out-of-date Garuda-linux installation)

edit : I realize I should probably clarify that I'm running bare-metal, netbird is literally just running as a service launched via systemctl on my actual linux desktop that's been powered on for a few weeks now, probably explaining why I have such a ludicrously high usage. After running "systemctl restart netbird" it seems locked at a stable 55.3M of memory and 30.2M of shared memory. Notably, when it WAS still at 10gb it was fluctuating a fair bit, on the order of hundreds of megabytes, whereas when it's running nominally now it's locked at a very stable 55.3. I wonder if this explains why some of my services have been getting borked by the OOMD shutting down random processes while my system is unattended for a while.

<!-- gh-comment-id:4018874052 --> @TheOverpassArsonist commented on GitHub (Mar 8, 2026): Strange, I'm running a fairly recent version right now and having this issue, even though it looks like it should've already been fixed. output of 'netbird status' > OS: linux/amd64 > Daemon version: 0.65.3 > CLI version: 0.66.2 meanwhile the netbird process is currently taking 10GB of ram. (Slightly out-of-date Garuda-linux installation) edit : I realize I should probably clarify that I'm running bare-metal, netbird is literally just running as a service launched via systemctl on my actual linux desktop that's been powered on for a few weeks now, probably explaining why I have such a ludicrously high usage. After running "systemctl restart netbird" it seems locked at a stable 55.3M of memory and 30.2M of shared memory. Notably, when it WAS still at 10gb it was fluctuating a fair bit, on the order of hundreds of megabytes, whereas when it's running nominally now it's locked at a very stable 55.3. I wonder if this explains why some of my services have been getting borked by the OOMD shutting down random processes while my system is unattended for a while.
Author
Owner

@meijeritbe commented on GitHub (Mar 24, 2026):

On Windows server 2016 (64Gb RAM), memory usage increased to a whopping 10Gb before crashing. After killing + restarting the service, memory usage drops to ~60-65Mb.

The client is on the latest version v0.67.0 with a clean install, C:\ProgramData\Netbird was removed before re-installation as this is happening over multiple versions.

The uptime of the service was ~10 hours.

The server is going to be decommissioned within ~2 weeks but nonetheless I consider this as a bug :) On server 2025, the service runs normally.

Unfortunately, I was not able to create a bundle.

Image

Image

<!-- gh-comment-id:4120254632 --> @meijeritbe commented on GitHub (Mar 24, 2026): On Windows server 2016 (64Gb RAM), memory usage increased to a whopping 10Gb before crashing. After killing + restarting the service, memory usage drops to ~60-65Mb. The client is on the latest version v0.67.0 with a clean install, C:\ProgramData\Netbird was removed before re-installation as this is happening over multiple versions. The uptime of the service was ~10 hours. The server is going to be decommissioned within ~2 weeks but nonetheless I consider this as a bug :) On server 2025, the service runs normally. Unfortunately, I was not able to create a bundle. ![Image](https://github.com/user-attachments/assets/17784c36-b7c3-4e1d-bf66-96120ff05b72) ![Image](https://github.com/user-attachments/assets/7cce6eec-d543-4258-8299-d4f53b0af6a6)
Author
Owner

@TheOverpassArsonist commented on GitHub (Apr 21, 2026):

On Windows server 2016 (64Gb RAM), memory usage increased to a whopping 10Gb before crashing. After killing + restarting the service, memory usage drops to ~60-65Mb.

Out of curiosity, are you using post-quantum encryption? I noticed the same apparent memory leak on my windows machine and I'm wondering if it it's a leak in the PQ encryption specifically, rather than netbird as a whole.

<!-- gh-comment-id:4285871320 --> @TheOverpassArsonist commented on GitHub (Apr 21, 2026): > On Windows server 2016 (64Gb RAM), memory usage increased to a whopping 10Gb before crashing. After killing + restarting the service, memory usage drops to ~60-65Mb. Out of curiosity, are you using post-quantum encryption? I noticed the same apparent memory leak on my windows machine and I'm wondering if it it's a leak in the PQ encryption specifically, rather than netbird as a whole.
Author
Owner

@ykorzikowski commented on GitHub (Apr 25, 2026):

experiencing same issue recently. Using netbird on some tiny cloud vm with 1GB Ram + 1GB SWAP. Works for 2 years until ~march 2026. Experience netbird consuming whole ram after 24h. Server only runs a nginx, using ~215MB in total after a fresh netbird restart.

Enabled a daily restart of netbird, which fixed the issue for 10 days. today I experience it again:

0  5    * * *   root    netbird service restart

Had quantum resistance enabled on this server, disabled it now. will continue testing.

Experienced same issue on one of my root servers. Netbird consumes 70% of 64gb RAM. Server uses ~6gb when working normally. Restarted netbird, problem did not come back. This server has rosenpass enabled too.

Using 0.67.4 client version.

<!-- gh-comment-id:4319800461 --> @ykorzikowski commented on GitHub (Apr 25, 2026): experiencing same issue recently. Using netbird on some tiny cloud vm with 1GB Ram + 1GB SWAP. Works for 2 years until ~march 2026. Experience netbird consuming whole ram after 24h. Server only runs a nginx, using ~215MB in total after a fresh netbird restart. Enabled a daily restart of netbird, which fixed the issue for 10 days. today I experience it again: ``` 0 5 * * * root netbird service restart ``` Had quantum resistance enabled on this server, disabled it now. will continue testing. Experienced same issue on one of my root servers. Netbird consumes 70% of 64gb RAM. Server uses ~6gb when working normally. Restarted netbird, problem did not come back. This server has rosenpass enabled too. Using 0.67.4 client version.
Author
Owner

@ykorzikowski commented on GitHub (Apr 25, 2026):

Are there any plans collecting client metrics and exposing them on the management /metrics path @mlsmaycon ?

<!-- gh-comment-id:4319814121 --> @ykorzikowski commented on GitHub (Apr 25, 2026): Are there any plans collecting client metrics and exposing them on the management /metrics path @mlsmaycon ?
Author
Owner

@ykorzikowski commented on GitHub (Apr 26, 2026):

Confirmed another memory issue event:

Image

Created a debug-bundle and restartet netbird. Also rosenpass was enabled on this host. If you think its helpful, I will provide you a private link.

<!-- gh-comment-id:4322269746 --> @ykorzikowski commented on GitHub (Apr 26, 2026): Confirmed another memory issue event: <img width="3844" height="492" alt="Image" src="https://github.com/user-attachments/assets/c50213b3-8ecd-4125-b055-50ef6e9840e0" /> Created a debug-bundle and restartet netbird. Also rosenpass was enabled on this host. If you think its helpful, I will provide you a private link.
Author
Owner

@lixmal commented on GitHub (Apr 26, 2026):

@ykorzikowski, please share with support@netbird.io or use the upload functionality and paste the upload key here

<!-- gh-comment-id:4322312129 --> @lixmal commented on GitHub (Apr 26, 2026): @ykorzikowski, please share with support@netbird.io or use the upload functionality and paste the upload key here
Author
Owner

@andrewjanssen commented on GitHub (Jul 24, 2026):

I found a NetBird 0.74.3 client using 15.7 GB RAM today.

Environment:

  • NetBird client: v0.74.3
  • OS: NixOS 25.11, x86_64
  • Kernel: 6.12.87
  • Swap: none
  • RosenpassEnabled: true
  • RosenpassPermissive: false
  • Peer fleet contained mixed client versions, including v0.71.4.

The daemon started at 2026-07-17 15:44:02 UTC. After approximately 6 days 20 hours it was still active but had grown to:

  • RSS: 15,298,180 KiB, almost entirely anonymous memory
  • systemd MemoryCurrent: 15,689,068,544 bytes
  • systemd peak: 15.7 GB
  • virtual size: 20,302,276 KiB
  • threads: 37
  • process CPU: approximately 65%

systemd reported 4 days 11 hours 46 minutes of accumulated CPU time, 76.1 GB incoming traffic, and 731.2 GB outgoing traffic.

During the final 72 seconds before restart, journald recorded 5,147 lines from the daemon. Repeated errors included:

Failed to handle message: session not found: `<session-id>`

There were 21 session not found errors during that interval, with the same session IDs recurring approximately every ten seconds. Logs also showed repeated WireGuard handshake timeouts and ICE/relay reconnection activity.

No OOM event occurred, and the host had no swap.

Restarting only netbird.service immediately reduced memory to roughly 61-70 MiB. Management, signaling, relays, and peer connectivity recovered. No further session not found errors appeared after restart.

<!-- gh-comment-id:5070073002 --> @andrewjanssen commented on GitHub (Jul 24, 2026): I found a NetBird 0.74.3 client using 15.7 GB RAM today. Environment: - NetBird client: v0.74.3 - OS: NixOS 25.11, x86_64 - Kernel: 6.12.87 - Swap: none - RosenpassEnabled: true - RosenpassPermissive: false - Peer fleet contained mixed client versions, including v0.71.4. The daemon started at 2026-07-17 15:44:02 UTC. After approximately 6 days 20 hours it was still active but had grown to: - RSS: 15,298,180 KiB, almost entirely anonymous memory - systemd MemoryCurrent: 15,689,068,544 bytes - systemd peak: 15.7 GB - virtual size: 20,302,276 KiB - threads: 37 - process CPU: approximately 65% `systemd` reported 4 days 11 hours 46 minutes of accumulated CPU time, 76.1 GB incoming traffic, and 731.2 GB outgoing traffic. During the final 72 seconds before restart, `journald` recorded 5,147 lines from the daemon. Repeated errors included: Failed to handle message: session not found: `<session-id>` There were 21 `session not found` errors during that interval, with the same session IDs recurring approximately every ten seconds. Logs also showed repeated WireGuard handshake timeouts and ICE/relay reconnection activity. No OOM event occurred, and the host had no swap. Restarting only `netbird.service` immediately reduced memory to roughly 61-70 MiB. Management, signaling, relays, and peer connectivity recovered. No further `session not found` errors appeared after restart.
Author
Owner

@MeijerIT commented on GitHub (Aug 5, 2026):

@TheOverpassArsonist

On Windows server 2016 (64Gb RAM), memory usage increased to a whopping 10Gb before crashing. After killing + restarting the service, memory usage drops to ~60-65Mb.

Out of curiosity, are you using post-quantum encryption? I noticed the same apparent memory leak on my windows machine and I'm wondering if it it's a leak in the PQ encryption specifically, rather than netbird as a whole.

I'm sorry for the delayed comment :) all optional features were disabled, only the bare minimum was configured :)

<!-- gh-comment-id:5187755906 --> @MeijerIT commented on GitHub (Aug 5, 2026): @TheOverpassArsonist >> On Windows server 2016 (64Gb RAM), memory usage increased to a whopping 10Gb before crashing. After killing + restarting the service, memory usage drops to ~60-65Mb. >Out of curiosity, are you using post-quantum encryption? I noticed the same apparent memory leak on my windows machine and I'm wondering if it it's a leak in the PQ encryption specifically, rather than netbird as a whole. I'm sorry for the delayed comment :) all optional features were disabled, only the bare minimum was configured :)
Sign in to join this conversation.
No Label bug
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#7721