Netbird sends gigabytes of packets via WireGuard interface with no reason #529

Closed
opened 2025-11-20 05:13:07 -05:00 by saavagebueno · 4 comments
Owner

Originally created by @s-zymon on GitHub (Dec 3, 2023).

I have noticed for some time that all network traffic regularly freezes at random times for a few minutes. This happens a couple of times a day. After doing a small investigation, I find that netbird is somehow involved in this issue to my assessment.

During such freezing I observe three things:

  • a couple of netbird processes in htop have a state marked as D, i.e. D Uninterruptible sleep (usually IO). See the attached screenshot.
  • Kernel threads, e.g. kworker/1:3+wg-crypt-wt0 is running at 100% CPU load. See attached screenshot.
  • The statistics of wt0 interface (ip -s -h link) which is netbird's wireguard interface has enormous number of transmitted packets and it abruptly increases during the freeze. See the attached screenshot.

To Reproduce
From my perspective it happens randomly, I have no clue how to force netbird into that state on demand.

I am using Arch Linux:

$ uname -a
Linux lazer 6.6.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 29 Nov 2023 00:37:40 +0000 x86_64 GNU/Linux

Netbird version:

$ paru -Si netbird-bin
Repository      : aur
Name            : netbird-bin
Version         : 0.24.3-1
Description     : WireGuard-based mesh network
URL             : https://netbird.io
AUR URL         : https://aur.archlinux.org/packages/netbird-bin
Groups          : None
Licenses        : BSD
Provides        : netbird  wiretrustee
Depends On      : glibc
Make Deps       : None
Check Deps      : None
Optional Deps   : resolvconf
Conflicts With  : netbird
Maintainer      : tarball
Votes           : 1
Popularity      : 0.00198
First Submitted : Mon, 30 May 2022 21:47:48
Last Modified   : Fri, 24 Nov 2023 13:09:24
Out Of Date     : No

Screenshots
netbird

Originally created by @s-zymon on GitHub (Dec 3, 2023). I have noticed for some time that all network traffic regularly freezes at random times for a few minutes. This happens a couple of times a day. After doing a small investigation, I find that `netbird` is somehow involved in this issue to my assessment. During such freezing I observe three things: - a couple of `netbird` processes in `htop` have a state marked as `D`, i.e. `D Uninterruptible sleep (usually IO)`. See the attached screenshot. - Kernel threads, e.g. `kworker/1:3+wg-crypt-wt0` is running at 100% CPU load. See attached screenshot. - The statistics of `wt0` interface (`ip -s -h link`) which is netbird's wireguard interface has enormous number of transmitted packets and it abruptly increases during the freeze. See the attached screenshot. **To Reproduce** From my perspective it happens randomly, I have no clue how to force `netbird` into that state on demand. I am using Arch Linux: ``` $ uname -a Linux lazer 6.6.3-arch1-1 #1 SMP PREEMPT_DYNAMIC Wed, 29 Nov 2023 00:37:40 +0000 x86_64 GNU/Linux ``` Netbird version: ``` $ paru -Si netbird-bin Repository : aur Name : netbird-bin Version : 0.24.3-1 Description : WireGuard-based mesh network URL : https://netbird.io AUR URL : https://aur.archlinux.org/packages/netbird-bin Groups : None Licenses : BSD Provides : netbird wiretrustee Depends On : glibc Make Deps : None Check Deps : None Optional Deps : resolvconf Conflicts With : netbird Maintainer : tarball Votes : 1 Popularity : 0.00198 First Submitted : Mon, 30 May 2022 21:47:48 Last Modified : Fri, 24 Nov 2023 13:09:24 Out Of Date : No ``` **Screenshots** ![netbird](https://github.com/netbirdio/netbird/assets/12126978/c5de6fb8-2221-4ad7-9c3a-b9a2ff6a6cc7)
saavagebueno added the bugagent labels 2025-11-20 05:13:07 -05:00
Author
Owner

@pappz commented on GitHub (Dec 4, 2023):

Hi @s-zymon
Could you send to me the logs with debug log level from your agent?
Do you have any relevant ACL or route settings for this machine?
An incorrect route settings can cause temporary network split and unexpected reconnect loop.

@pappz commented on GitHub (Dec 4, 2023): Hi @s-zymon Could you send to me the logs with debug log level from your agent? Do you have any relevant ACL or route settings for this machine? An incorrect route settings can cause temporary network split and unexpected reconnect loop.
Author
Owner

@s-zymon commented on GitHub (Dec 14, 2023):

@pappz, after your hints, I have investigated a bit more on my own, and indeed, it was one weird route that caused this mess. Thanks for your help.

Since it wasn't netbird's bug, I am closing it.

@s-zymon commented on GitHub (Dec 14, 2023): @pappz, after your hints, I have investigated a bit more on my own, and indeed, it was one weird route that caused this mess. Thanks for your help. Since it wasn't netbird's bug, I am closing it.
Author
Owner

@serhansevim commented on GitHub (Nov 4, 2024):

Hello @s-zymon, can you please explain further how you solved it? It would be great for people having the same issue followup with your solution. Thanks.

@serhansevim commented on GitHub (Nov 4, 2024): Hello @s-zymon, can you please explain further how you solved it? It would be great for people having the same issue followup with your solution. Thanks.
Author
Owner

@s-zymon commented on GitHub (Nov 23, 2024):

@serhansevim, in my case, the cause of this issue was netbird's network routes I set. If the netbird's routing was turned off, the issue no longer occurred. I would recommend you to check if it's the same in your case. If yes, find the route or routes that cause the issue. I just turned off and on each route and checked if the issue still occurs. After I found the routes that caused the issue, I reorganized routing rules in the network to be more sane. As @pappz suggested,

An incorrect route settings can cause a temporary network split and unexpected reconnect loop.

and I believe that was the case.

Feel free to reopen this issue and give more details about your setup. Maybe it's indeed a netbird's bug.

@s-zymon commented on GitHub (Nov 23, 2024): @serhansevim, in my case, the cause of this issue was netbird's network routes I set. If the netbird's routing was turned off, the issue no longer occurred. I would recommend you to check if it's the same in your case. If yes, find the route or routes that cause the issue. I just turned off and on each route and checked if the issue still occurs. After I found the routes that caused the issue, I reorganized routing rules in the network to be more sane. As @pappz suggested, > An incorrect route settings can cause a temporary network split and unexpected reconnect loop. and I believe that was the case. Feel free to reopen this issue and give more details about your setup. Maybe it's indeed a netbird's bug.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#529