[GH-ISSUE #3743] netbird v0.42.0 is frozen for few minutes #7934

Open
opened 2026-08-05 01:15:18 -04:00 by saavagebueno · 4 comments
Owner

Originally created by @jitbasemartin on GitHub (Apr 25, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/3743

Describe the problem

Sometimes my peer is not responding (e.g. netbird status hangs)
After investigating a little I saw that netbird was cleaning the rules but really slowly (500ms-1s per rule) as you can see in the following logs
I have around 200 rules on this peer

Are you using NetBird Cloud?

self-host

NetBird version

0.42.0

Is any other VPN software installed?

No

Debug output

...
2025-04-25T15:44:08Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.178.27.112/32-dcaf83e71c9ab6d5
2025-04-25T15:44:08Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.182.146.96/28-435c286a4b0cd0ca
2025-04-25T15:44:09Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.36.115.64/27-c4082dd3c6ccd8ac
2025-04-25T15:44:09Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 10.8.0.104/32-ba961dc5cd895ba8
2025-04-25T15:44:10Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 51.140.208.0/26-ab72054b0e36d3a5
2025-04-25T15:44:10Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.194.129.224/28-dc788f90f6ff7ad7
2025-04-25T15:44:11Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.138.68.41/32-fbd386789debd955
2025-04-25T15:44:11Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.78.195.160/27-32955a7bfe766ff2
2025-04-25T15:44:12Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.228.85.224/27-07bd3e1149ff7d12
2025-04-25T15:44:12Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.79.165.0/26-f0f225f3748db020
2025-04-25T15:44:13Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.175.18.134/32-961000a5ec40b749
2025-04-25T15:44:13Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 104.40.168.0/26-474e2e0bd66a3357
2025-04-25T15:44:14Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.153.15.0/24-e826d390b506af92
2025-04-25T15:44:14Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.79.175.0/26-cee64f073a06eafc
2025-04-25T15:44:15Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 65.52.251.96/27-475ba1bbcf11913a
2025-04-25T15:44:15Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 104.46.178.128/27-2aa735b06e3bbca3
2025-04-25T15:44:16Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.79.175.0/26-5f86c8f3cd5b0560
2025-04-25T15:44:16Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.169.218.0/32-87f16b1975b2efc9
2025-04-25T15:44:17Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 4.198.160.112/28-141ab51687e0c50a
2025-04-25T15:44:17Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.195.154.64/28-eebced112a3dc33d
2025-04-25T15:44:18Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.150.227.112/28-e95d612375816c3d
2025-04-25T15:44:18Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.74.146.192/27-168178b9059d3877
2025-04-25T15:44:19Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 74.176.1.48/30-43692edd01062614
2025-04-25T15:44:19Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 13.79.37.247/32-27e21159b8addcd0
2025-04-25T15:44:20Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.175.162.96/32-d5f646ea67ea9da4
2025-04-25T15:44:20Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.62.129.240/28-db325436cf40c6d1
2025-04-25T15:44:21Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.51.14.72/31-1063457f70bdfc73
2025-04-25T15:44:21Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 74.226.38.64/29-c92c8e0ec94e1ae6
2025-04-25T15:44:22Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.157.52.0/24-695dba1dff276f5d
2025-04-25T15:44:22Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.60.242.0/23-5950b0d2ecbb4361
2025-04-25T15:44:23Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 13.75.38.32/27-c8fd12f7c9c4ba0f
2025-04-25T15:44:23Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 13.71.175.32/27-248975752f091ac1
2025-04-25T15:44:24Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 191.234.147.0/26-e45dbc3f8ebbb1ba
2025-04-25T15:44:24Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.187.19.1/32-e754ac049c641674
2025-04-25T15:44:25Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 44.208.254.194/32-728aca4b7aaeb7e4
2025-04-25T15:44:25Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 172.173.10.80/28-0ddb90514a19615f
2025-04-25T15:44:26Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 135.225.177.112/28-9c12bf729baf8174
2025-04-25T15:44:26Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.195.64.192/27-8636216f2fea0d91
2025-04-25T15:44:27Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.226.169.45/32-2e40a9652dab29a7
...

Have you tried these troubleshooting steps?

  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @jitbasemartin on GitHub (Apr 25, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/3743 **Describe the problem** Sometimes my peer is not responding (e.g. `netbird status` hangs) After investigating a little I saw that netbird was cleaning the rules but really slowly (500ms-1s per rule) as you can see in the following logs I have around 200 rules on this peer **Are you using NetBird Cloud?** self-host **NetBird version** `0.42.0` **Is any other VPN software installed?** No **Debug output** ``` ... 2025-04-25T15:44:08Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.178.27.112/32-dcaf83e71c9ab6d5 2025-04-25T15:44:08Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.182.146.96/28-435c286a4b0cd0ca 2025-04-25T15:44:09Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.36.115.64/27-c4082dd3c6ccd8ac 2025-04-25T15:44:09Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 10.8.0.104/32-ba961dc5cd895ba8 2025-04-25T15:44:10Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 51.140.208.0/26-ab72054b0e36d3a5 2025-04-25T15:44:10Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.194.129.224/28-dc788f90f6ff7ad7 2025-04-25T15:44:11Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.138.68.41/32-fbd386789debd955 2025-04-25T15:44:11Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.78.195.160/27-32955a7bfe766ff2 2025-04-25T15:44:12Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.228.85.224/27-07bd3e1149ff7d12 2025-04-25T15:44:12Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.79.165.0/26-f0f225f3748db020 2025-04-25T15:44:13Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.175.18.134/32-961000a5ec40b749 2025-04-25T15:44:13Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 104.40.168.0/26-474e2e0bd66a3357 2025-04-25T15:44:14Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.153.15.0/24-e826d390b506af92 2025-04-25T15:44:14Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.79.175.0/26-cee64f073a06eafc 2025-04-25T15:44:15Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 65.52.251.96/27-475ba1bbcf11913a 2025-04-25T15:44:15Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 104.46.178.128/27-2aa735b06e3bbca3 2025-04-25T15:44:16Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.79.175.0/26-5f86c8f3cd5b0560 2025-04-25T15:44:16Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.169.218.0/32-87f16b1975b2efc9 2025-04-25T15:44:17Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 4.198.160.112/28-141ab51687e0c50a 2025-04-25T15:44:17Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.195.154.64/28-eebced112a3dc33d 2025-04-25T15:44:18Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.150.227.112/28-e95d612375816c3d 2025-04-25T15:44:18Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 40.74.146.192/27-168178b9059d3877 2025-04-25T15:44:19Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 74.176.1.48/30-43692edd01062614 2025-04-25T15:44:19Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 13.79.37.247/32-27e21159b8addcd0 2025-04-25T15:44:20Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.175.162.96/32-d5f646ea67ea9da4 2025-04-25T15:44:20Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.62.129.240/28-db325436cf40c6d1 2025-04-25T15:44:21Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.51.14.72/31-1063457f70bdfc73 2025-04-25T15:44:21Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 74.226.38.64/29-c92c8e0ec94e1ae6 2025-04-25T15:44:22Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.157.52.0/24-695dba1dff276f5d 2025-04-25T15:44:22Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.60.242.0/23-5950b0d2ecbb4361 2025-04-25T15:44:23Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 13.75.38.32/27-c8fd12f7c9c4ba0f 2025-04-25T15:44:23Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 13.71.175.32/27-248975752f091ac1 2025-04-25T15:44:24Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 191.234.147.0/26-e45dbc3f8ebbb1ba 2025-04-25T15:44:24Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.187.19.1/32-e754ac049c641674 2025-04-25T15:44:25Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 44.208.254.194/32-728aca4b7aaeb7e4 2025-04-25T15:44:25Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 172.173.10.80/28-0ddb90514a19615f 2025-04-25T15:44:26Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 135.225.177.112/28-9c12bf729baf8174 2025-04-25T15:44:26Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 20.195.64.192/27-8636216f2fea0d91 2025-04-25T15:44:27Z DEBG client/firewall/nftables/router_linux.go:556: removed route rule 52.226.169.45/32-2e40a9652dab29a7 ... ``` **Have you tried these troubleshooting steps?** - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2026-08-05 01:15:18 -04:00
Author
Owner

@jitbasemartin commented on GitHub (Apr 25, 2025):

At the end of the removed rules:

2025-04-25T21:49:05Z DEBG client/firewall/nftables/router_linux.go:537: Deleted unused ipset nb-8e291a3e
2025-04-25T21:49:05Z INFO client/internal/acl/manager.go:66: ACL rules processed in: 8m28.554706137s, total rules count: 68
2025-04-25T21:49:05Z DEBG client/internal/engine.go:983: got peers update from Management Service, total peers to connect to = 17
<!-- gh-comment-id:2831496732 --> @jitbasemartin commented on GitHub (Apr 25, 2025): At the end of the removed rules: ``` 2025-04-25T21:49:05Z DEBG client/firewall/nftables/router_linux.go:537: Deleted unused ipset nb-8e291a3e 2025-04-25T21:49:05Z INFO client/internal/acl/manager.go:66: ACL rules processed in: 8m28.554706137s, total rules count: 68 2025-04-25T21:49:05Z DEBG client/internal/engine.go:983: got peers update from Management Service, total peers to connect to = 17 ```
Author
Owner

@jitbasemartin commented on GitHub (Apr 30, 2025):

Still here in v0.43.1

2025-05-01T00:27:07Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.194.129.224/28-07b7642733ff7a55
2025-05-01T00:27:08Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 13.87.123.160/27-603c79ddeab2127a
2025-05-01T00:27:08Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 13.69.66.160/27-d21657f391694349
2025-05-01T00:27:08Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 104.16.98.215/32-e5664a843b4a6452
2025-05-01T00:27:08Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.48.197.112/30-a655bfb99388a887
2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 51.12.231.32/28-07184233b29e7edc
2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 52.146.132.96/27-1c2a1488042de7ff
2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 40.80.55.32/28-db3bf56e36a76e92
2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.150.100.0/24-66b662a58552cbf7
2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 52.228.96.5/32-eae2dba8a21093ee
2025-05-01T00:27:10Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.60.42.0/23-608f3d3a23ed0930
2025-05-01T00:27:10Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.62.129.0/27-5b53758db9aebee9
2025-05-01T00:27:10Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 40.79.179.128/27-9ec904ec4fe9889f
2025-05-01T00:27:10Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.18.183.32/28-cf51dbe6a0f20785
2025-05-01T00:27:11Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 135.225.177.112/28-65f95eadce52a078
2025-05-01T00:27:11Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 13.74.203.80/28-076562db3477f33e
2025-05-01T00:27:11Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 74.226.38.64/29-cb494d1e32b00870
2025-05-01T00:27:11Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 172.172.255.96/28-670a313f59811764
2025-05-01T00:27:12Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 40.79.149.192/26-170525d47ea3b7ca
2025-05-01T00:27:12Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 52.138.229.112/28-6221bbe33226dd8f
2025-05-01T00:27:12Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.61.99.128/28-25a4e4fa1c439f26
2025-05-01T00:27:12Z INFO client/internal/acl/manager.go:67: ACL rules processed in: 4m17.223596253s, total rules count: 64
<!-- gh-comment-id:2843795278 --> @jitbasemartin commented on GitHub (Apr 30, 2025): Still here in `v0.43.1` ``` 2025-05-01T00:27:07Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.194.129.224/28-07b7642733ff7a55 2025-05-01T00:27:08Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 13.87.123.160/27-603c79ddeab2127a 2025-05-01T00:27:08Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 13.69.66.160/27-d21657f391694349 2025-05-01T00:27:08Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 104.16.98.215/32-e5664a843b4a6452 2025-05-01T00:27:08Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.48.197.112/30-a655bfb99388a887 2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 51.12.231.32/28-07184233b29e7edc 2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 52.146.132.96/27-1c2a1488042de7ff 2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 40.80.55.32/28-db3bf56e36a76e92 2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.150.100.0/24-66b662a58552cbf7 2025-05-01T00:27:09Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 52.228.96.5/32-eae2dba8a21093ee 2025-05-01T00:27:10Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.60.42.0/23-608f3d3a23ed0930 2025-05-01T00:27:10Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.62.129.0/27-5b53758db9aebee9 2025-05-01T00:27:10Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 40.79.179.128/27-9ec904ec4fe9889f 2025-05-01T00:27:10Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.18.183.32/28-cf51dbe6a0f20785 2025-05-01T00:27:11Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 135.225.177.112/28-65f95eadce52a078 2025-05-01T00:27:11Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 13.74.203.80/28-076562db3477f33e 2025-05-01T00:27:11Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 74.226.38.64/29-cb494d1e32b00870 2025-05-01T00:27:11Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 172.172.255.96/28-670a313f59811764 2025-05-01T00:27:12Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 40.79.149.192/26-170525d47ea3b7ca 2025-05-01T00:27:12Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 52.138.229.112/28-6221bbe33226dd8f 2025-05-01T00:27:12Z DEBG client/firewall/nftables/router_linux.go:569: removed route rule 20.61.99.128/28-25a4e4fa1c439f26 2025-05-01T00:27:12Z INFO client/internal/acl/manager.go:67: ACL rules processed in: 4m17.223596253s, total rules count: 64 ```
Author
Owner

@mlsmaycon commented on GitHub (May 1, 2025):

Hello @jitbasemartin can you please run the following command to generate a debug bundle and upload them to our system?

netbird debug for <Minutes>m --system-info --upload-bundle --anonymize

Replace <Minutes> with the time it is taking for your client to stop.

This will output a local file location and an upload key; you can share the key with us.

<!-- gh-comment-id:2844223387 --> @mlsmaycon commented on GitHub (May 1, 2025): Hello @jitbasemartin can you please run the following command to generate a debug bundle and upload them to our system? ```shell netbird debug for <Minutes>m --system-info --upload-bundle --anonymize ``` > Replace `<Minutes>` with the time it is taking for your client to stop. This will output a local file location and an upload key; you can share the key with us.
Author
Owner

@jitbasemartin commented on GitHub (May 1, 2025):

Hello @mlsmaycon

Thank you for the instructions

Here the debug bundle when a new peer is added

$ netbird debug for 10m --system-info --upload-bundle --anonymize
Netbird down
Netbird up
Remaining time: 00:00:01
Duration completed
Creating debug bundle...
Local file:
/tmp/netbird.debug.1904067905.zip
Upload file key:
d7d9b9fb5d0594799bf41381feeddd8810a593a0d5f5522f7c8a547d82541d26/2225652e-f493-43e6-89a0-10f2fbc00dd6

PS: The netbird down took a long time also

<!-- gh-comment-id:2845155100 --> @jitbasemartin commented on GitHub (May 1, 2025): Hello @mlsmaycon Thank you for the instructions Here the debug bundle when a new peer is added ``` $ netbird debug for 10m --system-info --upload-bundle --anonymize Netbird down Netbird up Remaining time: 00:00:01 Duration completed Creating debug bundle... Local file: /tmp/netbird.debug.1904067905.zip Upload file key: d7d9b9fb5d0594799bf41381feeddd8810a593a0d5f5522f7c8a547d82541d26/2225652e-f493-43e6-89a0-10f2fbc00dd6 ``` PS: The `netbird down` took a long time also
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#7934