[GH-ISSUE #4578] NetBird routes and/or policies disappear (are removed by systemd-networkd) #8980

Open
opened 2026-08-05 01:20:17 -04:00 by saavagebueno · 1 comment
Owner

Originally created by @nazarewk on GitHub (Oct 2, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/4578

Describe the problem

This issue was discovered internally, but we are reporting it for visibility and the benefit of the community.

In short, systemd-networkd is gradually introducing the deletion of "foreign" (external to networkd) configurations upon startup. This is known to disrupt some networking (especially VPN) software, including NetBird.

We don't know the exact scenarios resulting in systemd-networkd.service restarting, but one possibility might be recovering from sleep/suspend.

The default configuration is as follows:

  • ManageForeignRoutes=true since v246 - possibly affects NetBird
  • 💣 ManageForeignRoutingPolicyRules=true since v249 - confirmed to affect NetBird
  • ManageForeignNextHops=true since v256 - I don't think we use this one as of writing the report

We are still considering a number of different countermeasures to take, but you can disable this behaviour yourself by creating a file at /etc/systemd/networkd.conf.d/10-dont-break-vpn.conf (the name doesn't really matter as long as it ends with .conf) with the following content:

[Network]
ManageForeignRoutes=false
ManageForeignRoutingPolicyRules=false
ManageForeignNextHops=false

To Reproduce

To check whether you are affected see if systemd-networkd is running:

> systemctl status systemd-networkd.service
● systemd-networkd.service - Network Configuration
     Loaded: loaded (/etc/systemd/system/systemd-networkd.service; enabled; preset: ignored)
...
     Active: active (running) since Thu 2025-10-02 11:16:22 CEST; 6h ago
...

Then try to reproduce the issue:

ip rule
netbird up
ip rule
systemctl restart systemd-networkd
ip rule

on the 2nd run of ip rule you will notice 2 new rules:

105:	from all lookup main suppress_prefixlength 0
110:	not from all fwmark 0x1bd00 lookup 7120

on the 3rd run of ip rule you will notice the above rules being deleted

Expected behavior

The rules are either preserved and/or recreated reasonably soon (a few seconds) after removal.

Are you using NetBird Cloud?

n/a

NetBird version

n/a

Is any other VPN software installed?

n/a

Debug output

n/a

Screenshots

n/a

Additional context

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @nazarewk on GitHub (Oct 2, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/4578 **Describe the problem** This issue was discovered internally, but we are reporting it for visibility and the benefit of the community. In short, `systemd-networkd` is gradually introducing the deletion of "foreign" (external to `networkd`) configurations upon startup. This is known to disrupt some networking (especially VPN) software, including NetBird. We don't know the exact scenarios resulting in `systemd-networkd.service` restarting, but one possibility might be recovering from sleep/suspend. The default configuration is as follows: - ❓ `ManageForeignRoutes=true` since `v246` - possibly affects NetBird - 💣 `ManageForeignRoutingPolicyRules=true` since `v249` - confirmed to affect NetBird - ❔ `ManageForeignNextHops=true` since `v256` - I don't think we use this one as of writing the report We are still considering a number of different countermeasures to take, but you can disable this behaviour yourself by creating a file at `/etc/systemd/networkd.conf.d/10-dont-break-vpn.conf` (the name doesn't really matter as long as it ends with `.conf`) with the following content: ```text [Network] ManageForeignRoutes=false ManageForeignRoutingPolicyRules=false ManageForeignNextHops=false ``` **To Reproduce** To check whether you are affected see if `systemd-networkd` is running: ```text > systemctl status systemd-networkd.service ● systemd-networkd.service - Network Configuration Loaded: loaded (/etc/systemd/system/systemd-networkd.service; enabled; preset: ignored) ... Active: active (running) since Thu 2025-10-02 11:16:22 CEST; 6h ago ... ``` Then try to reproduce the issue: ```shell ip rule netbird up ip rule systemctl restart systemd-networkd ip rule ``` on the 2nd run of `ip rule` you will notice 2 new rules: ```text 105: from all lookup main suppress_prefixlength 0 110: not from all fwmark 0x1bd00 lookup 7120 ``` on the 3rd run of `ip rule` you will notice the above rules being deleted **Expected behavior** The rules are either preserved and/or recreated reasonably soon (a few seconds) after removal. **Are you using NetBird Cloud?** n/a **NetBird version** n/a **Is any other VPN software installed?** n/a **Debug output** n/a **Screenshots** n/a **Additional context** - https://github.com/systemd/systemd/issues/19106 - https://web.archive.org/web/20250906155911/https://www.freedesktop.org/software/systemd/man/latest/networkd.conf.html#ManageForeignRoutingPolicyRules= - https://web.archive.org/web/20251001120906/https://wiki.archlinux.org/title/Systemd-networkd#Usage - https://web.archive.org/web/20250930224010/https://wiki.archlinux.org/title/WireGuard#Connection_lost_after_sleep_using_systemd-networkd **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the bugclientnetworking labels 2026-08-05 01:20:18 -04:00
Author
Owner

@heymoe commented on GitHub (Mar 26, 2026):

Wanted to add another scenario which can trigger this issue.

Running Netbird: 0.66.4

For the last 2 days I had several of our peers that utilize routes defined in NetBird to access remote devices go away (the routes not the peers). All these peers are running Ubuntu 22.04. Upon digging into the cause I found that there was a new update for systemd which got auto updated via the apt-daily-upgrade/unattended-upgrades service.

Start-Date: 2026-03-25  06:38:06
Commandline: /usr/bin/unattended-upgrade
Upgrade: libpam-systemd:amd64 (249.11-0ubuntu3.16, 249.11-0ubuntu3.19), libsystemd0:amd64 (249.11-0ubuntu3.16, 249.11-0ubuntu3.19), libnss-systemd:amd64 (24
9.11-0ubuntu3.16, 249.11-0ubuntu3.19), systemd:amd64 (249.11-0ubuntu3.16, 249.11-0ubuntu3.19), systemd-sysv:amd64 (249.11-0ubuntu3.16, 249.11-0ubuntu3.19)
End-Date: 2026-03-25  06:38:09

Upon doing the systemd upgrade, it looks like a bunch of systemd-* related services including systemd-networkd are stopped and then started.

Mar 25 06:38:00 XXXXX systemd[1]: Starting Daily apt upgrade and clean activities...
Mar 25 06:38:07 XXXXX systemd[1]: Reexecuting.
Mar 25 06:38:07 XXXXX systemd[1]: systemd 249.11-0ubuntu3.19 running in system mode (... +>
Mar 25 06:38:08 XXXXX systemd[1]: systemd-networkd-wait-online.service: Deactivated successfully.
Mar 25 06:38:08 XXXXX systemd[1]: Stopped Wait for Network to be Configured.
Mar 25 06:38:08 XXXXX systemd[1]: Stopping Wait for Network to be Configured...
Mar 25 06:38:08 XXXXX systemd[1]: Stopping Network Configuration...
Mar 25 06:38:08 XXXXX systemd-networkd[3924128]: ens5: DHCPv6 lease lost
Mar 25 06:38:08 XXXXX systemd[1]: systemd-networkd.service: Deactivated successfully.
Mar 25 06:38:08 XXXXX systemd[1]: Stopped Network Configuration.
Mar 25 06:38:08 XXXXX systemd[1]: systemd-networkd.service: Consumed 14.497s CPU time.
Mar 25 06:38:08 XXXXX systemd[1]: Starting Network Configuration...
Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: wt0: Link UP
Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: wt0: Gained carrier
Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: ens5: Link UP
Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: ens5: Gained carrier
Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: lo: Link UP
Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: lo: Gained carrier
Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: ens5: Gained IPv6LL
Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: Enumeration completed
Mar 25 06:38:08 XXXXX systemd[1]: Started Network Configuration.

And of course when systemd-networkd restarts, all the Netbird routes go away.

I did manually tested if stopping and starting (or just restarting) systemd-networkd caused the routes to go away and indeed it does.

The current work around I setup to prevent this from breaking things in the future is a cron job that checks if the Netbird routes are gone and if so restart Netbird.

Ideally it would be nice if the Netbird client itself can check if its routes are gone or not and if they are readd them or maybe monitor the status of systemd-networkd and if it detects it restarted to readd the routes.

<!-- gh-comment-id:4135633217 --> @heymoe commented on GitHub (Mar 26, 2026): Wanted to add another scenario which can trigger this issue. Running Netbird: 0.66.4 For the last 2 days I had several of our peers that utilize routes defined in NetBird to access remote devices go away (the routes not the peers). All these peers are running Ubuntu 22.04. Upon digging into the cause I found that there was a new update for systemd which got auto updated via the apt-daily-upgrade/unattended-upgrades service. ``` Start-Date: 2026-03-25 06:38:06 Commandline: /usr/bin/unattended-upgrade Upgrade: libpam-systemd:amd64 (249.11-0ubuntu3.16, 249.11-0ubuntu3.19), libsystemd0:amd64 (249.11-0ubuntu3.16, 249.11-0ubuntu3.19), libnss-systemd:amd64 (24 9.11-0ubuntu3.16, 249.11-0ubuntu3.19), systemd:amd64 (249.11-0ubuntu3.16, 249.11-0ubuntu3.19), systemd-sysv:amd64 (249.11-0ubuntu3.16, 249.11-0ubuntu3.19) End-Date: 2026-03-25 06:38:09 ``` Upon doing the systemd upgrade, it looks like a bunch of systemd-* related services including systemd-networkd are stopped and then started. ``` Mar 25 06:38:00 XXXXX systemd[1]: Starting Daily apt upgrade and clean activities... Mar 25 06:38:07 XXXXX systemd[1]: Reexecuting. Mar 25 06:38:07 XXXXX systemd[1]: systemd 249.11-0ubuntu3.19 running in system mode (... +> Mar 25 06:38:08 XXXXX systemd[1]: systemd-networkd-wait-online.service: Deactivated successfully. Mar 25 06:38:08 XXXXX systemd[1]: Stopped Wait for Network to be Configured. Mar 25 06:38:08 XXXXX systemd[1]: Stopping Wait for Network to be Configured... Mar 25 06:38:08 XXXXX systemd[1]: Stopping Network Configuration... Mar 25 06:38:08 XXXXX systemd-networkd[3924128]: ens5: DHCPv6 lease lost Mar 25 06:38:08 XXXXX systemd[1]: systemd-networkd.service: Deactivated successfully. Mar 25 06:38:08 XXXXX systemd[1]: Stopped Network Configuration. Mar 25 06:38:08 XXXXX systemd[1]: systemd-networkd.service: Consumed 14.497s CPU time. Mar 25 06:38:08 XXXXX systemd[1]: Starting Network Configuration... Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: wt0: Link UP Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: wt0: Gained carrier Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: ens5: Link UP Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: ens5: Gained carrier Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: lo: Link UP Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: lo: Gained carrier Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: ens5: Gained IPv6LL Mar 25 06:38:08 XXXXX systemd-networkd[3620337]: Enumeration completed Mar 25 06:38:08 XXXXX systemd[1]: Started Network Configuration. ``` And of course when systemd-networkd restarts, all the Netbird routes go away. I did manually tested if stopping and starting (or just restarting) systemd-networkd caused the routes to go away and indeed it does. The current work around I setup to prevent this from breaking things in the future is a cron job that checks if the Netbird routes are gone and if so restart Netbird. Ideally it would be nice if the Netbird client itself can check if its routes are gone or not and if they are readd them or maybe monitor the status of systemd-networkd and if it detects it restarted to readd the routes.
Sign in to join this conversation.
No Label bug client networking
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#8980