New 0.30.x ACL features break iptables-nft routing table #1325

Closed
opened 2025-11-20 05:28:30 -05:00 by saavagebueno · 9 comments
Owner

Originally created by @Spiritreader on GitHub (Oct 10, 2024).

Describe the problem

It is no longer possible to list the routing table when netbird is active because it inserts incompatible rules.
I cannot fully switch over to nftables because docker requires iptables and modifying existing chains via nft is discouraged.

There are also side effects where you cannot insert certain rules anymore (which affects services like docker when it is creating new networks as it needs to modify iptables rules). Some iptables insertion and delete commands depend on the table being parsable.

To Reproduce

While netbird is running

❯ sudo iptables -L
iptables v1.8.9 (nf_tables): table `filter' is incompatible, use 'nft' tool.
❯ netbird down
Disconnected
❯ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (3 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             172.19.0.2           tcp dpt:http
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:http
ACCEPT     tcp  --  anywhere             172.18.0.2           tcp dpt:https

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (3 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Expected behavior

iptables work as expected (before 0.30).
Netbird does not modify any tables managed by iptables-nft via nftables.

Netbird either uses iptables commands to modify iptables-nft-managed tables, or injects its own forward chain into `table ip netbird`` by setting a lower priority than the default of iprables-nft. won't work as outlined by https://github.com/netbirdio/netbird/issues/2725#issuecomment-2407408524

Are you using NetBird Cloud?

selfhosted

NetBird version

0.30.1

Do you face any (non-mobile) client issues?

The aforementioned issue affects linux clients with iptables installed.

Originally created by @Spiritreader on GitHub (Oct 10, 2024). **Describe the problem** It is no longer possible to list the routing table when netbird is active because it inserts incompatible rules. I cannot fully switch over to nftables because docker requires iptables and modifying existing chains via nft is discouraged. There are also side effects where you cannot insert certain rules anymore (which affects services like docker when it is creating new networks as it needs to modify iptables rules). Some iptables insertion and delete commands depend on the table being parsable. **To Reproduce** While netbird is running ``` ❯ sudo iptables -L iptables v1.8.9 (nf_tables): table `filter' is incompatible, use 'nft' tool. ``` ``` ❯ netbird down Disconnected ``` ``` ❯ sudo iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain DOCKER (3 references) target prot opt source destination ACCEPT tcp -- anywhere 172.19.0.2 tcp dpt:http ACCEPT tcp -- anywhere 172.18.0.2 tcp dpt:http ACCEPT tcp -- anywhere 172.18.0.2 tcp dpt:https Chain DOCKER-ISOLATION-STAGE-1 (1 references) target prot opt source destination DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-ISOLATION-STAGE-2 (3 references) target prot opt source destination DROP all -- anywhere anywhere DROP all -- anywhere anywhere DROP all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- anywhere anywhere ``` **Expected behavior** iptables work as expected (before 0.30). Netbird does not modify any tables managed by iptables-nft via nftables. Netbird either uses iptables commands to modify iptables-nft-managed tables, ~~or injects its own forward chain into `table ip netbird`` by setting a lower priority than the default of iprables-nft.~~ won't work as outlined by https://github.com/netbirdio/netbird/issues/2725#issuecomment-2407408524 **Are you using NetBird Cloud?** selfhosted **NetBird version** 0.30.1 **Do you face any (non-mobile) client issues?** The aforementioned issue affects linux clients with iptables installed.
saavagebueno added the triage-needed label 2025-11-20 05:28:30 -05:00
Author
Owner

@MDMeridio001 commented on GitHub (Oct 10, 2024):

I'm also having the same issue on version 0.30.0.
iptables v1.8.7 (nf_tables): table `filter' is incompatible, use 'nft' tool.

@MDMeridio001 commented on GitHub (Oct 10, 2024): I'm also having the same issue on version 0.30.0. ```iptables v1.8.7 (nf_tables): table `filter' is incompatible, use 'nft' tool.```
Author
Owner

@Spiritreader commented on GitHub (Oct 10, 2024):

Yeah, I'm also not quite sure which rule is causing the issue. I would assume any rule that is inserted into that chain via nftables causes the issue?

❯ sudo nft list table filter
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
        chain DOCKER {
                iifname != "br-2cee86ba52c6" oifname "br-2cee86ba52c6" ip daddr 172.19.0.2 tcp dport 80 counter packets 0 bytes 0 accept
                iifname != "br-4018d9fe98eb" oifname "br-4018d9fe98eb" ip daddr 172.18.0.2 tcp dport 80 counter packets 0 bytes 0 accept
                iifname != "br-4018d9fe98eb" oifname "br-4018d9fe98eb" ip daddr 172.18.0.2 tcp dport 443 counter packets 5 bytes 260 accept
        }

        chain DOCKER-ISOLATION-STAGE-1 {
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2
                iifname "br-4018d9fe98eb" oifname != "br-4018d9fe98eb" counter packets 2131 bytes 2672754 jump DOCKER-ISOLATION-STAGE-2
                iifname "br-2cee86ba52c6" oifname != "br-2cee86ba52c6" counter packets 794 bytes 46356 jump DOCKER-ISOLATION-STAGE-2
                counter packets 7013 bytes 18823363 return
        }

        chain DOCKER-ISOLATION-STAGE-2 {
                oifname "docker0" counter packets 0 bytes 0 drop
                oifname "br-4018d9fe98eb" counter packets 0 bytes 0 drop
                oifname "br-2cee86ba52c6" counter packets 0 bytes 0 drop
                counter packets 2925 bytes 2719110 return
        }

        chain FORWARD {
                type filter hook forward priority filter; policy drop;
                oifname "wt0" ct state established,related counter packets 33872 bytes 176020353 accept
                iifname "wt0" counter packets 124650 bytes 227839202 accept
                counter packets 7013 bytes 18823363 jump DOCKER-USER
                counter packets 7013 bytes 18823363 jump DOCKER-ISOLATION-STAGE-1
                oifname "docker0" ct state related,established counter packets 0 bytes 0 accept
                oifname "docker0" counter packets 0 bytes 0 jump DOCKER
                iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept
                iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
                oifname "br-4018d9fe98eb" ct state related,established counter packets 3186 bytes 7314863 accept
                oifname "br-4018d9fe98eb" counter packets 5 bytes 260 jump DOCKER
                iifname "br-4018d9fe98eb" oifname != "br-4018d9fe98eb" counter packets 2131 bytes 2672754 accept
                iifname "br-4018d9fe98eb" oifname "br-4018d9fe98eb" counter packets 0 bytes 0 accept
                oifname "br-2cee86ba52c6" ct state related,established counter packets 897 bytes 8789130 accept
                oifname "br-2cee86ba52c6" counter packets 0 bytes 0 jump DOCKER
                iifname "br-2cee86ba52c6" oifname != "br-2cee86ba52c6" counter packets 794 bytes 46356 accept
                iifname "br-2cee86ba52c6" oifname "br-2cee86ba52c6" counter packets 0 bytes 0 accept
        }

        chain DOCKER-USER {
                counter packets 7013 bytes 18823363 return
        }
}

It might be necessary to insert those rules

oifname "wt0" ct state established,related counter packets 33872 bytes 176020353 accept
iifname "wt0" counter packets 124650 bytes 227839202 accept

via iptables, something like

-A FORWARD -i wt0 -m state -j ACCEPT
-A FORWARD -o wt0 -m state --state RELATED,ESTABLISHED -j ACCEPT

instead of using nft add ...

@Spiritreader commented on GitHub (Oct 10, 2024): Yeah, I'm also not quite sure which rule is causing the issue. I would assume any rule that is inserted into that chain via nftables causes the issue? ``` ❯ sudo nft list table filter # Warning: table ip filter is managed by iptables-nft, do not touch! table ip filter { chain DOCKER { iifname != "br-2cee86ba52c6" oifname "br-2cee86ba52c6" ip daddr 172.19.0.2 tcp dport 80 counter packets 0 bytes 0 accept iifname != "br-4018d9fe98eb" oifname "br-4018d9fe98eb" ip daddr 172.18.0.2 tcp dport 80 counter packets 0 bytes 0 accept iifname != "br-4018d9fe98eb" oifname "br-4018d9fe98eb" ip daddr 172.18.0.2 tcp dport 443 counter packets 5 bytes 260 accept } chain DOCKER-ISOLATION-STAGE-1 { iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 jump DOCKER-ISOLATION-STAGE-2 iifname "br-4018d9fe98eb" oifname != "br-4018d9fe98eb" counter packets 2131 bytes 2672754 jump DOCKER-ISOLATION-STAGE-2 iifname "br-2cee86ba52c6" oifname != "br-2cee86ba52c6" counter packets 794 bytes 46356 jump DOCKER-ISOLATION-STAGE-2 counter packets 7013 bytes 18823363 return } chain DOCKER-ISOLATION-STAGE-2 { oifname "docker0" counter packets 0 bytes 0 drop oifname "br-4018d9fe98eb" counter packets 0 bytes 0 drop oifname "br-2cee86ba52c6" counter packets 0 bytes 0 drop counter packets 2925 bytes 2719110 return } chain FORWARD { type filter hook forward priority filter; policy drop; oifname "wt0" ct state established,related counter packets 33872 bytes 176020353 accept iifname "wt0" counter packets 124650 bytes 227839202 accept counter packets 7013 bytes 18823363 jump DOCKER-USER counter packets 7013 bytes 18823363 jump DOCKER-ISOLATION-STAGE-1 oifname "docker0" ct state related,established counter packets 0 bytes 0 accept oifname "docker0" counter packets 0 bytes 0 jump DOCKER iifname "docker0" oifname != "docker0" counter packets 0 bytes 0 accept iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept oifname "br-4018d9fe98eb" ct state related,established counter packets 3186 bytes 7314863 accept oifname "br-4018d9fe98eb" counter packets 5 bytes 260 jump DOCKER iifname "br-4018d9fe98eb" oifname != "br-4018d9fe98eb" counter packets 2131 bytes 2672754 accept iifname "br-4018d9fe98eb" oifname "br-4018d9fe98eb" counter packets 0 bytes 0 accept oifname "br-2cee86ba52c6" ct state related,established counter packets 897 bytes 8789130 accept oifname "br-2cee86ba52c6" counter packets 0 bytes 0 jump DOCKER iifname "br-2cee86ba52c6" oifname != "br-2cee86ba52c6" counter packets 794 bytes 46356 accept iifname "br-2cee86ba52c6" oifname "br-2cee86ba52c6" counter packets 0 bytes 0 accept } chain DOCKER-USER { counter packets 7013 bytes 18823363 return } } ``` It might be necessary to insert those rules ``` oifname "wt0" ct state established,related counter packets 33872 bytes 176020353 accept iifname "wt0" counter packets 124650 bytes 227839202 accept ``` via iptables, something like ``` -A FORWARD -i wt0 -m state -j ACCEPT -A FORWARD -o wt0 -m state --state RELATED,ESTABLISHED -j ACCEPT ``` instead of using `nft add ...`
Author
Owner

@horzadome commented on GitHub (Oct 11, 2024):

FWIW this completely broke Docker forwarding for me on multiple Ubuntu hosts, but not on all of them.
I have no idea how some of them were unaffected.
The best correlation that I see is that those that broke had docker, nftables and UFW enabled, while others didn't have UFW enabled.
But UFW kept working while Docker stopped exposing anything, presumably because iptables command return incompatible rule error.
Fixed it by stopping netbird, uninstalling nftables and libnftables, iptables flush and reboot. Everything correctly loaded its rules and iptables command worked again.

@horzadome commented on GitHub (Oct 11, 2024): FWIW this completely broke Docker forwarding for me on multiple Ubuntu hosts, but not on all of them. I have no idea how some of them were unaffected. The best correlation that I see is that those that broke had docker, nftables and UFW enabled, while others didn't have UFW enabled. But UFW kept working while Docker stopped exposing anything, presumably because iptables command return incompatible rule error. Fixed it by stopping netbird, uninstalling nftables and libnftables, iptables flush and reboot. Everything correctly loaded its rules and iptables command worked again.
Author
Owner

@Spiritreader commented on GitHub (Oct 11, 2024):

FWIW this completely broke Docker forwarding for me on multiple Ubuntu hosts, but not on all of them. I have no idea how some of them were unaffected. The best correlation that I see is that those that broke had docker, nftables and UFW enabled, while others didn't have UFW enabled. But UFW kept working while Docker stopped exposing anything, presumably because iptables command return incompatible rule error. Fixed it by stopping netbird, uninstalling nftables and libnftables, iptables flush and reboot. Everything correctly loaded its rules and iptables command worked again.

Oh you're right, it's no longer possible to add certain rules that rely on the table being parsable, I'll adjust the title. I did encounter that but since I didn't make any changes to docker yet after updating to 0.30, I haven't noticed so far.

As a workaround in the meantime if you don't want to downgrade, you can try running the netbird client in a docker container via network_mode=host
However that will come with added configuration to make DNS function correctly. That's what I'm doing on some of my machines and the host iptables remain intact this way.

@Spiritreader commented on GitHub (Oct 11, 2024): > FWIW this completely broke Docker forwarding for me on multiple Ubuntu hosts, but not on all of them. I have no idea how some of them were unaffected. The best correlation that I see is that those that broke had docker, nftables and UFW enabled, while others didn't have UFW enabled. But UFW kept working while Docker stopped exposing anything, presumably because iptables command return incompatible rule error. Fixed it by stopping netbird, uninstalling nftables and libnftables, iptables flush and reboot. Everything correctly loaded its rules and iptables command worked again. Oh you're right, it's no longer possible to add certain rules that rely on the table being parsable, I'll adjust the title. I did encounter that but since I didn't make any changes to docker yet after updating to 0.30, I haven't noticed so far. As a workaround in the meantime if you don't want to downgrade, you can try running the netbird client in a docker container via `network_mode=host` However that will come with added configuration to make DNS function correctly. That's what I'm doing on some of my machines and the host iptables remain intact this way.
Author
Owner

@lixmal commented on GitHub (Oct 11, 2024):

This will be fixed in https://github.com/netbirdio/netbird/pull/2727

Meanwhile you can try setting env NB_SKIP_NFTABLES_CHECK=true to use iptables only.

Netbird either uses iptables commands to modify iptables-nft-managed tables, or injects its own forward chain into `table ip netbird`` by setting a lower priority than the default of iprables-nft.

The latter is not feasible since an accept verdict won't stop the iptables-nft chain from being traversed, unlike a drop verdict. (We already have a filter chain in table netbird)

The culprit is -o wt0 -m state --state RELATED,ESTABLISHED, which must be a bug since the iptables-translate emits the same iptables-nft incompatible rule

@lixmal commented on GitHub (Oct 11, 2024): This will be fixed in https://github.com/netbirdio/netbird/pull/2727 Meanwhile you can try setting env `NB_SKIP_NFTABLES_CHECK=true` to use iptables only. >Netbird either uses iptables commands to modify iptables-nft-managed tables, or injects its own forward chain into `table ip netbird`` by setting a lower priority than the default of iprables-nft. The latter is not feasible since an `accept` verdict won't stop the iptables-nft chain from being traversed, unlike a `drop` verdict. (We already have a `filter` chain in table `netbird`) The culprit is `-o wt0 -m state --state RELATED,ESTABLISHED`, which must be a bug since the `iptables-translate` emits the same iptables-nft incompatible rule
Author
Owner

@MDMeridio001 commented on GitHub (Oct 11, 2024):

This will be fixed in #2727

Meanwhile you can try setting env NB_SKIP_NFTABLES_CHECK=true to use iptables only.

Netbird either uses iptables commands to modify iptables-nft-managed tables, or injects its own forward chain into `table ip netbird`` by setting a lower priority than the default of iprables-nft.

The latter is not feasible since an accept verdict won't stop the iptables-nft chain from being traversed, unlike a drop verdict. (We already have a filter chain in table netbird)

The culprit is -o wt0 -m state --state RELATED,ESTABLISHED, which must be a bug since the iptables-translate emits the same iptables-nft incompatible rule

I've tried the suggested workaround but I still get the same error from iptables.

❯ netbird down
Disconnected
❯ export NB_SKIP_NFTABLES_CHECK=true
❯ netbird up
Connected
❯ sudo iptables -L
iptables v1.8.7 (nf_tables): table `filter' is incompatible, use 'nft' tool.
@MDMeridio001 commented on GitHub (Oct 11, 2024): > This will be fixed in #2727 > > Meanwhile you can try setting env `NB_SKIP_NFTABLES_CHECK=true` to use iptables only. > > > Netbird either uses iptables commands to modify iptables-nft-managed tables, or injects its own forward chain into `table ip netbird`` by setting a lower priority than the default of iprables-nft. > > The latter is not feasible since an `accept` verdict won't stop the iptables-nft chain from being traversed, unlike a `drop` verdict. (We already have a `filter` chain in table `netbird`) > > The culprit is `-o wt0 -m state --state RELATED,ESTABLISHED`, which must be a bug since the `iptables-translate` emits the same iptables-nft incompatible rule I've tried the suggested workaround but I still get the same error from iptables. ``` ❯ netbird down Disconnected ``` ``` ❯ export NB_SKIP_NFTABLES_CHECK=true ``` ``` ❯ netbird up Connected ``` ``` ❯ sudo iptables -L iptables v1.8.7 (nf_tables): table `filter' is incompatible, use 'nft' tool. ```
Author
Owner

@lixmal commented on GitHub (Oct 11, 2024):

The env won't be picked up by the daemon, try it similar to here https://docs.netbird.io/how-to/troubleshooting-client#on-linux-with-systemd

@lixmal commented on GitHub (Oct 11, 2024): The env won't be picked up by the daemon, try it similar to here https://docs.netbird.io/how-to/troubleshooting-client#on-linux-with-systemd
Author
Owner

@MDMeridio001 commented on GitHub (Oct 11, 2024):

The env won't be picked up by the daemon, try it similar to here https://docs.netbird.io/how-to/troubleshooting-client#on-linux-with-systemd

Ok, can confirm that it works.

❯ sudo mkdir -p /etc/sysconfig
❯ echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a /etc/sysconfig/netbird
NB_SKIP_NFTABLES_CHECK=true
❯ sudo systemctl restart netbird
@MDMeridio001 commented on GitHub (Oct 11, 2024): > The env won't be picked up by the daemon, try it similar to here https://docs.netbird.io/how-to/troubleshooting-client#on-linux-with-systemd Ok, can confirm that it works. ``` ❯ sudo mkdir -p /etc/sysconfig ``` ``` ❯ echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a /etc/sysconfig/netbird NB_SKIP_NFTABLES_CHECK=true ``` ``` ❯ sudo systemctl restart netbird ```
Author
Owner

@Spiritreader commented on GitHub (Oct 17, 2024):

Fixed in 0.30.2

I un-set the abovementioned environment variable, and restarted the netbird service.
The iptables filter table now remains functional and the rest of the rules show up correctly if queried via nft.

Thanks a lot for the quick response and implementation!!

@Spiritreader commented on GitHub (Oct 17, 2024): Fixed in 0.30.2 I un-set the abovementioned environment variable, and restarted the netbird service. The iptables filter table now remains functional and the rest of the rules show up correctly if queried via `nft`. Thanks a lot for the quick response and implementation!!
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1325