SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-07-27 16:32:34 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

NETBIRD Disrupts NAT on CentOS 7 #919

New Issue
Closed
opened 2025-11-20 05:19:53 -05:00 by saavagebueno · 8 comments
saavagebueno commented 2025-11-20 05:19:53 -05:00
Owner
Copy Link

Originally created by @thorleifjacobsen on GitHub (May 22, 2024).

Description
When enabling NETBIRD on CentOS 7, NAT (Network Address Translation) stops functioning properly on two different networks (10.0.4.x and 10.0.2.x). Disabling NETBIRD restores NAT functionality. The interfaces 10.0.4.x and 10.0.2.x are not related to NETBIRD.

image

Current IPTables:

*nat
:PREROUTING ACCEPT [6415:852871]
:INPUT ACCEPT [6228:840858]
:OUTPUT ACCEPT [1942:137692]
:POSTROUTING ACCEPT [288:22279]
-A PREROUTING -p udp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024,1900,5353 -j DNAT --to-destination 10.0.4.236
-A PREROUTING -p tcp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024 -j DNAT --to-destination 10.0.4.236
-A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233
COMMIT
# Completed on Wed May 22 13:56:11 2024
# Generated by iptables-save v1.4.21 on Wed May 22 13:56:11 2024
*filter
:INPUT ACCEPT [98280:30756187]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [51952:10249852]
-A INPUT ! -i lo -p tcp -m tcp --dport 3306 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i oc__+ -p tcp -m multiport --dports 22 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i enp0s8 -p tcp -m multiport --dports 8082 -j REJECT --reject-with icmp-port-unreachable
-A INPUT -i enp0s9 -p tcp -m multiport --dports 8080,8083 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i enp0s8 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -d 10.0.4.236/32 -p udp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024,1900,5353 -j ACCEPT
-A FORWARD -d 10.0.4.236/32 -p tcp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024 -j ACCEPT

Steps to Reproduce
Set up two network interfaces with IP ranges 10.0.4.x and 10.0.2.x.
Enable NAT on these interfaces.
Install and enable NETBIRD.
Observe that NAT stops working on both interfaces.
Disable NETBIRD.
Observe that NAT functionality is restored.
Expected Behavior
Enabling NETBIRD should not affect NAT functionality on unrelated network interfaces.

Actual Behavior
Enabling NETBIRD disrupts NAT on the interfaces 10.0.4.x and 10.0.2.x. Disabling NETBIRD restores NAT functionality.

Troubleshooting Steps Taken

  • Verified that iptables rules remain unchanged before and after enabling NETBIRD.
  • Confirmed that firewalld is not installed.
  • Compared routing tables using ip route show before and after enabling NETBIRD (100 and 101 added ?)
  • Inspected network interfaces with ip addr show before and after enabling NETBIRD (no changes found other than wt0
  • Checked system logs (/var/log/messages and journalctl -xe) for relevant messages (no relevant entries found).
  • Reviewed NETBIRD configuration files for any settings related to network interfaces or routes (nothing unusual found).
  • Ensured NetworkManager is not making changes when NETBIRD is enabled.
  • Verified no other services or scripts are triggered by NETBIRD that could cause network changes.

System Information
OS: CentOS 7

Additional Information

  • No changes in MTU settings observed.
  • DNS settings is irrelevant
  • No errors or warnings in NETBIRD logs.
  • Network statistics show no anomalies before and after enabling NETBIRD.

Any assistance or tips to resolve this issue would be greatly appreciated.

Originally created by @thorleifjacobsen on GitHub (May 22, 2024). **Description** When enabling NETBIRD on CentOS 7, NAT (Network Address Translation) stops functioning properly on two different networks (10.0.4.x and 10.0.2.x). Disabling NETBIRD restores NAT functionality. The interfaces 10.0.4.x and 10.0.2.x are not related to NETBIRD. ![image](https://github.com/netbirdio/netbird/assets/8550684/e1adb4b5-faa8-483e-89ac-6709fdc519a4) **Current IPTables:** ``` *nat :PREROUTING ACCEPT [6415:852871] :INPUT ACCEPT [6228:840858] :OUTPUT ACCEPT [1942:137692] :POSTROUTING ACCEPT [288:22279] -A PREROUTING -p udp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024,1900,5353 -j DNAT --to-destination 10.0.4.236 -A PREROUTING -p tcp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024 -j DNAT --to-destination 10.0.4.236 -A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233 COMMIT # Completed on Wed May 22 13:56:11 2024 # Generated by iptables-save v1.4.21 on Wed May 22 13:56:11 2024 *filter :INPUT ACCEPT [98280:30756187] :FORWARD DROP [0:0] :OUTPUT ACCEPT [51952:10249852] -A INPUT ! -i lo -p tcp -m tcp --dport 3306 -j REJECT --reject-with icmp-port-unreachable -A INPUT -i oc__+ -p tcp -m multiport --dports 22 -j REJECT --reject-with icmp-port-unreachable -A INPUT -i enp0s8 -p tcp -m multiport --dports 8082 -j REJECT --reject-with icmp-port-unreachable -A INPUT -i enp0s9 -p tcp -m multiport --dports 8080,8083 -j REJECT --reject-with icmp-port-unreachable -A FORWARD -i enp0s8 -m state --state RELATED,ESTABLISHED -j ACCEPT -A FORWARD -d 10.0.4.236/32 -p udp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024,1900,5353 -j ACCEPT -A FORWARD -d 10.0.4.236/32 -p tcp -m mac --mac-source 96:4F:1D:XX:XX:XX -m multiport ! --dports 0:1024 -j ACCEPT ``` **Steps to Reproduce** Set up two network interfaces with IP ranges 10.0.4.x and 10.0.2.x. Enable NAT on these interfaces. Install and enable NETBIRD. Observe that NAT stops working on both interfaces. Disable NETBIRD. Observe that NAT functionality is restored. Expected Behavior Enabling NETBIRD should not affect NAT functionality on unrelated network interfaces. **Actual Behavior** Enabling NETBIRD disrupts NAT on the interfaces 10.0.4.x and 10.0.2.x. Disabling NETBIRD restores NAT functionality. **Troubleshooting Steps Taken** - Verified that iptables rules remain unchanged before and after enabling NETBIRD. - Confirmed that firewalld is not installed. - Compared routing tables using ip route show before and after enabling NETBIRD (100 and 101 added ?) - Inspected network interfaces with ip addr show before and after enabling NETBIRD (no changes found other than wt0 - Checked system logs (/var/log/messages and journalctl -xe) for relevant messages (no relevant entries found). - Reviewed NETBIRD configuration files for any settings related to network interfaces or routes (nothing unusual found). - Ensured NetworkManager is not making changes when NETBIRD is enabled. - Verified no other services or scripts are triggered by NETBIRD that could cause network changes. **System Information** OS: CentOS 7 **Additional Information** - No changes in MTU settings observed. - DNS settings is irrelevant - No errors or warnings in NETBIRD logs. - Network statistics show no anomalies before and after enabling NETBIRD. Any assistance or tips to resolve this issue would be greatly appreciated.
saavagebueno added the triage-needed label 2025-11-20 05:19:53 -05:00
saavagebueno commented 2025-11-20 05:19:53 -05:00
Author
Owner
Copy Link

@pascal-fischer commented on GitHub (May 22, 2024):

Hi @thorleifjacobsen,

this might be similar to https://github.com/netbirdio/netbird/issues/2015
Could you try the same solution and run

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird
@pascal-fischer commented on GitHub (May 22, 2024): Hi @thorleifjacobsen, this might be similar to https://github.com/netbirdio/netbird/issues/2015 Could you try the same solution and run ``` sudo mkdir -p /etc/sysconfig echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a /etc/sysconfig/netbird sudo systemctl restart netbird ```
saavagebueno commented 2025-11-20 05:19:53 -05:00
Author
Owner
Copy Link

@thorleifjacobsen commented on GitHub (May 23, 2024):

Hello! Thanks for the pointer, we did not think of checking the nft tables.

Indeed we found a problem, there is a rule in nft list tables ip netbird a chain called netbird-rt-nat by the look of it seems to prioritze over anything to accept all traffic. Which in part stops any special iptables natting.

table ip netbird {
        chain netbird-rt-fwd {
        }

       chain netbird-rt-nat {
                type nat hook postrouting priority 99; policy accept;
        }

        chain netbird-acl-input-rules {
                ip saddr & 0.0.0.0 == 0.0.0.0 accept
        }

        chain netbird-acl-output-rules {
                ip daddr & 0.0.0.0 == 0.0.0.0 accept
        }

        chain netbird-acl-input-filter {
                type filter hook input priority 0; policy accept;
                iifname "wt0" ip saddr 100.66.0.0/16 ip daddr != 100.66.0.0/16 accept
                iifname "wt0" ip saddr != 100.66.0.0/16 ip daddr 100.66.0.0/16 accept
                iifname "wt0" ip saddr 100.66.0.0/16 ip daddr 100.66.0.0/16 jump netbird-acl-input-rules
                iifname "wt0" drop
        }

        chain netbird-acl-output-filter {
                type filter hook output priority 0; policy accept;
                oifname "wt0" ip saddr != 100.66.0.0/16 ip daddr 100.66.0.0/16 accept
                oifname "wt0" ip saddr 100.66.0.0/16 ip daddr != 100.66.0.0/16 accept
                oifname "wt0" ip saddr 100.66.0.0/16 ip daddr 100.66.0.0/16 jump netbird-acl-output-rules
                oifname "wt0" drop
        }

        chain netbird-acl-forward-filter {
                type filter hook forward priority 0; policy accept;
                iifname "wt0" jump netbird-rt-fwd
                oifname "wt0" jump netbird-rt-fwd
                iifname "wt0" mark 0x000007e4 accept
                oifname "wt0" mark 0x000007e4 accept
                iifname "wt0" jump netbird-acl-input-rules
                iifname "wt0" drop
        }

        chain netbird-acl-prerouting-filter {
                type filter hook prerouting priority -150; policy accept;
                iifname "wt0" ip saddr != 100.66.0.0/16 ip daddr 100.66.121.137 mark set 0x000007e4
        }
}

Removing netbird-rt-nat made everything work again.

Your solution seems to work but I had to add =true as the code seems to be depended on this: os.Getenv(SKIP_NFTABLES_ENV) != "true"

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird

But is this a bug that nftables is breaking any further specialised natting? Having the rule with priority 99 which just accepts with no special spesifications is that OK? Maybe we have a real issue at hand here?

@thorleifjacobsen commented on GitHub (May 23, 2024): Hello! Thanks for the pointer, we did not think of checking the `nft` tables. Indeed we found a problem, there is a rule in `nft list tables ip netbird` a chain called `netbird-rt-nat` by the look of it seems to prioritze over anything to accept all traffic. Which in part stops any special iptables natting. ```bash table ip netbird { chain netbird-rt-fwd { } chain netbird-rt-nat { type nat hook postrouting priority 99; policy accept; } chain netbird-acl-input-rules { ip saddr & 0.0.0.0 == 0.0.0.0 accept } chain netbird-acl-output-rules { ip daddr & 0.0.0.0 == 0.0.0.0 accept } chain netbird-acl-input-filter { type filter hook input priority 0; policy accept; iifname "wt0" ip saddr 100.66.0.0/16 ip daddr != 100.66.0.0/16 accept iifname "wt0" ip saddr != 100.66.0.0/16 ip daddr 100.66.0.0/16 accept iifname "wt0" ip saddr 100.66.0.0/16 ip daddr 100.66.0.0/16 jump netbird-acl-input-rules iifname "wt0" drop } chain netbird-acl-output-filter { type filter hook output priority 0; policy accept; oifname "wt0" ip saddr != 100.66.0.0/16 ip daddr 100.66.0.0/16 accept oifname "wt0" ip saddr 100.66.0.0/16 ip daddr != 100.66.0.0/16 accept oifname "wt0" ip saddr 100.66.0.0/16 ip daddr 100.66.0.0/16 jump netbird-acl-output-rules oifname "wt0" drop } chain netbird-acl-forward-filter { type filter hook forward priority 0; policy accept; iifname "wt0" jump netbird-rt-fwd oifname "wt0" jump netbird-rt-fwd iifname "wt0" mark 0x000007e4 accept oifname "wt0" mark 0x000007e4 accept iifname "wt0" jump netbird-acl-input-rules iifname "wt0" drop } chain netbird-acl-prerouting-filter { type filter hook prerouting priority -150; policy accept; iifname "wt0" ip saddr != 100.66.0.0/16 ip daddr 100.66.121.137 mark set 0x000007e4 } } ``` Removing `netbird-rt-nat` made everything work again. Your solution seems to work but I had to add `=true` as the code seems to be depended on this: `os.Getenv(SKIP_NFTABLES_ENV) != "true"` ```bash sudo mkdir -p /etc/sysconfig echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a /etc/sysconfig/netbird sudo systemctl restart netbird ``` But is this a bug that nftables is breaking any further specialised natting? Having the rule with priority 99 which just accepts with no special spesifications is that OK? Maybe we have a real issue at hand here?
saavagebueno commented 2025-11-20 05:19:54 -05:00
Author
Owner
Copy Link

@thorleifjacobsen commented on GitHub (May 23, 2024):

My issue with NAT rules being ignored by IPTables, specifically with SNAT not working, was resolved with the script as shown above. This was the NAT rule which was previousl ignored:

-A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233

This IPTables rule was ignored, possibly due to the presence of this chain in nftables. After removing the chain, SNAT started working again:

# From CentOS NFTables
chain netbird-rt-nat {
    type nat hook postrouting priority 99; policy accept;
}

My next question is whether this is a configuration bug in Netbird. It seems to be incorrectly configured since it overrides other NAT rules.

My understanding is that this setup allows postrouting without passing through more specialized rules. Should the specific IPs involved be defined more clearly? What might be the underlying issue? Can this be fixed in a patch for Netbird, or do we need to enforce IPTables every time?

Additionally, why does this issue not occur on Ubuntu? After checking Ubunti I saw they have a lower priority on their chain in nftables?

Is this a bug which should be fixed for CentOS?

# From Ubuntu NFTables
chain netbird-rt-nat {
        type nat hook postrouting priority srcnat - 1; policy accept;
}
@thorleifjacobsen commented on GitHub (May 23, 2024): My issue with NAT rules being ignored by IPTables, specifically with SNAT not working, was resolved with the script as shown above. This was the NAT rule which was previousl ignored: ```css -A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233 ``` This IPTables rule was ignored, possibly due to the presence of this chain in nftables. After removing the chain, SNAT started working again: ```bash # From CentOS NFTables chain netbird-rt-nat { type nat hook postrouting priority 99; policy accept; } ``` My next question is whether this is a configuration bug in Netbird. It seems to be incorrectly configured since it overrides other NAT rules. My understanding is that this setup allows postrouting without passing through more specialized rules. Should the specific IPs involved be defined more clearly? What might be the underlying issue? Can this be fixed in a patch for Netbird, or do we need to enforce IPTables every time? Additionally, why does this issue not occur on Ubuntu? After checking Ubunti I saw they have a lower priority on their chain in nftables? Is this a bug which should be fixed for CentOS? ```bash # From Ubuntu NFTables chain netbird-rt-nat { type nat hook postrouting priority srcnat - 1; policy accept; } ```
saavagebueno commented 2025-11-20 05:19:54 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 23, 2024):

Thanks for the update, @thorleifjacobsen; yes. It is a bug; with iptables and nftables competing for precedence, this rule should not be in place in case there were no rules to be added to it.

It seems like the best approach might be to detect an old kernel or iptables versions that don't support nftables and set the client to use iptables instead.

@mlsmaycon commented on GitHub (May 23, 2024): Thanks for the update, @thorleifjacobsen; yes. It is a bug; with iptables and nftables competing for precedence, this rule should not be in place in case there were no rules to be added to it. It seems like the best approach might be to detect an old kernel or iptables versions that don't support nftables and set the client to use iptables instead.
saavagebueno commented 2025-11-20 05:19:54 -05:00
Author
Owner
Copy Link

@thorleifjacobsen commented on GitHub (May 23, 2024):

Keep in mind, this rule is available both on Ubuntu and CentOS (but it seems to be srcnat - 1 in priority vs 99 in CentOS).

When changing to iptables it adds some similar rules to iptables:

*nat
:PREROUTING ACCEPT [12:1383]
:INPUT ACCEPT [12:1383]
:OUTPUT ACCEPT [14:1225]
:POSTROUTING ACCEPT [14:1225]
:NETBIRD-RT-NAT - [0:0]
-A PREROUTING -p udp -m mac --mac-source 96:4F:1D:1F:69:CA -m multiport ! --dports 0:1024,1900,5353 -j DNAT --to-destination 10.0.4.236
-A PREROUTING -p tcp -m mac --mac-source 96:4F:1D:1F:69:CA -m multiport ! --dports 0:1024,8008,8443 -j DNAT --to-destination 10.0.4.236
-A POSTROUTING -j NETBIRD-RT-NAT
-A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233
-A NETBIRD-RT-NAT -j RETURN
@thorleifjacobsen commented on GitHub (May 23, 2024): Keep in mind, this rule is available both on Ubuntu and CentOS (but it seems to be `srcnat - 1` in priority vs `99` in CentOS). When changing to `iptables` it adds some similar rules to iptables: ```css *nat :PREROUTING ACCEPT [12:1383] :INPUT ACCEPT [12:1383] :OUTPUT ACCEPT [14:1225] :POSTROUTING ACCEPT [14:1225] :NETBIRD-RT-NAT - [0:0] -A PREROUTING -p udp -m mac --mac-source 96:4F:1D:1F:69:CA -m multiport ! --dports 0:1024,1900,5353 -j DNAT --to-destination 10.0.4.236 -A PREROUTING -p tcp -m mac --mac-source 96:4F:1D:1F:69:CA -m multiport ! --dports 0:1024,8008,8443 -j DNAT --to-destination 10.0.4.236 -A POSTROUTING -j NETBIRD-RT-NAT -A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233 -A NETBIRD-RT-NAT -j RETURN ```
saavagebueno commented 2025-11-20 05:19:54 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 23, 2024):

Thanks, @thorleifjacobsen; for the next release, we will verify if iptables and nftables can work together and fall back to iptables by default.

It is better to have a single firewall manager dealing with packets.

@mlsmaycon commented on GitHub (May 23, 2024): Thanks, @thorleifjacobsen; for the next release, we will verify if iptables and nftables can work together and fall back to iptables by default. It is better to have a single firewall manager dealing with packets.
saavagebueno commented 2025-11-20 05:19:54 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 23, 2024):

the release 0.27.8 contains the fix. Let us know if there is any other issue

@mlsmaycon commented on GitHub (May 23, 2024): the release 0.27.8 contains the fix. Let us know if there is any other issue
saavagebueno commented 2025-11-20 05:19:54 -05:00
Author
Owner
Copy Link

@thorleifjacobsen commented on GitHub (May 24, 2024):

0.27.9 installed and it works as expected. Thanks :)

@thorleifjacobsen commented on GitHub (May 24, 2024): 0.27.9 installed and it works as expected. Thanks :)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#919
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?