SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-06-08 00:11:58 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

BUG Maybe: After start netbird, all Docker containers can't connect network #907

New Issue
Closed
opened 2025-11-20 05:19:42 -05:00 by saavagebueno · 14 comments
saavagebueno commented 2025-11-20 05:19:42 -05:00
Owner
Copy Link

Originally created by @johnson7788 on GitHub (May 20, 2024).

Describe the problem
When I start netbird, No matter from netbird command or netbird docker, netbird works very well, But it leads my other docker container can't connect network. i suspect netbird conflict with docker deamon network.

Test on Linux: Ubuntu16.04 and CentOS7.

To Reproduce

Steps to reproduce the behavior:

  1. install docker services
  2. start netbird(docker or command)
  3. Test other docker, like below
  4. sudo docker run --rm --net=host -it busybox ping 123.150.76.218
  5. sudo docker run --rm busybox ping so.com
  6. both are failed
  7. stop netbird, try again step4 and step5, success network now.

Expected behavior

A clear and concise description of what you expected to happen.

Are you using NetBird Cloud?

Please specify whether you use NetBird Cloud or self-host NetBird's control plane.

NetBird version

netbird version
0.23.3 and 0.27.7

NetBird status -d output:

If applicable, add the `netbird status -d' command output.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Originally created by @johnson7788 on GitHub (May 20, 2024). **Describe the problem** When I start netbird, No matter from netbird command or netbird docker, netbird works very well, But it leads my other docker container can't connect network. i suspect netbird conflict with docker deamon network. Test on Linux: Ubuntu16.04 and CentOS7. **To Reproduce** Steps to reproduce the behavior: 1. install docker services 2. start netbird(docker or command) 3. Test other docker, like below 4. sudo docker run --rm --net=host -it busybox ping 123.150.76.218 5. sudo docker run --rm busybox ping so.com 6. both are failed 7. stop netbird, try again step4 and step5, success network now. **Expected behavior** A clear and concise description of what you expected to happen. **Are you using NetBird Cloud?** Please specify whether you use NetBird Cloud or self-host NetBird's control plane. **NetBird version** `netbird version` 0.23.3 and 0.27.7 **NetBird status -d output:** If applicable, add the `netbird status -d' command output. **Screenshots** If applicable, add screenshots to help explain your problem. **Additional context** Add any other context about the problem here.
saavagebueno added the triage-needed label 2025-11-20 05:19:42 -05:00
saavagebueno commented 2025-11-20 05:19:43 -05:00
Author
Owner
Copy Link

@enyachoke commented on GitHub (May 21, 2024):

@johnson7788 I also experienced this issue on CentOS 7, where the docker containers on the host could not connect to the Internet after installing Netbird.

i suspect netbird conflict with docker deamon network.

@johnson7788 have you found a way to run Netbird on a different IP range

@enyachoke commented on GitHub (May 21, 2024): @johnson7788 I also experienced this issue on CentOS 7, where the docker containers on the host could not connect to the Internet after installing Netbird. >i suspect netbird conflict with docker deamon network. @johnson7788 have you found a way to run Netbird on a different IP range
saavagebueno commented 2025-11-20 05:19:43 -05:00
Author
Owner
Copy Link

@johnson7788 commented on GitHub (May 21, 2024):

@enyachoke My docker network is 172.17.0.0/16, and netbird is 100.124.0.0/16, I think they are already in different IP segment.

@johnson7788 commented on GitHub (May 21, 2024): @enyachoke My docker network is 172.17.0.0/16, and netbird is 100.124.0.0/16, I think they are already in different IP segment.
saavagebueno commented 2025-11-20 05:19:43 -05:00
Author
Owner
Copy Link

@johnson7788 commented on GitHub (May 21, 2024):

I check docker network, iptables, and routes

Start Bird, the route changed
default via 192.168.50.1 dev eno1 proto static metric 100
100.124.0.0/16 dev wt0 proto kernel scope link src 100.124.182.112
169.254.0.0/16 dev eno1 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.50.0/24 dev eno1 proto kernel scope link src 192.168.50.179 metric 100

IPtables:
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere localhost tcp dpt:15672
ACCEPT tcp -- anywhere localhost tcp dpt:amqp

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere

Stop Bird, The route
default via 192.168.50.1 dev eno1 proto static metric 100
169.254.0.0/16 dev eno1 scope link metric 1000
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.50.0/24 dev eno1 proto kernel scope link src 192.168.50.179 metric 100

Iptables
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:15672
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:amqp

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere

@johnson7788 commented on GitHub (May 21, 2024): I check docker network, iptables, and routes Start Bird, the route changed default via 192.168.50.1 dev eno1 proto static metric 100 100.124.0.0/16 dev wt0 proto kernel scope link src 100.124.182.112 169.254.0.0/16 dev eno1 scope link metric 1000 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 192.168.50.0/24 dev eno1 proto kernel scope link src 192.168.50.179 metric 100 IPtables: Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain DOCKER (1 references) target prot opt source destination ACCEPT tcp -- anywhere localhost tcp dpt:15672 ACCEPT tcp -- anywhere localhost tcp dpt:amqp Chain DOCKER-ISOLATION-STAGE-1 (1 references) target prot opt source destination DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-ISOLATION-STAGE-2 (1 references) target prot opt source destination DROP all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- anywhere anywhere Stop Bird, The route default via 192.168.50.1 dev eno1 proto static metric 100 169.254.0.0/16 dev eno1 scope link metric 1000 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 192.168.50.0/24 dev eno1 proto kernel scope link src 192.168.50.179 metric 100 Iptables Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy DROP) target prot opt source destination DOCKER-USER all -- anywhere anywhere DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere ACCEPT all -- anywhere anywhere ACCEPT all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain DOCKER (1 references) target prot opt source destination ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:15672 ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:amqp Chain DOCKER-ISOLATION-STAGE-1 (1 references) target prot opt source destination DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-ISOLATION-STAGE-2 (1 references) target prot opt source destination DROP all -- anywhere anywhere RETURN all -- anywhere anywhere Chain DOCKER-USER (1 references) target prot opt source destination RETURN all -- anywhere anywhere
saavagebueno commented 2025-11-20 05:19:43 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 22, 2024):

@enyachoke @johnson7788, can you confirm whether you have any DNS or routes enabled in NetBird's dashboard?

Also can you try forcing the use of iptables instead of nftables with the following commands:

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird
@mlsmaycon commented on GitHub (May 22, 2024): @enyachoke @johnson7788, can you confirm whether you have any DNS or routes enabled in NetBird's dashboard? Also can you try forcing the use of iptables instead of nftables with the following commands: ```shell sudo mkdir -p /etc/sysconfig echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a /etc/sysconfig/netbird sudo systemctl restart netbird ```
saavagebueno commented 2025-11-20 05:19:43 -05:00
Author
Owner
Copy Link

@thorleifjacobsen commented on GitHub (May 23, 2024):

@enyachoke @johnson7788 the code @mlsmaycon wrote seems to be not working for me, I changed it to this to make it use iptables instead:

sudo mkdir -p /etc/sysconfig
echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a  /etc/sysconfig/netbird
sudo systemctl restart netbird

Ref issue #2030 - seems to be a common issue on C7 or machines using nftables? I do not see this table in ubuntu on nft or iptables.

@thorleifjacobsen commented on GitHub (May 23, 2024): @enyachoke @johnson7788 the code @mlsmaycon wrote seems to be not working for me, I changed it to this to make it use iptables instead: ```bash sudo mkdir -p /etc/sysconfig echo 'NB_SKIP_NFTABLES_CHECK=true' | sudo tee -a /etc/sysconfig/netbird sudo systemctl restart netbird ``` Ref issue #2030 - seems to be a common issue on C7 or machines using nftables? I do not see this table in ubuntu on nft or iptables.
saavagebueno commented 2025-11-20 05:19:43 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 23, 2024):

That's correct, I've updated the command.

The problem still happening with this flag enabled?

@mlsmaycon commented on GitHub (May 23, 2024): That's correct, I've updated the command. The problem still happening with this flag enabled?
saavagebueno commented 2025-11-20 05:19:44 -05:00
Author
Owner
Copy Link

@thorleifjacobsen commented on GitHub (May 23, 2024):

My issue with NAT rules being ignored by IPTables, specifically with SNAT not working, was resolved with the following flag:

-A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233

This IPTables rule was ignored, possibly due to the presence of this chain in nftables. After removing the chain, SNAT started working again:

chain netbird-rt-nat {
    type nat hook postrouting priority 99; policy accept;
}

I can't speak for others, but this fixed the issue for me. My next question is whether this is a configuration bug in Netbird. It seems to be incorrectly configured since it overrides other NAT rules.

My understanding is that this setup allows postrouting without passing through more specialized rules. Should the specific IPs involved be defined more clearly? What might be the underlying issue? Can this be fixed in a patch for Netbird, or do we need to enforce IPTables every time?

Additionally, why does this issue not occur on Ubuntu?

@thorleifjacobsen commented on GitHub (May 23, 2024): My issue with NAT rules being ignored by IPTables, specifically with SNAT not working, was resolved with the following flag: ```css -A POSTROUTING ! -d 224.0.0.251/32 -o enp0s8 -j SNAT --to-source 10.0.4.233 ``` This IPTables rule was ignored, possibly due to the presence of this chain in nftables. After removing the chain, SNAT started working again: ```bash chain netbird-rt-nat { type nat hook postrouting priority 99; policy accept; } ``` I can't speak for others, but this fixed the issue for me. My next question is whether this is a configuration bug in Netbird. It seems to be incorrectly configured since it overrides other NAT rules. My understanding is that this setup allows postrouting without passing through more specialized rules. Should the specific IPs involved be defined more clearly? What might be the underlying issue? Can this be fixed in a patch for Netbird, or do we need to enforce IPTables every time? Additionally, why does this issue not occur on Ubuntu?
saavagebueno commented 2025-11-20 05:19:44 -05:00
Author
Owner
Copy Link

@thorleifjacobsen commented on GitHub (May 23, 2024):

Another addition, in Ubuntu this is the same chain:

chain netbird-rt-nat {
        type nat hook postrouting priority srcnat - 1; policy accept;
}

Maybe CentOS NFTable Chains needs patching?

@thorleifjacobsen commented on GitHub (May 23, 2024): Another addition, in Ubuntu this is the same chain: ```bash chain netbird-rt-nat { type nat hook postrouting priority srcnat - 1; policy accept; } ``` Maybe CentOS NFTable Chains needs patching?
saavagebueno commented 2025-11-20 05:19:44 -05:00
Author
Owner
Copy Link

@johnson7788 commented on GitHub (May 23, 2024):

@mlsmaycon
I tried it ,but it not works, see below:
johnson@wacserver7:$ cat /etc/sysconfig/netbird
NB_SKIP_NFTABLES_CHECK=true
johnson@wacserver7:$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
371bce5deb08 netbirdio/netbird:latest "/usr/local/bin/netb…" 2 minutes ago Up 2 minutes netbirdapp
3fe3273373b6 rabbitmq:management "docker-entrypoint.s…" 3 years ago Up 3 months 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp rabbit

Docker started, but Peers ip is offline on https://app.netbird.io/peers.

cat /etc/release
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

After delete /etc/sysconfig/netbird
sudo rm /etc/sysconfig/netbird
sudo docker run --name netbirdapp --network host --privileged --rm -d -e NB_SETUP_KEY=xxxxxxx -v netbird-client:/etc/netbird netbirdio/netbird

Peers ip is online now, i think we can't turn off NB_SKIP_NFTABLES_CHECK. but when i turn off NB_SKIP_NFTABLES_CHECK, docker network is work, like below
docker run --rm --net=host -it busybox ping 123.150.76.218
PING 123.150.76.218 (123.150.76.218): 56 data bytes
64 bytes from 123.150.76.218: seq=0 ttl=52 time=10.719 ms
64 bytes from 123.150.76.218: seq=1 ttl=52 time=9.302 ms

@johnson7788 commented on GitHub (May 23, 2024): @mlsmaycon I tried it ,but it not works, see below: johnson@wacserver7:~$ cat /etc/sysconfig/netbird NB_SKIP_NFTABLES_CHECK=true johnson@wacserver7:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 371bce5deb08 netbirdio/netbird:latest "/usr/local/bin/netb…" 2 minutes ago Up 2 minutes netbirdapp 3fe3273373b6 rabbitmq:management "docker-entrypoint.s…" 3 years ago Up 3 months 4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 15691-15692/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp rabbit Docker started, but Peers ip is offline on https://app.netbird.io/peers. cat /etc/*release* DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS" After delete /etc/sysconfig/netbird sudo rm /etc/sysconfig/netbird sudo docker run --name netbirdapp --network host --privileged --rm -d -e NB_SETUP_KEY=xxxxxxx -v netbird-client:/etc/netbird netbirdio/netbird Peers ip is online now, i think we can't turn off NB_SKIP_NFTABLES_CHECK. but when i turn off NB_SKIP_NFTABLES_CHECK, docker network is work, like below docker run --rm --net=host -it busybox ping 123.150.76.218 PING 123.150.76.218 (123.150.76.218): 56 data bytes 64 bytes from 123.150.76.218: seq=0 ttl=52 time=10.719 ms 64 bytes from 123.150.76.218: seq=1 ttl=52 time=9.302 ms
saavagebueno commented 2025-11-20 05:19:44 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 23, 2024):

hello @johnson7788 for the container, please add the environment variable as part of its running config. e.g.:

sudo docker run --name netbirdapp --network host --privileged --rm -d -e NB_SETUP_KEY=xxxxxxx -e NB_SKIP_NFTABLES_CHECK=true -v netbird-client:/etc/netbird netbirdio/netbird
@mlsmaycon commented on GitHub (May 23, 2024): hello @johnson7788 for the container, please add the environment variable as part of its running config. e.g.: ```shell sudo docker run --name netbirdapp --network host --privileged --rm -d -e NB_SETUP_KEY=xxxxxxx -e NB_SKIP_NFTABLES_CHECK=true -v netbird-client:/etc/netbird netbirdio/netbird ```
saavagebueno commented 2025-11-20 05:19:44 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (May 23, 2024):

the release 0.27.8 contains the fix. Let us know if there is any other issue

@mlsmaycon commented on GitHub (May 23, 2024): the release 0.27.8 contains the fix. Let us know if there is any other issue
saavagebueno commented 2025-11-20 05:19:44 -05:00
Author
Owner
Copy Link

@johnson7788 commented on GitHub (May 23, 2024):

@mlsmaycon Great jobs!, thank you very much, it works for me, docker network is ok now. and netbird network also works. You are so amazing

sudo docker run --rm busybox ping so.com
PING so.com (180.163.237.15): 56 data bytes
64 bytes from 180.163.237.15: seq=0 ttl=51 time=29.483 ms

@johnson7788 commented on GitHub (May 23, 2024): @mlsmaycon Great jobs!, thank you very much, it works for me, docker network is ok now. and netbird network also works. You are so amazing sudo docker run --rm busybox ping so.com PING so.com (180.163.237.15): 56 data bytes 64 bytes from 180.163.237.15: seq=0 ttl=51 time=29.483 ms
saavagebueno commented 2025-11-20 05:19:45 -05:00
Author
Owner
Copy Link

@johnson7788 commented on GitHub (May 23, 2024):

the release 0.27.8 contains the fix. Let us know if there is any other issue

Yes, I install on an Centos server with latest 0.27.9, Both docker network and netbird network are working. That's great!!!

@johnson7788 commented on GitHub (May 23, 2024): > the release 0.27.8 contains the fix. Let us know if there is any other issue Yes, I install on an Centos server with latest 0.27.9, Both docker network and netbird network are working. That's great!!!
saavagebueno commented 2025-11-20 05:19:45 -05:00
Author
Owner
Copy Link

@johnson7788 commented on GitHub (May 23, 2024):

My issue all be fixed. thank you everyone! @mlsmaycon @thorleifjacobsen @enyachoke

@johnson7788 commented on GitHub (May 23, 2024): My issue all be fixed. thank you everyone! @mlsmaycon @thorleifjacobsen @enyachoke
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#907
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?