DYNR/netbird
2
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-10 03:55:25 -04:00
Code Issues 23.5k Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4470] Connectivity issues with Netbird on hosts running Kubernetes with Calico/Flannel CNI #8723

New Issue
Open
opened 2026-08-05 01:19:17 -04:00 by saavagebueno · 9 comments
saavagebueno commented 2026-08-05 01:19:17 -04:00
Owner
Copy Link

Originally created by @MichalisDBA on GitHub (Sep 9, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/4470

Description:

There is a connectivity issue when defining Access Control Policies on a host running Kubernetes with Calico or Flannel CNI plugins.

Steps to Reproduce:

  1. Install the Netbird client on a host machine running Kubernetes (not in a container).
  2. Ensure a CNI plugin (Calico or Flannel) is installed on the host, either manually or via Rancher.
  3. Set up the Netbird client—it should connect to the management panel successfully.
  4. Configure an Access Control Policy to allow another Netbird peer to reach this host (e.g., on port 22).
  5. Attempt to connect from the other peer.

Observed Behavior:

  • The connection fails despite the Access Control Policy being configured.
  • After inspecting iptables -t nat -L -n -v --line-numbers, the issue appears to originate from the CNI-HOSTPORT-MASQ chain.
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       12   756 cali-PREROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:6gwbT8clXdHdC1b1 */
2       12   756 NETBIRD-RT-RDR  all  --  *      *       0.0.0.0/0            0.0.0.0/0
3       71  4276 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
4        3   336 DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL
5        2   200 CNI-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       96  6352 cali-OUTPUT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:tVnHkvAo15HuiPy0 */
2      194 12313 KUBE-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
3       83  5004 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL
4       51  3418 CNI-HOSTPORT-DNAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       29  2100 NETBIRD-RT-NAT  all  --  *      *       0.0.0.0/0            0.0.0.0/0
2      137  8933 CNI-HOSTPORT-MASQ  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI portfwd requiring masquerade */
3      156  9841 KUBE-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
4        0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0
5       65  4225 FLANNEL-POSTRTG  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* flanneld masq */
6       58  3840 cali-POSTROUTING  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:0i8pjzKKPyA34aQD */

Chain CNI-DN-4a7ac74c95e4a0f411b94 (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       10.42.0.240          0.0.0.0/0            tcp dpt:80
2        0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:80
3        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:10.42.0.240:80
4        0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       10.42.0.240          0.0.0.0/0            tcp dpt:443
5        0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:443
6        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:10.42.0.240:443

Chain CNI-HOSTPORT-DNAT (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 CNI-DN-4a7ac74c95e4a0f411b94  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* dnat name: "k8s-pod-network" id: "431e755a5dce0f52b17d42db6307c4eec7a4b91b3ba74bc5ef0efebe7e2c53c5" */ multiport dports 80,443

Chain CNI-HOSTPORT-MASQ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        8   712 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000/0x2000

Chain CNI-HOSTPORT-SETMARK (4 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI portfwd masquerade mark */ MARK or 0x2000

Chain DOCKER (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0

Chain FLANNEL-POSTRTG (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* flanneld masq */
2        0     0 RETURN     all  --  *      *       10.42.0.0/24         10.42.0.0/16         /* flanneld masq */
3        0     0 RETURN     all  --  *      *       10.42.0.0/16         10.42.0.0/24         /* flanneld masq */
4       11   660 RETURN     all  --  *      *      !10.42.0.0/16         10.42.0.0/24         /* flanneld masq */
5        5   265 MASQUERADE  all  --  *      *       10.42.0.0/16        !224.0.0.0/4          /* flanneld masq */ random-fully
6        0     0 MASQUERADE  all  --  *      *      !10.42.0.0/16         10.42.0.0/16         /* flanneld masq */ random-fully

Chain KUBE-KUBELET-CANARY (0 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain KUBE-MARK-MASQ (12 references)
num   pkts bytes target     prot opt in     out     source               destination
1        6   360 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain KUBE-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        7   420 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match ! 0x4000/0x4000
2        6   360 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            MARK xor 0x4000
3        6   360 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ random-fully

Chain KUBE-PROXY-CANARY (0 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain KUBE-SEP-5OZY2IMZT2HZEWWY (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       10.42.0.240          0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */ tcp to:10.42.0.240:8443

Chain KUBE-SEP-BFSMZD7F6NYVNDLF (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       21  1260 KUBE-MARK-MASQ  all  --  *      *       192.168.74.123       0.0.0.0/0            /* default/kubernetes:https */
2       28  1680 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ tcp to:192.168.74.123:6443

Chain KUBE-SEP-O3LA2IPHNOPCHJL5 (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       10.42.0.242          0.0.0.0/0            /* kube-system/kube-dns:dns */
2        0     0 DNAT       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:10.42.0.242:53

Chain KUBE-SEP-RA3J745CCIDZ75M4 (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       10.42.0.242          0.0.0.0/0            /* kube-system/kube-dns:metrics */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics */ tcp to:10.42.0.242:9153

Chain KUBE-SEP-RE5HHH2FAJWXIYPH (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       10.42.0.242          0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
2        0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:10.42.0.242:53

Chain KUBE-SEP-VCE5AARVLJTAPBOL (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  all  --  *      *       10.42.0.239          0.0.0.0/0            /* kube-system/metrics-server:https */
2        6   360 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/metrics-server:https */ tcp to:10.42.0.239:10250

Chain KUBE-SERVICES (2 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-SVC-EZYNCFY2F7N6OQA2  tcp  --  *      *       0.0.0.0/0            10.43.132.78         /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
2        0     0 KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  *      *       0.0.0.0/0            10.43.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
3        0     0 KUBE-SVC-JD5MR3NA4I4DYORP  tcp  --  *      *       0.0.0.0/0            10.43.0.10           /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
4        0     0 KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  *      *       0.0.0.0/0            10.43.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
5        6   360 KUBE-SVC-Z4ANX4WAEWEBLCTM  tcp  --  *      *       0.0.0.0/0            10.43.199.36         /* kube-system/metrics-server:https cluster IP */ tcp dpt:443
6        0     0 KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  *      *       0.0.0.0/0            10.43.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
7        3   180 KUBE-NODEPORTS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !10.42.0.0/16         10.43.0.10           /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
2        0     0 KUBE-SEP-RE5HHH2FAJWXIYPH  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp -> 10.42.0.242:53 */

Chain KUBE-SVC-EZYNCFY2F7N6OQA2 (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !10.42.0.0/16         10.43.132.78         /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443
2        0     0 KUBE-SEP-5OZY2IMZT2HZEWWY  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* ingress-nginx/ingress-nginx-controller-admission:https-webhook -> 10.42.0.240:8443 */

Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  tcp  --  *      *      !10.42.0.0/16         10.43.0.10           /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
2        0     0 KUBE-SEP-RA3J745CCIDZ75M4  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:metrics -> 10.42.0.242:9153 */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       21  1260 KUBE-MARK-MASQ  tcp  --  *      *      !10.42.0.0/16         10.43.0.1            /* default/kubernetes:https cluster IP */ tcp dpt:443
2       28  1680 KUBE-SEP-BFSMZD7F6NYVNDLF  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https -> 192.168.74.123:6443 */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 KUBE-MARK-MASQ  udp  --  *      *      !10.42.0.0/16         10.43.0.10           /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
2        0     0 KUBE-SEP-O3LA2IPHNOPCHJL5  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns -> 10.42.0.242:53 */

Chain KUBE-SVC-Z4ANX4WAEWEBLCTM (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        6   360 KUBE-MARK-MASQ  tcp  --  *      *      !10.42.0.0/16         10.43.199.36         /* kube-system/metrics-server:https cluster IP */ tcp dpt:443
2        6   360 KUBE-SEP-VCE5AARVLJTAPBOL  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kube-system/metrics-server:https -> 10.42.0.239:10250 */

Chain NETBIRD-RT-NAT (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MASQUERADE  all  --  *      !lo     0.0.0.0/0            0.0.0.0/0            mark match 0x1bd21
2        0     0 MASQUERADE  all  --  *      wt0     0.0.0.0/0            0.0.0.0/0            mark match 0x1bd22

Chain NETBIRD-RT-RDR (1 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain cali-OUTPUT (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       96  6352 cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:GBTAv2p5CwevEyJm */

Chain cali-POSTROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       58  3840 cali-fip-snat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:Z-c7XtVd2Bq7s_hA */
2       58  3840 cali-nat-outgoing  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:nYKhEzDlr11Jccal */

Chain cali-PREROUTING (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1       22  1316 cali-fip-dnat  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:r6XmIziWUJsdOK6Z */

Chain cali-fip-dnat (2 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain cali-fip-snat (1 references)
num   pkts bytes target     prot opt in     out     source               destination

Chain cali-nat-outgoing (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* cali:flqWnvo8yq4ULQLa */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst random-fully

Details:

The CNI plugin marks all packets from Netbird originating from 127.0.0.1 and the loopback interface with 0x2000/0x2000. These packets then hit the POSTROUTING rule in the CNI-HOSTPORT-MASQ chain, which MASQUERADEs them. This breaks connectivity from other Netbird peers.

Example:

Chain CNI-HOSTPORT-MASQ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000/0x2000

For debugging, adding a rule to bypass MASQUERADE for local Netbird packets restores connectivity:

iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -m mark --mark 0x2000/0x2000 -j RETURN

Resulting chain:

Chain CNI-HOSTPORT-MASQ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 RETURN     all  --  *      *       127.0.0.1            0.0.0.0/0            mark match 0x2000/0x2000
2        0     0 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000/0x2000

Conclusion:

  • All Netbird packets are being marked by the CNI plugin and subsequently MASQUERADEd.
  • This prevents other Netbird peers from connecting, even though Access Control Policies are correctly set.
  • Currently, there is no proposed solution. Netbird uses marks 0x1BD21 and 0x1BD22, but these appear to be overwritten by the CNI plugin.

Impact:

Hosts running Kubernetes with Calico or Flannel CNI cannot be accessed from other Netbird peers when Access Control Policies are configured.

Originally created by @MichalisDBA on GitHub (Sep 9, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/4470 Description: There is a connectivity issue when defining Access Control Policies on a host running Kubernetes with Calico or Flannel CNI plugins. Steps to Reproduce: 1. Install the Netbird client on a host machine running Kubernetes (not in a container). 2. Ensure a CNI plugin (Calico or Flannel) is installed on the host, either manually or via Rancher. 3. Set up the Netbird client—it should connect to the management panel successfully. 4. Configure an Access Control Policy to allow another Netbird peer to reach this host (e.g., on port 22). 5. Attempt to connect from the other peer. Observed Behavior: - The connection fails despite the Access Control Policy being configured. - After inspecting `iptables -t nat -L -n -v --line-numbers`, the issue appears to originate from the `CNI-HOSTPORT-MASQ` chain. ``` Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 12 756 cali-PREROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6gwbT8clXdHdC1b1 */ 2 12 756 NETBIRD-RT-RDR all -- * * 0.0.0.0/0 0.0.0.0/0 3 71 4276 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */ 4 3 336 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL 5 2 200 CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL Chain INPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 96 6352 cali-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 */ 2 194 12313 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */ 3 83 5004 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL 4 51 3418 CNI-HOSTPORT-DNAT all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 29 2100 NETBIRD-RT-NAT all -- * * 0.0.0.0/0 0.0.0.0/0 2 137 8933 CNI-HOSTPORT-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI portfwd requiring masquerade */ 3 156 9841 KUBE-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes postrouting rules */ 4 0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0 5 65 4225 FLANNEL-POSTRTG all -- * * 0.0.0.0/0 0.0.0.0/0 /* flanneld masq */ 6 58 3840 cali-POSTROUTING all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:0i8pjzKKPyA34aQD */ Chain CNI-DN-4a7ac74c95e4a0f411b94 (1 references) num pkts bytes target prot opt in out source destination 1 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.42.0.240 0.0.0.0/0 tcp dpt:80 2 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:80 3 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:10.42.0.240:80 4 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.42.0.240 0.0.0.0/0 tcp dpt:443 5 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:443 6 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:10.42.0.240:443 Chain CNI-HOSTPORT-DNAT (2 references) num pkts bytes target prot opt in out source destination 1 0 0 CNI-DN-4a7ac74c95e4a0f411b94 tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* dnat name: "k8s-pod-network" id: "431e755a5dce0f52b17d42db6307c4eec7a4b91b3ba74bc5ef0efebe7e2c53c5" */ multiport dports 80,443 Chain CNI-HOSTPORT-MASQ (1 references) num pkts bytes target prot opt in out source destination 1 8 712 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x2000/0x2000 Chain CNI-HOSTPORT-SETMARK (4 references) num pkts bytes target prot opt in out source destination 1 0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI portfwd masquerade mark */ MARK or 0x2000 Chain DOCKER (2 references) num pkts bytes target prot opt in out source destination 1 0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0 Chain FLANNEL-POSTRTG (1 references) num pkts bytes target prot opt in out source destination 1 0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* flanneld masq */ 2 0 0 RETURN all -- * * 10.42.0.0/24 10.42.0.0/16 /* flanneld masq */ 3 0 0 RETURN all -- * * 10.42.0.0/16 10.42.0.0/24 /* flanneld masq */ 4 11 660 RETURN all -- * * !10.42.0.0/16 10.42.0.0/24 /* flanneld masq */ 5 5 265 MASQUERADE all -- * * 10.42.0.0/16 !224.0.0.0/4 /* flanneld masq */ random-fully 6 0 0 MASQUERADE all -- * * !10.42.0.0/16 10.42.0.0/16 /* flanneld masq */ random-fully Chain KUBE-KUBELET-CANARY (0 references) num pkts bytes target prot opt in out source destination Chain KUBE-MARK-MASQ (12 references) num pkts bytes target prot opt in out source destination 1 6 360 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK or 0x4000 Chain KUBE-NODEPORTS (1 references) num pkts bytes target prot opt in out source destination Chain KUBE-POSTROUTING (1 references) num pkts bytes target prot opt in out source destination 1 7 420 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 mark match ! 0x4000/0x4000 2 6 360 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 MARK xor 0x4000 3 6 360 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ random-fully Chain KUBE-PROXY-CANARY (0 references) num pkts bytes target prot opt in out source destination Chain KUBE-SEP-5OZY2IMZT2HZEWWY (1 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-MARK-MASQ all -- * * 10.42.0.240 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */ 2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook */ tcp to:10.42.0.240:8443 Chain KUBE-SEP-BFSMZD7F6NYVNDLF (1 references) num pkts bytes target prot opt in out source destination 1 21 1260 KUBE-MARK-MASQ all -- * * 192.168.74.123 0.0.0.0/0 /* default/kubernetes:https */ 2 28 1680 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https */ tcp to:192.168.74.123:6443 Chain KUBE-SEP-O3LA2IPHNOPCHJL5 (1 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-MARK-MASQ all -- * * 10.42.0.242 0.0.0.0/0 /* kube-system/kube-dns:dns */ 2 0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.42.0.242:53 Chain KUBE-SEP-RA3J745CCIDZ75M4 (1 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-MARK-MASQ all -- * * 10.42.0.242 0.0.0.0/0 /* kube-system/kube-dns:metrics */ 2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics */ tcp to:10.42.0.242:9153 Chain KUBE-SEP-RE5HHH2FAJWXIYPH (1 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-MARK-MASQ all -- * * 10.42.0.242 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ 2 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:10.42.0.242:53 Chain KUBE-SEP-VCE5AARVLJTAPBOL (1 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-MARK-MASQ all -- * * 10.42.0.239 0.0.0.0/0 /* kube-system/metrics-server:https */ 2 6 360 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/metrics-server:https */ tcp to:10.42.0.239:10250 Chain KUBE-SERVICES (2 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-SVC-EZYNCFY2F7N6OQA2 tcp -- * * 0.0.0.0/0 10.43.132.78 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443 2 0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.43.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53 3 0 0 KUBE-SVC-JD5MR3NA4I4DYORP tcp -- * * 0.0.0.0/0 10.43.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153 4 0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.43.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53 5 6 360 KUBE-SVC-Z4ANX4WAEWEBLCTM tcp -- * * 0.0.0.0/0 10.43.199.36 /* kube-system/metrics-server:https cluster IP */ tcp dpt:443 6 0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.43.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443 7 3 180 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-MARK-MASQ tcp -- * * !10.42.0.0/16 10.43.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53 2 0 0 KUBE-SEP-RE5HHH2FAJWXIYPH all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp -> 10.42.0.242:53 */ Chain KUBE-SVC-EZYNCFY2F7N6OQA2 (1 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-MARK-MASQ tcp -- * * !10.42.0.0/16 10.43.132.78 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook cluster IP */ tcp dpt:443 2 0 0 KUBE-SEP-5OZY2IMZT2HZEWWY all -- * * 0.0.0.0/0 0.0.0.0/0 /* ingress-nginx/ingress-nginx-controller-admission:https-webhook -> 10.42.0.240:8443 */ Chain KUBE-SVC-JD5MR3NA4I4DYORP (1 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-MARK-MASQ tcp -- * * !10.42.0.0/16 10.43.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153 2 0 0 KUBE-SEP-RA3J745CCIDZ75M4 all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:metrics -> 10.42.0.242:9153 */ Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references) num pkts bytes target prot opt in out source destination 1 21 1260 KUBE-MARK-MASQ tcp -- * * !10.42.0.0/16 10.43.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443 2 28 1680 KUBE-SEP-BFSMZD7F6NYVNDLF all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes:https -> 192.168.74.123:6443 */ Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references) num pkts bytes target prot opt in out source destination 1 0 0 KUBE-MARK-MASQ udp -- * * !10.42.0.0/16 10.43.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53 2 0 0 KUBE-SEP-O3LA2IPHNOPCHJL5 all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns -> 10.42.0.242:53 */ Chain KUBE-SVC-Z4ANX4WAEWEBLCTM (1 references) num pkts bytes target prot opt in out source destination 1 6 360 KUBE-MARK-MASQ tcp -- * * !10.42.0.0/16 10.43.199.36 /* kube-system/metrics-server:https cluster IP */ tcp dpt:443 2 6 360 KUBE-SEP-VCE5AARVLJTAPBOL all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/metrics-server:https -> 10.42.0.239:10250 */ Chain NETBIRD-RT-NAT (1 references) num pkts bytes target prot opt in out source destination 1 0 0 MASQUERADE all -- * !lo 0.0.0.0/0 0.0.0.0/0 mark match 0x1bd21 2 0 0 MASQUERADE all -- * wt0 0.0.0.0/0 0.0.0.0/0 mark match 0x1bd22 Chain NETBIRD-RT-RDR (1 references) num pkts bytes target prot opt in out source destination Chain cali-OUTPUT (1 references) num pkts bytes target prot opt in out source destination 1 96 6352 cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:GBTAv2p5CwevEyJm */ Chain cali-POSTROUTING (1 references) num pkts bytes target prot opt in out source destination 1 58 3840 cali-fip-snat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Z-c7XtVd2Bq7s_hA */ 2 58 3840 cali-nat-outgoing all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:nYKhEzDlr11Jccal */ Chain cali-PREROUTING (1 references) num pkts bytes target prot opt in out source destination 1 22 1316 cali-fip-dnat all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:r6XmIziWUJsdOK6Z */ Chain cali-fip-dnat (2 references) num pkts bytes target prot opt in out source destination Chain cali-fip-snat (1 references) num pkts bytes target prot opt in out source destination Chain cali-nat-outgoing (1 references) num pkts bytes target prot opt in out source destination 1 0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:flqWnvo8yq4ULQLa */ match-set cali40masq-ipam-pools src ! match-set cali40all-ipam-pools dst random-fully ``` Details: The CNI plugin marks all packets from Netbird originating from `127.0.0.1` and the loopback interface with `0x2000/0x2000`. These packets then hit the `POSTROUTING` rule in the `CNI-HOSTPORT-MASQ` chain, which MASQUERADEs them. This breaks connectivity from other Netbird peers. Example: ``` Chain CNI-HOSTPORT-MASQ (1 references) num pkts bytes target prot opt in out source destination 1 0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x2000/0x2000 ``` For debugging, adding a rule to bypass MASQUERADE for local Netbird packets restores connectivity: `iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -m mark --mark 0x2000/0x2000 -j RETURN` Resulting chain: ``` Chain CNI-HOSTPORT-MASQ (1 references) num pkts bytes target prot opt in out source destination 1 0 0 RETURN all -- * * 127.0.0.1 0.0.0.0/0 mark match 0x2000/0x2000 2 0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x2000/0x2000 ``` Conclusion: - All Netbird packets are being marked by the CNI plugin and subsequently MASQUERADEd. - This prevents other Netbird peers from connecting, even though Access Control Policies are correctly set. - Currently, there is no proposed solution. Netbird uses marks `0x1BD21` and `0x1BD22`, but these appear to be overwritten by the CNI plugin. Impact: Hosts running Kubernetes with Calico or Flannel CNI cannot be accessed from other Netbird peers when Access Control Policies are configured.
saavagebueno added the triage-needed label 2026-08-05 01:19:17 -04:00
saavagebueno commented 2026-08-05 01:19:18 -04:00
Author
Owner
Copy Link

@MichalisDBA commented on GitHub (Sep 10, 2025):

The strange part is that after applying iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -m mark --mark 0x2000/0x2000 -j RETURN and then test to see that connections from other netbird peers are established, you can remove this iptables rule and the connections still work until you restart the netbird service.

Edit: The connections are maintained because conntrack keeps a table of the active network connections from netbird peers that were used when iptables bypass rule was applied. If you clear the conntrack table with conntrack -D and try to initiate the connection from another netbird peer without the iptables rule, the connection won't be established again.

<!-- gh-comment-id:3273694914 --> @MichalisDBA commented on GitHub (Sep 10, 2025): The strange part is that after applying `iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -m mark --mark 0x2000/0x2000 -j RETURN` and then test to see that connections from other netbird peers are established, you can remove this `iptables` rule and the connections still work until you restart the netbird service. Edit: The connections are maintained because conntrack keeps a table of the active network connections from netbird peers that were used when `iptables` bypass rule was applied. If you clear the conntrack table with `conntrack -D` and try to initiate the connection from another netbird peer without the `iptables` rule, the connection won't be established again.
saavagebueno commented 2026-08-05 01:19:18 -04:00
Author
Owner
Copy Link

@MichalisDBA commented on GitHub (Sep 10, 2025):

I think i found the solution by simply adding one rule without affecting the CNI plugin.

Just add an iptables rule to return when the traffic goes to localhost and the wireguard destination port configured (default: 51820)

iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -p udp --dport 51820 -j RETURN

With this rule everything works.

<!-- gh-comment-id:3275322748 --> @MichalisDBA commented on GitHub (Sep 10, 2025): I think i found the solution by simply adding one rule without affecting the CNI plugin. Just add an iptables rule to return when the traffic goes to localhost and the wireguard destination port configured (default: 51820) `iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -p udp --dport 51820 -j RETURN` With this rule everything works.
saavagebueno commented 2026-08-05 01:19:18 -04:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Sep 10, 2025):

this is a curious case, I'll let the team know to analyze this further and maybe give some recommendations. Any idea whether that CNI-HOSTPORT-MASQ is managed by Calico, by Flanner or by k3s/rancher?

<!-- gh-comment-id:3275367461 --> @nazarewk commented on GitHub (Sep 10, 2025): this is a curious case, I'll let the team know to analyze this further and maybe give some recommendations. Any idea whether that `CNI-HOSTPORT-MASQ` is managed by Calico, by Flanner or by k3s/rancher?
saavagebueno commented 2026-08-05 01:19:18 -04:00
Author
Owner
Copy Link

@MichalisDBA commented on GitHub (Sep 11, 2025):

I think this rule CNI-HOSTPORT-MASQ is managed by Rancher because usualy Calico has it's named rules prefixed with cali-*

We have installed Netbird client on other Rancher nodes and they are working fine from the start. I do not know if this was an isolated issue in our environment but by adding this rule i mentioned everything worked again.

<!-- gh-comment-id:3278868357 --> @MichalisDBA commented on GitHub (Sep 11, 2025): I think this rule `CNI-HOSTPORT-MASQ` is managed by Rancher because usualy Calico has it's named rules prefixed with `cali-*` We have installed Netbird client on other Rancher nodes and they are working fine from the start. I do not know if this was an isolated issue in our environment but by adding this rule i mentioned everything worked again.
saavagebueno commented 2026-08-05 01:19:18 -04:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Sep 16, 2025):

@MichalisDBA it's not clear how this rule you added prevents the issue, in your output it doesn't have any hits and neither does the broader rule.

Can you provide the whole ruleset, ideally via the debug bundle (see new issue template)?

That chain seems to originate from https://www.cni.dev/plugins/current/meta/portmap/. Have you opened a ticket with them or rancher?

<!-- gh-comment-id:3299000273 --> @lixmal commented on GitHub (Sep 16, 2025): @MichalisDBA it's not clear how this rule you added prevents the issue, in your output it doesn't have any hits and neither does the broader rule. Can you provide the whole ruleset, ideally via the debug bundle (see [new issue template](https://github.com/netbirdio/netbird/issues/new?template=bug-issue-report.md))? That chain seems to originate from https://www.cni.dev/plugins/current/meta/portmap/. Have you opened a ticket with them or rancher?
saavagebueno commented 2026-08-05 01:19:18 -04:00
Author
Owner
Copy Link

@MichalisDBA commented on GitHub (Sep 17, 2025):

@lixmal I have not open a ticket with them.

The rule iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -p udp --dport 51820 -j RETURN prevents this because my guess is that packets are marked from CNI with '0x2000/0x2000' and are masqueraded because they go through the chain CNI-HOSTPORT-MASQ. That way NetBird can not see the correct ip address of the packets.

I do not know if anything else is happening but i tried with two systems that we had exactly the same problem and when i added the iptables rule iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -p udp --dport 51820 -j RETURN everything started to work again.

The udp listening socket at port 51820 is that NetBird initiates with wireguard i believe.

ss -lnu | grep ':51820'
UNCONN 0      0                                          0.0.0.0:51820      0.0.0.0:*
UNCONN 0      0                                             [::]:51820         [::]:*

I attach the bundles before and after the iptables fix

netbird.debug.3808407659-before-iptables-fix.zip
netbird.debug.1713813606-after-iptables-fix.zip

<!-- gh-comment-id:3302667093 --> @MichalisDBA commented on GitHub (Sep 17, 2025): @lixmal I have not open a ticket with them. The rule `iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -p udp --dport 51820 -j RETURN` prevents this because my guess is that packets are marked from CNI with '0x2000/0x2000' and are masqueraded because they go through the chain `CNI-HOSTPORT-MASQ`. That way NetBird can not see the correct ip address of the packets. I do not know if anything else is happening but i tried with two systems that we had exactly the same problem and when i added the iptables rule `iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -p udp --dport 51820 -j RETURN` everything started to work again. The udp listening socket at port 51820 is that NetBird initiates with wireguard i believe. ``` ss -lnu | grep ':51820' UNCONN 0 0 0.0.0.0:51820 0.0.0.0:* UNCONN 0 0 [::]:51820 [::]:* ``` I attach the bundles before and after the iptables fix [netbird.debug.3808407659-before-iptables-fix.zip](https://github.com/user-attachments/files/22384523/netbird.debug.3808407659-before-iptables-fix.zip) [netbird.debug.1713813606-after-iptables-fix.zip](https://github.com/user-attachments/files/22384524/netbird.debug.1713813606-after-iptables-fix.zip)
saavagebueno commented 2026-08-05 01:19:18 -04:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Sep 17, 2025):

Could it be that you have a faulty kernel/netfilter version that ignores fwmarks altogether?

The only chain that applies the mark 0x2000 is:

Chain CNI-HOSTPORT-SETMARK (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MARK       all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* CNI portfwd masquerade mark */ MARK or 0x2000

but it has 0 hits.

Besides, no traffic would match port 51820, these are the only rules referencing that chain:

    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       10.42.0.26           0.0.0.0/0            tcp dpt:80
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:80
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       10.42.0.26           0.0.0.0/0            tcp dpt:443
    0     0 CNI-HOSTPORT-SETMARK  tcp  --  *      *       127.0.0.1            0.0.0.0/0            tcp dpt:443

However the masq chain does have hits and it seems to match regardless of fwmark:

Chain CNI-HOSTPORT-MASQ (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   352 RETURN     udp  --  *      *       127.0.0.1            0.0.0.0/0            udp dpt:51820
   45  5244 MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            mark match 0x2000/0x2000

Have you tried a different kernel?

<!-- gh-comment-id:3303109342 --> @lixmal commented on GitHub (Sep 17, 2025): Could it be that you have a faulty kernel/netfilter version that ignores fwmarks altogether? The only chain that applies the mark 0x2000 is: ``` Chain CNI-HOSTPORT-SETMARK (4 references) pkts bytes target prot opt in out source destination 0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* CNI portfwd masquerade mark */ MARK or 0x2000 ``` but it has 0 hits. Besides, no traffic would match port 51820, these are the only rules referencing that chain: ``` 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.42.0.26 0.0.0.0/0 tcp dpt:80 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:80 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 10.42.0.26 0.0.0.0/0 tcp dpt:443 0 0 CNI-HOSTPORT-SETMARK tcp -- * * 127.0.0.1 0.0.0.0/0 tcp dpt:443 ``` However the masq chain does have hits and it seems to match regardless of fwmark: ``` Chain CNI-HOSTPORT-MASQ (1 references) pkts bytes target prot opt in out source destination 2 352 RETURN udp -- * * 127.0.0.1 0.0.0.0/0 udp dpt:51820 45 5244 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x2000/0x2000 ``` Have you tried a different kernel?
saavagebueno commented 2026-08-05 01:19:18 -04:00
Author
Owner
Copy Link

@MichalisDBA commented on GitHub (Sep 18, 2025):

@lixmal I tried with two hosts running Rancher with the same CNI config, two different versions of OS and Kernel. Same thing is happening unless i add the rule i mentioned. I can see that the MARK rule CNI-HOSTPORT-SETMARK does not set any fwmark but i can not understand then how in the chain CNI-HOSTPORT-MASQ flows packets. The thing for sure is that by adding iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -p udp --dport 51820 -j RETURN everything starts to work again in regard to NetBird and wireguard.

If you have any other idea please let me know to test it. Seems to be a very unusual and strange problem.

uname -a
Linux testing-rancher 5.15.0-153-generic #163-Ubuntu SMP Thu Aug 7 16:37:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

uname -a
Linux rancher 6.8.0-64-generic #67-Ubuntu SMP PREEMPT_DYNAMIC Sun Jun 15 20:23:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

cat /etc/os-release
PRETTY_NAME="Ubuntu 24.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.2 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Also after this "fix" rule is applied we can see some initial packets flowing to the RETURN rule in CNI-HOSTPORT-MASQ chain and then stop. I believe there are some specific packets so that conntrack table is populated and then kernel knows how to route the traffic between peers.

Chain CNI-HOSTPORT-MASQ (1 references)
num   pkts bytes target     prot opt in     out     source               destination
1        1   124 RETURN     udp  --  *      *       127.0.0.1            0.0.0.0/0            udp dpt:51820
<!-- gh-comment-id:3305592995 --> @MichalisDBA commented on GitHub (Sep 18, 2025): @lixmal I tried with two hosts running Rancher with the same CNI config, two different versions of OS and Kernel. Same thing is happening unless i add the rule i mentioned. I can see that the MARK rule `CNI-HOSTPORT-SETMARK` does not set any fwmark but i can not understand then how in the chain `CNI-HOSTPORT-MASQ` flows packets. The thing for sure is that by adding `iptables -t nat -I CNI-HOSTPORT-MASQ 1 -s 127.0.0.1/32 -p udp --dport 51820 -j RETURN` everything starts to work again in regard to NetBird and wireguard. If you have any other idea please let me know to test it. Seems to be a very unusual and strange problem. ``` uname -a Linux testing-rancher 5.15.0-153-generic #163-Ubuntu SMP Thu Aug 7 16:37:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux ``` ``` cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.5 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.5 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ``` ``` uname -a Linux rancher 6.8.0-64-generic #67-Ubuntu SMP PREEMPT_DYNAMIC Sun Jun 15 20:23:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux ``` ``` cat /etc/os-release PRETTY_NAME="Ubuntu 24.04.2 LTS" NAME="Ubuntu" VERSION_ID="24.04" VERSION="24.04.2 LTS (Noble Numbat)" VERSION_CODENAME=noble ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=noble LOGO=ubuntu-logo ``` Also after this "fix" rule is applied we can see some initial packets flowing to the RETURN rule in `CNI-HOSTPORT-MASQ` chain and then stop. I believe there are some specific packets so that conntrack table is populated and then kernel knows how to route the traffic between peers. ``` Chain CNI-HOSTPORT-MASQ (1 references) num pkts bytes target prot opt in out source destination 1 1 124 RETURN udp -- * * 127.0.0.1 0.0.0.0/0 udp dpt:51820 ```
saavagebueno commented 2026-08-05 01:19:18 -04:00
Author
Owner
Copy Link

@Nikita-Ma commented on GitHub (Dec 16, 2025):

Hi @MichalisDBA ,@nazarewk , @lixmal

I tried to use RKE2 with Calico and Flannel CNI, but it didn't work. Router peers (pods) in the cluster were not updating, and the information about Networks was not updated. However, I tried using Cilium, and it resolved the issue.
I hope this information is helpful.

Use this case
values

kubernetesAPI:
  enabled: true
router:
  enabled: true
#managementURL: "https://netbird.domain.com:443"
ingress:
  enabled: true
netbirdAPI:
  keyFromSecret:
    name: "netbird-mgmt-api-key"
    key: "NB_API_KEY"
default:
  name: Kubernetes Default Policy
  sourceGroups:
    - All
<!-- gh-comment-id:3662342511 --> @Nikita-Ma commented on GitHub (Dec 16, 2025): Hi @MichalisDBA ,@nazarewk , @lixmal I tried to use RKE2 with Calico and Flannel CNI, but it didn't work. Router peers (pods) in the cluster were not updating, and the information about Networks was not updated. However, I tried using Cilium, and it resolved the issue. I hope this information is helpful. [Use this case](https://docs.netbird.io/manage/integrations/kubernetes) values ``` kubernetesAPI: enabled: true router: enabled: true #managementURL: "https://netbird.domain.com:443" ingress: enabled: true netbirdAPI: keyFromSecret: name: "netbird-mgmt-api-key" key: "NB_API_KEY" default: name: Kubernetes Default Policy sourceGroups: - All ```
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2021 Q4
2021 Q4
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
accessibility
accessibility
accessibility
acl
acl
acl
agent
agent
agent
agent
agent
agent
Android
Android
Android
Android
Android
Android
api
api
api
authentik
authentik
authentik
automation
automation
automation
azure
azure
azure
battery-usage
battery-usage
battery-usage
bug
cache
cache
cache
client
client
client
client-ui
client-ui
client-ui
cloud
cloud
cloud
cloud-only
cloud-only
cloud-only
cloudflare
cloudflare
cloudflare
community
community
community
compatibility
compatibility
compatibility
config-idp
config-idp
config-idp
config-issue
config-issue
config-issue
connection
connection
connection
contribution
contribution
contribution
coturn
coturn
coturn
cross-vpn
cross-vpn
cross-vpn
dashboard
dashboard
dashboard
data-usage
data-usage
data-usage
distribution
distribution
distribution
dns
dns
dns
docker
docker
docker
documentation
documentation
documentation
duplicate
duplicate
duplicate
enhancement
enhancement
event-stream
event-stream
event-stream
feature-request
feature-request
feature-request
freebsd
freebsd
freebsd
getting-started
getting-started
getting-started
go
go
go
good first issue
good first issue
good first issue
gui
gui
gui
help wanted
help wanted
help wanted
home-assistant
home-assistant
home-assistant
idp
idp
idp
inconsistency
inconsistency
inconsistency
integration
integration
integration
integrations
integrations
integrations
ios
ios
ios
ipv6
ipv6
ipv6
jwt
jwt
jwt
k8s
k8s
k8s
keycloak
keycloak
keycloak
linux
linux
linux
login
login
login
macos
macos
macos
management-service
management-service
management-service
Medium
Medium
Medium
missing-docs
missing-docs
missing-docs
mobile
mobile
mobile
moved-internal
moved-internal
moved-internal
needs-review
needs-review
needs-review
netbird-ui
netbird-ui
netbird-ui
networking
networking
networking
new-platform
new-platform
new-platform
nginx
nginx
nginx
notification
notification
notification
okta
okta
okta
openwrt
openwrt
openwrt
P2
P2
P2
packaging
packaging
packaging
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
performance
performance
performance
postgres
postgres
postgres
posture-checks
posture-checks
posture-checks
psk
psk
psk
pull-request
Mirrored from GitHub Pull Request
 question
question
question
refactor
refactor
refactor
relay
relay
relay
release
release
release
rfc
rfc
rfc
routes
routes
routes
security
security
security
security-improvement
security-improvement
security-improvement
security-related
security-related
security-related
self-hosting
self-hosting
self-hosting
server
server
server
signal
signal
signal
sleep-issue
sleep-issue
sleep-issue
ssh
ssh
ssh
ssl
ssl
ssl
status
status
status
store
store
store
synology
synology
synology
system-compatibility-issue
system-compatibility-issue
system-compatibility-issue
test-suite
test-suite
test-suite
third-party-integration
third-party-integration
third-party-integration
triage
triage
triage
triage
triage
triage
triage-needed
triage-needed
triage-needed
troubleshooting
troubleshooting
troubleshooting
UX
UX
UX
waiting-feedback
waiting-feedback
waiting-feedback
windows
windows
windows
wontfix
wontfix
wontfix
zitadel
zitadel
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dynr-servers saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#8723
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?