DYNR/netbird
2
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-07 02:29:06 -04:00
Code Issues 23.5k Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #6022] NetBird netbird-acl-forward-filter drops k3s/kube-router-forwarded VPN traffic due to mark-bit collision with kube-router netpol #11461

New Issue
Open
opened 2026-08-05 01:29:46 -04:00 by saavagebueno · 3 comments
saavagebueno commented 2026-08-05 01:29:46 -04:00
Owner
Copy Link

Originally created by @snowzach on GitHub (Apr 28, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/6022

NetBird netbird-acl-forward-filter drops k3s/kube-router-forwarded VPN traffic due to mark-bit collision with kube-router netpol

I'm not gonna lie, the robots helped me figure it out but it does seem like k3s and netbird rules trample each other.

Version 0.70.0

TL;DR

NetBird's netbird-acl-forward-filter chain identifies "VPN-peer → local-host" traffic by a meta mark 0x0001bd20 set in netbird-mangle-prerouting. Bit 0x10000 is set inside that mark. kube-router's per-pod NetworkPolicy chain (KUBE-POD-FW-*) uses bit 0x10000 as its own "policy-permitted" flag and unconditionally clears it at the end of every per-pod evaluation. After kube-router's iptables FORWARD chain runs, the packet's mark is 0x0002bd20 instead of 0x0001bd20, so NetBird's meta mark 0x0001bd20 accept rule no longer matches and the packet falls through to the chain's catch-all iifname "wt0" drop.

The collision is silent — no rejects, no logs from either side.

NB_DISABLE_FIREWALL=true works around it by removing NetBird's nftables table entirely.

Environment

Component Version / detail
NetBird agent 0.70.0
Host OS Ubuntu (kernel uses nf_tables backend; iptables binary is iptables-nft)
Kubernetes k3s with kube-router as netpol controller, flannel CNI, default traefik ingress
Traefik Helm chart traefik-37.1.0, image docker.io/traefik:v3.5.1, Service type: ClusterIP, Deployment with hostPort: 80 → 8080 and hostPort: 443 → 8443 declared on the container; reachable from outside via the CNI portmap plugin writing CNI-DN-… DNAT rules in nat table PREROUTING/OUTPUT
NetBird interface wt0, 10.79.25.88/18, peer subnet 10.79.0.0/18
VPN client 10.79.43.248, attempting curl https://api-prod-tyo3-01:443

Symptom

  • VPN client SYN to <host-wt0-IP>:443 arrives on wt0 and times out (no SYN-ACK, no RST).
  • LAN clients on the host's primary interface and localhost succeed.
  • After systemctl restart netbird, VPN→host:443 works for ~60s then fails again. (Believed to coincide with kube-router's periodic iptables sync re-registering its base chains after NetBird's, flipping evaluation order at the same hook priority.)
  • NB_DISABLE_FIREWALL=true resolves the issue permanently.

Diagnostic trace

1. Packet enters and is correctly marked + DNAT'd

dmesg LOGs added at every netfilter hook (iptables -t … -I … 1 -j LOG) showed:

RAW-PRE:        IN=wt0 SRC=10.79.43.248 DST=10.79.25.88 SPT=…  DPT=443  SYN
MANGLE-PRE:     IN=wt0 SRC=10.79.43.248 DST=10.79.25.88 SPT=…  DPT=443  MARK=0x1bd20   ← NetBird sets the mark
NAT-PRE:        IN=wt0 SRC=10.79.43.248 DST=10.79.25.88 SPT=…  DPT=443  MARK=0x1bd20   ← unchanged
MANGLE-FWD-POST:IN=wt0 OUT=cni0 SRC=10.79.43.248 DST=10.42.0.17 SPT=…  DPT=8443 MARK=0x1bd20  ← CNI-DN DNAT'd to traefik pod
FILTER-FWD:     IN=wt0 OUT=cni0 SRC=10.79.43.248 DST=10.42.0.17 SPT=…  DPT=8443 MARK=0x1bd20

So far so good. NetBird's mark is the post-DNAT value still 0x1bd20.

2. Packet never reaches the egress interface

tcpdump -ni any 'host 10.79.43.248 or host 10.42.0.17' shows the packet on wt0 In only — never on cni0, never on any veth*. The pod never receives the SYN.

net.ipv4.ip_forward = 1, ip route get 10.42.0.17 → dev cni0, all interface-level forwarding sysctls = 1, rp_filter = 2 (loose). Routing is fine.

3. NetBird's nftables table

table ip netbird {
    set nb0000001 {
        type ipv4_addr; flags dynamic
        elements = { 10.79.43.248, … }              # all peers
    }

    chain netbird-mangle-prerouting {
        type filter hook prerouting priority mangle; policy accept;
        iifname "wt0" ct state new ct mark set 0x0001bd10
        iifname "wt0" ip saddr @nb0000001 fib daddr type local meta mark set 0x0001bd20
    }

    chain netbird-acl-forward-filter {
        type filter hook forward priority filter; policy accept;
        meta mark 0x0001bd20 accept                   # the only "permit VPN→local" rule
        iifname "wt0" jump netbird-rt-fwd             # only accepts established,related
        iifname "wt0" drop                            # catch-all
    }
    …
}

In isolation, the SYN's mark is 0x0001bd20 when entering the FORWARD hook, so meta mark 0x0001bd20 accept should match.

4. Mark collision with kube-router

iptables FORWARD also runs at hook forward priority filter (= 0) — same priority as netbird-acl-forward-filter. Its first rule is:

KUBE-ROUTER-FORWARD  /* kube-router netpol – TEMCG2JMHZYE7H7T */

…which dispatches into per-pod chains (KUBE-POD-FW-…). The traefik pod's chain ends with the standard kube-router epilogue:

NFLOG  ... mark match ! 0x10000/0x10000 ...
REJECT ... mark match ! 0x10000/0x10000 ...
MARK and 0xfffeffff               ← clears bit 0x10000
MARK or  0x20000                  ← sets  bit 0x20000   (kube-router "permitted")

The collision:

NetBird mark    : 0x0001bd20  =  0000 0001 1011 1101 0010 0000
                                          ^
                                          bit 0x10000 is SET as part of NetBird's tag

After KUBE-POD-FW :
  & 0xfffeffff  : 0x0000bd20  (bit 0x10000 cleared by kube-router)
  | 0x20000     : 0x0002bd20  (bit 0x20000 set)

So when control returns to netbird-acl-forward-filter, the packet's mark is 0x0002bd20, not 0x0001bd20. NetBird's meta mark 0x0001bd20 accept no longer matches; the packet falls through to iifname "wt0" drop and is silently dropped.

5. Why it works for ~60s after a restart

Both NetBird's netbird-acl-forward-filter and the iptables-compat FORWARD chain hook forward at the same priority. Order of evaluation between two base chains at the same hook+priority is determined by registration order.

  • Right after systemctl restart netbird, NetBird re-registers its chain after iptables — so iptables FORWARD evaluates first, kube-router clears bit 0x10000, and NetBird's mark-accept rule misses. It should already be broken.
  • However, in the brief window before kube-router/kube-proxy's next iptables reconcile, the chain ordering is such that NetBird's chain may evaluate first (still seeing the original mark and accepting), with the kube-router pass producing a verdict that doesn't override the prior accept.
  • On the next kube-router/kube-proxy reconcile (~60s default), kube-router does an iptables-restore, re-registers its base chain, and the eval order swaps. From that moment on the bit-clearing wins and NetBird drops the SYN.

(That's the most consistent explanation for the timing; the root cause — bit collision — is the real bug, regardless of which chain wins the race.)

Reproduction

  1. Single-node k3s with default flannel CNI + kube-router netpol + traefik (Helm chart with hostPort 80/443 enabled).
  2. NetBird ≥ 0.70.0 client on the same host, joined to a network with at least one peer.
  3. From a peer, curl https://<host-wt0-IP>:443.
  4. systemctl restart netbird → curl works briefly → fails within ~60s.

Workaround

Disable NetBird's firewall management:

sudo netbird service reconfigure --service-env NB_DISABLE_FIREWALL=true
sudo systemctl restart netbird

Verify: sudo nft list table ip netbird → "No such file or directory".

Suggested fix

The bit collision is the design flaw. Two options:

  1. Don't use bit 0x10000 in NetBird's marks. It's effectively reserved by kube-router and possibly other CNIs (Calico uses similar bit-ranges). Pick a mark that doesn't share any low-bit space with 0x00010000/0x00020000 (e.g. anything in the upper 16 bits). At minimum, mask only the NetBird bits when checking — e.g. tag a connection with 0x80000000 and match meta mark & 0x80000000 == 0x80000000. I would suggest either picking a different bit or a config option that allows overriding a different bit.

  2. Match on ct mark instead of meta mark for the forward-accept rule. NetBird already sets ct mark 0x0001bd10 on new connections in netbird-mangle-prerouting; that mark is stored in the conntrack entry, is not affected by kube-router's per-packet MARK ops, and survives DNAT and chain hops. Rewriting the rule as:

    meta nfproto ipv4 ct mark and 0x0000ff00 == 0x0000bd00 accept    # or similar

    would make the accept independent of any other component touching the per-packet mark.

The same class of mark-stomp can affect any environment combining NetBird with another netfilter consumer that uses --set-mark/--xset-mark on the per-packet mark (kube-router, calico-felix, kube-proxy KUBE-MARK-MASQ, fwmark-based policy routing daemons, etc.). The conntrack-mark approach defends against all of them.

Mark allocation reference

Mark Set by Meaning Cleared by
0x00010000 kube-router KUBE-NWPLCY-DEFAULT rule 4 "Network policy evaluated, permitted" kube-router MARK and 0xfffeffff at end of KUBE-POD-FW-*
0x00020000 kube-router MARK or 0x20000 at end of KUBE-POD-FW-* "Final accept" — matched by FORWARD line mark 0x20000/0x20000 ACCEPT (kept)
0x00004000 kube-proxy KUBE-MARK-MASQ "Will be masqueraded in POSTROUTING" (kept)
0x0001bd10 NetBird netbird-mangle-prerouting (ct mark) "wt0-inbound new connection" (ct mark – persistent)
0x0001bd11 NetBird netbird-mangle-postrouting (ct mark) "wt0-outbound new connection" (ct mark – persistent)
0x0001bd20 NetBird netbird-mangle-prerouting (meta mark) "wt0-inbound peer→local" kube-router clears bit 0x10000 of this mark while passing through KUBE-POD-FW-*
0x0001bd21 / 0x0001bd22 NetBird (meta mark) masquerade hint for routed traffic (unused in this report)

The 0x0001bd20 value contains bits 0x10000, 0x8000, 0x2000, 0x1000, 0x800, 0x100, 0x20 — the bit-16 overlap with kube-router's reserved bit is the failure mode.

Files / commands used during diagnosis

  • iptables -L FORWARD -nv --line-numbers — confirmed kube-router's KUBE-ROUTER-FORWARD is line 1 of FORWARD and ACCEPT in wt0 is line 3
  • iptables -t nat -S CNI-HOSTPORT-DNAT, iptables -t nat -S CNI-DN-… — confirmed traefik's hostPort DNAT is in place and source-agnostic
  • nft list table ip netbird — produced the rule set above
  • iptables -L KUBE-NWPLCY-DEFAULT -nv — confirmed final rule is MARK or 0x10000 for any traffic, i.e. the chain is permissive (no NetworkPolicy resource exists)
  • LOG instrumentation at raw/PREROUTING, mangle/PREROUTING, nat/PREROUTING, mangle/FORWARD, filter/FORWARD, nat/POSTROUTING
  • tcpdump -ni any 'host 10.79.43.248 or host 10.42.0.17'
  • sysctl net.ipv4.{ip_forward,conf.{all,wt0,cni0}.{forwarding,rp_filter}} — all confirmed correct

Related upstream issues

Notes

I'm willing to come up with a PR for changing the bits or overriding with a config option if you wish. I could also try to work on the connection mark one if that's the way you wanted to go. That seems possibly a lot more complicated though. Let me know which way you are leaning.

Originally created by @snowzach on GitHub (Apr 28, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/6022 # NetBird `netbird-acl-forward-filter` drops k3s/kube-router-forwarded VPN traffic due to mark-bit collision with kube-router netpol I'm not gonna lie, the robots helped me figure it out but it does seem like k3s and netbird rules trample each other. Version 0.70.0 ## TL;DR NetBird's `netbird-acl-forward-filter` chain identifies "VPN-peer → local-host" traffic by a `meta mark 0x0001bd20` set in `netbird-mangle-prerouting`. **Bit `0x10000` is set inside that mark.** kube-router's per-pod NetworkPolicy chain (`KUBE-POD-FW-*`) **uses bit `0x10000` as its own "policy-permitted" flag** and unconditionally **clears it** at the end of every per-pod evaluation. After kube-router's iptables FORWARD chain runs, the packet's mark is `0x0002bd20` instead of `0x0001bd20`, so NetBird's `meta mark 0x0001bd20 accept` rule no longer matches and the packet falls through to the chain's catch-all `iifname "wt0" drop`. The collision is silent — no rejects, no logs from either side. `NB_DISABLE_FIREWALL=true` works around it by removing NetBird's nftables table entirely. --- ## Environment | Component | Version / detail | |---|---| | NetBird agent | `0.70.0` | | Host OS | Ubuntu (kernel uses `nf_tables` backend; iptables binary is iptables-nft) | | Kubernetes | k3s with **kube-router** as netpol controller, **flannel** CNI, default **traefik** ingress | | Traefik | Helm chart `traefik-37.1.0`, image `docker.io/traefik:v3.5.1`, Service `type: ClusterIP`, Deployment with `hostPort: 80 → 8080` and `hostPort: 443 → 8443` declared on the container; reachable from outside via the **CNI portmap plugin** writing `CNI-DN-…` DNAT rules in `nat` table `PREROUTING`/`OUTPUT` | | NetBird interface | `wt0`, `10.79.25.88/18`, peer subnet `10.79.0.0/18` | | VPN client | `10.79.43.248`, attempting `curl https://api-prod-tyo3-01:443` | ## Symptom - VPN client SYN to `<host-wt0-IP>:443` arrives on `wt0` and times out (no SYN-ACK, no RST). - LAN clients on the host's primary interface and `localhost` succeed. - After `systemctl restart netbird`, VPN→host:443 **works for ~60s** then fails again. (Believed to coincide with kube-router's periodic iptables sync re-registering its base chains *after* NetBird's, flipping evaluation order at the same hook priority.) - `NB_DISABLE_FIREWALL=true` resolves the issue permanently. ## Diagnostic trace ### 1. Packet enters and is correctly marked + DNAT'd dmesg LOGs added at every netfilter hook (`iptables -t … -I … 1 -j LOG`) showed: ``` RAW-PRE: IN=wt0 SRC=10.79.43.248 DST=10.79.25.88 SPT=… DPT=443 SYN MANGLE-PRE: IN=wt0 SRC=10.79.43.248 DST=10.79.25.88 SPT=… DPT=443 MARK=0x1bd20 ← NetBird sets the mark NAT-PRE: IN=wt0 SRC=10.79.43.248 DST=10.79.25.88 SPT=… DPT=443 MARK=0x1bd20 ← unchanged MANGLE-FWD-POST:IN=wt0 OUT=cni0 SRC=10.79.43.248 DST=10.42.0.17 SPT=… DPT=8443 MARK=0x1bd20 ← CNI-DN DNAT'd to traefik pod FILTER-FWD: IN=wt0 OUT=cni0 SRC=10.79.43.248 DST=10.42.0.17 SPT=… DPT=8443 MARK=0x1bd20 ``` So far so good. NetBird's mark is the post-DNAT value still `0x1bd20`. ### 2. Packet never reaches the egress interface `tcpdump -ni any 'host 10.79.43.248 or host 10.42.0.17'` shows the packet on `wt0 In` only — never on `cni0`, never on any `veth*`. The pod never receives the SYN. `net.ipv4.ip_forward = 1`, `ip route get 10.42.0.17 → dev cni0`, all interface-level forwarding sysctls = 1, `rp_filter = 2` (loose). Routing is fine. ### 3. NetBird's nftables table ``` table ip netbird { set nb0000001 { type ipv4_addr; flags dynamic elements = { 10.79.43.248, … } # all peers } chain netbird-mangle-prerouting { type filter hook prerouting priority mangle; policy accept; iifname "wt0" ct state new ct mark set 0x0001bd10 iifname "wt0" ip saddr @nb0000001 fib daddr type local meta mark set 0x0001bd20 } chain netbird-acl-forward-filter { type filter hook forward priority filter; policy accept; meta mark 0x0001bd20 accept # the only "permit VPN→local" rule iifname "wt0" jump netbird-rt-fwd # only accepts established,related iifname "wt0" drop # catch-all } … } ``` In isolation, the SYN's mark **is** `0x0001bd20` when entering the FORWARD hook, so `meta mark 0x0001bd20 accept` *should* match. ### 4. Mark collision with kube-router iptables FORWARD also runs at `hook forward priority filter` (= 0) — same priority as `netbird-acl-forward-filter`. Its first rule is: ``` KUBE-ROUTER-FORWARD /* kube-router netpol – TEMCG2JMHZYE7H7T */ ``` …which dispatches into per-pod chains (`KUBE-POD-FW-…`). The traefik pod's chain ends with the standard kube-router epilogue: ``` NFLOG ... mark match ! 0x10000/0x10000 ... REJECT ... mark match ! 0x10000/0x10000 ... MARK and 0xfffeffff ← clears bit 0x10000 MARK or 0x20000 ← sets bit 0x20000 (kube-router "permitted") ``` **The collision:** ``` NetBird mark : 0x0001bd20 = 0000 0001 1011 1101 0010 0000 ^ bit 0x10000 is SET as part of NetBird's tag After KUBE-POD-FW : & 0xfffeffff : 0x0000bd20 (bit 0x10000 cleared by kube-router) | 0x20000 : 0x0002bd20 (bit 0x20000 set) ``` So when control returns to `netbird-acl-forward-filter`, the packet's mark is `0x0002bd20`, **not** `0x0001bd20`. NetBird's `meta mark 0x0001bd20 accept` no longer matches; the packet falls through to `iifname "wt0" drop` and is silently dropped. ### 5. Why it works for ~60s after a restart Both NetBird's `netbird-acl-forward-filter` and the iptables-compat `FORWARD` chain hook `forward` at the same priority. Order of evaluation between two base chains at the same hook+priority is determined by **registration order**. - Right after `systemctl restart netbird`, NetBird re-registers its chain *after* iptables — so iptables FORWARD evaluates first, kube-router clears bit `0x10000`, and NetBird's mark-accept rule *misses*. It should already be broken. - However, in the brief window before kube-router/kube-proxy's next iptables reconcile, the chain ordering is such that NetBird's chain may evaluate first (still seeing the original mark and accepting), with the kube-router pass producing a verdict that doesn't override the prior accept. - On the next kube-router/kube-proxy reconcile (~60s default), kube-router does an `iptables-restore`, **re-registers** its base chain, and the eval order swaps. From that moment on the bit-clearing wins and NetBird drops the SYN. (That's the most consistent explanation for the timing; the root cause — bit collision — is the real bug, regardless of which chain wins the race.) ## Reproduction 1. Single-node k3s with default flannel CNI + kube-router netpol + traefik (Helm chart with `hostPort` 80/443 enabled). 2. NetBird ≥ 0.70.0 client on the same host, joined to a network with at least one peer. 3. From a peer, `curl https://<host-wt0-IP>:443`. 4. `systemctl restart netbird` → curl works briefly → fails within ~60s. ## Workaround Disable NetBird's firewall management: ```bash sudo netbird service reconfigure --service-env NB_DISABLE_FIREWALL=true sudo systemctl restart netbird ``` Verify: `sudo nft list table ip netbird` → "No such file or directory". ## Suggested fix The bit collision is the design flaw. Two options: 1. **Don't use bit `0x10000` in NetBird's marks.** It's effectively reserved by kube-router and possibly other CNIs (Calico uses similar bit-ranges). Pick a mark that doesn't share any low-bit space with `0x00010000`/`0x00020000` (e.g. anything in the upper 16 bits). At minimum, mask only the *NetBird* bits when checking — e.g. tag a connection with `0x80000000` and match `meta mark & 0x80000000 == 0x80000000`. I would suggest either picking a different bit or a config option that allows overriding a different bit. 2. **Match on `ct mark` instead of `meta mark`** for the forward-accept rule. NetBird already sets `ct mark 0x0001bd10` on new connections in `netbird-mangle-prerouting`; that mark is stored in the conntrack entry, is not affected by kube-router's per-packet `MARK` ops, and survives DNAT and chain hops. Rewriting the rule as: ``` meta nfproto ipv4 ct mark and 0x0000ff00 == 0x0000bd00 accept # or similar ``` would make the accept independent of any other component touching the per-packet mark. The same class of mark-stomp can affect any environment combining NetBird with another netfilter consumer that uses `--set-mark`/`--xset-mark` on the per-packet mark (kube-router, calico-felix, kube-proxy `KUBE-MARK-MASQ`, fwmark-based policy routing daemons, etc.). The conntrack-mark approach defends against all of them. ## Mark allocation reference | Mark | Set by | Meaning | Cleared by | |---|---|---|---| | `0x00010000` | kube-router `KUBE-NWPLCY-DEFAULT` rule 4 | "Network policy evaluated, permitted" | kube-router `MARK and 0xfffeffff` at end of `KUBE-POD-FW-*` | | `0x00020000` | kube-router `MARK or 0x20000` at end of `KUBE-POD-FW-*` | "Final accept" — matched by FORWARD line `mark 0x20000/0x20000 ACCEPT` | (kept) | | `0x00004000` | kube-proxy `KUBE-MARK-MASQ` | "Will be masqueraded in POSTROUTING" | (kept) | | `0x0001bd10` | NetBird `netbird-mangle-prerouting` (ct mark) | "wt0-inbound new connection" | (ct mark – persistent) | | `0x0001bd11` | NetBird `netbird-mangle-postrouting` (ct mark) | "wt0-outbound new connection" | (ct mark – persistent) | | **`0x0001bd20`** | NetBird `netbird-mangle-prerouting` (meta mark) | "wt0-inbound peer→local" | **kube-router clears bit `0x10000` of this mark while passing through `KUBE-POD-FW-*`** | | `0x0001bd21` / `0x0001bd22` | NetBird (meta mark) | masquerade hint for routed traffic | (unused in this report) | The `0x0001bd20` value contains **bits 0x10000, 0x8000, 0x2000, 0x1000, 0x800, 0x100, 0x20** — the bit-16 overlap with kube-router's reserved bit is the failure mode. ## Files / commands used during diagnosis - `iptables -L FORWARD -nv --line-numbers` — confirmed kube-router's `KUBE-ROUTER-FORWARD` is line 1 of FORWARD and `ACCEPT in wt0` is line 3 - `iptables -t nat -S CNI-HOSTPORT-DNAT`, `iptables -t nat -S CNI-DN-…` — confirmed traefik's hostPort DNAT is in place and source-agnostic - `nft list table ip netbird` — produced the rule set above - `iptables -L KUBE-NWPLCY-DEFAULT -nv` — confirmed final rule is `MARK or 0x10000` for any traffic, i.e. the chain is permissive (no NetworkPolicy resource exists) - LOG instrumentation at `raw/PREROUTING`, `mangle/PREROUTING`, `nat/PREROUTING`, `mangle/FORWARD`, `filter/FORWARD`, `nat/POSTROUTING` - `tcpdump -ni any 'host 10.79.43.248 or host 10.42.0.17'` - `sysctl net.ipv4.{ip_forward,conf.{all,wt0,cni0}.{forwarding,rp_filter}}` — all confirmed correct ## Related upstream issues - [#2926 — netbird 0.32.0 breaks K3s 1.32.2+k3s1 with flannel due to iptables conflicts](https://github.com/netbirdio/netbird/issues/2926) - [#3363 — Add an option to force `nftables` usage](https://github.com/netbirdio/netbird/issues/3363) - [#4484 — Why use iptables in `client/firewall/nftags/router.linux.go`](https://github.com/netbirdio/netbird/issues/4484) - [#481 — Add return forward rule for network routes](https://github.com/netbirdio/netbird/issues/481) ## Notes I'm willing to come up with a PR for changing the bits or overriding with a config option if you wish. I could also try to work on the connection mark one if that's the way you wanted to go. That seems possibly a lot more complicated though. Let me know which way you are leaning.
saavagebueno added the triage-needed label 2026-08-05 01:29:46 -04:00
saavagebueno commented 2026-08-05 01:29:46 -04:00
Author
Owner
Copy Link

@alfadb commented on GitHub (May 30, 2026):

Hitting the same root cause from PR #5697 on a different stack — adding this as an additional reproducer in case it helps triage.

Environment

Component Version / detail
NetBird agent 0.71.4 (iptables-nft mode)
Kubernetes single-node, kubeadm-installed, flannel CNI, kube-proxy in nftables backend (no kube-router)
Ingress Envoy Gateway (LoadBalancer Service, MetalLB L2 announcing 192.168.50.20–99)
NetBird role of the node Network Router peer advertising the LAN subnet 192.168.50.0/24
Failing path Remote NetBird peer (overlay 100.108.227.115) → wt0 on the routing-peer node → kube-proxy DNAT to envoy pod (10.98.0.127:10443) → drop

Key difference from this issue as originally reported

The dropped packets carry mark 0x1bd21, not 0x1bd20. There is no kube-router stomping a bit here — the mark 0x1bd21 is set by NetBird itself in netbird-mangle-prerouting:

ip daddr 192.168.50.0/24 iifname "wt0" ct state new meta mark set 0x1bd21

Then in mangle FORWARD the guard from PR #5697 fires:

iifname "wt0" ct status dnat meta mark != 0x0001bd20 drop

0x1bd21 ≠ 0x1bd20 → SYN dropped. Counter increments cleanly with every retry of nc -vz <LB_IP> 443 from the remote peer.

Verified with iptables -t mangle -L FORWARD -nv:

2  9560  6000K  ACCEPT  all  --  wt0  *  0.0.0.0/0  0.0.0.0/0  ctstate RELATED,ESTABLISHED
3   141   8460  DROP    all  --  wt0  *  0.0.0.0/0  0.0.0.0/0  ctstate DNAT mark match ! 0x1bd20

141 dropped packets before I even started testing — all routed-peer → LB-IP TCP SYNs over many days.

Root cause interpretation

PR #5697's guard whitelists only the peer → local-host mark (0x1bd20). The peer → routed-network mark (0x1bd21, also set by NetBird) was not added to the whitelist. So any deployment where the NetBird routing peer is also a Kubernetes node that uses kube-proxy DNAT for Service traffic silently breaks — including single-node k3s/k8s with flannel/Cilium/Calico (any CNI), regardless of whether kube-router is involved.

This is broader than the mark-bit-collision angle. The fix proposal in this issue to switch the guard to ct mark would resolve both flavors. Alternatively, the simplest patch is to extend the whitelist:

iifname "wt0" ct status dnat meta mark != { 0x1bd20, 0x1bd21, 0x1bd22 } drop

Workaround that preserves NetBird ACL

NB_DISABLE_FIREWALL=true works but disables the entire NetBird ACL surface, which is a regression for anyone using NetBird Policies. A more surgical workaround is a separate nftables inet table at PREROUTING priority -101 (between NetBird mangle at -150 and kube-proxy nat at -100) that overwrites the meta mark to 0x1bd20 for the LB CIDR:

table inet netbird-k8s-fix {
    chain prerouting {
        type filter hook prerouting priority -101; policy accept;
        iifname "wt0" ip daddr 192.168.50.0/24 ct state new meta mark set 0x1bd20
    }
}

NetBird daemon does not touch the inet family, so this survives systemctl restart netbird and version upgrades. Tested working on 0.71.4. Posting it here in case it's useful for others hitting the same.

Happy to test any candidate patch.

<!-- gh-comment-id:4583516228 --> @alfadb commented on GitHub (May 30, 2026): Hitting the same root cause from PR #5697 on a different stack — adding this as an additional reproducer in case it helps triage. **Environment** | Component | Version / detail | |---|---| | NetBird agent | `0.71.4` (iptables-nft mode) | | Kubernetes | single-node, kubeadm-installed, **flannel** CNI, **kube-proxy in nftables backend** (no kube-router) | | Ingress | Envoy Gateway (LoadBalancer Service, MetalLB L2 announcing 192.168.50.20–99) | | NetBird role of the node | Network Router peer advertising the LAN subnet `192.168.50.0/24` | | Failing path | Remote NetBird peer (overlay `100.108.227.115`) → wt0 on the routing-peer node → kube-proxy DNAT to envoy pod (`10.98.0.127:10443`) → drop | **Key difference from this issue as originally reported** The dropped packets carry mark **`0x1bd21`**, **not** `0x1bd20`. There is no kube-router stomping a bit here — the mark `0x1bd21` is set by NetBird itself in `netbird-mangle-prerouting`: ``` ip daddr 192.168.50.0/24 iifname "wt0" ct state new meta mark set 0x1bd21 ``` Then in mangle FORWARD the guard from PR #5697 fires: ``` iifname "wt0" ct status dnat meta mark != 0x0001bd20 drop ``` `0x1bd21` ≠ `0x1bd20` → SYN dropped. Counter increments cleanly with every retry of `nc -vz <LB_IP> 443` from the remote peer. Verified with `iptables -t mangle -L FORWARD -nv`: ``` 2 9560 6000K ACCEPT all -- wt0 * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 3 141 8460 DROP all -- wt0 * 0.0.0.0/0 0.0.0.0/0 ctstate DNAT mark match ! 0x1bd20 ``` 141 dropped packets before I even started testing — all routed-peer → LB-IP TCP SYNs over many days. **Root cause interpretation** PR #5697's guard whitelists only the `peer → local-host` mark (`0x1bd20`). The `peer → routed-network` mark (`0x1bd21`, also set by NetBird) was not added to the whitelist. So **any deployment where the NetBird routing peer is also a Kubernetes node that uses kube-proxy DNAT for Service traffic** silently breaks — including single-node k3s/k8s with flannel/Cilium/Calico (any CNI), regardless of whether kube-router is involved. This is broader than the mark-bit-collision angle. The fix proposal in this issue to switch the guard to `ct mark` would resolve both flavors. Alternatively, the simplest patch is to extend the whitelist: ``` iifname "wt0" ct status dnat meta mark != { 0x1bd20, 0x1bd21, 0x1bd22 } drop ``` **Workaround that preserves NetBird ACL** `NB_DISABLE_FIREWALL=true` works but disables the entire NetBird ACL surface, which is a regression for anyone using NetBird Policies. A more surgical workaround is a separate nftables `inet` table at PREROUTING priority `-101` (between NetBird mangle at `-150` and kube-proxy nat at `-100`) that overwrites the meta mark to `0x1bd20` for the LB CIDR: ``` table inet netbird-k8s-fix { chain prerouting { type filter hook prerouting priority -101; policy accept; iifname "wt0" ip daddr 192.168.50.0/24 ct state new meta mark set 0x1bd20 } } ``` NetBird daemon does not touch the `inet` family, so this survives `systemctl restart netbird` and version upgrades. Tested working on 0.71.4. Posting it here in case it's useful for others hitting the same. Happy to test any candidate patch.
saavagebueno commented 2026-08-05 01:29:46 -04:00
Author
Owner
Copy Link

@alfadb commented on GitHub (Jun 2, 2026):

Follow-up to my previous comment with a refined workaround based on further investigation.

What I missed initially

My first proposed workaround was an inet table at PREROUTING priority dstnat - 1 (-101) that pre-stamps mark 0x1bd20. It worked for one peer (close to the routing peer, direct WireGuard tunnel) but broke for a cross-region peer hitting the same LB IPs. tcpdump+LOG instrumentation in mangle FORWARD revealed why:

USA1-FWD-IN: IN=wt0 OUT=cni0 SRC=100.108.243.225 DST=10.98.0.187 ... SYN MARK=0x1fd20
                                                                            ^^^^^^^

MARK=0x1fd20, not 0x1bd20.

The diff is bit 0x4000 — kube-proxy's KUBE-MARK-MASQ bit. kube-proxy ORs it on top of any existing mark during nat-prerouting for LoadBalancer/ClusterIP traffic that needs POSTROUTING SNAT (externalTrafficPolicy: Cluster). So even though we set mark = 0x1bd20 at priority -101 (before kube-proxy's -100), kube-proxy then mutates it to 0x1fd20, which fails NetBird's strict-equality meta mark != 0x1bd20 check at mangle FORWARD priority -150.

Net result: with the PREROUTING workaround in place, every LB-IP-bound flow from a remote NetBird peer routed through this node still gets silently dropped. We initially missed this because our first test peer happened to succeed (likely due to specific conntrack state on that machine at test time); a second peer added to the mesh exposed the regression immediately, with the LOG-instrumented dmesg evidence above.

Refined workaround (PREROUTING → FORWARD priority -151)

Move the rewrite to mangle FORWARD, one tick before NetBird's guard, after kube-proxy has finished touching the mark:

table inet netbird-k8s-fix {
    chain forward-fix {
        type filter hook forward priority mangle - 1; policy accept;
        iifname "wt0" ct status dnat \
            meta mark set 0x1bd20 \
            counter comment "rewrite mark for NetBird PR#5697 bypass"
    }
}

Properties of this hook position:

  1. Atomic for all upstream mark mutations. Whatever 0x1bd2X | 0x4000 | … combination kube-proxy produces, we overwrite the whole word to 0x1bd20. NetBird's strict-equality check then passes.
  2. No daddr / LB CIDR config needed. ct status dnat precisely scopes to kube-proxy DNAT'd flows. Non-DNAT'd plain LAN traffic (e.g. ping <a-LAN-host-that-NetBird-routes>) keeps its NetBird-assigned mark 0x1bd21 and gets normal NetBird LAN SNAT — important for return path of remote peers whose peer-to-peer tunnel is relayed/unreliable.
  3. CNI-agnostic. Works the same for flannel / Cilium / Calico / etc. — only ct status dnat matters, not which CNI does the DNAT.
  4. NetBird daemon never touches it. Lives in inet family; NetBird only manages ip family tables, so this survives systemctl restart netbird and version upgrades.

Tested working on NetBird 0.71.4, kernel netfilter mixed-backend host (iptables-legacy for NetBird's own rules + native nftables for kube-proxy nft-mode), single-node k8s with flannel, MetalLB L2.

Acknowledged side effect

kube-proxy no longer sees its 0x4000 bit after we overwrite the mark → it skips POSTROUTING SNAT for these connections. The pod (envoy in our case) sees the original NetBird overlay source IP 100.108.x.x instead of the node IP. Reply traffic returns via the routing peer's own 100.108.0.0/16 dev wt0 route, no SNAT needed.

For most workloads (HTTP applications wanting real client IPs, audit/access logging) this is actually preferable. Workloads that rely on seeing node IPs would need a different fix.

Why a ct mark based upstream fix is still the right direction

snowzach's earlier proposal in this issue (switch the FORWARD guard to a conntrack-mark check derived from netbird-mangle-prerouting's ct mark 0x1bd10) would handle both:

  • The kube-router bit-stomp variant originally reported
  • This kube-proxy MASQ-bit variant
  • Any other netfilter-aware component that mutates meta mark per packet

because ct mark is set once when the connection is created and not touched by per-packet MARK ops downstream. Happy to test a candidate patch.

Reproducer summary

For repro by maintainers, the minimal setup that triggers this:

  • Single-node k8s (kubeadm/k3s/kubernetes-in-docker) with any CNI
  • kube-proxy with default config (externalTrafficPolicy: Cluster on a LoadBalancer Service)
  • NetBird 0.70+ on the same host, set as routing peer for the LAN subnet
  • A remote NetBird peer (anywhere in the mesh)
  • From remote peer: curl https://<LB-IP>:<port> → times out
  • mangle FORWARD iifname "wt0" ct status dnat mark != 0x1bd20 counter increments per SYN attempt
<!-- gh-comment-id:4603013238 --> @alfadb commented on GitHub (Jun 2, 2026): Follow-up to my previous comment with a refined workaround based on further investigation. ## What I missed initially My first proposed workaround was an `inet` table at PREROUTING priority `dstnat - 1` (-101) that pre-stamps mark `0x1bd20`. It worked for one peer (close to the routing peer, direct WireGuard tunnel) but broke for a cross-region peer hitting the same LB IPs. tcpdump+LOG instrumentation in mangle FORWARD revealed why: ``` USA1-FWD-IN: IN=wt0 OUT=cni0 SRC=100.108.243.225 DST=10.98.0.187 ... SYN MARK=0x1fd20 ^^^^^^^ ``` **`MARK=0x1fd20`, not `0x1bd20`.** The diff is bit `0x4000` — kube-proxy's `KUBE-MARK-MASQ` bit. kube-proxy ORs it on top of any existing mark during `nat-prerouting` for LoadBalancer/ClusterIP traffic that needs POSTROUTING SNAT (`externalTrafficPolicy: Cluster`). So even though we set mark = `0x1bd20` at priority -101 (before kube-proxy's -100), kube-proxy then mutates it to `0x1fd20`, which fails NetBird's strict-equality `meta mark != 0x1bd20` check at mangle FORWARD priority -150. Net result: with the PREROUTING workaround in place, every LB-IP-bound flow from a remote NetBird peer routed through this node still gets silently dropped. We initially missed this because our first test peer happened to succeed (likely due to specific conntrack state on that machine at test time); a second peer added to the mesh exposed the regression immediately, with the LOG-instrumented dmesg evidence above. ## Refined workaround (PREROUTING → FORWARD priority -151) Move the rewrite to mangle FORWARD, one tick before NetBird's guard, after kube-proxy has finished touching the mark: ```nft table inet netbird-k8s-fix { chain forward-fix { type filter hook forward priority mangle - 1; policy accept; iifname "wt0" ct status dnat \ meta mark set 0x1bd20 \ counter comment "rewrite mark for NetBird PR#5697 bypass" } } ``` Properties of this hook position: 1. **Atomic for all upstream mark mutations.** Whatever `0x1bd2X | 0x4000 | …` combination kube-proxy produces, we overwrite the whole word to `0x1bd20`. NetBird's strict-equality check then passes. 2. **No daddr / LB CIDR config needed.** `ct status dnat` precisely scopes to kube-proxy DNAT'd flows. Non-DNAT'd plain LAN traffic (e.g. `ping <a-LAN-host-that-NetBird-routes>`) keeps its NetBird-assigned mark `0x1bd21` and gets normal NetBird LAN SNAT — important for return path of remote peers whose peer-to-peer tunnel is relayed/unreliable. 3. **CNI-agnostic.** Works the same for flannel / Cilium / Calico / etc. — only `ct status dnat` matters, not which CNI does the DNAT. 4. **NetBird daemon never touches it.** Lives in `inet` family; NetBird only manages `ip` family tables, so this survives `systemctl restart netbird` and version upgrades. Tested working on NetBird 0.71.4, kernel netfilter mixed-backend host (iptables-legacy for NetBird's own rules + native nftables for kube-proxy nft-mode), single-node k8s with flannel, MetalLB L2. ## Acknowledged side effect kube-proxy no longer sees its `0x4000` bit after we overwrite the mark → it skips POSTROUTING SNAT for these connections. The pod (envoy in our case) sees the original NetBird overlay source IP `100.108.x.x` instead of the node IP. Reply traffic returns via the routing peer's own `100.108.0.0/16 dev wt0` route, no SNAT needed. For most workloads (HTTP applications wanting real client IPs, audit/access logging) this is actually preferable. Workloads that rely on seeing node IPs would need a different fix. ## Why a `ct mark` based upstream fix is still the right direction snowzach's earlier proposal in this issue (switch the FORWARD guard to a conntrack-mark check derived from `netbird-mangle-prerouting`'s `ct mark 0x1bd10`) would handle both: - The kube-router bit-stomp variant originally reported - This kube-proxy MASQ-bit variant - Any other netfilter-aware component that mutates `meta mark` per packet because `ct mark` is set once when the connection is created and not touched by per-packet `MARK` ops downstream. Happy to test a candidate patch. ## Reproducer summary For repro by maintainers, the minimal setup that triggers this: - Single-node k8s (kubeadm/k3s/kubernetes-in-docker) with any CNI - kube-proxy with default config (`externalTrafficPolicy: Cluster` on a LoadBalancer Service) - NetBird 0.70+ on the same host, set as routing peer for the LAN subnet - A remote NetBird peer (anywhere in the mesh) - From remote peer: `curl https://<LB-IP>:<port>` → times out - mangle FORWARD `iifname "wt0" ct status dnat mark != 0x1bd20` counter increments per SYN attempt
saavagebueno commented 2026-08-05 01:29:46 -04:00
Author
Owner
Copy Link

@head1328 commented on GitHub (Jul 1, 2026):

Confirming this on NetBird 0.73.2 (k3s, flannel CNI, wt0 as the NetBird interface, kube-proxy in nft mode), and that @alfadb's FORWARD-mark workaround (comment from 2026-06-02) resolves it for us as well.

Same symptom: a mesh peer reaching a NodePort over wt0 is dropped by

table ip mangle, chain FORWARD (priority mangle):
  iifname "wt0" ct status dnat meta mark != 0x0001bd20 counter drop

because kube-proxy DNATs the packet to the pod IP and NetBird's routing mark 0x1bd20 is not present. Node-local access and non-DNAT'd 443 work.

We applied @alfadb's approach (own nft table, forward hook before NetBird's mangle FORWARD, rewrite the mark for ct status dnat). Our variant differs only in details: ip family instead of inet, and priority -160 instead of mangle - 1.
NetBird's reconcile manages only its own tables/chains and does not touch this separate table, so it survives policy/peer reconciles and restarts.

table ip nb-nodeport-fix {
  chain premark {
    type filter hook forward priority -160; policy accept;
    iifname "wt0" ct status dnat meta mark set 0x0001bd20
  }
}

Applied idempotently by a systemd oneshot unit ordered after the NetBird service (re-created on boot; NetBird never removes it at runtime):

# /usr/local/bin/netbird-nodeport-mark.sh
#!/bin/bash
nft list table ip nb-nodeport-fix >/dev/null 2>&1 && exit 0
nft add table ip nb-nodeport-fix
nft add chain ip nb-nodeport-fix premark '{ type filter hook forward priority -160 ; policy accept ; }'
nft add rule  ip nb-nodeport-fix premark iifname "wt0" ct status dnat meta mark set 0x0001bd20

Validated with >20k connections / 0 failures from a mesh peer to a NodePort while repeatedly triggering NetBird policy reconciles; the table stayed in place throughout. The ct mark-based upstream fix that snowzach and @alfadb describe is still the
clean solution, since it also covers the per-packet mark mutations (kube-router bit-stomp, kube-proxy MASQ bit) without an external table.

<!-- gh-comment-id:4853063297 --> @head1328 commented on GitHub (Jul 1, 2026): Confirming this on **NetBird 0.73.2** (k3s, flannel CNI, `wt0` as the NetBird interface, kube-proxy in nft mode), and that **@alfadb's FORWARD-mark workaround** (comment from 2026-06-02) resolves it for us as well. Same symptom: a mesh peer reaching a NodePort over `wt0` is dropped by ``` table ip mangle, chain FORWARD (priority mangle): iifname "wt0" ct status dnat meta mark != 0x0001bd20 counter drop ``` because kube-proxy DNATs the packet to the pod IP and NetBird's routing mark `0x1bd20` is not present. Node-local access and non-DNAT'd `443` work. We applied @alfadb's approach (own nft table, `forward` hook before NetBird's `mangle FORWARD`, rewrite the mark for `ct status dnat`). Our variant differs only in details: `ip` family instead of `inet`, and priority `-160` instead of `mangle - 1`. NetBird's reconcile manages only its own tables/chains and does not touch this separate table, so it survives policy/peer reconciles and restarts. ```nft table ip nb-nodeport-fix { chain premark { type filter hook forward priority -160; policy accept; iifname "wt0" ct status dnat meta mark set 0x0001bd20 } } ``` Applied idempotently by a systemd oneshot unit ordered after the NetBird service (re-created on boot; NetBird never removes it at runtime): ```bash # /usr/local/bin/netbird-nodeport-mark.sh #!/bin/bash nft list table ip nb-nodeport-fix >/dev/null 2>&1 && exit 0 nft add table ip nb-nodeport-fix nft add chain ip nb-nodeport-fix premark '{ type filter hook forward priority -160 ; policy accept ; }' nft add rule ip nb-nodeport-fix premark iifname "wt0" ct status dnat meta mark set 0x0001bd20 ``` Validated with >20k connections / 0 failures from a mesh peer to a NodePort while repeatedly triggering NetBird policy reconciles; the table stayed in place throughout. The `ct mark`-based upstream fix that snowzach and @alfadb describe is still the clean solution, since it also covers the per-packet mark mutations (kube-router bit-stomp, kube-proxy MASQ bit) without an external table.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2021 Q4
2021 Q4
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
accessibility
accessibility
accessibility
acl
acl
acl
agent
agent
agent
agent
agent
agent
Android
Android
Android
Android
Android
Android
api
api
api
authentik
authentik
authentik
automation
automation
automation
azure
azure
azure
battery-usage
battery-usage
battery-usage
bug
cache
cache
cache
client
client
client
client-ui
client-ui
client-ui
cloud
cloud
cloud
cloud-only
cloud-only
cloud-only
cloudflare
cloudflare
cloudflare
community
community
community
compatibility
compatibility
compatibility
config-idp
config-idp
config-idp
config-issue
config-issue
config-issue
connection
connection
connection
contribution
contribution
contribution
coturn
coturn
coturn
cross-vpn
cross-vpn
cross-vpn
dashboard
dashboard
dashboard
data-usage
data-usage
data-usage
distribution
distribution
distribution
dns
dns
dns
docker
docker
docker
documentation
documentation
documentation
duplicate
duplicate
duplicate
enhancement
enhancement
event-stream
event-stream
event-stream
feature-request
feature-request
feature-request
freebsd
freebsd
freebsd
getting-started
getting-started
getting-started
go
go
go
good first issue
good first issue
good first issue
gui
gui
gui
help wanted
help wanted
help wanted
home-assistant
home-assistant
home-assistant
idp
idp
idp
inconsistency
inconsistency
inconsistency
integration
integration
integration
integrations
integrations
integrations
ios
ios
ios
ipv6
ipv6
ipv6
jwt
jwt
jwt
k8s
k8s
k8s
keycloak
keycloak
keycloak
linux
linux
linux
login
login
login
macos
macos
macos
management-service
management-service
management-service
Medium
Medium
Medium
missing-docs
missing-docs
missing-docs
mobile
mobile
mobile
moved-internal
moved-internal
moved-internal
needs-review
needs-review
needs-review
netbird-ui
netbird-ui
netbird-ui
networking
networking
networking
new-platform
new-platform
new-platform
nginx
nginx
nginx
notification
notification
notification
okta
okta
okta
openwrt
openwrt
openwrt
P2
P2
P2
packaging
packaging
packaging
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
performance
performance
performance
postgres
postgres
postgres
posture-checks
posture-checks
posture-checks
psk
psk
psk
pull-request
Mirrored from GitHub Pull Request
 question
question
question
refactor
refactor
refactor
relay
relay
relay
release
release
release
rfc
rfc
rfc
routes
routes
routes
security
security
security
security-improvement
security-improvement
security-improvement
security-related
security-related
security-related
self-hosting
self-hosting
self-hosting
server
server
server
signal
signal
signal
sleep-issue
sleep-issue
sleep-issue
ssh
ssh
ssh
ssl
ssl
ssl
status
status
status
store
store
store
synology
synology
synology
system-compatibility-issue
system-compatibility-issue
system-compatibility-issue
test-suite
test-suite
test-suite
third-party-integration
third-party-integration
third-party-integration
triage
triage
triage
triage
triage
triage
triage-needed
triage-needed
triage-needed
troubleshooting
troubleshooting
troubleshooting
UX
UX
UX
waiting-feedback
waiting-feedback
waiting-feedback
windows
windows
windows
wontfix
wontfix
wontfix
zitadel
zitadel
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dynr-servers saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11461
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?