[GH-ISSUE #5595] Netbird client not working in versions 0.30+ on kubernetes #11773

Open
opened 2026-08-05 01:30:56 -04:00 by saavagebueno · 0 comments
Owner

Originally created by @Haydgoki on GitHub (Mar 13, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5595

Describe the problem

Hello, I'm reporting a problem that I get when it comes to use netbird client versions > 0.29 inside my kubernetes cluster.
Netbird clients are installed on the nodes via Daemonsets and are working with the hostNetwork at "true"

To Reproduce
More infos below

Expected behavioro
I can reach my kubernetes nodes through netbird network, use my ingresses etc.

Are you using NetBird Cloud?
Self hosted

NetBird version
0.29.3 and 0.66.3

Is any other VPN software installed?
No
Debug output
Error: failed to connect to daemon error: context deadline exceeded You will see why

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings

Architecture choice

Before explaining the problem and the architecture, I know there is a documentation on using the netbird client inside kubernetes through a deployment. But I need to install netbird on the nodes to peer them to the VPN.
It works in v 0.29.

Additionnal data :

  • Kubernetes version 1.33
  • Kubernetes provider : OVH MKS
  • CNI: Calico

I got a self hosted netbird server in a network A

I got my netbird clients on some kubernetes. I got 3 clusters in 3 differents Network, calling them B,C and D.
Each of that networks get Netbird clients installed as a Daemonsets, using the hostNetwork at "true".

I got other networks and othe folks that go their netbird client on their PC or VMs working properly with the Version 0.66

I need to get netbirds clients on the kube nodes, because I need to make those cluster private and I need those cluster to communicate each other in specific ports (Web ports and some extra). All was fine with this implementation, since I wanted to update netbird from version 0.29 to 0.30 or plus.

So for those experimentations, I tested with netbird versions 0.29.3 and 0.66.3, and sometimes versions between them when I needed.

Here is my Daemonset manifest (working before v0.30) :

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: netbird-client
  namespace: netbird
spec:
  selector:
    matchLabels:
      app: netbird-client
  template:
    metadata:
      labels:
        app: netbird-client
    spec:
      hostNetwork: true # To link the node network as the pod network
      automountServiceAccountToken: true
      containers:
      - env:
        - name: NB_SETUP_KEY
          valueFrom:
            secretKeyRef:
              key: setupKey
              name: <MY_SECRET>
        - name: NB_MANAGEMENT_URL
          value: <MY_URL>
        - name: NB_ADMIN_URL
          value: <MY_URL>
        - name: NB_HOSTNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: NB_LOG_LEVEL
          value: info
        image: netbirdio/netbird:0.29.3 # Or 0.66.3
        name: netbird-client
        securityContext:
          allowPrivilegeEscalation: true
          capabilities:
            add:
            - NET_ADMIN
            - SYS_RESOURCE
            - SYS_ADMIN
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst

Trouble Shooting

With this implementation it's not working on versions > 0.29
So If I upgrade, all my traffic to my kubernetes cluster just shut down.

What is working on both

On both versions :

  • The pods starts
  • My peer register to my netbird and I can see them on the dashboard
  • I can make my netbird commands on the pods

What is broken > 0.29

I can’t reach websites and services hosted on the kubernetes cluster (ingresses, NodePort services) from other NetBird peers. TCP connections hang indefinitely (SYN sent, no SYN-ACK received).

What is break on both versions

  • When making netbird routes list on pods, getting Error: failed to connect to daemon error: context deadline exceeded
  • I got the netbird iptable not existing on pods :
$ ip rule list
0:	from all lookup local 
100:	from all lookup main suppress_prefixlength 0 
110:	from all fwmark 0x1bd00 lookup 7120 
32766:	from all lookup main 
32767:	from all lookup default

$ ip route show table 7120
ip: invalid argument '7120' to 'table'

What I tried and was useless

  • Using different additional vars :
    • NB_WG_KERNEL_DISABLED = true
    • NB_DISABLE_CUSTOM_ROUTING = true
    • NB_SKIP_SOCKET_MARK = true
  • Adding different other capabilities :
    • SYS_MODULE
    • NET_RAW
  • Using those kube flags
    • hostPID: true
    • hostIPC: true
  • Adding and mounting those volumes:
    • /dev/net/tun
    • /var/lib/netbird

I got a try with the host network at false on the 0.66, but it worked like half of it. The think is that was the pod that was register in netbird and not the node. So I got problems when:

  • Trying to join netbirds IPs, it was not redirected to the node and trapped into the pod
  • Trying to use the VPN from another pod on the same cluster (can't just use the pod network without service or anything magicaly)

Why does this version breaks everything

I don't really know, I know the Biggest breaking change since 0.30 is The ACL policies.
So Maybe it's the netbird ps that wants to scrap all the traffic comming and outgoing from private networks addresses to check if it's policy friendly ? But with the network on the node and the netbird payload on its own pod, it may be crashing up ?

And if it's that or not I don't understand why it works in 0.29 but not on newer versions.

Originally created by @Haydgoki on GitHub (Mar 13, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5595 ## Describe the problem Hello, I'm reporting a problem that I get when it comes to use netbird client versions > 0.29 inside my kubernetes cluster. Netbird clients are installed on the nodes via Daemonsets and are working with the hostNetwork at "true" **To Reproduce** More infos below **Expected behavior**o I can reach my kubernetes nodes through netbird network, use my ingresses etc. **Are you using NetBird Cloud?** Self hosted **NetBird version** `0.29.3` and `0.66.3` **Is any other VPN software installed?** No **Debug output** `Error: failed to connect to daemon error: context deadline exceeded` You will see why **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings ## Architecture choice Before explaining the problem and the architecture, I know there is a documentation on using the netbird client inside kubernetes through a deployment. But I need to install netbird on the nodes to peer them to the VPN. It works in v 0.29. Additionnal data : * Kubernetes version 1.33 * Kubernetes provider : OVH MKS * CNI: Calico I got a **self hosted netbird server** in a network A I got my netbird clients on some kubernetes. I got 3 clusters in 3 differents Network, calling them B,C and D. Each of that networks get Netbird clients installed as a Daemonsets, using the hostNetwork at "true". I got other networks and othe folks that go their netbird client on their PC or VMs working properly with the Version 0.66 I need to get netbirds clients on the kube nodes, because I need to make those cluster private and I need those cluster to communicate each other in specific ports (Web ports and some extra). All was fine with this implementation, since I wanted to update netbird from version 0.29 to 0.30 or plus. So for those experimentations, I tested with netbird versions 0.29.3 and 0.66.3, and sometimes versions between them when I needed. Here is my Daemonset manifest (working before v0.30) : ```yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: netbird-client namespace: netbird spec: selector: matchLabels: app: netbird-client template: metadata: labels: app: netbird-client spec: hostNetwork: true # To link the node network as the pod network automountServiceAccountToken: true containers: - env: - name: NB_SETUP_KEY valueFrom: secretKeyRef: key: setupKey name: <MY_SECRET> - name: NB_MANAGEMENT_URL value: <MY_URL> - name: NB_ADMIN_URL value: <MY_URL> - name: NB_HOSTNAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName - name: NB_LOG_LEVEL value: info image: netbirdio/netbird:0.29.3 # Or 0.66.3 name: netbird-client securityContext: allowPrivilegeEscalation: true capabilities: add: - NET_ADMIN - SYS_RESOURCE - SYS_ADMIN privileged: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst ``` ## Trouble Shooting With this implementation it's not working on versions > 0.29 So If I upgrade, all my traffic to my kubernetes cluster just shut down. ### What is working on both On both versions : * The pods starts * My peer register to my netbird and I can see them on the dashboard * I can make my netbird commands on the pods ### What is broken > 0.29 I can’t reach websites and services hosted on the kubernetes cluster (ingresses, NodePort services) from other NetBird peers. TCP connections hang indefinitely (SYN sent, no SYN-ACK received). ### What is break on both versions * When making `netbird routes list` on pods, getting `Error: failed to connect to daemon error: context deadline exceeded` * I got the netbird iptable not existing on pods : ``` $ ip rule list 0: from all lookup local 100: from all lookup main suppress_prefixlength 0 110: from all fwmark 0x1bd00 lookup 7120 32766: from all lookup main 32767: from all lookup default $ ip route show table 7120 ip: invalid argument '7120' to 'table' ``` ### What I tried and was useless * Using different additional vars : * `NB_WG_KERNEL_DISABLED = true` * `NB_DISABLE_CUSTOM_ROUTING = true` * `NB_SKIP_SOCKET_MARK = true` * Adding different other capabilities : * `SYS_MODULE` * `NET_RAW` * Using those kube flags * `hostPID: true` * `hostIPC: true` * Adding and mounting those volumes: * `/dev/net/tun` * `/var/lib/netbird` I got a try with the host network at `false` on the 0.66, but it worked like half of it. The think is that was the pod that was register in netbird and not the node. So I got problems when: * Trying to join netbirds IPs, it was not redirected to the node and trapped into the pod * Trying to use the VPN from another pod on the same cluster (can't just use the pod network without service or anything magicaly) ## Why does this version breaks everything I don't really know, I know the Biggest breaking change since 0.30 is The ACL policies. So Maybe it's the netbird ps that wants to scrap all the traffic comming and outgoing from private networks addresses to check if it's policy friendly ? But with the network on the node and the netbird payload on its own pod, it may be crashing up ? And if it's that or not I don't understand why it works in 0.29 but not on newer versions.
saavagebueno added the triage-needed label 2026-08-05 01:30:56 -04:00
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11773