SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-05-17 13:42:11 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

User space wireguard not working? #1378

New Issue
Closed
opened 2025-11-20 05:29:16 -05:00 by saavagebueno · 30 comments
saavagebueno commented 2025-11-20 05:29:16 -05:00
Owner
Copy Link

Originally created by @Silex on GitHub (Oct 28, 2024).

First of all, thanks a lot for this awesome software! We are in the process of migrating everything to netbird and so far the experience is great.

I already succeed in running netbird on Teltonika routers, and now I'm trying to run netbird 0.28.4 on AXIS cameras. The goal is to be able to connect to the cameras (HTTP/RTSP) from other peers.

The problem is that the user running the application (ACAP) is very limited, but someone managed to run tailscale on them (https://github.com/Mo3he/Axis_Cam_Tailscale), so I'm pretty sure it'd be feasible to run netbird too.

Apparently they do it with --tun=userspace-networking when using tailscale (https://github.com/Mo3he/Axis_Cam_Tailscale/blob/main/aarch64/app/Tailscale_VPN).

So, I made a script like this:

#!/bin/sh

PACKAGE_DIR='/usr/local/packages/netbird'
TMP_DIR="$PACKAGE_DIR/tmp"

ARGS="--config $TMP_DIR/config.json --log-file $TMP_DIR/client.log --daemon-addr unix://$TMP_DIR/netbird.sock"
NETBIRD="$PACKAGE_DIR/lib/netbird $ARGS"

export NB_WG_KERNEL_DISABLED=true
# export NB_USE_NETSTACK_MODE=true
# export NB_SOCKS5_LISTENER_PORT=30000

echo '----------------------------------------'
echo 'Starting netbird'
mkdir -p $TMP_DIR
$NETBIRD service run &

echo 'Waiting 5 seconds'
sleep 5

echo 'Joining network'
$NETBIRD up --management-url https://netbird.foo.com --setup-key 123123123

echo 'Logs'
tail -f $TMP_DIR/client.log

This script runs fine as root (I can access my cameras), but when I run it as the application user (which will is the non-negociable default in latest firmares) then it outputs this:

2024-10-28T15:59:04.396+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Starting netbird
2024-10-28T15:59:04.398+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Waiting 5 seconds
2024-10-28T15:59:09.397+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Joining network
2024-10-28T15:59:12.398+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14493]: Connected
2024-10-28T15:59:12.406+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14426]: Waiting 5 seconds
2024-10-28T15:59:17.476+01:00 axis-accc8ede87b2 [ INFO    ] netbird[14863]: Error: status failed: create wg interface: operation not permitted

And while the peer shows up in the netbird ui, of course I cannot connect to it. So next if I uncomment the line with NB_USE_NETSTACK_MODE, then it connects fine and I can even ping the camera, but I cannot remotely access it.

With NB_USE_NETSTACK_MODE here is what it status prints:

stvs@axis-b8a44fb94617:/tmp/netbird/lib$ ./netbird $ARGS status
OS: linux/arm64
Daemon version: 0.28.4
CLI version: 0.28.4
Management: Connected
Signal: Connected
Relays: 2/2 Available
Nameservers: 0/0 Available
FQDN: (retracted)
NetBird IP: 100.82.41.54/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 5/9 Connected

Here's what the client.log show:

2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/cmd/service_controller.go:64: started daemon server: /usr/local/packages/netbird/tmp/netbird.sock
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:257: generated new SSH key
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:273: using default Wireguard port 51820
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:284: using default Wireguard interface wt0
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:337: filling in interface blacklist with defaults: [ wt0 wt utun tun0 zt ZeroTier wg ts Tailscale tailscale docker veth br- lo ]
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:383: using default DNS route interval 1m0s
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:02+01:00 INFO client/internal/config.go:216: new Management URL provided, updated to "https://netbird.foo.com" (old value "https://api.netbird.io:443")
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:03+01:00 WARN client/server/server.go:259: failed login: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:05+01:00 INFO client/internal/connect.go:119: starting NetBird client version 0.28.4 on linux/arm
2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:05+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 ERRO iface/uapi.go:15: failed to open uapi socket: remove /var/run/wireguard/wt0.sock: no such file or directory
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 ERRO iface/wg_configurer_usp.go:191: failed to open uapi listener: remove /var/run/wireguard/wt0.sock: no such file or directory
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/firewall/create_linux.go:58: no firewall manager found, trying to use userspace packet filtering firewall
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/connect.go:267: Netbird engine started, the IP is: 100.82.116.219/16
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/dns/host_unix.go:68: System DNS manager discovered: file
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/engine.go:1479: Network monitor is disabled, not starting
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO iface/tun_netstack.go:46: create netstack tun interface
2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO signal/client/grpc.go:158: connected to the Signal Service stream
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 ERRO client/internal/dns/server.go:322: unable to configure DNS for this peer using file manager without a nameserver group with all domains configured
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 208.831µs, total rules count: 2
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 INFO management/client/grpc.go:164: connected to the Management Service stream
2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO    ] netbird[23262]: 2024-10-28T16:35:07+01:00 WARN client/internal/engine.go:587: running SSH server is not permitted

Can you clarify:

  1. Wether NB_USE_NETSTACK_MODE is actually a dead end and won't help me connect from outside to my camera.
  2. What rights "userspace wireguard" requires so I can ask AXIS about what I should do.
Originally created by @Silex on GitHub (Oct 28, 2024). First of all, thanks a lot for this awesome software! We are in the process of migrating everything to netbird and so far the experience is great. I already succeed in running netbird on Teltonika routers, and now I'm trying to run netbird 0.28.4 on AXIS cameras. The goal is to be able to connect to the cameras (HTTP/RTSP) from other peers. The problem is that the user running the application (ACAP) is very limited, but someone managed to run tailscale on them (https://github.com/Mo3he/Axis_Cam_Tailscale), so I'm pretty sure it'd be feasible to run netbird too. Apparently they do it with `--tun=userspace-networking` when using tailscale (https://github.com/Mo3he/Axis_Cam_Tailscale/blob/main/aarch64/app/Tailscale_VPN). So, I made a script like this: ``` bash #!/bin/sh PACKAGE_DIR='/usr/local/packages/netbird' TMP_DIR="$PACKAGE_DIR/tmp" ARGS="--config $TMP_DIR/config.json --log-file $TMP_DIR/client.log --daemon-addr unix://$TMP_DIR/netbird.sock" NETBIRD="$PACKAGE_DIR/lib/netbird $ARGS" export NB_WG_KERNEL_DISABLED=true # export NB_USE_NETSTACK_MODE=true # export NB_SOCKS5_LISTENER_PORT=30000 echo '----------------------------------------' echo 'Starting netbird' mkdir -p $TMP_DIR $NETBIRD service run & echo 'Waiting 5 seconds' sleep 5 echo 'Joining network' $NETBIRD up --management-url https://netbird.foo.com --setup-key 123123123 echo 'Logs' tail -f $TMP_DIR/client.log ``` This script runs fine as root (I can access my cameras), but when I run it as the application user (which will is the non-negociable default in latest firmares) then it outputs this: ``` 2024-10-28T15:59:04.396+01:00 axis-accc8ede87b2 [ INFO ] netbird[14426]: Starting netbird 2024-10-28T15:59:04.398+01:00 axis-accc8ede87b2 [ INFO ] netbird[14426]: Waiting 5 seconds 2024-10-28T15:59:09.397+01:00 axis-accc8ede87b2 [ INFO ] netbird[14426]: Joining network 2024-10-28T15:59:12.398+01:00 axis-accc8ede87b2 [ INFO ] netbird[14493]: Connected 2024-10-28T15:59:12.406+01:00 axis-accc8ede87b2 [ INFO ] netbird[14426]: Waiting 5 seconds 2024-10-28T15:59:17.476+01:00 axis-accc8ede87b2 [ INFO ] netbird[14863]: Error: status failed: create wg interface: operation not permitted ``` And while the peer shows up in the netbird ui, of course I cannot connect to it. So next if I uncomment the line with `NB_USE_NETSTACK_MODE`, then it connects fine and I can even ping the camera, but I cannot remotely access it. With `NB_USE_NETSTACK_MODE` here is what it status prints: ``` shell stvs@axis-b8a44fb94617:/tmp/netbird/lib$ ./netbird $ARGS status OS: linux/arm64 Daemon version: 0.28.4 CLI version: 0.28.4 Management: Connected Signal: Connected Relays: 2/2 Available Nameservers: 0/0 Available FQDN: (retracted) NetBird IP: 100.82.41.54/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 5/9 Connected ``` Here's what the client.log show: ``` 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/cmd/service_controller.go:64: started daemon server: /usr/local/packages/netbird/tmp/netbird.sock 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:257: generated new SSH key 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:273: using default Wireguard port 51820 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:284: using default Wireguard interface wt0 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:337: filling in interface blacklist with defaults: [ wt0 wt utun tun0 zt ZeroTier wg ts Tailscale tailscale docker veth br- lo ] 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:34:57+01:00 INFO client/internal/config.go:383: using default DNS route interval 1m0s 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:02+01:00 INFO client/internal/config.go:216: new Management URL provided, updated to "https://netbird.foo.com" (old value "https://api.netbird.io:443") 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:03+01:00 WARN client/server/server.go:259: failed login: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:05+01:00 INFO client/internal/connect.go:119: starting NetBird client version 0.28.4 on linux/arm 2024-10-28T16:35:05.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:05+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service 2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:06+01:00 ERRO iface/uapi.go:15: failed to open uapi socket: remove /var/run/wireguard/wt0.sock: no such file or directory 2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:06+01:00 ERRO iface/wg_configurer_usp.go:191: failed to open uapi listener: remove /var/run/wireguard/wt0.sock: no such file or directory 2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/firewall/create_linux.go:58: no firewall manager found, trying to use userspace packet filtering firewall 2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/connect.go:267: Netbird engine started, the IP is: 100.82.116.219/16 2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/dns/host_unix.go:68: System DNS manager discovered: file 2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO client/internal/engine.go:1479: Network monitor is disabled, not starting 2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO iface/tun_netstack.go:46: create netstack tun interface 2024-10-28T16:35:07.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:06+01:00 INFO signal/client/grpc.go:158: connected to the Signal Service stream 2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:07+01:00 ERRO client/internal/dns/server.go:322: unable to configure DNS for this peer using file manager without a nameserver group with all domains configured 2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:07+01:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 208.831µs, total rules count: 2 2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:07+01:00 INFO management/client/grpc.go:164: connected to the Management Service stream 2024-10-28T16:35:08.314+01:00 axis-accc8ede87b2 [ INFO ] netbird[23262]: 2024-10-28T16:35:07+01:00 WARN client/internal/engine.go:587: running SSH server is not permitted ``` Can you clarify: 1) Wether `NB_USE_NETSTACK_MODE` is actually a dead end and won't help me connect from outside to my camera. 2) What rights "userspace wireguard" requires so I can ask AXIS about what I should do.
saavagebueno added the triage-needed label 2025-11-20 05:29:16 -05:00
saavagebueno commented 2025-11-20 05:29:17 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Oct 28, 2024):

We currently haven't implemented any forwarding from the netstack NIC to elsewhere (host or routes), so unfortunately, this access won't work in netstack mode.

@lixmal commented on GitHub (Oct 28, 2024): We currently haven't implemented any forwarding from the netstack NIC to elsewhere (host or routes), so unfortunately, this access won't work in netstack mode.
saavagebueno commented 2025-11-20 05:29:17 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Oct 28, 2024):

Thanks for the answer. Can you maybe clarify what netstack mode is for? I think it basically means "create a socks server that allows other applications to reach other peers through it", but that's "outgoing" only correct?

@Silex commented on GitHub (Oct 28, 2024): Thanks for the answer. Can you maybe clarify what netstack mode is for? I think it basically means "create a socks server that allows other applications to reach other peers through it", but that's "outgoing" only correct?
saavagebueno commented 2025-11-20 05:29:17 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Oct 28, 2024):

The intended use is in serverless environments: https://docs.netbird.io/how-to/netbird-on-faas

I think it basically means "create a socks server that allows other applications to reach other peers through it", but that's "outgoing" only correct?

correct

@lixmal commented on GitHub (Oct 28, 2024): The intended use is in serverless environments: https://docs.netbird.io/how-to/netbird-on-faas > I think it basically means "create a socks server that allows other applications to reach other peers through it", but that's "outgoing" only correct? correct
saavagebueno commented 2025-11-20 05:29:17 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Oct 28, 2024):

Thanks. Is it planned to allow for some --tun=userspace-networking like tailscale does?

I asked AXIS about what I can do with this limited user, we'll see what they say. Maybe I could create the wg interface as root and then netbird could work under the ACAP user.

@Silex commented on GitHub (Oct 28, 2024): Thanks. Is it planned to allow for some `--tun=userspace-networking` like tailscale does? I asked AXIS about what I can do with this limited user, we'll see what they say. Maybe I could create the wg interface as root and then netbird could work under the ACAP user.
saavagebueno commented 2025-11-20 05:29:17 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Oct 29, 2024):

@lixmal I'm a developer. How easy would it be to provide --tun=userspace-networking like tailscale does? I can maybe make a PR.

It's a shame because I find netbird much better than tailscale in pretty much every aspects, except I need a way to access remote cameras 😢

@Silex commented on GitHub (Oct 29, 2024): @lixmal I'm a developer. How easy would it be to provide `--tun=userspace-networking` like tailscale does? I can maybe make a PR. It's a shame because I find netbird much better than tailscale in pretty much every aspects, except I need a way to access remote cameras 😢
saavagebueno commented 2025-11-20 05:29:17 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Oct 29, 2024):

Thanks. Is it planned to allow for some --tun=userspace-networking like tailscale does?

We currently don't have any plans for that, there's been no request so far.

@lixmal I'm a developer. How easy would it be to provide --tun=userspace-networking like tailscale does? I can maybe make a PR.

It's not trivial. You'd have to implement a forwarder for tcp and udp (and possibly others like icmp, sctp, ...) packets that arrive on the wireguard netstack interface and decide where to forward them (= dial the destination address and copy over the payload).

Starting point would be somewhere here 10480eb52f/client/iface/device/device_netstack.go (L48)
That's when we get ahold of the netstack interface.

@lixmal commented on GitHub (Oct 29, 2024): >Thanks. Is it planned to allow for some --tun=userspace-networking like tailscale does? We currently don't have any plans for that, there's been no request so far. >@lixmal I'm a developer. How easy would it be to provide --tun=userspace-networking like tailscale does? I can maybe make a PR. It's not trivial. You'd have to implement a forwarder for tcp and udp (and possibly others like icmp, sctp, ...) packets that arrive on the wireguard netstack interface and decide where to forward them (= dial the destination address and copy over the payload). Starting point would be somewhere here https://github.com/netbirdio/netbird/blob/10480eb52f305cbe235ba7c63349f7e89db42bbc/client/iface/device/device_netstack.go#L48 That's when we get ahold of the netstack interface.
saavagebueno commented 2025-11-20 05:29:18 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Oct 29, 2024):

Thanks!

I noticed the AXIS camera sort the logs so they appear out of order. Here are the correct logs:

2024-10-29T16:34:48+01:00 INFO client/cmd/service_controller.go:24: starting Netbird service
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:147: generating new config /usr/local/packages/netbird/tmp/config.json
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:209: using default Management URL https://api.netbird.io:443
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:233: using default Admin URL https://api.netbird.io:443
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:251: generated new Wireguard key
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:257: generated new SSH key
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:273: using default Wireguard port 51820
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:284: using default Wireguard interface wt0
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:337: filling in interface blacklist with defaults: [ wt0 wt utun tun0 zt ZeroTier wg ts Tailscale tailscale docker veth br- lo ]
2024-10-29T16:34:48+01:00 INFO client/internal/config.go:383: using default DNS route interval 1m0s
2024-10-29T16:34:48+01:00 INFO client/cmd/service_controller.go:64: started daemon server: /usr/local/packages/netbird/tmp/netbird.sock
2024-10-29T16:34:53+01:00 INFO client/internal/config.go:216: new Management URL provided, updated to "https://netbird.foo.com" (old value "https://api.netbird.io:443")
2024-10-29T16:34:54+01:00 WARN client/server/server.go:259: failed login: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0
2024-10-29T16:34:55+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2024-10-29T16:34:55+01:00 INFO client/internal/connect.go:119: starting NetBird client version 0.28.4 on linux/arm64
2024-10-29T16:34:56+01:00 INFO iface/tun_usp_unix.go:33: using userspace bind mode
2024-10-29T16:34:56+01:00 WARN client/internal/routemanager/manager.go:122: Failed cleaning up routing: 4 errors occurred:
        * rule with suppress prefixlen v4: remove routing rule: operation not permitted
        * rule with suppress prefixlen v6: remove routing rule: operation not permitted
        * rule v4 netbird: remove routing rule: operation not permitted
        * rule v6 netbird: remove routing rule: operation not permitted
2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/systemops/systemops_linux.go:100: Error setting up sysctl: 1 errors occurred:
        * write sysctl net.ipv4.conf.all.src_valid_mark: open /proc/sys/net/ipv4/conf/all/src_valid_mark: permission denied
2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/systemops/systemops_linux.go:108: Error cleaning up routing: 4 errors occurred:
        * rule with suppress prefixlen v4: remove routing rule: operation not permitted
        * rule with suppress prefixlen v6: remove routing rule: operation not permitted
        * rule v4 netbird: remove routing rule: operation not permitted
        * rule v6 netbird: remove routing rule: operation not permitted
2024-10-29T16:34:56+01:00 ERRO client/internal/engine.go:322: Failed to initialize route manager: setup routing: rule with suppress prefixlen v4: add routing rule: operation not permitted
2024-10-29T16:34:56+01:00 INFO iface/tun_usp_unix.go:48: create tun interface
2024-10-29T16:34:56+01:00 ERRO client/internal/engine.go:332: failed creating tunnel interface wt0: [operation not permitted]
2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/manager.go:166: Error cleaning up routing: 4 errors occurred:
        * rule with suppress prefixlen v4: remove routing rule: operation not permitted
        * rule with suppress prefixlen v6: remove routing rule: operation not permitted
        * rule v4 netbird: remove routing rule: operation not permitted
        * rule v6 netbird: remove routing rule: operation not permitted
2024-10-29T16:34:56+01:00 ERRO client/internal/connect.go:263: error while starting Netbird Connection Engine: create wg interface: operation not permitted

It looks like the Management Service connects just fine, but then the tun interface creation fails. I forwarded this to AXIS in the hope they can losen up restrictions using some ACAP flags.

@Silex commented on GitHub (Oct 29, 2024): Thanks! I noticed the AXIS camera `sort` the logs so they appear out of order. Here are the correct logs: ``` 2024-10-29T16:34:48+01:00 INFO client/cmd/service_controller.go:24: starting Netbird service 2024-10-29T16:34:48+01:00 INFO client/internal/config.go:147: generating new config /usr/local/packages/netbird/tmp/config.json 2024-10-29T16:34:48+01:00 INFO client/internal/config.go:209: using default Management URL https://api.netbird.io:443 2024-10-29T16:34:48+01:00 INFO client/internal/config.go:233: using default Admin URL https://api.netbird.io:443 2024-10-29T16:34:48+01:00 INFO client/internal/config.go:251: generated new Wireguard key 2024-10-29T16:34:48+01:00 INFO client/internal/config.go:257: generated new SSH key 2024-10-29T16:34:48+01:00 INFO client/internal/config.go:273: using default Wireguard port 51820 2024-10-29T16:34:48+01:00 INFO client/internal/config.go:284: using default Wireguard interface wt0 2024-10-29T16:34:48+01:00 INFO client/internal/config.go:337: filling in interface blacklist with defaults: [ wt0 wt utun tun0 zt ZeroTier wg ts Tailscale tailscale docker veth br- lo ] 2024-10-29T16:34:48+01:00 INFO client/internal/config.go:383: using default DNS route interval 1m0s 2024-10-29T16:34:48+01:00 INFO client/cmd/service_controller.go:64: started daemon server: /usr/local/packages/netbird/tmp/netbird.sock 2024-10-29T16:34:53+01:00 INFO client/internal/config.go:216: new Management URL provided, updated to "https://netbird.foo.com" (old value "https://api.netbird.io:443") 2024-10-29T16:34:54+01:00 WARN client/server/server.go:259: failed login: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0 2024-10-29T16:34:55+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service 2024-10-29T16:34:55+01:00 INFO client/internal/connect.go:119: starting NetBird client version 0.28.4 on linux/arm64 2024-10-29T16:34:56+01:00 INFO iface/tun_usp_unix.go:33: using userspace bind mode 2024-10-29T16:34:56+01:00 WARN client/internal/routemanager/manager.go:122: Failed cleaning up routing: 4 errors occurred: * rule with suppress prefixlen v4: remove routing rule: operation not permitted * rule with suppress prefixlen v6: remove routing rule: operation not permitted * rule v4 netbird: remove routing rule: operation not permitted * rule v6 netbird: remove routing rule: operation not permitted 2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/systemops/systemops_linux.go:100: Error setting up sysctl: 1 errors occurred: * write sysctl net.ipv4.conf.all.src_valid_mark: open /proc/sys/net/ipv4/conf/all/src_valid_mark: permission denied 2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/systemops/systemops_linux.go:108: Error cleaning up routing: 4 errors occurred: * rule with suppress prefixlen v4: remove routing rule: operation not permitted * rule with suppress prefixlen v6: remove routing rule: operation not permitted * rule v4 netbird: remove routing rule: operation not permitted * rule v6 netbird: remove routing rule: operation not permitted 2024-10-29T16:34:56+01:00 ERRO client/internal/engine.go:322: Failed to initialize route manager: setup routing: rule with suppress prefixlen v4: add routing rule: operation not permitted 2024-10-29T16:34:56+01:00 INFO iface/tun_usp_unix.go:48: create tun interface 2024-10-29T16:34:56+01:00 ERRO client/internal/engine.go:332: failed creating tunnel interface wt0: [operation not permitted] 2024-10-29T16:34:56+01:00 ERRO client/internal/routemanager/manager.go:166: Error cleaning up routing: 4 errors occurred: * rule with suppress prefixlen v4: remove routing rule: operation not permitted * rule with suppress prefixlen v6: remove routing rule: operation not permitted * rule v4 netbird: remove routing rule: operation not permitted * rule v6 netbird: remove routing rule: operation not permitted 2024-10-29T16:34:56+01:00 ERRO client/internal/connect.go:263: error while starting Netbird Connection Engine: create wg interface: operation not permitted ``` It looks like the Management Service connects just fine, but then the tun interface creation fails. I forwarded this to AXIS in the hope they can losen up restrictions using some ACAP flags.
saavagebueno commented 2025-11-20 05:29:18 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Nov 4, 2024):

@lixmal just found out that tailscale works because it uses https://github.com/WireGuard/wireguard-go

How feasible would it be for netbird to fallback to wireguard-go as a backup strategy? I assume it'd be a lot of dev.

@Silex commented on GitHub (Nov 4, 2024): @lixmal just found out that tailscale works because it uses https://github.com/WireGuard/wireguard-go How feasible would it be for netbird to fallback to wireguard-go as a backup strategy? I assume it'd be a lot of dev.
saavagebueno commented 2025-11-20 05:29:18 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Nov 4, 2024):

Interface type: Userspace

Already uses wireguard-go

@lixmal commented on GitHub (Nov 4, 2024): > Interface type: Userspace Already uses `wireguard-go`
saavagebueno commented 2025-11-20 05:29:18 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Nov 4, 2024):

Then I'm confused about why tailscale works and netbird does not (without NB_USE_NETSTACK_MODE of course).

Will investigate more and report.

@Silex commented on GitHub (Nov 4, 2024): Then I'm confused about why tailscale works and netbird does not (without NB_USE_NETSTACK_MODE of course). Will investigate more and report.
saavagebueno commented 2025-11-20 05:29:18 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Nov 5, 2024):

Ok, so the crux is not wireguard-go, tailscale uses a Userspace Network Stack which allows it to bypass the need of CAP_NET_ADMIN.

Implementing the same in netbird is likely to be a lot of work.

@Silex commented on GitHub (Nov 5, 2024): Ok, so the crux is not wireguard-go, tailscale uses a Userspace Network Stack which allows it to bypass the need of CAP_NET_ADMIN. Implementing the same in netbird is likely to be a lot of work.
saavagebueno commented 2025-11-20 05:29:18 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Jan 2, 2025):

Hey @Silex, https://github.com/netbirdio/netbird/pull/3134/ will do what you want with netstack mode

@lixmal commented on GitHub (Jan 2, 2025): Hey @Silex, https://github.com/netbirdio/netbird/pull/3134/ will do what you want with netstack mode
saavagebueno commented 2025-11-20 05:29:18 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Jan 3, 2025):

Oh my it is christmas! 🥳

I'll continue the discussion there then

@Silex commented on GitHub (Jan 3, 2025): Oh my it *is* christmas! 🥳 I'll continue the discussion there then
saavagebueno commented 2025-11-20 05:29:19 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Jan 3, 2025):

Let's keep the discussion here please. The PR is for review comments.

@lixmal: wonderful, tell me if I can help with testing. I have cameras that only allow user space and thus tailscale was the only one working, I can quickly tell you if what you did works for netbird.

Are there binaries of this branch built somewhere? If not, what's the simplest way to build them?

I'll need to crosscompile for aarch64 and armv7hf (arm32) to test on the cameras. I'm used to cross-compile c++ libs there so it should be fairly straightforward.

For the record I also have routers that use mipsel but there userspace is not needed so the normal binaries work.

You can grab binaries here on the bottom https://github.com/netbirdio/netbird/actions/runs/12590171435

@lixmal commented on GitHub (Jan 3, 2025): Let's keep the discussion here please. The PR is for review comments. >@lixmal: wonderful, tell me if I can help with testing. I have cameras that only allow user space and thus tailscale was the only one working, I can quickly tell you if what you did works for netbird. >Are there binaries of this branch built somewhere? If not, what's the simplest way to build them? >I'll need to crosscompile for aarch64 and armv7hf (arm32) to test on the cameras. I'm used to cross-compile c++ libs there so it should be fairly straightforward. >For the record I also have routers that use mipsel but there userspace is not needed so the normal binaries work. You can grab binaries here on the bottom https://github.com/netbirdio/netbird/actions/runs/12590171435
saavagebueno commented 2025-11-20 05:29:19 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Jan 3, 2025):

Will test & report 👍

@Silex commented on GitHub (Jan 3, 2025): Will test & report 👍
saavagebueno commented 2025-11-20 05:29:19 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Jan 3, 2025):

image

Ok, initial tests show same behavior (it logs but it's not online). That said I have some problems with my installation, will continue testing monday.

@Silex commented on GitHub (Jan 3, 2025): ![image](https://github.com/user-attachments/assets/5f6bacd9-508b-457c-9100-73eb449732d0) Ok, initial tests show same behavior (it logs but it's not online). That said I have some problems with my installation, will continue testing monday.
saavagebueno commented 2025-11-20 05:29:19 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Jan 3, 2025):

You can grab a more recent binary here with some changes

https://github.com/netbirdio/netbird/actions/runs/12603370959

@lixmal commented on GitHub (Jan 3, 2025): You can grab a more recent binary here with some changes https://github.com/netbirdio/netbird/actions/runs/12603370959
saavagebueno commented 2025-11-20 05:29:19 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Jan 6, 2025):

It wooooooooooooooooooooooooooooooooooooooooooorks 🥳 🚀 😄

image

The other day it was just that I forgot to use NB_USE_NETSTACK_MODE 😅

@Silex commented on GitHub (Jan 6, 2025): It wooooooooooooooooooooooooooooooooooooooooooorks 🥳 🚀 😄 ![image](https://github.com/user-attachments/assets/6ba705d6-6ff5-479e-9961-031856745bd8) The other day it was just that I forgot to use `NB_USE_NETSTACK_MODE` 😅
saavagebueno commented 2025-11-20 05:29:19 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Jan 6, 2025):

Here are some logs in case they're helpful (redacted management url, logs might be a bit out of order because of a bug on these cameras with how things are logged):

2025-01-06T09:43:58.795+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:58+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2025-01-06T09:43:58.798+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:58+01:00 INFO client/internal/connect.go:115: starting NetBird client version 0.35.2-SNAPSHOT-ed5bc8c0 on linux/arm
2025-01-06T09:43:59.735+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 WARN client/iface/netstack/env.go:20: invalid socks5 listener port, unable to convert it to int, falling back to default: 1080
2025-01-06T09:43:59.737+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/iface/wgproxy/factory_usp.go:15: WireGuard Proxy Factory will produce bind proxy
2025-01-06T09:43:59.740+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/iface/device/device_netstack.go:46: create netstack tun interface
2025-01-06T09:43:59.746+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied
2025-01-06T09:43:59.746+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 ERRO client/iface/configurer/usp.go:192: failed to open uapi listener: mkdir /var/run/wireguard: permission denied
2025-01-06T09:43:59.822+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/firewall/create_linux.go:75: no firewall manager found, trying to use userspace packet filtering firewall
2025-01-06T09:43:59.822+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 WARN client/firewall/create_linux.go:48: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace
2025-01-06T09:43:59.826+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/internal/dns/host_unix.go:54: System DNS manager discovered: file
2025-01-06T09:43:59.856+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO signal/client/grpc.go:149: connected to the Signal Service stream
2025-01-06T09:43:59.857+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/internal/engine.go:1520: Network monitor is disabled, not starting
2025-01-06T09:43:59.858+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/internal/connect.go:273: Netbird engine started, the IP is: 100.82.74.159/16
2025-01-06T09:43:59.896+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO management/client/grpc.go:155: connected to the Management Service stream
2025-01-06T09:43:59.939+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO relay/client/manager.go:220: update relay server URLs: []
2025-01-06T09:43:59.939+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 WARN client/internal/engine.go:679: running SSH server is not permitted
2025-01-06T09:43:59.947+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/internal/acl/manager.go:61: ACL rules processed in: 217.792µs, total rules count: 2
2025-01-06T09:43:59.991+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:43:59+01:00 ERRO client/internal/dns/server.go:374: unable to configure DNS for this peer using file manager without a nameserver group with all domains configured
2025-01-06T09:44:00.110+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: VRO5ubnLxKToejzG61x2Ogv5sZ4v9N5MjgA+rB1TzGE=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer
2025-01-06T09:44:00.116+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK
2025-01-06T09:44:00.223+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: EfFsOYrYKgitmB1xi7gHzNp9fvNwXSpm0fBJEFcdPlY=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer
2025-01-06T09:44:00.224+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK
2025-01-06T09:44:00.265+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: b2YqqAb7LFzD8huM4ui2oEDhQA2v3XtgsRtVJILMxSA=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer
2025-01-06T09:44:00.267+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK
2025-01-06T09:44:00.367+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: EfFsOYrYKgitmB1xi7gHzNp9fvNwXSpm0fBJEFcdPlY=] client/internal/peer/conn.go:328: set ICE to active connection
2025-01-06T09:44:00.448+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: WQfps8zd4fyopipAoVsEgLc390hZwxNYXCCeEN8ssSE=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer
2025-01-06T09:44:00.461+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK
2025-01-06T09:44:00.806+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: b2YqqAb7LFzD8huM4ui2oEDhQA2v3XtgsRtVJILMxSA=] client/internal/peer/conn.go:328: set ICE to active connection
2025-01-06T09:44:02.194+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:02+01:00 INFO [peer: VRO5ubnLxKToejzG61x2Ogv5sZ4v9N5MjgA+rB1TzGE=] client/internal/peer/conn.go:328: set ICE to active connection
2025-01-06T09:44:02.744+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:02+01:00 INFO [peer: WQfps8zd4fyopipAoVsEgLc390hZwxNYXCCeEN8ssSE=] client/internal/peer/conn.go:328: set ICE to active connection
2025-01-06T09:44:03.053+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: WQfps8zd4fyopipAoVsEgLc390hZwxNYXCCeEN8ssSE=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-01-06T09:44:03.062+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: SdzSA45grWRggCfOh43zQZaGdtj7XRP5XTblvROlXEI=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-01-06T09:44:03.077+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: VRO5ubnLxKToejzG61x2Ogv5sZ4v9N5MjgA+rB1TzGE=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-01-06T09:44:03.111+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: +GfdDspu99RjD+0qOUwlWqeTz/meUwM3VI/yxTCZ9gQ=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-01-06T09:44:03.207+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: EfFsOYrYKgitmB1xi7gHzNp9fvNwXSpm0fBJEFcdPlY=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-01-06T09:44:03.209+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: b2YqqAb7LFzD8huM4ui2oEDhQA2v3XtgsRtVJILMxSA=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-01-06T09:44:03.266+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: g1ewRT90i082iGg3r1Ksub1YCGrgopJ/tpesP6kvsXM=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2025-01-06T09:44:23.528+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:23+01:00 INFO [peer: g1ewRT90i082iGg3r1Ksub1YCGrgopJ/tpesP6kvsXM=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer
2025-01-06T09:44:23.537+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:23+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK
2025-01-06T09:44:26.054+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:26+01:00 INFO [peer: g1ewRT90i082iGg3r1Ksub1YCGrgopJ/tpesP6kvsXM=] client/internal/peer/conn.go:328: set ICE to active connection
2025-01-06T09:44:27.823+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:26+01:00 ERRO Failed to create ICMP socket for {1 100.82.74.159 0 100.82.75.109}: listen ip4:icmp 0.0.0.0: socket: operation not permitted
2025-01-06T09:44:27.823+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: 2025-01-06T09:44:27+01:00 ERRO Failed to create ICMP socket for {1 100.82.74.159 0 100.82.75.109}: listen ip4:icmp 0.0.0.0: socket: operation not permitted
2025-01-06T09:46:00.745+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: )
2025-01-06T09:46:00.745+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: turnc ERROR: 2025/01/06 09:46:00 Fail to refresh permissions: CreatePermission error response (error 400: Bad Request
2025-01-06T09:46:01.286+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: )
2025-01-06T09:46:01.286+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: turnc ERROR: 2025/01/06 09:46:01 Fail to refresh permissions: CreatePermission error response (error 400: Bad Request
2025-01-06T09:46:01.352+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: )
2025-01-06T09:46:01.352+01:00 axis-accc8ede87b2 [ INFO    ] netbird[13709]: turnc ERROR: 2025/01/06 09:46:01 Fail to refresh permissions: CreatePermission error response (error 400: Bad Request
@Silex commented on GitHub (Jan 6, 2025): Here are some logs in case they're helpful (redacted management url, logs might be a bit out of order because of a bug on these cameras with how things are logged): ``` 2025-01-06T09:43:58.795+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:58+01:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service 2025-01-06T09:43:58.798+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:58+01:00 INFO client/internal/connect.go:115: starting NetBird client version 0.35.2-SNAPSHOT-ed5bc8c0 on linux/arm 2025-01-06T09:43:59.735+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 WARN client/iface/netstack/env.go:20: invalid socks5 listener port, unable to convert it to int, falling back to default: 1080 2025-01-06T09:43:59.737+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/iface/wgproxy/factory_usp.go:15: WireGuard Proxy Factory will produce bind proxy 2025-01-06T09:43:59.740+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/iface/device/device_netstack.go:46: create netstack tun interface 2025-01-06T09:43:59.746+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied 2025-01-06T09:43:59.746+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 ERRO client/iface/configurer/usp.go:192: failed to open uapi listener: mkdir /var/run/wireguard: permission denied 2025-01-06T09:43:59.822+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/firewall/create_linux.go:75: no firewall manager found, trying to use userspace packet filtering firewall 2025-01-06T09:43:59.822+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 WARN client/firewall/create_linux.go:48: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace 2025-01-06T09:43:59.826+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/internal/dns/host_unix.go:54: System DNS manager discovered: file 2025-01-06T09:43:59.856+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO signal/client/grpc.go:149: connected to the Signal Service stream 2025-01-06T09:43:59.857+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/internal/engine.go:1520: Network monitor is disabled, not starting 2025-01-06T09:43:59.858+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/internal/connect.go:273: Netbird engine started, the IP is: 100.82.74.159/16 2025-01-06T09:43:59.896+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO management/client/grpc.go:155: connected to the Management Service stream 2025-01-06T09:43:59.939+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO relay/client/manager.go:220: update relay server URLs: [] 2025-01-06T09:43:59.939+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 WARN client/internal/engine.go:679: running SSH server is not permitted 2025-01-06T09:43:59.947+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 INFO client/internal/acl/manager.go:61: ACL rules processed in: 217.792µs, total rules count: 2 2025-01-06T09:43:59.991+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:43:59+01:00 ERRO client/internal/dns/server.go:374: unable to configure DNS for this peer using file manager without a nameserver group with all domains configured 2025-01-06T09:44:00.110+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: VRO5ubnLxKToejzG61x2Ogv5sZ4v9N5MjgA+rB1TzGE=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer 2025-01-06T09:44:00.116+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK 2025-01-06T09:44:00.223+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: EfFsOYrYKgitmB1xi7gHzNp9fvNwXSpm0fBJEFcdPlY=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer 2025-01-06T09:44:00.224+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK 2025-01-06T09:44:00.265+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: b2YqqAb7LFzD8huM4ui2oEDhQA2v3XtgsRtVJILMxSA=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer 2025-01-06T09:44:00.267+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK 2025-01-06T09:44:00.367+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: EfFsOYrYKgitmB1xi7gHzNp9fvNwXSpm0fBJEFcdPlY=] client/internal/peer/conn.go:328: set ICE to active connection 2025-01-06T09:44:00.448+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: WQfps8zd4fyopipAoVsEgLc390hZwxNYXCCeEN8ssSE=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer 2025-01-06T09:44:00.461+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK 2025-01-06T09:44:00.806+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:00+01:00 INFO [peer: b2YqqAb7LFzD8huM4ui2oEDhQA2v3XtgsRtVJILMxSA=] client/internal/peer/conn.go:328: set ICE to active connection 2025-01-06T09:44:02.194+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:02+01:00 INFO [peer: VRO5ubnLxKToejzG61x2Ogv5sZ4v9N5MjgA+rB1TzGE=] client/internal/peer/conn.go:328: set ICE to active connection 2025-01-06T09:44:02.744+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:02+01:00 INFO [peer: WQfps8zd4fyopipAoVsEgLc390hZwxNYXCCeEN8ssSE=] client/internal/peer/conn.go:328: set ICE to active connection 2025-01-06T09:44:03.053+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: WQfps8zd4fyopipAoVsEgLc390hZwxNYXCCeEN8ssSE=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-01-06T09:44:03.062+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: SdzSA45grWRggCfOh43zQZaGdtj7XRP5XTblvROlXEI=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-01-06T09:44:03.077+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: VRO5ubnLxKToejzG61x2Ogv5sZ4v9N5MjgA+rB1TzGE=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-01-06T09:44:03.111+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: +GfdDspu99RjD+0qOUwlWqeTz/meUwM3VI/yxTCZ9gQ=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-01-06T09:44:03.207+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: EfFsOYrYKgitmB1xi7gHzNp9fvNwXSpm0fBJEFcdPlY=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-01-06T09:44:03.209+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: b2YqqAb7LFzD8huM4ui2oEDhQA2v3XtgsRtVJILMxSA=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-01-06T09:44:03.266+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:03+01:00 INFO [peer: g1ewRT90i082iGg3r1Ksub1YCGrgopJ/tpesP6kvsXM=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2025-01-06T09:44:23.528+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:23+01:00 INFO [peer: g1ewRT90i082iGg3r1Ksub1YCGrgopJ/tpesP6kvsXM=] client/internal/peer/worker_relay.go:61: Relay is not supported by remote peer 2025-01-06T09:44:23.537+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:23+01:00 INFO util/net/net_linux.go:61: Custom routing is disabled, skipping SO_MARK 2025-01-06T09:44:26.054+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:26+01:00 INFO [peer: g1ewRT90i082iGg3r1Ksub1YCGrgopJ/tpesP6kvsXM=] client/internal/peer/conn.go:328: set ICE to active connection 2025-01-06T09:44:27.823+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:26+01:00 ERRO Failed to create ICMP socket for {1 100.82.74.159 0 100.82.75.109}: listen ip4:icmp 0.0.0.0: socket: operation not permitted 2025-01-06T09:44:27.823+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: 2025-01-06T09:44:27+01:00 ERRO Failed to create ICMP socket for {1 100.82.74.159 0 100.82.75.109}: listen ip4:icmp 0.0.0.0: socket: operation not permitted 2025-01-06T09:46:00.745+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: ) 2025-01-06T09:46:00.745+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: turnc ERROR: 2025/01/06 09:46:00 Fail to refresh permissions: CreatePermission error response (error 400: Bad Request 2025-01-06T09:46:01.286+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: ) 2025-01-06T09:46:01.286+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: turnc ERROR: 2025/01/06 09:46:01 Fail to refresh permissions: CreatePermission error response (error 400: Bad Request 2025-01-06T09:46:01.352+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: ) 2025-01-06T09:46:01.352+01:00 axis-accc8ede87b2 [ INFO ] netbird[13709]: turnc ERROR: 2025/01/06 09:46:01 Fail to refresh permissions: CreatePermission error response (error 400: Bad Request ```
saavagebueno commented 2025-11-20 05:29:20 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Jan 6, 2025):

Glad it works for you. There are a few of errors in the log, but these are expected and don't interfere with operations. We'll eventually silence them

@lixmal commented on GitHub (Jan 6, 2025): Glad it works for you. There are a few of errors in the log, but these are expected and don't interfere with operations. We'll eventually silence them
saavagebueno commented 2025-11-20 05:29:20 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Jan 6, 2025):

Nice. I'll confirm it also work on arm64, if you need anything from me please do tell.

Out of curiosity, when do you expect this to reach standard releases?

@Silex commented on GitHub (Jan 6, 2025): Nice. I'll confirm it also work on arm64, if you need anything from me please do tell. Out of curiosity, when do you expect this to reach standard releases?
saavagebueno commented 2025-11-20 05:29:20 -05:00
Author
Owner
Copy Link

@callum-stakater commented on GitHub (Jan 17, 2025):

does this mean we can potentially run netbird in a kubernetes pod unprivileged as a subnet router to internal k8s pod subnets?

eg without needing:

          securityContext:
            capabilities:
              add:
                - NET_ADMIN
                - SYS_RESOURCE
                - SYS_ADMIN
@callum-stakater commented on GitHub (Jan 17, 2025): does this mean we can potentially run netbird in a kubernetes pod **unprivileged** as a subnet router to internal k8s pod subnets? eg without needing: ``` securityContext: capabilities: add: - NET_ADMIN - SYS_RESOURCE - SYS_ADMIN ```
saavagebueno commented 2025-11-20 05:29:20 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Jan 17, 2025):

@Silex

Nice. I'll confirm it also work on arm64, if you need anything from me please do tell.

Nothing so far, thanks!

If you pull a newer version, you'll have to set NB_ENABLE_NETSTACK_LOCAL_FORWARDING=true env var to allow access to the machine itself.

Out of curiosity, when do you expect this to reach standard releases?

Next release probably, v0.37.0

@callum-stakater

does this mean we can potentially run netbird in a kubernetes pod unprivileged as a subnet router to internal k8s pod subnets?

Yes, indeed. Although don't expect the same performance as with capabilities.

There's also a Dockerfile & tag for that mode:

3e836db1d1/client/Dockerfile-rootless (L1-L16)

@lixmal commented on GitHub (Jan 17, 2025): @Silex > Nice. I'll confirm it also work on arm64, if you need anything from me please do tell. Nothing so far, thanks! If you pull a newer version, you'll have to set `NB_ENABLE_NETSTACK_LOCAL_FORWARDING=true` env var to allow access to the machine itself. > Out of curiosity, when do you expect this to reach standard releases? Next release probably, `v0.37.0` @callum-stakater >does this mean we can potentially run netbird in a kubernetes pod unprivileged as a subnet router to internal k8s pod subnets? Yes, indeed. Although don't expect the same performance as with capabilities. There's also a Dockerfile & tag for that mode: https://github.com/netbirdio/netbird/blob/3e836db1d1bc29d13ff8e9f5df1e7f04a0bffb17/client/Dockerfile-rootless#L1-L16
saavagebueno commented 2025-11-20 05:29:20 -05:00
Author
Owner
Copy Link

@callum-stakater commented on GitHub (Jan 17, 2025):

Nice, performance vs security/compliance is sometimes an acceptable compromise, main usecases I have in mind is just as remote access to private clusters rather than service to service

Thanks for the efforts

@callum-stakater commented on GitHub (Jan 17, 2025): Nice, performance vs security/compliance is sometimes an acceptable compromise, main usecases I have in mind is just as remote access to private clusters rather than service to service Thanks for the efforts
saavagebueno commented 2025-11-20 05:29:20 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Feb 8, 2025):

I see the PR got merged, congratulations! 🚀 🎉

Should I close this?

@Silex commented on GitHub (Feb 8, 2025): I see the PR got merged, congratulations! 🚀 🎉 Should I close this?
saavagebueno commented 2025-11-20 05:29:20 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Feb 17, 2025):

@lixmal yay it was released in 0.36.6, congratulations!

Is there anything we need to set to enable this experimental feature?

@Silex commented on GitHub (Feb 17, 2025): @lixmal yay it was released in 0.36.6, congratulations! Is there anything we need to set to enable this experimental feature?
saavagebueno commented 2025-11-20 05:29:21 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Feb 17, 2025):

The routing part is enabled once you assign any routes.

for local access (your use case):

If you pull a newer version, you'll have to set NB_ENABLE_NETSTACK_LOCAL_FORWARDING=true env var to allow access to the machine itself.

Feel free to close the issue if the release works for you.

@lixmal commented on GitHub (Feb 17, 2025): The routing part is enabled once you assign any routes. for local access (your use case): >If you pull a newer version, you'll have to set NB_ENABLE_NETSTACK_LOCAL_FORWARDING=true env var to allow access to the machine itself. Feel free to close the issue if the release works for you.
saavagebueno commented 2025-11-20 05:29:21 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Feb 26, 2025):

@lixmal: version 37.1 works fine on the camera, found this in the logs which should interest you:

ERRO client/iface/netstack/tun.go:58: failed to parse NB_ETSTACK_SKIP_PROXY: strconv.ParseBool: parsing "": invalid syntax

Looks like both a typo (missing a N) & invalid env var parsing.

@Silex commented on GitHub (Feb 26, 2025): @lixmal: version 37.1 works fine on the camera, found this in the logs which should interest you: ``` ERRO client/iface/netstack/tun.go:58: failed to parse NB_ETSTACK_SKIP_PROXY: strconv.ParseBool: parsing "": invalid syntax ``` Looks like both a typo (missing a `N`) & invalid env var parsing.
saavagebueno commented 2025-11-20 05:29:21 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Feb 28, 2025):

Ok the typo is here:

https://github.com/netbirdio/netbird/blob/main/client/iface/netstack/tun.go#L58

@Silex commented on GitHub (Feb 28, 2025): Ok the typo is here: https://github.com/netbirdio/netbird/blob/main/client/iface/netstack/tun.go#L58
saavagebueno commented 2025-11-20 05:29:21 -05:00
Author
Owner
Copy Link

@Silex commented on GitHub (Feb 28, 2025):

Alright, PR is here #3415

Given this issue is solved I'll close it. I have technical questions about 4G routers and relayed connexions, ping me if by chance you're available for technical questions.

@Silex commented on GitHub (Feb 28, 2025): Alright, PR is here #3415 Given this issue is solved I'll close it. I have technical questions about 4G routers and relayed connexions, ping me if by chance you're available for technical questions.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1378
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?