[GH-ISSUE #5349] Reverse Proxy Bug: Possible permission issue within the netbird-proxy container and/or pre-shared key issue maybe? #11306

Open
opened 2026-08-05 01:29:13 -04:00 by saavagebueno · 15 comments
Owner

Originally created by @heymoe on GitHub (Feb 16, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5349

Describe the problem
I did a fresh install of 0.65.1. restored my /var/lib/netbird data to it and updated the config.yaml file with my previous keys. This process moved me from an external caddy + 4 container Netbird setup to the new default Traefik + unified netbird-server container setup. Netbird functionality and existing clients all were still up and running post fresh install + data restore. As a side note, I did update the Traefik config in the docker-compose to ues dnsChallenge instead of tlsChallenge for the LetsEncrypt certs.

Now that we have a base line where my setup is, I wanted to try out the new reverse proxy feature. I first added a Network and added a single IP host that is running a webserver. Assigned a Peer that was one the same network as this webserver host as a Routing Peer and was able to access the webserver from a remote host that otherwise would not have access to the webserver over the Netbird network. I then went to add a Reverse Proxy service. My test reverse proxy domain shows up in the drop down, which from the documentation is a good sign the netbird-proxy container is able to communicate with netbird-server. Filled in the subnet for my test: "testme". Clicked on add-target, opened the pull down to select a Peer or Resource, clicked on resource and added my webserver resource as the target. Clicked continue, didn't select any Authentication options so clicked continue again, left the 2 settings options disabled and clicked Add Service. Got the warning about not having any Auth options and clicked Add Service.

At that point it looked like the service was getting added as the status of the service was showing "issuing certificate" but that is where it stayed and never changed. If I go to edit the service I just added I noticed there is a message saying "Cluster [my proxy domain name] is offline. Make sure proxy server is running and connected to the right management address".

I checked the proxy.env file and the NB_PROXY_MANAGMENT_ADDRESS is set to the internal docker container name (http://netbird-server:80). I did try and change that to the public facing URL of my NetBird instance but after doing that my proxy domain no longer showed up in the drop down as an option when adding a reverse proxy service. So I changed it back to the internal container URL.

After putting the management URL back to what it was and restarting everything, my proxy domain shows up again as an option. I also looked at the container logs and saw this entry which I did not see when I tried using the public URL:

netbird-server     | 2026-02-16T18:11:09Z INFO [total_proxies: 1, proxy_id: netbird-proxy-20260216181107, address: myproxy.example.com, cluster_addr:myproxy.example.com] management/internals/shared/grpc/proxy.go:192: Proxy registered in cluster

OK, now to the permission issue. So after a fresh restart and not having any reverse proxy service defined, running "docker compose logs proxy" returns no logs for the proxy container. Once I add a reverse proxy service I see these error and warnings in the log:

netbird-proxy  | 2026-02-16T18:20:31Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.1834307082active_profile.json: permission denied
netbird-proxy  | 2026-02-16T18:20:31Z ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied
netbird-proxy  | 2026-02-16T18:20:31Z ERRO client/iface/configurer/usp.go:308: failed to open uapi listener: mkdir /var/run/wireguard: permission denied
netbird-proxy  | 2026-02-16T18:20:31Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace
netbird-proxy  | 2026-02-16T18:20:31Z ERRO shared/signal/client/worker.go:46: failed to handle message: wrongly addressed message <PEER_ID_FOR_REVERSE_PROXY_RESOUCE>
netbird-proxy  | 2026-02-16T18:20:35Z WARN [peer: <PEER_ID_FOR_REVERSE_PROXY_RESOUCE>] client/internal/peer/handshaker.go:133: skipping remote offer message because receiver not ready
netbird-proxy  | 2026-02-16T18:20:35Z WARN client/internal/peer/ice/agent.go:107: ICE agent close timed out after 3s, proceeding with cleanup
netbird-proxy  | 2026-02-16T18:21:01Z WARN [peer: <PEER_ID_FOR_REVERSE_PROXY_RESOUCE>] client/internal/peer/wg_watcher.go:127: WireGuard handshake timed out: 0001-01-01 00:00:00 +0000 UTC
netbird-proxy  | 2026-02-16T18:21:01Z WARN [peer: <PEER_ID_FOR_REVERSE_PROXY_RESOUCE>] client/internal/peer/conn.go:586: WireGuard handshake timeout detected, closing current connection
netbird-proxy  | 2026-02-16T18:21:01Z ERRO client/iface/wgproxy/bind/proxy.go:185: failed to read from remote conn: rels://my-public.netbird.url.com:443, use of closed network connection

I then logged into the netbird-proxy container using this command: "docker compose exec -it proxy sh". At the shell I ran a "ps auxwf" and see that the "/go/bin/netbird-proxy" process is running as the user "netbird" which is the same user I'm logged into the shell as (I ran whoami to check). I then changed into the /var/lib/netbird directory and the permission and owner ship of that directory is 755 uid: 0 gid: 0 and from the shell if I try to create a file in that directory (touch foo) I get a permission denied. The same goes for the /var/run directory too.

Now, after typing all of this up and reading all the log output above, there is one detail that popped into my head that might be causing a problem but likely not the permission problem which I guess could be normal?. All of my peers are running a preshared-key (same key on all peers) for added security. I didn't see an option anywhere to tell the proxy what that key is nor do I think that key is stored anywhere with in netbird-server / management config for the proxy to be able to pull it from. So, maybe that is an option that needs to get added to the proxy.env file?

To Reproduce

See above but let me know if you need more details.

Expected behavior

Being able to successfully add a reverse proxy service.

Are you using NetBird Cloud?

Self-Hosted

NetBird version

0.65.1

Is any other VPN software installed?

No

Debug output

Will provide if additional logging is needed

Screenshots

None at the moment

Additional context

Nothing to add at the moment.

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • [n/a] Disabled other VPN software
  • Checked firewall settings
Originally created by @heymoe on GitHub (Feb 16, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5349 **Describe the problem** I did a fresh install of 0.65.1. restored my /var/lib/netbird data to it and updated the config.yaml file with my previous keys. This process moved me from an external caddy + 4 container Netbird setup to the new default Traefik + unified netbird-server container setup. Netbird functionality and existing clients all were still up and running post fresh install + data restore. As a side note, I did update the Traefik config in the docker-compose to ues dnsChallenge instead of tlsChallenge for the LetsEncrypt certs. Now that we have a base line where my setup is, I wanted to try out the new reverse proxy feature. I first added a Network and added a single IP host that is running a webserver. Assigned a Peer that was one the same network as this webserver host as a Routing Peer and was able to access the webserver from a remote host that otherwise would not have access to the webserver over the Netbird network. I then went to add a Reverse Proxy service. My test reverse proxy domain shows up in the drop down, which from the documentation is a good sign the netbird-proxy container is able to communicate with netbird-server. Filled in the subnet for my test: "testme". Clicked on add-target, opened the pull down to select a Peer or Resource, clicked on resource and added my webserver resource as the target. Clicked continue, didn't select any Authentication options so clicked continue again, left the 2 settings options disabled and clicked Add Service. Got the warning about not having any Auth options and clicked Add Service. At that point it looked like the service was getting added as the status of the service was showing "issuing certificate" but that is where it stayed and never changed. If I go to edit the service I just added I noticed there is a message saying "Cluster [my proxy domain name] is offline. Make sure proxy server is running and connected to the right management address". I checked the proxy.env file and the NB_PROXY_MANAGMENT_ADDRESS is set to the internal docker container name (http://netbird-server:80). I did try and change that to the public facing URL of my NetBird instance but after doing that my proxy domain no longer showed up in the drop down as an option when adding a reverse proxy service. So I changed it back to the internal container URL. After putting the management URL back to what it was and restarting everything, my proxy domain shows up again as an option. I also looked at the container logs and saw this entry which I did not see when I tried using the public URL: ``` netbird-server | 2026-02-16T18:11:09Z INFO [total_proxies: 1, proxy_id: netbird-proxy-20260216181107, address: myproxy.example.com, cluster_addr:myproxy.example.com] management/internals/shared/grpc/proxy.go:192: Proxy registered in cluster ``` OK, now to the permission issue. So after a fresh restart and not having any reverse proxy service defined, running "docker compose logs proxy" returns no logs for the proxy container. Once I add a reverse proxy service I see these error and warnings in the log: ``` netbird-proxy | 2026-02-16T18:20:31Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.1834307082active_profile.json: permission denied netbird-proxy | 2026-02-16T18:20:31Z ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied netbird-proxy | 2026-02-16T18:20:31Z ERRO client/iface/configurer/usp.go:308: failed to open uapi listener: mkdir /var/run/wireguard: permission denied netbird-proxy | 2026-02-16T18:20:31Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace netbird-proxy | 2026-02-16T18:20:31Z ERRO shared/signal/client/worker.go:46: failed to handle message: wrongly addressed message <PEER_ID_FOR_REVERSE_PROXY_RESOUCE> netbird-proxy | 2026-02-16T18:20:35Z WARN [peer: <PEER_ID_FOR_REVERSE_PROXY_RESOUCE>] client/internal/peer/handshaker.go:133: skipping remote offer message because receiver not ready netbird-proxy | 2026-02-16T18:20:35Z WARN client/internal/peer/ice/agent.go:107: ICE agent close timed out after 3s, proceeding with cleanup netbird-proxy | 2026-02-16T18:21:01Z WARN [peer: <PEER_ID_FOR_REVERSE_PROXY_RESOUCE>] client/internal/peer/wg_watcher.go:127: WireGuard handshake timed out: 0001-01-01 00:00:00 +0000 UTC netbird-proxy | 2026-02-16T18:21:01Z WARN [peer: <PEER_ID_FOR_REVERSE_PROXY_RESOUCE>] client/internal/peer/conn.go:586: WireGuard handshake timeout detected, closing current connection netbird-proxy | 2026-02-16T18:21:01Z ERRO client/iface/wgproxy/bind/proxy.go:185: failed to read from remote conn: rels://my-public.netbird.url.com:443, use of closed network connection ``` I then logged into the netbird-proxy container using this command: "docker compose exec -it proxy sh". At the shell I ran a "ps auxwf" and see that the "/go/bin/netbird-proxy" process is running as the user "netbird" which is the same user I'm logged into the shell as (I ran whoami to check). I then changed into the /var/lib/netbird directory and the permission and owner ship of that directory is 755 uid: 0 gid: 0 and from the shell if I try to create a file in that directory (touch foo) I get a permission denied. The same goes for the /var/run directory too. Now, after typing all of this up and reading all the log output above, there is one detail that popped into my head that might be causing a problem but likely not the permission problem which I guess could be normal?. All of my peers are running a preshared-key (same key on all peers) for added security. I didn't see an option anywhere to tell the proxy what that key is nor do I think that key is stored anywhere with in netbird-server / management config for the proxy to be able to pull it from. So, maybe that is an option that needs to get added to the proxy.env file? **To Reproduce** See above but let me know if you need more details. **Expected behavior** Being able to successfully add a reverse proxy service. **Are you using NetBird Cloud?** Self-Hosted **NetBird version** 0.65.1 **Is any other VPN software installed?** No **Debug output** Will provide if additional logging is needed **Screenshots** None at the moment **Additional context** Nothing to add at the moment. **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [n/a] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2026-08-05 01:29:13 -04:00
Author
Owner

@xcslabs commented on GitHub (Feb 16, 2026):

I experience exactly the same issue.
Besides preshared key I've also enabled Rosenpass on all nodes.

<!-- gh-comment-id:3910559714 --> @xcslabs commented on GitHub (Feb 16, 2026): I experience exactly the same issue. Besides preshared key I've also enabled Rosenpass on all nodes.
Author
Owner

@heymoe commented on GitHub (Feb 16, 2026):

Related to my note about not being able to use the public URL of my Netbird instance for NB_PROXY_MANAGEMENT_ADDRESS in the proxy.env file. It looks like that is related to this other issue reported here (https://github.com/netbirdio/netbird/issues/5326).

I updated the traefik.http.routers.netbird-grpc rule for the netbird-server container in the docker-compose.yml from this:

- traefik.http.routers.netbird-grpc.rule=Host(`my-netbird.example.com`) && (PathPrefix(`/signalexchange.SignalExchange/`) || PathPrefix(`/management.ManagementService/`))

to this:

- traefik.http.routers.netbird-grpc.rule=Host(`my-netbird.example.com`) && (PathPrefix(`/signalexchange.SignalExchange/`) || PathPrefix(`/management.ManagementService/`) || PathPrefix(`/management.ProxyService/`))

and now the public URL works. At least from the point of view that the proxy now registers and the proxy domain shows up in the list.

<!-- gh-comment-id:3910824718 --> @heymoe commented on GitHub (Feb 16, 2026): Related to my note about not being able to use the public URL of my Netbird instance for NB_PROXY_MANAGEMENT_ADDRESS in the proxy.env file. It looks like that is related to this other issue reported here (https://github.com/netbirdio/netbird/issues/5326). I updated the traefik.http.routers.netbird-grpc rule for the netbird-server container in the docker-compose.yml from this: ``` - traefik.http.routers.netbird-grpc.rule=Host(`my-netbird.example.com`) && (PathPrefix(`/signalexchange.SignalExchange/`) || PathPrefix(`/management.ManagementService/`)) ``` to this: ``` - traefik.http.routers.netbird-grpc.rule=Host(`my-netbird.example.com`) && (PathPrefix(`/signalexchange.SignalExchange/`) || PathPrefix(`/management.ManagementService/`) || PathPrefix(`/management.ProxyService/`)) ``` and now the public URL works. At least from the point of view that the proxy now registers and the proxy domain shows up in the list.
Author
Owner

@pascal-fischer commented on GitHub (Feb 17, 2026):

Hi, the issue is the pre-shared key. The proxy internally uses an embedded netbird client to connect to your peers. Currently there is no support for the pre-shared key (this might be coming in the next couple of weeks).
Regarding the permission issues they are not necessarily causing issues. The main issue is
WireGuard handshake timed out which is due to the missing pre-shared key support.
On that note, rosenpass is also not supported yet.

<!-- gh-comment-id:3913616428 --> @pascal-fischer commented on GitHub (Feb 17, 2026): Hi, the issue is the pre-shared key. The proxy internally uses an embedded netbird client to connect to your peers. Currently there is no support for the pre-shared key (this might be coming in the next couple of weeks). Regarding the permission issues they are not necessarily causing issues. The main issue is `WireGuard handshake timed out` which is due to the missing pre-shared key support. On that note, rosenpass is also not supported yet.
Author
Owner

@pascal-fischer commented on GitHub (Feb 17, 2026):

Actually I just had a look and for selfhosted proxies it would be fairly easy to add pre-shared key support. So I assume we could manage to add this this week already.

<!-- gh-comment-id:3913657738 --> @pascal-fischer commented on GitHub (Feb 17, 2026): Actually I just had a look and for selfhosted proxies it would be fairly easy to add pre-shared key support. So I assume we could manage to add this this week already.
Author
Owner

@rakauchuk commented on GitHub (Feb 19, 2026):

Management
v0.65.2
Dashboard
v2.32.4

The same issue.

$ docker compose logs -f proxy
netbird-proxy  | 2026-02-19T11:46:52Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.3412122469active_profile.json: permission denied
netbird-proxy  | 2026/02/19 11:46:52 failed to sufficiently increase send buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
netbird-proxy  | 2026-02-19T11:46:52Z ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied
netbird-proxy  | 2026-02-19T11:46:52Z ERRO client/iface/configurer/usp.go:308: failed to open uapi listener: mkdir /var/run/wireguard: permission denied
netbird-proxy  | 2026-02-19T11:46:52Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace
netbird-proxy  | 2026-02-19T11:47:02Z WARN shared/management/client/grpc.go:157: exiting the Management service connection retry loop due to the unrecoverable error: context canceled
netbird-proxy  | 2026-02-19T11:47:02Z WARN [relay: rels://netbird.example.com:443] shared/relay/client/client.go:638: relay connection was already marked as not running
netbird-proxy  | 2026-02-19T11:47:05Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.1797134049active_profile.json: permission denied
netbird-proxy  | 2026-02-19T11:47:05Z ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied
netbird-proxy  | 2026-02-19T11:47:05Z ERRO client/iface/configurer/usp.go:308: failed to open uapi listener: mkdir /var/run/wireguard: permission denied
netbird-proxy  | 2026-02-19T11:47:05Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace
netbird-proxy  | 2026-02-19T11:47:11Z WARN shared/management/client/grpc.go:157: exiting the Management service connection retry loop due to the unrecoverable error: context canceled
netbird-proxy  | 2026-02-19T11:47:11Z WARN [relay: rels://netbird.example.com:443] shared/relay/client/client.go:638: relay connection was already marked as not running
netbird-proxy  | 2026-02-19T11:47:12Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.2184454836active_profile.json: permission denied
netbird-proxy  | 2026-02-19T11:47:12Z ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied
netbird-proxy  | 2026-02-19T11:47:12Z ERRO client/iface/configurer/usp.go:308: failed to open uapi listener: mkdir /var/run/wireguard: permission denied
netbird-proxy  | 2026-02-19T11:47:12Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace
netbird-proxy  | 2026-02-19T11:47:13Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-19T11:47:33Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-19T11:48:00Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-19T11:48:31Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-19T11:49:04Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-19T11:49:37Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet
$ docker compose exec -it proxy sh
/ $ id
uid=1000(netbird) gid=1000(netbird) groups=1000(netbird)
/ $ ls -la /var/lib/
dpkg/     misc/     netbird/
/ $ ls -la /var/lib/netbird/
total 8
drwxr-xr-x    2 0        0             4096 Feb 14 19:55 .
drwxr-xr-x    1 0        0             4096 Feb 14 19:55 ..
/ $ ls -la /var/lib/
total 20
drwxr-xr-x    1 0        0             4096 Feb 14 19:55 .
drwxr-xr-x    1 0        0             4096 Oct 29  2024 ..
drwxr-xr-x    1 0        0             4096 Oct 29  2024 dpkg
drwxr-xr-x    2 0        0             4096 Jan  2 12:35 misc
drwxr-xr-x    2 0        0             4096 Feb 14 19:55 netbird
<!-- gh-comment-id:3927322393 --> @rakauchuk commented on GitHub (Feb 19, 2026): Management v0.65.2 Dashboard v2.32.4 The same issue. ``` $ docker compose logs -f proxy netbird-proxy | 2026-02-19T11:46:52Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.3412122469active_profile.json: permission denied netbird-proxy | 2026/02/19 11:46:52 failed to sufficiently increase send buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details. netbird-proxy | 2026-02-19T11:46:52Z ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied netbird-proxy | 2026-02-19T11:46:52Z ERRO client/iface/configurer/usp.go:308: failed to open uapi listener: mkdir /var/run/wireguard: permission denied netbird-proxy | 2026-02-19T11:46:52Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace netbird-proxy | 2026-02-19T11:47:02Z WARN shared/management/client/grpc.go:157: exiting the Management service connection retry loop due to the unrecoverable error: context canceled netbird-proxy | 2026-02-19T11:47:02Z WARN [relay: rels://netbird.example.com:443] shared/relay/client/client.go:638: relay connection was already marked as not running netbird-proxy | 2026-02-19T11:47:05Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.1797134049active_profile.json: permission denied netbird-proxy | 2026-02-19T11:47:05Z ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied netbird-proxy | 2026-02-19T11:47:05Z ERRO client/iface/configurer/usp.go:308: failed to open uapi listener: mkdir /var/run/wireguard: permission denied netbird-proxy | 2026-02-19T11:47:05Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace netbird-proxy | 2026-02-19T11:47:11Z WARN shared/management/client/grpc.go:157: exiting the Management service connection retry loop due to the unrecoverable error: context canceled netbird-proxy | 2026-02-19T11:47:11Z WARN [relay: rels://netbird.example.com:443] shared/relay/client/client.go:638: relay connection was already marked as not running netbird-proxy | 2026-02-19T11:47:12Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.2184454836active_profile.json: permission denied netbird-proxy | 2026-02-19T11:47:12Z ERRO client/iface/configurer/uapi.go:15: failed to open uapi socket: mkdir /var/run/wireguard: permission denied netbird-proxy | 2026-02-19T11:47:12Z ERRO client/iface/configurer/usp.go:308: failed to open uapi listener: mkdir /var/run/wireguard: permission denied netbird-proxy | 2026-02-19T11:47:12Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace netbird-proxy | 2026-02-19T11:47:13Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet netbird-proxy | 2026-02-19T11:47:33Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet netbird-proxy | 2026-02-19T11:48:00Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet netbird-proxy | 2026-02-19T11:48:31Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet netbird-proxy | 2026-02-19T11:49:04Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet netbird-proxy | 2026-02-19T11:49:37Z WARN [peer: JvA1R9BsZ0Zmsa92HXOjECAbOcY9JDZ4Gd+4ojawIQw=] client/internal/peer/worker_ice.go:158: ICE Agent is not initialized yet ``` ``` $ docker compose exec -it proxy sh / $ id uid=1000(netbird) gid=1000(netbird) groups=1000(netbird) / $ ls -la /var/lib/ dpkg/ misc/ netbird/ / $ ls -la /var/lib/netbird/ total 8 drwxr-xr-x 2 0 0 4096 Feb 14 19:55 . drwxr-xr-x 1 0 0 4096 Feb 14 19:55 .. / $ ls -la /var/lib/ total 20 drwxr-xr-x 1 0 0 4096 Feb 14 19:55 . drwxr-xr-x 1 0 0 4096 Oct 29 2024 .. drwxr-xr-x 1 0 0 4096 Oct 29 2024 dpkg drwxr-xr-x 2 0 0 4096 Jan 2 12:35 misc drwxr-xr-x 2 0 0 4096 Feb 14 19:55 netbird ```
Author
Owner

@da-wilky commented on GitHub (Feb 19, 2026):

Is there a plan or a timeline to support the proxy while Rosenpass is enabled? Could we expect that feature soon?

<!-- gh-comment-id:3927556632 --> @da-wilky commented on GitHub (Feb 19, 2026): Is there a plan or a timeline to support the proxy while Rosenpass is enabled? Could we expect that feature soon?
Author
Owner

@t40mas commented on GitHub (Feb 23, 2026):

FYI: #5377

<!-- gh-comment-id:3945193636 --> @t40mas commented on GitHub (Feb 23, 2026): FYI: #5377
Author
Owner

@heymoe commented on GitHub (Feb 24, 2026):

@pascal-fischer : Does the pre-shared key (https://github.com/netbirdio/netbird/pull/5377) merge also make the proxy rosenpass aware too?

<!-- gh-comment-id:3952770723 --> @heymoe commented on GitHub (Feb 24, 2026): @pascal-fischer : Does the pre-shared key (https://github.com/netbirdio/netbird/pull/5377) merge also make the proxy rosenpass aware too?
Author
Owner

@sandroshu commented on GitHub (Feb 24, 2026):

I don't have a PSK but I still get permission issues and stuck at issuing cert indefinitely...

netbird-proxy  | 2026-02-24T17:57:26.935Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.2331260097active_profile.json: permission denied
netbird-proxy  | 2026/02/24 17:57:27 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
netbird-proxy  | 2026-02-24T17:57:27.311Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace
netbird-proxy  | 2026-02-24T17:57:27.523Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-24T17:58:42.421Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-24T17:59:51.505Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-24T17:59:51.506Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-24T17:59:51.507Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet
netbird-proxy  | 2026-02-24T17:59:51.507Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet

Can not access the proxy domain due to TLS handshake problem, probably because still no SSL configured.

Update: I spent 3 hours trying to get this working, giving up now. Deleted all proxies, tried to remove a peer and now I get: Code 412: Peer d6esmahvq3jc73ckd3sg is in use by proxy d6etufhvq3jc73d1hb90
No proxy configured...
Probably related: https://github.com/netbirdio/netbird/issues/5421

<!-- gh-comment-id:3953790979 --> @sandroshu commented on GitHub (Feb 24, 2026): I don't have a PSK but I still get permission issues and stuck at issuing cert indefinitely... ``` netbird-proxy | 2026-02-24T17:57:26.935Z WARN client/internal/profilemanager/service.go:360: failed to get active profile state: failed to set default active profile state: failed to set active profile to default: failed to write active profile state: create temp: open /var/lib/netbird/.2331260097active_profile.json: permission denied netbird-proxy | 2026/02/24 17:57:27 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details. netbird-proxy | 2026-02-24T17:57:27.311Z WARN client/firewall/create_linux.go:49: failed to create native firewall: create firewall: no firewall manager found. Proceeding with userspace netbird-proxy | 2026-02-24T17:57:27.523Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet netbird-proxy | 2026-02-24T17:58:42.421Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet netbird-proxy | 2026-02-24T17:59:51.505Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet netbird-proxy | 2026-02-24T17:59:51.506Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet netbird-proxy | 2026-02-24T17:59:51.507Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet netbird-proxy | 2026-02-24T17:59:51.507Z WARN [peer: h77wY0/KcLbz9vTVYwRZbfKkip9cgD6D+0orLRM1LWw=] client/internal/peer/worker_ice.go:160: ICE Agent is not initialized yet ``` Can not access the proxy domain due to TLS handshake problem, probably because still no SSL configured. Update: I spent 3 hours trying to get this working, giving up now. Deleted all proxies, tried to remove a peer and now I get: Code 412: Peer d6esmahvq3jc73ckd3sg is in use by proxy d6etufhvq3jc73d1hb90 No proxy configured... Probably related: https://github.com/netbirdio/netbird/issues/5421
Author
Owner

@heymoe commented on GitHub (Feb 24, 2026):

@sandroshu : I don't think the permission denied message in your logs (line 1) is causing any harm. Since they added the pre-shared key supported in 0.66.0, I still get the same permission denied but it looks like the proxy is now able to make a connection to the proxy service peer just fine and I no longer have the other logs about not being able to do the handshake or what not.

That being said, I'm still not able to add a reverse proxy service successfully. When I try, the new service now, it gets stuck at "Issuing Certificate". I now seem to be running into a Let's Encrypt / ACME problem where it is unable to do the verification challenge thus no valid SSL cert gets generated and just hangs. Because proxy service never fully initialize, Netbird currently isn't cleaning up the target entries it adds to the database thus not being able to remove a peer (your reference to the issue #5421). Hopefully that will get sorted out in a future update but that won't solve the Let's Encrypt / ACME issue I'm having now.

I'm working on opening another issue to see about adding support for ACME DNS-01 challenges to the proxy container. For some reason the "tls-alpn-01" or the "http-01" challenge methods are working for deployment. Even for the Traefik container BUT I was able to update the Traefik container to use a DNS challenge method to get certs issued to it but the proxy container seems to use its own ACME process to get certs.

This the the current error I'm getting related to the unsuccessful cert request:

netbird-proxy  | 2026-02-24T20:38:33.810Z WARN proxy/internal/acme/manager.go:139: prefetch certificate for domain "test3.proxy.example.com": acme/autocert: unable to satisfy "https://acme-v02.api.letsencrypt.org/acme/authz/XXXXXXX" for domain "test3.proxy.example.com": no viable challenge type found
<!-- gh-comment-id:3956663601 --> @heymoe commented on GitHub (Feb 24, 2026): @sandroshu : I don't think the permission denied message in your logs (line 1) is causing any harm. Since they added the pre-shared key supported in 0.66.0, I still get the same permission denied but it looks like the proxy is now able to make a connection to the proxy service peer just fine and I no longer have the other logs about not being able to do the handshake or what not. That being said, I'm still not able to add a reverse proxy service successfully. When I try, the new service now, it gets stuck at "Issuing Certificate". I now seem to be running into a Let's Encrypt / ACME problem where it is unable to do the verification challenge thus no valid SSL cert gets generated and just hangs. Because proxy service never fully initialize, Netbird currently isn't cleaning up the target entries it adds to the database thus not being able to remove a peer (your reference to the issue #5421). Hopefully that will get sorted out in a future update but that won't solve the Let's Encrypt / ACME issue I'm having now. I'm working on opening another issue to see about adding support for ACME DNS-01 challenges to the proxy container. For some reason the "tls-alpn-01" or the "http-01" challenge methods are working for deployment. Even for the Traefik container BUT I was able to update the Traefik container to use a DNS challenge method to get certs issued to it but the proxy container seems to use its own ACME process to get certs. This the the current error I'm getting related to the unsuccessful cert request: ``` netbird-proxy | 2026-02-24T20:38:33.810Z WARN proxy/internal/acme/manager.go:139: prefetch certificate for domain "test3.proxy.example.com": acme/autocert: unable to satisfy "https://acme-v02.api.letsencrypt.org/acme/authz/XXXXXXX" for domain "test3.proxy.example.com": no viable challenge type found ```
Author
Owner

@sandroshu commented on GitHub (Feb 25, 2026):

This the the current error I'm getting related to the unsuccessful cert request:

netbird-proxy  | 2026-02-24T20:38:33.810Z WARN proxy/internal/acme/manager.go:139: prefetch certificate for domain "test3.prox

I got the same result yesterday after a lot of retries. I followed everything in the docs to add/re-add the proxy.
Then I did a complete reset and using the install script I 'reinstalled' it. Turns out there are some more details in the proxy.env than in the docs.. or I just missed reading them.
Probably adding the last 3 would've solved my cert issue. If you don't have these in the proxy.env, try adding them.

Mine looks like this after a new install:

NB_PROXY_DEBUG_LOGS=false
NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80
NB_PROXY_ALLOW_INSECURE=true
NB_PROXY_DOMAIN=xxx.hu
NB_PROXY_ADDRESS=:8443
NB_PROXY_TOKEN=xxx
NB_PROXY_CERTIFICATE_DIRECTORY=/certs
NB_PROXY_ACME_CERTIFICATES=true
NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01
NB_PROXY_FORWARDED_PROTO=https
NB_PROXY_PROXY_PROTOCOL=true
NB_PROXY_TRUSTED_PROXIES=172.30.0.10

Before that:

NB_PROXY_DEBUG_LOGS=true
NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80
NB_PROXY_ALLOW_INSECURE=true
NB_PROXY_DOMAIN=xxx.hu
NB_PROXY_ADDRESS=:8443
NB_PROXY_TOKEN=xxx
NB_PROXY_CERTIFICATE_DIRECTORY=/certs
NB_PROXY_ACME_CERTIFICATES=true
NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01
<!-- gh-comment-id:3959927980 --> @sandroshu commented on GitHub (Feb 25, 2026): > This the the current error I'm getting related to the unsuccessful cert request: > > ``` > netbird-proxy | 2026-02-24T20:38:33.810Z WARN proxy/internal/acme/manager.go:139: prefetch certificate for domain "test3.prox > ``` I got the same result yesterday after a lot of retries. I followed everything in the docs to add/re-add the proxy. Then I did a complete reset and using the install script I 'reinstalled' it. Turns out there are some more details in the proxy.env than in the docs.. or I just missed reading them. Probably adding the last 3 would've solved my cert issue. If you don't have these in the proxy.env, try adding them. Mine looks like this after a new install: ``` NB_PROXY_DEBUG_LOGS=false NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80 NB_PROXY_ALLOW_INSECURE=true NB_PROXY_DOMAIN=xxx.hu NB_PROXY_ADDRESS=:8443 NB_PROXY_TOKEN=xxx NB_PROXY_CERTIFICATE_DIRECTORY=/certs NB_PROXY_ACME_CERTIFICATES=true NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01 NB_PROXY_FORWARDED_PROTO=https NB_PROXY_PROXY_PROTOCOL=true NB_PROXY_TRUSTED_PROXIES=172.30.0.10 ``` Before that: ``` NB_PROXY_DEBUG_LOGS=true NB_PROXY_MANAGEMENT_ADDRESS=http://netbird-server:80 NB_PROXY_ALLOW_INSECURE=true NB_PROXY_DOMAIN=xxx.hu NB_PROXY_ADDRESS=:8443 NB_PROXY_TOKEN=xxx NB_PROXY_CERTIFICATE_DIRECTORY=/certs NB_PROXY_ACME_CERTIFICATES=true NB_PROXY_ACME_CHALLENGE_TYPE=tls-alpn-01 ```
Author
Owner

@heymoe commented on GitHub (Feb 25, 2026):

I've noticed the same thing by doing a fresh install and then doing a diff on all the files (*.env, docker-compose.yml, etc..) to my current config to keep my deployment up to date with the current way they are doing things.

SSL Certs wise, I finally figured out why I couldn't get ACME to issue a cert using the tls-alpn-01 challenge. Found a write up on how the challenge works which basically consists of them checking the verification response from my host from 5 different locations on the Internet. 1 from the US and 4 from outside the US. The US site is required and I think it's either 2 or all 4 of the non-US sites have to also verify successfully otherwise the cert is not issued. It turns out I had a geoIP block on my firewall that was preventing the 4 non-US sites from connecting. Added those to a white list and the certs are now getting issued no problem.

<!-- gh-comment-id:3960555969 --> @heymoe commented on GitHub (Feb 25, 2026): I've noticed the same thing by doing a fresh install and then doing a diff on all the files (*.env, docker-compose.yml, etc..) to my current config to keep my deployment up to date with the current way they are doing things. SSL Certs wise, I finally figured out why I couldn't get ACME to issue a cert using the tls-alpn-01 challenge. Found a write up on how the challenge works which basically consists of them checking the verification response from my host from 5 different locations on the Internet. 1 from the US and 4 from outside the US. The US site is required and I think it's either 2 or all 4 of the non-US sites have to also verify successfully otherwise the cert is not issued. It turns out I had a geoIP block on my firewall that was preventing the 4 non-US sites from connecting. Added those to a white list and the certs are now getting issued no problem.
Author
Owner

@Dalewn commented on GitHub (Feb 26, 2026):

I can't get certs issued although I have no geoblocking enabled that I can think of.
Also tried switching to DNS challenge, but that didn't help either:


      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.propagation.delayBeforeChecks=5"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=8.8.8.8:53"
<!-- gh-comment-id:3965872665 --> @Dalewn commented on GitHub (Feb 26, 2026): I can't get certs issued although I have no geoblocking enabled that I can think of. Also tried switching to DNS challenge, but that didn't help either: ``` - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare" - "--certificatesresolvers.letsencrypt.acme.dnschallenge.propagation.delayBeforeChecks=5" - "--certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53" - "--certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=8.8.8.8:53"
Author
Owner

@heymoe commented on GitHub (Feb 26, 2026):

@Dalewn Checkout this feature request I put in (https://github.com/netbirdio/netbird/issues/5450), It has how I changed the Traefik container's docker-compose.yml config to enable cloudflare DNS challenge. The feature request is to add DNS challenge to the proxy container so you won't be able to use the new reverse proxy stuff yet though but will get the main Netbird functionality working.

<!-- gh-comment-id:3966730642 --> @heymoe commented on GitHub (Feb 26, 2026): @Dalewn Checkout this feature request I put in (https://github.com/netbirdio/netbird/issues/5450), It has how I changed the Traefik container's docker-compose.yml config to enable cloudflare DNS challenge. The feature request is to add DNS challenge to the proxy container so you won't be able to use the new reverse proxy stuff yet though but will get the main Netbird functionality working.
Author
Owner

@henrikekblad commented on GitHub (Mar 2, 2026):

I've finally found a fix for my problems issuing certificates and maybe this will help someone else struggeling to get it working. I saw exactly the same errors as the OP.

Before adding reverse proxy support I ran the migrate.sh script to use the unified netbird-server container. After going through the reverse-proxy setup according to the documentation I couldn't get past the spinning "Issuing certificate". Logs only showd the errors previously shown here.

So I did a fresh install on my laptop using getting-started script and compared the output. Noticed it produced a traefik-dynamic.yaml file not created when doing the migration and some new things on the traefic service.

When adding this to my docker compose it all started working!

Create new file traefik-dynamic.yaml

tcp:
  serversTransports:
    pp-v2:
      proxyProtocol:
        version: 2

docker-compose.yml (added lines)

services:
  # Traefik reverse proxy (automatic TLS via Let's Encrypt)
  traefik:
    command:
      - "--providers.file.filename=/etc/traefik/dynamic.yaml"
    volumes:
      - ./traefik-dynamic.yaml:/etc/traefik/dynamic.yaml:ro

<!-- gh-comment-id:3983395120 --> @henrikekblad commented on GitHub (Mar 2, 2026): I've finally found a fix for my problems issuing certificates and maybe this will help someone else struggeling to get it working. I saw exactly the same errors as the OP. Before adding reverse proxy support I ran the migrate.sh script to use the unified netbird-server container. After going through the reverse-proxy setup according to the documentation I couldn't get past the spinning "Issuing certificate". Logs only showd the errors previously shown here. So I did a fresh install on my laptop using getting-started script and compared the output. Noticed it produced a traefik-dynamic.yaml file not created when doing the migration and some new things on the traefic service. When adding this to my docker compose it all started working! Create new file traefik-dynamic.yaml ``` tcp: serversTransports: pp-v2: proxyProtocol: version: 2 ``` docker-compose.yml (added lines) ``` services: # Traefik reverse proxy (automatic TLS via Let's Encrypt) traefik: command: - "--providers.file.filename=/etc/traefik/dynamic.yaml" volumes: - ./traefik-dynamic.yaml:/etc/traefik/dynamic.yaml:ro ```
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11306