DYNR/netbird
2
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-05 00:15:26 -04:00
Code Issues 23.5k Packages Projects Releases 10 Wiki Activity

[GH-ISSUE #4813] SSH stopped working with v0.60.0 #9856

New Issue
Closed
opened 2026-08-05 01:23:44 -04:00 by saavagebueno · 14 comments
saavagebueno commented 2026-08-05 01:23:44 -04:00
Owner
Copy Link

Originally created by @Fridasbabe on GitHub (Nov 19, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/4813

Describe the problem

With the new update v0.60.0
To Reproduce
Update server and clients with reboot
Netbird down
netbird up --allow-server-ssh

A clear and concise description of what you expected to happen.

Are you using NetBird Cloud?

Using Self-host NetBird's control plane.

NetBird version

v0.60.0

`netbirdio/dashboard latest sha256:626c7987d1315e919ea6d38e890c1fd74822b343d48e36a8ad194ad505481397 a59b82abb946 16 hours ago 128MB

netbirdio/management latest sha256:78e02f88e8f3711b230468a2a0f6ad60015419b8da4a69f808a7009281f95d4a abdae89c5be5 16 hours ago 183MB

netbirdio/signal latest sha256:740b1b0dde24ebda6456b3b650f66300753dbb696c22ead79115b1d2e1dd826c b1fde1ae9824 16 hours ago 39.2MB

netbirdio/relay latest sha256:7c6cb67a8db441b3e19e8d33d6572754e111e056a0ce4b357a07c60522d335dd 5ee3155ca6e7 16 hours ago 44.4MB

coturn/coturn latest sha256:203b9bf51e84199f91025d411a7bcc7358d50c43a4d9c588079e282a3bbf4ba1 f44bafdab891 5 weeks ago 121MB`

Is any other VPN software installed?

No

Debug output

`"docker logs artifacts-relay-1"
2025-11-19T07:33:45Z INFO relay/server/listener/ws/listener.go:105: WS client connected from: 192.168.1.20:58158
2025-11-19T07:33:45Z INFO [peer_id: sha-GkH/GrsOiCEORZoE42lPin9jpXGKGhyh19ilv7yFNA0=] relay/server/relay.go:131: peer connected from: 192.168.1.20:58158

"docker logs artifacts-dashboard-1"
192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /peer/ssh.txt?id=d9c6z71ooluc754c07d90&user=root&port=22&_rsc=1uq4n HTTP/1.0" 200 4690 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5"
192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /peer/ssh.txt?_rsc=1yo34 HTTP/1.0" 200 4690 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5"
192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/8356-d004856c0d8d443f.js HTTP/1.0" 200 25190 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5"
192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/9829-e3f93d81728859ce.js HTTP/1.0" 200 116669 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5"
192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/app/(remote-access)/peer/ssh/page-2536b9fb9a009d81.js HTTP/1.0" 200 7869 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5"
192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/app/(remote-access)/layout-dade7d4f7a25d68d.js HTTP/1.0" 200 2254 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5"
192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/aaea2bcf-238c2184edf3cae3.js HTTP/1.0" 200 325087 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5"
192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/css/d8195771658dcda0.css HTTP/1.0" 200 3053 "https://netbird.mydomain.com/peer/ssh?id=d9c6z71ooluc754c07d90&user=root&port=22" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5"
192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /wasm_exec.js HTTP/1.0" 200 16992 "https://netbird.mydomain.com/peer/ssh?id=d9c6z71ooluc754c07d90&user=root&port=22" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5"
`

Additional context
Worked before now ssh is doesnt want to connect in GUI
I can connect through the terminal, but not login, tried setting "PermitRootLogin yes" in sshd_config without sucesss

Originally created by @Fridasbabe on GitHub (Nov 19, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/4813 **Describe the problem** With the new update v0.60.0 **To Reproduce** Update server and clients with reboot Netbird down netbird up --allow-server-ssh A clear and concise description of what you expected to happen. **Are you using NetBird Cloud?** Using Self-host NetBird's control plane. **NetBird version** `v0.60.0` `netbirdio/dashboard latest sha256:626c7987d1315e919ea6d38e890c1fd74822b343d48e36a8ad194ad505481397 a59b82abb946 16 hours ago 128MB netbirdio/management latest sha256:78e02f88e8f3711b230468a2a0f6ad60015419b8da4a69f808a7009281f95d4a abdae89c5be5 16 hours ago 183MB netbirdio/signal latest sha256:740b1b0dde24ebda6456b3b650f66300753dbb696c22ead79115b1d2e1dd826c b1fde1ae9824 16 hours ago 39.2MB netbirdio/relay latest sha256:7c6cb67a8db441b3e19e8d33d6572754e111e056a0ce4b357a07c60522d335dd 5ee3155ca6e7 16 hours ago 44.4MB coturn/coturn latest sha256:203b9bf51e84199f91025d411a7bcc7358d50c43a4d9c588079e282a3bbf4ba1 f44bafdab891 5 weeks ago 121MB` **Is any other VPN software installed?** No **Debug output** `"docker logs artifacts-relay-1" 2025-11-19T07:33:45Z INFO relay/server/listener/ws/listener.go:105: WS client connected from: 192.168.1.20:58158 2025-11-19T07:33:45Z INFO [peer_id: sha-GkH/GrsOiCEORZoE42lPin9jpXGKGhyh19ilv7yFNA0=] relay/server/relay.go:131: peer connected from: 192.168.1.20:58158 "docker logs artifacts-dashboard-1" 192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /peer/ssh.txt?id=d9c6z71ooluc754c07d90&user=root&port=22&_rsc=1uq4n HTTP/1.0" 200 4690 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5" 192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /peer/ssh.txt?_rsc=1yo34 HTTP/1.0" 200 4690 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5" 192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/8356-d004856c0d8d443f.js HTTP/1.0" 200 25190 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5" 192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/9829-e3f93d81728859ce.js HTTP/1.0" 200 116669 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5" 192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/app/(remote-access)/peer/ssh/page-2536b9fb9a009d81.js HTTP/1.0" 200 7869 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5" 192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/app/(remote-access)/layout-dade7d4f7a25d68d.js HTTP/1.0" 200 2254 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5" 192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/chunks/aaea2bcf-238c2184edf3cae3.js HTTP/1.0" 200 325087 "https://netbird.mydomain.com/peer/ssh" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5" 192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /_next/static/css/d8195771658dcda0.css HTTP/1.0" 200 3053 "https://netbird.mydomain.com/peer/ssh?id=d9c6z71ooluc754c07d90&user=root&port=22" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5" 192.168.1.20 - - [19/Nov/2025:07:35:40 +0000] "GET /wasm_exec.js HTTP/1.0" 200 16992 "https://netbird.mydomain.com/peer/ssh?id=d9c6z71ooluc754c07d90&user=root&port=22" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:145.0) Gecko/20100101 Firefox/145.0" "1.WANIP.2.3.4.5, 1.WANIP.2.3.4.5" ` **Additional context** Worked before now ssh is doesnt want to connect in GUI I can connect through the terminal, but not login, tried setting "PermitRootLogin yes" in sshd_config without sucesss
saavagebueno added the triage-needed label 2026-08-05 01:23:44 -04:00
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Nov 19, 2025):

Please take a look at upgrade instructions in the release for v0.60.0. You might be missing the --enable-ssh-root flag.
If the issue persists, please provide a debug bundle and logs from the browser console

<!-- gh-comment-id:3551663061 --> @lixmal commented on GitHub (Nov 19, 2025): Please take a look at upgrade instructions in the release for v0.60.0. You might be missing the `--enable-ssh-root` flag. If the issue persists, please provide a debug bundle and logs from the browser console
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@Fridasbabe commented on GitHub (Nov 19, 2025):

Tried it a couple of time without changes, the connections fails. I removed the ssh and enabled it both from gui and on the client.

netbird down
netbird up --allow-server-ssh --enable-ssh-root

<!-- gh-comment-id:3551817076 --> @Fridasbabe commented on GitHub (Nov 19, 2025): Tried it a couple of time without changes, the connections fails. I removed the ssh and enabled it both from gui and on the client. netbird down netbird up --allow-server-ssh --enable-ssh-root
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@cin3m commented on GitHub (Nov 19, 2025):

Do you try to use the web SSH access? I noticed while using this function that the JWT authentication does not work. To get it back working like before, I followed the documentation and ran the affected client with the --disable-ssh-auth parameter.

<!-- gh-comment-id:3552803560 --> @cin3m commented on GitHub (Nov 19, 2025): Do you try to use the web SSH access? I noticed while using this function that the JWT authentication does not work. To get it back working like before, I followed the documentation and ran the affected client with the `--disable-ssh-auth` parameter.
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@Fridasbabe commented on GitHub (Nov 19, 2025):

@cin3m That briefly worked but seems to be broken quickly after i tried it...

EDIT: Seems to happen after reboot

<!-- gh-comment-id:3552926178 --> @Fridasbabe commented on GitHub (Nov 19, 2025): @cin3m That briefly worked but seems to be broken quickly after i tried it... EDIT: Seems to happen after reboot
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@Fridasbabe commented on GitHub (Nov 19, 2025):

Updated to v0.60.1 and now it doesnt work at all

<!-- gh-comment-id:3553140908 --> @Fridasbabe commented on GitHub (Nov 19, 2025): Updated to v0.60.1 and now it doesnt work at all
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@jeditec commented on GitHub (Nov 19, 2025):

Do you try to use the web SSH access? I noticed while using this function that the JWT authentication does not work. To get it back working like before, I followed the documentation and ran the affected client with the --disable-ssh-auth parameter.

I can confirm that I had a similar issue the web console was not working,not connecting from the web just from the normal direct ssh, after using --disable-ssh-auth I am able to login without issue, I had 3 days trying now it is working, thank you very much.

The final command I used at the server i wanted to control was:

netbird up --allow-server-ssh --enable-ssh-root --disable-ssh-auth

<!-- gh-comment-id:3555265452 --> @jeditec commented on GitHub (Nov 19, 2025): > Do you try to use the web SSH access? I noticed while using this function that the JWT authentication does not work. To get it back working like before, I followed the documentation and ran the affected client with the `--disable-ssh-auth` parameter. I can confirm that I had a similar issue the web console was not working,not connecting from the web just from the normal direct ssh, after using --disable-ssh-auth I am able to login without issue, I had 3 days trying now it is working, thank you very much. The final command I used at the server i wanted to control was: netbird up --allow-server-ssh --enable-ssh-root --disable-ssh-auth
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@Fridasbabe commented on GitHub (Nov 19, 2025):

That was the exact command i was using. Which did work on v0.6.0 but failed to connect after a reboot.

On v0.6.1 it failed to connect no matter what.

Before i could reboot. Which is kind of important.

<!-- gh-comment-id:3555703411 --> @Fridasbabe commented on GitHub (Nov 19, 2025): That was the exact command i was using. Which did work on v0.6.0 but failed to connect after a reboot. On v0.6.1 it failed to connect no matter what. Before i could reboot. Which is kind of important.
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Nov 20, 2025):

It seems like there is a race condition on the browser client. We are still working on the root cause and will issue a fix afterwards.

Can you confirm if the issue happens using a regular netbird agent on your computer?

<!-- gh-comment-id:3556166784 --> @mlsmaycon commented on GitHub (Nov 20, 2025): It seems like there is a race condition on the browser client. We are still working on the root cause and will issue a fix afterwards. Can you confirm if the issue happens using a regular netbird agent on your computer?
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@mweissdigchg commented on GitHub (Nov 20, 2025):

Same here. It seems the JWT authentication process does not work. I am using Keycloak as IdP, but from the Keycloak logs I see the login process has completed. How can I increase the verbosity of netbirds JWT authentication/token process?

$ netbird ssh root@k8s-wb-1.netbird.<redacted>
SSH authentication required.
Please visit: https://auth.<redacted>.com/realms/<redacted>/device?login_hint=<redacted>%40<redacted>&user_code=BZCJ-NNOJ
Or visit: https://auth.<redacted>.com/realms/<redacted>/device?login_hint=<redacted>%40<redacted> and enter code: BZCJ-NNOJ
Waiting for authentication...
Failed to connect to root@k8s-wb-1.netbird.<redacted>:22

Troubleshooting steps:
  1. Check peer connectivity: netbird status -d
  2. Verify SSH server is enabled on the peer
  3. Ensure correct hostname/IP is used
Error: dial k8s-wb-1.netbird.<redacted>:22: request JWT token: wait for JWT token: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL
<!-- gh-comment-id:3556441016 --> @mweissdigchg commented on GitHub (Nov 20, 2025): Same here. It seems the JWT authentication process does not work. I am using Keycloak as IdP, but from the Keycloak logs I see the login process has completed. How can I increase the verbosity of netbirds JWT authentication/token process? ``` $ netbird ssh root@k8s-wb-1.netbird.<redacted> SSH authentication required. Please visit: https://auth.<redacted>.com/realms/<redacted>/device?login_hint=<redacted>%40<redacted>&user_code=BZCJ-NNOJ Or visit: https://auth.<redacted>.com/realms/<redacted>/device?login_hint=<redacted>%40<redacted> and enter code: BZCJ-NNOJ Waiting for authentication... Failed to connect to root@k8s-wb-1.netbird.<redacted>:22 Troubleshooting steps: 1. Check peer connectivity: netbird status -d 2. Verify SSH server is enabled on the peer 3. Ensure correct hostname/IP is used Error: dial k8s-wb-1.netbird.<redacted>:22: request JWT token: wait for JWT token: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL ```
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@horzadome commented on GitHub (Nov 20, 2025):

Figured out root login issue.
For me, ssh worked fine using netbird ssh command and regular user, but not using root user.
I've done --enable-ssh-root, but still got this when trying root@:

Nov 20 09:57:50 nas netbird[137549]: JWT authentication successful for user root (JWT user ID: google-oauth2|102XXXXXXXXXXXXXX41) from 100.77.140.68:41106
Nov 20 09:57:50 nas netbird[137549]: [session: root@100.77.140.68:41106-2a2bda76, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] SSH session started (JWT user: google-oauth2|102XXXXXXXXXXXXXX41)
Nov 20 09:57:50 nas netbird[137549]: [session: root@100.77.140.68:41106-2a2bda76, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] starting interactive shell: /usr/bin/login
Nov 20 09:57:50 nas netbird[137549]: [session: root@100.77.140.68:41106-2a2bda76, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] SSH session closed after 53ms

This was happening because local root login is disabled on my server by default (root password empty) so root user can't start /usr/bin/login .
I confirmed this by setting root password and sure enough - netbird ssh root@ works just fine.

I don't remember ssh internals all that well anymore, but I believe that this is happening because netbird depends on /usr/bin/login to start user's shell session, whereas regular SSH does not.
IIRC regular SSH instead has its own PAM module which doesn't touch local login at all, and instead launches user's default shell.

Update: on another server (this time debian-ish, not arch) I keep having the same not-starting-login-shell problem but with non-root user. This user does have local login privileges so I have no idea why it's failing :(

Nov 20 11:27:49 house netbird[630]: SSH connection from NetBird peer 100.77.140.68:55956 allowed
Nov 20 11:27:50 house netbird[630]: JWT authentication successful for user horza (JWT user ID: google-oauth2|102XXXXXXXXXXXXXX41) from 100.77.140.68:55956
Nov 20 11:27:50 house netbird[630]: [session: horza@100.77.140.68:55956-2a472899, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] SSH session started (JWT user: google-oauth2|102XXXXXXXXXXXXXX41)
Nov 20 11:27:50 house netbird[630]: [session: horza@100.77.140.68:55956-2a472899, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] starting interactive shell: /usr/bin/login
Nov 20 11:27:50 house netbird[630]: [session: horza@100.77.140.68:55956-2a472899, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] SSH session closed after 59ms
<!-- gh-comment-id:3557056236 --> @horzadome commented on GitHub (Nov 20, 2025): Figured out root login issue. For me, ssh worked fine using `netbird ssh` command and regular user, but not using root user. I've done --enable-ssh-root, but still got this when trying `root@`: ```Nov 20 09:57:50 nas netbird[137549]: SSH connection from NetBird peer 100.77.140.68:41106 allowed Nov 20 09:57:50 nas netbird[137549]: JWT authentication successful for user root (JWT user ID: google-oauth2|102XXXXXXXXXXXXXX41) from 100.77.140.68:41106 Nov 20 09:57:50 nas netbird[137549]: [session: root@100.77.140.68:41106-2a2bda76, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] SSH session started (JWT user: google-oauth2|102XXXXXXXXXXXXXX41) Nov 20 09:57:50 nas netbird[137549]: [session: root@100.77.140.68:41106-2a2bda76, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] starting interactive shell: /usr/bin/login Nov 20 09:57:50 nas netbird[137549]: [session: root@100.77.140.68:41106-2a2bda76, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] SSH session closed after 53ms ``` This was happening because local root login is disabled on my server by default (root password empty) so root user can't start /usr/bin/login . I confirmed this by setting root password and sure enough - `netbird ssh root@` works just fine. I don't remember ssh internals all that well anymore, but I believe that this is happening because netbird depends on /usr/bin/login to start user's shell session, whereas regular SSH does not. IIRC regular SSH instead has its own PAM module which doesn't touch local login at all, and instead launches user's default shell. Update: on another server (this time debian-ish, not arch) I keep having the same not-starting-login-shell problem but with non-root user. This user does have local login privileges so I have no idea why it's failing :( ```Nov 20 11:27:43 house netbird[630]: SSH connection from NetBird peer 100.77.140.68:55946 allowed Nov 20 11:27:49 house netbird[630]: SSH connection from NetBird peer 100.77.140.68:55956 allowed Nov 20 11:27:50 house netbird[630]: JWT authentication successful for user horza (JWT user ID: google-oauth2|102XXXXXXXXXXXXXX41) from 100.77.140.68:55956 Nov 20 11:27:50 house netbird[630]: [session: horza@100.77.140.68:55956-2a472899, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] SSH session started (JWT user: google-oauth2|102XXXXXXXXXXXXXX41) Nov 20 11:27:50 house netbird[630]: [session: horza@100.77.140.68:55956-2a472899, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] starting interactive shell: /usr/bin/login Nov 20 11:27:50 house netbird[630]: [session: horza@100.77.140.68:55956-2a472899, jwt_user: google-oauth2|102XXXXXXXXXXXXXX41] SSH session closed after 59ms ```
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@Fridasbabe commented on GitHub (Nov 20, 2025):

Unfortunately i can't proceed into debugging this issue, dashboard wont load.

https://github.com/netbirdio/netbird/issues/4778#issuecomment-3558064177

<!-- gh-comment-id:3558076823 --> @Fridasbabe commented on GitHub (Nov 20, 2025): Unfortunately i can't proceed into debugging this issue, dashboard wont load. https://github.com/netbirdio/netbird/issues/4778#issuecomment-3558064177
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@heisbrot commented on GitHub (Nov 21, 2025):

We just released dashboard version v2.22.2 which should fix the JWT authentication in the browser client SSH

<!-- gh-comment-id:3562212395 --> @heisbrot commented on GitHub (Nov 21, 2025): We just released dashboard version [v2.22.2](https://github.com/netbirdio/dashboard/releases/tag/v2.22.2) which should fix the JWT authentication in the browser client SSH
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@alexmoras commented on GitHub (Nov 22, 2025):

Also having issues with SSH on v0.60.2. Unsure if this is a related or separate issue, but I'll post here for the time being.

SSH setup as per instructions (flags added to netbird up command, SSH enabled in web interface, ACL added for port 22). The web-based SSH client never connects. Trying netbird ssh host from the CLI of a connected Fedora 43 client just returns a new line with no message. Running ssh root@host simply returns a new line saying Connection to host closed.

Running NetBird in debug mode on the server, I can see the following logs:

2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/worker_ice.go:391: ICE ConnectionState has changed to Connected
2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/worker_ice.go:259: agent dial succeeded
2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/worker_ice.go:373: selected candidate pair [local <-> remote] -> [udp4 host 10.X.X.5:56496 (resolved: 10.X.X.5:56496) candidate:FAZT5 <-> udp4 prflx 10.X.X.31:51820 related :0 (resolved: 10.X.X.31:51820) candidate:GP7sX4], peer NrgzA=
2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/worker_ice.go:294: on ICE conn is ready to use
2025-11-22T21:12:40Z INFO [peer: NrgzA=] client/internal/peer/worker_ice.go:296: connection succeeded with offer session: c9b840bb70
2025-11-22T21:12:40Z INFO [peer: NrgzA=] client/internal/peer/conn.go:339: set ICE to active connection
2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/wg_watcher.go:82: disable WireGuard watcher
2025-11-22T21:12:40Z INFO [peer: NrgzA=] client/internal/peer/conn.go:377: configure WireGuard endpoint to: 10.50.20.31:51820
2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/endpoint.go:45: configure up WireGuard as initiatr
2025-11-22T21:12:40Z DEBG client/iface/iface.go:158: updating interface wt0 peer NrgzA=, endpoint 10.X.X.31:51820, allowedIPs [100.X.X.92/32]
2025-11-22T21:12:40Z INFO [peer: NrgzA=] client/internal/peer/wg_watcher.go:119: WireGuard watcher stopped
2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/conn.go:386: redirect packets from relayed conn to WireGuard
2025-11-22T21:12:40Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42194 allowed
2025-11-22T21:12:40Z DEBG client/ssh/server/server.go:538: SSH connection failed for 100.X.X.92:42194: EOF
2025-11-22T21:12:40Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42204 allowed
2025-11-22T21:12:40Z INFO [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/session_handlers.go:35: SSH session started
2025-11-22T21:12:40Z INFO [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login
2025-11-22T21:12:40Z DEBG [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/command_execution_unix.go:252: Pty session cancelled, terminating command
2025-11-22T21:12:40Z DEBG [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/command_execution_unix.go:262: Pty command terminated after session cancellation with error: signal: killed
2025-11-22T21:12:40Z INFO [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/session_handlers.go:45: SSH session closed after 57ms

As can be seen from the logs, a PTY session is opened for 57ms and then closed. I can't work out exactly why this is happening. If I try and SSH to any of the other servers, from one another, it'll very rarely connect and every other time act like the above. After maybe 10 connection attempts over the space of five minutes, I will get one successful connection:

2025-11-22T21:53:46Z INFO [session: alex@100.X.X.92:42184-ef54558f] client/ssh/server/session_handlers.go:45: SSH session closed after 55ms
2025-11-22T21:53:47Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42192 allowed
2025-11-22T21:53:47Z DEBG client/ssh/server/server.go:538: SSH connection failed for 100.X.X.92:42192: EOF
2025-11-22T21:53:47Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42204 allowed
2025-11-22T21:53:47Z DEBG client/ssh/server/server.go:538: SSH connection failed for 100.X.X.92:42204: EOF
2025-11-22T21:53:47Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42210 allowed
2025-11-22T21:53:47Z INFO [session: alex@100.64.50.92:42210-659dfd19] client/ssh/server/session_handlers.go:35: SSH session started
2025-11-22T21:53:47Z INFO [session: alex@100.X.X.92:42210-659dfd19] client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login

Connecting from the Fedora client to the other servers returns the following, different, error:

SSH connection to NetBird server failed: SSH handshake: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain

They may be two completely separate errors but I thought it worth mentioning. For the moment, I will run without NetBird SSH since it hooks normal port 22 requests, I am unable to SSH in to any of my servers reliably.

Any suggestions would be greatly appreciated - also happy to help in trying to diagnose the above if there is anything you need me to do / logs to pull.

<!-- gh-comment-id:3567080292 --> @alexmoras commented on GitHub (Nov 22, 2025): Also having issues with SSH on v0.60.2. Unsure if this is a related or separate issue, but I'll post here for the time being. SSH setup as per instructions (flags added to `netbird up` command, SSH enabled in web interface, ACL added for port 22). The web-based SSH client never connects. Trying `netbird ssh host` from the CLI of a connected Fedora 43 client just returns a new line with no message. Running `ssh root@host` simply returns a new line saying `Connection to host closed.` Running NetBird in debug mode on the server, I can see the following logs: ``` 2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/worker_ice.go:391: ICE ConnectionState has changed to Connected 2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/worker_ice.go:259: agent dial succeeded 2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/worker_ice.go:373: selected candidate pair [local <-> remote] -> [udp4 host 10.X.X.5:56496 (resolved: 10.X.X.5:56496) candidate:FAZT5 <-> udp4 prflx 10.X.X.31:51820 related :0 (resolved: 10.X.X.31:51820) candidate:GP7sX4], peer NrgzA= 2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/worker_ice.go:294: on ICE conn is ready to use 2025-11-22T21:12:40Z INFO [peer: NrgzA=] client/internal/peer/worker_ice.go:296: connection succeeded with offer session: c9b840bb70 2025-11-22T21:12:40Z INFO [peer: NrgzA=] client/internal/peer/conn.go:339: set ICE to active connection 2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/wg_watcher.go:82: disable WireGuard watcher 2025-11-22T21:12:40Z INFO [peer: NrgzA=] client/internal/peer/conn.go:377: configure WireGuard endpoint to: 10.50.20.31:51820 2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/endpoint.go:45: configure up WireGuard as initiatr 2025-11-22T21:12:40Z DEBG client/iface/iface.go:158: updating interface wt0 peer NrgzA=, endpoint 10.X.X.31:51820, allowedIPs [100.X.X.92/32] 2025-11-22T21:12:40Z INFO [peer: NrgzA=] client/internal/peer/wg_watcher.go:119: WireGuard watcher stopped 2025-11-22T21:12:40Z DEBG [peer: NrgzA=] client/internal/peer/conn.go:386: redirect packets from relayed conn to WireGuard 2025-11-22T21:12:40Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42194 allowed 2025-11-22T21:12:40Z DEBG client/ssh/server/server.go:538: SSH connection failed for 100.X.X.92:42194: EOF 2025-11-22T21:12:40Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42204 allowed 2025-11-22T21:12:40Z INFO [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/session_handlers.go:35: SSH session started 2025-11-22T21:12:40Z INFO [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login 2025-11-22T21:12:40Z DEBG [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/command_execution_unix.go:252: Pty session cancelled, terminating command 2025-11-22T21:12:40Z DEBG [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/command_execution_unix.go:262: Pty command terminated after session cancellation with error: signal: killed 2025-11-22T21:12:40Z INFO [session: root@100.X.X.92:42204-457ef4ee] client/ssh/server/session_handlers.go:45: SSH session closed after 57ms ``` As can be seen from the logs, a PTY session is opened for 57ms and then closed. I can't work out exactly why this is happening. If I try and SSH to any of the other servers, from one another, it'll very rarely connect and every other time act like the above. After maybe 10 connection attempts over the space of five minutes, I will get one successful connection: ``` 2025-11-22T21:53:46Z INFO [session: alex@100.X.X.92:42184-ef54558f] client/ssh/server/session_handlers.go:45: SSH session closed after 55ms 2025-11-22T21:53:47Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42192 allowed 2025-11-22T21:53:47Z DEBG client/ssh/server/server.go:538: SSH connection failed for 100.X.X.92:42192: EOF 2025-11-22T21:53:47Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42204 allowed 2025-11-22T21:53:47Z DEBG client/ssh/server/server.go:538: SSH connection failed for 100.X.X.92:42204: EOF 2025-11-22T21:53:47Z INFO client/ssh/server/server.go:607: SSH connection from NetBird peer 100.X.X.92:42210 allowed 2025-11-22T21:53:47Z INFO [session: alex@100.64.50.92:42210-659dfd19] client/ssh/server/session_handlers.go:35: SSH session started 2025-11-22T21:53:47Z INFO [session: alex@100.X.X.92:42210-659dfd19] client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login ``` Connecting from the Fedora client to the other servers returns the following, different, error: ``` SSH connection to NetBird server failed: SSH handshake: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none password], no supported methods remain ``` They may be two completely separate errors but I thought it worth mentioning. For the moment, I will run without NetBird SSH since it hooks normal port 22 requests, I am unable to SSH in to any of my servers reliably. Any suggestions would be greatly appreciated - also happy to help in trying to diagnose the above if there is anything you need me to do / logs to pull.
saavagebueno commented 2026-08-05 01:23:45 -04:00
Author
Owner
Copy Link

@sgtaziz commented on GitHub (Nov 23, 2025):

For me, it seems like its overriding my issuer (authentik):

2025-11-23T09:09:17+03:00 ERRO shared/auth/jwt/validator.go:147: token could not be parsed: token has invalid claims: token has invalid issuer
2025-11-23T09:09:17+03:00 WARN client/ssh/server/server.go:501: JWT authentication failed for user root from 100.74.79.152:22915: validate token (expected issuer=https://my.authentik.local/, audience=...., actual issuer=https://my.authentik.local/application/o/netbird/, audience=....): token could not be parsed: token has invalid claims: token has invalid issuer

It's expecting the issuer to be the base domain. Meanwhile, my configuration of management.json as well as dashboard environment variables clearly state issuer to be https://my.authentik.local/application/o/netbird/.

Did some quick investigation, and I think there is a bug in management/internals/shared/grpc/conversion.go:372:

issuer := strings.TrimSpace(config.AuthIssuer)
if issuer == "" || deviceFlowConfig != nil {  // seems like this OR is unintended
    if d := deriveIssuerFromTokenEndpoint(deviceFlowConfig.ProviderConfig.TokenEndpoint); d != "" {
        issuer = d
    }
}

The OR condition means when deviceFlowConfig exists, it always overrides the configured issuer by deriving it from the token endpoint. The deriveIssuerFromTokenEndpoint() function strips the path:

return fmt.Sprintf("%s://%s/", u.Scheme, u.Host)  // https://my.authentik.local/application/o/token/ becomes https://my.authentik.local/

This breaks Authentik (and any IDP where the issuer has a path component).
Instead, I believe in line 372, we need to change || to &&:

if issuer == "" && deviceFlowConfig != nil {

Though maybe I'm misunderstanding the flow.

EDIT: I manually set DeviceAuthorizationFlow to null in management.json and confirmed this solves the issue, which further confirms my assumption.

<!-- gh-comment-id:3567561571 --> @sgtaziz commented on GitHub (Nov 23, 2025): For me, it seems like its overriding my issuer (authentik): ``` 2025-11-23T09:09:17+03:00 ERRO shared/auth/jwt/validator.go:147: token could not be parsed: token has invalid claims: token has invalid issuer 2025-11-23T09:09:17+03:00 WARN client/ssh/server/server.go:501: JWT authentication failed for user root from 100.74.79.152:22915: validate token (expected issuer=https://my.authentik.local/, audience=...., actual issuer=https://my.authentik.local/application/o/netbird/, audience=....): token could not be parsed: token has invalid claims: token has invalid issuer ``` It's expecting the issuer to be the base domain. Meanwhile, my configuration of management.json as well as dashboard environment variables clearly state issuer to be `https://my.authentik.local/application/o/netbird/`. Did some quick investigation, and I think there is a bug in `management/internals/shared/grpc/conversion.go:372`: ```go issuer := strings.TrimSpace(config.AuthIssuer) if issuer == "" || deviceFlowConfig != nil { // seems like this OR is unintended if d := deriveIssuerFromTokenEndpoint(deviceFlowConfig.ProviderConfig.TokenEndpoint); d != "" { issuer = d } } ``` The OR condition means when `deviceFlowConfig` exists, it always overrides the configured issuer by deriving it from the token endpoint. The `deriveIssuerFromTokenEndpoint()` function strips the path: ```go return fmt.Sprintf("%s://%s/", u.Scheme, u.Host) // https://my.authentik.local/application/o/token/ becomes https://my.authentik.local/ ``` This breaks Authentik (and any IDP where the issuer has a path component). Instead, I believe in line 372, we need to change `||` to `&&`: ```go if issuer == "" && deviceFlowConfig != nil { ``` Though maybe I'm misunderstanding the flow. EDIT: I manually set `DeviceAuthorizationFlow` to `null` in management.json and confirmed this solves the issue, which further confirms my assumption.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2021 Q4
2021 Q4
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
2022 Q1
accessibility
accessibility
accessibility
acl
acl
acl
agent
agent
agent
agent
agent
agent
Android
Android
Android
Android
Android
Android
api
api
api
authentik
authentik
authentik
automation
automation
automation
azure
azure
azure
battery-usage
battery-usage
battery-usage
bug
cache
cache
cache
client
client
client
client-ui
client-ui
client-ui
cloud
cloud
cloud
cloud-only
cloud-only
cloud-only
cloudflare
cloudflare
cloudflare
community
community
community
compatibility
compatibility
compatibility
config-idp
config-idp
config-idp
config-issue
config-issue
config-issue
connection
connection
connection
contribution
contribution
contribution
coturn
coturn
coturn
cross-vpn
cross-vpn
cross-vpn
dashboard
dashboard
dashboard
data-usage
data-usage
data-usage
distribution
distribution
distribution
dns
dns
dns
docker
docker
docker
documentation
documentation
documentation
duplicate
duplicate
duplicate
enhancement
enhancement
event-stream
event-stream
event-stream
feature-request
feature-request
feature-request
freebsd
freebsd
freebsd
getting-started
getting-started
getting-started
go
go
go
good first issue
good first issue
good first issue
gui
gui
gui
help wanted
help wanted
help wanted
home-assistant
home-assistant
home-assistant
idp
idp
idp
inconsistency
inconsistency
inconsistency
integration
integration
integration
integrations
integrations
integrations
ios
ios
ios
ipv6
ipv6
ipv6
jwt
jwt
jwt
k8s
k8s
k8s
keycloak
keycloak
keycloak
linux
linux
linux
login
login
login
macos
macos
macos
management-service
management-service
management-service
Medium
Medium
Medium
missing-docs
missing-docs
missing-docs
mobile
mobile
mobile
moved-internal
moved-internal
moved-internal
needs-review
needs-review
needs-review
netbird-ui
netbird-ui
netbird-ui
networking
networking
networking
new-platform
new-platform
new-platform
nginx
nginx
nginx
notification
notification
notification
okta
okta
okta
openwrt
openwrt
openwrt
P2
P2
P2
packaging
packaging
packaging
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
peer-management
performance
performance
performance
postgres
postgres
postgres
posture-checks
posture-checks
posture-checks
psk
psk
psk
pull-request
Mirrored from GitHub Pull Request
 question
question
question
refactor
refactor
refactor
relay
relay
relay
release
release
release
rfc
rfc
rfc
routes
routes
routes
security
security
security
security-improvement
security-improvement
security-improvement
security-related
security-related
security-related
self-hosting
self-hosting
self-hosting
server
server
server
signal
signal
signal
sleep-issue
sleep-issue
sleep-issue
ssh
ssh
ssh
ssl
ssl
ssl
status
status
status
store
store
store
synology
synology
synology
system-compatibility-issue
system-compatibility-issue
system-compatibility-issue
test-suite
test-suite
test-suite
third-party-integration
third-party-integration
third-party-integration
triage
triage
triage
triage
triage
triage
triage-needed
triage-needed
triage-needed
troubleshooting
troubleshooting
troubleshooting
UX
UX
UX
waiting-feedback
waiting-feedback
waiting-feedback
windows
windows
windows
wontfix
wontfix
wontfix
zitadel
zitadel
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dynr-servers saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#9856
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?