[GH-ISSUE #4869] SSH broken on Debian 13 due to new login package #9985

Closed
opened 2026-08-05 01:24:12 -04:00 by saavagebueno · 12 comments
Owner

Originally created by @afonsofrancof on GitHub (Nov 27, 2025).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/4869

Environment

  • NetBird Version: 0.60.3
  • OS: Debian 13 (Trixie)
  • Architecture: amd64
  • Installation Method: apt repository

Description

NetBird SSH connections fail immediately on Debian 13 (Trixie) systems. After successful authentication and PTY allocation, the session closes immediately without spawning an interactive shell. Non-interactive commands work correctly.

Root Cause

After looking through logs and debugging this issue for hours I finally found the cause.

NetBird's SSH server uses login with some flags for interactive sessions which works fine with shadow-login (Debian 12) but fails with util-linux login (Debian 13).

Debian 13 switched from shadow's to util-linux implementation of the login binary (check the right side of the page on those two links. It also shows it on the man page of the packages). The util-linux version behaves differently when invoked by NetBird and exits instead of spawning an interactive shell.

Behavior

  • Connection authenticates successfully
  • PTY allocation is accepted
  • Shell request is accepted
  • Session immediately closes
  • Non-interactive commands work: ssh user@peer 'echo test' succeeds

Receiving peer logs (without prefix because of horizontal space):

... client/ssh/server/server.go:607: SSH connection from NetBird peer 100.64.0.5:62942 allowed

... client/ssh/server/session_handlers.go:35: SSH session started

... client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login

Connecting peer output

❯ ssh afonso@togepi.netbird.selfhosted
Shared connection to togepi.netbird.selfhosted closed.

Workaround

The only way I could get NetBird's ssh to work again was to do the following:

1 - Edit /etc/apt/sources.list and change the repositories back to bookworm
2 - sudo apt update
3 - sudo apt install login=1:4.13+dfsg1-1+deb12u1
4 - Use NetBird's SSH (here I didn't use netbird.selfhosted suffix but it's the same)

Image

This is obviously not desired and I only used it for testing.

Change

Something needs to change here

aca0398105/client/ssh/server/userswitching_unix.go (L77-L100)

I will try to experiment with some of the login command flags and will report back in case I find something.

Hope this can be fixed.
I was going crazy trying to get this to work 😅

Originally created by @afonsofrancof on GitHub (Nov 27, 2025). Original GitHub issue: https://github.com/netbirdio/netbird/issues/4869 # Environment - **NetBird Version**: 0.60.3 - **OS**: Debian 13 (Trixie) - **Architecture**: amd64 - **Installation Method**: apt repository # Description NetBird SSH connections fail immediately on Debian 13 (Trixie) systems. After successful authentication and PTY allocation, the session closes immediately without spawning an interactive shell. Non-interactive commands work correctly. # Root Cause After looking through logs and debugging this issue for **hours** I finally found the cause. NetBird's SSH server uses `login` with some flags for interactive sessions which works fine with shadow-login (Debian 12) but fails with util-linux login (Debian 13). Debian 13 switched from [shadow's](https://packages.debian.org/bookworm/login) to [util-linux](https://packages.debian.org/trixie/login) implementation of the `login` binary (check the right side of the page on those two links. It also shows it on the man page of the packages). The util-linux version behaves differently when invoked by NetBird and exits instead of spawning an interactive shell. # Behavior - Connection authenticates successfully - PTY allocation is accepted - Shell request is accepted - Session immediately closes - Non-interactive commands work: `ssh user@peer 'echo test'` succeeds ## Receiving peer logs (without prefix because of horizontal space): ``` ... client/ssh/server/server.go:607: SSH connection from NetBird peer 100.64.0.5:62942 allowed ... client/ssh/server/session_handlers.go:35: SSH session started ... client/ssh/server/command_execution_unix.go:147: starting interactive shell: /usr/bin/login ``` ## Connecting peer output ``` ❯ ssh afonso@togepi.netbird.selfhosted Shared connection to togepi.netbird.selfhosted closed. ``` ### Workaround The only way I could get NetBird's ssh to work again was to do the following: 1 - Edit `/etc/apt/sources.list` and change the repositories back to `bookworm` 2 - `sudo apt update` 3 - `sudo apt install login=1:4.13+dfsg1-1+deb12u1` 4 - Use NetBird's SSH (here I didn't use `netbird.selfhosted` suffix but it's the same) <img width="930" height="246" alt="Image" src="https://github.com/user-attachments/assets/65c7654e-199c-4f60-b8df-dd799f33a614" /> This is obviously not desired and I only used it for testing. # Change Something needs to change here https://github.com/netbirdio/netbird/blob/aca0398105fd0662c09533e1368a8682310efd94/client/ssh/server/userswitching_unix.go#L77-L100 I will try to experiment with some of the `login` command flags and will report back in case I find something. Hope this can be fixed. I was going crazy trying to get this to work 😅
saavagebueno added the triage-needed label 2026-08-05 01:24:12 -04:00
Author
Owner

@afonsofrancof commented on GitHub (Nov 27, 2025):

Tailscale uses basically the same code as netbird for the login so I decided to look in their issues and found some things.

This tailscale issue might be relevant although it mentions musl vs glibc for shadow-utils, not shadow-utils vs util-linux.

There is also this tailscale issue
I already tried their solution, but it didn't work for me.

I also tried updating to the latest version of login from debian unstable, because of this bug report and of it's supposed fix, but it did nothing.

The only thing that worked for me was the downgrade I mentioned in the main post

<!-- gh-comment-id:3587698692 --> @afonsofrancof commented on GitHub (Nov 27, 2025): [Tailscale uses basically the same code as netbird for the login](https://github.com/tailscale/tailscale/blob/411cee0dc9dbff2bfcf68f7588e3276f44ff9c6c/ssh/tailssh/incubator.go#L1161-L1170) so I decided to look in their issues and found some things. [This tailscale issue](https://github.com/tailscale/tailscale/issues/16346) might be relevant although it mentions musl vs glibc for shadow-utils, not shadow-utils vs util-linux. There is also [this tailscale issue](https://github.com/tailscale/tailscale/issues/16983) I already tried [their solution](https://github.com/tailscale/tailscale/issues/16983#issuecomment-3326078422), but it didn't work for me. I also tried updating to the latest version of `login` from debian unstable, because of [this bug report](https://bugs-devel.debian.org/cgi-bin/bugreport.cgi?bug=1103923) and of [it's supposed fix](https://salsa.debian.org/debian/util-linux/-/commit/8a6af0188166c3080156c60f0ed8fa2158c38054), but it did nothing. The only thing that worked for me was the downgrade I mentioned in the main post
Author
Owner

@afonsofrancof commented on GitHub (Nov 28, 2025):

Found a fix!!
Using runuser instead of login works correctly for me!

runuserPath, err := exec.LookPath("runuser")
if err == nil {   
	return runuserPath, []string{"-l", username}, nil
}

I will try to find a way to check for debian 13 from golang and add that as a PR

<!-- gh-comment-id:3588919372 --> @afonsofrancof commented on GitHub (Nov 28, 2025): Found a fix!! Using `runuser` instead of login works correctly for me! ```go runuserPath, err := exec.LookPath("runuser") if err == nil { return runuserPath, []string{"-l", username}, nil } ``` I will try to find a way to check for debian 13 from golang and add that as a PR
Author
Owner

@afonsofrancof commented on GitHub (Nov 28, 2025):

Opened PR #4873 to fix this :)

<!-- gh-comment-id:3589027260 --> @afonsofrancof commented on GitHub (Nov 28, 2025): Opened PR #4873 to fix this :)
Author
Owner

@alexmoras commented on GitHub (Nov 29, 2025):

@afonsofrancof Exactly the same issue running Ubuntu Server 25.10 - this is running util-linux 2.41 which I believe is the same as Debian 13.

I think the PR will need to check more for whether util-linux is being used rather than which OS is installed. I know you can run login -V which will return login from util-linux 2.41 so we can parse that to see if it contains the relevant string?

<!-- gh-comment-id:3591724924 --> @alexmoras commented on GitHub (Nov 29, 2025): @afonsofrancof Exactly the same issue running Ubuntu Server 25.10 - this is running `util-linux 2.41` which I believe is the same as Debian 13. I think the PR will need to check more for whether `util-linux` is being used rather than which OS is installed. I know you can run `login -V` which will return `login from util-linux 2.41` so we can parse that to see if it contains the relevant string?
Author
Owner

@afonsofrancof commented on GitHub (Nov 29, 2025):

@alexmoras Hey!
Yes, I agree with you and even said so in my PR.
The login binary is the problem, so we should fix it on all util-linux distros.
I will change that now, just wanted a second opinion :)

Btw, Have you tried the PR? Does it work well for you?

<!-- gh-comment-id:3591730238 --> @afonsofrancof commented on GitHub (Nov 29, 2025): @alexmoras Hey! Yes, I agree with you and even said so in my PR. The login binary is the problem, so we should fix it on all util-linux distros. I will change that now, just wanted a second opinion :) Btw, Have you tried the PR? Does it work well for you?
Author
Owner

@alexmoras commented on GitHub (Nov 29, 2025):

Ah noted! Apologies, didn't think of looking at the PR. Standby, will test the PR now.

<!-- gh-comment-id:3591731006 --> @alexmoras commented on GitHub (Nov 29, 2025): Ah noted! Apologies, didn't think of looking at the PR. Standby, will test the PR now.
Author
Owner

@afonsofrancof commented on GitHub (Nov 29, 2025):

No need to apologize :)

<!-- gh-comment-id:3591731563 --> @afonsofrancof commented on GitHub (Nov 29, 2025): No need to apologize :)
Author
Owner

@alexmoras commented on GitHub (Nov 29, 2025):

@afonsofrancof Can confirm that the PR has appeared to have fixed the issue. Will comment separately on the PR now. Nice work!

<!-- gh-comment-id:3591760792 --> @alexmoras commented on GitHub (Nov 29, 2025): @afonsofrancof Can confirm that the PR has appeared to have fixed the issue. Will comment separately on the PR now. Nice work!
Author
Owner

@alexmoras commented on GitHub (Dec 2, 2025):

As noted in https://github.com/netbirdio/netbird/issues/4845#issuecomment-3603319266 this issue is also affecting modern Arch installs. From a very quick bit of research, it would appear that most non-LTS distros will be using util-linux now and therefore impact will be widespread.

<!-- gh-comment-id:3603386150 --> @alexmoras commented on GitHub (Dec 2, 2025): As noted in https://github.com/netbirdio/netbird/issues/4845#issuecomment-3603319266 this issue is also affecting modern Arch installs. From a very quick bit of research, it would appear that most non-LTS distros will be using `util-linux` now and therefore impact will be widespread.
Author
Owner

@lixmal commented on GitHub (Dec 3, 2025):

Hey folks, can you test this build https://github.com/netbirdio/netbird/pull/4900 (binaries here on the bottom) please?

<!-- gh-comment-id:3605596544 --> @lixmal commented on GitHub (Dec 3, 2025): Hey folks, can you test this build https://github.com/netbirdio/netbird/pull/4900 (binaries [here on the bottom](https://github.com/netbirdio/netbird/actions/runs/19886820589?pr=4900)) please?
Author
Owner

@afonsofrancof commented on GitHub (Dec 3, 2025):

@lixmal This fixes the problem for me completely.
It shows up correctly in w/who and the loginuid is set correctly.
I will close my PR.
Thanks!

<!-- gh-comment-id:3606785941 --> @afonsofrancof commented on GitHub (Dec 3, 2025): @lixmal This fixes the problem for me completely. It shows up correctly in `w`/`who` and the `loginuid` is set correctly. I will close my PR. Thanks!
Author
Owner

@jnfrati commented on GitHub (May 7, 2026):

@afonsofrancof closing this issue for now then!

If the current behavior still don't match what you were expecting feel free to open a new feature request or an issue and we'll take a look!

<!-- gh-comment-id:4396470899 --> @jnfrati commented on GitHub (May 7, 2026): @afonsofrancof closing this issue for now then! If the current behavior still don't match what you were expecting feel free to open a new [feature request or an issue](https://github.com/netbirdio/netbird/discussions/new/choose) and we'll take a look!
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#9985