Netbird SSH CLI Command failed #1968

Open
opened 2025-11-20 06:10:23 -05:00 by saavagebueno · 3 comments
Owner

Originally created by @2TAP2B on GitHub (Jun 16, 2025).

I'm unable to connect to another peer using the Netbird SSH command.

SSH is activated in the Netbird WebUI, and normal SSH to this peer over LAN is working, but Netbird SSH failed.

❯ sudo netbird ssh 100.70.103.72
Error: dial tcp 100.70.103.72:44338: connect: connection refused
Couldn't connect. Please check the connection status or if the ssh server is enabled on the other peer
You can verify the connection by running:

 netbird status

Error: dial tcp 100.70.103.72:44338: connect: connection refused

Are you using NetBird Cloud?

Netbird Selfhosted

NetBird version

❯ netbird version
0.46.0

netbird status -dA
Peers detail:
 tobias-macbookair.anon-gz160.domain:
  NetBird IP: 100.70.0.149
  Public key: ZzzOaLl+SMQm4243dU4kJpAx4nhQ6n6ZFPvED2xKaV8=
  Status: Connecting
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 11 minutes, 2 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

 adguard.anon-gz160.domain:
  NetBird IP: 100.70.9.67
  Public key: 1g4JAt1cw89zEH4JhKvoCYCqSCg8OpBwVVgNYXZoXno=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.70:51820
  Relay server address: rels://nb.anon-kcN8P.domain:443
  Last connection update: 11 minutes, 2 seconds ago
  Last WireGuard handshake: 29 seconds ago
  Transfer status (received/sent) 41.7 KiB/30.1 KiB
  Quantum resistance: false
  Networks: -
  Latency: 187.284186ms

 pxl-phn.anon-gz160.domain:
  NetBird IP: 100.70.50.94
  Public key: eTyv9oUJSjlDHCBnIY1aKlXBUYQcfNrN2iNzRbHh0xM=
  Status: Connecting
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 11 minutes, 2 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

 vm01.anon-gz160.domain:
  NetBird IP: 100.70.103.72
  Public key: DgZQwsPDkfIBQXEANNfnc3+w+O+CwiAldjcc9Wn1cjc=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.166:40868
  Relay server address: rels://nb.anon-kcN8P.domain:443
  Last connection update: 11 minutes, 1 seconds ago
  Last WireGuard handshake: 3 minutes, 1 seconds ago
  Transfer status (received/sent) 1.1 KiB/1.4 KiB
  Quantum resistance: false
  Networks: -
  Latency: 4.932799ms

 iphone-tobias.anon-gz160.domain:
  NetBird IP: 100.70.152.49
  Public key: euy4iZvRB/kk5P+n75a4bbicBgmp3xuS45yMG25XyBM=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.76:51820
  Relay server address: rels://nb.anon-kcN8P.domain:443
  Last connection update: 2 minutes, 3 seconds ago
  Last WireGuard handshake: 2 minutes, 2 seconds ago
  Transfer status (received/sent) 832 B/3.5 KiB
  Quantum resistance: false
  Networks: -
  Latency: 9.95454ms

 nfs01.anon-gz160.domain:
  NetBird IP: 100.70.153.57
  Public key: pt7t1z7jcZnkXWw2u+IFkUYON1Ib2ZTXSTzf9LwcEws=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.25:51820
  Relay server address: rels://nb.anon-kcN8P.domain:443
  Last connection update: 11 minutes, 2 seconds ago
  Last WireGuard handshake: 29 seconds ago
  Transfer status (received/sent) 1.9 KiB/952 B
  Quantum resistance: false
  Networks: -
  Latency: 1.708408ms

 lxd.anon-gz160.domain:
  NetBird IP: 100.70.185.55
  Public key: BWD2V9ofdI7eUaTqJdp5V56fKaiDq5IHwIiOopDYxAU=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.88:51820
  Relay server address: rels://nb.anon-kcN8P.domain:443
  Last connection update: 11 minutes, 1 seconds ago
  Last WireGuard handshake: 4 seconds ago
  Transfer status (received/sent) 1.3 KiB/1.3 KiB
  Quantum resistance: false
  Networks: -
  Latency: 9.460706ms

 pixel-8a.anon-gz160.domain:
  NetBird IP: 100.70.204.173
  Public key: q88DmOO8OJzzJAS6Ov0zJwo+uTsC4EnkcIICWwNZ7kw=
  Status: Connecting
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 11 minutes, 2 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Networks: -
  Latency: 0s

 subnet-router.anon-gz160.domain:
  NetBird IP: 100.70.210.176
  Public key: WGsTaHr82jUviEBZPfWmcuj/gDqkAyo2YSNHvRIQQgI=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.31:51820
  Relay server address: rels://nb.anon-kcN8P.domain:443
  Last connection update: 11 minutes, 2 seconds ago
  Last WireGuard handshake: 23 seconds ago
  Transfer status (received/sent) 27.4 KiB/25.0 KiB
  Quantum resistance: false
  Networks: -
  Latency: 2.080228ms

 hetzner01.anon-gz160.domain:
  NetBird IP: 100.70.235.147
  Public key: gsbTtIg8wOKURADea5D3i0+tkFunWtJtrCKuTyIS1mM=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:63018/198.51.100.1:51820
  Relay server address: rels://nb.anon-kcN8P.domain:443
  Last connection update: 11 minutes, 2 seconds ago
  Last WireGuard handshake: 35 seconds ago
  Transfer status (received/sent) 13.2 KiB/11.4 KiB
  Quantum resistance: false
  Networks: -
  Latency: 16.033891ms

Events:
  [WARNING] DNS (03fb4cff-9409-4866-8aa2-e71984e32474)
    Message: All upstream servers failed (probe failed)
    Time: 11 minutes, 2 seconds ago
    Metadata: upstreams: 100.70.9.67:53
  [INFO] SYSTEM (e2b61c0f-c447-474c-a38d-2e21beb3d1ae)
    Message: Network map updated
    Time: 11 minutes, 2 seconds ago
  [WARNING] DNS (80decfec-da44-4357-9181-f26c9bc1a97c)
    Message: All upstream servers failed (fail count exceeded)
    Time: 8 minutes, 36 seconds ago
    Metadata: upstreams: 100.70.9.67:53
  [WARNING] DNS (cb2f56c7-a6b0-4ef1-a03f-159c5571f794)
    Message: All upstream servers failed (fail count exceeded)
    Time: 8 minutes, 36 seconds ago
    Metadata: upstreams: 100.70.9.67:53
  [WARNING] DNS (1762103e-c41f-48d8-9993-f981b9580042)
    Message: All upstream servers failed (fail count exceeded)
    Time: 8 minutes, 36 seconds ago
    Metadata: upstreams: 100.70.9.67:53
OS: linux/amd64
Daemon version: 0.46.0
CLI version: 0.46.0
Management: Connected to https://nb.anon-kcN8P.domain:443
Signal: Connected to https://nb.anon-kcN8P.domain:443
Relays: 
  [stun:nb.anon-kcN8P.domain:3478] is Available
  [turn:nb.anon-kcN8P.domain:3478?transport=udp] is Available
  [rels://nb.anon-kcN8P.domain:443] is Available
Nameservers: 
  [100.70.9.67:53] for [.] is Available
FQDN: fedora.anon-gz160.domain
NetBird IP: 100.70.4.96/16
Interface type: Kernel
Quantum resistance: false
Lazy connection: false
Networks: -
Forwarding rules: 0
Peers count: 7/10 Connected

Create and upload a debug bundle, and share the returned file key:

netbird debug for 1m -AS -U

Uploaded files are automatically deleted after 30 days.

Alternatively, create the file only and attach it here manually:

netbird debug for 1m -AS

Originally created by @2TAP2B on GitHub (Jun 16, 2025). I'm unable to connect to another peer using the Netbird SSH command. SSH is activated in the Netbird WebUI, and normal SSH to this peer over LAN is working, but Netbird SSH failed. ``` ❯ sudo netbird ssh 100.70.103.72 Error: dial tcp 100.70.103.72:44338: connect: connection refused Couldn't connect. Please check the connection status or if the ssh server is enabled on the other peer You can verify the connection by running: netbird status Error: dial tcp 100.70.103.72:44338: connect: connection refused ``` **Are you using NetBird Cloud?** Netbird Selfhosted **NetBird version** ❯ netbird version 0.46.0 ``` netbird status -dA Peers detail: tobias-macbookair.anon-gz160.domain: NetBird IP: 100.70.0.149 Public key: ZzzOaLl+SMQm4243dU4kJpAx4nhQ6n6ZFPvED2xKaV8= Status: Connecting -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 11 minutes, 2 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Networks: - Latency: 0s adguard.anon-gz160.domain: NetBird IP: 100.70.9.67 Public key: 1g4JAt1cw89zEH4JhKvoCYCqSCg8OpBwVVgNYXZoXno= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/host ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.70:51820 Relay server address: rels://nb.anon-kcN8P.domain:443 Last connection update: 11 minutes, 2 seconds ago Last WireGuard handshake: 29 seconds ago Transfer status (received/sent) 41.7 KiB/30.1 KiB Quantum resistance: false Networks: - Latency: 187.284186ms pxl-phn.anon-gz160.domain: NetBird IP: 100.70.50.94 Public key: eTyv9oUJSjlDHCBnIY1aKlXBUYQcfNrN2iNzRbHh0xM= Status: Connecting -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 11 minutes, 2 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Networks: - Latency: 0s vm01.anon-gz160.domain: NetBird IP: 100.70.103.72 Public key: DgZQwsPDkfIBQXEANNfnc3+w+O+CwiAldjcc9Wn1cjc= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/prflx ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.166:40868 Relay server address: rels://nb.anon-kcN8P.domain:443 Last connection update: 11 minutes, 1 seconds ago Last WireGuard handshake: 3 minutes, 1 seconds ago Transfer status (received/sent) 1.1 KiB/1.4 KiB Quantum resistance: false Networks: - Latency: 4.932799ms iphone-tobias.anon-gz160.domain: NetBird IP: 100.70.152.49 Public key: euy4iZvRB/kk5P+n75a4bbicBgmp3xuS45yMG25XyBM= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/prflx ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.76:51820 Relay server address: rels://nb.anon-kcN8P.domain:443 Last connection update: 2 minutes, 3 seconds ago Last WireGuard handshake: 2 minutes, 2 seconds ago Transfer status (received/sent) 832 B/3.5 KiB Quantum resistance: false Networks: - Latency: 9.95454ms nfs01.anon-gz160.domain: NetBird IP: 100.70.153.57 Public key: pt7t1z7jcZnkXWw2u+IFkUYON1Ib2ZTXSTzf9LwcEws= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/host ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.25:51820 Relay server address: rels://nb.anon-kcN8P.domain:443 Last connection update: 11 minutes, 2 seconds ago Last WireGuard handshake: 29 seconds ago Transfer status (received/sent) 1.9 KiB/952 B Quantum resistance: false Networks: - Latency: 1.708408ms lxd.anon-gz160.domain: NetBird IP: 100.70.185.55 Public key: BWD2V9ofdI7eUaTqJdp5V56fKaiDq5IHwIiOopDYxAU= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/prflx ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.88:51820 Relay server address: rels://nb.anon-kcN8P.domain:443 Last connection update: 11 minutes, 1 seconds ago Last WireGuard handshake: 4 seconds ago Transfer status (received/sent) 1.3 KiB/1.3 KiB Quantum resistance: false Networks: - Latency: 9.460706ms pixel-8a.anon-gz160.domain: NetBird IP: 100.70.204.173 Public key: q88DmOO8OJzzJAS6Ov0zJwo+uTsC4EnkcIICWwNZ7kw= Status: Connecting -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 11 minutes, 2 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Networks: - Latency: 0s subnet-router.anon-gz160.domain: NetBird IP: 100.70.210.176 Public key: WGsTaHr82jUviEBZPfWmcuj/gDqkAyo2YSNHvRIQQgI= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/host ICE candidate endpoints (Local/Remote): 192.168.178.164:51820/192.168.178.31:51820 Relay server address: rels://nb.anon-kcN8P.domain:443 Last connection update: 11 minutes, 2 seconds ago Last WireGuard handshake: 23 seconds ago Transfer status (received/sent) 27.4 KiB/25.0 KiB Quantum resistance: false Networks: - Latency: 2.080228ms hetzner01.anon-gz160.domain: NetBird IP: 100.70.235.147 Public key: gsbTtIg8wOKURADea5D3i0+tkFunWtJtrCKuTyIS1mM= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): 198.51.100.0:63018/198.51.100.1:51820 Relay server address: rels://nb.anon-kcN8P.domain:443 Last connection update: 11 minutes, 2 seconds ago Last WireGuard handshake: 35 seconds ago Transfer status (received/sent) 13.2 KiB/11.4 KiB Quantum resistance: false Networks: - Latency: 16.033891ms Events: [WARNING] DNS (03fb4cff-9409-4866-8aa2-e71984e32474) Message: All upstream servers failed (probe failed) Time: 11 minutes, 2 seconds ago Metadata: upstreams: 100.70.9.67:53 [INFO] SYSTEM (e2b61c0f-c447-474c-a38d-2e21beb3d1ae) Message: Network map updated Time: 11 minutes, 2 seconds ago [WARNING] DNS (80decfec-da44-4357-9181-f26c9bc1a97c) Message: All upstream servers failed (fail count exceeded) Time: 8 minutes, 36 seconds ago Metadata: upstreams: 100.70.9.67:53 [WARNING] DNS (cb2f56c7-a6b0-4ef1-a03f-159c5571f794) Message: All upstream servers failed (fail count exceeded) Time: 8 minutes, 36 seconds ago Metadata: upstreams: 100.70.9.67:53 [WARNING] DNS (1762103e-c41f-48d8-9993-f981b9580042) Message: All upstream servers failed (fail count exceeded) Time: 8 minutes, 36 seconds ago Metadata: upstreams: 100.70.9.67:53 OS: linux/amd64 Daemon version: 0.46.0 CLI version: 0.46.0 Management: Connected to https://nb.anon-kcN8P.domain:443 Signal: Connected to https://nb.anon-kcN8P.domain:443 Relays: [stun:nb.anon-kcN8P.domain:3478] is Available [turn:nb.anon-kcN8P.domain:3478?transport=udp] is Available [rels://nb.anon-kcN8P.domain:443] is Available Nameservers: [100.70.9.67:53] for [.] is Available FQDN: fedora.anon-gz160.domain NetBird IP: 100.70.4.96/16 Interface type: Kernel Quantum resistance: false Lazy connection: false Networks: - Forwarding rules: 0 Peers count: 7/10 Connected ``` Create and upload a debug bundle, and share the returned file key: netbird debug for 1m -AS -U *Uploaded files are automatically deleted after 30 days.* Alternatively, create the file only and attach it here manually: netbird debug for 1m -AS <!-- Failed to upload "netbird.debug.1500200513.zip" -->
saavagebueno added the triage-needed label 2025-11-20 06:10:23 -05:00
Author
Owner

@nazarewk commented on GitHub (Jun 16, 2025):

In addition to allowing SSH in web ui, you also need to run the client with --allow-server-ssh (during up command), did you do that?

@nazarewk commented on GitHub (Jun 16, 2025): In addition to allowing SSH in web ui, you also need to run the client with `--allow-server-ssh` (during `up` command), did you do that?
Author
Owner

@2TAP2B commented on GitHub (Jun 16, 2025):

Okay, nice — thanks, it works for me, but only for one peer.

I really don’t get it...

I set up this VM with netbird down & netbird up --allow-server-ssh, and then ran:

╰─ sudo netbird ssh vm01.net.bird
Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-59-generic x86_64)

This one works fine.

But I did exactly the same on another VM, and it doesn’t work:

╰─ sudo netbird ssh hetzner01.net.bird
Error: dial tcp 100.70.235.147:44338: i/o timeout
Couldn't connect. Please check the connection status or if the ssh server is enabled on the other peer
You can verify the connection by running:

 netbird status

Error: dial tcp 100.70.235.147:44338: i/o timeout

###

╰─ ping hetzner01.net.bird                                                                                                                                                     ─╯
PING hetzner01.net.bird (100.70.235.147) 56(84) bytes of data.
64 bytes from hetzner01.net.bird (100.70.235.147): icmp_seq=1 ttl=64 time=16.5 ms
64 bytes from hetzner01.net.bird (100.70.235.147): icmp_seq=2 ttl=64 time=18.0 ms
64 bytes from hetzner01.net.bird (100.70.235.147): icmp_seq=3 ttl=64 time=17.6 ms
64 bytes from hetzner01.net.bird (100.70.235.147): icmp_seq=4 ttl=64 time=17.3 ms
^C
--- hetzner01.net.bird ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 16.525/17.344/17.992/0.536 ms

Any ideas what is going on here?

@2TAP2B commented on GitHub (Jun 16, 2025): Okay, nice — thanks, it works for me, but only for one peer. I really don’t get it... I set up this VM with `netbird down` & `netbird up --allow-server-ssh`, and then ran: ``` ╰─ sudo netbird ssh vm01.net.bird Welcome to Ubuntu 24.04.2 LTS (GNU/Linux 6.8.0-59-generic x86_64) ``` This one works fine. But I did exactly the same on another VM, and it doesn’t work: ``` ╰─ sudo netbird ssh hetzner01.net.bird Error: dial tcp 100.70.235.147:44338: i/o timeout Couldn't connect. Please check the connection status or if the ssh server is enabled on the other peer You can verify the connection by running: netbird status Error: dial tcp 100.70.235.147:44338: i/o timeout ### ╰─ ping hetzner01.net.bird ─╯ PING hetzner01.net.bird (100.70.235.147) 56(84) bytes of data. 64 bytes from hetzner01.net.bird (100.70.235.147): icmp_seq=1 ttl=64 time=16.5 ms 64 bytes from hetzner01.net.bird (100.70.235.147): icmp_seq=2 ttl=64 time=18.0 ms 64 bytes from hetzner01.net.bird (100.70.235.147): icmp_seq=3 ttl=64 time=17.6 ms 64 bytes from hetzner01.net.bird (100.70.235.147): icmp_seq=4 ttl=64 time=17.3 ms ^C --- hetzner01.net.bird ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3004ms rtt min/avg/max/mdev = 16.525/17.344/17.992/0.536 ms ``` Any ideas what is going on here?
Author
Owner

@mrg358 commented on GitHub (Jul 21, 2025):

ran into same issue but only for windows host

sudo netbird ssh 100.xxx.xx.xxx
Error: dial tcp 100.xxx.xx.xxx:44338: connect: connection refused
Couldn't connect. Please check the connection status or if the ssh server is enabled on the other peer
You can verify the connection by running:

 netbird status

Error: dial tcp 100.xxx.xx.xxx:44338: connect: connection refused

i don't see any status of ssh in netbird status output even for linux hosts where ssh works

netbird status
OS: linux/arm64
Daemon version: 0.51.1
CLI version: 0.51.1
Management: Connected
Signal: Connected
Relays: 2/3 Available
Nameservers: 0/0 Available
FQDN: linux-host.netbird.selfhosted
NetBird IP: 100.xxx.xx.xxx/16
Interface type: Kernel
Quantum resistance: false
Lazy connection: false
Networks: -
Forwarding rules: 0
Peers count: 3/4 Connected
@mrg358 commented on GitHub (Jul 21, 2025): ran into same issue but only for windows host ``` sudo netbird ssh 100.xxx.xx.xxx Error: dial tcp 100.xxx.xx.xxx:44338: connect: connection refused Couldn't connect. Please check the connection status or if the ssh server is enabled on the other peer You can verify the connection by running: netbird status Error: dial tcp 100.xxx.xx.xxx:44338: connect: connection refused ``` i don't see any status of ssh in `netbird status` output even for linux hosts where ssh works ``` netbird status OS: linux/arm64 Daemon version: 0.51.1 CLI version: 0.51.1 Management: Connected Signal: Connected Relays: 2/3 Available Nameservers: 0/0 Available FQDN: linux-host.netbird.selfhosted NetBird IP: 100.xxx.xx.xxx/16 Interface type: Kernel Quantum resistance: false Lazy connection: false Networks: - Forwarding rules: 0 Peers count: 3/4 Connected ```
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1968