[GH-ISSUE #2454] Netbird can't recover from macOS sleep #5183

New Issue

Originally created by @trbutler on GitHub (Aug 20, 2024).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/2454

Originally assigned to: @pappz on GitHub.

Even if I turn on Network Monitor, Netbird is rarely functional when I wake my Mac up from sleep.

To Reproduce

Put Mac to sleep for a short while, wake it up, try to access a Netbird peer. The computer will know the IP of the peer and netbird status will say it is connected, but any access to the other peer will fail. This happens on every macOS client I try it with, whether Network Monitor is enabled or not.

Expected behavior

One would expect the connection to recover shortly after waking from sleep.

Are you using NetBird Cloud?

Using the self-hosted version updated to the latest version.

NetBird version

0.28.7

Temporary Fix

Create a MacOS launchAgent for waking from sleep that will load a shell script that pings a known, always available Netbird peer and run the equivalent of netbird down && netbird up if ping fails:

#!/bin/bash

# Define the server to ping
SERVER="[path to known netbird peer]"

# Ping the server
ping -c 1 $SERVER > /dev/null 2>&1

# Check the exit status of the ping command
if [ $? -ne 0 ]; then
    # Run the command if the server is unreachable
    echo "Server is unreachable; reloading Netbird.\n";
    netbird down
    netbird up
fi

What I've done is install the netbird client on the server hosting the Netbird docker container so I have a reliable point to connect to, then check to see if I can in fact connect to a "peer" (and not just the server processes). Obviously, this is a hack and not a permanent solution; it'd be better if this were some sort of built-in check rather than a shell script.

It might be nice if such a check could be done every so often by the netbird client even between times sleeping: I've found that if my network connection is unstable, I'll keep losing access to netbird peers midsession as well. I'm thinking about having this test script run every so many minutes rather than just on wake for now.

Originally created by @trbutler on GitHub (Aug 20, 2024). Original GitHub issue: https://github.com/netbirdio/netbird/issues/2454 Originally assigned to: @pappz on GitHub. Even if I turn on Network Monitor, Netbird is rarely functional when I wake my Mac up from sleep. **To Reproduce** Put Mac to sleep for a short while, wake it up, try to access a Netbird peer. The computer will know the IP of the peer and `netbird status` will say it is connected, but any access to the other peer will fail. This happens on every macOS client I try it with, whether Network Monitor is enabled or not. **Expected behavior** One would expect the connection to recover shortly after waking from sleep. **Are you using NetBird Cloud?** Using the self-hosted version updated to the latest version. **NetBird version** 0.28.7 **Temporary Fix** Create a MacOS launchAgent for waking from sleep that will load a shell script that pings a known, always available Netbird peer and run the equivalent of `netbird down && netbird up` if ping fails: ``` #!/bin/bash # Define the server to ping SERVER="[path to known netbird peer]" # Ping the server ping -c 1 $SERVER > /dev/null 2>&1 # Check the exit status of the ping command if [ $? -ne 0 ]; then # Run the command if the server is unreachable echo "Server is unreachable; reloading Netbird.\n"; netbird down netbird up fi ``` What I've done is install the netbird client on the server hosting the Netbird docker container so I have a reliable point to connect to, then check to see if I can in fact connect to a "peer" (and not just the server processes). Obviously, this is a hack and not a permanent solution; it'd be better if this were some sort of built-in check rather than a shell script. It might be nice if such a check could be done every so often by the netbird client even between times sleeping: I've found that if my network connection is unstable, I'll keep losing access to netbird peers midsession as well. I'm thinking about having this test script run every so many minutes rather than just on wake for now.

saavagebueno added the bug client networking macos self-hosting sleep-issue labels 2026-08-05 01:01:51 -04:00

saavagebueno commented

@hurricanehrndz commented on GitHub (Aug 29, 2024):

@trbutler can you share the status of netbird on a bad wake up.

@hurricanehrndz commented on GitHub (Aug 29, 2024): @trbutler can you share the status of netbird on a bad wake up.

saavagebueno commented

@paularlott commented on GitHub (Sep 4, 2024):

We're seeing the same on 2 Macs.

When they have a bad wake up we get:

netbird status

OS: darwin/arm64
Daemon version: 0.28.9
CLI version: 0.28.9
Management: Connected
Signal: Connected
Relays: 0/0 Available
Nameservers: 0/1 Available
FQDN: seasmoke.*****
NetBird IP: 100.********/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 0/0 Connected

and

netbird status -d

Peers detail:
OS: darwin/arm64
Daemon version: 0.28.9
CLI version: 0.28.9
Management: Connected to https://mesh.***:443
Signal: Connected to https://mesh.***:443
Relays:
Nameservers:
  [172.******:53, 172.******:53] for [***.com, ***.au] is Unavailable, reason: 1 error occurred:
	* read udp 192.*****:50159->172.******:53: i/o timeout
FQDN: seasmoke.***
NetBird IP: 100.********/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 0/0 Connected

netbird down && netbird up seems to restore the connection every time.

Hopefully that will help but if I can supply more information please let me know what would be helpful.

@paularlott commented on GitHub (Sep 4, 2024): We're seeing the same on 2 Macs. When they have a bad wake up we get: ``` netbird status OS: darwin/arm64 Daemon version: 0.28.9 CLI version: 0.28.9 Management: Connected Signal: Connected Relays: 0/0 Available Nameservers: 0/1 Available FQDN: seasmoke.***** NetBird IP: 100.********/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 0/0 Connected ``` and ``` netbird status -d Peers detail: OS: darwin/arm64 Daemon version: 0.28.9 CLI version: 0.28.9 Management: Connected to https://mesh.***:443 Signal: Connected to https://mesh.***:443 Relays: Nameservers: [172.******:53, 172.******:53] for [***.com, ***.au] is Unavailable, reason: 1 error occurred: * read udp 192.*****:50159->172.******:53: i/o timeout FQDN: seasmoke.*** NetBird IP: 100.********/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 0/0 Connected ``` `netbird down && netbird up` seems to restore the connection every time. Hopefully that will help but if I can supply more information please let me know what would be helpful.

saavagebueno commented

https://github.com/netbirdio/netbird/actions/runs/10777251462?pr=2565

@hurricanehrndz commented on GitHub (Sep 10, 2024):

Version 29 is out that has a potential fix for this, if you still experience that with version 29, can you please test, the test version requires autoconnect and network monitor to both be on

@hurricanehrndz commented on GitHub (Sep 10, 2024): Version 29 is out that has a potential fix for this, if you still experience that with version 29, can you please test, the test version requires autoconnect and network monitor to both be on https://github.com/netbirdio/netbird/actions/runs/10777251462?pr=2565

saavagebueno commented

@paularlott commented on GitHub (Sep 12, 2024):

I'm finding that on 0.29.1 when a couple of the macs wake from sleep they are disconnected, selecting connect from the UI does nothing.

Running sudo netbird service restart causes a window to popup with the error:

rpc error: code = Unavailable desc = error reading from server: EOF

Thanks

@paularlott commented on GitHub (Sep 12, 2024): I'm finding that on 0.29.1 when a couple of the macs wake from sleep they are disconnected, selecting connect from the UI does nothing. Running `sudo netbird service restart` causes a window to popup with the error: ``` rpc error: code = Unavailable desc = error reading from server: EOF ``` Thanks

saavagebueno commented

@paularlott commented on GitHub (Sep 15, 2024):

Just to add to this, I think it's more to do with network change than wake from sleep, e.g. go to sleep on wifi 1, wake on wifi 2 or wired.

@paularlott commented on GitHub (Sep 15, 2024): Just to add to this, I think it's more to do with network change than wake from sleep, e.g. go to sleep on wifi 1, wake on wifi 2 or wired.

saavagebueno commented

@hurricanehrndz commented on GitHub (Sep 20, 2024):

29.3 should fix this

@hurricanehrndz commented on GitHub (Sep 20, 2024): 29.3 should fix this

saavagebueno commented

@paularlott commented on GitHub (Sep 21, 2024):

From our testing 29.3 is reconnecting to netbird every time on wake, thank you.

@paularlott commented on GitHub (Sep 21, 2024): From our testing 29.3 is reconnecting to netbird every time on wake, thank you.

saavagebueno commented

@trbutler commented on GitHub (Sep 22, 2024):

@trbutler can you share the status of netbird on a bad wake up.

Sorry, I missed your question. Typically it would look exactly as it should (with connections), they just wouldn't work. I updated to 0.29.4 from 0.29.2 this morning and will report back how it goes and share an example of the problematic status if it arises again.

@trbutler commented on GitHub (Sep 22, 2024): > @trbutler can you share the status of netbird on a bad wake up. Sorry, I missed your question. Typically it would look exactly as it should (with connections), they just wouldn't work. I updated to 0.29.4 from 0.29.2 this morning and will report back how it goes and share an example of the problematic status if it arises again.

saavagebueno commented

@juev commented on GitHub (Oct 3, 2024):

I installed version 0.29.4, and the laptop with macOS does not see the network after sleep.

Reconnecting brings it back to life.

:(

@juev commented on GitHub (Oct 3, 2024): I installed version 0.29.4, and the laptop with macOS does not see the network after sleep. Reconnecting brings it back to life. :(

saavagebueno commented

@juev commented on GitHub (Oct 3, 2024):

update: Interestingly, the retest did not reproduce the problem. I'll keep an eye on it.

@juev commented on GitHub (Oct 3, 2024): update: Interestingly, the retest did not reproduce the problem. I'll keep an eye on it.

saavagebueno commented

@hurricanehrndz commented on GitHub (Oct 3, 2024):

Please test the version in the PR above

Get Outlook for iOShttps://aka.ms/o0ukef

From: Evsyukov Denis @.>
Sent: Thursday, October 3, 2024 4:52:16 AM
To: netbirdio/netbird @.>
Cc: Carlos Hernandez @.>; Comment @.>
Subject: Re: [netbirdio/netbird] Netbird can't recover from macOS sleep (Issue #2454)

update: Interestingly, the retest did not reproduce the problem. I'll keep an eye on it.

—
Reply to this email directly, view it on GitHubhttps://github.com/netbirdio/netbird/issues/2454#issuecomment-2391109199, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABMJBTMV4MVEHOHA24J7HRLZZUOWBAVCNFSM6AAAAABM2MNULKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJRGEYDSMJZHE.
You are receiving this because you commented.Message ID: @.***>

@hurricanehrndz commented on GitHub (Oct 3, 2024): Please test the version in the PR above Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Evsyukov Denis ***@***.***> Sent: Thursday, October 3, 2024 4:52:16 AM To: netbirdio/netbird ***@***.***> Cc: Carlos Hernandez ***@***.***>; Comment ***@***.***> Subject: Re: [netbirdio/netbird] Netbird can't recover from macOS sleep (Issue #2454) update: Interestingly, the retest did not reproduce the problem. I'll keep an eye on it. — Reply to this email directly, view it on GitHub<https://github.com/netbirdio/netbird/issues/2454#issuecomment-2391109199>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABMJBTMV4MVEHOHA24J7HRLZZUOWBAVCNFSM6AAAAABM2MNULKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJRGEYDSMJZHE>. You are receiving this because you commented.Message ID: ***@***.***>

saavagebueno commented

https://github.com/netbirdio/netbird/pull/2676

@hurricanehrndz commented on GitHub (Oct 3, 2024):

Sorry this PR

@hurricanehrndz commented on GitHub (Oct 3, 2024): Sorry this PR https://github.com/netbirdio/netbird/pull/2676

saavagebueno commented

@paularlott commented on GitHub (Oct 19, 2024):

It looks like release 0.30.2 of the client has gone backwards, it now never reconnects after the mac wakes up and I have to do sudo netbird service restart after which I can get connected. The previous 0.29.x versions were working fine and always reconnecting after wake from sleep.

Is there any information that would help with debugging this?

@paularlott commented on GitHub (Oct 19, 2024): It looks like release 0.30.2 of the client has gone backwards, it now never reconnects after the mac wakes up and I have to do `sudo netbird service restart` after which I can get connected. The previous 0.29.x versions were working fine and always reconnecting after wake from sleep. Is there any information that would help with debugging this?

saavagebueno commented

@trbutler commented on GitHub (Oct 20, 2024):

I’m seeing the same thing. Or it’ll still be connected to 2-3 peers, but the other 8-12 only reconnect after bringing NetBird down and back up again.

@trbutler commented on GitHub (Oct 20, 2024): I’m seeing the same thing. Or it’ll still be connected to 2-3 peers, but the other 8-12 only reconnect after bringing NetBird down and back up again.

saavagebueno commented

@mlsmaycon commented on GitHub (Oct 20, 2024):

Hey folks, we are working on improving the recovery. The pr #2757 has a state handler for the DNS and routes added by the client. If possible, it would be great if you could download the builder binary and test it on your machine. See the steps below:

Download https://github.com/netbirdio/netbird/actions/runs/11405814951/artifacts/2074696862 (you need to be logged in to github).

Stop the service with sudo netbird service stop.

Extract, pick the correct arch and replace the netbird binary (which netbird) with the new one. e.g. when downloading and extracting the file in the Downloads folder:

sudo ln -s -f Downloads/macos-packages/netbird_darwin_all/netbird /usr/local/bin/netbird

Then sudo netbird service start

To recover the package version, stop the service again and run the following command:

sudo ln -s -f /Applications/NetBird.app/Contents/MacOS/netbird /usr/local/bin/netbird

@mlsmaycon commented on GitHub (Oct 20, 2024): Hey folks, we are working on improving the recovery. The pr #2757 has a state handler for the DNS and routes added by the client. If possible, it would be great if you could download the builder binary and test it on your machine. See the steps below: Download https://github.com/netbirdio/netbird/actions/runs/11405814951/artifacts/2074696862 (you need to be logged in to github). Stop the service with `sudo netbird service stop`. Extract, pick the correct arch and replace the netbird binary (which netbird) with the new one. e.g. when downloading and extracting the file in the Downloads folder: ```shell sudo ln -s -f Downloads/macos-packages/netbird_darwin_all/netbird /usr/local/bin/netbird ``` Then `sudo netbird service start` To recover the package version, stop the service again and run the following command: ```shell sudo ln -s -f /Applications/NetBird.app/Contents/MacOS/netbird /usr/local/bin/netbird ```

saavagebueno commented

@Eu-Arthur commented on GitHub (Oct 21, 2024):

i have same bug on windows client & linux client

@Eu-Arthur commented on GitHub (Oct 21, 2024): i have same bug on windows client & linux client

saavagebueno commented

@mlsmaycon commented on GitHub (Oct 21, 2024):

@Eu-Arthur you can use the following download links to validate them on your other clients:
Linux:
https://github.com/netbirdio/netbird/actions/runs/11408557648/artifacts/2075518167
Windows:
https://github.com/netbirdio/netbird/actions/runs/11408557648/artifacts/2075518258

The idea is similar, just check the correct Linux path with the command which netbird and on Windows the path will be C:\Program Files\NetBird\netbird.exe

@mlsmaycon commented on GitHub (Oct 21, 2024): @Eu-Arthur you can use the following download links to validate them on your other clients: Linux: https://github.com/netbirdio/netbird/actions/runs/11408557648/artifacts/2075518167 Windows: https://github.com/netbirdio/netbird/actions/runs/11408557648/artifacts/2075518258 The idea is similar, just check the correct Linux path with the command `which netbird` and on Windows the path will be `C:\Program Files\NetBird\netbird.exe`

saavagebueno commented

@Eu-Arthur commented on GitHub (Oct 21, 2024):

On windows, that doesn't resolv the probleme.
And that was not better on linux.

I have rollback on 29.4 that work

@Eu-Arthur commented on GitHub (Oct 21, 2024): On windows, that doesn't resolv the probleme. And that was not better on linux. I have rollback on 29.4 that work

saavagebueno commented

@mlsmaycon commented on GitHub (Oct 21, 2024):

On windows, that doesn't resolv the probleme. And that was not better on linux.

I have rollback on 29.4 that work

Thanks for the feedback @Eu-Arthur. Could you share a bit more about the tests that was performed and the behavior seem?

@mlsmaycon commented on GitHub (Oct 21, 2024): > On windows, that doesn't resolv the probleme. And that was not better on linux. > > I have rollback on 29.4 that work Thanks for the feedback @Eu-Arthur. Could you share a bit more about the tests that was performed and the behavior seem?

saavagebueno commented

@trbutler commented on GitHub (Oct 22, 2024):

Reference in new is

Download https://github.com/netbirdio/netbird/actions/runs/11405814951/artifacts/2074696862 (you need to be logged in to github).

One addition to get it working: I had to run xattr -dr com.apple.quarantine ~/Downloads/macos-packages/netbird_darwin_all/netbird . It's running at the moment; I'll report results back!

@trbutler commented on GitHub (Oct 22, 2024): > Reference in new is > Download https://github.com/netbirdio/netbird/actions/runs/11405814951/artifacts/2074696862 (you need to be logged in to github). One addition to get it working: I had to run `xattr -dr com.apple.quarantine ~/Downloads/macos-packages/netbird_darwin_all/netbird` . It's running at the moment; I'll report results back!

saavagebueno commented

@paularlott commented on GitHub (Oct 22, 2024):

On macOS Sequoia that PR appears to be solving the reconnect after wake

@paularlott commented on GitHub (Oct 22, 2024): On macOS Sequoia that PR appears to be solving the reconnect after wake

saavagebueno commented

@trbutler commented on GitHub (Oct 22, 2024):

Prior to the PR, I found that usually there'd only be 2/10 online peers connected after wake. After the PR, the system woke this morning to 4/10 peers. All the available peers were specifically in one location; the other 6 or the 10 online peers came online when I ran netbird down && netbird up.

Last night:

timothybutler@falstaff .ssh % netbird status -d
Peers detail:
 iphone-admin.anondomain2.tld:
  NetBird IP: 100.68.26.148/32
  Public key: S+4WdLdmo6im05ndW2ccPhdoQMVa7lvmQ4nXSMvIPHQ=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 1 hour ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 ipad-admin.anondomain2.tld:
  NetBird IP: 100.68.119.15/32
  Public key: oBnIFFtgnPA6SDOJcSHlR1BbIM5h7WudevvgmkDRLzQ=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 1 hour ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 ipad-admin-1.anondomain2.tld:
  NetBird IP: 100.68.95.42/32
  Public key: gYqQ1/ZrgCUzS2zKLuZJS7dHfb0SoS3YJaUKiPd3vQk=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 1 hour ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 l-h-live-stream.anondomain2.tld:
  NetBird IP: 100.68.40.244
  Public key: yBhdVf0uxhuvaAr4tVbFDLUWnTWg/JCOviH5T3KmphM=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/prflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:1740
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 11 minutes, 10 seconds ago
  Last WireGuard handshake: 1 minute, 14 seconds ago
  Transfer status (received/sent) 2.0 KiB/1.7 KiB
  Quantum resistance: true
  Routes: -
  Latency: 25.407083ms

 washington.anondomain2.tld:
  NetBird IP: 100.68.98.212
  Public key: KQIjQLtUaZM9J30rBp2AxHC4nrvn8neHA7Vg1DURkFg=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 franklin.anondomain2.tld:
  NetBird IP: 100.68.104.177
  Public key: v9F8qsB+L4fpvuTv9B8NiD27cx6h6dzVMC0XBwtw4WA=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:51820
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 59 minutes, 54 seconds ago
  Last WireGuard handshake: 1 minute, 32 seconds ago
  Transfer status (received/sent) 2.9 KiB/9.6 KiB
  Quantum resistance: true
  Routes: 192.168.4.0/22
  Latency: 20.697375ms

 barnabas.anondomain2.tld:
  NetBird IP: 100.68.127.149
  Public key: 8m/wrrUqMHd86sjWIIxQVpL9LH+hh7qHpRDZb1rx2TE=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:1729
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 59 minutes, 54 seconds ago
  Last WireGuard handshake: 1 minute, 2 seconds ago
  Transfer status (received/sent) 12.6 MiB/3.6 MiB
  Quantum resistance: true
  Routes: -
  Latency: 26.001625ms

 independence.anondomain2.tld:
  NetBird IP: 100.68.136.48
  Public key: wwROJuAi9t5d7W8DnF78sdMTm13iDZ9YcrtjjHtIYDM=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/z.z.z.z:1944
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 59 minutes, 53 seconds ago
  Last WireGuard handshake: 1 minute, 32 seconds ago
  Transfer status (received/sent) 2.7 KiB/9.6 KiB
  Quantum resistance: true
  Routes: -
  Latency: 32.979208ms

 spruce.anondomain2.tld:
  NetBird IP: 100.68.172.123
  Public key: 0sA1GjrlFs+yPKlh7CARYIoFA/Ydsa4Tq/jnpLw1axk=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/host
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/a.a.a.a:51820
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 59 minutes, 55 seconds ago
  Last WireGuard handshake: 1 minute, 32 seconds ago
  Transfer status (received/sent) 2.7 KiB/9.6 KiB
  Quantum resistance: true
  Routes: -
  Latency: 69.756291ms

 boaz.anondomain2.tld:
  NetBird IP: 100.68.191.18
  Public key: ydFumIBVUwCGBjx5Xh0pZPW1G6kFq2v+8DPNz1XYkRE=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/prflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:1739
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 5 minutes, 44 seconds ago
  Last WireGuard handshake: 1 minute, 35 seconds ago
  Transfer status (received/sent) 532 B/956 B
  Quantum resistance: true
  Routes: -
  Latency: 25.419666ms

 mesquite.anondomain2.tld:
  NetBird IP: 100.68.191.220
  Public key: vp6GLJc22GQXj2Ht5deowZp0OA8kG7XJS1kYl3zc6lI=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/b.b.b.b:51820
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 59 minutes, 54 seconds ago
  Last WireGuard handshake: 45 seconds ago
  Transfer status (received/sent) 3.1 KiB/10.2 KiB
  Quantum resistance: true
  Routes: -
  Latency: 61.260667ms

 touchstone.anondomain2.tld:
  NetBird IP: 100.68.218.63
  Public key: PAzQjGnO5xftL4rgeX9SdkajCjEJA3A+iViMbXoPgXE=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:1629
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 59 minutes, 53 seconds ago
  Last WireGuard handshake: 1 minute, 32 seconds ago
  Transfer status (received/sent) 2.6 KiB/9.6 KiB
  Quantum resistance: true
  Routes: -
  Latency: 22.926417ms

 rahab.anondomain2.tld:
  NetBird IP: 100.68.228.94
  Public key: hxczQ9TIXjpDAFHDVzwjH6aDPlC5l5GcTj0LEmhgfRQ=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:51820
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 21 minutes, 1 seconds ago
  Last WireGuard handshake: 1 minute, 34 seconds ago
  Transfer status (received/sent) 2.6 KiB/9.6 KiB
  Quantum resistance: true
  Routes: 192.168.10.0/24
  Latency: 27.587125ms

 madison.anondomain2.tld:
  NetBird IP: 100.68.238.72
  Public key: yIrcqYArbRwL8uN53oDXk8O2VPSerCd1dotme+q7uzQ=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 rosalind.anondomain2.tld:
  NetBird IP: 100.68.249.136
  Public key: 8ulzaG4yTm9RqIYMwRQXkw4LB7LDdhXy1ocdNCuEqBA=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:2242
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 59 minutes, 54 seconds ago
  Last WireGuard handshake: 1 minute, 26 seconds ago
  Transfer status (received/sent) 6.7 KiB/9.4 KiB
  Quantum resistance: true
  Routes: -
  Latency: 23.55075ms

OS: darwin/arm64
Daemon version: 0.30.2-SNAPSHOT-159f3f8b
CLI version: 0.30.2-SNAPSHOT-159f3f8b
Management: Connected to https://sugarmaple.anondomain1.tld:443
Signal: Connected to https://sugarmaple.anondomain1.tld:443
Relays: 
  [stun:sugarmaple.anondomain1.tld:3478] is Available
  [turn:sugarmaple.anondomain1.tld:3478?transport=udp] is Available
  [rels://sugarmaple.anondomain1.tld:443] is Available
Nameservers: 
FQDN: falstaff.anondomain2.tld
NetBird IP: 100.68.36.98/16
Interface type: Userspace
Quantum resistance: true (permissive)
Routes: -
Peers count: 10/15 Connected

In the morning:

timothybutler@falstaff .ssh % netbird status -d
Peers detail:
 iphone-admin.anondomain2.tld:
  NetBird IP: 100.68.26.148/32
  Public key: S+4WdLdmo6im05ndW2ccPhdoQMVa7lvmQ4nXSMvIPHQ=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 10 hours, 16 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 ipad-admin.anondomain2.tld:
  NetBird IP: 100.68.119.15/32
  Public key: oBnIFFtgnPA6SDOJcSHlR1BbIM5h7WudevvgmkDRLzQ=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 10 hours, 16 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 ipad-admin-1.anondomain2.tld:
  NetBird IP: 100.68.95.42/32
  Public key: gYqQ1/ZrgCUzS2zKLuZJS7dHfb0SoS3YJaUKiPd3vQk=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 10 hours, 16 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 l-h-live-stream.anondomain2.tld:
  NetBird IP: 100.68.40.244
  Public key: yBhdVf0uxhuvaAr4tVbFDLUWnTWg/JCOviH5T3KmphM=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 3 hours, 38 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: true
  Routes: -
  Latency: 31.308292ms

 washington.anondomain2.tld:
  NetBird IP: 100.68.98.212
  Public key: KQIjQLtUaZM9J30rBp2AxHC4nrvn8neHA7Vg1DURkFg=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 franklin.anondomain2.tld:
  NetBird IP: 100.68.104.177
  Public key: v9F8qsB+L4fpvuTv9B8NiD27cx6h6dzVMC0XBwtw4WA=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 6 hours, 21 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: true
  Routes: -
  Latency: 25.342166ms

 barnabas.anondomain2.tld:
  NetBird IP: 100.68.127.149
  Public key: 8m/wrrUqMHd86sjWIIxQVpL9LH+hh7qHpRDZb1rx2TE=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 8 hours, 24 minutes ago
  Last WireGuard handshake: 42 seconds ago
  Transfer status (received/sent) 12.7 MiB/3.6 MiB
  Quantum resistance: true
  Routes: -
  Latency: 24.913083ms

 independence.anondomain2.tld:
  NetBird IP: 100.68.136.48
  Public key: wwROJuAi9t5d7W8DnF78sdMTm13iDZ9YcrtjjHtIYDM=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 4 hours, 26 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: true
  Routes: -
  Latency: 21.285875ms

 spruce.anondomain2.tld:
  NetBird IP: 100.68.172.123
  Public key: 0sA1GjrlFs+yPKlh7CARYIoFA/Ydsa4Tq/jnpLw1axk=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 8 hours, 41 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: true
  Routes: -
  Latency: 79.169791ms

 boaz.anondomain2.tld:
  NetBird IP: 100.68.191.18
  Public key: ydFumIBVUwCGBjx5Xh0pZPW1G6kFq2v+8DPNz1XYkRE=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:1756
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 11 minutes, 40 seconds ago
  Last WireGuard handshake: 46 seconds ago
  Transfer status (received/sent) 1.2 KiB/1.9 KiB
  Quantum resistance: true
  Routes: -
  Latency: 23.269084ms

 mesquite.anondomain2.tld:
  NetBird IP: 100.68.191.220
  Public key: vp6GLJc22GQXj2Ht5deowZp0OA8kG7XJS1kYl3zc6lI=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 6 hours, 56 minutes ago
  Last WireGuard handshake: 41 seconds ago
  Transfer status (received/sent) 7.7 KiB/16.4 KiB
  Quantum resistance: true
  Routes: -
  Latency: 61.264333ms

 touchstone.anondomain2.tld:
  NetBird IP: 100.68.218.63
  Public key: PAzQjGnO5xftL4rgeX9SdkajCjEJA3A+iViMbXoPgXE=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 2 hours, 38 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: true
  Routes: -
  Latency: 30.977667ms

 rahab.anondomain2.tld:
  NetBird IP: 100.68.228.94
  Public key: hxczQ9TIXjpDAFHDVzwjH6aDPlC5l5GcTj0LEmhgfRQ=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:51820
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 14 minutes, 47 seconds ago
  Last WireGuard handshake: 8 seconds ago
  Transfer status (received/sent) 800 B/2.6 KiB
  Quantum resistance: true
  Routes: 192.168.10.0/24
  Latency: 28.161333ms

 madison.anondomain2.tld:
  NetBird IP: 100.68.238.72
  Public key: yIrcqYArbRwL8uN53oDXk8O2VPSerCd1dotme+q7uzQ=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 rosalind.anondomain2.tld:
  NetBird IP: 100.68.249.136
  Public key: 8ulzaG4yTm9RqIYMwRQXkw4LB7LDdhXy1ocdNCuEqBA=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 4 hours, 26 minutes ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: true
  Routes: -
  Latency: 28.039375ms

OS: darwin/arm64
Daemon version: 0.30.2-SNAPSHOT-159f3f8b
CLI version: 0.30.2-SNAPSHOT-159f3f8b
Management: Connected to https://sugarmaple.anondomain1.tld:443
Signal: Connected to https://sugarmaple.anondomain1.tld:443
Relays: 
  [stun:sugarmaple.anondomain1.tld:3478] is Available
  [turn:sugarmaple.anondomain1.tld:3478?transport=udp] is Available
  [rels://sugarmaple.anondomain1.tld:443] is Available
Nameservers: 
FQDN: falstaff.anondomain2.tld
NetBird IP: 100.68.36.98/16
Interface type: Userspace
Quantum resistance: true (permissive)
Routes: -
Peers count: 4/15 Connected

Restarting netbird and then trying again:

timothybutler@falstaff .ssh % netbird down && netbird up
Disconnected
Connected
timothybutler@falstaff .ssh % netbird status -d          
Peers detail:
 iphone-admin.anondomain2.tld:
  NetBird IP: 100.68.26.148/32
  Public key: S+4WdLdmo6im05ndW2ccPhdoQMVa7lvmQ4nXSMvIPHQ=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 1 minute, 27 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 ipad-admin.anondomain2.tld:
  NetBird IP: 100.68.119.15/32
  Public key: oBnIFFtgnPA6SDOJcSHlR1BbIM5h7WudevvgmkDRLzQ=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 1 minute, 27 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 ipad-admin-1.anondomain2.tld:
  NetBird IP: 100.68.95.42/32
  Public key: gYqQ1/ZrgCUzS2zKLuZJS7dHfb0SoS3YJaUKiPd3vQk=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: 1 minute, 27 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 l-h-live-stream.anondomain2.tld:
  NetBird IP: 100.68.40.244
  Public key: yBhdVf0uxhuvaAr4tVbFDLUWnTWg/JCOviH5T3KmphM=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:1749
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 6 seconds ago
  Last WireGuard handshake: 57 seconds ago
  Transfer status (received/sent) 484 B/692 B
  Quantum resistance: true
  Routes: -
  Latency: 31.515541ms

 washington.anondomain2.tld:
  NetBird IP: 100.68.98.212
  Public key: KQIjQLtUaZM9J30rBp2AxHC4nrvn8neHA7Vg1DURkFg=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 franklin.anondomain2.tld:
  NetBird IP: 100.68.104.177
  Public key: v9F8qsB+L4fpvuTv9B8NiD27cx6h6dzVMC0XBwtw4WA=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:51820
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 6 seconds ago
  Last WireGuard handshake: 1 minute, 7 seconds ago
  Transfer status (received/sent) 92 B/244 B
  Quantum resistance: true
  Routes: -
  Latency: 22.597417ms

 barnabas.anondomain2.tld:
  NetBird IP: 100.68.127.149
  Public key: 8m/wrrUqMHd86sjWIIxQVpL9LH+hh7qHpRDZb1rx2TE=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:1729
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 6 seconds ago
  Last WireGuard handshake: 1 minute, 7 seconds ago
  Transfer status (received/sent) 124 B/244 B
  Quantum resistance: true
  Routes: -
  Latency: 34.711458ms

 independence.anondomain2.tld:
  NetBird IP: 100.68.136.48
  Public key: wwROJuAi9t5d7W8DnF78sdMTm13iDZ9YcrtjjHtIYDM=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/z.z.z.z:1944
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 15 seconds ago
  Last WireGuard handshake: 1 minute, 16 seconds ago
  Transfer status (received/sent) 92 B/276 B
  Quantum resistance: true
  Routes: 192.168.4.0/22
  Latency: 25.726583ms

 spruce.anondomain2.tld:
  NetBird IP: 100.68.172.123
  Public key: 0sA1GjrlFs+yPKlh7CARYIoFA/Ydsa4Tq/jnpLw1axk=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/a.a.a.a:51820
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 6 seconds ago
  Last WireGuard handshake: 1 minute, 6 seconds ago
  Transfer status (received/sent) 124 B/244 B
  Quantum resistance: true
  Routes: -
  Latency: 70.688166ms

 boaz.anondomain2.tld:
  NetBird IP: 100.68.191.18
  Public key: ydFumIBVUwCGBjx5Xh0pZPW1G6kFq2v+8DPNz1XYkRE=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:1756
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 5 seconds ago
  Last WireGuard handshake: 1 minute, 8 seconds ago
  Transfer status (received/sent) 124 B/244 B
  Quantum resistance: true
  Routes: -
  Latency: 29.858375ms

 mesquite.anondomain2.tld:
  NetBird IP: 100.68.191.220
  Public key: vp6GLJc22GQXj2Ht5deowZp0OA8kG7XJS1kYl3zc6lI=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/b.b.b.b:51820
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 5 seconds ago
  Last WireGuard handshake: 1 minute, 6 seconds ago
  Transfer status (received/sent) 92 B/244 B
  Quantum resistance: true
  Routes: -
  Latency: 81.342833ms

 touchstone.anondomain2.tld:
  NetBird IP: 100.68.218.63
  Public key: PAzQjGnO5xftL4rgeX9SdkajCjEJA3A+iViMbXoPgXE=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/z.z.z.z:1629
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 14 seconds ago
  Last WireGuard handshake: 1 minute, 15 seconds ago
  Transfer status (received/sent) 124 B/276 B
  Quantum resistance: true
  Routes: -
  Latency: 34.070333ms

 rahab.anondomain2.tld:
  NetBird IP: 100.68.228.94
  Public key: hxczQ9TIXjpDAFHDVzwjH6aDPlC5l5GcTj0LEmhgfRQ=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/srflx
  ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:51820
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 5 seconds ago
  Last WireGuard handshake: 1 minute, 8 seconds ago
  Transfer status (received/sent) 92 B/244 B
  Quantum resistance: true
  Routes: 192.168.10.0/24
  Latency: 28.613125ms

 madison.anondomain2.tld:
  NetBird IP: 100.68.238.72
  Public key: yIrcqYArbRwL8uN53oDXk8O2VPSerCd1dotme+q7uzQ=
  Status: Disconnected
  -- detail --
  Connection type: 
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: 
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false (remote didn't enable quantum resistance)
  Routes: -
  Latency: 0s

 rosalind.anondomain2.tld:
  NetBird IP: 100.68.249.136
  Public key: 8ulzaG4yTm9RqIYMwRQXkw4LB7LDdhXy1ocdNCuEqBA=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:2242
  Relay server address: rels://sugarmaple.anondomain1.tld:443
  Last connection update: 1 minute, 6 seconds ago
  Last WireGuard handshake: 1 minute, 7 seconds ago
  Transfer status (received/sent) 188 B/180 B
  Quantum resistance: true
  Routes: -
  Latency: 23.094ms

OS: darwin/arm64
Daemon version: 0.30.2-SNAPSHOT-159f3f8b
CLI version: 0.30.2-SNAPSHOT-159f3f8b
Management: Connected to https://sugarmaple.anondomain1.tld:443
Signal: Connected to https://sugarmaple.anondomain1.tld:443
Relays: 
  [stun:sugarmaple.anondomain1.tld:3478] is Available
  [turn:sugarmaple.anondomain1.tld:3478?transport=udp] is Available
  [rels://sugarmaple.anondomain1.tld:443] is Available
Nameservers: 
FQDN: falstaff.anondomain2.tld
NetBird IP: 100.68.36.98/16
Interface type: Userspace
Quantum resistance: true (permissive)
Routes: -
Peers count: 10/15 Connected
timothybutler@falstaff .ssh %

@trbutler commented on GitHub (Oct 22, 2024): Prior to the PR, I found that usually there'd only be 2/10 online peers connected after wake. After the PR, the system woke this morning to 4/10 peers. All the available peers were specifically in one location; the other 6 or the 10 online peers came online when I ran `netbird down && netbird up`. Last night: ```` timothybutler@falstaff .ssh % netbird status -d Peers detail: iphone-admin.anondomain2.tld: NetBird IP: 100.68.26.148/32 Public key: S+4WdLdmo6im05ndW2ccPhdoQMVa7lvmQ4nXSMvIPHQ= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 1 hour ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s ipad-admin.anondomain2.tld: NetBird IP: 100.68.119.15/32 Public key: oBnIFFtgnPA6SDOJcSHlR1BbIM5h7WudevvgmkDRLzQ= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 1 hour ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s ipad-admin-1.anondomain2.tld: NetBird IP: 100.68.95.42/32 Public key: gYqQ1/ZrgCUzS2zKLuZJS7dHfb0SoS3YJaUKiPd3vQk= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 1 hour ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s l-h-live-stream.anondomain2.tld: NetBird IP: 100.68.40.244 Public key: yBhdVf0uxhuvaAr4tVbFDLUWnTWg/JCOviH5T3KmphM= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/prflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:1740 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 11 minutes, 10 seconds ago Last WireGuard handshake: 1 minute, 14 seconds ago Transfer status (received/sent) 2.0 KiB/1.7 KiB Quantum resistance: true Routes: - Latency: 25.407083ms washington.anondomain2.tld: NetBird IP: 100.68.98.212 Public key: KQIjQLtUaZM9J30rBp2AxHC4nrvn8neHA7Vg1DURkFg= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s franklin.anondomain2.tld: NetBird IP: 100.68.104.177 Public key: v9F8qsB+L4fpvuTv9B8NiD27cx6h6dzVMC0XBwtw4WA= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:51820 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 59 minutes, 54 seconds ago Last WireGuard handshake: 1 minute, 32 seconds ago Transfer status (received/sent) 2.9 KiB/9.6 KiB Quantum resistance: true Routes: 192.168.4.0/22 Latency: 20.697375ms barnabas.anondomain2.tld: NetBird IP: 100.68.127.149 Public key: 8m/wrrUqMHd86sjWIIxQVpL9LH+hh7qHpRDZb1rx2TE= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:1729 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 59 minutes, 54 seconds ago Last WireGuard handshake: 1 minute, 2 seconds ago Transfer status (received/sent) 12.6 MiB/3.6 MiB Quantum resistance: true Routes: - Latency: 26.001625ms independence.anondomain2.tld: NetBird IP: 100.68.136.48 Public key: wwROJuAi9t5d7W8DnF78sdMTm13iDZ9YcrtjjHtIYDM= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/z.z.z.z:1944 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 59 minutes, 53 seconds ago Last WireGuard handshake: 1 minute, 32 seconds ago Transfer status (received/sent) 2.7 KiB/9.6 KiB Quantum resistance: true Routes: - Latency: 32.979208ms spruce.anondomain2.tld: NetBird IP: 100.68.172.123 Public key: 0sA1GjrlFs+yPKlh7CARYIoFA/Ydsa4Tq/jnpLw1axk= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/host ICE candidate endpoints (Local/Remote): x.x.x.x:51820/a.a.a.a:51820 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 59 minutes, 55 seconds ago Last WireGuard handshake: 1 minute, 32 seconds ago Transfer status (received/sent) 2.7 KiB/9.6 KiB Quantum resistance: true Routes: - Latency: 69.756291ms boaz.anondomain2.tld: NetBird IP: 100.68.191.18 Public key: ydFumIBVUwCGBjx5Xh0pZPW1G6kFq2v+8DPNz1XYkRE= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/prflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:1739 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 5 minutes, 44 seconds ago Last WireGuard handshake: 1 minute, 35 seconds ago Transfer status (received/sent) 532 B/956 B Quantum resistance: true Routes: - Latency: 25.419666ms mesquite.anondomain2.tld: NetBird IP: 100.68.191.220 Public key: vp6GLJc22GQXj2Ht5deowZp0OA8kG7XJS1kYl3zc6lI= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/b.b.b.b:51820 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 59 minutes, 54 seconds ago Last WireGuard handshake: 45 seconds ago Transfer status (received/sent) 3.1 KiB/10.2 KiB Quantum resistance: true Routes: - Latency: 61.260667ms touchstone.anondomain2.tld: NetBird IP: 100.68.218.63 Public key: PAzQjGnO5xftL4rgeX9SdkajCjEJA3A+iViMbXoPgXE= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:1629 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 59 minutes, 53 seconds ago Last WireGuard handshake: 1 minute, 32 seconds ago Transfer status (received/sent) 2.6 KiB/9.6 KiB Quantum resistance: true Routes: - Latency: 22.926417ms rahab.anondomain2.tld: NetBird IP: 100.68.228.94 Public key: hxczQ9TIXjpDAFHDVzwjH6aDPlC5l5GcTj0LEmhgfRQ= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:51820 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 21 minutes, 1 seconds ago Last WireGuard handshake: 1 minute, 34 seconds ago Transfer status (received/sent) 2.6 KiB/9.6 KiB Quantum resistance: true Routes: 192.168.10.0/24 Latency: 27.587125ms madison.anondomain2.tld: NetBird IP: 100.68.238.72 Public key: yIrcqYArbRwL8uN53oDXk8O2VPSerCd1dotme+q7uzQ= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s rosalind.anondomain2.tld: NetBird IP: 100.68.249.136 Public key: 8ulzaG4yTm9RqIYMwRQXkw4LB7LDdhXy1ocdNCuEqBA= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:2242 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 59 minutes, 54 seconds ago Last WireGuard handshake: 1 minute, 26 seconds ago Transfer status (received/sent) 6.7 KiB/9.4 KiB Quantum resistance: true Routes: - Latency: 23.55075ms OS: darwin/arm64 Daemon version: 0.30.2-SNAPSHOT-159f3f8b CLI version: 0.30.2-SNAPSHOT-159f3f8b Management: Connected to https://sugarmaple.anondomain1.tld:443 Signal: Connected to https://sugarmaple.anondomain1.tld:443 Relays: [stun:sugarmaple.anondomain1.tld:3478] is Available [turn:sugarmaple.anondomain1.tld:3478?transport=udp] is Available [rels://sugarmaple.anondomain1.tld:443] is Available Nameservers: FQDN: falstaff.anondomain2.tld NetBird IP: 100.68.36.98/16 Interface type: Userspace Quantum resistance: true (permissive) Routes: - Peers count: 10/15 Connected ```` In the morning: ```` timothybutler@falstaff .ssh % netbird status -d Peers detail: iphone-admin.anondomain2.tld: NetBird IP: 100.68.26.148/32 Public key: S+4WdLdmo6im05ndW2ccPhdoQMVa7lvmQ4nXSMvIPHQ= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 10 hours, 16 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s ipad-admin.anondomain2.tld: NetBird IP: 100.68.119.15/32 Public key: oBnIFFtgnPA6SDOJcSHlR1BbIM5h7WudevvgmkDRLzQ= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 10 hours, 16 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s ipad-admin-1.anondomain2.tld: NetBird IP: 100.68.95.42/32 Public key: gYqQ1/ZrgCUzS2zKLuZJS7dHfb0SoS3YJaUKiPd3vQk= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 10 hours, 16 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s l-h-live-stream.anondomain2.tld: NetBird IP: 100.68.40.244 Public key: yBhdVf0uxhuvaAr4tVbFDLUWnTWg/JCOviH5T3KmphM= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 3 hours, 38 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: true Routes: - Latency: 31.308292ms washington.anondomain2.tld: NetBird IP: 100.68.98.212 Public key: KQIjQLtUaZM9J30rBp2AxHC4nrvn8neHA7Vg1DURkFg= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s franklin.anondomain2.tld: NetBird IP: 100.68.104.177 Public key: v9F8qsB+L4fpvuTv9B8NiD27cx6h6dzVMC0XBwtw4WA= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 6 hours, 21 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: true Routes: - Latency: 25.342166ms barnabas.anondomain2.tld: NetBird IP: 100.68.127.149 Public key: 8m/wrrUqMHd86sjWIIxQVpL9LH+hh7qHpRDZb1rx2TE= Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 8 hours, 24 minutes ago Last WireGuard handshake: 42 seconds ago Transfer status (received/sent) 12.7 MiB/3.6 MiB Quantum resistance: true Routes: - Latency: 24.913083ms independence.anondomain2.tld: NetBird IP: 100.68.136.48 Public key: wwROJuAi9t5d7W8DnF78sdMTm13iDZ9YcrtjjHtIYDM= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 4 hours, 26 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: true Routes: - Latency: 21.285875ms spruce.anondomain2.tld: NetBird IP: 100.68.172.123 Public key: 0sA1GjrlFs+yPKlh7CARYIoFA/Ydsa4Tq/jnpLw1axk= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 8 hours, 41 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: true Routes: - Latency: 79.169791ms boaz.anondomain2.tld: NetBird IP: 100.68.191.18 Public key: ydFumIBVUwCGBjx5Xh0pZPW1G6kFq2v+8DPNz1XYkRE= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/prflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:1756 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 11 minutes, 40 seconds ago Last WireGuard handshake: 46 seconds ago Transfer status (received/sent) 1.2 KiB/1.9 KiB Quantum resistance: true Routes: - Latency: 23.269084ms mesquite.anondomain2.tld: NetBird IP: 100.68.191.220 Public key: vp6GLJc22GQXj2Ht5deowZp0OA8kG7XJS1kYl3zc6lI= Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 6 hours, 56 minutes ago Last WireGuard handshake: 41 seconds ago Transfer status (received/sent) 7.7 KiB/16.4 KiB Quantum resistance: true Routes: - Latency: 61.264333ms touchstone.anondomain2.tld: NetBird IP: 100.68.218.63 Public key: PAzQjGnO5xftL4rgeX9SdkajCjEJA3A+iViMbXoPgXE= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 2 hours, 38 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: true Routes: - Latency: 30.977667ms rahab.anondomain2.tld: NetBird IP: 100.68.228.94 Public key: hxczQ9TIXjpDAFHDVzwjH6aDPlC5l5GcTj0LEmhgfRQ= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:51820 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 14 minutes, 47 seconds ago Last WireGuard handshake: 8 seconds ago Transfer status (received/sent) 800 B/2.6 KiB Quantum resistance: true Routes: 192.168.10.0/24 Latency: 28.161333ms madison.anondomain2.tld: NetBird IP: 100.68.238.72 Public key: yIrcqYArbRwL8uN53oDXk8O2VPSerCd1dotme+q7uzQ= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s rosalind.anondomain2.tld: NetBird IP: 100.68.249.136 Public key: 8ulzaG4yTm9RqIYMwRQXkw4LB7LDdhXy1ocdNCuEqBA= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 4 hours, 26 minutes ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: true Routes: - Latency: 28.039375ms OS: darwin/arm64 Daemon version: 0.30.2-SNAPSHOT-159f3f8b CLI version: 0.30.2-SNAPSHOT-159f3f8b Management: Connected to https://sugarmaple.anondomain1.tld:443 Signal: Connected to https://sugarmaple.anondomain1.tld:443 Relays: [stun:sugarmaple.anondomain1.tld:3478] is Available [turn:sugarmaple.anondomain1.tld:3478?transport=udp] is Available [rels://sugarmaple.anondomain1.tld:443] is Available Nameservers: FQDN: falstaff.anondomain2.tld NetBird IP: 100.68.36.98/16 Interface type: Userspace Quantum resistance: true (permissive) Routes: - Peers count: 4/15 Connected ```` Restarting netbird and then trying again: ```` timothybutler@falstaff .ssh % netbird down && netbird up Disconnected Connected timothybutler@falstaff .ssh % netbird status -d Peers detail: iphone-admin.anondomain2.tld: NetBird IP: 100.68.26.148/32 Public key: S+4WdLdmo6im05ndW2ccPhdoQMVa7lvmQ4nXSMvIPHQ= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 1 minute, 27 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s ipad-admin.anondomain2.tld: NetBird IP: 100.68.119.15/32 Public key: oBnIFFtgnPA6SDOJcSHlR1BbIM5h7WudevvgmkDRLzQ= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 1 minute, 27 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s ipad-admin-1.anondomain2.tld: NetBird IP: 100.68.95.42/32 Public key: gYqQ1/ZrgCUzS2zKLuZJS7dHfb0SoS3YJaUKiPd3vQk= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: 1 minute, 27 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s l-h-live-stream.anondomain2.tld: NetBird IP: 100.68.40.244 Public key: yBhdVf0uxhuvaAr4tVbFDLUWnTWg/JCOviH5T3KmphM= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:1749 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 6 seconds ago Last WireGuard handshake: 57 seconds ago Transfer status (received/sent) 484 B/692 B Quantum resistance: true Routes: - Latency: 31.515541ms washington.anondomain2.tld: NetBird IP: 100.68.98.212 Public key: KQIjQLtUaZM9J30rBp2AxHC4nrvn8neHA7Vg1DURkFg= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s franklin.anondomain2.tld: NetBird IP: 100.68.104.177 Public key: v9F8qsB+L4fpvuTv9B8NiD27cx6h6dzVMC0XBwtw4WA= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:51820 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 6 seconds ago Last WireGuard handshake: 1 minute, 7 seconds ago Transfer status (received/sent) 92 B/244 B Quantum resistance: true Routes: - Latency: 22.597417ms barnabas.anondomain2.tld: NetBird IP: 100.68.127.149 Public key: 8m/wrrUqMHd86sjWIIxQVpL9LH+hh7qHpRDZb1rx2TE= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/y.y.y.y:1729 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 6 seconds ago Last WireGuard handshake: 1 minute, 7 seconds ago Transfer status (received/sent) 124 B/244 B Quantum resistance: true Routes: - Latency: 34.711458ms independence.anondomain2.tld: NetBird IP: 100.68.136.48 Public key: wwROJuAi9t5d7W8DnF78sdMTm13iDZ9YcrtjjHtIYDM= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/z.z.z.z:1944 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 15 seconds ago Last WireGuard handshake: 1 minute, 16 seconds ago Transfer status (received/sent) 92 B/276 B Quantum resistance: true Routes: 192.168.4.0/22 Latency: 25.726583ms spruce.anondomain2.tld: NetBird IP: 100.68.172.123 Public key: 0sA1GjrlFs+yPKlh7CARYIoFA/Ydsa4Tq/jnpLw1axk= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/host ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/a.a.a.a:51820 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 6 seconds ago Last WireGuard handshake: 1 minute, 6 seconds ago Transfer status (received/sent) 124 B/244 B Quantum resistance: true Routes: - Latency: 70.688166ms boaz.anondomain2.tld: NetBird IP: 100.68.191.18 Public key: ydFumIBVUwCGBjx5Xh0pZPW1G6kFq2v+8DPNz1XYkRE= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:1756 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 5 seconds ago Last WireGuard handshake: 1 minute, 8 seconds ago Transfer status (received/sent) 124 B/244 B Quantum resistance: true Routes: - Latency: 29.858375ms mesquite.anondomain2.tld: NetBird IP: 100.68.191.220 Public key: vp6GLJc22GQXj2Ht5deowZp0OA8kG7XJS1kYl3zc6lI= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/b.b.b.b:51820 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 5 seconds ago Last WireGuard handshake: 1 minute, 6 seconds ago Transfer status (received/sent) 92 B/244 B Quantum resistance: true Routes: - Latency: 81.342833ms touchstone.anondomain2.tld: NetBird IP: 100.68.218.63 Public key: PAzQjGnO5xftL4rgeX9SdkajCjEJA3A+iViMbXoPgXE= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/z.z.z.z:1629 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 14 seconds ago Last WireGuard handshake: 1 minute, 15 seconds ago Transfer status (received/sent) 124 B/276 B Quantum resistance: true Routes: - Latency: 34.070333ms rahab.anondomain2.tld: NetBird IP: 100.68.228.94 Public key: hxczQ9TIXjpDAFHDVzwjH6aDPlC5l5GcTj0LEmhgfRQ= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/srflx ICE candidate endpoints (Local/Remote): x.x.x.x:51820/y.y.y.y:51820 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 5 seconds ago Last WireGuard handshake: 1 minute, 8 seconds ago Transfer status (received/sent) 92 B/244 B Quantum resistance: true Routes: 192.168.10.0/24 Latency: 28.613125ms madison.anondomain2.tld: NetBird IP: 100.68.238.72 Public key: yIrcqYArbRwL8uN53oDXk8O2VPSerCd1dotme+q7uzQ= Status: Disconnected -- detail -- Connection type: ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false (remote didn't enable quantum resistance) Routes: - Latency: 0s rosalind.anondomain2.tld: NetBird IP: 100.68.249.136 Public key: 8ulzaG4yTm9RqIYMwRQXkw4LB7LDdhXy1ocdNCuEqBA= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.0.97:51820/z.z.z.z:2242 Relay server address: rels://sugarmaple.anondomain1.tld:443 Last connection update: 1 minute, 6 seconds ago Last WireGuard handshake: 1 minute, 7 seconds ago Transfer status (received/sent) 188 B/180 B Quantum resistance: true Routes: - Latency: 23.094ms OS: darwin/arm64 Daemon version: 0.30.2-SNAPSHOT-159f3f8b CLI version: 0.30.2-SNAPSHOT-159f3f8b Management: Connected to https://sugarmaple.anondomain1.tld:443 Signal: Connected to https://sugarmaple.anondomain1.tld:443 Relays: [stun:sugarmaple.anondomain1.tld:3478] is Available [turn:sugarmaple.anondomain1.tld:3478?transport=udp] is Available [rels://sugarmaple.anondomain1.tld:443] is Available Nameservers: FQDN: falstaff.anondomain2.tld NetBird IP: 100.68.36.98/16 Interface type: Userspace Quantum resistance: true (permissive) Routes: - Peers count: 10/15 Connected timothybutler@falstaff .ssh % ````

saavagebueno commented

@paularlott commented on GitHub (Oct 22, 2024):

I commented too soon, this morning the mac only connected to 2 out of 57 peers, I had to restart netbird and then the other peers all showed as available.

@paularlott commented on GitHub (Oct 22, 2024): I commented too soon, this morning the mac only connected to 2 out of 57 peers, I had to restart netbird and then the other peers all showed as available.

saavagebueno commented

@Eu-Arthur commented on GitHub (Oct 25, 2024):

On windows, and linux the last update fix this

@Eu-Arthur commented on GitHub (Oct 25, 2024): On windows, and linux the last update fix this

saavagebueno commented

@paularlott commented on GitHub (Nov 12, 2024):

It seems the issue was resolved in 0.30.0 but back in 0.31.1, on the mac I have to restart netbird every time it wakes again.

@paularlott commented on GitHub (Nov 12, 2024): It seems the issue was resolved in 0.30.0 but back in 0.31.1, on the mac I have to restart netbird every time it wakes again.

saavagebueno commented

@mlsmaycon commented on GitHub (Nov 13, 2024):

@paularlott can you share the logs from:

/var/log/netbird and /var/log/netbird.err?

@mlsmaycon commented on GitHub (Nov 13, 2024): @paularlott can you share the logs from: /var/log/netbird and /var/log/netbird.err?

saavagebueno commented

@ser commented on GitHub (Nov 13, 2024):

With my setup, no single peer connects after waking from sleep. I even added alias to my zshrc:

alias nb=netbird down&&netbird up

i really recommend it!

@ser commented on GitHub (Nov 13, 2024): With my setup, no single peer connects after waking from sleep. I even added alias to my zshrc: ```alias nb=netbird down&&netbird up``` i really recommend it!

saavagebueno commented

@paularlott commented on GitHub (Nov 13, 2024):

@mlsmaycon I'm more than happy to share the logs privately but there's some things in there from my pushing netbird to replace all our VPNs that I'm not keen to share publicly, what's the best way to get an archive of them to you?

@paularlott commented on GitHub (Nov 13, 2024): @mlsmaycon I'm more than happy to share the logs privately but there's some things in there from my pushing netbird to replace all our VPNs that I'm not keen to share publicly, what's the best way to get an archive of them to you?

saavagebueno commented

@mlsmaycon commented on GitHub (Nov 14, 2024):

@mlsmaycon I'm more than happy to share the logs privately but there's some things in there from my pushing netbird to replace all our VPNs that I'm not keen to share publicly, what's the best way to get an archive of them to you?

Can you join our slack and share through there?

NetBird slack

@mlsmaycon commented on GitHub (Nov 14, 2024): > @mlsmaycon I'm more than happy to share the logs privately but there's some things in there from my pushing netbird to replace all our VPNs that I'm not keen to share publicly, what's the best way to get an archive of them to you? Can you join our slack and share through there? [NetBird slack](https://join.slack.com/t/netbirdio/shared_invite/zt-2p5zwhm4g-8fHollzrQa5y4PZF5AEpvQ)

saavagebueno commented

@mlsmaycon commented on GitHub (Nov 14, 2024):

@ser all your peers are in the latest 0.31.1 version? Can you share the same logs?

@mlsmaycon commented on GitHub (Nov 14, 2024): @ser all your peers are in the latest 0.31.1 version? Can you share the same logs?

saavagebueno commented

@mlsmaycon commented on GitHub (Nov 19, 2024):

Hello @paularlott. Thanks for sharing the logs in Slack. It seems like you’ve faced an issue with our state manager:

2024-11-14T16:24:23-03:00 ERRO client/internal/statemanager/manager.go:162: failed to persist state: context deadline exceeded

We’ve fixed this issue in the new 0.32.0 version

Can you try this version?

@mlsmaycon commented on GitHub (Nov 19, 2024): Hello @paularlott. Thanks for sharing the logs in Slack. It seems like you’ve faced an issue with our state manager: ``` 2024-11-14T16:24:23-03:00 ERRO client/internal/statemanager/manager.go:162: failed to persist state: context deadline exceeded ``` We’ve fixed this issue in the new 0.32.0 version Can you try this version?

saavagebueno commented

@trbutler commented on GitHub (Dec 9, 2024):

This issue seems to have regressed in the last release or two. I believe since 0.34.0, I've started noticing that some, but not all, peers become unavailable during sleep. Right now I can ping one system at my office, but not the system directly next to it. Sometimes network routes continue to work, but depending on which peers quit working, even HA routes can stop working because it seems to get stuck on one of the unavailable peers despite Netbird still being able to talk to one of the available peers.

As with earlier versions that exhibited this problem, netbird down && netbird up on the Mac that was sleeping resolves the issue.

Possible point on this: the system that remained available to the newly awakened Mac is a Linux system running Netbird 0.32.0. The ones that were unavailable until the Mac's Netbird was taken down and brought back up are running 0.34.0 or 0.34.1.

@trbutler commented on GitHub (Dec 9, 2024): This issue seems to have regressed in the last release or two. I believe since 0.34.0, I've started noticing that some, but not all, peers become unavailable during sleep. Right now I can ping one system at my office, but not the system directly next to it. Sometimes network routes continue to work, but depending on which peers quit working, even HA routes can stop working because it seems to get stuck on one of the unavailable peers despite Netbird still being able to talk to one of the available peers. As with earlier versions that exhibited this problem, `netbird down && netbird up` on the Mac that was sleeping resolves the issue. Possible point on this: the system that remained available to the newly awakened Mac is a Linux system running Netbird 0.32.0. The ones that were unavailable until the Mac's Netbird was taken down and brought back up are running 0.34.0 or 0.34.1.

saavagebueno commented

@trbutler commented on GitHub (Feb 21, 2025):

As I mentioned back in December, there seemed to be a regression after 0.32.0. If I keep my Linux servers on client version 0.32.0, I can reliably connect to them from my MacOS client that is running 0.36.7. However, as soon as I move the Debian systems to the latest Netbird, often the systems erratically disconnect even without going into sleep first. There's nothing in client.log that seems to evidence what is going on, but after netbird down && netbird up, the more recent (beyond 0.32.0) clients will be available, then vanish after a minute or two. Those rolled back to 0.32.0 continue to be available, however.

@trbutler commented on GitHub (Feb 21, 2025): As I mentioned back in December, there seemed to be a regression after 0.32.0. If I keep my Linux servers on client version 0.32.0, I can reliably connect to them from my MacOS client that is running 0.36.7. However, as soon as I move the Debian systems to the latest Netbird, often the systems erratically disconnect even without going into sleep first. There's nothing in client.log that seems to evidence what is going on, but after `netbird down && netbird up`, the more recent (beyond 0.32.0) clients will be available, then vanish after a minute or two. Those rolled back to 0.32.0 continue to be available, however.

saavagebueno commented

@trbutler commented on GitHub (Feb 22, 2025):

Continuing to try to locate the exact problem, I've noticed whenever I downgrade a system to 0.32.0 (something thereafter might be OK, but it is the version I've found consistent results with), I have to take Netbird down and back up my MacOS system with 0.36.7 before it will see the systems again. However, once I do that, the link becomes more reliable. I did notice if I do a roll back on a third machine, it may be that my system running 0.36.7 has to be restarted to see either remote host again.

The self-hosted installation of Netbird should be up-to-date -- I went through the upgrade process the middle of last week.

@trbutler commented on GitHub (Feb 22, 2025): Continuing to try to locate the exact problem, I've noticed whenever I downgrade a system to 0.32.0 (something thereafter might be OK, but it is the version I've found consistent results with), I have to take Netbird down and back up my MacOS system with 0.36.7 before it will see the systems again. However, once I do that, the link becomes more reliable. I did notice if I do a roll back on a third machine, it may be that my system running 0.36.7 has to be restarted to see _either_ remote host again. The self-hosted installation of Netbird should be up-to-date -- I went through the upgrade process the middle of last week.

saavagebueno commented

@trbutler commented on GitHub (Mar 3, 2025):

I'm trying to figure out if one client struggling has a profound impact on the other clients. What I have noticed if running 0.37.1 is that the problem seems to have gotten more extreme. On an M3 MacBook Air, on-wake Netbird's resource usage skyrockets as it gets stuck. This morning, it was taking 150% CPU on Activity Monitor and, as per usual with this situation, unable to communicate with other peers.

Here's some output from /var/log/netbird.out.log which seems to balloon to many gigabytes in size on my Mac peers, requiring me to remember to delete it or it will literally fill up the disk. The contents just repeat over and over:

time=2025-03-03T11:31:59.591-06:00 level=DEBUG msg="Handling message" type=InitHello
time=2025-03-03T11:31:59.672-06:00 level=DEBUG msg="Sending message" pid="DZfmMxU7KEkG2wA4Wm5btUTxoCaee1twjHGY/veuH5o=" type=RespHello
time=2025-03-03T11:31:59.674-06:00 level=DEBUG msg="Handling message" type=RespHello
time=2025-03-03T11:31:59.674-06:00 level=ERROR msg="Failed to handle message" error="session not found: f271a4d1"
time=2025-03-03T11:31:59.675-06:00 level=DEBUG msg="Handling message" type=InitHello
time=2025-03-03T11:31:59.755-06:00 level=DEBUG msg="Sending message" pid="DZfmMxU7KEkG2wA4Wm5btUTxoCaee1twjHGY/veuH5o=" type=RespHello
time=2025-03-03T11:31:59.757-06:00 level=DEBUG msg="Handling message" type=RespHello
time=2025-03-03T11:31:59.757-06:00 level=ERROR msg="Failed to handle message" error="session not found: 7d232fdd"
time=2025-03-03T11:31:59.758-06:00 level=DEBUG msg="Handling message" type=InitHello

Here's what the /var/netbird/client.log shows on the same system running the latest Netbird:

2025-03-03T11:30:36-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:510: free up connection to peer: sha-UW1BjKkhMDn3n5jsJSx2wLEF8soSW/Yu7L5c2p5mdWo=
2025-03-03T11:30:36-06:00 ERRO client/iface/wgproxy/bind/proxy.go:118: failed to read from remote conn: rels://[REDACTED].com:443, use of closed network connection
2025-03-03T11:30:45-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:214: open connection to peer: sha-UW1BjKkhMDn3n5jsJSx2wLEF8soSW/Yu7L5c2p5mdWo=
2025-03-03T11:30:45-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:446: created new wgProxy for relay connection: 127.1.88.48:51820
2025-03-03T11:30:45-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started
2025-03-03T11:30:45-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:475: start to communicate with peer via relay
2025-03-03T11:31:03-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection
2025-03-03T11:31:03-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped
2025-03-03T11:31:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection
2025-03-03T11:31:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started
2025-03-03T11:31:59-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection
2025-03-03T11:31:59-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped
2025-03-03T11:32:00-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer
2025-03-03T11:32:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection
2025-03-03T11:32:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started
2025-03-03T11:32:42-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection
2025-03-03T11:32:42-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped
2025-03-03T11:33:24-06:00 INFO client/internal/peer/guard/sr_watcher.go:94: network changes detected by ICE agent
2025-03-03T11:33:32-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection
2025-03-03T11:33:32-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started
2025-03-03T11:33:50-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection
2025-03-03T11:33:50-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped
2025-03-03T11:34:05-06:00 WARN [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/wg_watcher.go:135: WireGuard handshake timed out, closing relay connection: 1969-12-31 18:00:00 -0600 CST
2025-03-03T11:34:05-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:510: free up connection to peer: sha-NaHy5H143GjYGlDXvsAI7fj0oJLtlA9nBCJoD5KBV6c=
2025-03-03T11:34:05-06:00 ERRO client/iface/wgproxy/bind/proxy.go:118: failed to read from remote conn: rels://[REDACTED].com:443, use of closed network connection
2025-03-03T11:34:07-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:214: open connection to peer: sha-NaHy5H143GjYGlDXvsAI7fj0oJLtlA9nBCJoD5KBV6c=
2025-03-03T11:34:07-06:00 INFO [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/conn.go:446: created new wgProxy for relay connection: 127.1.126.239:51820
2025-03-03T11:34:07-06:00 INFO [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started
2025-03-03T11:34:07-06:00 INFO [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/conn.go:475: start to communicate with peer via relay
2025-03-03T11:34:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection
2025-03-03T11:34:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started
2025-03-03T11:35:03-06:00 WARN [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:129: WireGuard handshake timed out, closing relay connection: 1969-12-31 18:00:00 -0600 CST
2025-03-03T11:35:03-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:510: free up connection to peer: sha-UW1BjKkhMDn3n5jsJSx2wLEF8soSW/Yu7L5c2p5mdWo=
2025-03-03T11:35:03-06:00 ERRO client/iface/wgproxy/bind/proxy.go:118: failed to read from remote conn: rels://[REDACTED].com:443, use of closed network connection
2025-03-03T11:35:07-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:214: open connection to peer: sha-UW1BjKkhMDn3n5jsJSx2wLEF8soSW/Yu7L5c2p5mdWo=
2025-03-03T11:35:07-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:446: created new wgProxy for relay connection: 127.1.88.48:51820
2025-03-03T11:35:07-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started
2025-03-03T11:35:07-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:475: start to communicate with peer via relay
2025-03-03T11:35:08-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection
2025-03-03T11:35:08-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped
2025-03-03T11:35:14-06:00 INFO [peer: 1HoPtRJ9mJ9v9Ut/m8pTBfmF4HJcRq/aweObXh1l6AQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection
2025-03-03T11:35:14-06:00 INFO [peer: 1HoPtRJ9mJ9v9Ut/m8pTBfmF4HJcRq/aweObXh1l6AQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started
2025-03-03T11:35:30-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection
2025-03-03T11:35:30-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started

This all fixes itself if I restart Netbird, until (usually) the next time the system sleeps. Sometimes it'll malfunction again before going to sleep, but sleep is the most predictable cause of the problem.

@trbutler commented on GitHub (Mar 3, 2025): I'm trying to figure out if one client struggling has a profound impact on the other clients. What I have noticed if running 0.37.1 is that the problem seems to have gotten more extreme. On an M3 MacBook Air, on-wake Netbird's resource usage skyrockets as it gets stuck. This morning, it was taking 150% CPU on Activity Monitor and, as per usual with this situation, unable to communicate with other peers. Here's some output from /var/log/netbird.out.log which seems to balloon to many gigabytes in size on my Mac peers, requiring me to remember to delete it or it will literally fill up the disk. The contents just repeat over and over: > time=2025-03-03T11:31:59.591-06:00 level=DEBUG msg="Handling message" type=InitHello > time=2025-03-03T11:31:59.672-06:00 level=DEBUG msg="Sending message" pid="DZfmMxU7KEkG2wA4Wm5btUTxoCaee1twjHGY/veuH5o=" type=RespHello > time=2025-03-03T11:31:59.674-06:00 level=DEBUG msg="Handling message" type=RespHello > time=2025-03-03T11:31:59.674-06:00 level=ERROR msg="Failed to handle message" error="session not found: f271a4d1" > time=2025-03-03T11:31:59.675-06:00 level=DEBUG msg="Handling message" type=InitHello > time=2025-03-03T11:31:59.755-06:00 level=DEBUG msg="Sending message" pid="DZfmMxU7KEkG2wA4Wm5btUTxoCaee1twjHGY/veuH5o=" type=RespHello > time=2025-03-03T11:31:59.757-06:00 level=DEBUG msg="Handling message" type=RespHello > time=2025-03-03T11:31:59.757-06:00 level=ERROR msg="Failed to handle message" error="session not found: 7d232fdd" > time=2025-03-03T11:31:59.758-06:00 level=DEBUG msg="Handling message" type=InitHello Here's what the /var/netbird/client.log shows on the same system running the latest Netbird: > 2025-03-03T11:30:36-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:510: free up connection to peer: sha-UW1BjKkhMDn3n5jsJSx2wLEF8soSW/Yu7L5c2p5mdWo= > 2025-03-03T11:30:36-06:00 ERRO client/iface/wgproxy/bind/proxy.go:118: failed to read from remote conn: rels://[REDACTED].com:443, use of closed network connection > 2025-03-03T11:30:45-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:214: open connection to peer: sha-UW1BjKkhMDn3n5jsJSx2wLEF8soSW/Yu7L5c2p5mdWo= > 2025-03-03T11:30:45-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:446: created new wgProxy for relay connection: 127.1.88.48:51820 > 2025-03-03T11:30:45-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started > 2025-03-03T11:30:45-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:475: start to communicate with peer via relay > 2025-03-03T11:31:03-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection > 2025-03-03T11:31:03-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped > 2025-03-03T11:31:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection > 2025-03-03T11:31:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started > 2025-03-03T11:31:59-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection > 2025-03-03T11:31:59-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped > 2025-03-03T11:32:00-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/worker_relay.go:50: Relay is not supported by remote peer > 2025-03-03T11:32:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection > 2025-03-03T11:32:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started > 2025-03-03T11:32:42-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection > 2025-03-03T11:32:42-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped > 2025-03-03T11:33:24-06:00 INFO client/internal/peer/guard/sr_watcher.go:94: network changes detected by ICE agent > 2025-03-03T11:33:32-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection > 2025-03-03T11:33:32-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started > 2025-03-03T11:33:50-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection > 2025-03-03T11:33:50-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped > 2025-03-03T11:34:05-06:00 WARN [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/wg_watcher.go:135: WireGuard handshake timed out, closing relay connection: 1969-12-31 18:00:00 -0600 CST > 2025-03-03T11:34:05-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:510: free up connection to peer: sha-NaHy5H143GjYGlDXvsAI7fj0oJLtlA9nBCJoD5KBV6c= > 2025-03-03T11:34:05-06:00 ERRO client/iface/wgproxy/bind/proxy.go:118: failed to read from remote conn: rels://[REDACTED].com:443, use of closed network connection > 2025-03-03T11:34:07-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:214: open connection to peer: sha-NaHy5H143GjYGlDXvsAI7fj0oJLtlA9nBCJoD5KBV6c= > 2025-03-03T11:34:07-06:00 INFO [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/conn.go:446: created new wgProxy for relay connection: 127.1.126.239:51820 > 2025-03-03T11:34:07-06:00 INFO [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started > 2025-03-03T11:34:07-06:00 INFO [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/conn.go:475: start to communicate with peer via relay > 2025-03-03T11:34:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection > 2025-03-03T11:34:33-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started > 2025-03-03T11:35:03-06:00 WARN [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:129: WireGuard handshake timed out, closing relay connection: 1969-12-31 18:00:00 -0600 CST > 2025-03-03T11:35:03-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:510: free up connection to peer: sha-UW1BjKkhMDn3n5jsJSx2wLEF8soSW/Yu7L5c2p5mdWo= > 2025-03-03T11:35:03-06:00 ERRO client/iface/wgproxy/bind/proxy.go:118: failed to read from remote conn: rels://[REDACTED].com:443, use of closed network connection > 2025-03-03T11:35:07-06:00 INFO [relay: rels://[REDACTED].com:443] relay/client/client.go:214: open connection to peer: sha-UW1BjKkhMDn3n5jsJSx2wLEF8soSW/Yu7L5c2p5mdWo= > 2025-03-03T11:35:07-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:446: created new wgProxy for relay connection: 127.1.88.48:51820 > 2025-03-03T11:35:07-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started > 2025-03-03T11:35:07-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:475: start to communicate with peer via relay > 2025-03-03T11:35:08-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:327: set ICE to active connection > 2025-03-03T11:35:08-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:111: WireGuard watcher stopped > 2025-03-03T11:35:14-06:00 INFO [peer: 1HoPtRJ9mJ9v9Ut/m8pTBfmF4HJcRq/aweObXh1l6AQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection > 2025-03-03T11:35:14-06:00 INFO [peer: 1HoPtRJ9mJ9v9Ut/m8pTBfmF4HJcRq/aweObXh1l6AQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started > 2025-03-03T11:35:30-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/conn.go:395: ICE disconnected, set Relay to active connection > 2025-03-03T11:35:30-06:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/wg_watcher.go:87: WireGuard watcher started This all fixes itself if I restart Netbird, until (usually) the next time the system sleeps. Sometimes it'll malfunction again before going to sleep, but sleep is the most predictable cause of the problem.

saavagebueno commented

@mlsmaycon commented on GitHub (Mar 4, 2025):

Hello @trbutler, we are working on fixing a race condition that happens after sleeping at 0.37+. In the meantime, a workaround is to either downgrade to 0.36.7 or disable the network monitor.

@mlsmaycon commented on GitHub (Mar 4, 2025): Hello @trbutler, we are working on fixing a race condition that happens after sleeping at 0.37+. In the meantime, a workaround is to either downgrade to 0.36.7 or disable the network monitor.

saavagebueno commented

@trbutler commented on GitHub (Mar 6, 2025):

Thanks @mlsmaycon. I reverted back to 0.36.7 per your advice. Interestingly so far 0.36.7 is proving the most stable of any release I've used since 0.32.0 concerning the broader problem of recovering after sleep.

@trbutler commented on GitHub (Mar 6, 2025): Thanks @mlsmaycon. I reverted back to 0.36.7 per your advice. Interestingly so far 0.36.7 is proving the most stable of any release I've used since 0.32.0 concerning the broader problem of recovering after sleep.

saavagebueno commented

@trbutler commented on GitHub (Mar 7, 2025):

I don't know if this is relevant to the overall problem of restoring connections after sleep, but I've noticed even on 0.36.7, after sleep Netbird uses 2-4x more resources than before sleep. It seems to idle at about 5% CPU on the M3, but after sleep it easily tops 20% until I restart Netbird.

@trbutler commented on GitHub (Mar 7, 2025): I don't know if this is relevant to the overall problem of restoring connections after sleep, but I've noticed even on 0.36.7, after sleep Netbird uses 2-4x more resources than before sleep. It seems to idle at about 5% CPU on the M3, but after sleep it easily tops 20% until I restart Netbird.

saavagebueno commented

@trbutler commented on GitHub (Mar 12, 2025):

So, 0.38.0 still does not resolve the sleep issue and also seem to continue to show signs of the bug present in 0.37.x. Upon wake from sleep, Netbird is using 60%+ of the M3, spitting out this error into the log at high speed:

time=2025-03-12T17:44:45.996-05:00 level=ERROR msg="Failed to send" pid="On27+F7XY7YokUTfcmoC6f+afXAjUmnAyJciRnTOzjM=" error="write udp [::]:53939->100.68.237.111:56892: use of closed network connection"
time=2025-03-12T17:44:46.004-05:00 level=ERROR msg="Failed to send" pid="wwRjAqzcl2/eXICHMZYp+YajCu3abQVEmiEHD0f7f0E=" error="write udp [::]:53939->100.68.253.229:41753: use of closed network connection"
time=2025-03-12T17:44:46.007-05:00 level=ERROR msg="Failed to send" pid="On27+F7XY7YokUTfcmoC6f+afXAjUmnAyJciRnTOzjM=" error="write udp [::]:53939->100.68.237.111:56892: use of closed network connection"
time=2025-03-12T17:44:46.012-05:00 level=ERROR msg="Failed to send" pid="0S3E31wMigMeqUajW0RWWexGNnO9w8PcHe3Tcx6YL/E=" error="write udp [::]:53939->100.68.163.110:52546: use of closed network connection"
time=2025-03-12T17:44:46.013-05:00 level=ERROR msg="Failed to send" pid="rqXwsgRW0f21aToatSjXc+pGd/dIYJRnTgdtPjKAMAI=" error="write udp [::]:57498->100.68.88.48:60231: use of closed network connection"

After restarting Netbird, usage goes down to about 6%.

@trbutler commented on GitHub (Mar 12, 2025): So, 0.38.0 still does not resolve the sleep issue and also seem to continue to show signs of the bug present in 0.37.x. Upon wake from sleep, Netbird is using 60%+ of the M3, spitting out this error into the log at high speed: ``` time=2025-03-12T17:44:45.996-05:00 level=ERROR msg="Failed to send" pid="On27+F7XY7YokUTfcmoC6f+afXAjUmnAyJciRnTOzjM=" error="write udp [::]:53939->100.68.237.111:56892: use of closed network connection" time=2025-03-12T17:44:46.004-05:00 level=ERROR msg="Failed to send" pid="wwRjAqzcl2/eXICHMZYp+YajCu3abQVEmiEHD0f7f0E=" error="write udp [::]:53939->100.68.253.229:41753: use of closed network connection" time=2025-03-12T17:44:46.007-05:00 level=ERROR msg="Failed to send" pid="On27+F7XY7YokUTfcmoC6f+afXAjUmnAyJciRnTOzjM=" error="write udp [::]:53939->100.68.237.111:56892: use of closed network connection" time=2025-03-12T17:44:46.012-05:00 level=ERROR msg="Failed to send" pid="0S3E31wMigMeqUajW0RWWexGNnO9w8PcHe3Tcx6YL/E=" error="write udp [::]:53939->100.68.163.110:52546: use of closed network connection" time=2025-03-12T17:44:46.013-05:00 level=ERROR msg="Failed to send" pid="rqXwsgRW0f21aToatSjXc+pGd/dIYJRnTgdtPjKAMAI=" error="write udp [::]:57498->100.68.88.48:60231: use of closed network connection" ``` After restarting Netbird, usage goes down to about 6%.

saavagebueno commented

@trbutler commented on GitHub (Mar 25, 2025):

So the issues remain in 0.39.1 and may have actually worsened. After waking from sleep Netbird was using 65% of a processor core. /var/log/netbird/client.log hadn't shown any updates at all since 5 a.m. (several hours after I put the computer to sleep for the night), but /var/log/netbird.out.log was going wild again:

time=2025-03-25T14:08:22.278-05:00 level=ERROR msg="Failed to send" pid="zE7RGWFXUgp+VY6vq1okJNzVZC7FiQQJFDU/fquZAFc=" error="write udp [::]:59445->100.68.36.98:51500: use of closed network connection"
time=2025-03-25T14:08:22.302-05:00 level=DEBUG msg="Handling message" type=InitHello
time=2025-03-25T14:08:22.341-05:00 level=DEBUG msg="Sending message" pid="oRSVlj1QGB2+o2Lb+NpXpsQ3iTuLFMoaznC7WqLwPB4=" type=RespHello
time=2025-03-25T14:08:22.342-05:00 level=ERROR msg="Failed to send" pid="W2zmqmuTObiXHXXcuiGKAv5SXsV1EJlprAEL0CC+1T8=" error="write udp [::]:59445->100.68.126.239:37142: use of closed network connection"
time=2025-03-25T14:08:22.342-05:00 level=ERROR msg="Failed to send" pid="zE7RGWFXUgp+VY6vq1okJNzVZC7FiQQJFDU/fquZAFc=" error="write udp [::]:59445->100.68.36.98:51500: use of closed network connection"
time=2025-03-25T14:08:22.342-05:00 level=DEBUG msg="Handling message" type=RespHello
time=2025-03-25T14:08:22.342-05:00 level=ERROR msg="Failed to handle message" error="session not found: e906089c"
time=2025-03-25T14:08:22.344-05:00 level=ERROR msg="Failed to send" pid="W2zmqmuTObiXHXXcuiGKAv5SXsV1EJlprAEL0CC+1T8=" error="write udp [::]:59445->100.68.126.239:37142: use of closed network connection"
time=2025-03-25T14:08:22.348-05:00 level=DEBUG msg="Handling message" type=RespHello
time=2025-03-25T14:08:22.348-05:00 level=ERROR msg="Failed to handle message" error="session not found: f2d216cf"
time=2025-03-25T14:08:22.350-05:00 level=ERROR msg="Failed to send" pid="wwRjAqzcl2/eXICHMZYp+YajCu3abQVEmiEHD0f7f0E=" error="write udp [::]:59445->100.68.253.229:41753: use of closed network connection"
time=2025-03-25T14:08:22.372-05:00 level=DEBUG msg="Handling message" type=InitHello
time=2025-03-25T14:08:22.376-05:00 level=ERROR msg="Failed to send" pid="wwRjAqzcl2/eXICHMZYp+YajCu3abQVEmiEHD0f7f0E=" error="write udp [::]:59445->100.68.253.229:41753: use of closed network connection"
time=2025-03-25T14:08:22.408-05:00 level=DEBUG msg="Sending message" pid="Pe+rqDOp6tx5iXr2C4sZ9SnF4+ryqXlM5e3v1kWz9G8=" type=RespHello
time=2025-03-25T14:08:22.409-05:00 level=DEBUG msg="Handling message" type=InitHello

@trbutler commented on GitHub (Mar 25, 2025): So the issues remain in 0.39.1 and may have actually worsened. After waking from sleep Netbird was using 65% of a processor core. /var/log/netbird/client.log hadn't shown any updates at all since 5 a.m. (several hours after I put the computer to sleep for the night), but /var/log/netbird.out.log was going wild again: ```` time=2025-03-25T14:08:22.278-05:00 level=ERROR msg="Failed to send" pid="zE7RGWFXUgp+VY6vq1okJNzVZC7FiQQJFDU/fquZAFc=" error="write udp [::]:59445->100.68.36.98:51500: use of closed network connection" time=2025-03-25T14:08:22.302-05:00 level=DEBUG msg="Handling message" type=InitHello time=2025-03-25T14:08:22.341-05:00 level=DEBUG msg="Sending message" pid="oRSVlj1QGB2+o2Lb+NpXpsQ3iTuLFMoaznC7WqLwPB4=" type=RespHello time=2025-03-25T14:08:22.342-05:00 level=ERROR msg="Failed to send" pid="W2zmqmuTObiXHXXcuiGKAv5SXsV1EJlprAEL0CC+1T8=" error="write udp [::]:59445->100.68.126.239:37142: use of closed network connection" time=2025-03-25T14:08:22.342-05:00 level=ERROR msg="Failed to send" pid="zE7RGWFXUgp+VY6vq1okJNzVZC7FiQQJFDU/fquZAFc=" error="write udp [::]:59445->100.68.36.98:51500: use of closed network connection" time=2025-03-25T14:08:22.342-05:00 level=DEBUG msg="Handling message" type=RespHello time=2025-03-25T14:08:22.342-05:00 level=ERROR msg="Failed to handle message" error="session not found: e906089c" time=2025-03-25T14:08:22.344-05:00 level=ERROR msg="Failed to send" pid="W2zmqmuTObiXHXXcuiGKAv5SXsV1EJlprAEL0CC+1T8=" error="write udp [::]:59445->100.68.126.239:37142: use of closed network connection" time=2025-03-25T14:08:22.348-05:00 level=DEBUG msg="Handling message" type=RespHello time=2025-03-25T14:08:22.348-05:00 level=ERROR msg="Failed to handle message" error="session not found: f2d216cf" time=2025-03-25T14:08:22.350-05:00 level=ERROR msg="Failed to send" pid="wwRjAqzcl2/eXICHMZYp+YajCu3abQVEmiEHD0f7f0E=" error="write udp [::]:59445->100.68.253.229:41753: use of closed network connection" time=2025-03-25T14:08:22.372-05:00 level=DEBUG msg="Handling message" type=InitHello time=2025-03-25T14:08:22.376-05:00 level=ERROR msg="Failed to send" pid="wwRjAqzcl2/eXICHMZYp+YajCu3abQVEmiEHD0f7f0E=" error="write udp [::]:59445->100.68.253.229:41753: use of closed network connection" time=2025-03-25T14:08:22.408-05:00 level=DEBUG msg="Sending message" pid="Pe+rqDOp6tx5iXr2C4sZ9SnF4+ryqXlM5e3v1kWz9G8=" type=RespHello time=2025-03-25T14:08:22.409-05:00 level=DEBUG msg="Handling message" type=InitHello ````

saavagebueno commented

@trbutler commented on GitHub (Mar 27, 2025):

A few details I've noticed that may be relevant beyond the resource usage upon wake from sleep. The Netbird clients I have that are running in a datacenter remain available from my Mac laptop. But the clients I have connected to SOHO-grade Internet are not (unfortunately this didn't dawn on me before I restarted Netbird, so I can't say if clients on my local network remain available or not). Restarting netbird on my reawakened Mac restored access to the clients at two different locations outside of a datacenter.

This came up while providing more information for #3576, in which I noticed that the file /var/logs/netbird.out on the reawakened MacOS system had no entries in it for today, but /var/logs/netbird/netbird.out.log on one of the remote systems did have a bunch of entries in it for today.

@trbutler commented on GitHub (Mar 27, 2025): A few details I've noticed that may be relevant beyond the resource usage upon wake from sleep. The Netbird clients I have that are running in a datacenter remain available from my Mac laptop. But the clients I have connected to SOHO-grade Internet are not (unfortunately this didn't dawn on me before I restarted Netbird, so I can't say if clients on my local network remain available or not). Restarting netbird on my reawakened Mac restored access to the clients at two different locations outside of a datacenter. This came up while providing more information for #3576, in which I noticed that the file /var/logs/netbird.out on the reawakened MacOS system had no entries in it for today, but /var/logs/netbird/netbird.out.log on one of the remote systems *did* have a bunch of entries in it for today.

saavagebueno commented

@trbutler commented on GitHub (Mar 31, 2025):

On previous releases, I found that frequently using netbird down && netbird up would both restore access to the peers and reduce the load Netbird put on the system after sleep. However, on 0.39.2, something has degraded further and I have to run sudo netbird service restart or I'm seeing Netbird take as much as 140% CPU on an M1 Max after waking from sleep.

@trbutler commented on GitHub (Mar 31, 2025): On previous releases, I found that frequently using `netbird down && netbird up` would both restore access to the peers and reduce the load Netbird put on the system after sleep. However, on 0.39.2, something has degraded further and I have to run `sudo netbird service restart` or I'm seeing Netbird take as much as 140% CPU on an M1 Max after waking from sleep.

saavagebueno commented

@nazarewk commented on GitHub (Apr 3, 2025):

@trbutler could you enable the trace logs and send us (support at netbird.io or kdn @ public Slack) the debug bundle gathered while the issue is happening?

@nazarewk commented on GitHub (Apr 3, 2025): @trbutler could you [enable the trace logs](https://docs.netbird.io/how-to/troubleshooting-client#on-other-linux-and-mac-os) and send us (support at netbird.io or kdn @ public Slack) the debug bundle gathered while the issue is happening?

saavagebueno commented

@trbutler commented on GitHub (Apr 4, 2025):

@nazarewk I'll update with more contents because today it is technically functioning, but I have been observing the high resource usage that happens when Netbird hasn't been restarted after sleep. I'll send the requested logs to that e-mail address. Thank you! Here's what client.log shows at the moment:

2025-04-04T13:31:05-05:00 INFO [peer: 8m/wrrUqMHd86sjWIIxQVpL9LH+hh7qHpRDZb1rx2TE=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 269, RemoteOffer: 13, RemoteAnswer: 227, RemoteCandidate: 391, P2PConnected: 73, SwitchToRelay: 71, WGCheckSuccess: 5, RelayConnected: 67, LocalProxies: 67
2025-04-04T13:31:05-05:00 INFO [peer: J9NfNe3S6ZsH8UtCNf/Cbpe+dVPvU8bq4CgySpQVLhk=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 189, RemoteOffer: 64, RemoteAnswer: 156, RemoteCandidate: 383, P2PConnected: 74, SwitchToRelay: 72, WGCheckSuccess: 0, RelayConnected: 67, LocalProxies: 67
2025-04-04T13:31:05-05:00 INFO [peer: cnc4Yydo4GjkMivfNmgdi0Z2Sa90wbtINeY/+4x6lio=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 248, RemoteOffer: 25, RemoteAnswer: 210, RemoteCandidate: 273, P2PConnected: 74, SwitchToRelay: 72, WGCheckSuccess: 3, RelayConnected: 69, LocalProxies: 69
2025-04-04T13:31:05-05:00 INFO [peer: hg3w5qD2xhVXQUhh254eNXwB4GUvyHwO64Qe57BhDU4=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 259, RemoteOffer: 21, RemoteAnswer: 206, RemoteCandidate: 189, P2PConnected: 71, SwitchToRelay: 70, WGCheckSuccess: 7, RelayConnected: 66, LocalProxies: 66
2025-04-04T13:31:05-05:00 INFO [peer: lTARJvsn7HviMdIIyAbeVToO7w6x3qlykcex1A1fL2M=] client/internal/peer/state_dump.go:79: Dump stat: Status: Disconnected, SentOffer: 1286, RemoteOffer: 0, RemoteAnswer: 0, RemoteCandidate: 0, P2PConnected: 0, SwitchToRelay: 0, WGCheckSuccess: 0, RelayConnected: 0, LocalProxies: 0
2025-04-04T13:31:05-05:00 INFO [peer: 3d1HqfQI/nnpRVIZ9v+PP5GZif11hyS+oT7XGxaW2C0=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 263, RemoteOffer: 23, RemoteAnswer: 205, RemoteCandidate: 188, P2PConnected: 73, SwitchToRelay: 72, WGCheckSuccess: 0, RelayConnected: 69, LocalProxies: 69
2025-04-04T13:31:05-05:00 INFO [peer: UiC64Vk4V0ODz2MBxmC1z6lmSqSt1eUKYZS2WmTjFSw=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 436, RemoteOffer: 13, RemoteAnswer: 390, RemoteCandidate: 324, P2PConnected: 37, SwitchToRelay: 31, WGCheckSuccess: 23, RelayConnected: 72, LocalProxies: 72
2025-04-04T13:31:05-05:00 INFO [peer: 2DoM1p3SXCX+MrNKEVfeR+T+PHHIzJ6sZQ78SvPLE1Q=] client/internal/peer/state_dump.go:79: Dump stat: Status: Disconnected, SentOffer: 617, RemoteOffer: 42, RemoteAnswer: 81, RemoteCandidate: 234, P2PConnected: 44, SwitchToRelay: 43, WGCheckSuccess: 0, RelayConnected: 32, LocalProxies: 32
2025-04-04T13:31:05-05:00 INFO [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 260, RemoteOffer: 17, RemoteAnswer: 222, RemoteCandidate: 175, P2PConnected: 69, SwitchToRelay: 67, WGCheckSuccess: 1, RelayConnected: 67, LocalProxies: 67
2025-04-04T13:31:05-05:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/state_dump.go:79: Dump stat: Status: Disconnected, SentOffer: 1508, RemoteOffer: 12, RemoteAnswer: 10, RemoteCandidate: 32, P2PConnected: 4, SwitchToRelay: 4, WGCheckSuccess: 111, RelayConnected: 10, LocalProxies: 10
2025-04-04T13:31:05-05:00 INFO [peer: GzKcfOOsMz5OHmPXWaVUlNxemtHLUvXBqFbUKRI7aFU=] client/internal/peer/state_dump.go:79: Dump stat: Status: Disconnected, SentOffer: 1292, RemoteOffer: 0, RemoteAnswer: 0, RemoteCandidate: 0, P2PConnected: 0, SwitchToRelay: 0, WGCheckSuccess: 0, RelayConnected: 0, LocalProxies: 0
2025-04-04T13:31:16-05:00 INFO [peer: KQIjQLtUaZM9J30rBp2AxHC4nrvn8neHA7Vg1DURkFg=] client/internal/peer/conn.go:548: send offer to peer
2025-04-04T13:31:18-05:00 INFO [peer: oBnIFFtgnPA6SDOJcSHlR1BbIM5h7WudevvgmkDRLzQ=] client/internal/peer/conn.go:548: send offer to peer

@trbutler commented on GitHub (Apr 4, 2025): @nazarewk I'll update with more contents because today it is technically functioning, but I have been observing the high resource usage that happens when Netbird hasn't been restarted after sleep. I'll send the requested logs to that e-mail address. Thank you! Here's what client.log shows at the moment: ```` 2025-04-04T13:31:05-05:00 INFO [peer: 8m/wrrUqMHd86sjWIIxQVpL9LH+hh7qHpRDZb1rx2TE=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 269, RemoteOffer: 13, RemoteAnswer: 227, RemoteCandidate: 391, P2PConnected: 73, SwitchToRelay: 71, WGCheckSuccess: 5, RelayConnected: 67, LocalProxies: 67 2025-04-04T13:31:05-05:00 INFO [peer: J9NfNe3S6ZsH8UtCNf/Cbpe+dVPvU8bq4CgySpQVLhk=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 189, RemoteOffer: 64, RemoteAnswer: 156, RemoteCandidate: 383, P2PConnected: 74, SwitchToRelay: 72, WGCheckSuccess: 0, RelayConnected: 67, LocalProxies: 67 2025-04-04T13:31:05-05:00 INFO [peer: cnc4Yydo4GjkMivfNmgdi0Z2Sa90wbtINeY/+4x6lio=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 248, RemoteOffer: 25, RemoteAnswer: 210, RemoteCandidate: 273, P2PConnected: 74, SwitchToRelay: 72, WGCheckSuccess: 3, RelayConnected: 69, LocalProxies: 69 2025-04-04T13:31:05-05:00 INFO [peer: hg3w5qD2xhVXQUhh254eNXwB4GUvyHwO64Qe57BhDU4=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 259, RemoteOffer: 21, RemoteAnswer: 206, RemoteCandidate: 189, P2PConnected: 71, SwitchToRelay: 70, WGCheckSuccess: 7, RelayConnected: 66, LocalProxies: 66 2025-04-04T13:31:05-05:00 INFO [peer: lTARJvsn7HviMdIIyAbeVToO7w6x3qlykcex1A1fL2M=] client/internal/peer/state_dump.go:79: Dump stat: Status: Disconnected, SentOffer: 1286, RemoteOffer: 0, RemoteAnswer: 0, RemoteCandidate: 0, P2PConnected: 0, SwitchToRelay: 0, WGCheckSuccess: 0, RelayConnected: 0, LocalProxies: 0 2025-04-04T13:31:05-05:00 INFO [peer: 3d1HqfQI/nnpRVIZ9v+PP5GZif11hyS+oT7XGxaW2C0=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 263, RemoteOffer: 23, RemoteAnswer: 205, RemoteCandidate: 188, P2PConnected: 73, SwitchToRelay: 72, WGCheckSuccess: 0, RelayConnected: 69, LocalProxies: 69 2025-04-04T13:31:05-05:00 INFO [peer: UiC64Vk4V0ODz2MBxmC1z6lmSqSt1eUKYZS2WmTjFSw=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 436, RemoteOffer: 13, RemoteAnswer: 390, RemoteCandidate: 324, P2PConnected: 37, SwitchToRelay: 31, WGCheckSuccess: 23, RelayConnected: 72, LocalProxies: 72 2025-04-04T13:31:05-05:00 INFO [peer: 2DoM1p3SXCX+MrNKEVfeR+T+PHHIzJ6sZQ78SvPLE1Q=] client/internal/peer/state_dump.go:79: Dump stat: Status: Disconnected, SentOffer: 617, RemoteOffer: 42, RemoteAnswer: 81, RemoteCandidate: 234, P2PConnected: 44, SwitchToRelay: 43, WGCheckSuccess: 0, RelayConnected: 32, LocalProxies: 32 2025-04-04T13:31:05-05:00 INFO [peer: oO58e18RBlgzZtE0R3QqcXz7z4EAka4Gfg/YPK+fwQ4=] client/internal/peer/state_dump.go:79: Dump stat: Status: Connected, SentOffer: 260, RemoteOffer: 17, RemoteAnswer: 222, RemoteCandidate: 175, P2PConnected: 69, SwitchToRelay: 67, WGCheckSuccess: 1, RelayConnected: 67, LocalProxies: 67 2025-04-04T13:31:05-05:00 INFO [peer: uxBPJx0lXlDYLmtLnXS09qQLkvnGtPL4jFuKlfQWvRQ=] client/internal/peer/state_dump.go:79: Dump stat: Status: Disconnected, SentOffer: 1508, RemoteOffer: 12, RemoteAnswer: 10, RemoteCandidate: 32, P2PConnected: 4, SwitchToRelay: 4, WGCheckSuccess: 111, RelayConnected: 10, LocalProxies: 10 2025-04-04T13:31:05-05:00 INFO [peer: GzKcfOOsMz5OHmPXWaVUlNxemtHLUvXBqFbUKRI7aFU=] client/internal/peer/state_dump.go:79: Dump stat: Status: Disconnected, SentOffer: 1292, RemoteOffer: 0, RemoteAnswer: 0, RemoteCandidate: 0, P2PConnected: 0, SwitchToRelay: 0, WGCheckSuccess: 0, RelayConnected: 0, LocalProxies: 0 2025-04-04T13:31:16-05:00 INFO [peer: KQIjQLtUaZM9J30rBp2AxHC4nrvn8neHA7Vg1DURkFg=] client/internal/peer/conn.go:548: send offer to peer 2025-04-04T13:31:18-05:00 INFO [peer: oBnIFFtgnPA6SDOJcSHlR1BbIM5h7WudevvgmkDRLzQ=] client/internal/peer/conn.go:548: send offer to peer ````

saavagebueno commented

@trbutler commented on GitHub (Apr 18, 2025):

The issue continues to exist in 0.41.3. In fact, on the M3, I saw the worst post-sleep performance yet with Netbird claiming 120% CPU in Activity Monitor.

@trbutler commented on GitHub (Apr 18, 2025): The issue continues to exist in 0.41.3. In fact, on the M3, I saw the worst post-sleep performance yet with Netbird claiming 120% CPU in Activity Monitor.

saavagebueno commented

@trbutler commented on GitHub (Apr 24, 2025):

The issue persists in 0.42.0 with the same ~ 110-120% CPU as 0.41.x. This is a significant continued regression from even the earlier problem where it was only 20-30% CPU upon wake.

@trbutler commented on GitHub (Apr 24, 2025): The issue persists in 0.42.0 with the same ~ 110-120% CPU as 0.41.x. This is a significant continued regression from even the earlier problem where it was only 20-30% CPU upon wake. <img width="1072" alt="Image" src="https://github.com/user-attachments/assets/38876b64-519f-43cd-bc8b-28d88748911a" />

saavagebueno commented

@lixmal commented on GitHub (Apr 24, 2025):

@trbutler Next time this happens, can you run

pkill -SIGUSR1 netbird

This might be able to generate a debug bundle in /tmp (while netbird status is stuck)

@lixmal commented on GitHub (Apr 24, 2025): @trbutler Next time this happens, can you run ``` pkill -SIGUSR1 netbird ``` This might be able to generate a debug bundle in `/tmp` (while `netbird status` is stuck)

saavagebueno commented

@trbutler commented on GitHub (Apr 25, 2025):

Got it. Where should I send it? Thanks!

@trbutler commented on GitHub (Apr 25, 2025): Got it. Where should I send it? Thanks!

saavagebueno commented

@lixmal commented on GitHub (Apr 25, 2025):

support@netbird.io or on slack

@lixmal commented on GitHub (Apr 25, 2025): support@netbird.io or on slack

saavagebueno commented

@trbutler commented on GitHub (Apr 25, 2025):

Thanks, @lixmal . I sent that debug bundle to the address.

@trbutler commented on GitHub (Apr 25, 2025): Thanks, @lixmal . I sent that debug bundle to the address.

saavagebueno commented

@lixmal commented on GitHub (Apr 26, 2025):

@trbutler We haven't received anything yet

@lixmal commented on GitHub (Apr 26, 2025): @trbutler We haven't received anything yet

saavagebueno commented

@Gauss23 commented on GitHub (May 15, 2025):

Any progress here? The Netbird client on MacOS should cope with sleep/hibernation of the machine. That's a common usecase.

@Gauss23 commented on GitHub (May 15, 2025): Any progress here? The Netbird client on MacOS should cope with sleep/hibernation of the machine. That's a common usecase.

saavagebueno commented

@trbutler commented on GitHub (May 15, 2025):

@trbutler We haven't received anything yet

That's weird -- I'll try to send it again. Please let me know if you receive it. Thanks!

I can confirm this is continuing in 0.43.3, incidentally.

@trbutler commented on GitHub (May 15, 2025): > [@trbutler](https://github.com/trbutler) We haven't received anything yet That's weird -- I'll try to send it again. Please let me know if you receive it. Thanks! I can confirm this is continuing in 0.43.3, incidentally.

saavagebueno commented

@pappz commented on GitHub (May 19, 2025):

@trbutler
Could you run the netbird debug bundle -S when the connection and CPU usage are normal?

@pappz commented on GitHub (May 19, 2025): @trbutler Could you run the **netbird debug bundle -S** when the connection and CPU usage are normal?

saavagebueno commented

@trbutler commented on GitHub (May 25, 2025):

@pappz I sent the requested bundle on Slack. This is a particularly interesting moment of it working, too: it was a rare occasion the MacBook resumed from sleep without Netbird going crazy on resource usage, so I thought it might offer something helpful. Notably, I've see 0.45.1 using fewer resources when it is working well, sometimes as little as 2-5%. It can still use up to 15% of the M3, which seems a bit high given how well it works onconsiderably less powerful processors such as a Raspberry Pi's or an anemic Atom x5-Z8350 (it often idles at about the same utilization on the Atom as on the M3). But, the previous range was 12-20%, so it is an improvement! It can still get into the mess after waking where it uses 100% until I restart the service.

@trbutler commented on GitHub (May 25, 2025): @pappz I sent the requested bundle on Slack. This is a particularly interesting moment of it working, too: it was a rare occasion the MacBook resumed from sleep without Netbird going crazy on resource usage, so I thought it might offer something helpful. Notably, I've see 0.45.1 using fewer resources when it is working well, sometimes as little as 2-5%. It can still use up to 15% of the M3, which seems a bit high given how well it works onconsiderably less powerful processors such as a Raspberry Pi's or an anemic Atom x5-Z8350 (it often idles at about the same utilization on the Atom as on the M3). But, the previous range was 12-20%, so it is an improvement! It can still get into the mess after waking where it uses 100% until I restart the service.

saavagebueno commented