SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-05-12 11:12:00 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Netbird seems to "die" on my server #1510

New Issue
Open
opened 2025-11-20 05:31:57 -05:00 by saavagebueno · 18 comments
saavagebueno commented 2025-11-20 05:31:57 -05:00
Owner
Copy Link

Originally created by @GronbergInfo on GitHub (Dec 23, 2024).

I run
sudo netbird down && sudo netbird up --allow-server-ssh

On the server, and it starts to work, then after a uncertain time, it just stops communicate, resolve dns and such until i run that command again

client and server has netbird 0.35.0.

If i ssh into the server using lan ip and check netbird status, it looks ok/normal

> netbird status
OS: linux/amd64
Daemon version: 0.35.0
CLI version: 0.35.0
Management: Connected
Signal: Connected
Relays: 3/3 Available
Nameservers: 1/1 Available
FQDN: <servername>.netbird.cloud
NetBird IP: 100.85.x.y/16
Interface type: Kernel
Quantum resistance: true
Routes: -
Networks: -
Peers count: 2/2 Connected

EDIT: just added 2 peers (laptop + android tablet) and it died for the server again.

Originally created by @GronbergInfo on GitHub (Dec 23, 2024). I run `sudo netbird down && sudo netbird up --allow-server-ssh ` On the server, and it starts to work, then after a uncertain time, it just stops communicate, resolve dns and such until i run that command again client and server has netbird 0.35.0. If i ssh into the server using lan ip and check netbird status, it looks ok/normal ``` > netbird status OS: linux/amd64 Daemon version: 0.35.0 CLI version: 0.35.0 Management: Connected Signal: Connected Relays: 3/3 Available Nameservers: 1/1 Available FQDN: <servername>.netbird.cloud NetBird IP: 100.85.x.y/16 Interface type: Kernel Quantum resistance: true Routes: - Networks: - Peers count: 2/2 Connected ``` EDIT: just added 2 peers (laptop + android tablet) and it died for the server again.
saavagebueno added the bugclientconnection labels 2025-11-20 05:31:57 -05:00
saavagebueno commented 2025-11-20 05:31:58 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Dec 24, 2024):

Hello @GronbergInfo thanks for reporting the issue, can you please run the following commands on the server and one of the clients?

Enable debug information

# enable trace logs:
netbird debug log level trace
# enable network map report
netbird debug persistence on

When the issue happens again:

netbird debug bundle -SA

This will generate an anonymized bundle file with log information. Then, please share them with us.

@mlsmaycon commented on GitHub (Dec 24, 2024): Hello @GronbergInfo thanks for reporting the issue, can you please run the following commands on the server and one of the clients? Enable debug information ```shell # enable trace logs: netbird debug log level trace # enable network map report netbird debug persistence on ``` When the issue happens again: ```shell netbird debug bundle -SA ``` This will generate an anonymized bundle file with log information. Then, please share them with us.
saavagebueno commented 2025-11-20 05:31:58 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Dec 24, 2024):

Also, can you confirm that the issue happens on SSH connections with the server or, in general, the peer connection 'dies' and you can't even ping it? and how long does it take to have issues?

@mlsmaycon commented on GitHub (Dec 24, 2024): Also, can you confirm that the issue happens on SSH connections with the server or, in general, the peer connection 'dies' and you can't even ping it? and how long does it take to have issues?
saavagebueno commented 2025-11-20 05:31:58 -05:00
Author
Owner
Copy Link

@GronbergInfo commented on GitHub (Dec 24, 2024):

Hello @GronbergInfo thanks for reporting the issue, can you please run the following commands on the server and one of the clients?

Enable debug information

# enable trace logs:
netbird debug log level trace
# enable network map report

netbird debug persistence on


When the issue happens again:


netbird debug bundle -SA

This will generate an anonymized bundle file with log information. Then, please share them with us.

From client: netbird.debug.3148610165.zip
netbird.debug.3148610165.zip

From Server: netbird.debug.1025569393.zip
netbird.debug.1025569393.zip

@GronbergInfo commented on GitHub (Dec 24, 2024): > Hello @GronbergInfo thanks for reporting the issue, can you please run the following commands on the server and one of the clients? > > Enable debug information > > ```shell > # enable trace logs: > netbird debug log level trace > # enable network map report > netbird debug persistence on > ``` > > When the issue happens again: > > ```shell > netbird debug bundle -SA > ``` > > This will generate an anonymized bundle file with log information. Then, please share them with us. From client: netbird.debug.3148610165.zip [netbird.debug.3148610165.zip](https://github.com/user-attachments/files/18238140/netbird.debug.3148610165.zip) From Server: netbird.debug.1025569393.zip [netbird.debug.1025569393.zip](https://github.com/user-attachments/files/18238173/netbird.debug.1025569393.zip)
saavagebueno commented 2025-11-20 05:31:58 -05:00
Author
Owner
Copy Link

@GronbergInfo commented on GitHub (Dec 24, 2024):

Also, can you confirm that the issue happens on SSH connections with the server or, in general, the peer connection 'dies' and you can't even ping it? and how long does it take to have issues?

It seems everything "dies" but hostname lookup

> nslookup hades10gbe
Server:         100.85.228.40
Address:        100.85.228.40#53

Non-authoritative answer:
Name:   hades10gbe.netbird.cloud
Address: 100.85.119.90
** server can't find hades10gbe.netbird.cloud: NXDOMAIN
@GronbergInfo commented on GitHub (Dec 24, 2024): > Also, can you confirm that the issue happens on SSH connections with the server or, in general, the peer connection 'dies' and you can't even ping it? and how long does it take to have issues? It seems everything "dies" but hostname lookup ``` > nslookup hades10gbe Server: 100.85.228.40 Address: 100.85.228.40#53 Non-authoritative answer: Name: hades10gbe.netbird.cloud Address: 100.85.119.90 ** server can't find hades10gbe.netbird.cloud: NXDOMAIN ```
saavagebueno commented 2025-11-20 05:31:58 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Dec 24, 2024):

@GronbergInfo it doesn't seem like you executed the commands as shared in https://github.com/netbirdio/netbird/issues/3107#issuecomment-2560846327. Can you please do the exact steps?

Also, can you provide an answer to the following questions?

  1. when the connection dies, you can run netbird status -d, and you will see a connection to signal and management, but no connection to the peer?
  2. how long until the issue happens?
  3. Does the issue happen if you have rosenpass integration disabled?
@mlsmaycon commented on GitHub (Dec 24, 2024): @GronbergInfo it doesn't seem like you executed the commands as shared in https://github.com/netbirdio/netbird/issues/3107#issuecomment-2560846327. Can you please do the exact steps? Also, can you provide an answer to the following questions? 1. when the connection dies, you can run netbird status -d, and you will see a connection to signal and management, but no connection to the peer? 2. how long until the issue happens? 3. Does the issue happen if you have rosenpass integration disabled?
saavagebueno commented 2025-11-20 05:31:58 -05:00
Author
Owner
Copy Link

@GronbergInfo commented on GitHub (Dec 24, 2024):

@GronbergInfo it doesn't seem like you executed the commands as shared in #3107 (comment). Can you please do the exact steps?

Also, can you provide an answer to the following questions?

  1. when the connection dies, you can run netbird status -d, and you will see a connection to signal and management, but no connection to the peer?
  2. how long until the issue happens?
  3. Does the issue happen if you have rosenpass integration disabled?

nbd.txt - client
nbd2.txt - server
nbd.txt
nbd2.txt

  1. it feels random, can work for 15-30-60 minutes, uncertain, it can stop working within a heart beat almost.

  2. Disabled rosenpass on client and server, mounted an nfs share, started to copy a 600GB vm vmware folder, changed group for my android tablet and applied ACL for this group. After 10 or so minutes, it is still running ok.

@GronbergInfo commented on GitHub (Dec 24, 2024): > @GronbergInfo it doesn't seem like you executed the commands as shared in [#3107 (comment)](https://github.com/netbirdio/netbird/issues/3107#issuecomment-2560846327). Can you please do the exact steps? > > Also, can you provide an answer to the following questions? > > 1. when the connection dies, you can run netbird status -d, and you will see a connection to signal and management, but no connection to the peer? > 2. how long until the issue happens? > 3. Does the issue happen if you have rosenpass integration disabled? 1) nbd.txt - client nbd2.txt - server [nbd.txt](https://github.com/user-attachments/files/18238617/nbd.txt) [nbd2.txt](https://github.com/user-attachments/files/18238618/nbd2.txt) 2) it feels random, can work for 15-30-60 minutes, uncertain, it can stop working within a heart beat almost. 3) Disabled rosenpass on client and server, mounted an nfs share, started to copy a 600GB vm vmware folder, changed group for my android tablet and applied ACL for this group. After 10 or so minutes, it is still running ok.
saavagebueno commented 2025-11-20 05:31:58 -05:00
Author
Owner
Copy Link

@GronbergInfo commented on GitHub (Dec 24, 2024):

still 15 or so minutes later, it still up and running.
So it seems to be attached to the rosenpass function.

@GronbergInfo commented on GitHub (Dec 24, 2024): still 15 or so minutes later, it still up and running. So it seems to be attached to the rosenpass function.
saavagebueno commented 2025-11-20 05:31:59 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Dec 24, 2024):

It is caused by the key not being rotated due to a firewall issue. We are missing a default allow rule for rosenpass.

If this continues, can you add a UDP rule like the one in the image and try again? We will be working on a permanent fix.

image
@mlsmaycon commented on GitHub (Dec 24, 2024): It is caused by the key not being rotated due to a firewall issue. We are missing a default allow rule for rosenpass. If this continues, can you add a UDP rule like the one in the image and try again? We will be working on a permanent fix. <img width="751" alt="image" src="https://github.com/user-attachments/assets/4516e758-effc-46f8-9427-9b54760e9543" />
saavagebueno commented 2025-11-20 05:31:59 -05:00
Author
Owner
Copy Link

@GronbergInfo commented on GitHub (Dec 24, 2024):

It is caused by the key not being rotated due to a firewall issue. We are missing a default allow rule for rosenpass.

If this continues, can you add a UDP rule like the one in the image and try again? We will be working on a permanent fix.

image

Even with this setting, it seems it dies with quantum enabled.

@GronbergInfo commented on GitHub (Dec 24, 2024): > It is caused by the key not being rotated due to a firewall issue. We are missing a default allow rule for rosenpass. > > If this continues, can you add a UDP rule like the one in the image and try again? We will be working on a permanent fix. > > <img alt="image" width="751" src="https://private-user-images.githubusercontent.com/7747744/398398373-4516e758-effc-46f8-9427-9b54760e9543.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzUwMzk2NjcsIm5iZiI6MTczNTAzOTM2NywicGF0aCI6Ii83NzQ3NzQ0LzM5ODM5ODM3My00NTE2ZTc1OC1lZmZjLTQ2ZjgtOTQyNy05YjU0NzYwZTk1NDMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MTIyNCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDEyMjRUMTEyMjQ3WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YTBlN2RmMmM5YjVjYzk2MDhkN2QwZTQxZmMzZWRmODVkNDU2MmM0ZjRhMTA0MzY5NjA4M2NmNTk5ZDMxNTlhYiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QifQ.sP9UN5r2eF4xtiCPmetfZNsxixWadonms70w75UmZVE"> Even with this setting, it seems it dies with quantum enabled.
saavagebueno commented 2025-11-20 05:31:59 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Dec 24, 2024):

Thanks @GronbergInfo , can you please enable the logs and share them with us once the connection fail? we are running the same test on our side too.

@mlsmaycon commented on GitHub (Dec 24, 2024): Thanks @GronbergInfo , can you please enable the logs and share them with us once the connection fail? we are running the same test on our side too.
saavagebueno commented 2025-11-20 05:31:59 -05:00
Author
Owner
Copy Link

@GronbergInfo commented on GitHub (Dec 24, 2024):

Thanks @GronbergInfo , can you please enable the logs and share them with us once the connection fail? we are running the same test on our side too.

Did the log thing on both ends.

client
netbird.debug.2882577087.zip

server
netbird.debug.2497331844.zip

this time the connection died nearly instantly when switching both sides to --enable-rosenpass=true

@GronbergInfo commented on GitHub (Dec 24, 2024): > Thanks @GronbergInfo , can you please enable the logs and share them with us once the connection fail? we are running the same test on our side too. Did the log thing on both ends. client [netbird.debug.2882577087.zip](https://github.com/user-attachments/files/18239318/netbird.debug.2882577087.zip) server [netbird.debug.2497331844.zip](https://github.com/user-attachments/files/18239320/netbird.debug.2497331844.zip) this time the connection died nearly instantly when switching both sides to --enable-rosenpass=true
saavagebueno commented 2025-11-20 05:31:59 -05:00
Author
Owner
Copy Link

@rihards-simanovics commented on GitHub (Dec 27, 2024):

Update: forgot to mention the downtime detection was at around 2024-12-27 06:30:31

I am not sure if this is related, but each time, one of my servers, which does auto-upgrade at around 6 pm, dies after upgrading to 0.35.1. Right now, it would seem that at least for me, client 0.29.4 is the last stable version I can rely on to work well:

/var/log/apt/history.log

Start-Date: 2024-12-27  06:26:12
Upgrade: netbird:amd64 (0.29.4, 0.35.1)
End-Date: 2024-12-27  06:26:17

Start-Date: 2024-12-27  06:35:49
Commandline: apt install netbird=0.29.4
Downgrade: netbird:amd64 (0.35.1, 0.29.4)
End-Date: 2024-12-27  06:35:54

cat /var/log/netbird/client.log

2024-12-27T06:26:13Z INFO client/internal/engine.go:251: Network monitor: stopped
2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:521: closing all peer connections
2024-12-27T06:26:13Z INFO client/internal/connect.go:300: stopped NetBird client
2024-12-27T06:26:13Z INFO client/cmd/root.go:191: shutdown signal received
2024-12-27T06:26:13Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:330: start to Relay read loop exit
2024-12-27T06:26:13Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:529: waiting for read loop to close
2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:531: relay connection closed
2024-12-27T06:26:13Z WARN [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:516: relay connection was already marked as not running
2024-12-27T06:26:13Z INFO [peer: part-redactedinternal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redactedinternal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [peer: part-redactedinternal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redactedinternal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redactedwTT0=] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redactedc8aELBa96mmk=] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z INFO [peer: part-redactedinternal/peer/conn.go:214: close peer connection
2024-12-27T06:26:13Z WARN [peer: part-redactedinternal/peer/worker_relay.go:156: failed to close relay connection: connection already closed
2024-12-27T06:26:13Z ERRO client/internal/wgproxy/proxy_ebpf.go:176: failed to read UDP pkg from WG: read udp 127.0.0.1:3128: use of closed network connection
2024-12-27T06:26:13Z INFO client/internal/dns/systemd_linux.go:161: reverting link settings and flushing cache
2024-12-27T06:26:13Z INFO client/internal/routemanager/manager.go:177: Routing cleanup complete
2024-12-27T06:26:13Z INFO iface/iface.go:192: interface wt0 has been removed
2024-12-27T06:26:13Z INFO client/internal/engine.go:271: stopped Netbird Engine
2024-12-27T06:26:13Z INFO client/server/server.go:625: service is down
2024-12-27T06:26:15Z INFO client/cmd/service_controller.go:90: stopped Netbird service
2024-12-27T06:26:17Z INFO client/cmd/service_controller.go:24: starting Netbird service
2024-12-27T06:26:17Z INFO client/cmd/service_controller.go:68: started daemon server: /var/run/netbird.sock
2024-12-27T06:26:17Z INFO client/internal/connect.go:115: starting NetBird client version 0.35.1 on linux/amd64
2024-12-27T06:26:18Z INFO client/internal/connect.go:247: connecting to the Relay service(s): rels://vpn-subdomain.domain.com:443
2024-12-27T06:26:18Z INFO relay/client/picker.go:72: try to connecting to relay server: rels://vpn-subdomain.domain.com:443
2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:165: create new relay connection: local peerID: redacted, local peer hashedID: part-redacted
2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:171: connecting to relay server
2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:188: relay connection established
2024-12-27T06:26:18Z INFO relay/client/picker.go:90: connected to Relay server: rels://vpn-subdomain.domain.com:443
2024-12-27T06:26:18Z INFO relay/client/picker.go:64: chosen home Relay server: rels://vpn-subdomain.domain.com:443
2024-12-27T06:26:18Z INFO client/iface/wgproxy/ebpf/proxy.go:91: local wg proxy listening on: 3128
2024-12-27T06:26:18Z INFO client/iface/wgproxy/factory_kernel.go:29: WireGuard Proxy Factory will produce eBPF proxy
2024-12-27T06:26:18Z INFO client/internal/engine.go:352: rosenpass is enabled
2024-12-27T06:26:18Z INFO client/internal/engine.go:354: running rosenpass in permissive mode
2024-12-27T06:26:18Z INFO client/internal/rosenpass/manager.go:158: starting rosenpass server on port 49146
2024-12-27T06:26:18Z INFO client/internal/routemanager/manager.go:163: Routing setup complete
2024-12-27T06:26:18Z INFO client/firewall/create_linux.go:72: creating an nftables firewall manager
2024-12-27T06:26:18Z INFO client/internal/dns/host_unix.go:54: System DNS manager discovered: systemd
2024-12-27T06:26:18Z INFO client/internal/peer/guard/sr_watcher.go:106: reconnected to Signal or Relay server
2024-12-27T06:26:18Z INFO signal/client/grpc.go:149: connected to the Signal Service stream
2024-12-27T06:26:18Z INFO client/internal/engine.go:1472: Network monitor is disabled, not starting
2024-12-27T06:26:18Z INFO client/internal/connect.go:273: Netbird engine started, the IP is: 100.90.79.155/16
2024-12-27T06:26:18Z INFO management/client/grpc.go:155: connected to the Management Service stream
2024-12-27T06:26:18Z INFO relay/client/manager.go:220: update relay server URLs: [rels://vpn-subdomain.domain.com:443]
2024-12-27T06:26:18Z WARN client/internal/engine.go:644: running SSH server is not permitted
2024-12-27T06:26:18Z INFO client/internal/acl/manager.go:61: ACL rules processed in: 48.875679ms, total rules count: 276
2024-12-27T06:26:18Z ERRO client/internal/dns/server.go:374: unable to parse ip address, error: ParseAddr(""): unable to parse IP
2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedbXBhk6Jn4dYzMiCQCqXRZ5WNMFgMemSTPyk=
2024-12-27T06:26:18Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :1
2024-12-27T06:26:18Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:1
2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedqA3YQ=
2024-12-27T06:26:18Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :2
2024-12-27T06:26:18Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:2
2024-12-27T06:26:18Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:138: start listen for reconnect events...
2024-12-27T06:26:18Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:18Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:138: start listen for reconnect events...
2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedCEiFwKvDM+LYhWTbhypMRxrKU=
2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :3
2024-12-27T06:26:19Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:3
2024-12-27T06:26:19Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedtxUqu9wJBfRot5k/SiGBB3eZbmbxrYQ=
2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :4
2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:4
2024-12-27T06:26:19Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:19Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedAUM=
2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :5
2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:5
2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedp50ILYJOXmWcMN6ci43tI=
2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :6
2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:6
2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redacted
2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :7
2024-12-27T06:26:19Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:7
2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:19Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedteYT8KKEKmSGB8HbQMMVnR4c=
2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :8
2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:8
2024-12-27T06:26:19Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedF7j8MAO5OITBusHC4Lh/ww=
2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :9
2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:9
2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:20Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedYCOQWBtzvOM=
2024-12-27T06:26:20Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :10
2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:10
2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:20Z INFO [peer: part-redactedinternal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:26:21Z INFO [peer: part-redactedinternal/peer/guard/guard.go:84: start reconnect loop...
2024-12-27T06:26:21Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:84: start reconnect loop...
2024-12-27T06:26:21Z INFO [peer: part-redactedwTT0=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2024-12-27T06:26:21Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2024-12-27T06:26:21Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:84: start reconnect loop...
2024-12-27T06:26:21Z INFO [peer: part-redactedinternal/peer/guard/guard.go:84: start reconnect loop...
2024-12-27T06:26:22Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:84: start reconnect loop...
2024-12-27T06:26:22Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/guard/guard.go:84: start reconnect loop...
2024-12-27T06:26:22Z INFO [peer: part-redactedinternal/peer/guard/guard.go:84: start reconnect loop...
2024-12-27T06:26:33Z INFO [peer: part-redactedinternal/peer/conn.go:328: set ICE to active connection
2024-12-27T06:31:18Z INFO client/internal/peer/guard/sr_watcher.go:94: network changes detected by ICE agent
2024-12-27T06:35:30Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A=
2024-12-27T06:35:30Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :11
2024-12-27T06:35:30Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:11
2024-12-27T06:35:30Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay
2024-12-27T06:35:50Z INFO client/cmd/root.go:192: shutdown signal received
2024-12-27T06:35:50Z INFO client/internal/engine.go:279: Network monitor: stopped
2024-12-27T06:35:50Z INFO client/internal/dns/systemd_linux.go:166: reverting link settings and flushing cache
2024-12-27T06:35:50Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:536: closing all peer connections
2024-12-27T06:35:50Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:345: start to Relay read loop exit
2024-12-27T06:35:50Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:544: waiting for read loop to close
2024-12-27T06:35:50Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:546: relay connection closed
2024-12-27T06:35:50Z WARN [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:531: relay connection was already marked as not running
2024-12-27T06:35:50Z INFO client/internal/routemanager/manager.go:219: Routing cleanup complete
2024-12-27T06:35:50Z INFO [peer: part-redactedinternal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redactedinternal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redactedwTT0=] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redactedinternal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redactedinternal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redactedc8aELBa96mmk=] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redactedinternal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redactedinternal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection
2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection
2024-12-27T06:35:50Z ERRO client/iface/bind/udp_mux_universal.go:93: error while reading packet: shared socked stopped
2024-12-27T06:35:50Z INFO client/iface/iface.go:220: interface wt0 has been removed
2024-12-27T06:35:51Z INFO client/internal/engine.go:317: stopped Netbird Engine
2024-12-27T06:35:51Z INFO client/server/server.go:648: service is down
2024-12-27T06:35:51Z INFO client/internal/connect.go:296: stopped NetBird client
2024-12-27T06:35:53Z INFO client/cmd/service_controller.go:94: stopped Netbird service
2024-12-27T06:35:54Z INFO client/cmd/service_controller.go:24: starting Netbird service
2024-12-27T06:35:54Z INFO client/cmd/service_controller.go:66: started daemon server: /var/run/netbird.sock
2024-12-27T06:35:54Z INFO client/internal/connect.go:117: starting NetBird client version 0.29.4 on linux/amd64
2024-12-27T06:35:54Z INFO client/internal/connect.go:252: connecting to the Relay service(s): rels://vpn-subdomain.domain.com:443
2024-12-27T06:35:54Z INFO relay/client/picker.go:66: try to connecting to relay server: rels://vpn-subdomain.domain.com:443
2024-12-27T06:35:54Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:165: create new relay connection: local peerID: redacted, local peer hashedID: part-redacted
2024-12-27T06:35:54Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:171: connecting to relay server
2024-12-27T06:35:54Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:192: relay connection established
2024-12-27T06:35:54Z INFO relay/client/picker.go:84: connected to Relay server: rels://vpn-subdomain.domain.com:443
2024-12-27T06:35:54Z INFO relay/client/picker.go:58: chosen home Relay server: rels://vpn-subdomain.domain.com:443
2024-12-27T06:35:54Z INFO client/internal/wgproxy/proxy_ebpf.go:89: local wg proxy listening on: 3128
2024-12-27T06:35:54Z INFO client/internal/engine.go:298: rosenpass is enabled
2024-12-27T06:35:54Z INFO client/internal/engine.go:300: running rosenpass in permissive mode
2024-12-27T06:35:55Z ERRO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:418: peer not found: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A=
2024-12-27T06:35:55Z INFO client/internal/rosenpass/manager.go:158: starting rosenpass server on port 34272
2024-12-27T06:35:55Z INFO client/internal/routemanager/manager.go:142: Routing setup complete
2024-12-27T06:35:55Z INFO client/firewall/create_linux.go:51: creating an nftables firewall manager
2024-12-27T06:35:55Z INFO client/internal/dns/host_unix.go:68: System DNS manager discovered: systemd
2024-12-27T06:35:55Z INFO signal/client/grpc.go:147: connected to the Signal Service stream
2024-12-27T06:35:55Z INFO client/internal/engine.go:1374: Network monitor is disabled, not starting
2024-12-27T06:35:55Z INFO client/internal/connect.go:286: Netbird engine started, the IP is: 100.90.79.155/16
2024-12-27T06:35:55Z INFO management/client/grpc.go:155: connected to the Management Service stream
2024-12-27T06:35:55Z WARN client/internal/engine.go:554: running SSH server is not permitted
2024-12-27T06:35:55Z ERRO client/internal/dns/server.go:322: unable to parse ip address, error: ParseAddr(""): unable to parse IP
2024-12-27T06:35:55Z INFO client/internal/acl/manager.go:52: ACL rules processed in: 39.879012ms, total rules count: 350
2024-12-27T06:35:55Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redacted
2024-12-27T06:35:55Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :1
2024-12-27T06:35:55Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:1
2024-12-27T06:35:55Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A=
2024-12-27T06:35:55Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :2
2024-12-27T06:35:55Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:2
2024-12-27T06:35:55Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedqA3YQ=
2024-12-27T06:35:55Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :3
2024-12-27T06:35:55Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:3
2024-12-27T06:35:55Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:55Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedp50ILYJOXmWcMN6ci43tI=
2024-12-27T06:35:55Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :4
2024-12-27T06:35:55Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:4
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedAUM=
2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :5
2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:5
2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedteYT8KKEKmSGB8HbQMMVnR4c=
2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :6
2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:6
2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedF7j8MAO5OITBusHC4Lh/ww=
2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :7
2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:7
2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedbXBhk6Jn4dYzMiCQCqXRZ5WNMFgMemSTPyk=
2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :8
2024-12-27T06:35:56Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:8
2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedCEiFwKvDM+LYhWTbhypMRxrKU=
2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :9
2024-12-27T06:35:56Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:9
2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedYCOQWBtzvOM=
2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :10
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:10
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedtxUqu9wJBfRot5k/SiGBB3eZbmbxrYQ=
2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :11
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:11
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:56Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:57Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:35:57Z INFO [peer: part-redactedinternal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:57Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:57Z INFO [peer: part-redactedinternal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:57Z INFO [peer: part-redactedinternal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:35:57Z INFO [peer: part-redactedinternal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:35:58Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:36:04Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:36:05Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:36:08Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:36:22Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:36:37Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:36:38Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:36:45Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:36:46Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:36:59Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:37:00Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:37:03Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:37:13Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:37:15Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:37:42Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:39:25Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:187: WireGuard handshake timed out, closing relay connection: 2024-12-27 06:35:55.939451497 +0000 GMT
2024-12-27T06:39:25Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:501: free up connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A=
2024-12-27T06:39:44Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A=
2024-12-27T06:39:44Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :12
2024-12-27T06:39:44Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:12
2024-12-27T06:39:44Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:39:48Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:39:50Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:39:52Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:39:53Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:39:53Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:39:59Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:41:05Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:41:06Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:41:07Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection
2024-12-27T06:43:14Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:187: WireGuard handshake timed out, closing relay connection: 2024-12-27 06:39:44.652828966 +0000 GMT
2024-12-27T06:43:14Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:501: free up connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A=
2024-12-27T06:45:35Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A=
2024-12-27T06:45:35Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :13
2024-12-27T06:45:35Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:13
2024-12-27T06:45:36Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay
2024-12-27T06:45:39Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:45:41Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:45:44Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:45:51Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:46:03Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:46:26Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:47:04Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:47:46Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:48:27Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:49:06Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:49:08Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:49:11Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:49:18Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:49:30Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:49:55Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:50:34Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:50:35Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:50:38Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:50:44Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:50:56Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:51:22Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:52:55Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:52:59Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:53:01Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:53:04Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:53:10Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:53:23Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:53:49Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:54:28Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:55:08Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:55:48Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:56:27Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:57:07Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:57:47Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:58:26Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:59:06Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:59:45Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:59:59Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection
2024-12-27T06:59:59Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection
@rihards-simanovics commented on GitHub (Dec 27, 2024): Update: forgot to mention the downtime detection was at around `2024-12-27 06:30:31` I am not sure if this is related, but each time, one of my servers, which does auto-upgrade at around 6 pm, dies after upgrading to 0.35.1. Right now, it would seem that at least for me, client 0.29.4 is the last stable version I can rely on to work well: /var/log/apt/history.log ```log Start-Date: 2024-12-27 06:26:12 Upgrade: netbird:amd64 (0.29.4, 0.35.1) End-Date: 2024-12-27 06:26:17 Start-Date: 2024-12-27 06:35:49 Commandline: apt install netbird=0.29.4 Downgrade: netbird:amd64 (0.35.1, 0.29.4) End-Date: 2024-12-27 06:35:54 ``` cat /var/log/netbird/client.log ```log 2024-12-27T06:26:13Z INFO client/internal/engine.go:251: Network monitor: stopped 2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:521: closing all peer connections 2024-12-27T06:26:13Z INFO client/internal/connect.go:300: stopped NetBird client 2024-12-27T06:26:13Z INFO client/cmd/root.go:191: shutdown signal received 2024-12-27T06:26:13Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:330: start to Relay read loop exit 2024-12-27T06:26:13Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:529: waiting for read loop to close 2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:531: relay connection closed 2024-12-27T06:26:13Z WARN [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:516: relay connection was already marked as not running 2024-12-27T06:26:13Z INFO [peer: part-redactedinternal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redactedinternal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [peer: part-redactedinternal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redactedinternal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [peer: part-redacted] client/internal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redactedwTT0=] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redactedc8aELBa96mmk=] client/internal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z INFO [peer: part-redactedinternal/peer/conn.go:214: close peer connection 2024-12-27T06:26:13Z WARN [peer: part-redactedinternal/peer/worker_relay.go:156: failed to close relay connection: connection already closed 2024-12-27T06:26:13Z ERRO client/internal/wgproxy/proxy_ebpf.go:176: failed to read UDP pkg from WG: read udp 127.0.0.1:3128: use of closed network connection 2024-12-27T06:26:13Z INFO client/internal/dns/systemd_linux.go:161: reverting link settings and flushing cache 2024-12-27T06:26:13Z INFO client/internal/routemanager/manager.go:177: Routing cleanup complete 2024-12-27T06:26:13Z INFO iface/iface.go:192: interface wt0 has been removed 2024-12-27T06:26:13Z INFO client/internal/engine.go:271: stopped Netbird Engine 2024-12-27T06:26:13Z INFO client/server/server.go:625: service is down 2024-12-27T06:26:15Z INFO client/cmd/service_controller.go:90: stopped Netbird service 2024-12-27T06:26:17Z INFO client/cmd/service_controller.go:24: starting Netbird service 2024-12-27T06:26:17Z INFO client/cmd/service_controller.go:68: started daemon server: /var/run/netbird.sock 2024-12-27T06:26:17Z INFO client/internal/connect.go:115: starting NetBird client version 0.35.1 on linux/amd64 2024-12-27T06:26:18Z INFO client/internal/connect.go:247: connecting to the Relay service(s): rels://vpn-subdomain.domain.com:443 2024-12-27T06:26:18Z INFO relay/client/picker.go:72: try to connecting to relay server: rels://vpn-subdomain.domain.com:443 2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:165: create new relay connection: local peerID: redacted, local peer hashedID: part-redacted 2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:171: connecting to relay server 2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:188: relay connection established 2024-12-27T06:26:18Z INFO relay/client/picker.go:90: connected to Relay server: rels://vpn-subdomain.domain.com:443 2024-12-27T06:26:18Z INFO relay/client/picker.go:64: chosen home Relay server: rels://vpn-subdomain.domain.com:443 2024-12-27T06:26:18Z INFO client/iface/wgproxy/ebpf/proxy.go:91: local wg proxy listening on: 3128 2024-12-27T06:26:18Z INFO client/iface/wgproxy/factory_kernel.go:29: WireGuard Proxy Factory will produce eBPF proxy 2024-12-27T06:26:18Z INFO client/internal/engine.go:352: rosenpass is enabled 2024-12-27T06:26:18Z INFO client/internal/engine.go:354: running rosenpass in permissive mode 2024-12-27T06:26:18Z INFO client/internal/rosenpass/manager.go:158: starting rosenpass server on port 49146 2024-12-27T06:26:18Z INFO client/internal/routemanager/manager.go:163: Routing setup complete 2024-12-27T06:26:18Z INFO client/firewall/create_linux.go:72: creating an nftables firewall manager 2024-12-27T06:26:18Z INFO client/internal/dns/host_unix.go:54: System DNS manager discovered: systemd 2024-12-27T06:26:18Z INFO client/internal/peer/guard/sr_watcher.go:106: reconnected to Signal or Relay server 2024-12-27T06:26:18Z INFO signal/client/grpc.go:149: connected to the Signal Service stream 2024-12-27T06:26:18Z INFO client/internal/engine.go:1472: Network monitor is disabled, not starting 2024-12-27T06:26:18Z INFO client/internal/connect.go:273: Netbird engine started, the IP is: 100.90.79.155/16 2024-12-27T06:26:18Z INFO management/client/grpc.go:155: connected to the Management Service stream 2024-12-27T06:26:18Z INFO relay/client/manager.go:220: update relay server URLs: [rels://vpn-subdomain.domain.com:443] 2024-12-27T06:26:18Z WARN client/internal/engine.go:644: running SSH server is not permitted 2024-12-27T06:26:18Z INFO client/internal/acl/manager.go:61: ACL rules processed in: 48.875679ms, total rules count: 276 2024-12-27T06:26:18Z ERRO client/internal/dns/server.go:374: unable to parse ip address, error: ParseAddr(""): unable to parse IP 2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedbXBhk6Jn4dYzMiCQCqXRZ5WNMFgMemSTPyk= 2024-12-27T06:26:18Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :1 2024-12-27T06:26:18Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:1 2024-12-27T06:26:18Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedqA3YQ= 2024-12-27T06:26:18Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :2 2024-12-27T06:26:18Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:2 2024-12-27T06:26:18Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:138: start listen for reconnect events... 2024-12-27T06:26:18Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:18Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:138: start listen for reconnect events... 2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedCEiFwKvDM+LYhWTbhypMRxrKU= 2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :3 2024-12-27T06:26:19Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:3 2024-12-27T06:26:19Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedtxUqu9wJBfRot5k/SiGBB3eZbmbxrYQ= 2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :4 2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:4 2024-12-27T06:26:19Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:19Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedAUM= 2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :5 2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:5 2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedp50ILYJOXmWcMN6ci43tI= 2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :6 2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:6 2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redacted 2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :7 2024-12-27T06:26:19Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:7 2024-12-27T06:26:19Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:19Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedteYT8KKEKmSGB8HbQMMVnR4c= 2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :8 2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:8 2024-12-27T06:26:19Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:19Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedF7j8MAO5OITBusHC4Lh/ww= 2024-12-27T06:26:19Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :9 2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:9 2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:19Z INFO [peer: part-redactedinternal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:20Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redactedYCOQWBtzvOM= 2024-12-27T06:26:20Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :10 2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:10 2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:26:20Z INFO [peer: part-redacted] client/internal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:20Z INFO [peer: part-redactedinternal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:26:21Z INFO [peer: part-redactedinternal/peer/guard/guard.go:84: start reconnect loop... 2024-12-27T06:26:21Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:84: start reconnect loop... 2024-12-27T06:26:21Z INFO [peer: part-redactedwTT0=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2024-12-27T06:26:21Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2024-12-27T06:26:21Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:84: start reconnect loop... 2024-12-27T06:26:21Z INFO [peer: part-redactedinternal/peer/guard/guard.go:84: start reconnect loop... 2024-12-27T06:26:22Z INFO [peer: part-redacted] client/internal/peer/guard/guard.go:84: start reconnect loop... 2024-12-27T06:26:22Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/guard/guard.go:84: start reconnect loop... 2024-12-27T06:26:22Z INFO [peer: part-redactedinternal/peer/guard/guard.go:84: start reconnect loop... 2024-12-27T06:26:33Z INFO [peer: part-redactedinternal/peer/conn.go:328: set ICE to active connection 2024-12-27T06:31:18Z INFO client/internal/peer/guard/sr_watcher.go:94: network changes detected by ICE agent 2024-12-27T06:35:30Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:217: open connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A= 2024-12-27T06:35:30Z INFO client/iface/wgproxy/ebpf/proxy.go:102: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :11 2024-12-27T06:35:30Z INFO [peer: part-redacted] client/internal/peer/conn.go:444: created new wgProxy for relay connection: 127.0.0.1:11 2024-12-27T06:35:30Z INFO [peer: part-redacted] client/internal/peer/conn.go:473: start to communicate with peer via relay 2024-12-27T06:35:50Z INFO client/cmd/root.go:192: shutdown signal received 2024-12-27T06:35:50Z INFO client/internal/engine.go:279: Network monitor: stopped 2024-12-27T06:35:50Z INFO client/internal/dns/systemd_linux.go:166: reverting link settings and flushing cache 2024-12-27T06:35:50Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:536: closing all peer connections 2024-12-27T06:35:50Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:345: start to Relay read loop exit 2024-12-27T06:35:50Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:544: waiting for read loop to close 2024-12-27T06:35:50Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:546: relay connection closed 2024-12-27T06:35:50Z WARN [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:531: relay connection was already marked as not running 2024-12-27T06:35:50Z INFO client/internal/routemanager/manager.go:219: Routing cleanup complete 2024-12-27T06:35:50Z INFO [peer: part-redactedinternal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redactedinternal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redactedwTT0=] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redactedinternal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redactedinternal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redactedc8aELBa96mmk=] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redactedinternal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redactedinternal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z INFO [peer: part-redacted] client/internal/peer/conn.go:215: close peer connection 2024-12-27T06:35:50Z WARN [peer: part-redacted] client/internal/peer/worker_relay.go:155: failed to close relay connection: use of closed network connection 2024-12-27T06:35:50Z ERRO client/iface/bind/udp_mux_universal.go:93: error while reading packet: shared socked stopped 2024-12-27T06:35:50Z INFO client/iface/iface.go:220: interface wt0 has been removed 2024-12-27T06:35:51Z INFO client/internal/engine.go:317: stopped Netbird Engine 2024-12-27T06:35:51Z INFO client/server/server.go:648: service is down 2024-12-27T06:35:51Z INFO client/internal/connect.go:296: stopped NetBird client 2024-12-27T06:35:53Z INFO client/cmd/service_controller.go:94: stopped Netbird service 2024-12-27T06:35:54Z INFO client/cmd/service_controller.go:24: starting Netbird service 2024-12-27T06:35:54Z INFO client/cmd/service_controller.go:66: started daemon server: /var/run/netbird.sock 2024-12-27T06:35:54Z INFO client/internal/connect.go:117: starting NetBird client version 0.29.4 on linux/amd64 2024-12-27T06:35:54Z INFO client/internal/connect.go:252: connecting to the Relay service(s): rels://vpn-subdomain.domain.com:443 2024-12-27T06:35:54Z INFO relay/client/picker.go:66: try to connecting to relay server: rels://vpn-subdomain.domain.com:443 2024-12-27T06:35:54Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:165: create new relay connection: local peerID: redacted, local peer hashedID: part-redacted 2024-12-27T06:35:54Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:171: connecting to relay server 2024-12-27T06:35:54Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:192: relay connection established 2024-12-27T06:35:54Z INFO relay/client/picker.go:84: connected to Relay server: rels://vpn-subdomain.domain.com:443 2024-12-27T06:35:54Z INFO relay/client/picker.go:58: chosen home Relay server: rels://vpn-subdomain.domain.com:443 2024-12-27T06:35:54Z INFO client/internal/wgproxy/proxy_ebpf.go:89: local wg proxy listening on: 3128 2024-12-27T06:35:54Z INFO client/internal/engine.go:298: rosenpass is enabled 2024-12-27T06:35:54Z INFO client/internal/engine.go:300: running rosenpass in permissive mode 2024-12-27T06:35:55Z ERRO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:418: peer not found: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A= 2024-12-27T06:35:55Z INFO client/internal/rosenpass/manager.go:158: starting rosenpass server on port 34272 2024-12-27T06:35:55Z INFO client/internal/routemanager/manager.go:142: Routing setup complete 2024-12-27T06:35:55Z INFO client/firewall/create_linux.go:51: creating an nftables firewall manager 2024-12-27T06:35:55Z INFO client/internal/dns/host_unix.go:68: System DNS manager discovered: systemd 2024-12-27T06:35:55Z INFO signal/client/grpc.go:147: connected to the Signal Service stream 2024-12-27T06:35:55Z INFO client/internal/engine.go:1374: Network monitor is disabled, not starting 2024-12-27T06:35:55Z INFO client/internal/connect.go:286: Netbird engine started, the IP is: 100.90.79.155/16 2024-12-27T06:35:55Z INFO management/client/grpc.go:155: connected to the Management Service stream 2024-12-27T06:35:55Z WARN client/internal/engine.go:554: running SSH server is not permitted 2024-12-27T06:35:55Z ERRO client/internal/dns/server.go:322: unable to parse ip address, error: ParseAddr(""): unable to parse IP 2024-12-27T06:35:55Z INFO client/internal/acl/manager.go:52: ACL rules processed in: 39.879012ms, total rules count: 350 2024-12-27T06:35:55Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redacted 2024-12-27T06:35:55Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :1 2024-12-27T06:35:55Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:1 2024-12-27T06:35:55Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A= 2024-12-27T06:35:55Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :2 2024-12-27T06:35:55Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:2 2024-12-27T06:35:55Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedqA3YQ= 2024-12-27T06:35:55Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :3 2024-12-27T06:35:55Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:3 2024-12-27T06:35:55Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:55Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedp50ILYJOXmWcMN6ci43tI= 2024-12-27T06:35:55Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :4 2024-12-27T06:35:55Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:4 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedAUM= 2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :5 2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:5 2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedteYT8KKEKmSGB8HbQMMVnR4c= 2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :6 2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:6 2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedF7j8MAO5OITBusHC4Lh/ww= 2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :7 2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:7 2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedbXBhk6Jn4dYzMiCQCqXRZ5WNMFgMemSTPyk= 2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :8 2024-12-27T06:35:56Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:8 2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedCEiFwKvDM+LYhWTbhypMRxrKU= 2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :9 2024-12-27T06:35:56Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:9 2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [peer: part-redactedwTT0=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:56Z INFO [peer: part-redactedinternal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedYCOQWBtzvOM= 2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :10 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:10 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:35:56Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redactedtxUqu9wJBfRot5k/SiGBB3eZbmbxrYQ= 2024-12-27T06:35:56Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :11 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:11 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:35:56Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:56Z INFO [peer: part-redactedc8aELBa96mmk=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:57Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:35:57Z INFO [peer: part-redactedinternal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:57Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:57Z INFO [peer: part-redactedinternal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:57Z INFO [peer: part-redactedinternal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:35:57Z INFO [peer: part-redactedinternal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:35:58Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:36:04Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:36:05Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:36:08Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:36:22Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:36:37Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:36:38Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:36:45Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:36:46Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:36:59Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:37:00Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:37:03Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:37:13Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:37:15Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:37:42Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:39:25Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:187: WireGuard handshake timed out, closing relay connection: 2024-12-27 06:35:55.939451497 +0000 GMT 2024-12-27T06:39:25Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:501: free up connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A= 2024-12-27T06:39:44Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A= 2024-12-27T06:39:44Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :12 2024-12-27T06:39:44Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:12 2024-12-27T06:39:44Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:39:48Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:39:50Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:39:52Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:39:53Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:39:53Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:39:59Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:41:05Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:41:06Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:41:07Z INFO [peer: part-redacted0MimoQhLOtQCq554kE=] client/internal/peer/conn.go:430: set ICE to active connection 2024-12-27T06:43:14Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:187: WireGuard handshake timed out, closing relay connection: 2024-12-27 06:39:44.652828966 +0000 GMT 2024-12-27T06:43:14Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:501: free up connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A= 2024-12-27T06:45:35Z INFO [relay: rels://vpn-subdomain.domain.com:443] relay/client/client.go:214: open connection to peer: part-redacted7DCtO3oEPv1dxzUDun+O4wBAbyZ9A= 2024-12-27T06:45:35Z INFO client/internal/wgproxy/proxy_ebpf.go:101: turn conn added to wg proxy store: rels://vpn-subdomain.domain.com:443, endpoint port: :13 2024-12-27T06:45:35Z INFO [peer: part-redacted] client/internal/peer/conn.go:536: created new wgProxy for relay connection: 127.0.0.1:13 2024-12-27T06:45:36Z INFO [peer: part-redacted] client/internal/peer/conn.go:577: start to communicate with peer via relay 2024-12-27T06:45:39Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:45:41Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:45:44Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:45:51Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:46:03Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:46:26Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:47:04Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:47:46Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:48:27Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:49:06Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:49:08Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:49:11Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:49:18Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:49:30Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:49:55Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:50:34Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:50:35Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:50:38Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:50:44Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:50:56Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:51:22Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:52:55Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:52:59Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:53:01Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:53:04Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:53:10Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:53:23Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:53:49Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:54:28Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:55:08Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:55:48Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:56:27Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:57:07Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:57:47Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:58:26Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:59:06Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:59:45Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:59:59Z INFO [peer: part-redacted] client/internal/peer/worker_relay.go:77: do not need to reopen relay connection 2024-12-27T06:59:59Z INFO [peer: part-redacted] client/internal/peer/conn.go:430: set ICE to active connection ```
saavagebueno commented 2025-11-20 05:31:59 -05:00
Author
Owner
Copy Link

@wakawakaaa commented on GitHub (Dec 28, 2024):

same issue I have.

@wakawakaaa commented on GitHub (Dec 28, 2024): same issue I have.
saavagebueno commented 2025-11-20 05:32:00 -05:00
Author
Owner
Copy Link

@Gauss23 commented on GitHub (Dec 30, 2024):

Same for me. Seems to be something from 0.31 upwards. With 0.31 I don't see those problems. Maybe it's related to the fix for this issue: https://github.com/netbirdio/netbird/issues/2973 ?
Before this the keys for Rosenpass didn't seem to rotate and therefore it just worked.

@Gauss23 commented on GitHub (Dec 30, 2024): Same for me. Seems to be something from 0.31 upwards. With 0.31 I don't see those problems. Maybe it's related to the fix for this issue: https://github.com/netbirdio/netbird/issues/2973 ? Before this the keys for Rosenpass didn't seem to rotate and therefore it just worked.
saavagebueno commented 2025-11-20 05:32:00 -05:00
Author
Owner
Copy Link

@cdestefano commented on GitHub (Jan 4, 2025):

@Gauss23 the fix for that linked ticket was introduced in 0.34 so it wouldn't affect > 0.31 && < 0.34. The rotation is working and I haven't had any issues with the client's losing communication.

Rosenpass by design is supposed to rotate the keys. If you want a static key, you can use a pre-shared key without rosenpass.

@cdestefano commented on GitHub (Jan 4, 2025): @Gauss23 the fix for that linked ticket was introduced in 0.34 so it wouldn't affect > 0.31 && < 0.34. The rotation is working and I haven't had any issues with the client's losing communication. Rosenpass by design is supposed to rotate the keys. If you want a static key, you can use a pre-shared key without rosenpass.
saavagebueno commented 2025-11-20 05:32:00 -05:00
Author
Owner
Copy Link

@pizzaandcheese commented on GitHub (Feb 28, 2025):

I to am experiencing this issue when self-hosting.

I am using different domains for the dashboard, relay, signal, management, turn, and stun addresses. The deshboard, Management, and Signal exist on the same server while the stun, turn, and relay services exist on another.

netbird status -d shows all endpoints are connected and updated configs are getting pushed from the netbird management endpoint but when quantum resistance is enabled on any client they end up working for around 2 -4 minutes before connectivity is lost

@pizzaandcheese commented on GitHub (Feb 28, 2025): I to am experiencing this issue when self-hosting. I am using different domains for the dashboard, relay, signal, management, turn, and stun addresses. The deshboard, Management, and Signal exist on the same server while the stun, turn, and relay services exist on another. netbird status -d shows all endpoints are connected and updated configs are getting pushed from the netbird management endpoint but when quantum resistance is enabled on any client they end up working for around 2 -4 minutes before connectivity is lost
saavagebueno commented 2025-11-20 05:32:00 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Jun 1, 2025):

Guys, is this issue still present in recent versions?

@mlsmaycon commented on GitHub (Jun 1, 2025): Guys, is this issue still present in recent versions?
saavagebueno commented 2025-11-20 05:32:00 -05:00
Author
Owner
Copy Link

@rihards-simanovics commented on GitHub (Jun 3, 2025):

Guys, is this issue still present in recent versions?

Hey @mlsmaycon, I wouldn't know at this point, most of my old instability issues appeared to have been caused by the Rosenpass options. Now that it is disabled, the network is very stable.

@rihards-simanovics commented on GitHub (Jun 3, 2025): > Guys, is this issue still present in recent versions? Hey @mlsmaycon, I wouldn't know at this point, most of my old instability issues appeared to have been caused by the Rosenpass options. Now that it is disabled, the network is very stable.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label bug client connection
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1510
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?