[GH-ISSUE #2566] New Relay public thread - Q&A and Issues discussions #5276

New Issue

Originally created by @mlsmaycon on GitHub (Sep 9, 2024).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/2566

Hello folks, this issue is open to any questions or problems regarding the new relay implementation.

Originally created by @mlsmaycon on GitHub (Sep 9, 2024). Original GitHub issue: https://github.com/netbirdio/netbird/issues/2566 Hello folks, this issue is open to any questions or problems regarding the new relay implementation.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 9, 2024):

Status information to confirm relay usage:

Peers detail:
 relay-test-ip-172-20-1-178-rly.netbird.selfhosted:
  NetBird IP: 100.89.101.6
  Public key: CdRpcUnzq2LM9v97VnU7JiiqE0Y4wXp379mXju0efjk=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://relay-eu1.stage.netbird.io <--------------- indicates the relay used to connect to the remote peer
  Last connection update: 2 seconds ago
  Last WireGuard handshake: 3 seconds ago
  Transfer status (received/sent) 92 B/180 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 relay-test-ip-172-20-14-148.netbird.selfhosted:
  NetBird IP: 100.89.212.227
  Public key: bhSrOMLvN+5cMnjWyL4gB+o9En2a1AvAGWNB5N+gEGw=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/srflx
  ICE candidate endpoints (Local/Remote): 192.168.178.38:51820/1.2.3.4:51820
  Relay server address: rels://relay-eu2.stage.netbird.io. <--------------- indicates the relay used to connect to the remote peer ( there is a bug which this needs to be cleaned after P2P connection)
  Last connection update: 2 seconds ago
  Last WireGuard handshake: 3 seconds ago
  Transfer status (received/sent) 92 B/180 B
  Quantum resistance: false
  Routes: 34.160.111.145/32
  Latency: 28.5755ms

OS: darwin/arm64
Daemon version: 0.29.0
CLI version: 0.29.0
Management: Connected to [https://test.stage.netbird.io:443](https://test.stage.netbird.io/)
Signal: Connected to [https://signal.stage.netbird.io:443](https://signal.stage.netbird.io/)
Relays:
  [stun:test.stage.netbird.io:3478] is Available
  [turn:test.stage.netbird.io:3478?transport=udp] is Available
  [rels://relay-eu1.stage.netbird.io] is Available.    <--------------- indicates the relay used by your local client (the home relay)
Nameservers:
  [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: maycons-macbook-pro-2-1.netbird.selfhosted
NetBird IP: 100.89.107.107/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 2/2 Connected

@mlsmaycon commented on GitHub (Sep 9, 2024): Status information to confirm relay usage: ```shell Peers detail: relay-test-ip-172-20-1-178-rly.netbird.selfhosted: NetBird IP: 100.89.101.6 Public key: CdRpcUnzq2LM9v97VnU7JiiqE0Y4wXp379mXju0efjk= Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://relay-eu1.stage.netbird.io <--------------- indicates the relay used to connect to the remote peer Last connection update: 2 seconds ago Last WireGuard handshake: 3 seconds ago Transfer status (received/sent) 92 B/180 B Quantum resistance: false Routes: - Latency: 0s relay-test-ip-172-20-14-148.netbird.selfhosted: NetBird IP: 100.89.212.227 Public key: bhSrOMLvN+5cMnjWyL4gB+o9En2a1AvAGWNB5N+gEGw= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/srflx ICE candidate endpoints (Local/Remote): 192.168.178.38:51820/1.2.3.4:51820 Relay server address: rels://relay-eu2.stage.netbird.io. <--------------- indicates the relay used to connect to the remote peer ( there is a bug which this needs to be cleaned after P2P connection) Last connection update: 2 seconds ago Last WireGuard handshake: 3 seconds ago Transfer status (received/sent) 92 B/180 B Quantum resistance: false Routes: 34.160.111.145/32 Latency: 28.5755ms OS: darwin/arm64 Daemon version: 0.29.0 CLI version: 0.29.0 Management: Connected to [https://test.stage.netbird.io:443](https://test.stage.netbird.io/) Signal: Connected to [https://signal.stage.netbird.io:443](https://signal.stage.netbird.io/) Relays: [stun:test.stage.netbird.io:3478] is Available [turn:test.stage.netbird.io:3478?transport=udp] is Available [rels://relay-eu1.stage.netbird.io] is Available. <--------------- indicates the relay used by your local client (the home relay) Nameservers: [8.8.8.8:53, 8.8.4.4:53] for [.] is Available FQDN: maycons-macbook-pro-2-1.netbird.selfhosted NetBird IP: 100.89.107.107/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 2/2 Connected ```

saavagebueno commented

@allroundtechie commented on GitHub (Sep 9, 2024):

Hi,

I have some questions about the new relay which are not clear to me.

In the release notes you wrote "We are moving away from the TURN relay (coturn) to our own relay implementation based on WebSocket".
If I take that literally this means that "only" the TURN part of coturn gets replaced but not the STUN part. Is this correct and the release only the first step to replace coturn completely or is the STUN part also already replaced with the new relay?
In the example mentioned above which indicates the relay is used it is I guess active in a secured version but in the release notes only this part is mentioned: "Addresses": ["rel://:"]
Can you enable TLS in the new relay and if yes how? Or is this something for a future release?
I am using Traefik as a reverse proxy and also have implemented Netbird like described in the documentation which works well. I am missing documentation around the new relay and a reverse proxy.

Thanks in advance and also many thanks for your awesome work in building this great software stack!

@allroundtechie commented on GitHub (Sep 9, 2024): Hi, I have some questions about the new relay which are not clear to me. 1. In the release notes you wrote "We are moving away from the TURN relay (coturn) to our own relay implementation based on WebSocket". If I take that literally this means that "only" the TURN part of coturn gets replaced but not the STUN part. Is this correct and the release only the first step to replace coturn completely or is the STUN part also already replaced with the new relay? 2. In the example mentioned above which indicates the relay is used it is I guess active in a secured version but in the release notes only this part is mentioned: "Addresses": ["rel://<DOMAIN>:<PORT>"] Can you enable TLS in the new relay and if yes how? Or is this something for a future release? 3. I am using Traefik as a reverse proxy and also have implemented Netbird like described in the documentation which works well. I am missing documentation around the new relay and a reverse proxy. Thanks in advance and also many thanks for your awesome work in building this great software stack!

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 9, 2024):

@landmass-deftly-reptile-budget:

Stun is still going to be required for the P2P discovery. Also, for retro-compatibility, TURN is still required.
The supported URLs are rel:// and rels://, where rels is used for TLS connections. Like signal and management, the relay have Let's Encrypt support, and you can use the environment variables below to enable it:

NB_EXPOSED_ADDRESS=rels://relay.example.com:443  # update the port configuration to match it
NB_LETSENCRYPT_DOMAINS=relay.example.com # should match the exposed address
NB_LETSENCRYPT_DATA_DIR=/etc/letsencrypt # mount this directory for persistency
NB_LETSENCRYPT_EMAIL=admin@relay.example.com
#NB_LETSENCRYPT_AWS_ROUTE53=true # in case you want to use route 53 for issuing the certificate

It also supports certificate files with:

NB_TLS_CERT_FILE=/etc/certificates/cert.crt
NB_TLS_KEy_FILE=/etc/certificates/cert.key

Once this is done, add the exposed address to the management.json file and restart the file.

Relay should work fine behind traefik. We are missing the configuration, but the traffic to the service can be routed with either a domain or with the /relay path prefix.

@mlsmaycon commented on GitHub (Sep 9, 2024): @landmass-deftly-reptile-budget: 1. Stun is still going to be required for the P2P discovery. Also, for retro-compatibility, TURN is still required. 2. The supported URLs are `rel://` and `rels://`, where `rels` is used for TLS connections. Like signal and management, the relay have Let's Encrypt support, and you can use the environment variables below to enable it: ```shell NB_EXPOSED_ADDRESS=rels://relay.example.com:443 # update the port configuration to match it NB_LETSENCRYPT_DOMAINS=relay.example.com # should match the exposed address NB_LETSENCRYPT_DATA_DIR=/etc/letsencrypt # mount this directory for persistency NB_LETSENCRYPT_EMAIL=admin@relay.example.com #NB_LETSENCRYPT_AWS_ROUTE53=true # in case you want to use route 53 for issuing the certificate ``` It also supports certificate files with: ```shell NB_TLS_CERT_FILE=/etc/certificates/cert.crt NB_TLS_KEy_FILE=/etc/certificates/cert.key ``` Once this is done, add the exposed address to the `management.json` file and restart the file. 3. Relay should work fine behind traefik. We are missing the configuration, but the traffic to the service can be routed with either a domain or with the `/relay` path prefix.

saavagebueno commented

@ismail0234 commented on GitHub (Sep 9, 2024):

Hello, I have 2 questions. I am undecided whether to upgrade or not.

I don't fully understand the new Relay Feature. How will it benefit us?
What was the reason to switch to our own relay application? Was there something that the existing system did not meet?

@ismail0234 commented on GitHub (Sep 9, 2024): Hello, I have 2 questions. I am undecided whether to upgrade or not. 1. I don't fully understand the new Relay Feature. How will it benefit us? 2. What was the reason to switch to our own relay application? Was there something that the existing system did not meet?

saavagebueno commented

@bryanjuho commented on GitHub (Sep 9, 2024):

Is it okay to update to 0.29.0 without actually running the new relay image and changing management.json?

@bryanjuho commented on GitHub (Sep 9, 2024): Is it okay to update to 0.29.0 without actually running the new relay image and changing management.json?

saavagebueno commented

@rudradevpal commented on GitHub (Sep 10, 2024):

For new relay to work is there any new openwrt package released?

@rudradevpal commented on GitHub (Sep 10, 2024): For new relay to work is there any new openwrt package released?

saavagebueno commented

@Marcus1Pierce commented on GitHub (Sep 10, 2024):

Is it okay to use same domain for management, signal, coturn and relay?

Example: If i use domain netbird.domain.com and i want to use this domain for all services but with different port is that okay?

@Marcus1Pierce commented on GitHub (Sep 10, 2024): Is it okay to use same domain for management, signal, coturn and relay? Example: If i use domain netbird.domain.com and i want to use this domain for all services but with different port is that okay?

saavagebueno commented

@Zaunei commented on GitHub (Sep 10, 2024):

Also, for retro-compatibility, TURN is still required.

If I don't care about old clients, I can ignore TURN completely, right?

Otherwise, this sounds very promising, especially with Kubernetes, the port ranges of TURN have always made the setup a bit more complex. I will definitely give it a try and report back.

STUN will continue to be used in the future?

@Zaunei commented on GitHub (Sep 10, 2024): > 1. Also, for retro-compatibility, TURN is still required. If I don't care about old clients, I can ignore TURN completely, right? Otherwise, this sounds very promising, especially with Kubernetes, the port ranges of TURN have always made the setup a bit more complex. I will definitely give it a try and report back. STUN will continue to be used in the future?

saavagebueno commented

@WolfgangDpunkt commented on GitHub (Sep 10, 2024):

Replace PORT and DOMAIN according to your deployment.

I have used the automatic setup script, so I am probably using the default values for ports, so what do I need to specify here for PORT in the compose file?

@WolfgangDpunkt commented on GitHub (Sep 10, 2024): > Replace **PORT** and **DOMAIN** according to your deployment. I have used the automatic setup script, so I am probably using the default values for ports, so what do I need to specify here for **PORT** in the compose file?

saavagebueno commented

@ndziuba commented on GitHub (Sep 10, 2024):

Replace PORT and DOMAIN according to your deployment.

I have used the automatic setup script, so I am probably using the default values for ports, so what do I need to specify here for PORT in the compose file?

It can be found in the setup.env file.
The default port is 33080

@ndziuba commented on GitHub (Sep 10, 2024): > > Replace **PORT** and **DOMAIN** according to your deployment. > > I have used the automatic setup script, so I am probably using the default values for ports, so what do I need to specify here for **PORT** in the compose file? It can be found in the setup.env file. The default port is 33080

saavagebueno commented

@MDMeridio001 commented on GitHub (Sep 10, 2024):

Hello,

Is it possible to run the relay behind nginx acting as a proxy? I have tried by adding the following to my nginx configuration file, but it results in clients recieving a 400 error when trying to establish a connection to the relay. A direct connection without nginx in front works perfectly fine.

upstream relay-upstream {
    server 127.0.0.1:33080;
}

[...]

# Proxy Relay http endpoint
    location /relay/ {
        proxy_pass http://relay-upstream/relay;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
    }

@MDMeridio001 commented on GitHub (Sep 10, 2024): Hello, Is it possible to run the relay behind nginx acting as a proxy? I have tried by adding the following to my nginx configuration file, but it results in clients recieving a 400 error when trying to establish a connection to the relay. A direct connection without nginx in front works perfectly fine. ``` upstream relay-upstream { server 127.0.0.1:33080; } [...] # Proxy Relay http endpoint location /relay/ { proxy_pass http://relay-upstream/relay; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; proxy_set_header Host $host; } ```

saavagebueno commented

@mvivaldi commented on GitHub (Sep 10, 2024):

Hello,

Is it possible to run the relay behind nginx acting as a proxy? I have tried by adding the following to my nginx configuration file, but it results in clients recieving a 400 error when trying to establish a connection to the relay. A direct connection without nginx in front works perfectly fine.
upstream relay-upstream {
    server 127.0.0.1:33080;
}

[...]

# Proxy Relay http endpoint
    location /relay/ {
        proxy_pass http://relay-upstream/relay;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
    }

try add these:

      proxy_set_header Host            $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_cache_bypass $http_upgrade;

and delete the directive:

proxy_set_header Host $host;

@mvivaldi commented on GitHub (Sep 10, 2024): > Hello, > > Is it possible to run the relay behind nginx acting as a proxy? I have tried by adding the following to my nginx configuration file, but it results in clients recieving a 400 error when trying to establish a connection to the relay. A direct connection without nginx in front works perfectly fine. > > ``` > upstream relay-upstream { > server 127.0.0.1:33080; > } > > [...] > > # Proxy Relay http endpoint > location /relay/ { > proxy_pass http://relay-upstream/relay; > proxy_http_version 1.1; > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection "Upgrade"; > proxy_set_header Host $host; > } > ``` try add these: ``` proxy_set_header Host $http_host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $remote_addr; proxy_set_header X-Forwarded-Host $http_host; proxy_cache_bypass $http_upgrade; ``` and delete the directive: ``` proxy_set_header Host $host; ```

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 10, 2024):

Hello, I have 2 questions. I am undecided whether to upgrade or not.

I don't fully understand the new Relay Feature. How will it benefit us?

What was the reason to switch to our own relay application? Was there something that the existing system did not meet?

@ismail0234 some of the benefits of the new relay over Coturn:

More efficient relay connection for multiple peers: The ICE mode with TURN opens a connection with the TURN server for each peer connection. That consumes more resources on the client and on the coturn server.
The NetBird connection with the new relay is up to 15% faster than coturn.
The service is easier to run on self-hosted environments, since need to configure a single port.
Built-in TLS/SSL support

The main idea is to have a more efficient relay system for NetBird. Turn/Coturn is a really good system for short-term connections. As a connection via VPN usually lasts many hours or days, we need a more efficient system that can easily be scaled.

@mlsmaycon commented on GitHub (Sep 10, 2024): > Hello, I have 2 questions. I am undecided whether to upgrade or not. > > 1. I don't fully understand the new Relay Feature. How will it benefit us? > 2. What was the reason to switch to our own relay application? Was there something that the existing system did not meet? @ismail0234 some of the benefits of the new relay over Coturn: - More efficient relay connection for multiple peers: The ICE mode with TURN opens a connection with the TURN server for each peer connection. That consumes more resources on the client and on the coturn server. - The NetBird connection with the new relay is up to 15% faster than coturn. - The service is easier to run on self-hosted environments, since need to configure a single port. - Built-in TLS/SSL support The main idea is to have a more efficient relay system for NetBird. Turn/Coturn is a really good system for short-term connections. As a connection via VPN usually lasts many hours or days, we need a more efficient system that can easily be scaled.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 10, 2024):

Is it okay to update to 0.29.0 without actually running the new relay image and changing management.json?

Yes it is. You don't need to update or configure anything if you don't want. It should be fully compatible with older versions of the management.json file.

@mlsmaycon commented on GitHub (Sep 10, 2024): > Is it okay to update to 0.29.0 without actually running the new relay image and changing management.json? Yes it is. You don't need to update or configure anything if you don't want. It should be fully compatible with older versions of the management.json file.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 10, 2024):

For new relay to work is there any new openwrt package released?

We will look into updating the openwrt version.

@mlsmaycon commented on GitHub (Sep 10, 2024): > For new relay to work is there any new openwrt package released? We will look into updating the openwrt version.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 10, 2024):

Is it okay to use same domain for management, signal, coturn and relay?

Example: If i use domain netbird.domain.com and i want to use this domain for all services but with different port is that okay?

Yes it is possible.

@mlsmaycon commented on GitHub (Sep 10, 2024): > Is it okay to use same domain for management, signal, coturn and relay? > > Example: If i use domain netbird.domain.com and i want to use this domain for all services but with different port is that okay? Yes it is possible.

saavagebueno commented

@MDMeridio001 commented on GitHub (Sep 10, 2024):

Hello,
Is it possible to run the relay behind nginx acting as a proxy? I have tried by adding the following to my nginx configuration file, but it results in clients recieving a 400 error when trying to establish a connection to the relay. A direct connection without nginx in front works perfectly fine.
upstream relay-upstream {
    server 127.0.0.1:33080;
}

[...]

# Proxy Relay http endpoint
    location /relay/ {
        proxy_pass http://relay-upstream/relay;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
    }
try add these:
      proxy_set_header Host            $http_host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header X-Forwarded-Host $http_host;
      proxy_cache_bypass $http_upgrade;
and delete the directive:
proxy_set_header Host $host;

I added them but I am still getting the same error. I don't know if it is of any help but this is what I added to the docker-compose.yml file:

# Relay
  relay:
    image: netbirdio/relay:latest
    restart: unless-stopped
    environment:
    - NB_LOG_LEVEL=info
    - NB_LISTEN_ADDRESS=:33080
    - NB_EXPOSED_ADDRESS=netbird.mydomain.com:443
    - NB_AUTH_SECRET=<MYSECRET>
    ports:
      - 127.0.0.1:33080:33080
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

And this is what I added to management.json:

"Relay": {
        "Addresses": ["rel://netbird.mydomain.com:443/relay"],
        "CredentialsTTL": "24h",
        "Secret": "<MYSECRET>"
},

@MDMeridio001 commented on GitHub (Sep 10, 2024): > > Hello, > > Is it possible to run the relay behind nginx acting as a proxy? I have tried by adding the following to my nginx configuration file, but it results in clients recieving a 400 error when trying to establish a connection to the relay. A direct connection without nginx in front works perfectly fine. > > ``` > > upstream relay-upstream { > > server 127.0.0.1:33080; > > } > > > > [...] > > > > # Proxy Relay http endpoint > > location /relay/ { > > proxy_pass http://relay-upstream/relay; > > proxy_http_version 1.1; > > proxy_set_header Upgrade $http_upgrade; > > proxy_set_header Connection "Upgrade"; > > proxy_set_header Host $host; > > } > > ``` > > try add these: > > ``` > proxy_set_header Host $http_host; > proxy_set_header X-Forwarded-Proto $scheme; > proxy_set_header X-Forwarded-For $remote_addr; > proxy_set_header X-Forwarded-Host $http_host; > proxy_cache_bypass $http_upgrade; > ``` > > and delete the directive: > > ``` > proxy_set_header Host $host; > ``` I added them but I am still getting the same error. I don't know if it is of any help but this is what I added to the docker-compose.yml file: ``` # Relay relay: image: netbirdio/relay:latest restart: unless-stopped environment: - NB_LOG_LEVEL=info - NB_LISTEN_ADDRESS=:33080 - NB_EXPOSED_ADDRESS=netbird.mydomain.com:443 - NB_AUTH_SECRET=<MYSECRET> ports: - 127.0.0.1:33080:33080 logging: driver: "json-file" options: max-size: "500m" max-file: "2" ``` And this is what I added to management.json: ``` "Relay": { "Addresses": ["rel://netbird.mydomain.com:443/relay"], "CredentialsTTL": "24h", "Secret": "<MYSECRET>" }, ```

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 10, 2024):

@MDMeridio001 it seems like you are using nginx for SSL termination too, in that case, try this:

    - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443

and

"Relay": {
        "Addresses": ["rels://netbird.mydomain.com:443"],
        "CredentialsTTL": "24h",
        "Secret": "<MYSECRET>"
},

@mlsmaycon commented on GitHub (Sep 10, 2024): @MDMeridio001 it seems like you are using nginx for SSL termination too, in that case, try this: ```yaml - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443 ``` and ```json "Relay": { "Addresses": ["rels://netbird.mydomain.com:443"], "CredentialsTTL": "24h", "Secret": "<MYSECRET>" }, ```

saavagebueno commented

@MDMeridio001 commented on GitHub (Sep 10, 2024):

@MDMeridio001 it seems like you are using nginx for SSL termination too, in that case, try this:

    - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443

and

"Relay": {
        "Addresses": ["rels://netbird.mydomain.com:443"],
        "CredentialsTTL": "24h",
        "Secret": "<MYSECRET>"
},

I completely forgot I needed to add "rels://", thank you so much, it's working fine now.

@MDMeridio001 commented on GitHub (Sep 10, 2024): > @MDMeridio001 it seems like you are using nginx for SSL termination too, in that case, try this: > > ```yaml > - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443 > ``` > > and > > ```json > "Relay": { > "Addresses": ["rels://netbird.mydomain.com:443"], > "CredentialsTTL": "24h", > "Secret": "<MYSECRET>" > }, > ``` I completely forgot I needed to add "rels://", thank you so much, it's working fine now.

saavagebueno commented

@rgdev commented on GitHub (Sep 10, 2024):

Assuming a brand new deployment and all clients running 0.29+ where does coturn fit in the picture ? Can we just run coturn with --stun-only if retrocompability is no concern ?

@rgdev commented on GitHub (Sep 10, 2024): Assuming a brand new deployment and all clients running 0.29+ where does coturn fit in the picture ? Can we just run coturn with `--stun-only` if retrocompability is no concern ?

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 10, 2024):

@rgdev With a new deployment, it is very likely that Coturn will only be used with mobile clients until we update them.

@mlsmaycon commented on GitHub (Sep 10, 2024): @rgdev With a new deployment, it is very likely that Coturn will only be used with mobile clients until we update them.

saavagebueno commented

@Roeda commented on GitHub (Sep 10, 2024):

@rgdev With a new deployment, it is very likely that Coturn will only be used with mobile clients until we update them.

Excuse my confusion, but since you say that you still use STUN for peer discovery, and at the same time Coturn won’t be used when the mobile apps are updated. Does that mean that the STUN service is baked into the new Relay now (or the management service) ? (Would we be ultimately able to remove Coturn from docker compose and the management.json ?)
Thank you very much for this new implementation it sounds cool and production friendly

@Roeda commented on GitHub (Sep 10, 2024): > @rgdev With a new deployment, it is very likely that Coturn will only be used with mobile clients until we update them. Excuse my confusion, but since you say that you still use STUN for peer discovery, and at the same time Coturn won’t be used when the mobile apps are updated. Does that mean that the STUN service is baked into the new Relay now (or the management service) ? (Would we be ultimately able to remove Coturn from docker compose and the management.json ?) Thank you very much for this new implementation it sounds cool and production friendly

saavagebueno commented

@wehagy commented on GitHub (Sep 10, 2024):

For new relay to work is there any new openwrt package released?

We will look into updating the openwrt version.

I am updating netbird package against openwrt snapshot for months, and I have no problem so far, in fact I have built the new version 0.29.0 and is working fine, and open a PR https://github.com/openwrt/packages/pull/24950, for now I just see the error 2024-09-10T15:25:09-03:00 INFO [peer: [ REDACTED ]=] client/internal/peer/worker_relay.go:59: Relay is not supported by remote peer, probably because I'm not selfhosting, and from release notes:

Cloud support for the new relay feature is coming soon*.

But I'm not backporting to openwrt 23.05, one of my targets is supported only on openwrt snapshot.

And to be honest someone open a issue https://github.com/openwrt/packages/issues/24569#issuecomment-2246451384 on openwrt repo to backport a new version, I offered my help to the person if he could test it, but I got no response.

@wehagy commented on GitHub (Sep 10, 2024): > > For new relay to work is there any new openwrt package released? > > We will look into updating the openwrt version. I am updating `netbird` package against `openwrt snapshot` for months, and I have no problem so far, in fact I have built the new version `0.29.0` and is working fine, and open a PR https://github.com/openwrt/packages/pull/24950, for now I just see the error `2024-09-10T15:25:09-03:00 INFO [peer: [ REDACTED ]=] client/internal/peer/worker_relay.go:59: Relay is not supported by remote peer`, probably because I'm not selfhosting, and from release notes: > - Cloud support for the new relay feature is coming soon*. But I'm not backporting to `openwrt 23.05`, one of my targets is supported only on `openwrt snapshot`. And to be honest someone open a issue https://github.com/openwrt/packages/issues/24569#issuecomment-2246451384 on `openwrt` repo to backport a new version, I offered my help to the person if he could test it, but I got no response.

saavagebueno commented

@ismail0234 commented on GitHub (Sep 10, 2024):

@mlsmaycon Thanks for the explanation. Do you think about optimization on the api side? The api slows down after 200 peers connected to the system. After 500 peers, it slows down a lot. Each request takes more than 1-2 seconds.

In the test measurements I made, these are the response times returned from the api according to the number of peers connected to the system.

20 Peers: 200-300 ms
100 Peers 300-600 ms
200 Peers: 500-1000 ms
500 Peers: 1500-3000 ms

@ismail0234 commented on GitHub (Sep 10, 2024): @mlsmaycon Thanks for the explanation. Do you think about optimization on the api side? The api slows down after 200 peers connected to the system. After 500 peers, it slows down a lot. Each request takes more than 1-2 seconds. In the test measurements I made, these are the response times returned from the api according to the number of peers connected to the system. 20 Peers: 200-300 ms 100 Peers 300-600 ms 200 Peers: 500-1000 ms 500 Peers: 1500-3000 ms

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 11, 2024):

Hey folks, we have a new release, 0.29.1. This release improves the relay with better authentication messages. To ensure your system is working properly, you should upgrade your relay and management servers before upgrading your clients.

@mlsmaycon commented on GitHub (Sep 11, 2024): Hey folks, we have a new release, 0.29.1. This release improves the relay with better authentication messages. To ensure your system is working properly, you should upgrade your relay and management servers before upgrading your clients.

saavagebueno commented

@allroundtechie commented on GitHub (Sep 11, 2024):

Works like a charm, thanks!

@allroundtechie commented on GitHub (Sep 11, 2024): Works like a charm, thanks!

saavagebueno commented

@marcportabellaclotet-mt commented on GitHub (Sep 11, 2024):

Thanks for improving the relay functionality.
I can't find the relay repo in netbirdio github. Will it be private or closed source?

@marcportabellaclotet-mt commented on GitHub (Sep 11, 2024): Thanks for improving the relay functionality. I can't find the relay repo in [netbirdio github](https://github.com/netbirdio/). Will it be private or closed source?

saavagebueno commented

https://github.com/netbirdio/netbird/tree/main/relay

@allroundtechie commented on GitHub (Sep 11, 2024):

@marcportabellaclotet-mt

@allroundtechie commented on GitHub (Sep 11, 2024): @marcportabellaclotet-mt https://github.com/netbirdio/netbird/tree/main/relay

saavagebueno commented

@ptpu commented on GitHub (Sep 11, 2024):

A short example for traefik which is working fine for me:

docker-compose.yml

relay:
    image: "netbirdio/relay:latest"
    container_name: netbird-relay
    restart: unless-stopped
    env_file:
      - relay.env
      - common.env
    labels:
      traefik.enable: 'true'
      traefik.http.routers.netbird-relay.rule: 'Host("netbird.mydomain.com") && PathPrefix("/relay")'
      traefik.http.routers.netbird-relay.entrypoints: websecure
      traefik.http.routers.netbird-relay.service: netbird-relay-service
      traefik.http.services.netbird-relay-service.loadbalancer.server.port: 33080

relay.env

NB_LOG_LEVEL=info
NB_LISTEN_ADDRESS=:33080
NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443/relay
NB_AUTH_SECRET=secret

management.json

"Relay": {
        "Addresses": ["rels://netbird.mydomain.com:443/relay"],
        "CredentialsTTL": "24h",
        "Secret": "secret"
    },

@ptpu commented on GitHub (Sep 11, 2024): A short example for traefik which is working fine for me: docker-compose.yml ``` relay: image: "netbirdio/relay:latest" container_name: netbird-relay restart: unless-stopped env_file: - relay.env - common.env labels: traefik.enable: 'true' traefik.http.routers.netbird-relay.rule: 'Host("netbird.mydomain.com") && PathPrefix("/relay")' traefik.http.routers.netbird-relay.entrypoints: websecure traefik.http.routers.netbird-relay.service: netbird-relay-service traefik.http.services.netbird-relay-service.loadbalancer.server.port: 33080 ``` relay.env ``` NB_LOG_LEVEL=info NB_LISTEN_ADDRESS=:33080 NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443/relay NB_AUTH_SECRET=secret ``` management.json ``` "Relay": { "Addresses": ["rels://netbird.mydomain.com:443/relay"], "CredentialsTTL": "24h", "Secret": "secret" }, ```

saavagebueno commented

@pugnobellum commented on GitHub (Sep 11, 2024):

Relay compose file

  relay:
    image: netbirdio/relay:latest
    container_name: netbird_relay
    restart: unless-stopped
    environment:
    - NB_LOG_LEVEL=info
    - NB_LISTEN_ADDRESS=:33080
    - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443
    - NB_AUTH_SECRET=secret
    ports:
      - 33080:33080
    networks:
      - proxynet
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

management.json

   "Relay": {
    "Addresses": ["rels://netbird.mydomain.com:443"],
    "CredentialsTTL": "24h",
    "Secret": "secret"
    },

netbird.subdomain.conf

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name netbird.mydomain.com;

    include /config/nginx/ssl3.conf;

    client_max_body_size 128M;
    client_header_timeout 1d;
    client_body_timeout 1d;

    location / {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app netbird_dashboard;
        set $upstream_port 80;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /api {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app netbird_management;
        set $upstream_port 443;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /signalexchange.SignalExchange/ {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;

        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;

        set $upstream_app netbird_signal;
        set $upstream_port 80;
        set $upstream_proto grpc;
        grpc_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /management.ManagementService/ {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;

        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;

        set $upstream_app netbird_management;
        set $upstream_port 443;
        set $upstream_proto grpc;
        grpc_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /relay/ {
        proxy_pass http://netbird_relay:33080/relay;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        
        # Forward headers
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout settings
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
        proxy_connect_timeout 60s;

        # Handle upstream errors
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    }


}

I use SWAG reverse proxy which just bundles nginx and lets encrypt, my config files are above. I'm trying to add the new relay service. When I fire up my docker client/agent I get this error in the logs for it:

UPDATE: the current relay location I have now works.

@pugnobellum commented on GitHub (Sep 11, 2024): Relay compose file ``` relay: image: netbirdio/relay:latest container_name: netbird_relay restart: unless-stopped environment: - NB_LOG_LEVEL=info - NB_LISTEN_ADDRESS=:33080 - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443 - NB_AUTH_SECRET=secret ports: - 33080:33080 networks: - proxynet logging: driver: "json-file" options: max-size: "500m" max-file: "2" ``` management.json ``` "Relay": { "Addresses": ["rels://netbird.mydomain.com:443"], "CredentialsTTL": "24h", "Secret": "secret" }, ``` netbird.subdomain.conf ``` server { listen 443 ssl; listen [::]:443 ssl; server_name netbird.mydomain.com; include /config/nginx/ssl3.conf; client_max_body_size 128M; client_header_timeout 1d; client_body_timeout 1d; location / { include /config/nginx/proxy.conf; include /config/nginx/resolver.conf; set $upstream_app netbird_dashboard; set $upstream_port 80; set $upstream_proto http; proxy_pass $upstream_proto://$upstream_app:$upstream_port; } location /api { include /config/nginx/proxy.conf; include /config/nginx/resolver.conf; set $upstream_app netbird_management; set $upstream_port 443; set $upstream_proto http; proxy_pass $upstream_proto://$upstream_app:$upstream_port; } location /signalexchange.SignalExchange/ { include /config/nginx/proxy.conf; include /config/nginx/resolver.conf; grpc_read_timeout 1d; grpc_send_timeout 1d; grpc_socket_keepalive on; set $upstream_app netbird_signal; set $upstream_port 80; set $upstream_proto grpc; grpc_pass $upstream_proto://$upstream_app:$upstream_port; } location /management.ManagementService/ { include /config/nginx/proxy.conf; include /config/nginx/resolver.conf; grpc_read_timeout 1d; grpc_send_timeout 1d; grpc_socket_keepalive on; set $upstream_app netbird_management; set $upstream_port 443; set $upstream_proto grpc; grpc_pass $upstream_proto://$upstream_app:$upstream_port; } location /relay/ { proxy_pass http://netbird_relay:33080/relay; # WebSocket support proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "Upgrade"; # Forward headers proxy_set_header Host $http_host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Timeout settings proxy_read_timeout 3600s; proxy_send_timeout 3600s; proxy_connect_timeout 60s; # Handle upstream errors proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504; } } ``` I use SWAG reverse proxy which just bundles nginx and lets encrypt, my config files are above. I'm trying to add the new relay service. When I fire up my docker client/agent I get this error in the logs for it: UPDATE: the current relay location I have now works.

saavagebueno commented

@EdouardVanbelle commented on GitHub (Sep 11, 2024):

Good for me using traefik as a proxy (my config is similar to @ptpu's one )

@mlsmaycon according to your sample, I guess I can spawn multiple relay instances for redundancy reasons:

meaning can I have this case:

    "Relay": {
            "Addresses": ["rels://relay-a.mydomain", "rels://relay-b.mydomain"],

and you confirm that I cannot have this case: relay.mydomain resolving to multiple IPs ?

    "Relay": {
            "Addresses": ["rels://relay.mydomain"],

@EdouardVanbelle commented on GitHub (Sep 11, 2024): Good for me using *traefik* as a proxy (my config is similar to @ptpu's one ) @mlsmaycon according to your sample, I guess I can spawn multiple relay instances for redundancy reasons: meaning can I have this case: ```json "Relay": { "Addresses": ["rels://relay-a.mydomain", "rels://relay-b.mydomain"], ``` and you confirm that I cannot have this case: `relay.mydomain` resolving to multiple IPs ? ```json "Relay": { "Addresses": ["rels://relay.mydomain"], ```

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 11, 2024):

Good for me using traefik as a proxy (my config is similar to @ptpu's one )

@mlsmaycon according to your sample, I guess I can spawn multiple relay instances for redundancy reasons:

meaning can I have this case:
    "Relay": {
            "Addresses": ["rels://relay-a.mydomain", "rels://relay-b.mydomain"],       
and you confirm that I cannot have this case: relay.mydomain resolving to multiple IPs ?
    "Relay": {
            "Addresses": ["rels://relay.mydomain"],       

Hey @EdouardVanbelle , you can have both, the first one means that the client will try both endpoints at the same time and use the one that responds first.

The second one implies a single node or a load balancer endpoint, in the first case, that should work fine, but for a LB, the nodes would need to expose their own addresses configured in the expose address configuration and it should either point directly to them or point to the LB with a Host routing rule to ensure that the traffic would go the correct node.

@mlsmaycon commented on GitHub (Sep 11, 2024): > Good for me using *traefik* as a proxy (my config is similar to @ptpu's one ) > > @mlsmaycon according to your sample, I guess I can spawn multiple relay instances for redundancy reasons: > > meaning can I have this case: > > ```json > "Relay": { > "Addresses": ["rels://relay-a.mydomain", "rels://relay-b.mydomain"], > ``` > > and you confirm that I cannot have this case: `relay.mydomain` resolving to multiple IPs ? > > ```json > "Relay": { > "Addresses": ["rels://relay.mydomain"], > ``` > > Hey @EdouardVanbelle , you can have both, the first one means that the client will try both endpoints at the same time and use the one that responds first. The second one implies a single node or a load balancer endpoint, in the first case, that should work fine, but for a LB, the nodes would need to expose their own addresses configured in the expose address configuration and it should either point directly to them or point to the LB with a Host routing rule to ensure that the traffic would go the correct node.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 12, 2024):

@rgdev With a new deployment, it is very likely that Coturn will only be used with mobile clients until we update them.

Excuse my confusion, but since you say that you still use STUN for peer discovery, and at the same time Coturn won’t be used when the mobile apps are updated. Does that mean that the STUN service is baked into the new Relay now (or the management service) ? (Would we be ultimately able to remove Coturn from docker compose and the management.json ?) Thank you very much for this new implementation it sounds cool and production friendly

@Roeda We are studying this option. We have two systems that can hold the stun role, signal and relay, which are both involved in the connection discovery, we will have a decision soon. But the idea of not having coturn in our self-hosted scripts and templates is something that will be applied after 2-3 major (v0.X.0) releases.

@mlsmaycon commented on GitHub (Sep 12, 2024): > > @rgdev With a new deployment, it is very likely that Coturn will only be used with mobile clients until we update them. > > Excuse my confusion, but since you say that you still use STUN for peer discovery, and at the same time Coturn won’t be used when the mobile apps are updated. Does that mean that the STUN service is baked into the new Relay now (or the management service) ? (Would we be ultimately able to remove Coturn from docker compose and the management.json ?) Thank you very much for this new implementation it sounds cool and production friendly @Roeda We are studying this option. We have two systems that can hold the stun role, signal and relay, which are both involved in the connection discovery, we will have a decision soon. But the idea of not having coturn in our self-hosted scripts and templates is something that will be applied after 2-3 major (v0.X.0) releases.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 12, 2024):

For new relay to work is there any new openwrt package released?

We will look into updating the openwrt version.

I am updating netbird package against openwrt snapshot for months, and I have no problem so far, in fact I have built the new version 0.29.0 and is working fine, and open a PR openwrt/packages#24950, for now I just see the error 2024-09-10T15:25:09-03:00 INFO [peer: [ REDACTED ]=] client/internal/peer/worker_relay.go:59: Relay is not supported by remote peer, probably because I'm not selfhosting, and from release notes:

Cloud support for the new relay feature is coming soon*.

But I'm not backporting to openwrt 23.05, one of my targets is supported only on openwrt snapshot.

And to be honest someone open a issue openwrt/packages#24569 (comment) on openwrt repo to backport a new version, I offered my help to the person if he could test it, but I got no response.

Thanks for the contribution @wehagy. I've asked the user again if they have some time to test it out.

@mlsmaycon commented on GitHub (Sep 12, 2024): > > > For new relay to work is there any new openwrt package released? > > > > > > We will look into updating the openwrt version. > > I am updating `netbird` package against `openwrt snapshot` for months, and I have no problem so far, in fact I have built the new version `0.29.0` and is working fine, and open a PR [openwrt/packages#24950](https://github.com/openwrt/packages/pull/24950), for now I just see the error `2024-09-10T15:25:09-03:00 INFO [peer: [ REDACTED ]=] client/internal/peer/worker_relay.go:59: Relay is not supported by remote peer`, probably because I'm not selfhosting, and from release notes: > > > * Cloud support for the new relay feature is coming soon*. > > But I'm not backporting to `openwrt 23.05`, one of my targets is supported only on `openwrt snapshot`. > > And to be honest someone open a issue [openwrt/packages#24569 (comment)](https://github.com/openwrt/packages/issues/24569#issuecomment-2246451384) on `openwrt` repo to backport a new version, I offered my help to the person if he could test it, but I got no response. Thanks for the contribution @wehagy. I've asked the user again if they have some time to test it out.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 12, 2024):

@mlsmaycon Thanks for the explanation. Do you think about optimization on the api side? The api slows down after 200 peers connected to the system. After 500 peers, it slows down a lot. Each request takes more than 1-2 seconds.

In the test measurements I made, these are the response times returned from the api according to the number of peers connected to the system.

20 Peers: 200-300 ms 100 Peers 300-600 ms 200 Peers: 500-1000 ms 500 Peers: 1500-3000 ms

@ismail0234 We are working on some optimization around our database access. It would be helpful if you can share the exact setup you have, VM size and database you are using, and the API calls you are using to measure it.

Another thing that might affect a self-hosted installation is if you have users in the system that got removed from your IDP, that would cause lots of IDP requests to fetch new data to be cached.

@mlsmaycon commented on GitHub (Sep 12, 2024): > @mlsmaycon Thanks for the explanation. Do you think about optimization on the api side? The api slows down after 200 peers connected to the system. After 500 peers, it slows down a lot. Each request takes more than 1-2 seconds. > > In the test measurements I made, these are the response times returned from the api according to the number of peers connected to the system. > > 20 Peers: 200-300 ms 100 Peers 300-600 ms 200 Peers: 500-1000 ms 500 Peers: 1500-3000 ms @ismail0234 We are working on some optimization around our database access. It would be helpful if you can share the exact setup you have, VM size and database you are using, and the API calls you are using to measure it. Another thing that might affect a self-hosted installation is if you have users in the system that got removed from your IDP, that would cause lots of IDP requests to fetch new data to be cached.

saavagebueno commented

@ismail0234 commented on GitHub (Sep 12, 2024):

@mlsmaycon Thanks for the explanation. Do you think about optimization on the api side? The api slows down after 200 peers connected to the system. After 500 peers, it slows down a lot. Each request takes more than 1-2 seconds.
In the test measurements I made, these are the response times returned from the api according to the number of peers connected to the system.
20 Peers: 200-300 ms 100 Peers 300-600 ms 200 Peers: 500-1000 ms 500 Peers: 1500-3000 ms

@ismail0234 We are working on some optimization around our database access. It would be helpful if you can share the exact setup you have, VM size and database you are using, and the API calls you are using to measure it.

Another thing that might affect a self-hosted installation is if you have users in the system that got removed from your IDP, that would cause lots of IDP requests to fetch new data to be cached.

@mlsmaycon

I'm using the standard Netbird installation, I haven't made any extra settings, so I guess sqllite is used by default. Users connect to the network via setup-key. Is there a way to find out the database size?

I also don't know what an IDP is. JWT Group sync and User group propagation feature are also disabled. I do not use these features. Right now, I'm kicking all unconnected peers off the network every hour to keep the api running fast, so my problems are completely fixed. As the number of peers in the network increases, api calls become very slow and some calls return 0 http response.

The api calls I use are as follows;

“List all Peers”
“List all Groups”

Since Netbird does not share any information on the client side to find the id of the user in api calls, I need to match the netbird ip. For this, it is necessary to go through all peers and match the correct ip.

@ismail0234 commented on GitHub (Sep 12, 2024): > > @mlsmaycon Thanks for the explanation. Do you think about optimization on the api side? The api slows down after 200 peers connected to the system. After 500 peers, it slows down a lot. Each request takes more than 1-2 seconds. > > In the test measurements I made, these are the response times returned from the api according to the number of peers connected to the system. > > 20 Peers: 200-300 ms 100 Peers 300-600 ms 200 Peers: 500-1000 ms 500 Peers: 1500-3000 ms > > @ismail0234 We are working on some optimization around our database access. It would be helpful if you can share the exact setup you have, VM size and database you are using, and the API calls you are using to measure it. > > Another thing that might affect a self-hosted installation is if you have users in the system that got removed from your IDP, that would cause lots of IDP requests to fetch new data to be cached. @mlsmaycon I'm using the standard Netbird installation, I haven't made any extra settings, so I guess sqllite is used by default. Users connect to the network via setup-key. Is there a way to find out the database size? I also don't know what an IDP is. JWT Group sync and User group propagation feature are also disabled. I do not use these features. Right now, I'm kicking all unconnected peers off the network every hour to keep the api running fast, so my problems are completely fixed. As the number of peers in the network increases, api calls become very slow and some calls return 0 http response. The api calls I use are as follows; 1. “List all Peers” 2. “List all Groups” Since Netbird does not share any information on the client side to find the id of the user in api calls, I need to match the netbird ip. For this, it is necessary to go through all peers and match the correct ip.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 12, 2024):

@ismail0234 IDP is the identity provider (authentik, Zitadel, Google and others).

NetBird has an integration with the tool of your choice that mainly gets the user name and email and cache so that you can see them as human readable in the dashboard.

Something is really off about your performance. You can try disabling your IDP integration by updating the management.json and setting the following key to "none."

    "IdpManagerConfig": {
        "ManagerType": "zitadel",

to

    "IdpManagerConfig": {
        "ManagerType": "none",

Then you can restart your management server and test. You will see users as IDs during this test.

Also, can you create a ticket for this problem so we can continue there?

@mlsmaycon commented on GitHub (Sep 12, 2024): @ismail0234 IDP is the identity provider (authentik, Zitadel, Google and others). NetBird has an integration with the tool of your choice that mainly gets the user name and email and cache so that you can see them as human readable in the dashboard. Something is really off about your performance. You can try disabling your IDP integration by updating the management.json and setting the following key to "none." ```json "IdpManagerConfig": { "ManagerType": "zitadel", ``` to ```json "IdpManagerConfig": { "ManagerType": "none", ``` Then you can restart your management server and test. You will see users as IDs during this test. Also, can you create a ticket for this problem so we can continue there?

saavagebueno commented

@ismail0234 commented on GitHub (Sep 12, 2024):

@mlsmaycon Where do I create the ticket? Github or slack?

@ismail0234 commented on GitHub (Sep 12, 2024): @mlsmaycon Where do I create the ticket? Github or slack?

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 12, 2024):

@ismail0234 Github, but feel free to reach out on Slack for a faster iteration.

@mlsmaycon commented on GitHub (Sep 12, 2024): @ismail0234 Github, but feel free to reach out on Slack for a faster iteration.

saavagebueno commented

@ndziuba commented on GitHub (Sep 12, 2024):

For People that are using Caddy (based on the zitadel starter script)

Caddyfile:

  :80, netbird.example.com:443 {
          import security_headers
          reverse_proxy /relay* relay:80
          reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000
          reverse_proxy /api/* management:80
          reverse_proxy /management.ManagementService/* h2c://management:80
          reverse_proxy /* dashboard:80
  }

relay.env

  NB_LOG_LEVEL=info
  NB_LISTEN_ADDRESS=:80
  NB_EXPOSED_ADDRESS=rels://netbird.example.com:443
  NB_AUTH_SECRET="secret"

managment.json

  "Relay": {
          "Addresses": ["rels://netbird.example.com:443/relay"],
          "CredentialsTTL": "24h",
          "Secret": "secret"
  },

docker-compose.yml

  #Relay
  relay:
    image: netbirdio/relay:latest
    container_name: relay
    restart: unless-stopped
    env_file:
      - ./relay.env
    networks:
      - netbird #If you use a network
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

@ndziuba commented on GitHub (Sep 12, 2024): For People that are using Caddy (based on the zitadel starter script) Caddyfile: ``` :80, netbird.example.com:443 { import security_headers reverse_proxy /relay* relay:80 reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000 reverse_proxy /api/* management:80 reverse_proxy /management.ManagementService/* h2c://management:80 reverse_proxy /* dashboard:80 } ``` relay.env ``` NB_LOG_LEVEL=info NB_LISTEN_ADDRESS=:80 NB_EXPOSED_ADDRESS=rels://netbird.example.com:443 NB_AUTH_SECRET="secret" ``` managment.json ``` "Relay": { "Addresses": ["rels://netbird.example.com:443/relay"], "CredentialsTTL": "24h", "Secret": "secret" }, ``` docker-compose.yml ``` #Relay relay: image: netbirdio/relay:latest container_name: relay restart: unless-stopped env_file: - ./relay.env networks: - netbird #If you use a network logging: driver: "json-file" options: max-size: "500m" max-file: "2" ```

saavagebueno commented

@pellz0r commented on GitHub (Sep 12, 2024):

I followed the above settings for Caddy (as I've used the Zitadel starter script once upon a time), but when a node with the latest client tries to connect I get the following:

2024-09-12T19:35:10Z DEBG client/internal/connect.go:176: connecting to the Management service netbird.mydomain.se:443
2024-09-12T19:35:10Z DEBG util/net/dialer_nonios.go:52: Dialing tcp netbird.mydomain.se:443
2024-09-12T19:35:10Z DEBG client/internal/connect.go:184: connected to the Management service netbird.mydomain.se:443
2024-09-12T19:35:11Z DEBG util/net/dialer_nonios.go:52: Dialing tcp netbird.mydomain.se:443
2024-09-12T19:35:11Z DEBG signal/client/grpc.go:81: connected to Signal Service: netbird.mydomain.se:443
2024-09-12T19:35:11Z INFO client/internal/connect.go:251: connecting to the Relay service(s): rels://netbird.mydomain.se:443/relay
2024-09-12T19:35:11Z DEBG relay/client/manager.go:93: starting relay client manager with [rels://netbird.mydomain.se:443/relay] relay servers
2024-09-12T19:35:11Z INFO [client_id: sha-WnVmoeH7RuspQpTopd/8RnZhD9vXJTf3J9VTglSeyGk=] relay/client/client.go:141: connecting to relay server: rels://netbird.mydomain.se:443/relay
2024-09-12T19:35:11Z DEBG util/net/dialer_nonios.go:52: Dialing tcp netbird.mydomain.se:443
2024-09-12T19:35:11Z ERRO relay/client/dialer/ws/ws.go:36: failed to dial to Relay server 'wss://netbird.mydomain.se:443/relay': failed to WebSocket dial: expected handshake response status code 101 but got 404
2024-09-12T19:35:11Z WARN relay/client/manager.go:130: Connection attempt failed: failed to connect to rels://netbird.mydomain.se:443/relay: failed to WebSocket dial: expected handshake response status code 101 but got 404
2024-09-12T19:35:11Z ERRO client/internal/connect.go:253: failed to connect to any relay server: all attempts failed

Not really sure what I'm missing

EDIT: Ah, I messed up and didn't pull / restart all the containers. :)

@pellz0r commented on GitHub (Sep 12, 2024): I followed the above settings for Caddy (as I've used the Zitadel starter script once upon a time), but when a node with the latest client tries to connect I get the following: 2024-09-12T19:35:10Z DEBG client/internal/connect.go:176: connecting to the Management service netbird.mydomain.se:443 2024-09-12T19:35:10Z DEBG util/net/dialer_nonios.go:52: Dialing tcp netbird.mydomain.se:443 2024-09-12T19:35:10Z DEBG client/internal/connect.go:184: connected to the Management service netbird.mydomain.se:443 2024-09-12T19:35:11Z DEBG util/net/dialer_nonios.go:52: Dialing tcp netbird.mydomain.se:443 2024-09-12T19:35:11Z DEBG signal/client/grpc.go:81: connected to Signal Service: netbird.mydomain.se:443 2024-09-12T19:35:11Z INFO client/internal/connect.go:251: connecting to the Relay service(s): rels://netbird.mydomain.se:443/relay 2024-09-12T19:35:11Z DEBG relay/client/manager.go:93: starting relay client manager with [rels://netbird.mydomain.se:443/relay] relay servers 2024-09-12T19:35:11Z INFO [client_id: sha-WnVmoeH7RuspQpTopd/8RnZhD9vXJTf3J9VTglSeyGk=] relay/client/client.go:141: connecting to relay server: rels://netbird.mydomain.se:443/relay 2024-09-12T19:35:11Z DEBG util/net/dialer_nonios.go:52: Dialing tcp netbird.mydomain.se:443 2024-09-12T19:35:11Z ERRO relay/client/dialer/ws/ws.go:36: failed to dial to Relay server 'wss://netbird.mydomain.se:443/relay': failed to WebSocket dial: expected handshake response status code 101 but got 404 2024-09-12T19:35:11Z WARN relay/client/manager.go:130: Connection attempt failed: failed to connect to rels://netbird.mydomain.se:443/relay: failed to WebSocket dial: expected handshake response status code 101 but got 404 2024-09-12T19:35:11Z ERRO client/internal/connect.go:253: failed to connect to any relay server: all attempts failed Not really sure what I'm missing EDIT: Ah, I messed up and didn't pull / restart all the containers. :)

saavagebueno commented

@1nerdyguy commented on GitHub (Sep 13, 2024):

To confirm:

With the new relay, I still need to have the Coturn instance for STUN at this time?

But I can deploy out multiple relay instances, update the config file accordingly, and it will use those?

@1nerdyguy commented on GitHub (Sep 13, 2024): To confirm: With the new relay, I still need to have the Coturn instance for STUN at this time? But I can deploy out multiple relay instances, update the config file accordingly, and it will use those?

saavagebueno commented

@tienlq2011 commented on GitHub (Sep 16, 2024):

Could there be a guide to deploying them on Kubernetes? Thank you!

@tienlq2011 commented on GitHub (Sep 16, 2024): Could there be a guide to deploying them on Kubernetes? Thank you!

saavagebueno commented

@1nerdyguy commented on GitHub (Sep 16, 2024):

Could there be a guide to deploying them on Kubernetes? Thank you!

To my understanding, it looks like it's just it's own container. So you'd just spin them up, map the port, and then update the management.json file and rebuild the containers

@1nerdyguy commented on GitHub (Sep 16, 2024): > Could there be a guide to deploying them on Kubernetes? Thank you! To my understanding, it looks like it's just it's own container. So you'd just spin them up, map the port, and then update the management.json file and rebuild the containers

saavagebueno commented

@rgdev commented on GitHub (Sep 16, 2024):

Relay on k8s (behind a ingress-nginx reverse since it's websockets) :

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
  name: netbird-relay-ingress
  namespace: netbird
spec:
  ingressClassName: nginx
  rules:
  - host: netbird.company.com
    http:
      paths:
      - backend:
          service:
            name: netbird-relay
            port:
              number: 80
        path: /relay
        pathType: Prefix
  tls:
  - hosts:
    - netbird.company.com
    secretName: netbird-tls

Service

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: netbird-relay
  name: netbird-relay
  namespace: netbird
spec:
  ports:
  - name: relay
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app.kubernetes.io/name: netbird-relay

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: netbird-relay
  name: netbird-relay
  namespace: netbird
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: netbird-relay
  template:
    metadata:
      labels:
        app.kubernetes.io/component: relay
        app.kubernetes.io/instance: netbird-relay
        app.kubernetes.io/name: netbird-relay
        app.kubernetes.io/part-of: netbird
    spec:
      containers:
      - env:
        - name: NB_LOG_LEVEL
          value: info
        - name: NB_LISTEN_ADDRESS
          value: :80
        - name: NB_AUTH_SECRET
          valueFrom:
            secretKeyRef:
              key: auth_secret
              name: netbird-relay-authkey
        - name: NB_EXPOSED_ADDRESS
          value: rels://netbird.company.com:443
        image: netbirdio/relay:0.29.2
        imagePullPolicy: IfNotPresent
        name: netbird-relay
        ports:
        - containerPort: 80
          name: relay
          protocol: TCP

The deployment references a netbird-relay-authkey Secret you need to provide it with a key of your choice.

@rgdev commented on GitHub (Sep 16, 2024): Relay on k8s (behind a ingress-nginx reverse since it's websockets) : Ingress ``` apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: cert-manager.io/cluster-issuer: letsencrypt nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" name: netbird-relay-ingress namespace: netbird spec: ingressClassName: nginx rules: - host: netbird.company.com http: paths: - backend: service: name: netbird-relay port: number: 80 path: /relay pathType: Prefix tls: - hosts: - netbird.company.com secretName: netbird-tls ``` Service ``` apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/name: netbird-relay name: netbird-relay namespace: netbird spec: ports: - name: relay port: 80 protocol: TCP targetPort: 80 selector: app.kubernetes.io/name: netbird-relay ``` Deployment ``` apiVersion: apps/v1 kind: Deployment metadata: labels: app.kubernetes.io/name: netbird-relay name: netbird-relay namespace: netbird spec: selector: matchLabels: app.kubernetes.io/name: netbird-relay template: metadata: labels: app.kubernetes.io/component: relay app.kubernetes.io/instance: netbird-relay app.kubernetes.io/name: netbird-relay app.kubernetes.io/part-of: netbird spec: containers: - env: - name: NB_LOG_LEVEL value: info - name: NB_LISTEN_ADDRESS value: :80 - name: NB_AUTH_SECRET valueFrom: secretKeyRef: key: auth_secret name: netbird-relay-authkey - name: NB_EXPOSED_ADDRESS value: rels://netbird.company.com:443 image: netbirdio/relay:0.29.2 imagePullPolicy: IfNotPresent name: netbird-relay ports: - containerPort: 80 name: relay protocol: TCP ``` The deployment references a `netbird-relay-authkey` Secret you need to provide it with a key of your choice.

saavagebueno commented

@marcportabellaclotet-mt commented on GitHub (Sep 16, 2024):

Relay performance question...
I was testing netbird speed using direct connection (opening wg ports) and using relay, and it seems that there is a big performance penalty. Anyone have similars results?
Direct connection : 150Mbit speed
Using Relay: 30 Mbit speed.
I haven't a turn setup, so I can not compare.

@marcportabellaclotet-mt commented on GitHub (Sep 16, 2024): Relay performance question... I was testing netbird speed using direct connection (opening wg ports) and using relay, and it seems that there is a big performance penalty. Anyone have similars results? Direct connection : 150Mbit speed Using Relay: 30 Mbit speed. I haven't a turn setup, so I can not compare.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 16, 2024):

Hey @marcportabellaclotet-mt can you check with different MTU configurations for the NetBird interface on both ends of the connection?

Also, can you share which tool you used for the test?

@mlsmaycon commented on GitHub (Sep 16, 2024): Hey @marcportabellaclotet-mt can you check with different MTU configurations for the NetBird interface on both ends of the connection? Also, can you share which tool you used for the test?

saavagebueno commented

@marcportabellaclotet-mt commented on GitHub (Sep 16, 2024):

I am using iperf and speedtest.
MTU is 1500 in both sides.
Relay app is deployed as a lxc container.

@marcportabellaclotet-mt commented on GitHub (Sep 16, 2024): I am using iperf and [speedtest](https://github.com/librespeed/speedtest). MTU is 1500 in both sides. Relay app is deployed as a lxc container.

saavagebueno commented

@1nerdyguy commented on GitHub (Sep 16, 2024):

@marcportabellaclotet-mt
Does the relay have adequate upload/download bandwidth? Since all traffic on a relayed connection flows 'through' it, you're limited by the download/upload of the relay. It may also impact the latency between clients, depending how far the relay is from their point of presence.

@1nerdyguy commented on GitHub (Sep 16, 2024): @marcportabellaclotet-mt Does the relay have adequate upload/download bandwidth? Since all traffic on a relayed connection flows 'through' it, you're limited by the download/upload of the relay. It may also impact the latency between clients, depending how far the relay is from their point of presence.

saavagebueno commented

@marcportabellaclotet-mt commented on GitHub (Sep 16, 2024):

I am testing the relay service in the same network where netbird client is hostes, so there is no BW restriction.
My test setup is:

netbird client deployed at main office network.
relay app deployed at main office network
netbird client installed in my laptop at home.

@marcportabellaclotet-mt commented on GitHub (Sep 16, 2024): I am testing the relay service in the same network where netbird client is hostes, so there is no BW restriction. My test setup is: - netbird client deployed at main office network. - relay app deployed at main office network - netbird client installed in my laptop at home.

saavagebueno commented

@mlsmaycon commented on GitHub (Sep 17, 2024):

@marcportabellaclotet-mt, the wireguard interface created by NetBird has an MTU of 1280, which can influence the performance and concurrent transfer of other peers' connections in the relay.

A good starting test could be to update your peer's MTU size to 1420 for the wt0 or utun100(macOS) interfaces and test again.

@mlsmaycon commented on GitHub (Sep 17, 2024): @marcportabellaclotet-mt, the wireguard interface created by NetBird has an MTU of 1280, which can influence the performance and concurrent transfer of other peers' connections in the relay. A good starting test could be to update your peer's MTU size to 1420 for the wt0 or utun100(macOS) interfaces and test again.

saavagebueno commented

@marcportabellaclotet-mt commented on GitHub (Sep 17, 2024):

Thanks for answering @mlsmaycon . I will try some debug during the weekend.

@marcportabellaclotet-mt commented on GitHub (Sep 17, 2024): Thanks for answering @mlsmaycon . I will try some debug during the weekend.

saavagebueno commented

@SamB-GB commented on GitHub (Sep 17, 2024):

Relay compose file

  relay:
    image: netbirdio/relay:latest
    container_name: netbird_relay
    restart: unless-stopped
    environment:
    - NB_LOG_LEVEL=info
    - NB_LISTEN_ADDRESS=:33080
    - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443
    - NB_AUTH_SECRET=secret
    ports:
      - 33080:33080
    networks:
      - proxynet
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

management.json

   "Relay": {
    "Addresses": ["rels://netbird.mydomain.com:443"],
    "CredentialsTTL": "24h",
    "Secret": "secret"
    },

netbird.subdomain.conf

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name netbird.mydomain.com;

    include /config/nginx/ssl3.conf;

    client_max_body_size 128M;
    client_header_timeout 1d;
    client_body_timeout 1d;

    location / {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app netbird_dashboard;
        set $upstream_port 80;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /api {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app netbird_management;
        set $upstream_port 443;
        set $upstream_proto http;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /signalexchange.SignalExchange/ {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;

        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;

        set $upstream_app netbird_signal;
        set $upstream_port 80;
        set $upstream_proto grpc;
        grpc_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /management.ManagementService/ {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;

        grpc_read_timeout 1d;
        grpc_send_timeout 1d;
        grpc_socket_keepalive on;

        set $upstream_app netbird_management;
        set $upstream_port 443;
        set $upstream_proto grpc;
        grpc_pass $upstream_proto://$upstream_app:$upstream_port;

    }

    location /relay/ {
        proxy_pass http://netbird_relay:33080/relay;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        
        # Forward headers
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Timeout settings
        proxy_read_timeout 3600s;
        proxy_send_timeout 3600s;
        proxy_connect_timeout 60s;

        # Handle upstream errors
        proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    }


}

I use SWAG reverse proxy which just bundles nginx and lets encrypt, my config files are above. I'm trying to add the new relay service. When I fire up my docker client/agent I get this error in the logs for it:

UPDATE: the current relay location I have now works.

Thanks @pugnobellum adding the /relay onto the proxy pass location fixed my issue.

@SamB-GB commented on GitHub (Sep 17, 2024): > Relay compose file > > ``` > relay: > image: netbirdio/relay:latest > container_name: netbird_relay > restart: unless-stopped > environment: > - NB_LOG_LEVEL=info > - NB_LISTEN_ADDRESS=:33080 > - NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443 > - NB_AUTH_SECRET=secret > ports: > - 33080:33080 > networks: > - proxynet > logging: > driver: "json-file" > options: > max-size: "500m" > max-file: "2" > ``` > > management.json > > ``` > "Relay": { > "Addresses": ["rels://netbird.mydomain.com:443"], > "CredentialsTTL": "24h", > "Secret": "secret" > }, > ``` > > netbird.subdomain.conf > > ``` > server { > listen 443 ssl; > listen [::]:443 ssl; > > server_name netbird.mydomain.com; > > include /config/nginx/ssl3.conf; > > client_max_body_size 128M; > client_header_timeout 1d; > client_body_timeout 1d; > > location / { > include /config/nginx/proxy.conf; > include /config/nginx/resolver.conf; > set $upstream_app netbird_dashboard; > set $upstream_port 80; > set $upstream_proto http; > proxy_pass $upstream_proto://$upstream_app:$upstream_port; > > } > > location /api { > include /config/nginx/proxy.conf; > include /config/nginx/resolver.conf; > set $upstream_app netbird_management; > set $upstream_port 443; > set $upstream_proto http; > proxy_pass $upstream_proto://$upstream_app:$upstream_port; > > } > > location /signalexchange.SignalExchange/ { > include /config/nginx/proxy.conf; > include /config/nginx/resolver.conf; > > grpc_read_timeout 1d; > grpc_send_timeout 1d; > grpc_socket_keepalive on; > > set $upstream_app netbird_signal; > set $upstream_port 80; > set $upstream_proto grpc; > grpc_pass $upstream_proto://$upstream_app:$upstream_port; > > } > > location /management.ManagementService/ { > include /config/nginx/proxy.conf; > include /config/nginx/resolver.conf; > > grpc_read_timeout 1d; > grpc_send_timeout 1d; > grpc_socket_keepalive on; > > set $upstream_app netbird_management; > set $upstream_port 443; > set $upstream_proto grpc; > grpc_pass $upstream_proto://$upstream_app:$upstream_port; > > } > > location /relay/ { > proxy_pass http://netbird_relay:33080/relay; > > # WebSocket support > proxy_http_version 1.1; > proxy_set_header Upgrade $http_upgrade; > proxy_set_header Connection "Upgrade"; > > # Forward headers > proxy_set_header Host $http_host; > proxy_set_header X-Real-IP $remote_addr; > proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; > proxy_set_header X-Forwarded-Proto $scheme; > > # Timeout settings > proxy_read_timeout 3600s; > proxy_send_timeout 3600s; > proxy_connect_timeout 60s; > > # Handle upstream errors > proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504; > } > > > } > ``` > > I use SWAG reverse proxy which just bundles nginx and lets encrypt, my config files are above. I'm trying to add the new relay service. When I fire up my docker client/agent I get this error in the logs for it: > > UPDATE: the current relay location I have now works. Thanks @pugnobellum adding the /relay onto the proxy pass location fixed my issue.

saavagebueno commented

@vampywiz17 commented on GitHub (Sep 18, 2024):

@mlsmaycon

It possible to use cloud hosted Netbird (free tier), but self hosted relay?

@vampywiz17 commented on GitHub (Sep 18, 2024): @mlsmaycon It possible to use cloud hosted Netbird (free tier), but self hosted relay?

saavagebueno commented

@drnkknt commented on GitHub (Sep 20, 2024):

@mlsmaycon

hi maycom, i want to confirm if running a relay service alongside Coturn will cause connection issues on the user client or maybe connection between service? Currently, many of my users are still using versions below 29

@drnkknt commented on GitHub (Sep 20, 2024): @mlsmaycon hi maycom, i want to confirm if running a relay service alongside Coturn will cause connection issues on the user client or maybe connection between service? Currently, many of my users are still using versions below 29

saavagebueno commented

@thorstenkramm commented on GitHub (Sep 21, 2024):

Just if someone uses Caddy as reverse proxy, here is my config, that works fine with the relay container.

# Reverse proxy for Netbird UI, Relay, Signal and API
netbird.example.com: {
    # World accessible GRPC endpoint for all access nodes and clients
    handle /management.ManagementService/* {
        reverse_proxy h2c://127.0.0.1:33073
    }
    # World accessible relay endpoint
    handle /relay* {
        reverse_proxy http://127.0.0.1:33080 # Relay
    }

    # Protected management routes
    handle {
        @denied not client_ip <YOUR_ALLOWED_IP_FOR_MGMT>
        abort @denied

        reverse_proxy /api/* http://127.0.0.1:33073 # Management
        reverse_proxy /* http://127.0.0.1:8080 # Dashboard Singepage App
        header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        header Referrer-Policy no-referrer
        header Permissions-Policy "geolocation=(), microphone=()"
    }
    log {
	    output file /var/log/caddy/door1.az.dimedis.net.log
    }
}

In the mangement.json I appended:

"Relay": {
        "Addresses": ["rels://netbird.example.com:443"],
        "CredentialsTTL": "24h",
        "Secret": "xyz"
    }

The part

@denied not client_ip <YOUR_ALLOWED_IP_FOR_MGMT>
abort @denied

is, of course, optional. I wanted to make the Management API and UI accessible for a known IP address only.

@thorstenkramm commented on GitHub (Sep 21, 2024): Just if someone uses Caddy as reverse proxy, here is my config, that works fine with the relay container. ``` # Reverse proxy for Netbird UI, Relay, Signal and API netbird.example.com: { # World accessible GRPC endpoint for all access nodes and clients handle /management.ManagementService/* { reverse_proxy h2c://127.0.0.1:33073 } # World accessible relay endpoint handle /relay* { reverse_proxy http://127.0.0.1:33080 # Relay } # Protected management routes handle { @denied not client_ip <YOUR_ALLOWED_IP_FOR_MGMT> abort @denied reverse_proxy /api/* http://127.0.0.1:33073 # Management reverse_proxy /* http://127.0.0.1:8080 # Dashboard Singepage App header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" header Referrer-Policy no-referrer header Permissions-Policy "geolocation=(), microphone=()" } log { output file /var/log/caddy/door1.az.dimedis.net.log } } ``` In the mangement.json I appended: ``` "Relay": { "Addresses": ["rels://netbird.example.com:443"], "CredentialsTTL": "24h", "Secret": "xyz" } ``` The part ``` @denied not client_ip <YOUR_ALLOWED_IP_FOR_MGMT> abort @denied ``` is, of course, optional. I wanted to make the Management API and UI accessible for a known IP address only.

saavagebueno commented

@drnkknt commented on GitHub (Sep 25, 2024):

thankyou @thorstenkramm

Just if someone uses Caddy as reverse proxy, here is my config, that works fine with the relay container.
# Reverse proxy for Netbird UI, Relay, Signal and API

i can confirm this config is working if you using caddy, for me i prefer using relay instead of http://127.0.0.1 and change rel:// to rels:// in management.json

Caddyfile

:80, netbird.domain.com:443 {
    import security_headers
    # Relay
    handle /relay* {
        reverse_proxy relay:33080 # Relay
    }

    # Signal
    reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000
    # Management
    reverse_proxy /api/* management:80
    reverse_proxy /management.ManagementService/* h2c://management:80
    # Zitadel
    reverse_proxy /zitadel.admin.v1.AdminService/* h2c://zitadel:8080
    reverse_proxy /admin/v1/* h2c://zitadel:8080

docker-compose.yaml

 # Relay
  relay:
    image: netbirdio/relay:latest
    restart: unless-stopped
    networks: [netbird]
    env_file:
      - ./relay.env
    ports:
      - 33080:33080
    logging:
      driver: "json-file"
      options:
        max-size: "500m"
        max-file: "2"

management.json

 "Relay": {
        "Addresses": ["rels://netbird.domain.com:443"],
        "CredentialsTTL": "24h",
        "Secret": "secret"
    }

relay.env

NB_LOG_LEVEL=info
NB_LISTEN_ADDRESS=:33080
NB_EXPOSED_ADDRESS=rels://netbird.domain.com:443
NB_AUTH_SECRET=secret

@drnkknt commented on GitHub (Sep 25, 2024): thankyou @thorstenkramm > Just if someone uses Caddy as reverse proxy, here is my config, that works fine with the relay container. > > ``` > # Reverse proxy for Netbird UI, Relay, Signal and API i can confirm this config is working if you using caddy, for me i prefer using relay instead of http://127.0.0.1 and change rel:// to rels:// in management.json _Caddyfile_ ``` :80, netbird.domain.com:443 { import security_headers # Relay handle /relay* { reverse_proxy relay:33080 # Relay } # Signal reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000 # Management reverse_proxy /api/* management:80 reverse_proxy /management.ManagementService/* h2c://management:80 # Zitadel reverse_proxy /zitadel.admin.v1.AdminService/* h2c://zitadel:8080 reverse_proxy /admin/v1/* h2c://zitadel:8080 ``` _docker-compose.yaml_ ``` # Relay relay: image: netbirdio/relay:latest restart: unless-stopped networks: [netbird] env_file: - ./relay.env ports: - 33080:33080 logging: driver: "json-file" options: max-size: "500m" max-file: "2" ``` _management.json_ ``` "Relay": { "Addresses": ["rels://netbird.domain.com:443"], "CredentialsTTL": "24h", "Secret": "secret" } ``` _relay.env_ ``` NB_LOG_LEVEL=info NB_LISTEN_ADDRESS=:33080 NB_EXPOSED_ADDRESS=rels://netbird.domain.com:443 NB_AUTH_SECRET=secret ```

saavagebueno commented

@daifeilail commented on GitHub (Sep 29, 2024):

I recommend continuing to use COTURN as the relay. COTURN can be further developed as needed to meet NETBIRD's requirements.

The reasons are as follows:

QoS Control Issues:

Merging all requests into a single TCP connection can make Quality of Service (QoS) difficult to manage effectively.
Complexity of Relay Networks:

The complexity of relay networks is influenced by factors such as firewalls and QoS. Proprietary protocols may encounter unusual issues, such as being blocked, in complex network environments.
Mature Telecom-Grade Solutions:

While many relay solutions exist, currently only TURN can be effectively deployed in telecom-grade solutions. Similar to VXLAN and EVPN, which require collaboration among various vendors to implement based on a standard, TURN stands out as the viable option for reliable relay networks.
Development of Low-Level Communication Protocols:

Developing low-level communication protocols requires many years of accumulation. I believe that adding features at the application layer is much more cost-effective than investing in technologies that may fail during the R&D phase.

@daifeilail commented on GitHub (Sep 29, 2024): I recommend continuing to use COTURN as the relay. COTURN can be further developed as needed to meet NETBIRD's requirements. The reasons are as follows: QoS Control Issues: Merging all requests into a single TCP connection can make Quality of Service (QoS) difficult to manage effectively. Complexity of Relay Networks: The complexity of relay networks is influenced by factors such as firewalls and QoS. Proprietary protocols may encounter unusual issues, such as being blocked, in complex network environments. Mature Telecom-Grade Solutions: While many relay solutions exist, currently only TURN can be effectively deployed in telecom-grade solutions. Similar to VXLAN and EVPN, which require collaboration among various vendors to implement based on a standard, TURN stands out as the viable option for reliable relay networks. Development of Low-Level Communication Protocols: Developing low-level communication protocols requires many years of accumulation. I believe that adding features at the application layer is much more cost-effective than investing in technologies that may fail during the R&D phase.

saavagebueno commented

@Spiritreader commented on GitHub (Sep 29, 2024):

What is the benefit of enabling TLS compared to leaving it off?
If I understand this correctly the purpose of the relay is to relay wireguard traffic, which is encrypted already and the relay server endpoints are statically configured via management.json

So other than somebody hijacking your DNS to point to a malicious relay while at the same time having stolen the relay secret, why should I enable TLS?

@Spiritreader commented on GitHub (Sep 29, 2024): What is the benefit of enabling TLS compared to leaving it off? If I understand this correctly the purpose of the relay is to relay wireguard traffic, which is encrypted already and the relay server endpoints are statically configured via `management.json` So other than somebody hijacking your DNS to point to a malicious relay while at the same time having stolen the relay secret, why should I enable TLS?

saavagebueno commented

@PeterWang-dev commented on GitHub (Oct 2, 2024):

@mlsmaycon

It possible to use cloud hosted Netbird (free tier), but self hosted relay?

Same question here. A self-host-able DERP like relay server is very critical for low latency access.

@PeterWang-dev commented on GitHub (Oct 2, 2024): > @mlsmaycon > > It possible to use cloud hosted Netbird (free tier), but self hosted relay? Same question here. A self-host-able DERP like relay server is very critical for low latency access.

saavagebueno commented

@Roeda commented on GitHub (Oct 6, 2024):

What is the benefit of enabling TLS compared to leaving it off? If I understand this correctly the purpose of the relay is to relay wireguard traffic, which is encrypted already and the relay server endpoints are statically configured via management.json

So other than somebody hijacking your DNS to point to a malicious relay while at the same time having stolen the relay secret, why should I enable TLS?

I wonder essentially the same, what a the recommendation to best secure the Relay service. Normally we put the management cluster behind a Reverse Proxy/API gateway + waf & API protection (with support for grpc). but this configuration will increase latency and create problems for Relay trafic.

so what needs to be protected in the relay service, and what is the official recommendation for security layers to add in production ?
@mlsmaycon
any input guys is welcome. thank you very much in advance

@Roeda commented on GitHub (Oct 6, 2024): > What is the benefit of enabling TLS compared to leaving it off? If I understand this correctly the purpose of the relay is to relay wireguard traffic, which is encrypted already and the relay server endpoints are statically configured via `management.json` > > So other than somebody hijacking your DNS to point to a malicious relay while at the same time having stolen the relay secret, why should I enable TLS? I wonder essentially the same, what a the recommendation to best secure the Relay service. Normally we put the management cluster behind a Reverse Proxy/API gateway + waf & API protection (with support for grpc). but this configuration will increase latency and create problems for Relay trafic. so what needs to be protected in the relay service, and what is the official recommendation for security layers to add in production ? @mlsmaycon any input guys is welcome. thank you very much in advance

saavagebueno commented

@thorstenkramm commented on GitHub (Oct 15, 2024):

Problem: Relayed connections not working

Problem

I have two peers, both on Debian 12 Linux with Netbird version 0.30.1
Problem started to appear with version 0.29, but I wouldn't say it started directly after the update to 0.30.

Both peers show

odroid.example.vpn:
  NetBird IP: 100.125.107.139
  Public key: xxx
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://door1.az.example.net:443
  Last connection update: 7 seconds ago
  Last WireGuard handshake: -
  Transfer status (received/sent) 296 B/332 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

But there is no working connection between the peers. No ping. No nothing.
The connection worked flawlessly for weeks with connection type P2P. Suddenly, the P2P stopped functioning.
I'm not aware of changes to the firewall or the routing.

What makes me wonder is the connection status to the management host.

OS: linux/amd64
Daemon version: 0.30.1
CLI version: 0.30.1
Management: Connected to https://door1.az.example.net:443
Signal: Connected to http://door1.az.example.net:10000
Relays: 
  [stun:door1.az.example.net:3478] is Available
  [turn:door1.az.example.net:3478?transport=udp] is Unavailable, reason: allocate: attribute not found
  [rels://door1.az.example.net:443] is Available

I have a couple of other peers. They are all connected via P2P and all works flawlessly.

The management host is an MS Azure VM.

$mgmt: docker exec netbird-management-1 /go/bin/netbird-mgmt --version
netbird-mgmt version 0.30.1

Relay appears to be the latest version, too. (No version option available)

docker logs netbird-relay-1 -f
2024-10-14T06:40:48Z INFO relay/cmd/root.go:149: server will be available on: rels://door1.az.example.net:443
2024-10-14T06:40:48Z INFO relay/cmd/root.go:124: running metrics server: :9090/metrics
2024-10-14T06:40:48Z INFO relay/server/listener/ws/listener.go:39: WS server listening address: :33080
2024-10-14T06:40:50Z INFO [peer_id: sha-xx+DGnBKca4vsfSnTA=] relay/server/relay.go:120: peer connected from: 172.21.0.1:38154
2024-10-14T06:40:50Z INFO [peer_id: sha-xx=] relay/server/relay.go:120: peer connected from: 172.21.0.1:38164
2024-10-14T06:40:50Z INFO [peer_id: sha-xx/eIdZqPbFv/xx=] relay/server/relay.go:120: peer connected from: 172.21.0.1:38180
2024-10-14T06:40:50Z INFO [peer_id: sha-xx/xx=] relay/server/relay.go:120: peer connected from: 172.21.0.1:38194
2024-10-14T06:53:39Z INFO [peer_id: sha-xx/xx+xx=] relay/server/relay.go:120: peer connected from: 172.21.0.1:37250

Caddy reverse proxy

It all runs behind a caddy reverse proxy.

door1.az.example.net: {
    # World accessible GRPC endpoint for all access nodes and clients
    handle /management.ManagementService/* {
        reverse_proxy h2c://127.0.0.1:33073
    }
    # World accessible relay endpoint
    handle /relay* {
        reverse_proxy http://127.0.0.1:33080 # Relay
    }

    # Protected management routes
    handle {
        @denied not client_ip x.x.x.x
        abort @denied

        reverse_proxy /api/* http://127.0.0.1:33073 # Management
        reverse_proxy /* http://127.0.0.1:8080 # Dashboard Singepage App
        header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        header Referrer-Policy no-referrer
        header Permissions-Policy "geolocation=(), microphone=()"
    }
    log {
	    output file /var/log/caddy/door1.az.example.net.log
    }
}

Questions

How to investigate further

why P2P stopped working?
why are relayed connection failing?

Is the status of the management host [turn:door1.az.example.net:3478?transport=udp] is Unavailable, reason: allocate: attribute not found something I should worry about?
Firewall is open. Turn server is listening.

Any help is much appreciated.

@thorstenkramm commented on GitHub (Oct 15, 2024): Problem: Relayed connections not working ## Problem I have two peers, both on Debian 12 Linux with Netbird version 0.30.1 Problem started to appear with version 0.29, but I wouldn't say it started directly after the update to 0.30. Both peers show ``` odroid.example.vpn: NetBird IP: 100.125.107.139 Public key: xxx Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://door1.az.example.net:443 Last connection update: 7 seconds ago Last WireGuard handshake: - Transfer status (received/sent) 296 B/332 B Quantum resistance: false Routes: - Latency: 0s ``` But there is no working connection between the peers. No ping. No nothing. The connection worked flawlessly for weeks with connection type P2P. Suddenly, the P2P stopped functioning. I'm not aware of changes to the firewall or the routing. What makes me wonder is the connection status to the management host. ``` OS: linux/amd64 Daemon version: 0.30.1 CLI version: 0.30.1 Management: Connected to https://door1.az.example.net:443 Signal: Connected to http://door1.az.example.net:10000 Relays: [stun:door1.az.example.net:3478] is Available [turn:door1.az.example.net:3478?transport=udp] is Unavailable, reason: allocate: attribute not found [rels://door1.az.example.net:443] is Available ``` I have a couple of other peers. They are all connected via P2P and all works flawlessly. The management host is an MS Azure VM. ``` $mgmt: docker exec netbird-management-1 /go/bin/netbird-mgmt --version netbird-mgmt version 0.30.1 ``` Relay appears to be the latest version, too. (No `version` option available) ``` docker logs netbird-relay-1 -f 2024-10-14T06:40:48Z INFO relay/cmd/root.go:149: server will be available on: rels://door1.az.example.net:443 2024-10-14T06:40:48Z INFO relay/cmd/root.go:124: running metrics server: :9090/metrics 2024-10-14T06:40:48Z INFO relay/server/listener/ws/listener.go:39: WS server listening address: :33080 2024-10-14T06:40:50Z INFO [peer_id: sha-xx+DGnBKca4vsfSnTA=] relay/server/relay.go:120: peer connected from: 172.21.0.1:38154 2024-10-14T06:40:50Z INFO [peer_id: sha-xx=] relay/server/relay.go:120: peer connected from: 172.21.0.1:38164 2024-10-14T06:40:50Z INFO [peer_id: sha-xx/eIdZqPbFv/xx=] relay/server/relay.go:120: peer connected from: 172.21.0.1:38180 2024-10-14T06:40:50Z INFO [peer_id: sha-xx/xx=] relay/server/relay.go:120: peer connected from: 172.21.0.1:38194 2024-10-14T06:53:39Z INFO [peer_id: sha-xx/xx+xx=] relay/server/relay.go:120: peer connected from: 172.21.0.1:37250 ``` ## Caddy reverse proxy It all runs behind a caddy reverse proxy. ``` door1.az.example.net: { # World accessible GRPC endpoint for all access nodes and clients handle /management.ManagementService/* { reverse_proxy h2c://127.0.0.1:33073 } # World accessible relay endpoint handle /relay* { reverse_proxy http://127.0.0.1:33080 # Relay } # Protected management routes handle { @denied not client_ip x.x.x.x abort @denied reverse_proxy /api/* http://127.0.0.1:33073 # Management reverse_proxy /* http://127.0.0.1:8080 # Dashboard Singepage App header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" header Referrer-Policy no-referrer header Permissions-Policy "geolocation=(), microphone=()" } log { output file /var/log/caddy/door1.az.example.net.log } } ``` ## Questions How to investigate further * why P2P stopped working? * why are relayed connection failing? Is the status of the management host `[turn:door1.az.example.net:3478?transport=udp] is Unavailable, reason: allocate: attribute not found` something I should worry about? Firewall is open. Turn server is listening. Any help is much appreciated.

saavagebueno commented

@marcportabellaclotet-mt commented on GitHub (Oct 16, 2024):

Back to the stun topic, is it planned to remove the stun requirement in the future, to make the deployment simpler? Will relay service be able to manage the p2p discovery by itself?

@marcportabellaclotet-mt commented on GitHub (Oct 16, 2024): Back to the stun topic, is it planned to remove the stun requirement in the future, to make the deployment simpler? Will relay service be able to manage the p2p discovery by itself?

saavagebueno commented

@rudradevpal commented on GitHub (Oct 20, 2024):

@wehagy can you please let me know where can i find Latest openwrt packages

@rudradevpal commented on GitHub (Oct 20, 2024): @wehagy can you please let me know where can i find Latest openwrt packages

saavagebueno commented

@wehagy commented on GitHub (Oct 21, 2024):

@wehagy can you please let me know where can i find Latest openwrt packages

@rudradevpal you can find the most up-to-date netbird packages for openwrt using the snapshot version of openwrt. Probably, but I haven't tested it and I might be wrong, you can download the latest .ipk package of netbird from the openwrt snapshot repository and install it on the stable version of openwrt.

@wehagy commented on GitHub (Oct 21, 2024): > @wehagy can you please let me know where can i find Latest openwrt packages @rudradevpal you can find the most up-to-date `netbird` packages for `openwrt` using the snapshot version of `openwrt`. Probably, but I haven't tested it and I might be wrong, you can download the latest `.ipk` package of netbird from the `openwrt snapshot` repository and install it on the stable version of `openwrt`.

saavagebueno commented

@benniekiss commented on GitHub (Nov 9, 2024):

I'm experiencing issues with proxying the relay service with caddy, but I am able to do so with nginx, and I was wondering if anyone had configuration advice.

The relay service is using self signed certificates, so the Caddyfile looks like this:

...
        handle /relay* {
                reverse_proxy {RELAY_ADDRESS}:{RELAY_PORT} {
                        transport http {
                                tls
                                tls_insecure_skip_verify
                                read_timeout 86400s
                                write_timeout 86400s
                                keepalive_interval 75s
                        }
                }
        }

And I'm getting this consistently in my logs -- both caddy and the relay service

{"level":"error","ts":1731175600.2423942,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"stream error: stream ID 5; CANCEL"}

{"level":"error","ts":1731175600.2424295,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:10000","duration":0.011144689,"request":{"remote_ip":"10.10.10.1","remote_port":"26566","client_ip":"10.10.10.1","proto":"HTTP/2.0","method":"POST","host":"{DOMAIN}:443","uri":"/signalexchange.SignalExchange/ConnectStream","headers":{"User-Agent":["grpc-go/1.64.1"],"Te":["trailers"],"X-Forwarded-For":["10.10.10.1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["{DOMAIN}:443"],"X-Wiretrustee-Peer-Id":["{ID}"],"Accept-Encoding":["gzip"],"Content-Type":["application/grpc"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":""}},"error":"reading: stream error: stream ID 5; CANCEL"}

2024-11-09T13:06:53-05:00 ERRO [peer_id: ] relay/server/peer.go:61: failed to read message: failed to get reader: failed to read frame header: EOF

2024-11-09T13:06:53-05:00 DEBG [peer_id: ] relay/server/relay.go:137: relay connection closed

@benniekiss commented on GitHub (Nov 9, 2024): I'm experiencing issues with proxying the relay service with caddy, but I am able to do so with nginx, and I was wondering if anyone had configuration advice. The relay service is using self signed certificates, so the Caddyfile looks like this: ``` ... handle /relay* { reverse_proxy {RELAY_ADDRESS}:{RELAY_PORT} { transport http { tls tls_insecure_skip_verify read_timeout 86400s write_timeout 86400s keepalive_interval 75s } } } ``` And I'm getting this consistently in my logs -- both caddy and the relay service ``` {"level":"error","ts":1731175600.2423942,"logger":"http.handlers.reverse_proxy","msg":"reading from backend","error":"stream error: stream ID 5; CANCEL"} {"level":"error","ts":1731175600.2424295,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:10000","duration":0.011144689,"request":{"remote_ip":"10.10.10.1","remote_port":"26566","client_ip":"10.10.10.1","proto":"HTTP/2.0","method":"POST","host":"{DOMAIN}:443","uri":"/signalexchange.SignalExchange/ConnectStream","headers":{"User-Agent":["grpc-go/1.64.1"],"Te":["trailers"],"X-Forwarded-For":["10.10.10.1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["{DOMAIN}:443"],"X-Wiretrustee-Peer-Id":["{ID}"],"Accept-Encoding":["gzip"],"Content-Type":["application/grpc"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":""}},"error":"reading: stream error: stream ID 5; CANCEL"} 2024-11-09T13:06:53-05:00 ERRO [peer_id: ] relay/server/peer.go:61: failed to read message: failed to get reader: failed to read frame header: EOF 2024-11-09T13:06:53-05:00 DEBG [peer_id: ] relay/server/relay.go:137: relay connection closed ```

saavagebueno commented

@ghost commented on GitHub (Jan 7, 2025):

A short example for traefik which is working fine for me:

docker-compose.yml

relay:
    image: "netbirdio/relay:latest"
    container_name: netbird-relay
    restart: unless-stopped
    env_file:
      - relay.env
      - common.env
    labels:
      traefik.enable: 'true'
      traefik.http.routers.netbird-relay.rule: 'Host("netbird.mydomain.com") && PathPrefix("/relay")'
      traefik.http.routers.netbird-relay.entrypoints: websecure
      traefik.http.routers.netbird-relay.service: netbird-relay-service
      traefik.http.services.netbird-relay-service.loadbalancer.server.port: 33080

relay.env

NB_LOG_LEVEL=info
NB_LISTEN_ADDRESS=:33080
NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443/relay
NB_AUTH_SECRET=secret

management.json

"Relay": {
        "Addresses": ["rels://netbird.mydomain.com:443/relay"],
        "CredentialsTTL": "24h",
        "Secret": "secret"
    },

Would it be possible to get in touch with you via Discord or something similar? I’m still having issues with Traefik and Netbird and would be very grateful if you could share your complete configuration (regarding the compose file) with me.

@ghost commented on GitHub (Jan 7, 2025): > A short example for traefik which is working fine for me: > > docker-compose.yml > > ``` > relay: > image: "netbirdio/relay:latest" > container_name: netbird-relay > restart: unless-stopped > env_file: > - relay.env > - common.env > labels: > traefik.enable: 'true' > traefik.http.routers.netbird-relay.rule: 'Host("netbird.mydomain.com") && PathPrefix("/relay")' > traefik.http.routers.netbird-relay.entrypoints: websecure > traefik.http.routers.netbird-relay.service: netbird-relay-service > traefik.http.services.netbird-relay-service.loadbalancer.server.port: 33080 > ``` > > relay.env > > ``` > NB_LOG_LEVEL=info > NB_LISTEN_ADDRESS=:33080 > NB_EXPOSED_ADDRESS=rels://netbird.mydomain.com:443/relay > NB_AUTH_SECRET=secret > ``` > > management.json > > ``` > "Relay": { > "Addresses": ["rels://netbird.mydomain.com:443/relay"], > "CredentialsTTL": "24h", > "Secret": "secret" > }, > ``` Would it be possible to get in touch with you via Discord or something similar? I’m still having issues with Traefik and Netbird and would be very grateful if you could share your complete configuration (regarding the compose file) with me.

saavagebueno commented

@Djjvb commented on GitHub (Jan 16, 2025):

It has become a blur for me, since I have no IT background (willing to learn). followed the Jim's Garage video and the official netbird documentation. But:

The script which downloads all the latest stable files has a traefik docker compose file. But this doesn't include any relay lines. So nothing in there when you run the script.
The github traefik docker compose file , does have relay lines. But has no traefik labels, what so ever.
This thread does mention traefik labels of some contributors.

So now I am lost. I can't follow the Jim's garage video and files no more due to the relay. And the official installation seem to not be consistent with the github files. And have had no succes fixing it myself with limited knowledge (and chatgpt has not much input to help). To begin I have 2 questions:

In the video of Jim's garage he has put the internal IP of his docker server in the NETBIRD_TURN_EXTERNAL_IP field. Is that something that still is something that you need to do with the relay? (And do you need a turnserver.conf file then?)
Does the relay container need any traefik labels? And if so, can they be added to the traefik compose file?

@Djjvb commented on GitHub (Jan 16, 2025): It has become a blur for me, since I have no IT background (willing to learn). followed the Jim's Garage video and the official netbird documentation. But: - The script which downloads all the latest stable files has a traefik docker compose file. But this doesn't include any relay lines. So nothing in there when you run the script. - The github traefik docker compose file , does have relay lines. But has no traefik labels, what so ever. - This thread does mention traefik labels of some contributors. So now I am lost. I can't follow the Jim's garage video and files no more due to the relay. And the official installation seem to not be consistent with the github files. And have had no succes fixing it myself with limited knowledge (and chatgpt has not much input to help). To begin I have 2 questions: - In the video of Jim's garage he has put the internal IP of his docker server in the NETBIRD_TURN_EXTERNAL_IP field. Is that something that still is something that you need to do with the relay? (And do you need a turnserver.conf file then?) - Does the relay container need any traefik labels? And if so, can they be added to the traefik compose file?

saavagebueno commented

@alexdellabruna commented on GitHub (Feb 12, 2025):

Hi, I'm trying to setup new relay on a self-hosted installation, but I'm not able to get it work.
The current config doesn't have Coturn integrated.
Netbird status output:

Relays: 
  [rels://netbird.domain.com:443/relay] is Unavailable, reason: relay client not connected

From the client.log I get:

2025-02-12T17:42:10+01:00 INFO management/client/grpc.go:155: connected to the Management Service stream
2025-02-12T17:42:10+01:00 ERRO management/client/grpc.go:262: failed handling an update message received from Management Service: update TURNs: invalid port
2025-02-12T17:42:10+01:00 WARN management/client/grpc.go:170: disconnected from the Management service but will retry silently. Reason: update TURNs: invalid port

This is my actual config:

env:
    NB_LOG_LEVEL: info
    NB_LISTEN_ADDRESS: ":33080"
    NB_EXPOSED_ADDRESS: rels://netbird.domain.com:443/relay

The management points STUN and TURN to rels://netbird.domain.com:443/relay

All the deployment is behind Nginx Ingress Controller

Does the new version still need Coturn? If that's the case, what is the relay current scope?

@alexdellabruna commented on GitHub (Feb 12, 2025): Hi, I'm trying to setup new relay on a self-hosted installation, but I'm not able to get it work. The current config doesn't have Coturn integrated. Netbird status output: ``` Relays: [rels://netbird.domain.com:443/relay] is Unavailable, reason: relay client not connected ``` From the client.log I get: ``` 2025-02-12T17:42:10+01:00 INFO management/client/grpc.go:155: connected to the Management Service stream 2025-02-12T17:42:10+01:00 ERRO management/client/grpc.go:262: failed handling an update message received from Management Service: update TURNs: invalid port 2025-02-12T17:42:10+01:00 WARN management/client/grpc.go:170: disconnected from the Management service but will retry silently. Reason: update TURNs: invalid port ``` This is my actual config: ``` env: NB_LOG_LEVEL: info NB_LISTEN_ADDRESS: ":33080" NB_EXPOSED_ADDRESS: rels://netbird.domain.com:443/relay ``` The management points STUN and TURN to `rels://netbird.domain.com:443/relay` All the deployment is behind Nginx Ingress Controller Does the new version still need Coturn? If that's the case, what is the relay current scope?

saavagebueno commented

@1nerdyguy commented on GitHub (Feb 12, 2025):

Yes, you need Coturn still.

Relay is used by clients who can support it. Android, for example, does not.

@1nerdyguy commented on GitHub (Feb 12, 2025): Yes, you need Coturn still. Relay is used by clients who can support it. Android, for example, does not.

saavagebueno commented

@Roeda commented on GitHub (Feb 23, 2025):

Hello @mlsmaycon ,
any news on the relay feature development progress ?

@Roeda commented on GitHub (Feb 23, 2025): Hello @mlsmaycon , any news on the relay feature development progress ?

saavagebueno commented

@himekifee commented on GitHub (Mar 29, 2025):

Is it possible to force use a relay for a peer? Currently, my local machine to a remote node uses Connection type: P2P but the connection quality is really bad. My relay server has good connectivity to both nodes so I would like to prefer to use relay instead of P2P. Setting one of the nodes to globally only use relay is also acceptable if specifying a single peer connection to use relay is not feasible.

@himekifee commented on GitHub (Mar 29, 2025): Is it possible to force use a relay for a peer? Currently, my local machine to a remote node uses `Connection type: P2P` but the connection quality is really bad. My relay server has good connectivity to both nodes so I would like to prefer to use relay instead of P2P. Setting one of the nodes to globally only use relay is also acceptable if specifying a single peer connection to use relay is not feasible.

saavagebueno commented

@papaj-na-wrotkach commented on GitHub (Apr 18, 2025):

I am wondering if the relay binary could support setting auth secret from a file (the NB_AUTH_SECRET variable). It would be cool if we could use Docker secrets to store the auth secret. Many images on Docker Hub support it by having a _FILE suffixed version of every variable (or most of them).

Authentik does it another way - there are no additional variables. It checks if the variable value starts with env:// or file:// and treats them as URLs to the value. file:// reads the value from specified file and env:// binds the value to the value of another variable. If it does not start with any of them, Authentik treats it as a simple value.

@papaj-na-wrotkach commented on GitHub (Apr 18, 2025): I am wondering if the relay binary could support setting auth secret from a file (the `NB_AUTH_SECRET` variable). It would be cool if we could use Docker secrets to store the auth secret. Many images on Docker Hub support it by having a `_FILE` suffixed version of every variable (or most of them). Authentik does it another way - there are no additional variables. It checks if the variable value starts with `env://` or `file://` and treats them as URLs to the value. `file://` reads the value from specified file and `env://` binds the value to the value of another variable. If it does not start with any of them, Authentik treats it as a simple value.

saavagebueno commented

@iamspido commented on GitHub (Jun 29, 2025):

QUIC Support with Traefik Reverse Proxy - Need Help

Hi everyone,

I'm trying to get QUIC working with NetBird relay behind a Traefik reverse proxy. The relay works perfectly with WebSocket/HTTPS, but QUIC connections are failing.

Current Setup

NetBird relay running in Docker
Traefik v3.4.3 as reverse proxy
HTTPS/TLS working fine via Traefik with Let's Encrypt
WebSocket connections work without issues

Error Messages

netbird-client | failed to resolve UDP address: lookup udp/443/relay: unknown port
netbird-client | [relay: rels://netbird.mydomain.de:443/relay] failed to dial via quic: lookup udp/443/relay: unknown port

What I've Tried

Attempt 1: UDP routing through Traefik

Traefik configuration:

traefik:
  ports:
    - "443:443/udp"
  command:
    - "--entrypoints.websecure-udp.address=:443/udp" # for QUIC Netbird Relay

Relay labels:

relay:
  labels:
    - "traefik.udp.routers.netbird-relay-udp.entrypoints=websecure-udp"
    - "traefik.udp.routers.netbird-relay-udp.service=netbird-relay-udp@docker"
    - "traefik.udp.services.netbird-relay-udp.loadbalancer.server.port=33080"

Attempt 2: Direct UDP port mapping

Relay configuration:

relay:
  ports:
    - "443:33080/udp"

Current Status

UDP port 443 is confirmed open: netstat -ulnp | grep :443 shows docker-proxy listening
Firewall allows UDP 443
NetBird functionality works (can establish SSH connections via WebSocket fallback)
Only QUIC connections are failing

Questions

Has anyone successfully configured QUIC with Traefik for NetBird relay?
Should I use UDP routing through Traefik or direct port mapping?
Is there a specific NB_EXPOSED_ADDRESS format required for QUIC?
Are there any additional Traefik or NetBird configuration parameters I'm missing?

Any help or working configurations would be greatly appreciated!

Environment:

NetBird relay: ghcr.io/netbirdio/relay:0.49.0
Traefik: v3.4.3

@iamspido commented on GitHub (Jun 29, 2025): # QUIC Support with Traefik Reverse Proxy - Need Help Hi everyone, I'm trying to get QUIC working with NetBird relay behind a Traefik reverse proxy. The relay works perfectly with WebSocket/HTTPS, but QUIC connections are failing. ## Current Setup - NetBird relay running in Docker - Traefik v3.4.3 as reverse proxy - HTTPS/TLS working fine via Traefik with Let's Encrypt - WebSocket connections work without issues ## Error Messages ``` netbird-client | failed to resolve UDP address: lookup udp/443/relay: unknown port netbird-client | [relay: rels://netbird.mydomain.de:443/relay] failed to dial via quic: lookup udp/443/relay: unknown port ``` ## What I've Tried ### Attempt 1: UDP routing through Traefik **Traefik configuration:** ```yaml traefik: ports: - "443:443/udp" command: - "--entrypoints.websecure-udp.address=:443/udp" # for QUIC Netbird Relay ``` **Relay labels:** ```yaml relay: labels: - "traefik.udp.routers.netbird-relay-udp.entrypoints=websecure-udp" - "traefik.udp.routers.netbird-relay-udp.service=netbird-relay-udp@docker" - "traefik.udp.services.netbird-relay-udp.loadbalancer.server.port=33080" ``` ### Attempt 2: Direct UDP port mapping **Relay configuration:** ```yaml relay: ports: - "443:33080/udp" ``` ## Current Status - UDP port 443 is confirmed open: `netstat -ulnp | grep :443` shows docker-proxy listening - Firewall allows UDP 443 - NetBird functionality works (can establish SSH connections via WebSocket fallback) - Only QUIC connections are failing ## Questions 1. Has anyone successfully configured QUIC with Traefik for NetBird relay? 2. Should I use UDP routing through Traefik or direct port mapping? 3. Is there a specific `NB_EXPOSED_ADDRESS` format required for QUIC? 4. Are there any additional Traefik or NetBird configuration parameters I'm missing? Any help or working configurations would be greatly appreciated! **Environment:** - NetBird relay: `ghcr.io/netbirdio/relay:0.49.0` - Traefik: `v3.4.3`

saavagebueno commented

@berkant commented on GitHub (Aug 16, 2025):

Is there a way to fall back to the old TURN-based setup? I have just found myself in a very peculiar spot where I have to rely on TURN for connectivity and new Relay system wouldn't work.

@berkant commented on GitHub (Aug 16, 2025): Is there a way to fall back to the old TURN-based setup? I have just found myself in a very peculiar spot where I have to rely on TURN for connectivity and new Relay system wouldn't work.

saavagebueno commented

@mrwsl commented on GitHub (Aug 21, 2025):

Periodically, I have disconnects from netbird peers that I am connected to via SSH. When I check netbird status -dA I can see that the relay is not available from time to time:

Relays: 
  [stun:internal.anon-WnXR0.domain:3478] is Available
  [turn:internal.anon-WnXR0.domain:3478?transport=udp] is Available
  [rel://internal.anon-WnXR0.domain:33080] is Unavailable, reason: relay connection is not established

The logs show:

2025-08-21T09:46:34+02:00 WARN [peer: ujDju1lSJ8xO2jcdVrD4uxI6oPTmYRHM0o7XTQ5DjF0=] client/internal/peer/conn.go:738: failed to close deprecated wg proxy conn: 1 error occurred:
	* remote conn: conn reference mismatch
2025-08-21T09:46:34+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:234: remote peer is available, prepare the relayed connection: sha-Ca7hauHkTK6yN4fqf7vzfIOlk8SHrq+nCv7eRx1XoDs=
2025-08-21T09:46:34+02:00 INFO [peer: kIqCZQ6h0jsEKGQ3qaWxJGnu0aPQyUw8b3h01qWBPFU=] client/internal/peer/conn.go:478: created new wgProxy for relay connection: 127.0.0.1:41722
2025-08-21T09:46:34+02:00 WARN [peer: kIqCZQ6h0jsEKGQ3qaWxJGnu0aPQyUw8b3h01qWBPFU=] client/internal/peer/conn.go:738: failed to close deprecated wg proxy conn: 1 error occurred:
	* remote conn: conn reference mismatch
2025-08-21T09:46:35+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:367: start to Relay read loop exit
2025-08-21T09:46:35+02:00 ERRO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:370: failed to read message from relay server: failed to get reader: failed to read frame header: EOF
2025-08-21T09:46:35+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:594: closing all peer connections
2025-08-21T09:46:35+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:602: waiting for read loop to close
2025-08-21T09:46:35+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:604: relay connection closed
2025-08-21T09:46:35+02:00 ERRO client/iface/wgproxy/udp/proxy.go:219: failed to read from remote conn: unknown, use of closed network connection

After that, the client reconnects to the relay. I was unable to find anything related in other issues so I wonder if someone has a similar problem?

@mrwsl commented on GitHub (Aug 21, 2025): Periodically, I have disconnects from netbird peers that I am connected to via SSH. When I check `netbird status -dA` I can see that the relay is not available from time to time: ``` Relays: [stun:internal.anon-WnXR0.domain:3478] is Available [turn:internal.anon-WnXR0.domain:3478?transport=udp] is Available [rel://internal.anon-WnXR0.domain:33080] is Unavailable, reason: relay connection is not established ``` The logs show: ``` 2025-08-21T09:46:34+02:00 WARN [peer: ujDju1lSJ8xO2jcdVrD4uxI6oPTmYRHM0o7XTQ5DjF0=] client/internal/peer/conn.go:738: failed to close deprecated wg proxy conn: 1 error occurred: * remote conn: conn reference mismatch 2025-08-21T09:46:34+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:234: remote peer is available, prepare the relayed connection: sha-Ca7hauHkTK6yN4fqf7vzfIOlk8SHrq+nCv7eRx1XoDs= 2025-08-21T09:46:34+02:00 INFO [peer: kIqCZQ6h0jsEKGQ3qaWxJGnu0aPQyUw8b3h01qWBPFU=] client/internal/peer/conn.go:478: created new wgProxy for relay connection: 127.0.0.1:41722 2025-08-21T09:46:34+02:00 WARN [peer: kIqCZQ6h0jsEKGQ3qaWxJGnu0aPQyUw8b3h01qWBPFU=] client/internal/peer/conn.go:738: failed to close deprecated wg proxy conn: 1 error occurred: * remote conn: conn reference mismatch 2025-08-21T09:46:35+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:367: start to Relay read loop exit 2025-08-21T09:46:35+02:00 ERRO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:370: failed to read message from relay server: failed to get reader: failed to read frame header: EOF 2025-08-21T09:46:35+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:594: closing all peer connections 2025-08-21T09:46:35+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:602: waiting for read loop to close 2025-08-21T09:46:35+02:00 INFO [relay: rel://internal.tld.com:33080] shared/relay/client/client.go:604: relay connection closed 2025-08-21T09:46:35+02:00 ERRO client/iface/wgproxy/udp/proxy.go:219: failed to read from remote conn: unknown, use of closed network connection ``` After that, the client reconnects to the relay. I was unable to find anything related in other issues so I wonder if someone has a similar problem?

saavagebueno commented

@Sonlis commented on GitHub (Aug 30, 2025):

Exact same problem as @mrwsl, my linux client disconnects and reconnects to relay almost every second due to the following:

2025-08-30T14:30:56+02:00 INFO shared/relay/client/guard.go:82: try to reconnect to Relay server: rel://netbird.bastincloud.com:30080
2025-08-30T14:30:56+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:173: connecting to relay server
2025-08-30T14:30:56+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/dialer/race_dialer.go:66: dialing Relay server via quic
2025-08-30T14:30:56+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/dialer/race_dialer.go:66: dialing Relay server via WS
2025-08-30T14:30:57+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/dialer/race_dialer.go:91: successfully dialed via: WS
2025-08-30T14:30:57+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/dialer/race_dialer.go:77: connection attempt aborted via: quic
2025-08-30T14:30:57+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:195: relay connection established
2025-08-30T14:30:57+02:00 INFO [peer: 4YCGK2pM3/3BT/Si022A2bpyYsXURfPkil675OzR43c=] client/internal/peer/conn.go:470: created new wgProxy for relay connection: 127.0.0.1:49651
2025-08-30T14:30:57+02:00 ERRO [peer: 4YCGK2pM3/3BT/Si022A2bpyYsXURfPkil675OzR43c=] client/internal/peer/wg_watcher.go:53: WireGuard watcher already enabled
2025-08-30T14:30:57+02:00 INFO [peer: 4YCGK2pM3/3BT/Si022A2bpyYsXURfPkil675OzR43c=] client/internal/peer/conn.go:501: start to communicate with peer via relay
 Disconnected
2025-08-30T14:30:58+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:367: start to Relay read loop exit
2025-08-30T14:30:58+02:00 ERRO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:370: failed to read message from relay server: failed to get reader: failed to read frame header: EOF
2025-08-30T14:30:58+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:594: closing all peer connections
2025-08-30T14:30:58+02:00 ERRO client/iface/wgproxy/udp/proxy.go:219: failed to read from remote conn: unknown, use of closed network connection
2025-08-30T14:30:58+02:00 ERRO client/iface/wgproxy/udp/proxy.go:219: failed to read from remote conn: unknown, use of closed network connection

And every second, from the netbird relay container:

2025-08-30T13:13:26Z INFO relay/server/listener/ws/listener.go:100: WS client connected from: 192.168.188.186:58702
2025-08-30T13:13:26Z INFO [peer_id: sha-tTY8Lh0vH5xTFGNy0DTlAhAIFuKSKC28ZaL/HpOU1kw=] relay/server/relay.go:131: peer connected from: 192.168.188.186:58702
2025-08-30T13:13:28Z INFO relay/server/listener/ws/listener.go:100: WS client connected from: 192.168.188.186:58712
2025-08-30T13:13:28Z INFO [peer_id: sha-tTY8Lh0vH5xTFGNy0DTlAhAIFuKSKC28ZaL/HpOU1kw=] relay/server/relay.go:131: peer connected from: 192.168.188.186:58712

Having the relay container behind nginx or directly publishing its port and listening to incoming connections delivers the same results.

EDIT: I realised I had 2 instances of netbird running, and I am honestly not too sure how which got installed.

➜  ~ sudo systemctl | grep netb
  netbird@main.service                                                                       loaded active running   Netbird Client (main)
  system-netbird.slice                                                                       loaded active active    Slice /system/netbird

➜  ~ systemctl status netbird@main
● netbird@main.service - Netbird Client (main)
     Loaded: loaded (/usr/lib/systemd/system/netbird@.service; enabled; preset: disabled)
     Active: active (running) since Sun 2025-08-31 15:48:24 CEST; 2min 33s ago
 Invocation: 3883e0353ed449c7a6586987e3287fcf
       Docs: https://netbird.io/docs
   Main PID: 3852 (netbird)
      Tasks: 17 (limit: 37809)
     Memory: 47.7M (peak: 48M)
        CPU: 1.698s
     CGroup: /system.slice/system-netbird.slice/netbird@main.service
             └─3852 /usr/bin/netbird service run --log-file /var/log/netbird/client-main.log --daemon-addr unix:///var/run/netbird/main.sock
➜  ~ sudo systemctl status netbird
○ netbird.service - NetBird mesh network client
     Loaded: loaded (/etc/systemd/system/netbird.service; enabled; preset: disabled)
     Active: inactive (dead)

Aug 31 15:08:58 workibast systemd[1]: Started NetBird mesh network client.
Aug 31 15:48:59 workibast systemd[1]: Stopping NetBird mesh network client...
Aug 31 15:49:02 workibast systemd[1]: netbird.service: Deactivated successfully.
Aug 31 15:49:02 workibast systemd[1]: Stopped NetBird mesh network client.
Aug 31 15:49:02 workibast systemd[1]: netbird.service: Consumed 1min 1.307s CPU time, 99.3M memory peak.

I disabled and stop netbird@main, and since no more problem with relay.

@Sonlis commented on GitHub (Aug 30, 2025): Exact same problem as @mrwsl, my linux client disconnects and reconnects to relay almost every second due to the following: ``` 2025-08-30T14:30:56+02:00 INFO shared/relay/client/guard.go:82: try to reconnect to Relay server: rel://netbird.bastincloud.com:30080 2025-08-30T14:30:56+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:173: connecting to relay server 2025-08-30T14:30:56+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/dialer/race_dialer.go:66: dialing Relay server via quic 2025-08-30T14:30:56+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/dialer/race_dialer.go:66: dialing Relay server via WS 2025-08-30T14:30:57+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/dialer/race_dialer.go:91: successfully dialed via: WS 2025-08-30T14:30:57+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/dialer/race_dialer.go:77: connection attempt aborted via: quic 2025-08-30T14:30:57+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:195: relay connection established 2025-08-30T14:30:57+02:00 INFO [peer: 4YCGK2pM3/3BT/Si022A2bpyYsXURfPkil675OzR43c=] client/internal/peer/conn.go:470: created new wgProxy for relay connection: 127.0.0.1:49651 2025-08-30T14:30:57+02:00 ERRO [peer: 4YCGK2pM3/3BT/Si022A2bpyYsXURfPkil675OzR43c=] client/internal/peer/wg_watcher.go:53: WireGuard watcher already enabled 2025-08-30T14:30:57+02:00 INFO [peer: 4YCGK2pM3/3BT/Si022A2bpyYsXURfPkil675OzR43c=] client/internal/peer/conn.go:501: start to communicate with peer via relay Disconnected 2025-08-30T14:30:58+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:367: start to Relay read loop exit 2025-08-30T14:30:58+02:00 ERRO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:370: failed to read message from relay server: failed to get reader: failed to read frame header: EOF 2025-08-30T14:30:58+02:00 INFO [relay: rel://netbird.example.com:30080] shared/relay/client/client.go:594: closing all peer connections 2025-08-30T14:30:58+02:00 ERRO client/iface/wgproxy/udp/proxy.go:219: failed to read from remote conn: unknown, use of closed network connection 2025-08-30T14:30:58+02:00 ERRO client/iface/wgproxy/udp/proxy.go:219: failed to read from remote conn: unknown, use of closed network connection ``` And every second, from the netbird relay container: ``` 2025-08-30T13:13:26Z INFO relay/server/listener/ws/listener.go:100: WS client connected from: 192.168.188.186:58702 2025-08-30T13:13:26Z INFO [peer_id: sha-tTY8Lh0vH5xTFGNy0DTlAhAIFuKSKC28ZaL/HpOU1kw=] relay/server/relay.go:131: peer connected from: 192.168.188.186:58702 2025-08-30T13:13:28Z INFO relay/server/listener/ws/listener.go:100: WS client connected from: 192.168.188.186:58712 2025-08-30T13:13:28Z INFO [peer_id: sha-tTY8Lh0vH5xTFGNy0DTlAhAIFuKSKC28ZaL/HpOU1kw=] relay/server/relay.go:131: peer connected from: 192.168.188.186:58712 ``` Having the relay container behind nginx or directly publishing its port and listening to incoming connections delivers the same results. EDIT: I realised I had 2 instances of netbird running, and I am honestly not too sure how which got installed. ``` ➜ ~ sudo systemctl | grep netb netbird@main.service loaded active running Netbird Client (main) system-netbird.slice loaded active active Slice /system/netbird ➜ ~ systemctl status netbird@main ● netbird@main.service - Netbird Client (main) Loaded: loaded (/usr/lib/systemd/system/netbird@.service; enabled; preset: disabled) Active: active (running) since Sun 2025-08-31 15:48:24 CEST; 2min 33s ago Invocation: 3883e0353ed449c7a6586987e3287fcf Docs: https://netbird.io/docs Main PID: 3852 (netbird) Tasks: 17 (limit: 37809) Memory: 47.7M (peak: 48M) CPU: 1.698s CGroup: /system.slice/system-netbird.slice/netbird@main.service └─3852 /usr/bin/netbird service run --log-file /var/log/netbird/client-main.log --daemon-addr unix:///var/run/netbird/main.sock ➜ ~ sudo systemctl status netbird ○ netbird.service - NetBird mesh network client Loaded: loaded (/etc/systemd/system/netbird.service; enabled; preset: disabled) Active: inactive (dead) Aug 31 15:08:58 workibast systemd[1]: Started NetBird mesh network client. Aug 31 15:48:59 workibast systemd[1]: Stopping NetBird mesh network client... Aug 31 15:49:02 workibast systemd[1]: netbird.service: Deactivated successfully. Aug 31 15:49:02 workibast systemd[1]: Stopped NetBird mesh network client. Aug 31 15:49:02 workibast systemd[1]: netbird.service: Consumed 1min 1.307s CPU time, 99.3M memory peak. ``` I disabled and stop netbird@main, and since no more problem with relay.

saavagebueno commented

@mrwsl commented on GitHub (Aug 31, 2025):

@Sonlis Thanks for sharing your solution. It also fixed my constant reconnecting.

@mrwsl commented on GitHub (Aug 31, 2025): @Sonlis Thanks for sharing your solution. It also fixed my constant reconnecting.

saavagebueno commented

@linkpad commented on GitHub (Sep 5, 2025):

Same as @Sonlis I had 2 instances of netbird running and got the disconnects to relay really often.
I disabled and stopped netbird@main service and the issue is gone.

I installed netbird through the install script curl -fsSL https://pkgs.netbird.io/install.sh | sh on Manjaro (arch linux).

Another issue I had (probably not related to the relay), is DNS resolution inside docker container was only working for peers configured in netbird. Forcing docker to read resolv.conf live fixed the issue :
Stop Docker: sudo systemctl stop docker
Point Docker to the live config: ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
Start Docker again: sudo systemctl start docker

@linkpad commented on GitHub (Sep 5, 2025): Same as @Sonlis I had 2 instances of netbird running and got the disconnects to relay really often. I disabled and stopped netbird@main service and the issue is gone. I installed netbird through the install script `curl -fsSL https://pkgs.netbird.io/install.sh | sh` on Manjaro (arch linux). Another issue I had (probably not related to the relay), is DNS resolution inside docker container was only working for peers configured in netbird. Forcing docker to read resolv.conf live fixed the issue : Stop Docker: `sudo systemctl stop docker` Point Docker to the live config: `ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf` Start Docker again: `sudo systemctl start docker`

saavagebueno commented

@IvaskevychYuriy commented on GitHub (Sep 25, 2025):

With new ability to configure MTU on the client (I'm running 0.56.1 & also tried 0.58.1 on Ubuntu Server 24.04) I did increased MTU - currently to 1340 for example.
But now clients do not even attempt to connect via QUIC - Websockets only. Message from logs:
"MTU 1340 exceeds default (1280), forcing WebSocket transport to avoid DATAGRAM frame size issues" - coming from https://github.com/netbirdio/netbird/blob/main/shared/relay/client/client.go#L302

Why is there a hard restriction on 1280 MTU for QUIC? I usually see the exact opposite - for example from Tailscale where people set MTU even to 1350 to get some sites working with QUIC over VPN: https://github.com/tailscale/tailscale/issues/2633

@IvaskevychYuriy commented on GitHub (Sep 25, 2025): With new ability to configure MTU on the client (I'm running 0.56.1 & also tried 0.58.1 on Ubuntu Server 24.04) I did increased MTU - currently to 1340 for example. But now clients do not even attempt to connect via QUIC - Websockets only. Message from logs: `"MTU 1340 exceeds default (1280), forcing WebSocket transport to avoid DATAGRAM frame size issues"` - coming from https://github.com/netbirdio/netbird/blob/main/shared/relay/client/client.go#L302 Why is there a hard restriction on 1280 MTU for QUIC? I usually see the exact opposite - for example from Tailscale where people set MTU even to 1350 to get some sites working with QUIC over VPN: https://github.com/tailscale/tailscale/issues/2633

saavagebueno commented

@fede843 commented on GitHub (Oct 1, 2025):

QUIC Support with Traefik Reverse Proxy - Need Help

Hi everyone,

I'm trying to get QUIC working with NetBird relay behind a Traefik reverse proxy. The relay works perfectly with WebSocket/HTTPS, but QUIC connections are failing.

Current Setup

NetBird relay running in Docker

Traefik v3.4.3 as reverse proxy

HTTPS/TLS working fine via Traefik with Let's Encrypt

WebSocket connections work without issues

Error Messages
netbird-client | failed to resolve UDP address: lookup udp/443/relay: unknown port
netbird-client | [relay: rels://netbird.mydomain.de:443/relay] failed to dial via quic: lookup udp/443/relay: unknown port
...

Hi @iamspido have you find a way around this? I am in the same situation here.

@fede843 commented on GitHub (Oct 1, 2025): > # QUIC Support with Traefik Reverse Proxy - Need Help > Hi everyone, > > I'm trying to get QUIC working with NetBird relay behind a Traefik reverse proxy. The relay works perfectly with WebSocket/HTTPS, but QUIC connections are failing. > > ## Current Setup > * NetBird relay running in Docker > * Traefik v3.4.3 as reverse proxy > * HTTPS/TLS working fine via Traefik with Let's Encrypt > * WebSocket connections work without issues > - > ## Error Messages > ``` > netbird-client | failed to resolve UDP address: lookup udp/443/relay: unknown port > netbird-client | [relay: rels://netbird.mydomain.de:443/relay] failed to dial via quic: lookup udp/443/relay: unknown port > ``` > ... Hi @iamspido have you find a way around this? I am in the same situation here.

saavagebueno commented

@iamspido commented on GitHub (Oct 1, 2025):

QUIC Support with Traefik Reverse Proxy - Need Help

Hi everyone,
I'm trying to get QUIC working with NetBird relay behind a Traefik reverse proxy. The relay works perfectly with WebSocket/HTTPS, but QUIC connections are failing.

Current Setup

NetBird relay running in Docker

Traefik v3.4.3 as reverse proxy

HTTPS/TLS working fine via Traefik with Let's Encrypt

WebSocket connections work without issues

Error Messages
netbird-client | failed to resolve UDP address: lookup udp/443/relay: unknown port
netbird-client | [relay: rels://netbird.mydomain.de:443/relay] failed to dial via quic: lookup udp/443/relay: unknown port
...
Hi @iamspido have you find a way around this? I am in the same situation here.

Hey @fede843, unfortunately, I haven’t found a solution or workaround for this yet.

@iamspido commented on GitHub (Oct 1, 2025): > > # QUIC Support with Traefik Reverse Proxy - Need Help > > Hi everyone, > > I'm trying to get QUIC working with NetBird relay behind a Traefik reverse proxy. The relay works perfectly with WebSocket/HTTPS, but QUIC connections are failing. > > ## Current Setup > > > > * NetBird relay running in Docker > > * Traefik v3.4.3 as reverse proxy > > * HTTPS/TLS working fine via Traefik with Let's Encrypt > > * WebSocket connections work without issues > > > > > > > > > > ## Error Messages > > ``` > > netbird-client | failed to resolve UDP address: lookup udp/443/relay: unknown port > > netbird-client | [relay: rels://netbird.mydomain.de:443/relay] failed to dial via quic: lookup udp/443/relay: unknown port > > ``` > > > > > > > > > > > > > > > > > > > > > > > > ... > > Hi [@iamspido](https://github.com/iamspido) have you find a way around this? I am in the same situation here. Hey @fede843, unfortunately, I haven’t found a solution or workaround for this yet.

saavagebueno commented

@fede843 commented on GitHub (Oct 1, 2025):

Hi @lixmal @iamspido
yeap, I've working on this the whole morning. QUIC wants to terminate the TLS by itself.
I was able to set it up without the proxy, just using straight let's encrypt certs.
I guess you can still try to split WS behind the proxy, which it works just fine, and the QUIC udp straight to the container. But it is making things complicated.
For the moment I decided to leave an extra port open for this. We'll see how it goes.
Thanks!

@fede843 commented on GitHub (Oct 1, 2025): Hi @lixmal @iamspido yeap, I've working on this the whole morning. QUIC wants to terminate the TLS by itself. I was able to set it up without the proxy, just using straight let's encrypt certs. I guess you can still try to split WS behind the proxy, which it works just fine, and the QUIC udp straight to the container. But it is making things complicated. For the moment I decided to leave an extra port open for this. We'll see how it goes. Thanks!

saavagebueno commented

@lixmal commented on GitHub (Oct 1, 2025):

@IvaskevychYuriy

With new ability to configure MTU on the client (I'm running 0.56.1 & also tried 0.58.1 on Ubuntu Server 24.04) I did increased MTU - currently to 1340 for example. But now clients do not even attempt to connect via QUIC - Websockets only. Message from logs: "MTU 1340 exceeds default (1280), forcing WebSocket transport to avoid DATAGRAM frame size issues" - coming from https://github.com/netbirdio/netbird/blob/main/shared/relay/client/client.go#L302

Why is there a hard restriction on 1280 MTU for QUIC? I usually see the exact opposite - for example from Tailscale where people set MTU even to 1350 to get some sites working with QUIC over VPN: tailscale/tailscale#2633

Your example is for QUIC running via the overlay network. Increasing wireguard interface MTU there makes sense to allow QUIC datagrams to carry whatever other protocol is encapsulated.

However, in our case QUIC is used for the underlay, it encapsulates wireguard packets. So increasing the wireguard MTU will make wireguard packets bigger, which means they might exceed QUIC capacity (not configurable at the moment) and/or physical MTU.

That's how the layers look in an example if you'd use QUIC in both overlay and underlay (from inner to outer layers):

HTTPS -> QUIC -> UDP -> IP -> Wireguard [MTU here]-> QUIC -> UDP -> IP [MTU here again]-> ...

@fede843 @iamspido
The URL looks incorrect; there shouldn't even be a path. Please note that NetBird uses plain QUIC (with custom next proto), not http/3 (see layer example above)!
That means you cannot use an http proxy, you'll need a L4 proxy or expose the port directly.

Edit: had to resubmit the comment, had an issue

@lixmal commented on GitHub (Oct 1, 2025): @IvaskevychYuriy > With new ability to configure MTU on the client (I'm running 0.56.1 & also tried 0.58.1 on Ubuntu Server 24.04) I did increased MTU - currently to 1340 for example. But now clients do not even attempt to connect via QUIC - Websockets only. Message from logs: `"MTU 1340 exceeds default (1280), forcing WebSocket transport to avoid DATAGRAM frame size issues"` - coming from https://github.com/netbirdio/netbird/blob/main/shared/relay/client/client.go#L302 > > Why is there a hard restriction on 1280 MTU for QUIC? I usually see the exact opposite - for example from Tailscale where people set MTU even to 1350 to get some sites working with QUIC over VPN: [tailscale/tailscale#2633](https://github.com/tailscale/tailscale/issues/2633) Your example is for QUIC running via the overlay network. Increasing wireguard interface MTU there makes sense to allow QUIC datagrams to carry whatever other protocol is encapsulated. However, in our case QUIC is used for the underlay, it encapsulates wireguard packets. So increasing the wireguard MTU will make wireguard packets *bigger*, which means they might exceed QUIC capacity (not configurable at the moment) and/or physical MTU. That's how the layers look in an example if you'd use QUIC in both overlay and underlay (from inner to outer layers): HTTPS -> QUIC -> UDP -> IP -> Wireguard [MTU here]-> QUIC -> UDP -> IP [MTU here again]-> ... @fede843 @iamspido The URL looks incorrect; there shouldn't even be a path. Please note that NetBird uses plain QUIC (with custom next proto), **not** http/3 (see layer example above)! That means you cannot use an http proxy, you'll need a L4 proxy or expose the port directly. Edit: had to resubmit the comment, had an issue

saavagebueno commented

@IvaskevychYuriy commented on GitHub (Oct 1, 2025):

@lixmal thanks for the details! So basically, with netbird up --mtu <X> I'm just setting cap on "Wireguard [MTU here]" part. Then the question would be: can this limit (1280) be also configurable?

The use case is that an overlay network with just 1280 MTU can have some limitations (as seen from linked Tailscale issue). But when I'm in environment where I control the client and I know that my interface is capable of 1500 MTU then when I want to use QUIC relay I can expect the inner (Netbird overlay) network to carry 1500 - QUIC overhead, so low 1400 for netbird up --mtu 14XX.

Does this make sense?

@IvaskevychYuriy commented on GitHub (Oct 1, 2025): @lixmal thanks for the details! So basically, with `netbird up --mtu <X>` I'm just setting cap on "Wireguard [MTU here]" part. Then the question would be: can this limit (`1280`) be also configurable? The use case is that an overlay network with just 1280 MTU can have some limitations (as seen from linked Tailscale issue). But when I'm in environment where I control the client and I know that my interface is capable of 1500 MTU then when I want to use QUIC relay I can expect the inner (Netbird overlay) network to carry `1500 - QUIC overhead`, so low 1400 for `netbird up --mtu 14XX`. Does this make sense?

saavagebueno commented

@vespersio commented on GitHub (Oct 24, 2025):

I ran into the same issue and found that the QUIC relay requires a valid TLS certificate and key to work properly. After configuring the relay with proper TLS settings, the connection started working. Here is the working docker-compose snippet for the relay:

---
  relay:
    image: netbirdio/relay:latest
    container_name: netbird-relay
    restart: unless-stopped
    volumes:
      - ./config/etc/letsencrypt:/etc/letsencrypt:ro
    environment:
      - NB_TLS_CERT_FILE=/etc/letsencrypt/live/vpn.example.com/fullchain.pem
      - NB_TLS_KEY_FILE=/etc/letsencrypt/live/vpn.example.com/privkey.pem
      - NB_LOG_LEVEL=error
      - NB_LISTEN_ADDRESS=:33080
      - NB_METRICS_PORT=9091
      - NB_HEALTH_LISTEN_ADDRESS=:9001
      - NB_EXPOSED_ADDRESS=rels://vpn.example.com
      - NB_AUTH_SECRET=*****
    ports:
      - 172.17.0.1:33080:33080
      - 443:33080/udp

Important note:
To use QUIC, the relay must be accessible via a hostname with valid TLS certs. Using the relay’s FQDN works for me when setting it in management.json like this: rels://vpn.example.com:443
After applying this configuration, the relay started working correctly with QUIC.

@vespersio commented on GitHub (Oct 24, 2025): I ran into the same issue and found that the QUIC relay requires a valid TLS certificate and key to work properly. After configuring the relay with proper TLS settings, the connection started working. Here is the working docker-compose snippet for the relay: ```yaml --- relay: image: netbirdio/relay:latest container_name: netbird-relay restart: unless-stopped volumes: - ./config/etc/letsencrypt:/etc/letsencrypt:ro environment: - NB_TLS_CERT_FILE=/etc/letsencrypt/live/vpn.example.com/fullchain.pem - NB_TLS_KEY_FILE=/etc/letsencrypt/live/vpn.example.com/privkey.pem - NB_LOG_LEVEL=error - NB_LISTEN_ADDRESS=:33080 - NB_METRICS_PORT=9091 - NB_HEALTH_LISTEN_ADDRESS=:9001 - NB_EXPOSED_ADDRESS=rels://vpn.example.com - NB_AUTH_SECRET=***** ports: - 172.17.0.1:33080:33080 - 443:33080/udp ``` Important note: To use QUIC, the relay must be accessible via a hostname with valid TLS certs. Using the relay’s FQDN works for me when setting it in management.json like this: `rels://vpn.example.com:443` After applying this configuration, the relay started working correctly with QUIC.

saavagebueno commented

@devopskupryk commented on GitHub (Nov 10, 2025):

Hi,

Is it possible to specify multiple relays?

I've specified two relays in my config:

      "Relay": {
        "Addresses": [
          "rels://netbird-relay-1.example.tech:443/relay",
          "rels://netbird-relay-2.example.tech:443/relay"
        ],
        "CredentialsTTL": "24h",
        "Secret": "{{ .RELAY_PASSWORD }}"
      },

Вut netbird status -d shows only the first one:

Relays: 
  [rels://netbird-relay-1.example.tech:443/relay] is Available

@devopskupryk commented on GitHub (Nov 10, 2025): Hi, Is it possible to specify multiple relays? I've specified two relays in my config: ``` "Relay": { "Addresses": [ "rels://netbird-relay-1.example.tech:443/relay", "rels://netbird-relay-2.example.tech:443/relay" ], "CredentialsTTL": "24h", "Secret": "{{ .RELAY_PASSWORD }}" }, ``` Вut netbird status -d shows only the first one: ``` Relays: [rels://netbird-relay-1.example.tech:443/relay] is Available ```

saavagebueno commented

@fede843 commented on GitHub (Nov 10, 2025):

Hi,

Is it possible to specify multiple relays?

I've specified two relays in my config:

      "Relay": {
        "Addresses": [
          "rels://netbird-relay-1.example.tech:443/relay",
          "rels://netbird-relay-2.example.tech:443/relay"
        ],
        "CredentialsTTL": "24h",
        "Secret": "{{ .RELAY_PASSWORD }}"
      },

Вut netbird status -d shows only the first one:

Relays: 
  [rels://netbird-relay-1.example.tech:443/relay] is Available

I think based on latency that is the one the client picks up. From the pool chooses the "closest" one.

@fede843 commented on GitHub (Nov 10, 2025): > Hi, > > Is it possible to specify multiple relays? > > I've specified two relays in my config: > > ``` > "Relay": { > "Addresses": [ > "rels://netbird-relay-1.example.tech:443/relay", > "rels://netbird-relay-2.example.tech:443/relay" > ], > "CredentialsTTL": "24h", > "Secret": "{{ .RELAY_PASSWORD }}" > }, > ``` > > Вut netbird status -d shows only the first one: > > ``` > Relays: > [rels://netbird-relay-1.example.tech:443/relay] is Available > ``` I think based on latency that is the one the client picks up. From the pool chooses the "closest" one.

saavagebueno commented

@devopskupryk commented on GitHub (Nov 11, 2025):

I think based on latency that is the one the client picks up. From the pool chooses the "closest" one.

No. I've checked with peer which is near tech second relay. It also sees only the first relay.

One more question.
Do I have to install coturn (stun and turn) if I going to use the new relay?

@devopskupryk commented on GitHub (Nov 11, 2025): > I think based on latency that is the one the client picks up. From the pool chooses the "closest" one. No. I've checked with peer which is near tech second relay. It also sees only the first relay. One more question. Do I have to install coturn (stun and turn) if I going to use the new relay?

saavagebueno commented

@fede843 commented on GitHub (Nov 11, 2025):

@devopskupryk about the relay not sure how can I help. Don't know of any CLI command to debug that.

About Coturn, STUN is needed to determine the public IP, that I understand is a mandatory component. TURN we opted to not deploy it. We are using only the new relay system. There might be some nasty LAN/Firewall scenarios where a peer cannot connect, and in those you might need TURN, but we have not seen it, so decided to kept it simple.
I think the plan is eventually to keep only the new relay around.

@fede843 commented on GitHub (Nov 11, 2025): @devopskupryk about the relay not sure how can I help. Don't know of any CLI command to debug that. About Coturn, STUN is needed to determine the public IP, that I understand is a mandatory component. TURN we opted to not deploy it. We are using only the new relay system. There might be some nasty LAN/Firewall scenarios where a peer cannot connect, and in those you might need TURN, but we have not seen it, so decided to kept it simple. I think the plan is eventually to keep only the new relay around.

saavagebueno commented