SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-05-13 19:52:13 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Clients fail to connect on self hosted server - possibly signal certificate issue following #4644? #2443

New Issue
Open
opened 2025-11-20 07:09:54 -05:00 by saavagebueno · 4 comments
saavagebueno commented 2025-11-20 07:09:54 -05:00
Owner
Copy Link

Originally created by @fishboy25uk on GitHub (Nov 6, 2025).

Attempted to deploy a new server on a VPS this morning (version 0.59.12, Azure as IdP and Let's Encrypt certs which I've obtained on host using certbot myself). Although I can authenticate ok and clients get registered, clients then fail to connect to the network:

Client log:
2025-11-06T12:26:50Z INFO ./caller_not_available:0: 2025/11/06 12:26:50 WARNING: [core] [Channel #116 SubChannel #117]grpc: addrConn.createTransport failed to connect to {Addr: "<SERVER IP>:33073", ServerName: "vpn.<SERVER DOMAIN>:33073", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: nbnet.NewDialer().DialContext: dial tcp <SERVER IP>:33073: connectex: No connection could be made because the target machine actively refused it."

I've modified the docker-compose.yml to change volumes from netbird-letsencrypt to /etc/netbird for dashboard, signal, and management containers. Otherwise it's the standard docker-compose as generated, following the Netbird self hosted guide for Azure IdP

Certificates appear to work correctly for dashboard at least, as I can successfully log in and I can see the "offline" client, but certs are not working with signal container. Initially signal logs showing it couldn't find the certs due to certs on docker host being symbolically linked, but even if I address this, client connection still fails. This is the output from the signal container:

2025-11-06T09:42:39Z INFO signal/cmd/run.go:236: setting up TLS with custom certificates.
2025-11-06T09:42:39Z INFO signal/cmd/run.go:150: running HTTPS server with WebSocket proxy: [::]:443
2025-11-06T09:42:39Z INFO signal/cmd/run.go:161: running gRPC backward compatibility server: [::]:10000
2025-11-06T09:42:39Z INFO signal/cmd/run.go:164: signal server version 0.59.12
2025-11-06T09:42:39Z INFO signal/cmd/run.go:165: started Signal Service
2025-11-06T09:42:39Z INFO signal/cmd/run.go:111: running metrics server: :9090/metrics

Using exactly the same setup/config, works fine using 0.59.8 so I believe this is an issue introduced by "[signal] Fix HTTP/WebSocket proxy not using custom certificates #4644" (https://github.com/netbirdio/netbird/pull/4644)

Originally created by @fishboy25uk on GitHub (Nov 6, 2025). Attempted to deploy a new server on a VPS this morning (version 0.59.12, Azure as IdP and Let's Encrypt certs which I've obtained on host using certbot myself). Although I can authenticate ok and clients get registered, clients then fail to connect to the network: Client log: `2025-11-06T12:26:50Z INFO ./caller_not_available:0: 2025/11/06 12:26:50 WARNING: [core] [Channel #116 SubChannel #117]grpc: addrConn.createTransport failed to connect to {Addr: "<SERVER IP>:33073", ServerName: "vpn.<SERVER DOMAIN>:33073", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: nbnet.NewDialer().DialContext: dial tcp <SERVER IP>:33073: connectex: No connection could be made because the target machine actively refused it."` I've modified the docker-compose.yml to change volumes from netbird-letsencrypt to /etc/netbird for dashboard, signal, and management containers. Otherwise it's the standard docker-compose as generated, following the Netbird self hosted guide for Azure IdP Certificates appear to work correctly for dashboard at least, as I can successfully log in and I can see the "offline" client, but certs are not working with signal container. Initially signal logs showing it couldn't find the certs due to certs on docker host being symbolically linked, but even if I address this, client connection still fails. This is the output from the signal container: ``` 2025-11-06T09:42:39Z INFO signal/cmd/run.go:236: setting up TLS with custom certificates. 2025-11-06T09:42:39Z INFO signal/cmd/run.go:150: running HTTPS server with WebSocket proxy: [::]:443 2025-11-06T09:42:39Z INFO signal/cmd/run.go:161: running gRPC backward compatibility server: [::]:10000 2025-11-06T09:42:39Z INFO signal/cmd/run.go:164: signal server version 0.59.12 2025-11-06T09:42:39Z INFO signal/cmd/run.go:165: started Signal Service 2025-11-06T09:42:39Z INFO signal/cmd/run.go:111: running metrics server: :9090/metrics ``` Using exactly the same setup/config, works fine using 0.59.8 so I believe this is an issue introduced by "[signal] Fix HTTP/WebSocket proxy not using custom certificates #4644" (https://github.com/netbirdio/netbird/pull/4644)
saavagebueno added the triage-needed label 2025-11-20 07:09:54 -05:00
saavagebueno commented 2025-11-20 07:09:56 -05:00
Author
Owner
Copy Link

@fishboy25uk commented on GitHub (Nov 6, 2025):

Full docker-compose.yml:

x-default: &default
  restart: 'unless-stopped'
  logging:
    driver: 'json-file'
    options:
      max-size: '500m'
      max-file: '2'

services:
  # UI dashboard
  dashboard:
    <<: *default
    image: netbirdio/dashboard:latest
    ports:
      - 80:80
      - 443:443
    environment:
      # Endpoints
      - NETBIRD_MGMT_API_ENDPOINT=https://vpn.<SERVER DOMAIN>:33073
      - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://vpn.<SERVER DOMAIN>:33073
      # OIDC
      - AUTH_AUDIENCE=<APP ID>
      - AUTH_CLIENT_ID=<APP ID>
      - AUTH_CLIENT_SECRET=
      - AUTH_AUTHORITY=https://login.microsoftonline.com/<TENANT ID>/v2.0
      - USE_AUTH0=false
      - AUTH_SUPPORTED_SCOPES=openid profile email offline_access User.Read api://<APP ID>/api
      - AUTH_REDIRECT_URI=/auth
      - AUTH_SILENT_REDIRECT_URI=/silent-auth
      - NETBIRD_TOKEN_SOURCE=idToken
      # SSL
      - NGINX_SSL_PORT=443
      # Letsencrypt
      - LETSENCRYPT_DOMAIN=vpn.<SERVER DOMAIN>
      - LETSENCRYPT_EMAIL=<MY EMAIL>
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt/

  # Signal
  signal:
    <<: *default
    image: netbirdio/signal:latest
    depends_on:
          - dashboard
    volumes:
      - netbird-signal:/var/lib/netbird
      - /etc/letsencrypt:/etc/letsencrypt:ro
    ports:
      - 10000:80
  #      # port and command for Let's Encrypt validation
  #      - 443:443
  #    command: ["--letsencrypt-domain", "vpn.<SERVER DOMAIN>", "--log-file", "console"]
    command: [
      "--cert-file", "/etc/letsencrypt/live/vpn.<SERVER DOMAIN>/fullchain.pem",
      "--cert-key", "/etc/letsencrypt/live/vpn.<SERVER DOMAIN>/privkey.pem",
      "--log-file", "console"
    ]

  # Relay
  relay:
    <<: *default
    image: netbirdio/relay:latest
    environment:
    - NB_LOG_LEVEL=info
    - NB_LISTEN_ADDRESS=:33080
    - NB_EXPOSED_ADDRESS=rel://vpn.<SERVER DOMAIN>:33080
    # todo: change to a secure secret
    - NB_AUTH_SECRET=<SECRET>
    ports:
      - 33080:33080

  # Management
  management:
    <<: *default
    image: netbirdio/management:latest
    depends_on:
      - dashboard
    volumes:
      - netbird-mgmt:/var/lib/netbird
      - netbird-letsencrypt:/etc/letsencrypt:ro
      - ./management.json:/etc/netbird/management.json
    ports:
      - 33073:443 #API port
  #    # command for Let's Encrypt validation without dashboard container
  #    command: ["--letsencrypt-domain", "vpn.<SERVER DOMAIN>", "--log-file", "console"]
    command: [
      "--port", "443",
      "--log-file", "console",
      "--log-level", "info",
      "--disable-anonymous-metrics=false",
      "--single-account-mode-domain=remotevpn.<SERVER DOMAIN>",
      "--dns-domain=netbird.selfhosted"
      ]
    environment:
      - NETBIRD_STORE_ENGINE_POSTGRES_DSN=
      - NETBIRD_STORE_ENGINE_MYSQL_DSN=
      
  # Coturn
  coturn:
    <<: *default
    image: coturn/coturn:latest
    #domainname: vpn.<SERVER DOMAIN> # only needed when TLS is enabled
    volumes:
      - ./turnserver.conf:/etc/turnserver.conf:ro
    #      - ./privkey.pem:/etc/coturn/private/privkey.pem:ro
    #      - ./cert.pem:/etc/coturn/certs/cert.pem:ro
    network_mode: host
    command:
      - -c /etc/turnserver.conf

volumes:
  netbird-mgmt:
  netbird-signal:
  netbird-letsencrypt:
@fishboy25uk commented on GitHub (Nov 6, 2025): Full docker-compose.yml: ``` x-default: &default restart: 'unless-stopped' logging: driver: 'json-file' options: max-size: '500m' max-file: '2' services: # UI dashboard dashboard: <<: *default image: netbirdio/dashboard:latest ports: - 80:80 - 443:443 environment: # Endpoints - NETBIRD_MGMT_API_ENDPOINT=https://vpn.<SERVER DOMAIN>:33073 - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://vpn.<SERVER DOMAIN>:33073 # OIDC - AUTH_AUDIENCE=<APP ID> - AUTH_CLIENT_ID=<APP ID> - AUTH_CLIENT_SECRET= - AUTH_AUTHORITY=https://login.microsoftonline.com/<TENANT ID>/v2.0 - USE_AUTH0=false - AUTH_SUPPORTED_SCOPES=openid profile email offline_access User.Read api://<APP ID>/api - AUTH_REDIRECT_URI=/auth - AUTH_SILENT_REDIRECT_URI=/silent-auth - NETBIRD_TOKEN_SOURCE=idToken # SSL - NGINX_SSL_PORT=443 # Letsencrypt - LETSENCRYPT_DOMAIN=vpn.<SERVER DOMAIN> - LETSENCRYPT_EMAIL=<MY EMAIL> volumes: - /etc/letsencrypt:/etc/letsencrypt/ # Signal signal: <<: *default image: netbirdio/signal:latest depends_on: - dashboard volumes: - netbird-signal:/var/lib/netbird - /etc/letsencrypt:/etc/letsencrypt:ro ports: - 10000:80 # # port and command for Let's Encrypt validation # - 443:443 # command: ["--letsencrypt-domain", "vpn.<SERVER DOMAIN>", "--log-file", "console"] command: [ "--cert-file", "/etc/letsencrypt/live/vpn.<SERVER DOMAIN>/fullchain.pem", "--cert-key", "/etc/letsencrypt/live/vpn.<SERVER DOMAIN>/privkey.pem", "--log-file", "console" ] # Relay relay: <<: *default image: netbirdio/relay:latest environment: - NB_LOG_LEVEL=info - NB_LISTEN_ADDRESS=:33080 - NB_EXPOSED_ADDRESS=rel://vpn.<SERVER DOMAIN>:33080 # todo: change to a secure secret - NB_AUTH_SECRET=<SECRET> ports: - 33080:33080 # Management management: <<: *default image: netbirdio/management:latest depends_on: - dashboard volumes: - netbird-mgmt:/var/lib/netbird - netbird-letsencrypt:/etc/letsencrypt:ro - ./management.json:/etc/netbird/management.json ports: - 33073:443 #API port # # command for Let's Encrypt validation without dashboard container # command: ["--letsencrypt-domain", "vpn.<SERVER DOMAIN>", "--log-file", "console"] command: [ "--port", "443", "--log-file", "console", "--log-level", "info", "--disable-anonymous-metrics=false", "--single-account-mode-domain=remotevpn.<SERVER DOMAIN>", "--dns-domain=netbird.selfhosted" ] environment: - NETBIRD_STORE_ENGINE_POSTGRES_DSN= - NETBIRD_STORE_ENGINE_MYSQL_DSN= # Coturn coturn: <<: *default image: coturn/coturn:latest #domainname: vpn.<SERVER DOMAIN> # only needed when TLS is enabled volumes: - ./turnserver.conf:/etc/turnserver.conf:ro # - ./privkey.pem:/etc/coturn/private/privkey.pem:ro # - ./cert.pem:/etc/coturn/certs/cert.pem:ro network_mode: host command: - -c /etc/turnserver.conf volumes: netbird-mgmt: netbird-signal: netbird-letsencrypt: ```
saavagebueno commented 2025-11-20 07:09:56 -05:00
Author
Owner
Copy Link

@fishboy25uk commented on GitHub (Nov 6, 2025):

pinging @bcmmbaga . Thanks in advance!

@fishboy25uk commented on GitHub (Nov 6, 2025): pinging @bcmmbaga . Thanks in advance!
saavagebueno commented 2025-11-20 07:09:56 -05:00
Author
Owner
Copy Link

@welbow commented on GitHub (Nov 15, 2025):

I too am having this problem. I deployed a new Netbird instance this month and ran into this with my first client. After troubleshooting, it looks like the docker port is mapping 10000 to port 80 inside the container, but I do not see anything listening inside the container on port 80.

root@myhost:~/netbird/infrastructure_files/artifacts# docker compose exec signal sh
/ # netstat -antlp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:6060          0.0.0.0:*               LISTEN      1/netbird-signal
tcp        0      0 127.0.0.11:39981        0.0.0.0:*               LISTEN      -
tcp        0      0 :::10000                :::*                    LISTEN      1/netbird-signal
tcp        0      0 :::9090                 :::*                    LISTEN      1/netbird-signal
tcp        0      0 :::443                  :::*                    LISTEN      1/netbird-signal
/ #

root@myhost:~/netbird/infrastructure_files/artifacts# docker compose port signal 80
0.0.0.0:10000

I did update my netbird repository to the latest tag & re-ran configure.sh just in case the docker-compose template file had been updated.

@welbow commented on GitHub (Nov 15, 2025): I too am having this problem. I deployed a new Netbird instance this month and ran into this with my first client. After troubleshooting, it looks like the docker port is mapping 10000 to port 80 inside the container, but I do not see anything listening inside the container on port 80. ``` root@myhost:~/netbird/infrastructure_files/artifacts# docker compose exec signal sh / # netstat -antlp Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:6060 0.0.0.0:* LISTEN 1/netbird-signal tcp 0 0 127.0.0.11:39981 0.0.0.0:* LISTEN - tcp 0 0 :::10000 :::* LISTEN 1/netbird-signal tcp 0 0 :::9090 :::* LISTEN 1/netbird-signal tcp 0 0 :::443 :::* LISTEN 1/netbird-signal / # root@myhost:~/netbird/infrastructure_files/artifacts# docker compose port signal 80 0.0.0.0:10000 ``` I did update my netbird repository to the latest tag & re-ran configure.sh just in case the docker-compose template file had been updated.
saavagebueno commented 2025-11-20 07:09:56 -05:00
Author
Owner
Copy Link

@davidpryke commented on GitHub (Nov 15, 2025):

I setup a self-hosted netbird this week, and finally got to connecting a client today and ran into this problem, which brought me here.

But, this thread was super helpful:

root@myhost:~/netbird/infrastructure_files/artifacts# docker compose port signal 80
0.0.0.0:10000

Thank you for your investigation and sharing this, @welbow ! This was the key to resolving this for me.

  • First, I ran docker compose down
  • Then I edited docker-compose.yml
    • under the signal section, changed the ports configuration from - 10000:80 to - 10000:10000
  • Then I ran docker compose up - d and tested again and it all worked!

So that is the key. I won't have time to submit any changes or pull requests or anything this weekend, but I wanted to report back that making this change worked for me.

Thank you, thank you!

@davidpryke commented on GitHub (Nov 15, 2025): I setup a self-hosted netbird this week, and finally got to connecting a client today and ran into this problem, which brought me here. But, this thread was super helpful: > root@myhost:~/netbird/infrastructure_files/artifacts# docker compose port signal 80 > 0.0.0.0:10000 > ``` Thank you for your investigation and sharing this, @welbow ! This was the key to resolving this for me. * First, I ran `docker compose down` * Then I edited `docker-compose.yml` * under the signal section, changed the ports configuration from ```- 10000:80``` to ```- 10000:10000``` * Then I ran `docker compose up - d` and tested again and it all worked! So that *is* the key. I won't have time to submit any changes or pull requests or anything this weekend, but I wanted to report back that making this change worked for me. Thank you, thank you!
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#2443
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?