NetBird behind Traefik results in 404 page not found #1476

Closed
opened 2025-11-20 05:31:19 -05:00 by saavagebueno · 2 comments
Owner

Originally created by @den5o on GitHub (Dec 6, 2024).

Describe the problem

I followed the advanced guide to self host NetBird behind Traefik but I get 404 page not found on https://netbird.domain.tld. I am running everything in rootless podman. Previous test installation using auth0, but without a reverse proxy, was working fine without issue in rootless podman.

To Reproduce

  1. Set up Traefik v3.2.1
  2. Set up Keycloak hosted at https://auth.domain.tld behind Traefik on the same VPS.
  3. Configure Keycloak as per docs.
  4. Rename docker-compose.yml.tmpl.traefik to docker-compose.yml.tmpl and fill info in setup.env
  5. Run configure.sh
  6. Allow ports 80, 443, 33080, 3478:udp, 49152-65535:udp through firewall. (I also tried allowing all ports listed in the guide but the issue persists).
  7. Deploy container stack.

Expected behavior

Visiting https://netbird.domain.tld should load NetBird dashboard and ask for authentication but instead results in 404.

NetBird version

0.34.1

Traefik version

3.2.1

Configuration

NetBird

services:
  #UI dashboard
  dashboard:
    image: docker.io/netbirdio/dashboard:latest
    container_name: netbird_dashboard
    restart: unless-stopped
    networks:
      - netbird
    dns:
      - "9.9.9.9"
      - "149.112.112.112"
    environment:
      # Endpoints
      - NETBIRD_MGMT_API_ENDPOINT=https://netbird.domain.tld:443
      - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://netbird.domain.tld:443
      # OIDC
      - AUTH_AUDIENCE=netbird-client
      - AUTH_CLIENT_ID=netbird-client
      - AUTH_CLIENT_SECRET=
      - AUTH_AUTHORITY=https://auth.domain.tld/realms/netbird
      - USE_AUTH0=false
      - AUTH_SUPPORTED_SCOPES=openid profile email offline_access api
      - AUTH_REDIRECT_URI=
      - AUTH_SILENT_REDIRECT_URI=
      - NETBIRD_TOKEN_SOURCE=accessToken
      # SSL
      - NGINX_SSL_PORT=443
      # Letsencrypt
      - LETSENCRYPT_DOMAIN=
      - LETSENCRYPT_EMAIL=
    volumes:
      - netbird_letsencrypt:/etc/letsencrypt/
    labels:
    - traefik.enable=true
    - traefik.http.routers.netbird-dashboard.rule=Host(`netbird.domain.tld`)
    - traefik.http.services.netbird-dashboard.loadbalancer.server.port=80

  # Signal
  signal:
    image: docker.io/netbirdio/signal:latest
    container_name: netbird_signal
    restart: unless-stopped
    networks:
      - netbird
    dns:
      - "9.9.9.9"
      - "149.112.112.112"
    volumes:
      - netbird_signal:/var/lib/netbird
    labels:
    - traefik.enable=true
    - traefik.http.routers.netbird-signal.rule=Host(`netbird.domain.tld`) && PathPrefix(`/signalexchange.SignalExchange/`)
    - traefik.http.services.netbird-signal.loadbalancer.server.port=80
    - traefik.http.services.netbird-signal.loadbalancer.server.scheme=h2c

  # Management
  management:
    image: docker.io/netbirdio/management:latest
    container_name: netbird_management
    restart: unless-stopped
    depends_on:
      - dashboard
    networks:
      - netbird
    dns:
      - "9.9.9.9"
      - "149.112.112.112"
    volumes:
      - netbird_mgmt:/var/lib/netbird
      - netbird_letsencrypt:/etc/letsencrypt:ro
      - ./management.json:/etc/netbird/management.json
    command: [
      "--port", "443",
      "--log-file", "console",
      "--disable-anonymous-metrics=true",
      "--single-account-mode-domain=netbird.domain.tld",
      "--dns-domain=netbird.badhippo",
      "--user-delete-from-idp",
      "--disable-single-account-mode"
      ]
    labels:
    - traefik.enable=true
    - traefik.http.routers.netbird-api.rule=Host(`netbird.domain.tld`) && PathPrefix(`/api`)
    - traefik.http.routers.netbird-api.service=netbird-api
    - traefik.http.services.netbird-api.loadbalancer.server.port=443
    - traefik.http.routers.netbird-management.rule=Host(`netbird.domain.tld`) && PathPrefix(`/management.ManagementService/`)
    - traefik.http.routers.netbird-management.service=netbird-management
    - traefik.http.services.netbird-management.loadbalancer.server.port=443
    - traefik.http.services.netbird-management.loadbalancer.server.scheme=h2c
    environment:
      - NETBIRD_STORE_ENGINE_POSTGRES_DSN=
  # Coturn
  coturn:
    image: docker.io/coturn/coturn:latest
    container_name: netbird_coturn
    restart: unless-stopped
    domainname: netbird.domain.tld
    network_mode: host
    volumes:
      - ./turnserver.conf:/etc/turnserver.conf:ro
    command:
      - -c /etc/turnserver.conf

networks:
  netbird:
    external: true

volumes:
  netbird_mgmt:
    external: true
  netbird_signal:
    external: true
  netbird_letsencrypt:
    external: true

management.json

{
    "Stuns": [
        {
            "Proto": "udp",
            "URI": "stun:netbird.domain.tld:3478",
            "Username": "",
            "Password": ""
        }
    ],
    "TURNConfig": {
        "TimeBasedCredentials": false,
        "CredentialsTTL": "12h0m0s",
        "Secret": "secret",
        "Turns": [
            {
                "Proto": "udp",
                "URI": "turn:netbird.domain.tld:3478",
                "Username": "self",
                "Password": "redacted"
            }
        ]
    },
    "Relay": {
        "Addresses": [
            "rel://netbird.domain.tld:33080"
        ],
        "CredentialsTTL": "24h0m0s",
        "Secret": "redacted"
    },
    "Signal": {
        "Proto": "https",
        "URI": "netbird.domain.tld:443",
        "Username": "",
        "Password": ""
    },
    "Datadir": "/var/lib/netbird/",
    "DataStoreEncryptionKey": "redacted",
    "HttpConfig": {
        "LetsEncryptDomain": "",
        "CertFile": "",
        "CertKey": "",
        "AuthAudience": "",
        "AuthIssuer": "https://auth.domain.tld/realms/netbird",
        "AuthUserIDClaim": "",
        "AuthKeysLocation": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/certs",
        "OIDCConfigEndpoint": "https://auth.domain.tld/realms/netbird/.well-known/openid-configuration",
        "IdpSignKeyRefreshEnabled": false,
        "ExtraAuthAudience": ""
    },
    "IdpManagerConfig": {
        "ManagerType": "keycloak",
        "ClientConfig": {
            "Issuer": "https://auth.domain.tld/realms/netbird",
            "TokenEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/token",
            "ClientID": "netbird-backend",
            "ClientSecret": "redacted",
            "GrantType": "client_credentials"
        },
        "ExtraConfig": {
            "AdminEndpoint": "https://auth.domain.tld/admin/realms/netbird"
        },
        "Auth0ClientCredentials": null,
        "AzureClientCredentials": null,
        "KeycloakClientCredentials": null,
        "ZitadelClientCredentials": null
    },
    "DeviceAuthorizationFlow": {
        "Provider": "hosted",
        "ProviderConfig": {
            "ClientID": "netbird-client",
            "ClientSecret": "",
            "Domain": "auth.domain.tld",
            "Audience": "netbird-client",
            "TokenEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/token",
            "DeviceAuthEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth/device",
            "AuthorizationEndpoint": "",
            "Scope": "openid",
            "UseIDToken": false,
            "RedirectURLs": null
        }
    },
    "PKCEAuthorizationFlow": {
        "ProviderConfig": {
            "ClientID": "netbird-client",
            "ClientSecret": "",
            "Domain": "",
            "Audience": "",
            "TokenEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/token",
            "DeviceAuthEndpoint": "",
            "AuthorizationEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth",
            "Scope": "openid profile email offline_access api",
            "UseIDToken": false,
            "RedirectURLs": [
                "http://localhost:53000"
            ]
        }
    },
    "StoreConfig": {
        "Engine": "sqlite"
    },
    "ReverseProxy": {
        "TrustedHTTPProxies": [],
        "TrustedHTTPProxiesCount": 0,
        "TrustedPeers": [
            "0.0.0.0/0"
        ]
    }
}

turnserver.conf

listening-port=3478
tls-listening-port=5349
external-ip=redacted
min-port=49152
max-port=65535
fingerprint
lt-cred-mech
user=self:redacted
realm=wiretrustee.com
cert=/etc/coturn/certs/cert.pem
pkey=/etc/coturn/private/privkey.pem
log-file=stdout
no-software-attribute
pidfile="/var/tmp/turnserver.pid"
no-cli

Traefik

services:
  traefik:
    image: docker.io/library/traefik:v3.2.1
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    networks:
      - frontend
      - netbird
    ports:
      - 80:80
      - 443:443
    environment:
      HOST_DOMAIN: ${HOST_DOMAIN}
      PORKBUN_API_KEY_FILE: /run/secrets/porkbun_api_key
      PORKBUN_SECRET_API_KEY_FILE: /run/secrets/porkbun_secret_api_key
      TRAEFIK_DASHBOARD_CREDENTIALS: ${TRAEFIK_DASHBOARD_CREDENTIALS}
    env_file: .env
    volumes:
      - /run/user/1002/podman/podman.sock:/var/run/docker.sock
      - ./config/traefik.yaml:/traefik.yaml:ro
      - ./data/acme.json:/acme.json:rw
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.entrypoints=web"
      - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.${HOST_DOMAIN}`)"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_DASHBOARD_CREDENTIALS}"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=websecure"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.${HOST_DOMAIN}`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=porkbun"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=${HOST_DOMAIN}"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.${HOST_DOMAIN}"
      - "traefik.http.routers.traefik-secure.service=api@internal"
    secrets:
      - porkbun_api_key
      - porkbun_secret_api_key

networks:
  frontend:
    external: true
  netbird:
    external: true

secrets:
  porkbun_api_key:
    external: true
  porkbun_secret_api_key:
    external: true

Logs

Dashboard

+ LETSENCRYPT_DOMAIN=none
+ LETSENCRYPT_EMAIL=example@local
+ NGINX_SSL_PORT=443
+ '[' none-x == none-x ']'
+ exit 0
NetBird latest version: 
+ LETSENCRYPT_DOMAIN=none
+ LETSENCRYPT_EMAIL=example@local
+ NGINX_SSL_PORT=443
+ '[' none-x == none-x ']'
+ exit 0

Management

INFO [context: SYSTEM] management/cmd/management.go:497: loading OIDC configuration from the provided IDP configuration endpoint https://auth.domain.tld/realms/netbird/.well-known/openid-configuration
INFO [context: SYSTEM] management/cmd/management.go:502: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.domain.tld/realms/netbird/.well-known/openid-configuration
INFO [context: SYSTEM] management/cmd/management.go:504: overriding HttpConfig.AuthIssuer with a new value https://auth.domain.tld/realms/netbird, previously configured value: https://auth.domain.tld/realms/netbird
INFO [context: SYSTEM] management/cmd/management.go:508: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.domain.tld/realms/netbird/protocol/openid-connect/certs, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/certs
INFO [context: SYSTEM] management/cmd/management.go:513: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/token, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/token
INFO [context: SYSTEM] management/cmd/management.go:516: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth/device, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth/device
INFO [context: SYSTEM] management/cmd/management.go:524: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.domain.tld, previously configured value: 
INFO [context: SYSTEM] management/cmd/management.go:534: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/token, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/token
INFO [context: SYSTEM] management/cmd/management.go:537: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth
INFO management/cmd/management.go:544: Relay addresses: [rel://netbird.domain.tld:33080]
INFO [context: SYSTEM] management/server/telemetry/app_metrics.go:193: enabled application metrics and exposing on http://0.0.0.0:9090
INFO [context: SYSTEM] management/server/store.go:209: using SQLite store engine
INFO [context: SYSTEM] management/server/sql_store.go:83: Set max open db connections to 1
INFO management/server/migration/migration.go:120: Table for peer.Peer does not exist, no migration needed
INFO management/server/migration/migration.go:120: Table for peer.Peer does not exist, no migration needed
INFO [context: SYSTEM] management/cmd/management.go:175: update config with activity store key
INFO [context: SYSTEM] management/server/geolocation/database.go:34: Geolocation database file GeoLite2-City_20241115.mmdb not found, file will be downloaded
INFO [context: SYSTEM] management/server/geolocation/database.go:34: Geolocation database file geonames_20241115.db not found, file will be downloaded
INFO [context: SYSTEM] management/cmd/management.go:187: geolocation service has been initialized from /var/lib/netbird/
INFO [context: SYSTEM] management/server/account_request_buffer.go:42: set account request buffer interval to 100ms
INFO [context: SYSTEM] management/server/account.go:1076: single account mode disabled, accounts number 0
INFO [context: SYSTEM] management/cmd/management.go:305: running gRPC backward compatibility server: [::]:33073
INFO [context: SYSTEM] management/cmd/management.go:337: management server version 0.34.1
INFO [context: SYSTEM] management/cmd/management.go:338: running HTTP server and gRPC server on the same port: [::]:443
INFO [context: SYSTEM] management/server/account.go:1348: 1 entries received from IdP management
INFO [context: SYSTEM] management/server/account.go:1379: warmed up IDP cache with 0 entries for 0 accounts

Signal

INFO signal/cmd/run.go:181: running without TLS
INFO signal/cmd/run.go:127: running gRPC server: [::]:80
INFO signal/cmd/run.go:137: running gRPC backward compatibility server: [::]:10000
INFO signal/cmd/run.go:140: signal server version 0.34.1
INFO signal/cmd/run.go:141: started Signal Service
INFO signal/cmd/run.go:99: running metrics server: :9090/metrics

Coturn

INFO: System cpu num is 64
INFO: log file opened: /var/tmp/turn_1_2024-12-06.log
INFO: System enable num is 1
INFO: Coturn Version Coturn-4.6.2 'Gorst'
INFO: Coturn Version Coturn-4.6.2 'Gorst'
INFO: Max number of open files/sockets allowed for this process: 524288
INFO: Due to the open files/sockets limitation, max supported number of TURN Sessions possible is: 262000 (approximately)
INFO: 
==== Show him the instruments, Practical Frost: ====
INFO: OpenSSL compile-time version: OpenSSL 3.0.14 4 Jun 2024 (0x300000e0)
INFO: TLS 1.3 supported
INFO: DTLS 1.2 supported
INFO: TURN/STUN ALPN supported
INFO: Third-party authorization (oAuth) supported
INFO: GCM (AEAD) supported
INFO: SQLite supported, default database location is /var/lib/coturn/turndb
INFO: Redis supported
INFO: PostgreSQL supported
INFO: MySQL supported
INFO: MongoDB supported
INFO: Default Net Engine version: 3 (UDP thread per CPU core)
INFO: Domain name: 
INFO: Default realm: wiretrustee.com
WARNING: cannot find certificate file: /etc/coturn/certs/cert.pem (1)
WARNING: cannot start TLS and DTLS listeners because certificate file is not set properly
WARNING: cannot find private key file: /etc/coturn/private/privkey.pem (1)
WARNING: cannot start TLS and DTLS listeners because private key file is not set properly
INFO: Certificate file found: /etc/coturn/certs/cert.pem
INFO: Private key file found: /etc/coturn/private/privkey.pem
WARNING: NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED
INFO: ===========Discovering listener addresses: =========
INFO: Listener address to use: 127.0.0.1
INFO: Listener address to use: redacted
INFO: Listener address to use: ::1
INFO: Listener address to use: redacted
INFO: =====================================================
INFO: Total: 2 'real' addresses discovered
INFO: =====================================================
WARNING: NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED
INFO: ===========Discovering relay addresses: =============
INFO: Relay address to use: redacted
INFO: Relay address to use: redacted
INFO: =====================================================
INFO: Total: 2 relay addresses discovered
INFO: =====================================================
INFO: pid file created: /var/tmp/turnserver.pid
INFO: IO method: epoll (with changelist)
WARNING: STUN CHANGE_REQUEST not supported: only one IP address is provided
INFO: Wait for relay ports initialization...
INFO:   relay redacted initialization...
INFO:   relay redacted initialization done
INFO:   relay redacted initialization...
INFO:   relay redacted initialization done
INFO: Relay ports initialization done
DEBUG: turn server id=2 created
...
DEBUG: turn server id=46 created
INFO: Total General servers: 64
DEBUG: turn server id=47 created
...
DEBUG: turn server id=57 created
INFO: Total auth threads: 33
INFO: prometheus collector disabled, not started

Traefik

INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:103 > Traefik version 3.2.1 built on 2024-11-20T17:39:47Z version=3.2.1
DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:110 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"debug":true},"certificatesResolvers":{"porkbun":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"porkbun","resolvers":["1.1.1.1:53","8.8.8.8:53"]},"email":"admin@domain.tld","keyType":"EC256","storage":"acme.json"}}},"entryPoints":{"web":{"address":":80","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576,"redirections":{"entryPoint":{"permanent":true,"priority":9223372036854775806,"scheme":"https","to":"websecure"}}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":443","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"global":{},"log":{"format":"common","level":"DEBUG"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","watch":true},"file":{"filename":"/config.yml","watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}}
INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator *aggregator.ProviderAggregator
DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web
DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=websecure
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config={"filename":"/config.yml","watch":true}
DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /
DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /config.yml
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config={}
DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"redirect-web-to-websecure":{"redirectScheme":{"permanent":true,"port":"443","scheme":"https"}}},"routers":{"web-to-websecure":{"entryPoints":["web"],"middlewares":["redirect-web-to-websecure"],"priority":9223372036854775806,"rule":"HostRegexp(`^.+$`)","ruleSyntax":"v3","service":"noop@internal"}},"serversTransports":{"default":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200}},"services":{"api":{},"dashboard":{},"noop":{}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"tls":{},"udp":{}} providerName=internal
DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"services":{"cockpit":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://redacted:9090"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=file
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *docker.Provider
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *docker.Provider provider configuration config={"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","watch":true}
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={}
INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider
DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"porkbun","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"porkbun","resolvers":["1.1.1.1:53","8.8.8.8:53"]},"email":"admin@domain.tld","keyType":"EC256","storage":"acme.json","store":{}}
DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:221 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=porkbun.acme
INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:859 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=porkbun.acme
DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=porkbun.acme
DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) *.domain.tld,domain.tld
DBG github.com/traefik/traefik/v3/pkg/provider/docker/pdocker.go:90 > Provider connection established with docker 5.2.2 (API 1.41) providerName=docker
DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:184 > Filtering disabled container container=coturn-netbird-9c86101ecaf97e9a01101768057e46c043841f2e1d6b0e29674490b1766056a4 providerName=docker
DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"sslheader":{"headers":{"customRequestHeaders":{"X-Forwarded-Proto":"https"}}},"traefik-auth":{"basicAuth":{"users":["admin:$apr1$kRTh8V.I$zEM0xhyt6GBpLRPVoIzt30"]}},"traefik-https-redirect":{"redirectScheme":{"scheme":"https"}}},"routers":{"cockpit":{"entryPoints":["websecure"],"priority":1000,"rule":"Host(`cockpit.domain.tld`)","service":"cockpit@file","tls":{"certResolver":"porkbun"}},"keycloak":{"entryPoints":["websecure"],"rule":"Host(`auth.domain.tld`)","service":"cloud-network-keycloak","tls":{}},"netbird-api":{"rule":"Host(`netbird.domain.tld`) \u0026\u0026 PathPrefix(`/api`)","service":"netbird-api"},"netbird-dashboard":{"rule":"Host(`netbird.domain.tld`)","service":"netbird-dashboard"},"netbird-management":{"rule":"Host(`netbird.domain.tld`) \u0026\u0026 PathPrefix(`/management.ManagementService/`)","service":"netbird-management"},"netbird-signal":{"rule":"Host(`netbird.domain.tld`) \u0026\u0026 PathPrefix(`/signalexchange.SignalExchange/`)","service":"netbird-signal"},"traefik":{"entryPoints":["web"],"middlewares":["traefik-https-redirect"],"rule":"Host(`traefik-dashboard.domain.tld`)","service":"traefik-traefik"},"traefik-secure":{"entryPoints":["websecure"],"middlewares":["traefik-auth"],"rule":"Host(`traefik-dashboard.domain.tld`)","service":"api@internal","tls":{"certResolver":"porkbun","domains":[{"main":"domain.tld","sans":["*.domain.tld"]}]}}},"services":{"cloud-network-keycloak":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.89.3.3:8080"}]}},"netbird-api":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.89.2.4:443"}]}},"netbird-dashboard":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.89.2.2:80"}]}},"netbird-management":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"h2c://10.89.2.4:443"}]}},"netbird-signal":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"h2c://10.89.2.3:80"}]}},"traefik-traefik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.89.3.7:80"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=docker
DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
DBG github.com/traefik/traefik/v3/pkg/server/aggregator.go:51 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["web","websecure"] routerName=netbird-api
DBG github.com/traefik/traefik/v3/pkg/server/aggregator.go:51 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["web","websecure"] routerName=netbird-management
DBG github.com/traefik/traefik/v3/pkg/server/aggregator.go:51 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["web","websecure"] routerName=netbird-signal
DBG github.com/traefik/traefik/v3/pkg/server/aggregator.go:51 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["web","websecure"] routerName=netbird-dashboard
DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) *.domain.tld,domain.tld
DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=traefik-https-redirect@docker middlewareType=RedirectScheme routerName=traefik@docker
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https  entryPointName=web middlewareName=traefik-https-redirect@docker middlewareType=RedirectScheme routerName=traefik@docker
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:318 > Creating load-balancer entryPointName=web routerName=netbird-api@docker serviceName=netbird-api@docker
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:355 > Creating server entryPointName=web routerName=netbird-api@docker serverName=573a1fd9790d212e serviceName=netbird-api@docker target=http://10.89.2.4:443
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:318 > Creating load-balancer entryPointName=web routerName=netbird-signal@docker serviceName=netbird-signal@docker
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:355 > Creating server entryPointName=web routerName=netbird-signal@docker serverName=b802a5f2e429e875 serviceName=netbird-signal@docker target=h2c://10.89.2.3:80
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:318 > Creating load-balancer entryPointName=web routerName=netbird-dashboard@docker serviceName=netbird-dashboard@docker
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:355 > Creating server entryPointName=web routerName=netbird-dashboard@docker serverName=a7c19bef807bb55f serviceName=netbird-dashboard@docker target=http://10.89.2.2:80
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:318 > Creating load-balancer entryPointName=web routerName=netbird-management@docker serviceName=netbird-management@docker
DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:355 > Creating server entryPointName=web routerName=netbird-management@docker serverName=3d644fa6d1781c2f serviceName=netbird-management@docker target=h2c://10.89.2.4:443
DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for auth.domain.tld with TLS options default entryPointName=websecure

Screenshots of Traefik Dashboard

Routers
routers
Services
services
Additional context

  • Tried advice from this old Reddit post for Traefik v2.
  • Configure.sh results in this entry in the compose file:
      # Endpoints
      - NETBIRD_MGMT_API_ENDPOINT=https://netbird.domain.tld:33073
      - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://netbird.domain.tld:33073

Opening port 33073 and mapping it to Traefik did not resolve the issue.

  • I also tried:
      # Endpoints
      - NETBIRD_MGMT_API_ENDPOINT=https://netbird.domain.tld:443
      - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://netbird.domain.tld:443
  • management.json also results in a lot of entries such as rel://netbird.domain.tld:33080. I tried too many combinations to remember but nothing worked.
Originally created by @den5o on GitHub (Dec 6, 2024). **Describe the problem** I followed the [advanced guide](https://docs.netbird.io/selfhosted/selfhosted-guide) to self host NetBird behind Traefik but I get `404 page not found` on https://netbird.domain.tld. I am running everything in rootless podman. Previous _test installation_ using auth0, but **without a reverse proxy**, was working fine without issue in rootless podman. **To Reproduce** 1. Set up Traefik `v3.2.1` 2. Set up Keycloak hosted at `https://auth.domain.tld` behind Traefik on the same VPS. 3. Configure Keycloak as per [docs](https://docs.netbird.io/selfhosted/identity-providers#keycloak). 4. Rename `docker-compose.yml.tmpl.traefik` to `docker-compose.yml.tmpl` and fill info in `setup.env` 5. Run `configure.sh` 6. Allow ports `80`, `443`, `33080`, `3478:udp`, `49152-65535:udp` through firewall. (I also tried allowing all ports listed in the guide but the issue persists). 7. Deploy container stack. **Expected behavior** Visiting https://netbird.domain.tld should load NetBird dashboard and ask for authentication but instead results in 404. **NetBird version** `0.34.1` **Traefik version** `3.2.1` **Configuration** ***NetBird*** ``` services: #UI dashboard dashboard: image: docker.io/netbirdio/dashboard:latest container_name: netbird_dashboard restart: unless-stopped networks: - netbird dns: - "9.9.9.9" - "149.112.112.112" environment: # Endpoints - NETBIRD_MGMT_API_ENDPOINT=https://netbird.domain.tld:443 - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://netbird.domain.tld:443 # OIDC - AUTH_AUDIENCE=netbird-client - AUTH_CLIENT_ID=netbird-client - AUTH_CLIENT_SECRET= - AUTH_AUTHORITY=https://auth.domain.tld/realms/netbird - USE_AUTH0=false - AUTH_SUPPORTED_SCOPES=openid profile email offline_access api - AUTH_REDIRECT_URI= - AUTH_SILENT_REDIRECT_URI= - NETBIRD_TOKEN_SOURCE=accessToken # SSL - NGINX_SSL_PORT=443 # Letsencrypt - LETSENCRYPT_DOMAIN= - LETSENCRYPT_EMAIL= volumes: - netbird_letsencrypt:/etc/letsencrypt/ labels: - traefik.enable=true - traefik.http.routers.netbird-dashboard.rule=Host(`netbird.domain.tld`) - traefik.http.services.netbird-dashboard.loadbalancer.server.port=80 # Signal signal: image: docker.io/netbirdio/signal:latest container_name: netbird_signal restart: unless-stopped networks: - netbird dns: - "9.9.9.9" - "149.112.112.112" volumes: - netbird_signal:/var/lib/netbird labels: - traefik.enable=true - traefik.http.routers.netbird-signal.rule=Host(`netbird.domain.tld`) && PathPrefix(`/signalexchange.SignalExchange/`) - traefik.http.services.netbird-signal.loadbalancer.server.port=80 - traefik.http.services.netbird-signal.loadbalancer.server.scheme=h2c # Management management: image: docker.io/netbirdio/management:latest container_name: netbird_management restart: unless-stopped depends_on: - dashboard networks: - netbird dns: - "9.9.9.9" - "149.112.112.112" volumes: - netbird_mgmt:/var/lib/netbird - netbird_letsencrypt:/etc/letsencrypt:ro - ./management.json:/etc/netbird/management.json command: [ "--port", "443", "--log-file", "console", "--disable-anonymous-metrics=true", "--single-account-mode-domain=netbird.domain.tld", "--dns-domain=netbird.badhippo", "--user-delete-from-idp", "--disable-single-account-mode" ] labels: - traefik.enable=true - traefik.http.routers.netbird-api.rule=Host(`netbird.domain.tld`) && PathPrefix(`/api`) - traefik.http.routers.netbird-api.service=netbird-api - traefik.http.services.netbird-api.loadbalancer.server.port=443 - traefik.http.routers.netbird-management.rule=Host(`netbird.domain.tld`) && PathPrefix(`/management.ManagementService/`) - traefik.http.routers.netbird-management.service=netbird-management - traefik.http.services.netbird-management.loadbalancer.server.port=443 - traefik.http.services.netbird-management.loadbalancer.server.scheme=h2c environment: - NETBIRD_STORE_ENGINE_POSTGRES_DSN= # Coturn coturn: image: docker.io/coturn/coturn:latest container_name: netbird_coturn restart: unless-stopped domainname: netbird.domain.tld network_mode: host volumes: - ./turnserver.conf:/etc/turnserver.conf:ro command: - -c /etc/turnserver.conf networks: netbird: external: true volumes: netbird_mgmt: external: true netbird_signal: external: true netbird_letsencrypt: external: true ``` ***management.json*** ``` { "Stuns": [ { "Proto": "udp", "URI": "stun:netbird.domain.tld:3478", "Username": "", "Password": "" } ], "TURNConfig": { "TimeBasedCredentials": false, "CredentialsTTL": "12h0m0s", "Secret": "secret", "Turns": [ { "Proto": "udp", "URI": "turn:netbird.domain.tld:3478", "Username": "self", "Password": "redacted" } ] }, "Relay": { "Addresses": [ "rel://netbird.domain.tld:33080" ], "CredentialsTTL": "24h0m0s", "Secret": "redacted" }, "Signal": { "Proto": "https", "URI": "netbird.domain.tld:443", "Username": "", "Password": "" }, "Datadir": "/var/lib/netbird/", "DataStoreEncryptionKey": "redacted", "HttpConfig": { "LetsEncryptDomain": "", "CertFile": "", "CertKey": "", "AuthAudience": "", "AuthIssuer": "https://auth.domain.tld/realms/netbird", "AuthUserIDClaim": "", "AuthKeysLocation": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/certs", "OIDCConfigEndpoint": "https://auth.domain.tld/realms/netbird/.well-known/openid-configuration", "IdpSignKeyRefreshEnabled": false, "ExtraAuthAudience": "" }, "IdpManagerConfig": { "ManagerType": "keycloak", "ClientConfig": { "Issuer": "https://auth.domain.tld/realms/netbird", "TokenEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/token", "ClientID": "netbird-backend", "ClientSecret": "redacted", "GrantType": "client_credentials" }, "ExtraConfig": { "AdminEndpoint": "https://auth.domain.tld/admin/realms/netbird" }, "Auth0ClientCredentials": null, "AzureClientCredentials": null, "KeycloakClientCredentials": null, "ZitadelClientCredentials": null }, "DeviceAuthorizationFlow": { "Provider": "hosted", "ProviderConfig": { "ClientID": "netbird-client", "ClientSecret": "", "Domain": "auth.domain.tld", "Audience": "netbird-client", "TokenEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/token", "DeviceAuthEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth/device", "AuthorizationEndpoint": "", "Scope": "openid", "UseIDToken": false, "RedirectURLs": null } }, "PKCEAuthorizationFlow": { "ProviderConfig": { "ClientID": "netbird-client", "ClientSecret": "", "Domain": "", "Audience": "", "TokenEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/token", "DeviceAuthEndpoint": "", "AuthorizationEndpoint": "https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth", "Scope": "openid profile email offline_access api", "UseIDToken": false, "RedirectURLs": [ "http://localhost:53000" ] } }, "StoreConfig": { "Engine": "sqlite" }, "ReverseProxy": { "TrustedHTTPProxies": [], "TrustedHTTPProxiesCount": 0, "TrustedPeers": [ "0.0.0.0/0" ] } } ``` ***turnserver.conf*** ``` listening-port=3478 tls-listening-port=5349 external-ip=redacted min-port=49152 max-port=65535 fingerprint lt-cred-mech user=self:redacted realm=wiretrustee.com cert=/etc/coturn/certs/cert.pem pkey=/etc/coturn/private/privkey.pem log-file=stdout no-software-attribute pidfile="/var/tmp/turnserver.pid" no-cli ``` ***Traefik*** ``` services: traefik: image: docker.io/library/traefik:v3.2.1 container_name: traefik restart: unless-stopped security_opt: - no-new-privileges:true networks: - frontend - netbird ports: - 80:80 - 443:443 environment: HOST_DOMAIN: ${HOST_DOMAIN} PORKBUN_API_KEY_FILE: /run/secrets/porkbun_api_key PORKBUN_SECRET_API_KEY_FILE: /run/secrets/porkbun_secret_api_key TRAEFIK_DASHBOARD_CREDENTIALS: ${TRAEFIK_DASHBOARD_CREDENTIALS} env_file: .env volumes: - /run/user/1002/podman/podman.sock:/var/run/docker.sock - ./config/traefik.yaml:/traefik.yaml:ro - ./data/acme.json:/acme.json:rw labels: - "traefik.enable=true" - "traefik.http.routers.traefik.entrypoints=web" - "traefik.http.routers.traefik.rule=Host(`traefik-dashboard.${HOST_DOMAIN}`)" - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_DASHBOARD_CREDENTIALS}" - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https" - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https" - "traefik.http.routers.traefik.middlewares=traefik-https-redirect" - "traefik.http.routers.traefik-secure.entrypoints=websecure" - "traefik.http.routers.traefik-secure.rule=Host(`traefik-dashboard.${HOST_DOMAIN}`)" - "traefik.http.routers.traefik-secure.middlewares=traefik-auth" - "traefik.http.routers.traefik-secure.tls=true" - "traefik.http.routers.traefik-secure.tls.certresolver=porkbun" - "traefik.http.routers.traefik-secure.tls.domains[0].main=${HOST_DOMAIN}" - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.${HOST_DOMAIN}" - "traefik.http.routers.traefik-secure.service=api@internal" secrets: - porkbun_api_key - porkbun_secret_api_key networks: frontend: external: true netbird: external: true secrets: porkbun_api_key: external: true porkbun_secret_api_key: external: true ``` **Logs** ***Dashboard*** ``` + LETSENCRYPT_DOMAIN=none + LETSENCRYPT_EMAIL=example@local + NGINX_SSL_PORT=443 + '[' none-x == none-x ']' + exit 0 NetBird latest version: + LETSENCRYPT_DOMAIN=none + LETSENCRYPT_EMAIL=example@local + NGINX_SSL_PORT=443 + '[' none-x == none-x ']' + exit 0 ``` ***Management*** ``` INFO [context: SYSTEM] management/cmd/management.go:497: loading OIDC configuration from the provided IDP configuration endpoint https://auth.domain.tld/realms/netbird/.well-known/openid-configuration INFO [context: SYSTEM] management/cmd/management.go:502: loaded OIDC configuration from the provided IDP configuration endpoint: https://auth.domain.tld/realms/netbird/.well-known/openid-configuration INFO [context: SYSTEM] management/cmd/management.go:504: overriding HttpConfig.AuthIssuer with a new value https://auth.domain.tld/realms/netbird, previously configured value: https://auth.domain.tld/realms/netbird INFO [context: SYSTEM] management/cmd/management.go:508: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://auth.domain.tld/realms/netbird/protocol/openid-connect/certs, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/certs INFO [context: SYSTEM] management/cmd/management.go:513: overriding DeviceAuthorizationFlow.TokenEndpoint with a new value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/token, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/token INFO [context: SYSTEM] management/cmd/management.go:516: overriding DeviceAuthorizationFlow.DeviceAuthEndpoint with a new value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth/device, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth/device INFO [context: SYSTEM] management/cmd/management.go:524: overriding DeviceAuthorizationFlow.ProviderConfig.Domain with a new value: auth.domain.tld, previously configured value: INFO [context: SYSTEM] management/cmd/management.go:534: overriding PKCEAuthorizationFlow.TokenEndpoint with a new value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/token, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/token INFO [context: SYSTEM] management/cmd/management.go:537: overriding PKCEAuthorizationFlow.AuthorizationEndpoint with a new value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth, previously configured value: https://auth.domain.tld/realms/netbird/protocol/openid-connect/auth INFO management/cmd/management.go:544: Relay addresses: [rel://netbird.domain.tld:33080] INFO [context: SYSTEM] management/server/telemetry/app_metrics.go:193: enabled application metrics and exposing on http://0.0.0.0:9090 INFO [context: SYSTEM] management/server/store.go:209: using SQLite store engine INFO [context: SYSTEM] management/server/sql_store.go:83: Set max open db connections to 1 INFO management/server/migration/migration.go:120: Table for peer.Peer does not exist, no migration needed INFO management/server/migration/migration.go:120: Table for peer.Peer does not exist, no migration needed INFO [context: SYSTEM] management/cmd/management.go:175: update config with activity store key INFO [context: SYSTEM] management/server/geolocation/database.go:34: Geolocation database file GeoLite2-City_20241115.mmdb not found, file will be downloaded INFO [context: SYSTEM] management/server/geolocation/database.go:34: Geolocation database file geonames_20241115.db not found, file will be downloaded INFO [context: SYSTEM] management/cmd/management.go:187: geolocation service has been initialized from /var/lib/netbird/ INFO [context: SYSTEM] management/server/account_request_buffer.go:42: set account request buffer interval to 100ms INFO [context: SYSTEM] management/server/account.go:1076: single account mode disabled, accounts number 0 INFO [context: SYSTEM] management/cmd/management.go:305: running gRPC backward compatibility server: [::]:33073 INFO [context: SYSTEM] management/cmd/management.go:337: management server version 0.34.1 INFO [context: SYSTEM] management/cmd/management.go:338: running HTTP server and gRPC server on the same port: [::]:443 INFO [context: SYSTEM] management/server/account.go:1348: 1 entries received from IdP management INFO [context: SYSTEM] management/server/account.go:1379: warmed up IDP cache with 0 entries for 0 accounts ``` ***Signal*** ``` INFO signal/cmd/run.go:181: running without TLS INFO signal/cmd/run.go:127: running gRPC server: [::]:80 INFO signal/cmd/run.go:137: running gRPC backward compatibility server: [::]:10000 INFO signal/cmd/run.go:140: signal server version 0.34.1 INFO signal/cmd/run.go:141: started Signal Service INFO signal/cmd/run.go:99: running metrics server: :9090/metrics ``` ***Coturn*** ``` INFO: System cpu num is 64 INFO: log file opened: /var/tmp/turn_1_2024-12-06.log INFO: System enable num is 1 INFO: Coturn Version Coturn-4.6.2 'Gorst' INFO: Coturn Version Coturn-4.6.2 'Gorst' INFO: Max number of open files/sockets allowed for this process: 524288 INFO: Due to the open files/sockets limitation, max supported number of TURN Sessions possible is: 262000 (approximately) INFO: ==== Show him the instruments, Practical Frost: ==== INFO: OpenSSL compile-time version: OpenSSL 3.0.14 4 Jun 2024 (0x300000e0) INFO: TLS 1.3 supported INFO: DTLS 1.2 supported INFO: TURN/STUN ALPN supported INFO: Third-party authorization (oAuth) supported INFO: GCM (AEAD) supported INFO: SQLite supported, default database location is /var/lib/coturn/turndb INFO: Redis supported INFO: PostgreSQL supported INFO: MySQL supported INFO: MongoDB supported INFO: Default Net Engine version: 3 (UDP thread per CPU core) INFO: Domain name: INFO: Default realm: wiretrustee.com WARNING: cannot find certificate file: /etc/coturn/certs/cert.pem (1) WARNING: cannot start TLS and DTLS listeners because certificate file is not set properly WARNING: cannot find private key file: /etc/coturn/private/privkey.pem (1) WARNING: cannot start TLS and DTLS listeners because private key file is not set properly INFO: Certificate file found: /etc/coturn/certs/cert.pem INFO: Private key file found: /etc/coturn/private/privkey.pem WARNING: NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED INFO: ===========Discovering listener addresses: ========= INFO: Listener address to use: 127.0.0.1 INFO: Listener address to use: redacted INFO: Listener address to use: ::1 INFO: Listener address to use: redacted INFO: ===================================================== INFO: Total: 2 'real' addresses discovered INFO: ===================================================== WARNING: NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED INFO: ===========Discovering relay addresses: ============= INFO: Relay address to use: redacted INFO: Relay address to use: redacted INFO: ===================================================== INFO: Total: 2 relay addresses discovered INFO: ===================================================== INFO: pid file created: /var/tmp/turnserver.pid INFO: IO method: epoll (with changelist) WARNING: STUN CHANGE_REQUEST not supported: only one IP address is provided INFO: Wait for relay ports initialization... INFO: relay redacted initialization... INFO: relay redacted initialization done INFO: relay redacted initialization... INFO: relay redacted initialization done INFO: Relay ports initialization done DEBUG: turn server id=2 created ... DEBUG: turn server id=46 created INFO: Total General servers: 64 DEBUG: turn server id=47 created ... DEBUG: turn server id=57 created INFO: Total auth threads: 33 INFO: prometheus collector disabled, not started ``` ***Traefik*** ``` INF github.com/traefik/traefik/v3/cmd/traefik/traefik.go:103 > Traefik version 3.2.1 built on 2024-11-20T17:39:47Z version=3.2.1 DBG github.com/traefik/traefik/v3/cmd/traefik/traefik.go:110 > Static configuration loaded [json] staticConfiguration={"api":{"dashboard":true,"debug":true},"certificatesResolvers":{"porkbun":{"acme":{"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"porkbun","resolvers":["1.1.1.1:53","8.8.8.8:53"]},"email":"admin@domain.tld","keyType":"EC256","storage":"acme.json"}}},"entryPoints":{"web":{"address":":80","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576,"redirections":{"entryPoint":{"permanent":true,"priority":9223372036854775806,"scheme":"https","to":"websecure"}}},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}},"websecure":{"address":":443","forwardedHeaders":{},"http":{"maxHeaderBytes":1048576},"http2":{"maxConcurrentStreams":250},"transport":{"lifeCycle":{"graceTimeOut":"10s"},"respondingTimeouts":{"idleTimeout":"3m0s","readTimeout":"1m0s"}},"udp":{"timeout":"3s"}}},"global":{},"log":{"format":"common","level":"DEBUG"},"providers":{"docker":{"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","watch":true},"file":{"filename":"/config.yml","watch":true},"providersThrottleDuration":"2s"},"serversTransport":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200},"tcpServersTransport":{"dialKeepAlive":"15s","dialTimeout":"30s"}} INF github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:73 > Starting provider aggregator *aggregator.ProviderAggregator DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=web DBG github.com/traefik/traefik/v3/pkg/server/server_entrypoint_tcp.go:231 > Starting TCP Server entryPointName=websecure INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *file.Provider DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *file.Provider provider configuration config={"filename":"/config.yml","watch":true} DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: / DBG github.com/traefik/traefik/v3/pkg/provider/file/file.go:122 > add watcher on: /config.yml INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *traefik.Provider DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *traefik.Provider provider configuration config={} DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"redirect-web-to-websecure":{"redirectScheme":{"permanent":true,"port":"443","scheme":"https"}}},"routers":{"web-to-websecure":{"entryPoints":["web"],"middlewares":["redirect-web-to-websecure"],"priority":9223372036854775806,"rule":"HostRegexp(`^.+$`)","ruleSyntax":"v3","service":"noop@internal"}},"serversTransports":{"default":{"insecureSkipVerify":true,"maxIdleConnsPerHost":200}},"services":{"api":{},"dashboard":{},"noop":{}}},"tcp":{"serversTransports":{"default":{"dialKeepAlive":"15s","dialTimeout":"30s"}}},"tls":{},"udp":{}} providerName=internal DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"services":{"cockpit":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://redacted:9090"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=file INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *docker.Provider DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *docker.Provider provider configuration config={"defaultRule":"Host(`{{ normalize .Name }}`)","endpoint":"unix:///var/run/docker.sock","watch":true} INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.ChallengeTLSALPN DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.ChallengeTLSALPN provider configuration config={} INF github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:202 > Starting provider *acme.Provider DBG github.com/traefik/traefik/v3/pkg/provider/aggregator/aggregator.go:203 > *acme.Provider provider configuration config={"HTTPChallengeProvider":{},"ResolverName":"porkbun","TLSChallengeProvider":{},"caServer":"https://acme-v02.api.letsencrypt.org/directory","certificatesDuration":2160,"dnsChallenge":{"provider":"porkbun","resolvers":["1.1.1.1:53","8.8.8.8:53"]},"email":"admin@domain.tld","keyType":"EC256","storage":"acme.json","store":{}} DBG github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:221 > Attempt to renew certificates "720h0m0s" before expiry and check every "24h0m0s" acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=porkbun.acme INF github.com/traefik/traefik/v3/pkg/provider/acme/provider.go:859 > Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=porkbun.acme DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{},"tcp":{},"tls":{},"udp":{}} providerName=porkbun.acme DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) *.domain.tld,domain.tld DBG github.com/traefik/traefik/v3/pkg/provider/docker/pdocker.go:90 > Provider connection established with docker 5.2.2 (API 1.41) providerName=docker DBG github.com/traefik/traefik/v3/pkg/provider/docker/config.go:184 > Filtering disabled container container=coturn-netbird-9c86101ecaf97e9a01101768057e46c043841f2e1d6b0e29674490b1766056a4 providerName=docker DBG github.com/traefik/traefik/v3/pkg/server/configurationwatcher.go:227 > Configuration received config={"http":{"middlewares":{"sslheader":{"headers":{"customRequestHeaders":{"X-Forwarded-Proto":"https"}}},"traefik-auth":{"basicAuth":{"users":["admin:$apr1$kRTh8V.I$zEM0xhyt6GBpLRPVoIzt30"]}},"traefik-https-redirect":{"redirectScheme":{"scheme":"https"}}},"routers":{"cockpit":{"entryPoints":["websecure"],"priority":1000,"rule":"Host(`cockpit.domain.tld`)","service":"cockpit@file","tls":{"certResolver":"porkbun"}},"keycloak":{"entryPoints":["websecure"],"rule":"Host(`auth.domain.tld`)","service":"cloud-network-keycloak","tls":{}},"netbird-api":{"rule":"Host(`netbird.domain.tld`) \u0026\u0026 PathPrefix(`/api`)","service":"netbird-api"},"netbird-dashboard":{"rule":"Host(`netbird.domain.tld`)","service":"netbird-dashboard"},"netbird-management":{"rule":"Host(`netbird.domain.tld`) \u0026\u0026 PathPrefix(`/management.ManagementService/`)","service":"netbird-management"},"netbird-signal":{"rule":"Host(`netbird.domain.tld`) \u0026\u0026 PathPrefix(`/signalexchange.SignalExchange/`)","service":"netbird-signal"},"traefik":{"entryPoints":["web"],"middlewares":["traefik-https-redirect"],"rule":"Host(`traefik-dashboard.domain.tld`)","service":"traefik-traefik"},"traefik-secure":{"entryPoints":["websecure"],"middlewares":["traefik-auth"],"rule":"Host(`traefik-dashboard.domain.tld`)","service":"api@internal","tls":{"certResolver":"porkbun","domains":[{"main":"domain.tld","sans":["*.domain.tld"]}]}}},"services":{"cloud-network-keycloak":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.89.3.3:8080"}]}},"netbird-api":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.89.2.4:443"}]}},"netbird-dashboard":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.89.2.2:80"}]}},"netbird-management":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"h2c://10.89.2.4:443"}]}},"netbird-signal":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"h2c://10.89.2.3:80"}]}},"traefik-traefik":{"loadBalancer":{"passHostHeader":true,"responseForwarding":{"flushInterval":"100ms"},"servers":[{"url":"http://10.89.3.7:80"}]}}}},"tcp":{},"tls":{},"udp":{}} providerName=docker DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery DBG github.com/traefik/traefik/v3/pkg/server/aggregator.go:51 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["web","websecure"] routerName=netbird-api DBG github.com/traefik/traefik/v3/pkg/server/aggregator.go:51 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["web","websecure"] routerName=netbird-management DBG github.com/traefik/traefik/v3/pkg/server/aggregator.go:51 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["web","websecure"] routerName=netbird-signal DBG github.com/traefik/traefik/v3/pkg/server/aggregator.go:51 > No entryPoint defined for this router, using the default one(s) instead entryPointName=["web","websecure"] routerName=netbird-dashboard DBG github.com/traefik/traefik/v3/pkg/tls/certificate.go:132 > Adding certificate for domain(s) *.domain.tld,domain.tld DBG github.com/traefik/traefik/v3/pkg/tls/tlsmanager.go:321 > No default certificate, fallback to the internal generated certificate tlsStoreName=default DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=traefik-https-redirect@docker middlewareType=RedirectScheme routerName=traefik@docker DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https entryPointName=web middlewareName=traefik-https-redirect@docker middlewareType=RedirectScheme routerName=traefik@docker DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:29 > Creating middleware entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal DBG github.com/traefik/traefik/v3/pkg/middlewares/redirect/redirect_scheme.go:30 > Setting up redirection to https 443 entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:318 > Creating load-balancer entryPointName=web routerName=netbird-api@docker serviceName=netbird-api@docker DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:355 > Creating server entryPointName=web routerName=netbird-api@docker serverName=573a1fd9790d212e serviceName=netbird-api@docker target=http://10.89.2.4:443 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:318 > Creating load-balancer entryPointName=web routerName=netbird-signal@docker serviceName=netbird-signal@docker DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:355 > Creating server entryPointName=web routerName=netbird-signal@docker serverName=b802a5f2e429e875 serviceName=netbird-signal@docker target=h2c://10.89.2.3:80 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:318 > Creating load-balancer entryPointName=web routerName=netbird-dashboard@docker serviceName=netbird-dashboard@docker DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:355 > Creating server entryPointName=web routerName=netbird-dashboard@docker serverName=a7c19bef807bb55f serviceName=netbird-dashboard@docker target=http://10.89.2.2:80 DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:318 > Creating load-balancer entryPointName=web routerName=netbird-management@docker serviceName=netbird-management@docker DBG github.com/traefik/traefik/v3/pkg/server/service/service.go:355 > Creating server entryPointName=web routerName=netbird-management@docker serverName=3d644fa6d1781c2f serviceName=netbird-management@docker target=h2c://10.89.2.4:443 DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery DBG github.com/traefik/traefik/v3/pkg/middlewares/recovery/recovery.go:25 > Creating middleware entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery DBG github.com/traefik/traefik/v3/pkg/server/router/tcp/manager.go:237 > Adding route for auth.domain.tld with TLS options default entryPointName=websecure ``` **Screenshots of Traefik Dashboard** ***Routers*** ![routers](https://github.com/user-attachments/assets/3a5e4235-b51b-461d-9a70-416e2eb20288) ***Services*** ![services](https://github.com/user-attachments/assets/0c4db002-8ad3-4e25-bf47-5c0eaa87f6df) **Additional context** - Tried advice from this [old Reddit post](https://www.reddit.com/r/selfhosted/comments/xpju6p/comment/iq7myvo/) for `Traefik v2`. - Configure.sh results in this entry in the compose file: ``` # Endpoints - NETBIRD_MGMT_API_ENDPOINT=https://netbird.domain.tld:33073 - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://netbird.domain.tld:33073 ``` Opening port `33073` and mapping it to Traefik did not resolve the issue. - I also tried: ``` # Endpoints - NETBIRD_MGMT_API_ENDPOINT=https://netbird.domain.tld:443 - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://netbird.domain.tld:443 ``` - `management.json` also results in a lot of entries such as `rel://netbird.domain.tld:33080`. I tried too many combinations to remember but nothing worked.
saavagebueno added the triage-needed label 2025-11-20 05:31:19 -05:00
Author
Owner

@epicteammate commented on GitHub (Dec 6, 2024):

Attempted to stand this up today and encountered the same issue but with Authentik.

@epicteammate commented on GitHub (Dec 6, 2024): Attempted to stand this up today and encountered the same issue but with Authentik.
Author
Owner

@den5o commented on GitHub (Dec 15, 2024):

I neglected this for days and came back to it just now. It turns out the template for traefik is missing one crucial label in the compose file, that being:
- traefik.http.routers.netbird-{service}.tls=true

After enabling it on all 3 services dashboard, management, and signal the dashboard loads and redirects correctly to keycloak for sign in. Glaring omission, simple fix. Please update template.

Although I am facing a new issue with status code 401 Error: Invalid Token, it is not related to Traefik so this issue can be closed.

@den5o commented on GitHub (Dec 15, 2024): I neglected this for days and came back to it just now. It turns out the template for traefik is missing one crucial label in the compose file, that being: `- traefik.http.routers.netbird-{service}.tls=true` After enabling it on all 3 services `dashboard`, `management`, and `signal` the dashboard loads and redirects correctly to keycloak for sign in. Glaring omission, simple fix. Please update template. Although I am facing a new issue with status code 401 Error: Invalid Token, it is not related to Traefik so this issue can be closed.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1476