[GH-ISSUE #5519] NetBird Management Server 0.66.0 fails to connect to external PostgreSQL database, attempting to use Unix socket /tmp/.s.PGSQL.15432 #11663

Closed
opened 2026-08-05 01:30:20 -04:00 by saavagebueno · 1 comment
Owner

Originally created by @folarinmartins on GitHub (Mar 5, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5519

Describe the problem

NetBird Management Server 0.66.0 fails to connect to external PostgreSQL database, attempting to use Unix socket /tmp/.s.PGSQL.15432 instead of the configured TCP connection. Despite setting correct environment variables and JSON configuration, the server ignores the TCP host/port and defaults to Unix socket connection, resulting in fatal error on startup.

To Reproduce

Steps to reproduce the behavior:

  1. Deploy NetBird Management Server 0.66.0 in Docker Swarm

  2. Configure external PostgreSQL 16 database on custom port (15432)

  3. Set environment variables:

    NETBIRD_STORE_CONFIG_ENGINE=postgres
    NETBIRD_STORE_ENGINE_POSTGRES_DSN=host=192.168.1.100 user=netbird password=SecurePass123 dbname=netbird port=15432 sslmode=disable
    NB_ACTIVITY_EVENT_STORE_ENGINE=postgres
    NB_ACTIVITY_EVENT_POSTGRES_DSN=host=192.168.1.100 user=netbird password=SecurePass123 dbname=netbird port=15432 sslmode=disable
    
  4. Configure management.json with Store section:

    {
      "Store": {
        "Engine": "postgres"
      }
    }
    
  5. Start the management server

  6. Observe fatal error attempting Unix socket connection

Expected behavior

NetBird Management Server should connect to PostgreSQL via TCP using the host and port specified in NETBIRD_STORE_ENGINE_POSTGRES_DSN environment variable.

Are you using NetBird Cloud?

No, self-hosting NetBird's control plane.

NetBird version

Management Server: 0.66.0
Deployment: Docker Swarm (official image netbirdio/management:0.66.0)

Is any other VPN software installed?

No other VPN software is installed on the host.

Debug output

Management server logs showing the error:

2026-03-06T01:14:59.572Z INFO management/cmd/management.go:320: running with the embedded IdP: https://vpn.example.com/oauth2
2026-03-06T01:14:59.572Z INFO management/cmd/management.go:323: Relay addresses: [rels://vpn.example.com:443]
2026-03-06T01:14:59.574Z INFO management/server/store/store.go:354: using Postgres store engine
2026-03-06T01:14:59.577Z FATL management/internals/server/boot.go:64: failed to create store: failed to connect to `host=/tmp user=netbird database=netbird`: dial error (dial unix /tmp/.s.PGSQL.15432: connect: no such file or directory)

Note: The log shows "using Postgres store engine" but then attempts connection to /tmp Unix socket instead of the configured TCP host.

Configuration Details

Docker Compose Configuration:

services:
  management:
    image: netbirdio/management:0.66.0
    command:
      - "--port=80"
      - "--log-file=console"
      - "--log-level=info"
      - "--disable-anonymous-metrics=true"
      - "--single-account-mode-domain=netbird.selfhosted"
      - "--idp-sign-key-refresh-enabled"
    environment:
      - NETBIRD_STORE_CONFIG_ENGINE=postgres
      - NETBIRD_STORE_ENGINE_POSTGRES_DSN=host=192.168.1.100 user=netbird password=SecurePass123 dbname=netbird port=15432 sslmode=disable
      - NB_ACTIVITY_EVENT_STORE_ENGINE=postgres
      - NB_ACTIVITY_EVENT_POSTGRES_DSN=host=192.168.1.100 user=netbird password=SecurePass123 dbname=netbird port=15432 sslmode=disable
    configs:
      - source: netbird-management-config
        target: /etc/netbird/management.json
    volumes:
      - netbird-mgmt-data:/var/lib/netbird

management.json:

{
    "Stuns": [
        {
            "Proto": "udp",
            "URI": "stun:vpn.example.com:3478"
        }
    ],
    "Relay": {
        "Addresses": ["rels://vpn.example.com:443"],
        "CredentialsTTL": "24h",
        "Secret": "AEM6CU3RJDa/LwK+jqUqYh0F8Dsr5xRSCOhet6ijp5c"
    },
    "Signal": {
        "Proto": "https",
        "URI": "vpn.example.com:443"
    },
    "Datadir": "/var/lib/netbird",
    "DataStoreEncryptionKey": "PJ4b/j757v/vfuo+Lvy5vJiB+WmYh7l0MXQUtcyKioM=",
    "Store": {
        "Engine": "postgres"
    },
    "EmbeddedIdP": {
        "Enabled": true,
        "Issuer": "https://vpn.example.com/oauth2",
        "DashboardRedirectURIs": [
            "https://vpn.example.com/nb-auth",
            "https://vpn.example.com/nb-silent-auth"
        ]
    }
}

Additional context

  • PostgreSQL 16 is running and accessible on 192.168.1.100:15432
  • Connection verified working with psql using identical connection parameters
  • Database netbird exists with proper schema
  • User netbird has all necessary permissions
  • The management server successfully reads other configuration from management.json (Relay, Signal, IdP settings)
  • Event store correctly defaults to SQLite (expected behavior)
  • Tested with both URL-encoded and non-encoded passwords in DSN - same result
  • Tested with and without Store.Config section in JSON - same result
  • The error message shows the DSN is being completely ignored and defaults to Unix socket

Troubleshooting attempts:

  1. Verified PostgreSQL connectivity from container host
  2. Tested multiple DSN formats (key=value pairs, postgresql:// URI)
  3. Tried with/without password URL encoding
  4. Verified environment variables are set in container
  5. Tested with different combinations of JSON config and env vars
  6. Checked PostgreSQL logs - no connection attempts from NetBird
  7. Verified no firewall blocking port 15432

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (N/A - management server issue)
  • Checked for newer NetBird versions (0.66.0 is current stable)
  • Searched for similar issues on GitHub (no exact matches found)
  • Restarted the NetBird management server (multiple times)
  • Disabled other VPN software (none installed)
  • Checked firewall settings (PostgreSQL port accessible)

Questions:

  1. Is there a specific order or format required for the DSN parameters?
  2. Should the Store.Engine be set in JSON, environment variable, or both?
  3. Why does the log show "using Postgres store engine" but then attempt Unix socket?
  4. Is there additional debug logging that can show how the DSN is being parsed?
  5. Are there known issues with PostgreSQL on non-standard ports?
Originally created by @folarinmartins on GitHub (Mar 5, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5519 **Describe the problem** NetBird Management Server 0.66.0 fails to connect to external PostgreSQL database, attempting to use Unix socket `/tmp/.s.PGSQL.15432` instead of the configured TCP connection. Despite setting correct environment variables and JSON configuration, the server ignores the TCP host/port and defaults to Unix socket connection, resulting in fatal error on startup. **To Reproduce** Steps to reproduce the behavior: 1. Deploy NetBird Management Server 0.66.0 in Docker Swarm 2. Configure external PostgreSQL 16 database on custom port (15432) 3. Set environment variables: ``` NETBIRD_STORE_CONFIG_ENGINE=postgres NETBIRD_STORE_ENGINE_POSTGRES_DSN=host=192.168.1.100 user=netbird password=SecurePass123 dbname=netbird port=15432 sslmode=disable NB_ACTIVITY_EVENT_STORE_ENGINE=postgres NB_ACTIVITY_EVENT_POSTGRES_DSN=host=192.168.1.100 user=netbird password=SecurePass123 dbname=netbird port=15432 sslmode=disable ``` 4. Configure management.json with Store section: ```json { "Store": { "Engine": "postgres" } } ``` 5. Start the management server 6. Observe fatal error attempting Unix socket connection **Expected behavior** NetBird Management Server should connect to PostgreSQL via TCP using the host and port specified in `NETBIRD_STORE_ENGINE_POSTGRES_DSN` environment variable. **Are you using NetBird Cloud?** No, self-hosting NetBird's control plane. **NetBird version** Management Server: `0.66.0` Deployment: Docker Swarm (official image `netbirdio/management:0.66.0`) **Is any other VPN software installed?** No other VPN software is installed on the host. **Debug output** Management server logs showing the error: ``` 2026-03-06T01:14:59.572Z INFO management/cmd/management.go:320: running with the embedded IdP: https://vpn.example.com/oauth2 2026-03-06T01:14:59.572Z INFO management/cmd/management.go:323: Relay addresses: [rels://vpn.example.com:443] 2026-03-06T01:14:59.574Z INFO management/server/store/store.go:354: using Postgres store engine 2026-03-06T01:14:59.577Z FATL management/internals/server/boot.go:64: failed to create store: failed to connect to `host=/tmp user=netbird database=netbird`: dial error (dial unix /tmp/.s.PGSQL.15432: connect: no such file or directory) ``` Note: The log shows "using Postgres store engine" but then attempts connection to `/tmp` Unix socket instead of the configured TCP host. **Configuration Details** **Docker Compose Configuration:** ```yaml services: management: image: netbirdio/management:0.66.0 command: - "--port=80" - "--log-file=console" - "--log-level=info" - "--disable-anonymous-metrics=true" - "--single-account-mode-domain=netbird.selfhosted" - "--idp-sign-key-refresh-enabled" environment: - NETBIRD_STORE_CONFIG_ENGINE=postgres - NETBIRD_STORE_ENGINE_POSTGRES_DSN=host=192.168.1.100 user=netbird password=SecurePass123 dbname=netbird port=15432 sslmode=disable - NB_ACTIVITY_EVENT_STORE_ENGINE=postgres - NB_ACTIVITY_EVENT_POSTGRES_DSN=host=192.168.1.100 user=netbird password=SecurePass123 dbname=netbird port=15432 sslmode=disable configs: - source: netbird-management-config target: /etc/netbird/management.json volumes: - netbird-mgmt-data:/var/lib/netbird ``` **management.json:** ```json { "Stuns": [ { "Proto": "udp", "URI": "stun:vpn.example.com:3478" } ], "Relay": { "Addresses": ["rels://vpn.example.com:443"], "CredentialsTTL": "24h", "Secret": "AEM6CU3RJDa/LwK+jqUqYh0F8Dsr5xRSCOhet6ijp5c" }, "Signal": { "Proto": "https", "URI": "vpn.example.com:443" }, "Datadir": "/var/lib/netbird", "DataStoreEncryptionKey": "PJ4b/j757v/vfuo+Lvy5vJiB+WmYh7l0MXQUtcyKioM=", "Store": { "Engine": "postgres" }, "EmbeddedIdP": { "Enabled": true, "Issuer": "https://vpn.example.com/oauth2", "DashboardRedirectURIs": [ "https://vpn.example.com/nb-auth", "https://vpn.example.com/nb-silent-auth" ] } } ``` **Additional context** - PostgreSQL 16 is running and accessible on `192.168.1.100:15432` - Connection verified working with `psql` using identical connection parameters - Database `netbird` exists with proper schema - User `netbird` has all necessary permissions - The management server successfully reads other configuration from `management.json` (Relay, Signal, IdP settings) - Event store correctly defaults to SQLite (expected behavior) - Tested with both URL-encoded and non-encoded passwords in DSN - same result - Tested with and without `Store.Config` section in JSON - same result - The error message shows the DSN is being completely ignored and defaults to Unix socket **Troubleshooting attempts:** 1. ✅ Verified PostgreSQL connectivity from container host 2. ✅ Tested multiple DSN formats (key=value pairs, postgresql:// URI) 3. ✅ Tried with/without password URL encoding 4. ✅ Verified environment variables are set in container 5. ✅ Tested with different combinations of JSON config and env vars 6. ✅ Checked PostgreSQL logs - no connection attempts from NetBird 7. ✅ Verified no firewall blocking port 15432 **Have you tried these troubleshooting steps?** - [x] Reviewed client troubleshooting (N/A - management server issue) - [x] Checked for newer NetBird versions (0.66.0 is current stable) - [x] Searched for similar issues on GitHub (no exact matches found) - [x] Restarted the NetBird management server (multiple times) - [x] Disabled other VPN software (none installed) - [x] Checked firewall settings (PostgreSQL port accessible) **Questions:** 1. Is there a specific order or format required for the DSN parameters? 2. Should the `Store.Engine` be set in JSON, environment variable, or both? 3. Why does the log show "using Postgres store engine" but then attempt Unix socket? 4. Is there additional debug logging that can show how the DSN is being parsed? 5. Are there known issues with PostgreSQL on non-standard ports?
saavagebueno added the triage-needed label 2026-08-05 01:30:20 -04:00
Author
Owner

@folarinmartins commented on GitHub (Mar 7, 2026):

[RESOLVED] Issue fixed: A combination of Docker quoting, Env Var prefixes, and Swarm Networking

I managed to completely resolve this issue. It turned out to be a cascading failure caused by three separate configuration traps between NetBird, Docker Compose, and Docker Swarm networking.

For anyone else who runs into the dial unix /tmp/.s.PGSQL.15432: connect: no such file or directory error or a subsequent connection timed out error when using PostgreSQL, here is exactly what is happening and how to fix it.

The Root Causes

  1. The /tmp Socket Fallback (Docker Quotes & Prefixes):
    NetBird’s Go PostgreSQL driver falls back to a local unix socket (/tmp) when it receives an invalid or empty connection string. This happened for two reasons:
  • Bypassing the Entrypoint: Because I was overriding the default container command: [...] in my compose file, the configure.sh wrapper script was bypassed. This meant NETBIRD_ prefixed setup variables were ignored. I had to use the NB_ runtime prefixes directly.
  • Literal Single Quotes: In my docker-compose.yml, I wrapped the DSN in single quotes (- NB_STORE_ENGINE_POSTGRES_DSN='host=...'). Docker Compose passes these literal quote characters into the container, which completely breaks the Go DSN parser, resulting in an empty connection string and triggering the /tmp fallback.
  • JSON Config Limit: NetBird does not currently support reading the PostgreSQL DSN parameters from a nested "Config" block inside management.json. It must be passed via environment variables.
  1. The connection timed out Error (Hairpin NAT in Swarm):
    Once the DSN was parsed correctly, the container tried to reach the database via the host's public IP (192.168.1.33). Because the container is inside an isolated Docker virtual network, routing out to the public internet and back into the same host/swarm is blocked by default firewall rules (Hairpin NAT/UFW).

The Complete Solution

To fix this, I stripped the quotes, switched to the NB_ prefixes, and put both services on a shared internal Docker network so they can communicate via DNS service names instead of public IPs.

**1. The Corrected management.json**
Remove any nested database config parameters. Just tell it to use Postgres.

{
    "StoreConfig": {
        "Engine": "postgres"
    },
    "Datadir": "/var/lib/netbird",
    "DataStoreEncryptionKey": "YOUR_BASE64_KEY="
    // ... rest of your config (Signal, Relay, Stuns, etc.)
}

**2. The Corrected docker-compose.yml**
Notice: No single quotes around the DSN variables, using NB_ prefixes, and pointing the DSN host to the internal Docker service name (e.g., postgres-db).

version: "3.8"

networks:
  netbird-net:
    driver: overlay # Or 'bridge' if not using Swarm

services:
  # NetBird Management Server
  management:
    image: network-core/netbird-management:0.66.0
    command: [
      "--port", "80",
      "--log-file", "console",
      "--log-level", "info",
      "--disable-anonymous-metrics=true",
      "--single-account-mode-domain=netbird.selfhosted",
      "--idp-sign-key-refresh-enabled"
    ]
    environment:
      - NB_PORT=80
      - NB_LOG_FILE=console
      - NB_LOG_LEVEL=info
      
      # --- Database Configuration ---
      # Crucial: DO NOT wrap the DSN in single or double quotes
      # Crucial: Use the internal docker service name as the host (e.g., postgres-db)
      - NB_STORE_CONFIG_ENGINE=postgres
      - NB_STORE_ENGINE_POSTGRES_DSN=host=postgres-db port=15432 user=netbird password=YourPassword dbname=netbird sslmode=disable
      
      - NB_ACTIVITY_EVENT_STORE_ENGINE=postgres
      - NB_ACTIVITY_EVENT_POSTGRES_DSN=host=postgres-db port=15432 user=netbird password=YourPassword dbname=netbird sslmode=disable
      
    networks:
      - netbird-net
    configs:
      - source: netbird-management-config
        target: /etc/netbird/management.json
    volumes:
      - netbird-mgmt-data:/var/lib/netbird

  # Ensure your database is on the same network
  postgres-db:
    image: postgres:16
    networks:
      - netbird-net
    # ... other db settings

After deploying these changes, the management server successfully connected via TCP, ran its auto-migrations, and the control plane is now fully operational. Closing this issue!

<!-- gh-comment-id:4016006547 --> @folarinmartins commented on GitHub (Mar 7, 2026): ### **[RESOLVED] Issue fixed: A combination of Docker quoting, Env Var prefixes, and Swarm Networking** I managed to completely resolve this issue. It turned out to be a cascading failure caused by three separate configuration traps between NetBird, Docker Compose, and Docker Swarm networking. For anyone else who runs into the `dial unix /tmp/.s.PGSQL.15432: connect: no such file or directory` error or a subsequent `connection timed out` error when using PostgreSQL, here is exactly what is happening and how to fix it. #### **The Root Causes** 1. **The `/tmp` Socket Fallback (Docker Quotes & Prefixes):** NetBird’s Go PostgreSQL driver falls back to a local unix socket (`/tmp`) when it receives an invalid or empty connection string. This happened for two reasons: * **Bypassing the Entrypoint:** Because I was overriding the default container `command: [...]` in my compose file, the `configure.sh` wrapper script was bypassed. This meant `NETBIRD_` prefixed setup variables were ignored. I had to use the `NB_` runtime prefixes directly. * **Literal Single Quotes:** In my `docker-compose.yml`, I wrapped the DSN in single quotes (`- NB_STORE_ENGINE_POSTGRES_DSN='host=...'`). Docker Compose passes these literal quote characters into the container, which completely breaks the Go DSN parser, resulting in an empty connection string and triggering the `/tmp` fallback. * **JSON Config Limit:** NetBird does not currently support reading the PostgreSQL DSN parameters from a nested `"Config"` block inside `management.json`. It *must* be passed via environment variables. 2. **The `connection timed out` Error (Hairpin NAT in Swarm):** Once the DSN was parsed correctly, the container tried to reach the database via the host's public IP (`192.168.1.33`). Because the container is inside an isolated Docker virtual network, routing out to the public internet and back into the same host/swarm is blocked by default firewall rules (Hairpin NAT/UFW). #### **The Complete Solution** To fix this, I stripped the quotes, switched to the `NB_` prefixes, and put both services on a shared internal Docker network so they can communicate via DNS service names instead of public IPs. **1. The Corrected `management.json**` Remove any nested database config parameters. Just tell it to use Postgres. ```json { "StoreConfig": { "Engine": "postgres" }, "Datadir": "/var/lib/netbird", "DataStoreEncryptionKey": "YOUR_BASE64_KEY=" // ... rest of your config (Signal, Relay, Stuns, etc.) } ``` **2. The Corrected `docker-compose.yml**` *Notice: No single quotes around the DSN variables, using `NB_` prefixes, and pointing the DSN host to the internal Docker service name (e.g., `postgres-db`).* ```yaml version: "3.8" networks: netbird-net: driver: overlay # Or 'bridge' if not using Swarm services: # NetBird Management Server management: image: network-core/netbird-management:0.66.0 command: [ "--port", "80", "--log-file", "console", "--log-level", "info", "--disable-anonymous-metrics=true", "--single-account-mode-domain=netbird.selfhosted", "--idp-sign-key-refresh-enabled" ] environment: - NB_PORT=80 - NB_LOG_FILE=console - NB_LOG_LEVEL=info # --- Database Configuration --- # Crucial: DO NOT wrap the DSN in single or double quotes # Crucial: Use the internal docker service name as the host (e.g., postgres-db) - NB_STORE_CONFIG_ENGINE=postgres - NB_STORE_ENGINE_POSTGRES_DSN=host=postgres-db port=15432 user=netbird password=YourPassword dbname=netbird sslmode=disable - NB_ACTIVITY_EVENT_STORE_ENGINE=postgres - NB_ACTIVITY_EVENT_POSTGRES_DSN=host=postgres-db port=15432 user=netbird password=YourPassword dbname=netbird sslmode=disable networks: - netbird-net configs: - source: netbird-management-config target: /etc/netbird/management.json volumes: - netbird-mgmt-data:/var/lib/netbird # Ensure your database is on the same network postgres-db: image: postgres:16 networks: - netbird-net # ... other db settings ``` After deploying these changes, the management server successfully connected via TCP, ran its auto-migrations, and the control plane is now fully operational. Closing this issue!
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11663