[GH-ISSUE #1970] [SELF-HOSTED] Geographically dispersed peers get the exact same "Public IP" - And the "Public IP" is actually a Private IP on the 10.x.x.x subnet! #3891

Open
opened 2026-08-05 00:54:50 -04:00 by saavagebueno · 9 comments
Owner

Originally created by @pomology on GitHub (May 12, 2024).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/1970

The problem I'm having is that all my peers are getting the very same "Public IP" which is 10.89.1.53. The "Public IP" is actually a private IP, and every single peer gets exactly the same "Public IP" address! Needless to say, none of my peers can talk to each other, nor ping each other, etc, etc.

The peers are on widely geographically dispersed networks, and yet they all have the same problem.

Here is my Caddyfile, from NetBird's default configs:

{
  debug
	servers :80,:443 {
    protocols h1 h2c
  }
}

(security_headers) {
    header * {
     
        Strict-Transport-Security "max-age=3600; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "DENY"
        X-XSS-Protection "1; mode=block"
        -Server
        Referrer-Policy strict-origin-when-cross-origin
    }
}

:80, REDACTED.REDACTED.com:443 {
    import security_headers
    reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000
    reverse_proxy /api/* management:80
    reverse_proxy /management.ManagementService/* h2c://management:80
    reverse_proxy /* dashboard:80
}


And here's my compose.yml, again just built on NetBird's default:

version: "3"
services:
 # Caddy reverse proxy
  caddy:
    image: caddy:latest
    restart: unless-stopped
    networks:
      - netbird
    ports:
      - '443:443'
      - '80:80'
    volumes:
      - netbird_caddy_data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile

  #UI dashboard
  dashboard:
    image: netbirdio/dashboard:latest
    restart: unless-stopped
    networks:
      - netbird
    environment:
      # Endpoints
      - NETBIRD_MGMT_API_ENDPOINT=https://REDACTED.REDACTED.com:443
      - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://REDACTED.REDACTED.com:443
      # OIDC
      - AUTH_AUDIENCE=REDACTED
      - AUTH_CLIENT_ID=REDACTED
      - AUTH_CLIENT_SECRET=
      - AUTH_AUTHORITY=https://REDACTED.okta.com
      - USE_AUTH0=false
      - AUTH_SUPPORTED_SCOPES=openid profile email
      - AUTH_REDIRECT_URI=/auth
      - AUTH_SILENT_REDIRECT_URI=/silent-auth
      - NETBIRD_TOKEN_SOURCE=idToken

  # Signal
  signal:
    image: netbirdio/signal:latest
    restart: unless-stopped
    networks:
      - netbird
   

  # Management
  management:
    image: netbirdio/management:latest
    restart: unless-stopped
    networks:
      - netbird
    depends_on:
      - dashboard
    volumes:
      - netbird-mgmt:/var/lib/netbird
      - ./management.json:/etc/netbird/management.json
    command: [
      "--port", "80",
      "--log-file", "console",
      "--log-level", "info",
      "--disable-anonymous-metrics=false",
      "--single-account-mode-domain=REDACTED.REDACTED.com",
      "--dns-domain=netbird.selfhosted",
      "--idp-sign-key-refresh-enabled"
]

  # Coturn
  coturn:
    image: coturn/coturn:latest
    restart: unless-stopped
    domainname: REDACTED.REDACTED.com
    volumes:
      - ./turnserver.conf:/etc/turnserver.conf:ro
    network_mode: host
    command:
      - -c /etc/turnserver.conf

volumes:
  netbird-mgmt:
  netbird_caddy_data:

networks:
  netbird:

Originally created by @pomology on GitHub (May 12, 2024). Original GitHub issue: https://github.com/netbirdio/netbird/issues/1970 The problem I'm having is that all my peers are getting the very same "Public IP" which is 10.89.1.53. The "Public IP" is actually a private IP, and every single peer gets exactly the same "Public IP" address! Needless to say, none of my peers can talk to each other, nor ping each other, etc, etc. The peers are on widely geographically dispersed networks, and yet they all have the same problem. Here is my Caddyfile, from NetBird's default configs: ``` { debug servers :80,:443 { protocols h1 h2c } } (security_headers) { header * { Strict-Transport-Security "max-age=3600; includeSubDomains; preload" X-Content-Type-Options "nosniff" X-Frame-Options "DENY" X-XSS-Protection "1; mode=block" -Server Referrer-Policy strict-origin-when-cross-origin } } :80, REDACTED.REDACTED.com:443 { import security_headers reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000 reverse_proxy /api/* management:80 reverse_proxy /management.ManagementService/* h2c://management:80 reverse_proxy /* dashboard:80 } ``` And here's my compose.yml, again just built on NetBird's default: ``` version: "3" services: # Caddy reverse proxy caddy: image: caddy:latest restart: unless-stopped networks: - netbird ports: - '443:443' - '80:80' volumes: - netbird_caddy_data:/data - ./Caddyfile:/etc/caddy/Caddyfile #UI dashboard dashboard: image: netbirdio/dashboard:latest restart: unless-stopped networks: - netbird environment: # Endpoints - NETBIRD_MGMT_API_ENDPOINT=https://REDACTED.REDACTED.com:443 - NETBIRD_MGMT_GRPC_API_ENDPOINT=https://REDACTED.REDACTED.com:443 # OIDC - AUTH_AUDIENCE=REDACTED - AUTH_CLIENT_ID=REDACTED - AUTH_CLIENT_SECRET= - AUTH_AUTHORITY=https://REDACTED.okta.com - USE_AUTH0=false - AUTH_SUPPORTED_SCOPES=openid profile email - AUTH_REDIRECT_URI=/auth - AUTH_SILENT_REDIRECT_URI=/silent-auth - NETBIRD_TOKEN_SOURCE=idToken # Signal signal: image: netbirdio/signal:latest restart: unless-stopped networks: - netbird # Management management: image: netbirdio/management:latest restart: unless-stopped networks: - netbird depends_on: - dashboard volumes: - netbird-mgmt:/var/lib/netbird - ./management.json:/etc/netbird/management.json command: [ "--port", "80", "--log-file", "console", "--log-level", "info", "--disable-anonymous-metrics=false", "--single-account-mode-domain=REDACTED.REDACTED.com", "--dns-domain=netbird.selfhosted", "--idp-sign-key-refresh-enabled" ] # Coturn coturn: image: coturn/coturn:latest restart: unless-stopped domainname: REDACTED.REDACTED.com volumes: - ./turnserver.conf:/etc/turnserver.conf:ro network_mode: host command: - -c /etc/turnserver.conf volumes: netbird-mgmt: netbird_caddy_data: networks: netbird: ```
saavagebueno added the management-servicesecurityself-hosting labels 2026-08-05 00:54:50 -04:00
Author
Owner

@lixmal commented on GitHub (May 13, 2024):

If your setup is behind another proxy (cloudflare etc) you might need to accept X-Forwarded-For headers from there to display the real public IP. See https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#defaults

As for

Needless to say, none of my peers can talk to each other, nor ping each other, etc, etc.

Could you please provide a netbird status -dA from one of the peers? I suspect they are not connecting to each other

<!-- gh-comment-id:2107262678 --> @lixmal commented on GitHub (May 13, 2024): If your setup is behind another proxy (cloudflare etc) you might need to accept `X-Forwarded-For` headers from there to display the real public IP. See https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#defaults As for > Needless to say, none of my peers can talk to each other, nor ping each other, etc, etc. Could you please provide a `netbird status -dA` from one of the peers? I suspect they are not connecting to each other
Author
Owner

@pomology commented on GitHub (May 13, 2024):

Thanks! My server is running on AWS EC2, so not behind any other proxy. Just to be sure, I tried accepting X-Forwarded-For headers in my Caddyfile, but still got the same result....

Here's a netbird status -dA for one of the peers:

Peers detail:
 ip-172-45-16-167.connect.anon-IbY1H.domain:
  NetBird IP: 100.120.11.112
  Public key: REDACTED
  Status: Disconnected
  -- detail --
  Connection type: 
  Direct: false
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Last connection update: -
  Last WireGuard handshake: -
  Transfer status (received/sent) 0 B/0 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

OS: darwin/arm64
Daemon version: 0.27.5
CLI version: 0.27.5
Management: Connected to https://connect.anon-IbY1H.domain:443
Signal: Connected to https://connect.anon-IbY1H.domain:443
Relays: 
  [stun:connect.anon-IbY1H.domain:3478] is Unavailable, reason: dial: failed to listen: dial: dial udp: lookup connect.anon-IbY1H.domain: no such host
  [turn:connect.anon-IbY1H.domain:3478?transport=udp] is Unavailable, reason: create client: lookup connect.anon-IbY1H.domain: no such host
Nameservers: 
FQDN: rmacbook-pro.connect.anon-IbY1H.domain
NetBird IP: 100.120.67.54/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 0/1 Connected
<!-- gh-comment-id:2107462055 --> @pomology commented on GitHub (May 13, 2024): Thanks! My server is running on AWS EC2, so not behind any other proxy. Just to be sure, I tried accepting `X-Forwarded-For` headers in my Caddyfile, but still got the same result.... Here's a `netbird status -dA` for one of the peers: ``` Peers detail: ip-172-45-16-167.connect.anon-IbY1H.domain: NetBird IP: 100.120.11.112 Public key: REDACTED Status: Disconnected -- detail -- Connection type: Direct: false ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Last connection update: - Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B Quantum resistance: false Routes: - Latency: 0s OS: darwin/arm64 Daemon version: 0.27.5 CLI version: 0.27.5 Management: Connected to https://connect.anon-IbY1H.domain:443 Signal: Connected to https://connect.anon-IbY1H.domain:443 Relays: [stun:connect.anon-IbY1H.domain:3478] is Unavailable, reason: dial: failed to listen: dial: dial udp: lookup connect.anon-IbY1H.domain: no such host [turn:connect.anon-IbY1H.domain:3478?transport=udp] is Unavailable, reason: create client: lookup connect.anon-IbY1H.domain: no such host Nameservers: FQDN: rmacbook-pro.connect.anon-IbY1H.domain NetBird IP: 100.120.67.54/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 0/1 Connected ```
Author
Owner

@lixmal commented on GitHub (May 13, 2024):

Thanks! My server is running on AWS EC2, so not behind any other proxy. Just to be sure, I tried accepting X-Forwarded-For headers in my Caddyfile, but still got the same result....

Do you run a loadbalancer in front (ELB/ALB or some ingress controller)? That would have the same result

Here's a netbird status -dA for one of the peers:

It seems there is some issue with the stun/turn server connection
Do you happen to have the same domain for netbird internally as well as for management/signal etc?

<!-- gh-comment-id:2107564220 --> @lixmal commented on GitHub (May 13, 2024): > Thanks! My server is running on AWS EC2, so not behind any other proxy. Just to be sure, I tried accepting X-Forwarded-For headers in my Caddyfile, but still got the same result.... Do you run a loadbalancer in front (ELB/ALB or some ingress controller)? That would have the same result > Here's a netbird status -dA for one of the peers: It seems there is some issue with the stun/turn server connection Do you happen to have the same domain for netbird internally as well as for management/signal etc?
Author
Owner

@pomology commented on GitHub (May 13, 2024):

So it appears that this only occurs when using Podman. I just now tried rootfull docker instead and the issue goes away, I get correct external IP's that display normally and clients can ping each other. Is Netbird known to be incompatible with Podman? Any idea what's going on?

<!-- gh-comment-id:2108869481 --> @pomology commented on GitHub (May 13, 2024): So it appears that this only occurs when using Podman. I just now tried rootfull docker instead and the issue goes away, I get correct external IP's that display normally and clients can ping each other. Is Netbird known to be incompatible with Podman? Any idea what's going on?
Author
Owner

@pomology commented on GitHub (May 13, 2024):

For testing purposes, I just rebuilt the whole thing with rootless docker, and I get the same issue of an internal IP being shown where the External IP should be, and no communication between peers. It seems these configs simply don't work for operating in a rootless container.

Rootfull containers seem to work fine, but running rootfull is arguably very dangerous.

What do you guys recommend for running rootless? Any recommended configs? Thank you!

<!-- gh-comment-id:2109001313 --> @pomology commented on GitHub (May 13, 2024): For testing purposes, I just rebuilt the whole thing with rootless docker, and I get the same issue of an internal IP being shown where the External IP should be, and no communication between peers. It seems these configs simply don't work for operating in a rootless container. Rootfull containers seem to work fine, but running rootfull is arguably very dangerous. What do you guys recommend for running rootless? Any recommended configs? Thank you!
Author
Owner

@pomology commented on GitHub (May 13, 2024):

Per the below website, it appears this is expected behavior.

https://rootlesscontaine.rs/how-it-works/netns/incoming/

There are apparently ways to get around it, but quite complicated. I hope the devs can work toward as much rootless implementation as possible, to further improve security.

<!-- gh-comment-id:2109121228 --> @pomology commented on GitHub (May 13, 2024): Per the below website, it appears this is expected behavior. https://rootlesscontaine.rs/how-it-works/netns/incoming/ There are apparently ways to get around it, but quite complicated. I hope the devs can work toward as much rootless implementation as possible, to further improve security.
Author
Owner

@AIMI-SAYO commented on GitHub (Jan 24, 2026):

I managed to fix this! I was seeing the internal IP of my Nginx Proxy Manager container instead of the clients' real Public IPs.

The fix required two steps:

1. Update management.json Tell NetBird to trust your proxy's IP.

File: /etc/netbird/management.json

{
    // ... existing config ...
    "ReverseProxy": {
        "TrustedHTTPProxies": ["192.168.50.50/32"], 
        "TrustedHTTPProxiesCount": 1,
        "TrustedPeers": ["0.0.0.0/0"]
    }
}

(Replace 192.168.50.50 with your Proxy IP. The /32 CIDR is required).

2. Update Nginx Configuration Default Nginx configs often miss forwarding headers for gRPC traffic. In your Nginx/NPM config, add grpc_set_header to the gRPC location blocks:

# Add these headers to /signalexchange.SignalExchange/ AND /management.ManagementService/
location /signalexchange.SignalExchange/ {
    grpc_pass grpc://netbird-signal:10000;
    # ... other settings ...
    grpc_set_header X-Real-IP $remote_addr;
    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

After these changes, the real Public IP appeared immediately after just restarting the clients (netbird down && netbird up).

<!-- gh-comment-id:3794618507 --> @AIMI-SAYO commented on GitHub (Jan 24, 2026): **I managed to fix this!** I was seeing the internal IP of my Nginx Proxy Manager container instead of the clients' real Public IPs. The fix required two steps: **1. Update `management.json`** Tell NetBird to trust your proxy's IP. **File:** `/etc/netbird/management.json` { // ... existing config ... "ReverseProxy": { "TrustedHTTPProxies": ["192.168.50.50/32"], "TrustedHTTPProxiesCount": 1, "TrustedPeers": ["0.0.0.0/0"] } } > (Replace `192.168.50.50` with your Proxy IP. The `/32` CIDR is required). **2. Update Nginx Configuration** Default Nginx configs often miss forwarding headers for **gRPC** traffic. In your Nginx/NPM config, add `grpc_set_header` to the gRPC location blocks: # Add these headers to /signalexchange.SignalExchange/ AND /management.ManagementService/ location /signalexchange.SignalExchange/ { grpc_pass grpc://netbird-signal:10000; # ... other settings ... grpc_set_header X-Real-IP $remote_addr; grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } After these changes, the real Public IP appeared immediately after just restarting the clients (netbird down && netbird up).
Author
Owner

@Muska-Ami commented on GitHub (Jan 26, 2026):

I've tried set ReverseProxy.TrustedHTTPProxies to 0.0.0.0/0 and ::/0, but I only got the docker's network gateway IP for netbird.

I've configure the OpenResty config like this:

location <ENDPOINT> ~^ {
    # ...

    # Common proxy headers
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Scheme $scheme;
    proxy_set_header X-Forwarded-Host $host;
    grpc_set_header X-Real-IP $remote_addr;
    grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
<!-- gh-comment-id:3798469723 --> @Muska-Ami commented on GitHub (Jan 26, 2026): I've tried set `ReverseProxy.TrustedHTTPProxies` to `0.0.0.0/0` and `::/0`, but I only got the docker's network gateway IP for netbird. I've configure the OpenResty config like this: ``` location <ENDPOINT> ~^ { # ... # Common proxy headers proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Scheme $scheme; proxy_set_header X-Forwarded-Host $host; grpc_set_header X-Real-IP $remote_addr; grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } ```
Author
Owner

@Muska-Ami commented on GitHub (Jan 26, 2026):

When I remove ReverseProxy.TrustedHTTPProxies configuration, it works, but netbird doesn't resolve the X-Forwarded-For header correctly.

This is a example XFF header:

X-Forwarded-For: 1.1.1.1, 8.8.8.8
# X-Forwarded-For: <client>, <proxy1>, <proxy2>, ...

As the XFF header defintion, we should use the first IP as client IP (1.1.1.1), but netbird use the last one (8.8.8.8), finally it get the proxy's IP(LB or CDN) and treat it as client IP.

PS: I've removed X-Real-IP header and only X-Forwarded-For header present. So I'm sure this is a bug related to netbird.

<!-- gh-comment-id:3798500658 --> @Muska-Ami commented on GitHub (Jan 26, 2026): When I remove `ReverseProxy.TrustedHTTPProxies` configuration, it works, but netbird doesn't resolve the `X-Forwarded-For` header correctly. This is a example XFF header: ``` X-Forwarded-For: 1.1.1.1, 8.8.8.8 # X-Forwarded-For: <client>, <proxy1>, <proxy2>, ... ``` As the XFF header defintion, we should use the first IP as client IP (1.1.1.1), but netbird use the last one (8.8.8.8), finally it get the proxy's IP(LB or CDN) and treat it as client IP. PS: I've removed `X-Real-IP` header and only `X-Forwarded-For` header present. So I'm sure this is a bug related to netbird.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#3891