SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-06-10 17:31:53 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Route handling regression: HTTP traffic fails on LAN IPs while other protocols work #1423

New Issue
Open
opened 2025-11-20 05:30:06 -05:00 by saavagebueno · 11 comments
saavagebueno commented 2025-11-20 05:30:06 -05:00
Owner
Copy Link

Originally created by @cipherw0lf on GitHub (Nov 18, 2024).

Describe the problem

HTTP/HTTPS traffic fails when accessing a NetBird client using its LAN IP address, despite routes being correctly configured. This issue appeared after a recent NetBird update and has been reproduced in two independent environments.

  • NetBird client on LAN (192.168.20.23/24)
  • Route configured for entire subnet (192.168.20.0/24)
  • Two independent test environments with different firewalls

Current Behavior

  1. HTTP/HTTPS Traffic:

    • Fails/times out when accessing LAN IP (192.168.20.23)
    • Works perfectly when using NetBird IP (100.90.x.x)
    • Other HTTP resources on same subnet (e.g., 192.168.20.8) work fine

  2. Other Protocols:

    • SSH (tcp/22) to 192.168.20.23 works correctly
    • iperf3 (tcp/5201) to 192.168.20.23 works correctly (correct throughput)
    • Only HTTP/HTTPS traffic is affected

To Reproduce

Steps to reproduce the behavior:

  1. Configure NetBird client on LAN (192.168.20.23/24)
  2. Add route for 192.168.20.0/24 in NetBird dashboard
  3. From another NetBird client, attempt to access:
   # These fail
   curl http://192.168.20.23
   curl https://192.168.20.23
   curl http://192.168.20.23:81
   curl https://192.168.20.23:9443

   # These work
   ssh user@192.168.20.23
   curl http://100.90.x.x    # NetBird IP of same host
   curl http://192.168.20.100 (Different host on same LAN)

Expected behavior

  • Access to HTTP/HTTPS resources on the NetBird client should work using its LAN IP (192.168.20.23)
  • All traffic should route correctly as per configured routes
  • Behavior should match what was working for the past 4 months

Are you using NetBird Cloud?

No. Self hosted using getting-started-with-zitadel.sh script

NetBird version

Client node version

OS: darwin/amd64
Daemon version: 0.32.0
CLI version: 0.32.0
Management: Connected to https://netbird.anon-PDpeG.domain:443
Signal: Connected to https://netbird.anon-PDpeG.domain:443
Relays:
[stun:netbird.anon-PDpeG.domain:3478] is Available
[turn:netbird.anon-PDpeG.domain:3478?transport=udp] is Available
Nameservers:
[192.168.0.8:53, 192.168.0.9:53] for [anon-H9yIg.domain] is Available
[192.168.40.3:53, 192.168.40.7:53] for [anon-sVKwC.domain] is Available
[192.168.20.8:53, 192.168.30.4:53] for [anon-wHSfX.domain] is Available
FQDN: rm-mbp.netbird.selfhosted
NetBird IP: 100.90.44.172/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 8/9 Connected

Node with route host

OS: linux/amd64
Daemon version: 0.32.0
CLI version: 0.32.0
Management: Connected to https://netbird.anon-glhvW.domain:443
Signal: Connected to https://netbird.anon-glhvW.domain:443
Relays:
[stun:netbird.anon-glhvW.domain:3478] is Available
[turn:netbird.anon-glhvW.domain:3478?transport=udp] is Available
Nameservers:
FQDN: selby-ubuntu-docker.netbird.selfhosted
NetBird IP: 100.90.109.225/16
Interface type: Kernel
Quantum resistance: false
Routes: 192.168.20.0/24
Peers count: 8/9 Connected

Originally created by @cipherw0lf on GitHub (Nov 18, 2024). **Describe the problem** HTTP/HTTPS traffic fails when accessing a NetBird client using its LAN IP address, despite routes being correctly configured. This issue appeared after a recent NetBird update and has been reproduced in two independent environments. - NetBird client on LAN (192.168.20.23/24) - Route configured for entire subnet (192.168.20.0/24) - Two independent test environments with different firewalls **Current Behavior** 1. HTTP/HTTPS Traffic: - Fails/times out when accessing LAN IP (192.168.20.23) - Works perfectly when using NetBird IP (100.90.x.x) - Other HTTP resources on same subnet (e.g., 192.168.20.8) work fine 2. Other Protocols: - SSH (tcp/22) to 192.168.20.23 works correctly - iperf3 (tcp/5201) to 192.168.20.23 works correctly (correct throughput) - Only HTTP/HTTPS traffic is affected **To Reproduce** Steps to reproduce the behavior: 1. Configure NetBird client on LAN (192.168.20.23/24) 2. Add route for 192.168.20.0/24 in NetBird dashboard 3. From another NetBird client, attempt to access: ```bash # These fail curl http://192.168.20.23 curl https://192.168.20.23 curl http://192.168.20.23:81 curl https://192.168.20.23:9443 # These work ssh user@192.168.20.23 curl http://100.90.x.x # NetBird IP of same host curl http://192.168.20.100 (Different host on same LAN) ``` **Expected behavior** - Access to HTTP/HTTPS resources on the NetBird client should work using its LAN IP (192.168.20.23) - All traffic should route correctly as per configured routes - Behavior should match what was working for the past 4 months **Are you using NetBird Cloud?** No. Self hosted using `getting-started-with-zitadel.sh` script **NetBird version** #### Client node version OS: darwin/amd64 Daemon version: 0.32.0 CLI version: 0.32.0 Management: Connected to https://netbird.anon-PDpeG.domain:443 Signal: Connected to https://netbird.anon-PDpeG.domain:443 Relays: [stun:netbird.anon-PDpeG.domain:3478] is Available [turn:netbird.anon-PDpeG.domain:3478?transport=udp] is Available Nameservers: [192.168.0.8:53, 192.168.0.9:53] for [anon-H9yIg.domain] is Available [192.168.40.3:53, 192.168.40.7:53] for [anon-sVKwC.domain] is Available [192.168.20.8:53, 192.168.30.4:53] for [anon-wHSfX.domain] is Available FQDN: rm-mbp.netbird.selfhosted NetBird IP: 100.90.44.172/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 8/9 Connected #### Node with route host OS: linux/amd64 Daemon version: 0.32.0 CLI version: 0.32.0 Management: Connected to https://netbird.anon-glhvW.domain:443 Signal: Connected to https://netbird.anon-glhvW.domain:443 Relays: [stun:netbird.anon-glhvW.domain:3478] is Available [turn:netbird.anon-glhvW.domain:3478?transport=udp] is Available Nameservers: FQDN: selby-ubuntu-docker.netbird.selfhosted NetBird IP: 100.90.109.225/16 Interface type: Kernel Quantum resistance: false Routes: 192.168.20.0/24 Peers count: 8/9 Connected
saavagebueno added the waiting-feedbackroutestriage-needed labels 2025-11-20 05:30:06 -05:00
saavagebueno commented 2025-11-20 05:30:07 -05:00
Author
Owner
Copy Link

@cipherw0lf commented on GitHub (Nov 18, 2024):

I have narrowed the issue down to docker containers' exposed services. These netbird peers are installed on docker hosts.
However the containers' exposed ports are bound to all interfaces (0.0.0.0) and this has worked fine for 4 months since deployment of netbird.

@cipherw0lf commented on GitHub (Nov 18, 2024): I have narrowed the issue down to docker containers' exposed services. These netbird peers are installed on docker hosts. However the containers' exposed ports are bound to all interfaces (0.0.0.0) and this has worked fine for 4 months since deployment of netbird.
saavagebueno commented 2025-11-20 05:30:07 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Nov 18, 2024):

Hi @cipherw0lf, what does your access policy look like? if you have specific ports there you might need to add the listening port in the container, additionally to the exposed port

@lixmal commented on GitHub (Nov 18, 2024): Hi @cipherw0lf, what does your access policy look like? if you have specific ports there you might need to add the listening port in the container, additionally to the exposed port
saavagebueno commented 2025-11-20 05:30:08 -05:00
Author
Owner
Copy Link

@mikhail-markov commented on GitHub (Nov 19, 2024):

I have this problem too. My access policy is All-to-All allow. The problem is with access to docker container ports running on one of the gateway servers to the remote local network. The problem appeared after update to Netbird version 0.32.0 on my PC and this gateway server.

@mikhail-markov commented on GitHub (Nov 19, 2024): I have this problem too. My access policy is All-to-All allow. The problem is with access to docker container ports running on one of the gateway servers to the remote local network. The problem appeared after update to Netbird version 0.32.0 on my PC and this gateway server.
saavagebueno commented 2025-11-20 05:30:08 -05:00
Author
Owner
Copy Link

@Codixer commented on GitHub (Nov 21, 2024):

Same issue, updated all my clients to 0.32.0 and now I cannot access my servers over HTTP/HTTPS anymore, unless I use the Netbird IP adress. My firewall has an ALL-ALL policy, and a route to my 10.0.0.0/24 network.

@Codixer commented on GitHub (Nov 21, 2024): Same issue, updated all my clients to 0.32.0 and now I cannot access my servers over HTTP/HTTPS anymore, unless I use the Netbird IP adress. My firewall has an ALL-ALL policy, and a route to my 10.0.0.0/24 network.
saavagebueno commented 2025-11-20 05:30:08 -05:00
Author
Owner
Copy Link

@Codixer commented on GitHub (Nov 21, 2024):

Can additionally confirm SSH also works over the IP adress:
image

But any service that I use over web protocols do NOT work. Even if I use non-docker instances. (Proxmox for example)
image

Client:

OS: windows/amd64
Daemon version: 0.32.0
CLI version: 0.32.0
Management: Connected to https://vpn.stefanocoding.me:443/
Signal: Connected to https://vpn.stefanocoding.me:443
Relays:
  [stun:vpn.stefanocoding.me:3478] is Available
  [turn:vpn.stefanocoding.me:3478?transport=udp] is Available
  [rel://vpn.stefanocoding.me:8443] is Available
Nameservers:
  [100.77.68.136:53] for [home.stefanocoding.me, media.home.stefanocoding.me] is Available
  [1.1.1.1:53, 1.0.0.1:53] for [.] is Available
  [100.77.50.140:53] for [stefanocoding.me] is Available
  [100.77.50.140:53] for [hovedstadenrp.com] is Available
FQDN: win-5cg3353zq4.sc.local
NetBird IP: 100.77.79.252/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 5/10 Connected

Server:

OS: linux/amd64
Daemon version: 0.32.0
CLI version: 0.32.0
Management: Connected to https://vpn.stefanocoding.me:443
Signal: Connected to https://vpn.stefanocoding.me:443
Relays:
  [stun:vpn.stefanocoding.me:3478] is Available
  [turn:vpn.stefanocoding.me:3478?transport=udp] is Available
  [rel://vpn.stefanocoding.me:8443] is Available
Nameservers:
FQDN: home-server.sc.local
NetBird IP: 100.77.68.136/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 5/10 Connected

Route:
image
image
image

@Codixer commented on GitHub (Nov 21, 2024): Can additionally confirm SSH also works over the IP adress: ![image](https://github.com/user-attachments/assets/77d7320b-6a76-4c76-8ece-ebe65a5c374a) But any service that I use over web protocols do NOT work. Even if I use non-docker instances. (Proxmox for example) ![image](https://github.com/user-attachments/assets/c12a1b01-d8f2-4729-b382-78b719475d81) Client: ``` OS: windows/amd64 Daemon version: 0.32.0 CLI version: 0.32.0 Management: Connected to https://vpn.stefanocoding.me:443/ Signal: Connected to https://vpn.stefanocoding.me:443 Relays: [stun:vpn.stefanocoding.me:3478] is Available [turn:vpn.stefanocoding.me:3478?transport=udp] is Available [rel://vpn.stefanocoding.me:8443] is Available Nameservers: [100.77.68.136:53] for [home.stefanocoding.me, media.home.stefanocoding.me] is Available [1.1.1.1:53, 1.0.0.1:53] for [.] is Available [100.77.50.140:53] for [stefanocoding.me] is Available [100.77.50.140:53] for [hovedstadenrp.com] is Available FQDN: win-5cg3353zq4.sc.local NetBird IP: 100.77.79.252/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 5/10 Connected ``` Server: ``` OS: linux/amd64 Daemon version: 0.32.0 CLI version: 0.32.0 Management: Connected to https://vpn.stefanocoding.me:443 Signal: Connected to https://vpn.stefanocoding.me:443 Relays: [stun:vpn.stefanocoding.me:3478] is Available [turn:vpn.stefanocoding.me:3478?transport=udp] is Available [rel://vpn.stefanocoding.me:8443] is Available Nameservers: FQDN: home-server.sc.local NetBird IP: 100.77.68.136/16 Interface type: Kernel Quantum resistance: false Routes: - Peers count: 5/10 Connected ``` Route: ![image](https://github.com/user-attachments/assets/e7f784e6-ae51-4efa-bd3a-35f2d05b4348) ![image](https://github.com/user-attachments/assets/9f9793ad-efba-44fa-856d-4e7cbfc1a498) ![image](https://github.com/user-attachments/assets/635baab7-fef5-4988-aedc-9811a4e7925f)
saavagebueno commented 2025-11-20 05:30:08 -05:00
Author
Owner
Copy Link

@cipherw0lf commented on GitHub (Nov 22, 2024):

Hi @cipherw0lf, what does your access policy look like? if you have specific ports there you might need to add the listening port in the container, additionally to the exposed port

I have no access policy rules:
image

@cipherw0lf commented on GitHub (Nov 22, 2024): > Hi @cipherw0lf, what does your access policy look like? if you have specific ports there you might need to add the listening port in the container, additionally to the exposed port I have no access policy rules: <img width="1557" alt="image" src="https://github.com/user-attachments/assets/f09c3c03-4d99-4a8e-8de1-701d680c7c23">
saavagebueno commented 2025-11-20 05:30:08 -05:00
Author
Owner
Copy Link

@mgarces commented on GitHub (Nov 28, 2024):

hi

@Codixer I think your issue is not related, since you have no policies, but you do declare them in you route (Access Control Groups). You'll need to create a policy where the destination group name matches your ACG Stefancoding:*

@cipherw0lf I have replicated your setup, exposing a remote subnet 192.168.100.0/24; I've ran nginx proxy manager using docker compose:

services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt

I'm able to reach 80, 81 & 443 via both internal IP address (192.168.100.124) but also via NetBird's IP address. I've even locked down my route using Access Control Group, and both scenarios worked for me (netbird v0.33.0)

Can you please share your network route setup for 192.168.20.0/24? Also, the output for netstat -tunelp on a peer you are running docker containers.

@mgarces commented on GitHub (Nov 28, 2024): hi @Codixer I think your issue is not related, since you have no policies, but you do declare them in you route (Access Control Groups). You'll need to create a policy where the destination group name matches your ACG `Stefancoding:*` @cipherw0lf I have replicated your setup, exposing a remote subnet `192.168.100.0/24`; I've ran nginx proxy manager using docker compose: ``` services: app: image: 'jc21/nginx-proxy-manager:latest' restart: unless-stopped ports: - '80:80' - '81:81' - '443:443' volumes: - ./data:/data - ./letsencrypt:/etc/letsencrypt ``` I'm able to reach `80`, `81` & `443` via both internal IP address (`192.168.100.124`) but also via NetBird's IP address. I've even locked down my route using Access Control Group, and both scenarios worked for me (`netbird v0.33.0`) Can you please share your network route setup for `192.168.20.0/24`? Also, the output for `netstat -tunelp` on a peer you are running docker containers.
saavagebueno commented 2025-11-20 05:30:08 -05:00
Author
Owner
Copy Link

@Codixer commented on GitHub (Nov 28, 2024):

@mgarces
image
image
image

@Codixer commented on GitHub (Nov 28, 2024): @mgarces ![image](https://github.com/user-attachments/assets/2d573db4-ee3e-42f0-9600-039d5454d705) ![image](https://github.com/user-attachments/assets/82eec376-82eb-4fa4-ad92-52bae01b11df) ![image](https://github.com/user-attachments/assets/695b6c9b-6dc2-4241-b9b5-c1a0cd60aef0)
saavagebueno commented 2025-11-20 05:30:08 -05:00
Author
Owner
Copy Link

@Codixer commented on GitHub (Dec 9, 2024):

So another interesting tidbit of information @mgarces & @cipherw0lf (Does this happen for you?)
When I use the local IP (10.0.0.200) in my hosts file (on the domains that I use), it'll refuse to connect. However, when I use the IP address of the netbird client address (100.77.55.26), it returns the service that I am trying to reach. Any idea how to fix this?

@Codixer commented on GitHub (Dec 9, 2024): So another interesting tidbit of information @mgarces & @cipherw0lf (Does this happen for you?) When I use the local IP (10.0.0.200) in my hosts file (on the domains that I use), it'll refuse to connect. However, when I use the IP address of the netbird client address (100.77.55.26), it returns the service that I am trying to reach. Any idea how to fix this?
saavagebueno commented 2025-11-20 05:30:09 -05:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Apr 28, 2025):

Hello @cipherw0lf,

We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.

Could you please confirm if the issue is still there?

We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.

Thanks for your contribution to improving the project!

@nazarewk commented on GitHub (Apr 28, 2025): Hello @cipherw0lf, We're currently reviewing our open issues and would like to verify if this problem still exists in the [latest NetBird version](https://github.com/netbirdio/netbird/releases). Could you please confirm if the issue is still there? We may close this issue temporarily if we don't hear back from you within **2 weeks**, but feel free to reopen it with updated information. Thanks for your contribution to improving the project!
saavagebueno commented 2025-11-20 05:30:09 -05:00
Author
Owner
Copy Link

@cipherw0lf commented on GitHub (Apr 28, 2025):

I believe the issue is resolved in the current version .Will do more testing when I get a chance.Sent from my iPhoneOn 28 Apr 2025, at 17:40, Krzysztof Nazarewski (kdn) @.***> wrote:
nazarewk left a comment (netbirdio/netbird#2903)
Hello @cipherw0lf,
We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.
Could you please confirm if the issue is still there?
We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.
Thanks for your contribution to improving the project!

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

@cipherw0lf commented on GitHub (Apr 28, 2025): I believe the issue is resolved in the current version .Will do more testing when I get a chance.Sent from my iPhoneOn 28 Apr 2025, at 17:40, Krzysztof Nazarewski (kdn) ***@***.***> wrote: nazarewk left a comment (netbirdio/netbird#2903) Hello @cipherw0lf, We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version. Could you please confirm if the issue is still there? We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information. Thanks for your contribution to improving the project! —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label routes triage-needed waiting-feedback
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1423
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?