SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-04 19:55:09 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Routes not updated when destination group peers appear after source peer startup #2111

New Issue
Open
opened 2025-11-20 07:04:04 -05:00 by saavagebueno · 0 comments
saavagebueno commented 2025-11-20 07:04:04 -05:00
Owner
Copy Link

Originally created by @ddrugeon on GitHub (Jul 24, 2025).

Describe the problem

Routes are not automatically updated when destination group peers appear after source peer startup. When a NetBird peer connects and the destination group in an access policy is empty (no connected peers), routes are not automatically updated when peers later join the destination group. This requires manual intervention to force route synchronization.

To Reproduce

Steps to reproduce the behavior:

  1. Create an access policy allowing Group A (group_a) to access Group B (group_b)
  2. Create a setup key and associate it to group_a (setup key ephemeral and reusable)
  3. Create a setup key and associate it to group_b (setup key ephemeral and reusable)
  4. Start a NetBird agent that belongs to Group A when Group B has no connected peers
  5. Verify that Group A peer connects successfully but has no routes to Group B (expected behavior)
  6. Start a NetBird agent that belongs to Group B (with network routes configured for 10.99.0.0/16 for instance)
  7. Verify that Group B peer connects and appears as "Connected" in the management interface
  8. Check routing table on Group A peer - no routes to Group B resources appear
  9. Test connectivity from Group A peer to Group B resources - connection fails
  10. Manually restart NetBird agent on Group A peer with netbird down && netbird up or update Access policy to force route update
  11. Routes now appear and connectivity works

Expected behavior

When a new peer joins Group B:

  • All peers in Group A should automatically receive route updates without manual intervention
  • Network connectivity should be established immediately
  • Routes should appear in the routing table of Group A peers
  • ACL rules should be re-evaluated and distributed

Are you using NetBird Cloud?

No, I am using a self-hosted NetBird control plane.

NetBird version

netbird version: 0.51.2

Is any other VPN software installed?

No, NetBird is the only VPN software installed. Running in Alpine Linux containers in Kubernetes.

Debug output

Group A peer logs (no route updates received when Group B peer connects):

2025-07-24T07:01:26Z INFO relay/client/manager.go:212: update relay server URLs: [rels://109-232-233-166.nip.io:443]
2025-07-24T07:01:26Z INFO client/internal/engine.go:807: SSH server is not enabled
2025-07-24T07:01:26Z INFO client/internal/dns/file_unix.go:109: created a NetBird managed /etc/resolv.conf file with the DNS settings
2025-07-24T07:01:26Z INFO client/internal/acl/manager.go:72: ACL rules processed in: 1.057536ms, total rules count: 0
2025-07-24T07:18:03Z INFO client/internal/acl/manager.go:72: ACL rules processed in: 946.073µs, total rules count: 0
# No route updates logged when Group B peer connects later

Interface and routing information from Group A peer:

/ # ip addr show wt0
6: wt0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none
    inet 100.121.35.67/16 brd 100.121.255.255 scope global wt0
       valid_lft forever preferred_lft forever

/ # ip route show | grep wt0
# No routes via wt0 interface appear until manual restart

/ # ping 100.121.73.128  # Group B peer IP
PING 100.121.73.128 (100.121.73.128): 56 data bytes
ping: sendto: Required key not available

Note: Due to security restrictions, I cannot run netbird status -dA or netbird debug from within the container environment, but can provide additional logs if needed.

Additional context

Use case: This occurs in a Cluster API (CAPI) controller environment where:

The CAPI controllers run in a Kubernetes pod with a NetBird sidecar container
The controller needs to access dynamically created cluster private load balancers in a VPC
Routing peers may restart or be replaced during operations
Manual intervention for route updates is not feasible in automated environments

Network Configuration:

Group A: group_a - Contains Kubernetes sidecar containers
Group B: group_b - Contains routing peers with network routes
Network Route: 10.99.0.0/16 via Group B peers (masqueraded)
Policy: Allow Group A → Group B (all protocols, bidirectional)

Workarounds that force synchronization:

Restart NetBird agent: netbird down && netbird up
Toggle policy via API (disable/enable)
Restart the container/pod

Impact: This affects dynamic environments where peers join/leave groups frequently, infrastructure automation, and high availability scenarios.
Potential root cause: The management service may not be triggering network map redistribution when group membership changes for already-connected peers.

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @ddrugeon on GitHub (Jul 24, 2025). **Describe the problem** Routes are not automatically updated when destination group peers appear after source peer startup. When a NetBird peer connects and the destination group in an access policy is empty (no connected peers), routes are not automatically updated when peers later join the destination group. This requires manual intervention to force route synchronization. **To Reproduce** Steps to reproduce the behavior: 1. Create an access policy allowing Group A (group_a) to access Group B (group_b) 2. Create a setup key and associate it to group_a (setup key ephemeral and reusable) 3. Create a setup key and associate it to group_b (setup key ephemeral and reusable) 5. Start a NetBird agent that belongs to Group A when Group B has no connected peers 6. Verify that Group A peer connects successfully but has no routes to Group B (expected behavior) 7. Start a NetBird agent that belongs to Group B (with network routes configured for 10.99.0.0/16 for instance) 8. Verify that Group B peer connects and appears as "Connected" in the management interface 9. Check routing table on Group A peer - no routes to Group B resources appear 10. Test connectivity from Group A peer to Group B resources - connection fails 11. Manually restart NetBird agent on Group A peer with netbird down && netbird up or update Access policy to force route update 12. Routes now appear and connectivity works **Expected behavior** When a new peer joins Group B: - All peers in Group A should automatically receive route updates without manual intervention - Network connectivity should be established immediately - Routes should appear in the routing table of Group A peers - ACL rules should be re-evaluated and distributed **Are you using NetBird Cloud?** No, I am using a self-hosted NetBird control plane. **NetBird version** `netbird version: 0.51.2` **Is any other VPN software installed?** No, NetBird is the only VPN software installed. Running in Alpine Linux containers in Kubernetes. **Debug output** Group A peer logs (no route updates received when Group B peer connects): ``` 2025-07-24T07:01:26Z INFO relay/client/manager.go:212: update relay server URLs: [rels://109-232-233-166.nip.io:443] 2025-07-24T07:01:26Z INFO client/internal/engine.go:807: SSH server is not enabled 2025-07-24T07:01:26Z INFO client/internal/dns/file_unix.go:109: created a NetBird managed /etc/resolv.conf file with the DNS settings 2025-07-24T07:01:26Z INFO client/internal/acl/manager.go:72: ACL rules processed in: 1.057536ms, total rules count: 0 2025-07-24T07:18:03Z INFO client/internal/acl/manager.go:72: ACL rules processed in: 946.073µs, total rules count: 0 # No route updates logged when Group B peer connects later ``` Interface and routing information from Group A peer: ``` / # ip addr show wt0 6: wt0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000 link/none inet 100.121.35.67/16 brd 100.121.255.255 scope global wt0 valid_lft forever preferred_lft forever / # ip route show | grep wt0 # No routes via wt0 interface appear until manual restart / # ping 100.121.73.128 # Group B peer IP PING 100.121.73.128 (100.121.73.128): 56 data bytes ping: sendto: Required key not available ``` Note: Due to security restrictions, I cannot run netbird status -dA or netbird debug from within the container environment, but can provide additional logs if needed. **Additional context** Use case: This occurs in a Cluster API (CAPI) controller environment where: The CAPI controllers run in a Kubernetes pod with a NetBird sidecar container The controller needs to access dynamically created cluster private load balancers in a VPC Routing peers may restart or be replaced during operations Manual intervention for route updates is not feasible in automated environments Network Configuration: Group A: group_a - Contains Kubernetes sidecar containers Group B: group_b - Contains routing peers with network routes Network Route: 10.99.0.0/16 via Group B peers (masqueraded) Policy: Allow Group A → Group B (all protocols, bidirectional) Workarounds that force synchronization: Restart NetBird agent: netbird down && netbird up Toggle policy via API (disable/enable) Restart the container/pod Impact: This affects dynamic environments where peers join/leave groups frequently, infrastructure automation, and high availability scenarios. Potential root cause: The management service may not be triggering network map redistribution when group membership changes for already-connected peers. **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2025-11-20 07:04:04 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#2111
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?