SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-07-27 16:32:34 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Optimize Peer Joining Messages #508

New Issue
Open
opened 2025-11-20 05:12:41 -05:00 by saavagebueno · 4 comments
saavagebueno commented 2025-11-20 05:12:41 -05:00
Owner
Copy Link

Originally created by @clarkmcc on GitHub (Nov 10, 2023).

As documented here, we have an arrangement between administrators and peers, where there may be 1-5 administrators and 10,000+ peers. As I understand it, some messages like "peer joining" messages must be distributed to all the peers in a mesh network, however, our mesh according to the ACLs do not allow for "peer" to "peer" communication, therefore it would be wasteful to distribute peer joining messages to all the "peers" rather than just the administrators.

Originally created by @clarkmcc on GitHub (Nov 10, 2023). As [documented here](https://netbirdio.slack.com/archives/C028VPB34NB/p1691336732336609), we have an arrangement between administrators and peers, where there may be 1-5 administrators and 10,000+ peers. As I understand it, some messages like "peer joining" messages must be distributed to all the peers in a mesh network, however, our mesh according to the ACLs do not allow for "peer" to "peer" communication, therefore it would be wasteful to distribute peer joining messages to all the "peers" rather than just the administrators.
saavagebueno added the enhancementfeature-requestclientwaiting-feedbacknetworking labels 2025-11-20 05:12:41 -05:00
saavagebueno commented 2025-11-20 05:12:42 -05:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Apr 17, 2025):

@clarkmcc Is this still relevant for you? Could you elaborate?

I am closing this for lack of activity, but will surely reopen if you get back with more information :)

@nazarewk commented on GitHub (Apr 17, 2025): @clarkmcc Is this still relevant for you? Could you elaborate? I am closing this for lack of activity, but will surely reopen if you get back with more information :)
saavagebueno commented 2025-11-20 05:12:42 -05:00
Author
Owner
Copy Link

@clarkmcc commented on GitHub (Apr 17, 2025):

Yes, this is still relevant for us. In a large mesh, distributing irrelevant messages to all the peers as frequently as those messages would be sent in a large mesh would be wasteful. But I can understand that maybe we have a niche use-case.

@clarkmcc commented on GitHub (Apr 17, 2025): Yes, this is still relevant for us. In a large mesh, distributing irrelevant messages to all the peers as frequently as those messages would be sent in a large mesh would be wasteful. But I can understand that maybe we have a niche use-case.
saavagebueno commented 2025-11-20 05:12:42 -05:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Apr 17, 2025):

I'm not sure what you think/mean by "administrator" and "peers" distinction, but in NetBird, all Peers are equal. They distribute messages only within the scope of pairs of Peers maintaining a single connection. This applies equally to the "peer joining" and any other messages, there are no globally distributed messages involved anywhere in the process of maintaining connectivity.

Depending how you look at it there 2-3 caveats/exceptions to the above:

  • centralized management service (which is NOT a member of the network) distributes "network updated" packets to the relevant & currently connected peers,
  • it is relatively easy to establish a full-mesh with All group on both sides of the ACL policy,
  • it is relatively easy to make a single Peer establish a connectivity with the rest of the network by using All group on one side of the ACL policy,

Periodical "keepalive" packets are required to determine the state of the so-called "connections". Wireguard uses UDP as transport. Then, UDP concerns itself with the scope of a single packet, which does not guarantee any kind of stable or persistent connection (unlike TCP). Increasing the frequency of those packets correlates directly with increased robustness and quicker recovery upon connectivity drops.

The above, combined with a lot of consumer & enterprise-grade stateful firewalls interfering with and/or forgetting UDP packets too far apart from each other. Makes the "chatter" essential to the stability of the connection between NetBird Peers (or any WG-based VPN).

Having that out of the way, we are actively working on a "lazy connections" feature (see https://github.com/netbirdio/netbird/pull/3675), which will make the Peers establish connectivity upon first use instead of maintaining them across whole lifetime of the Peer. This is probably highly relevant to your use case.

Does this answer your doubts or do you still have some ideas to improve the current state?

@nazarewk commented on GitHub (Apr 17, 2025): I'm not sure what you think/mean by "administrator" and "peers" distinction, but in NetBird, all Peers are equal. They distribute messages only within the scope of pairs of Peers maintaining a single connection. This applies equally to the "peer joining" and any other messages, there are no globally distributed messages involved anywhere in the process of maintaining connectivity. Depending how you look at it there 2-3 caveats/exceptions to the above: - centralized management service (which is NOT a member of the network) distributes "network updated" packets to the relevant & currently connected peers, - it is relatively easy to establish a full-mesh with `All` group on both sides of the ACL policy, - it is relatively easy to make a single Peer establish a connectivity with the rest of the network by using `All` group on one side of the ACL policy, Periodical "keepalive" packets are required to determine the state of the so-called "connections". Wireguard uses UDP as transport. Then, UDP concerns itself with the scope of a single packet, which does not guarantee any kind of stable or persistent connection (unlike TCP). Increasing the frequency of those packets correlates directly with increased robustness and quicker recovery upon connectivity drops. The above, combined with a lot of consumer & enterprise-grade stateful firewalls interfering with and/or forgetting UDP packets too far apart from each other. Makes the "chatter" essential to the stability of the connection between NetBird Peers (or any WG-based VPN). Having that out of the way, we are actively working on a "lazy connections" feature (see https://github.com/netbirdio/netbird/pull/3675), which will make the Peers establish connectivity upon first use instead of maintaining them across whole lifetime of the Peer. This is probably highly relevant to your use case. Does this answer your doubts or do you still have some ideas to improve the current state?
saavagebueno commented 2025-11-20 05:12:42 -05:00
Author
Owner
Copy Link

@clarkmcc commented on GitHub (Apr 28, 2025):

Sure, let me clarify. I understand that all peers are equal in a mesh arrangement like Netbird, but it's not uncommon to impose a hub-and-spoke architecture on top of a mesh using ACLs. In our specific case, we might have 1-5 peers that are administrators and 10,000+ peers that are IoT devices. We don't want the IoT devices to talk to each other, but we do want the administrators to be able to talk to the devices.

We can express this relationship using ACLs, but at the time I opened this ticket, I found that IoT devices were receiving peer joining messages for other IoT devices. The ACLs would indicate that the administrators should see every IoT device's peer joining message, but because the ACLs prevented IoT devices from talking to other IoT devices, it seems like distributing 10,000 peer joining messages to 10,000 peers when the 10,001th peer joined was unnecessary.

@clarkmcc commented on GitHub (Apr 28, 2025): Sure, let me clarify. I understand that all peers are equal in a mesh arrangement like Netbird, but it's not uncommon to impose a hub-and-spoke architecture on top of a mesh using ACLs. In our specific case, we might have 1-5 peers that are administrators and 10,000+ peers that are IoT devices. We don't want the IoT devices to talk to each other, but we do want the administrators to be able to talk to the devices. We can express this relationship using ACLs, but at the time I opened this ticket, I found that IoT devices were receiving peer joining messages for other IoT devices. The ACLs would indicate that the administrators should see every IoT device's peer joining message, but because the ACLs prevented IoT devices from talking to other IoT devices, it seems like distributing 10,000 peer joining messages to 10,000 peers when the 10,001th peer joined was unnecessary.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label client enhancement feature-request networking waiting-feedback
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#508
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?