[PR #6774] [client] Size relay data-path socket buffers for throughput #26994

Open
opened 2026-08-05 07:08:04 -04:00 by saavagebueno · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/netbirdio/netbird/pull/6774
Author: @Silex
Created: 7/15/2026
Status: 🔄 Open

Base: mainHead: relay-socket-buffers


📝 Commits (1)

  • a318cf1 [client] Size relay data-path socket buffers for throughput

📊 Changes

8 files changed (+313 additions, -0 deletions)

View changed files

📝 client/iface/wgproxy/ebpf/proxy.go (+1 -0)
📝 client/iface/wgproxy/udp/proxy.go (+3 -0)
client/net/sockbuffer.go (+68 -0)
client/net/sockbuffer_linux.go (+73 -0)
client/net/sockbuffer_linux_test.go (+73 -0)
client/net/sockbuffer_others.go (+13 -0)
client/net/sockbuffer_test.go (+78 -0)
📝 shared/relay/client/dialer/quic/quic.go (+4 -0)

📄 Description

Describe your changes

NetBird never sizes the UDP sockets on the relayed data path, so they run at the OS default (net.core.rmem_default, typically ~208 KiB). A single fast relayed flow overruns the sending peer's local WG proxy socket: the kernel drops packets, the tunnelled TCP interprets that as path loss, and its congestion window collapses. In our measurements this caps single-flow relayed throughput at ~95–100 Mbit/s on a path whose individual legs have 5–10× that capacity. #6021 reports the same ceiling and even noticed the missing setsockopt calls via strace.

This PR sizes the receive/send buffers of the sockets that carry relayed WireGuard data:

  • the WG proxy's local socket (both the UDP and eBPF proxy variants),
  • the QUIC dialer's UDP socket (quic-go asks for 7 MiB itself but is clamped by rmem_max and logs a warning; netbird usually runs privileged and can do better).

On Linux it first tries SO_RCVBUFFORCE/SO_SNDBUFFORCE, which bypass net.core.rmem_max/wmem_max when the process has the privilege (the daemon typically runs as root); otherwise it falls back to the portable SetReadBuffer/SetWriteBuffer (clamped by the sysctls, still an improvement over the default). The default size is 7 MiB — matching what quic-go requests — and NB_WGPROXY_SOCKET_BUFFER overrides it in bytes (0 disables the sizing entirely).

Measured effect (two 1 Gbit Linux hosts, ~5 ms RTT via a dedicated relay VPS running the stock relay image, single iperf3 flow, cubic, stock sysctls, receiver-side Mbit/s):

up down
stock 0.74.2 97.6 96.5
with this change 180 150

Retransmissions on the tunnelled TCP drop from hundreds per 30 s run to zero. The change is client-only and wire-compatible: no relay/server change, no configuration needed.

Memory note: SO_RCVBUF/SO_SNDBUF set a limit, not an allocation — memory is only consumed while packets are queued, so the worst case (~2×7 MiB per relayed connection) is reached only under sustained overload, which is exactly when the headroom is needed.

For completeness, other avenues we explored while investigating (details and numbers in #6021) and deliberately left out of this PR:

  • recvmmsg batch reads + packing several packets per relay frame (WS transport): a further 180 → 288 Mbit/s, but it needs a small relay protocol extension — potential follow-up if there's interest.
  • striping one flow across several relay connections: no measurable gain.
  • BBR on the inner TCP: useful as a diagnostic, not a fix.

#6021

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)
  • This change does not modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — OR I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See CONTRIBUTING.md. — discussion anchor: #6021

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

  • I added/updated documentation for this change
  • Documentation is not needed for this change (explain why)

The change is automatic and default-on; no configuration is required. NB_WGPROXY_SOCKET_BUFFER is an advanced escape hatch for tuning/debugging, consistent with other undocumented NB_* tunables in the client.

Summary by CodeRabbit

  • Performance Improvements
    • Improved UDP socket buffer management for relayed WireGuard and QUIC traffic.
    • Added configurable buffer sizing, with a default capacity and an option to disable it.
    • Helps reduce datagram loss during busy relay connections.
  • Compatibility
    • Added platform-appropriate handling across supported operating systems.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/netbirdio/netbird/pull/6774 **Author:** [@Silex](https://github.com/Silex) **Created:** 7/15/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `relay-socket-buffers` --- ### 📝 Commits (1) - [`a318cf1`](https://github.com/netbirdio/netbird/commit/a318cf1519d81900fa815fa967656f665cf6543a) [client] Size relay data-path socket buffers for throughput ### 📊 Changes **8 files changed** (+313 additions, -0 deletions) <details> <summary>View changed files</summary> 📝 `client/iface/wgproxy/ebpf/proxy.go` (+1 -0) 📝 `client/iface/wgproxy/udp/proxy.go` (+3 -0) ➕ `client/net/sockbuffer.go` (+68 -0) ➕ `client/net/sockbuffer_linux.go` (+73 -0) ➕ `client/net/sockbuffer_linux_test.go` (+73 -0) ➕ `client/net/sockbuffer_others.go` (+13 -0) ➕ `client/net/sockbuffer_test.go` (+78 -0) 📝 `shared/relay/client/dialer/quic/quic.go` (+4 -0) </details> ### 📄 Description ## Describe your changes NetBird never sizes the UDP sockets on the relayed data path, so they run at the OS default (`net.core.rmem_default`, typically ~208 KiB). A single fast relayed flow overruns the sending peer's local WG proxy socket: the kernel drops packets, the tunnelled TCP interprets that as path loss, and its congestion window collapses. In our measurements this caps single-flow relayed throughput at ~95–100 Mbit/s on a path whose individual legs have 5–10× that capacity. #6021 reports the same ceiling and even noticed the missing `setsockopt` calls via strace. This PR sizes the receive/send buffers of the sockets that carry relayed WireGuard data: - the WG proxy's local socket (both the UDP and eBPF proxy variants), - the QUIC dialer's UDP socket (quic-go asks for 7 MiB itself but is clamped by `rmem_max` and logs a warning; netbird usually runs privileged and can do better). On Linux it first tries `SO_RCVBUFFORCE`/`SO_SNDBUFFORCE`, which bypass `net.core.rmem_max`/`wmem_max` when the process has the privilege (the daemon typically runs as root); otherwise it falls back to the portable `SetReadBuffer`/`SetWriteBuffer` (clamped by the sysctls, still an improvement over the default). The default size is 7 MiB — matching what quic-go requests — and `NB_WGPROXY_SOCKET_BUFFER` overrides it in bytes (`0` disables the sizing entirely). Measured effect (two 1 Gbit Linux hosts, ~5 ms RTT via a dedicated relay VPS running the stock relay image, single iperf3 flow, cubic, stock sysctls, receiver-side Mbit/s): | | up | down | |---|---|---| | stock 0.74.2 | 97.6 | 96.5 | | with this change | 180 | 150 | Retransmissions on the tunnelled TCP drop from hundreds per 30 s run to zero. The change is client-only and wire-compatible: no relay/server change, no configuration needed. Memory note: `SO_RCVBUF`/`SO_SNDBUF` set a limit, not an allocation — memory is only consumed while packets are queued, so the worst case (~2×7 MiB per relayed connection) is reached only under sustained overload, which is exactly when the headroom is needed. For completeness, other avenues we explored while investigating (details and numbers in #6021) and deliberately left out of this PR: - recvmmsg batch reads + packing several packets per relay frame (WS transport): a further 180 → 288 Mbit/s, but it needs a small relay protocol extension — potential follow-up if there's interest. - striping one flow across several relay connections: no measurable gain. - BBR on the inner TCP: useful as a diagnostic, not a fix. ## Issue ticket number and link #6021 ## Stack <!-- branch-stack --> ### Checklist - [ ] Is it a bug fix - [ ] Is a typo/documentation fix - [x] Is a feature enhancement - [ ] It is a refactor - [x] Created tests that fail without the change (if possible) - [x] This change does **not** modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — **OR** I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See [CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#discuss-changes-with-the-netbird-team-first). — discussion anchor: #6021 > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [ ] I added/updated documentation for this change - [x] Documentation is **not needed** for this change (explain why) The change is automatic and default-on; no configuration is required. `NB_WGPROXY_SOCKET_BUFFER` is an advanced escape hatch for tuning/debugging, consistent with other undocumented `NB_*` tunables in the client. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Performance Improvements** * Improved UDP socket buffer management for relayed WireGuard and QUIC traffic. * Added configurable buffer sizing, with a default capacity and an option to disable it. * Helps reduce datagram loss during busy relay connections. * **Compatibility** * Added platform-appropriate handling across supported operating systems. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
saavagebueno added the pull-request label 2026-08-05 07:08:04 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#26994