[PR #6776] [client,relay] Coalesce relayed packets into batch frames (RFC) #28952

Open
opened 2026-08-05 08:07:12 -04:00 by saavagebueno · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/netbirdio/netbird/pull/6776
Author: @Silex
Created: 7/15/2026
Status: 🔄 Open

Base: mainHead: relay-frame-coalescing


📝 Commits (10+)

  • a318cf1 [client] Size relay data-path socket buffers for throughput
  • b7211ae [client] Apply bounded backpressure before dropping inbound relayed messages
  • 823ee82 [client] Size relay conn inbound channel for the path bandwidth-delay product
  • e15cc5b [client] Expose relay inbound message drops in RelayConnState
  • 674cab0 [relay] Add relay_transport_queue_drops_total metric
  • 598a7e9 [relay] Route relayed messages through a per-peer write queue
  • b7b7517 [relay] Add peer forwarding unit tests
  • bfb34e2 [relay] Add a benchmark harness for the relayed data path
  • af0c9a7 [relay] Add ReadBatch to the relay client connection
  • 2af24fc [client] Batch WireGuard inject syscalls via sendmmsg

📊 Changes

21 files changed (+2070 additions, -52 deletions)

View changed files

📝 client/iface/wgproxy/ebpf/proxy.go (+91 -1)
📝 client/iface/wgproxy/ebpf/wrapper.go (+131 -0)
📝 client/iface/wgproxy/udp/proxy.go (+64 -0)
client/net/sockbuffer.go (+68 -0)
client/net/sockbuffer_linux.go (+73 -0)
client/net/sockbuffer_linux_test.go (+73 -0)
client/net/sockbuffer_others.go (+13 -0)
client/net/sockbuffer_test.go (+78 -0)
📝 relay/metrics/realy.go (+32 -17)
📝 relay/server/peer.go (+116 -18)
relay/server/peer_test.go (+286 -0)
shared/relay/client/bench_pipe_test.go (+332 -0)
📝 shared/relay/client/client.go (+265 -15)
📝 shared/relay/client/conn.go (+46 -0)
shared/relay/client/conn_container_test.go (+259 -0)
📝 shared/relay/client/dialer/capability.go (+10 -0)
📝 shared/relay/client/dialer/quic/conn.go (+4 -0)
📝 shared/relay/client/dialer/quic/quic.go (+4 -0)
📝 shared/relay/client/manager.go (+4 -1)
📝 shared/relay/messages/message.go (+67 -0)

...and 1 more files

📄 Description

Describe your changes

Draft / RFC — this is the one slice of the #6021 throughput work that extends the relay wire protocol, so we'd like design guidance before polishing it (questions at the bottom). Stacked on #6775 (which stacks on #6774).

What it does: adds MsgTypeTransportBatch, a frame that packs several packets destined to the same peer as repeated [uint16 length][payload] entries after the normal transport header. The header layout deliberately matches a plain transport message, so the relay server rewrites only the destination peer ID and forwards the whole frame opaquely — no per-packet parsing on the server's hot path. The client packs opportunistically: whatever is already queued gets coalesced up to MaxMessageSize, so it adds no latency under light load and packs full frames exactly when it matters. Opt-in via NB_RELAY_COALESCE (off by default).

Coalescing only engages on stream transports (WebSocket). QUIC datagrams are capped by path MTU, so they can't carry multi-packet frames — one more reason WS wins on relayed throughput once buffers are sized.

Measured effect (same setup as #6774: two 1 Gbit Linux hosts, ~5 ms RTT, single iperf3 flow, cubic, stock sysctls, receiver-side Mbit/s, WS transport):

up down
#6774 + batch I/O 227 184
+ this PR 288 ~180

Cumulative vs stock 0.74.2: 97.6 → 288 Mbit/s single-flow (~3×), with zero retransmissions on the tunnelled TCP.

Why draft — the compatibility question. A batch frame must be understood by both the relay server (it accepts the type) and the receiving peer (frames are forwarded opaquely, so the receiver unpacks them). The protocol currently has no capability negotiation: the version byte is hard-validated to 1 everywhere, and AuthResponse carries only the server address. As-is, the feature is safe purely because it's opt-in — an operator enables it once the relay and all participating peers run a supporting version. That's workable for self-hosted fleets but not a general answer. Options we see:

  1. Keep it operator-coordinated (this PR as-is): env-gated everywhere, documented constraint. Simplest; no handshake change.
  2. Protocol version bump to 2. The relay learns each client's version at auth time; it forwards batch frames opaquely to v2 receivers and unpacks them into single transport messages for v1 receivers; senders enable coalescing only when the relay advertises v2. Backward-compatible in both directions, but old relays reject a v2 auth outright, so new clients need a one-shot fallback reconnect at v1, and the relay grows an unpack path for mixed fleets.
  3. Explicit capability flag in the handshake (new field or message) rather than overloading the version byte.

Which direction would you take? We'll rework the PR accordingly.

#6021

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)
  • This change does not modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — OR I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See CONTRIBUTING.md. — discussion anchor: #6021; this PR is a draft specifically to have that design discussion.

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

  • I added/updated documentation for this change
  • Documentation is not needed for this change (explain why)

Draft under design discussion; NB_RELAY_COALESCE is an opt-in experimental gate. Docs to be added once the negotiation design is settled.


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/netbirdio/netbird/pull/6776 **Author:** [@Silex](https://github.com/Silex) **Created:** 7/15/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `relay-frame-coalescing` --- ### 📝 Commits (10+) - [`a318cf1`](https://github.com/netbirdio/netbird/commit/a318cf1519d81900fa815fa967656f665cf6543a) [client] Size relay data-path socket buffers for throughput - [`b7211ae`](https://github.com/netbirdio/netbird/commit/b7211ae7e6be7323fcb3d5f57a43dc225f25d299) [client] Apply bounded backpressure before dropping inbound relayed messages - [`823ee82`](https://github.com/netbirdio/netbird/commit/823ee82f5451e3edacbea9a41b991b7784ff3e87) [client] Size relay conn inbound channel for the path bandwidth-delay product - [`e15cc5b`](https://github.com/netbirdio/netbird/commit/e15cc5b8a98469d561c3c3a409fa5472044cd0e4) [client] Expose relay inbound message drops in RelayConnState - [`674cab0`](https://github.com/netbirdio/netbird/commit/674cab0d6baa5109bf57969c863eb59d8dd80d8d) [relay] Add relay_transport_queue_drops_total metric - [`598a7e9`](https://github.com/netbirdio/netbird/commit/598a7e96b01245100ff65b7ec035c0f76a333877) [relay] Route relayed messages through a per-peer write queue - [`b7b7517`](https://github.com/netbirdio/netbird/commit/b7b7517665958e76b69f0d31606a6025da08cb97) [relay] Add peer forwarding unit tests - [`bfb34e2`](https://github.com/netbirdio/netbird/commit/bfb34e214d2751ccd97f3b5c98af1f07c2fae8a1) [relay] Add a benchmark harness for the relayed data path - [`af0c9a7`](https://github.com/netbirdio/netbird/commit/af0c9a767b2c88bd8b675009d1384df8405c9dca) [relay] Add ReadBatch to the relay client connection - [`2af24fc`](https://github.com/netbirdio/netbird/commit/2af24fce7186d5e4c9f16688907eba922d700846) [client] Batch WireGuard inject syscalls via sendmmsg ### 📊 Changes **21 files changed** (+2070 additions, -52 deletions) <details> <summary>View changed files</summary> 📝 `client/iface/wgproxy/ebpf/proxy.go` (+91 -1) 📝 `client/iface/wgproxy/ebpf/wrapper.go` (+131 -0) 📝 `client/iface/wgproxy/udp/proxy.go` (+64 -0) ➕ `client/net/sockbuffer.go` (+68 -0) ➕ `client/net/sockbuffer_linux.go` (+73 -0) ➕ `client/net/sockbuffer_linux_test.go` (+73 -0) ➕ `client/net/sockbuffer_others.go` (+13 -0) ➕ `client/net/sockbuffer_test.go` (+78 -0) 📝 `relay/metrics/realy.go` (+32 -17) 📝 `relay/server/peer.go` (+116 -18) ➕ `relay/server/peer_test.go` (+286 -0) ➕ `shared/relay/client/bench_pipe_test.go` (+332 -0) 📝 `shared/relay/client/client.go` (+265 -15) 📝 `shared/relay/client/conn.go` (+46 -0) ➕ `shared/relay/client/conn_container_test.go` (+259 -0) 📝 `shared/relay/client/dialer/capability.go` (+10 -0) 📝 `shared/relay/client/dialer/quic/conn.go` (+4 -0) 📝 `shared/relay/client/dialer/quic/quic.go` (+4 -0) 📝 `shared/relay/client/manager.go` (+4 -1) 📝 `shared/relay/messages/message.go` (+67 -0) _...and 1 more files_ </details> ### 📄 Description ## Describe your changes **Draft / RFC** — this is the one slice of the #6021 throughput work that extends the relay wire protocol, so we'd like design guidance before polishing it (questions at the bottom). Stacked on #6775 (which stacks on #6774). **What it does:** adds `MsgTypeTransportBatch`, a frame that packs several packets destined to the same peer as repeated `[uint16 length][payload]` entries after the normal transport header. The header layout deliberately matches a plain transport message, so the relay server rewrites only the destination peer ID and forwards the whole frame **opaquely** — no per-packet parsing on the server's hot path. The client packs opportunistically: whatever is already queued gets coalesced up to `MaxMessageSize`, so it adds no latency under light load and packs full frames exactly when it matters. Opt-in via `NB_RELAY_COALESCE` (off by default). Coalescing only engages on stream transports (WebSocket). QUIC datagrams are capped by path MTU, so they can't carry multi-packet frames — one more reason WS wins on relayed throughput once buffers are sized. Measured effect (same setup as #6774: two 1 Gbit Linux hosts, ~5 ms RTT, single iperf3 flow, cubic, stock sysctls, receiver-side Mbit/s, WS transport): | | up | down | |---|---|---| | #6774 + batch I/O | 227 | 184 | | + this PR | 288 | ~180 | Cumulative vs stock 0.74.2: 97.6 → 288 Mbit/s single-flow (~3×), with zero retransmissions on the tunnelled TCP. **Why draft — the compatibility question.** A batch frame must be understood by **both** the relay server (it accepts the type) and the **receiving peer** (frames are forwarded opaquely, so the receiver unpacks them). The protocol currently has no capability negotiation: the version byte is hard-validated to 1 everywhere, and `AuthResponse` carries only the server address. As-is, the feature is safe purely because it's opt-in — an operator enables it once the relay and all participating peers run a supporting version. That's workable for self-hosted fleets but not a general answer. Options we see: 1. **Keep it operator-coordinated** (this PR as-is): env-gated everywhere, documented constraint. Simplest; no handshake change. 2. **Protocol version bump to 2.** The relay learns each client's version at auth time; it forwards batch frames opaquely to v2 receivers and unpacks them into single transport messages for v1 receivers; senders enable coalescing only when the relay advertises v2. Backward-compatible in both directions, but old relays reject a v2 auth outright, so new clients need a one-shot fallback reconnect at v1, and the relay grows an unpack path for mixed fleets. 3. **Explicit capability flag** in the handshake (new field or message) rather than overloading the version byte. Which direction would you take? We'll rework the PR accordingly. ## Issue ticket number and link #6021 ## Stack <!-- branch-stack --> ### Checklist - [ ] Is it a bug fix - [ ] Is a typo/documentation fix - [x] Is a feature enhancement - [ ] It is a refactor - [x] Created tests that fail without the change (if possible) - [x] This change does **not** modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — **OR** I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See [CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#discuss-changes-with-the-netbird-team-first). — discussion anchor: #6021; this PR is a draft specifically to have that design discussion. > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [ ] I added/updated documentation for this change - [x] Documentation is **not needed** for this change (explain why) Draft under design discussion; `NB_RELAY_COALESCE` is an opt-in experimental gate. Docs to be added once the negotiation design is settled. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
saavagebueno added the pull-request label 2026-08-05 08:07:12 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#28952