[PR #6775] [client,relay] Reduce per-packet overhead on the relay data path #26992

Open
opened 2026-08-05 07:08:04 -04:00 by saavagebueno · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/netbirdio/netbird/pull/6775
Author: @Silex
Created: 7/15/2026
Status: 🔄 Open

Base: mainHead: relay-datapath-perf


📝 Commits (10+)

  • a318cf1 [client] Size relay data-path socket buffers for throughput
  • b7211ae [client] Apply bounded backpressure before dropping inbound relayed messages
  • 823ee82 [client] Size relay conn inbound channel for the path bandwidth-delay product
  • e15cc5b [client] Expose relay inbound message drops in RelayConnState
  • 674cab0 [relay] Add relay_transport_queue_drops_total metric
  • 598a7e9 [relay] Route relayed messages through a per-peer write queue
  • b7b7517 [relay] Add peer forwarding unit tests
  • bfb34e2 [relay] Add a benchmark harness for the relayed data path
  • af0c9a7 [relay] Add ReadBatch to the relay client connection
  • 2af24fc [client] Batch WireGuard inject syscalls via sendmmsg

📊 Changes

19 files changed (+1772 additions, -45 deletions)

View changed files

📝 client/iface/wgproxy/ebpf/proxy.go (+89 -1)
📝 client/iface/wgproxy/ebpf/wrapper.go (+131 -0)
📝 client/iface/wgproxy/udp/proxy.go (+62 -0)
client/net/sockbuffer.go (+68 -0)
client/net/sockbuffer_linux.go (+73 -0)
client/net/sockbuffer_linux_test.go (+73 -0)
client/net/sockbuffer_others.go (+13 -0)
client/net/sockbuffer_test.go (+78 -0)
📝 relay/metrics/realy.go (+32 -17)
📝 relay/server/peer.go (+112 -17)
relay/server/peer_test.go (+286 -0)
shared/relay/client/bench_pipe_test.go (+332 -0)
📝 shared/relay/client/client.go (+96 -9)
📝 shared/relay/client/conn.go (+46 -0)
shared/relay/client/conn_container_test.go (+259 -0)
📝 shared/relay/client/dialer/capability.go (+10 -0)
📝 shared/relay/client/dialer/quic/conn.go (+4 -0)
📝 shared/relay/client/dialer/quic/quic.go (+4 -0)
📝 shared/relay/client/manager.go (+4 -1)

📄 Description

Describe your changes

Second slice of the relay-throughput work from #6021 (first slice: #6774). Everything in this PR is wire-compatible — no protocol change, an old client and a new relay (or vice versa) interoperate unchanged.

Four independent layers, bottom of the stack first:

1. Client: bounded backpressure on the inbound relay path. Instead of dropping a relayed message the moment the per-peer channel is full, the reader applies bounded backpressure before giving up, and the channel is sized for the path's bandwidth-delay product rather than a fixed small constant. Drops that do happen are now visible: RelayConnState exposes an InboundMsgDrops counter instead of losing packets silently.

2. Relay server: per-peer write queue. Forwarding used to write synchronously into the destination peer's connection, so one slow receiver could stall the shared forwarding path. Writes now go through a bounded per-peer queue with a drop counter exported as relay_transport_queue_drops_total on the metrics endpoint. Unit tests included.

3. Batch I/O on the data path (opt-in). Three syscall-batching levers, each gated by an env var and off by default:

  • NB_RELAY_BATCH_READ — drain the local WG proxy socket with recvmmsg instead of one read per packet,
  • NB_RELAY_INJECT_BATCH — inject received packets into WireGuard with sendmmsg,
  • plus a ReadBatch on the relay client connection feeding the inject path.

4. A benchmark harness (TestPipe... in shared/relay/client) that exercises the full client→relay→client data path in-process, so data-path changes can be measured without a lab setup.

Measured effect on the same pair as #6774 (two 1 Gbit Linux hosts, ~5 ms RTT via a dedicated relay, single iperf3 flow, cubic, stock sysctls, receiver-side Mbit/s), with the batch gates enabled and the WebSocket transport (NB_RELAY_TRANSPORT=ws):

up down
#6774 alone 180 150
+ this PR 227 184

(Stock 0.74.2 baseline was 97.6/96.5.) A side observation from the same runs: once buffers are sized, the WS transport consistently outperformed QUIC on this path (227 vs 183 up) — QUIC datagram queues drop silently under bursts.

The batch gates ship off exactly as they were tested; happy to flip any of them default-on if that's the direction you prefer after your own validation.

Stacked on #6774 — the first commit here is that PR's commit; review the rest.

#6021

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)
  • This change does not modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — OR I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See CONTRIBUTING.md. — discussion anchor: #6021

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

  • I added/updated documentation for this change
  • Documentation is not needed for this change (explain why)

The default-on parts (backpressure, write queue, metric) need no operator action; the batch env vars are advanced opt-in tuning knobs, consistent with other undocumented NB_* tunables. Can add docs for the metric name if desired.

Summary by CodeRabbit

  • New Features
    • Added optional packet batching to improve relay throughput and reduce processing overhead.
    • Added configurable UDP socket buffer sizing, including an option to disable it.
    • Added inbound message drop reporting and transport queue drop metrics.
  • Bug Fixes
    • Improved backpressure handling so slow destinations no longer block message intake.
    • Added bounded queues and safer cleanup to prevent stalls during congestion or connection closure.
  • Performance
    • Improved bidirectional relay performance with batched reads and writes.
    • Added support for detecting datagram-capable connections.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/netbirdio/netbird/pull/6775 **Author:** [@Silex](https://github.com/Silex) **Created:** 7/15/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `relay-datapath-perf` --- ### 📝 Commits (10+) - [`a318cf1`](https://github.com/netbirdio/netbird/commit/a318cf1519d81900fa815fa967656f665cf6543a) [client] Size relay data-path socket buffers for throughput - [`b7211ae`](https://github.com/netbirdio/netbird/commit/b7211ae7e6be7323fcb3d5f57a43dc225f25d299) [client] Apply bounded backpressure before dropping inbound relayed messages - [`823ee82`](https://github.com/netbirdio/netbird/commit/823ee82f5451e3edacbea9a41b991b7784ff3e87) [client] Size relay conn inbound channel for the path bandwidth-delay product - [`e15cc5b`](https://github.com/netbirdio/netbird/commit/e15cc5b8a98469d561c3c3a409fa5472044cd0e4) [client] Expose relay inbound message drops in RelayConnState - [`674cab0`](https://github.com/netbirdio/netbird/commit/674cab0d6baa5109bf57969c863eb59d8dd80d8d) [relay] Add relay_transport_queue_drops_total metric - [`598a7e9`](https://github.com/netbirdio/netbird/commit/598a7e96b01245100ff65b7ec035c0f76a333877) [relay] Route relayed messages through a per-peer write queue - [`b7b7517`](https://github.com/netbirdio/netbird/commit/b7b7517665958e76b69f0d31606a6025da08cb97) [relay] Add peer forwarding unit tests - [`bfb34e2`](https://github.com/netbirdio/netbird/commit/bfb34e214d2751ccd97f3b5c98af1f07c2fae8a1) [relay] Add a benchmark harness for the relayed data path - [`af0c9a7`](https://github.com/netbirdio/netbird/commit/af0c9a767b2c88bd8b675009d1384df8405c9dca) [relay] Add ReadBatch to the relay client connection - [`2af24fc`](https://github.com/netbirdio/netbird/commit/2af24fce7186d5e4c9f16688907eba922d700846) [client] Batch WireGuard inject syscalls via sendmmsg ### 📊 Changes **19 files changed** (+1772 additions, -45 deletions) <details> <summary>View changed files</summary> 📝 `client/iface/wgproxy/ebpf/proxy.go` (+89 -1) 📝 `client/iface/wgproxy/ebpf/wrapper.go` (+131 -0) 📝 `client/iface/wgproxy/udp/proxy.go` (+62 -0) ➕ `client/net/sockbuffer.go` (+68 -0) ➕ `client/net/sockbuffer_linux.go` (+73 -0) ➕ `client/net/sockbuffer_linux_test.go` (+73 -0) ➕ `client/net/sockbuffer_others.go` (+13 -0) ➕ `client/net/sockbuffer_test.go` (+78 -0) 📝 `relay/metrics/realy.go` (+32 -17) 📝 `relay/server/peer.go` (+112 -17) ➕ `relay/server/peer_test.go` (+286 -0) ➕ `shared/relay/client/bench_pipe_test.go` (+332 -0) 📝 `shared/relay/client/client.go` (+96 -9) 📝 `shared/relay/client/conn.go` (+46 -0) ➕ `shared/relay/client/conn_container_test.go` (+259 -0) 📝 `shared/relay/client/dialer/capability.go` (+10 -0) 📝 `shared/relay/client/dialer/quic/conn.go` (+4 -0) 📝 `shared/relay/client/dialer/quic/quic.go` (+4 -0) 📝 `shared/relay/client/manager.go` (+4 -1) </details> ### 📄 Description ## Describe your changes Second slice of the relay-throughput work from #6021 (first slice: #6774). Everything in this PR is **wire-compatible** — no protocol change, an old client and a new relay (or vice versa) interoperate unchanged. Four independent layers, bottom of the stack first: **1. Client: bounded backpressure on the inbound relay path.** Instead of dropping a relayed message the moment the per-peer channel is full, the reader applies bounded backpressure before giving up, and the channel is sized for the path's bandwidth-delay product rather than a fixed small constant. Drops that do happen are now visible: `RelayConnState` exposes an `InboundMsgDrops` counter instead of losing packets silently. **2. Relay server: per-peer write queue.** Forwarding used to write synchronously into the destination peer's connection, so one slow receiver could stall the shared forwarding path. Writes now go through a bounded per-peer queue with a drop counter exported as `relay_transport_queue_drops_total` on the metrics endpoint. Unit tests included. **3. Batch I/O on the data path (opt-in).** Three syscall-batching levers, each gated by an env var and **off by default**: - `NB_RELAY_BATCH_READ` — drain the local WG proxy socket with `recvmmsg` instead of one read per packet, - `NB_RELAY_INJECT_BATCH` — inject received packets into WireGuard with `sendmmsg`, - plus a `ReadBatch` on the relay client connection feeding the inject path. **4. A benchmark harness** (`TestPipe...` in `shared/relay/client`) that exercises the full client→relay→client data path in-process, so data-path changes can be measured without a lab setup. Measured effect on the same pair as #6774 (two 1 Gbit Linux hosts, ~5 ms RTT via a dedicated relay, single iperf3 flow, cubic, stock sysctls, receiver-side Mbit/s), with the batch gates enabled and the WebSocket transport (`NB_RELAY_TRANSPORT=ws`): | | up | down | |---|---|---| | #6774 alone | 180 | 150 | | + this PR | 227 | 184 | (Stock 0.74.2 baseline was 97.6/96.5.) A side observation from the same runs: once buffers are sized, the WS transport consistently outperformed QUIC on this path (227 vs 183 up) — QUIC datagram queues drop silently under bursts. The batch gates ship off exactly as they were tested; happy to flip any of them default-on if that's the direction you prefer after your own validation. Stacked on #6774 — the first commit here is that PR's commit; review the rest. ## Issue ticket number and link #6021 ## Stack <!-- branch-stack --> ### Checklist - [ ] Is it a bug fix - [ ] Is a typo/documentation fix - [x] Is a feature enhancement - [ ] It is a refactor - [x] Created tests that fail without the change (if possible) - [x] This change does **not** modify the public API, gRPC protocols, functionality behavior, CLI / service flags, or introduce a new feature — **OR** I have discussed it with the NetBird team beforehand (link the issue / Slack thread in the description). See [CONTRIBUTING.md](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTING.md#discuss-changes-with-the-netbird-team-first). — discussion anchor: #6021 > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [ ] I added/updated documentation for this change - [x] Documentation is **not needed** for this change (explain why) The default-on parts (backpressure, write queue, metric) need no operator action; the batch env vars are advanced opt-in tuning knobs, consistent with other undocumented `NB_*` tunables. Can add docs for the metric name if desired. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional packet batching to improve relay throughput and reduce processing overhead. * Added configurable UDP socket buffer sizing, including an option to disable it. * Added inbound message drop reporting and transport queue drop metrics. * **Bug Fixes** * Improved backpressure handling so slow destinations no longer block message intake. * Added bounded queues and safer cleanup to prevent stalls during congestion or connection closure. * **Performance** * Improved bidirectional relay performance with batched reads and writes. * Added support for detecting datagram-capable connections. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
saavagebueno added the pull-request label 2026-08-05 07:08:04 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#26992