[GH-ISSUE #6021] Relay protocol selection (WS vs QUIC) + per-pair throughput ceiling (~25 Mbps with 76% Go runtime overhead) #11462

New Issue

saavagebueno · 2026-08-05T01:29:46-04:00

saavagebueno commented

2026-08-05 01:29:46 -04:00

Originally created by @dlf-dds on GitHub (Apr 28, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/6021

We've been doing a load-test campaign on a self-hosted netbird deployment (v0.70.0 daemon + v0.70.0 relay, official netbirdio/relay:latest image, t3.micro relay VPS in eu-central-1 / il-central-1, two peer-pairs behind separate AWS NAT Gateways forcing relay routing) and surfaced two related observations that we'd like to raise upstream. Both are reproducible.

Original prompt for the investigation: hyper-derp's announcement post (https://hyper-derp.dev/blog/hyper-derp-announcement/), which benchmarks Netbird's relay code as part of the author's preamble. Our findings below partially confirm and partially refine the claims in that post against our own deployment.

1. Daemon picks WebSocket exclusively even when QUIC is available

The relay binary listens for both WebSocket (TCP) and QUIC (UDP) on the same port (:33073):

relay/server/listener/quic/listener.go:39: QUIC server listening on address: :33073
relay/server/listener/ws/listener.go:44:   WS server listening address: :33073

The netbird daemon binary contains the quic-go library (verified via strings | grep quic-go). Yet across our entire deployment (38 active peer connections during the observation window) 100% of relay connections are WebSocket; 0% are QUIC. Every entry in the relay log is WS client connected from <ip>, none are QUIC client connected from <ip>.

Questions for the maintainers:

Why does the daemon prefer WebSocket over QUIC when both are available?
Is there a config field, env var, or CLI flag to express a protocol preference? netbird up --help shows none; we found QUIC_GO_DISABLE_GSO and QUIC_GO_DISABLE_ECN (quic-go library tunables), but nothing controlling daemon-side protocol selection.
Would maintainers consider exposing a relay-protocol-selection interface (env var, config field, or CLI flag) so operators can choose their preferred transport? A prefer_quic toggle would let deployments take advantage of QUIC's congestion-control and HOL-blocking-resistance properties where the network supports UDP.

We'd be happy to send a PR if there's appetite — please point us at the protocol-selection codepath if so.

2. Per-pair throughput caps at ~25 Mbps with 76% of CPU time in Go runtime overhead

iperf3-over-WireGuard between two peers forced into the relay path consistently caps at ~25-30 Mbps single-flow, regardless of:

Peer hardware (also tested with bigger peer instances; same ceiling)
Linux TCP receive/send buffer sizes (tcp_rmem/tcp_wmem tuned to 16 MB on peers — only ~10% improvement)
Docker / docker-proxy presence (extracted relay binary, ran natively on host with --network=host equivalent — single-flow throughput identical to containerized)
Kernel rmem_max (strace -c shows ZERO setsockopt calls — relay relies on Linux TCP autotuning, doesn't cap itself)
Multi-flow concurrency (4-stream collapses to ~22 Mbps total, 16-stream collapses to ~10 Mbps total — the inverse of expected behavior)

strace -c -f -p <relay-pid> for a 20-second window during sustained load shows:

Syscall	Calls	Time (s)	% of time
futex	16,697	13.53	51.82%
nanosleep	51,581	6.33	24.26%
write	140,320	3.65	13.96%
read	88,565	1.59	6.08%
epoll_pwait	42,860	0.99	3.78%

76% of the relay process's measured CPU time is in futex + nanosleep — Go runtime mutex contention and scheduler park/unpark. Only ~24% is actual network I/O syscalls. Relay process CPU peaked at 33% on the 2-vCPU instance — not CPU-bound at the processor level. This empirically aligns with the "fighting the Go runtime" framing in the hyper-derp post linked above.

We confirmed inter-thread futex contention is part of the picture by sweeping GOMAXPROCS:

`GOMAXPROCS`	Single-flow Mbps
1	36.6
2 (default on 2 vCPU)	27-30
4	34.7

Setting GOMAXPROCS=1 on a 2-vCPU relay yields ~25% improvement — collapses to a single OS thread, eliminates inter-thread futex traffic.

Also worth flagging: when the relay starts with QUIC enabled, quic-go logs:

failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB)

The quic-go library tries to setsockopt the UDP receive buffer to 7 MB but is capped by Linux's default net.core.rmem_max=212992 (208 KB). For deployments that DO use QUIC, this is likely a meaningful per-flow cap — could be worth either a deployment-doc note or adjusting the relay container's expectations.

Questions for the maintainers:

Is the per-pair ~25 Mbps ceiling expected? It's roughly 13× lower than direct TCP between the same hosts (which sustains 330 Mbps single-flow / 1 Gbps 4-parallel).
Has the GOMAXPROCS=1 improvement been observed by others, and is it documented anywhere we missed?
Would the project consider documenting expected per-pair relay throughput as a function of relay instance shape, so operators can plan capacity for sustained-bandwidth workloads (peer-to-peer video, file transfer through relay, etc.)?
Is there interest in upstream contributions toward batched-syscall I/O (recvmmsg/sendmmsg) or kTLS offload, which our profiling suggests would help the 24% I/O slice?

Reproducer

The above is summarized from a multi-day RCA campaign — happy to share specific reproducer commands, full strace output, full iperf3 ramps, or run additional measurements if any of this is helpful. Please reply on this issue and we'll provide whatever level of detail is useful.

Thanks for the project!

Originally created by @dlf-dds on GitHub (Apr 28, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/6021 We've been doing a load-test campaign on a self-hosted netbird deployment (v0.70.0 daemon + v0.70.0 relay, official `netbirdio/relay:latest` image, t3.micro relay VPS in eu-central-1 / il-central-1, two peer-pairs behind separate AWS NAT Gateways forcing relay routing) and surfaced two related observations that we'd like to raise upstream. Both are reproducible. Original prompt for the investigation: hyper-derp's announcement post (https://hyper-derp.dev/blog/hyper-derp-announcement/), which benchmarks Netbird's relay code as part of the author's preamble. Our findings below partially confirm and partially refine the claims in that post against our own deployment. ## 1. Daemon picks WebSocket exclusively even when QUIC is available The relay binary listens for **both** WebSocket (TCP) and QUIC (UDP) on the same port (`:33073`): ``` relay/server/listener/quic/listener.go:39: QUIC server listening on address: :33073 relay/server/listener/ws/listener.go:44: WS server listening address: :33073 ``` The netbird daemon binary contains the `quic-go` library (verified via `strings | grep quic-go`). Yet across our entire deployment (38 active peer connections during the observation window) **100% of relay connections are WebSocket; 0% are QUIC**. Every entry in the relay log is `WS client connected from <ip>`, none are `QUIC client connected from <ip>`. **Questions for the maintainers:** - Why does the daemon prefer WebSocket over QUIC when both are available? - Is there a config field, env var, or CLI flag to express a protocol preference? `netbird up --help` shows none; we found `QUIC_GO_DISABLE_GSO` and `QUIC_GO_DISABLE_ECN` (quic-go library tunables), but nothing controlling daemon-side protocol selection. - Would maintainers consider exposing a relay-protocol-selection interface (env var, config field, or CLI flag) so operators can choose their preferred transport? A `prefer_quic` toggle would let deployments take advantage of QUIC's congestion-control and HOL-blocking-resistance properties where the network supports UDP. We'd be happy to send a PR if there's appetite — please point us at the protocol-selection codepath if so. ## 2. Per-pair throughput caps at ~25 Mbps with 76% of CPU time in Go runtime overhead iperf3-over-WireGuard between two peers forced into the relay path consistently caps at ~25-30 Mbps single-flow, regardless of: - Peer hardware (also tested with bigger peer instances; same ceiling) - Linux TCP receive/send buffer sizes (`tcp_rmem`/`tcp_wmem` tuned to 16 MB on peers — only ~10% improvement) - Docker / docker-proxy presence (extracted relay binary, ran natively on host with `--network=host` equivalent — single-flow throughput **identical** to containerized) - Kernel `rmem_max` (`strace -c` shows ZERO `setsockopt` calls — relay relies on Linux TCP autotuning, doesn't cap itself) - Multi-flow concurrency (4-stream collapses to ~22 Mbps total, 16-stream collapses to ~10 Mbps total — the inverse of expected behavior) `strace -c -f -p <relay-pid>` for a 20-second window during sustained load shows: | Syscall | Calls | Time (s) | % of time | |---|---|---|---| | **futex** | 16,697 | 13.53 | **51.82%** | | **nanosleep** | 51,581 | 6.33 | **24.26%** | | write | 140,320 | 3.65 | 13.96% | | read | 88,565 | 1.59 | 6.08% | | epoll_pwait | 42,860 | 0.99 | 3.78% | **76% of the relay process's measured CPU time is in `futex` + `nanosleep`** — Go runtime mutex contention and scheduler park/unpark. Only ~24% is actual network I/O syscalls. Relay process CPU peaked at 33% on the 2-vCPU instance — not CPU-bound at the processor level. This empirically aligns with the "fighting the Go runtime" framing in the hyper-derp post linked above. We confirmed inter-thread futex contention is part of the picture by sweeping `GOMAXPROCS`: | `GOMAXPROCS` | Single-flow Mbps | |---|---| | 1 | **36.6** | | 2 (default on 2 vCPU) | 27-30 | | 4 | 34.7 | **Setting `GOMAXPROCS=1` on a 2-vCPU relay yields ~25% improvement** — collapses to a single OS thread, eliminates inter-thread futex traffic. Also worth flagging: when the relay starts with QUIC enabled, quic-go logs: ``` failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB) ``` The quic-go library tries to setsockopt the UDP receive buffer to 7 MB but is capped by Linux's default `net.core.rmem_max=212992` (208 KB). For deployments that DO use QUIC, this is likely a meaningful per-flow cap — could be worth either a deployment-doc note or adjusting the relay container's expectations. **Questions for the maintainers:** - Is the per-pair ~25 Mbps ceiling expected? It's roughly 13× lower than direct TCP between the same hosts (which sustains 330 Mbps single-flow / 1 Gbps 4-parallel). - Has the `GOMAXPROCS=1` improvement been observed by others, and is it documented anywhere we missed? - Would the project consider documenting expected per-pair relay throughput as a function of relay instance shape, so operators can plan capacity for sustained-bandwidth workloads (peer-to-peer video, file transfer through relay, etc.)? - Is there interest in upstream contributions toward batched-syscall I/O (`recvmmsg`/`sendmmsg`) or `kTLS` offload, which our profiling suggests would help the 24% I/O slice? ## Reproducer The above is summarized from a multi-day RCA campaign — happy to share specific reproducer commands, full strace output, full iperf3 ramps, or run additional measurements if any of this is helpful. Please reply on this issue and we'll provide whatever level of detail is useful. Thanks for the project!

saavagebueno commented

2026-08-05 01:29:46 -04:00

@jfrconley commented on GitHub (May 28, 2026):

Not affiliated with netbird, but you should never do benchmarking of any kind on a burstable instance. t3.micro is going to burn through the burst credits very quickly and then throttle like mad. You should be using a non-burstable compute tuned instance for this.

You probably got better performance on single thread because you burned through the burst limit slower. Until these results can be validated on a non-burstable instance, they should be considered invalid. This isn't to say that the netbird relay is particularly performant, just that these results as presented are not useful in proving that.

The linked article is interesting, with what appears to be good data. The LLM sounding prose does concern me though...

@jfrconley commented on GitHub (May 28, 2026): Not affiliated with netbird, but you should never do benchmarking of any kind on a burstable instance. t3.micro is going to burn through the burst credits very quickly and then throttle like mad. You should be using a non-burstable compute tuned instance for this. You probably got better performance on single thread because you burned through the burst limit slower. Until these results can be validated on a non-burstable instance, they should be considered invalid. This isn't to say that the netbird relay is particularly performant, just that these results as presented are not useful in proving that. The linked article is interesting, with what appears to be good data. The LLM sounding prose does concern me though...

saavagebueno commented

2026-08-05 01:29:46 -04:00

@jfrconley commented on GitHub (May 29, 2026):

Upon follow up, un-scientific testing of my own CPU usage I can confirm two things:

Your benchmarks are likely impacted by burstable instance use
CPU usage of the relay is still astoundingly high

Additionally, QUIC seems to be working fine for my testing. I already had net.core.rmem_max set appropriately.

Testing from hetzner dedicated AMD instance in hetzner with 3 vcpu, I consistently get 200-230 mbps and cpu usage of the relay container is at 200% (I run an unconsolidated container deployment). When setting GOMAXPROCS=1 I get roughly the same bandwidth but with only 100% cpu usage.

I see a couple of issues with this. For one, cpu usage (effective cost) to bandwidth is simply not economical. Cloudflare realtime turn would be an order of magnitude cheaper assuming you rely heavily on relay traffic. Likely faster as well. I had already suspected this was the case and only adopted the relay once external WebRTC was marked as "legacy". I have some doubts about the wisdom of the custom relay approach.

Second, even when this gets fixed there must be a mechanism for metering this usage to relay clients. Allowing a single client to blow out your entire relay budget doesn't make any sense. A default of 40mbps per peer seems reasonable to me.

@jfrconley commented on GitHub (May 29, 2026): Upon follow up, un-scientific testing of my own CPU usage I can confirm two things: 1. Your benchmarks are likely impacted by burstable instance use 2. CPU usage of the relay is still astoundingly high Additionally, QUIC seems to be working fine for my testing. I already had net.core.rmem_max set appropriately. Testing from hetzner dedicated AMD instance in hetzner with 3 vcpu, I consistently get 200-230 mbps and cpu usage of the relay container is at 200% (I run an unconsolidated container deployment). When setting GOMAXPROCS=1 I get roughly the same bandwidth but with only 100% cpu usage. I see a couple of issues with this. For one, cpu usage (effective cost) to bandwidth is simply not economical. Cloudflare realtime turn would be an order of magnitude cheaper assuming you rely heavily on relay traffic. Likely faster as well. I had already suspected this was the case and only adopted the relay once external WebRTC was marked as "legacy". I have some doubts about the wisdom of the custom relay approach. Second, even when this gets fixed there must be a mechanism for metering this usage to relay clients. Allowing a single client to blow out your entire relay budget doesn't make any sense. A default of 40mbps per peer seems reasonable to me.

saavagebueno commented

2026-08-05 01:29:46 -04:00

@dlf-dds commented on GitHub (May 31, 2026):

Thanks for digging in, and especially for the dedicated-box numbers — that's the most useful part of this.

On burstable: fair, and I'll concede it for the single-flow figure specifically. But I don't think the
burst-credit mechanism explains the GOMAXPROCS result, and two things cut against it:

In my t3.micro runs the relay CPU never went above ~45% (avg ~28%), and a 30s iperf can't drain a fresh
t3.micro's CPU credits. More importantly, throughput dropped as CPU dropped — down to ~14% at 16 streams.
Credit throttling does the opposite: it pins CPU at baseline while you try to push harder. What I saw is
the TCP-over-TCP head-of-line-blocking signature — WireGuard's UDP-shaped retries fighting the outer
TLS-over-TCP retransmit, buffers ballooning (12 MB → 520 MB in two minutes), throughput collapsing.
Your own result is the cleanest refutation of the credit theory: you got the GOMAXPROCS=1 effect (same
bandwidth, half the CPU) on a non-burstable dedicated AMD box. If burst credits were the cause it wouldn't
show up there. It matches an strace I ran showing ~76% of relay time in the Go scheduler (futex +
nanosleep) — inter-thread contention that GOMAXPROCS=1 removes by collapsing to a single OS thread.

And your 200–230 Mbps single-flow actually answers an open question I had. My 31.8 Mbps matched the t3.nano
peer network baseline exactly, so I couldn't tell whether I was measuring the relay or the peer NIC. Your
number says it was the peer NIC — the relay clears 200+ on adequate hardware. So the headline isn't "the
relay caps at 32 Mbps," it's "the relay burns a lot of CPU per Mbps and falls over under multi-flow." Which
is your point, and I agree with it.

So I'd push back on "results should be considered invalid": the CPU-isn't-the-bottleneck-at-multi-flow
finding, the multi-flow collapse, and the GOMAXPROCS win are all hardware-independent, and you confirmed
two of them on bare metal yourself. The one absolute number worth re-running on non-burstable peers is the
single-flow ceiling — and you've basically handed me that already.

GOMAXPROCS=1 is a free ~25% on a 2-vCPU relay until the data plane gets reworked, so worth setting
regardless.

Agree on per-peer metering — one peer saturating a shared relay is a real problem, and a sensible default
cap would help.

On Cloudflare Realtime TURN: agree it's cheaper if you can offload relaying to a third party. Some
deployments can't (trust/sovereignty constraints), so a self-hosted relay still has a reason to exist — it
just needs to stop being this expensive per Mbps.

@dlf-dds commented on GitHub (May 31, 2026): Thanks for digging in, and especially for the dedicated-box numbers — that's the most useful part of this. On burstable: fair, and I'll concede it for the single-flow figure specifically. But I don't think the burst-credit mechanism explains the GOMAXPROCS result, and two things cut against it: - In my t3.micro runs the relay CPU never went above ~45% (avg ~28%), and a 30s iperf can't drain a fresh t3.micro's CPU credits. More importantly, throughput dropped as CPU dropped — down to ~14% at 16 streams. Credit throttling does the opposite: it pins CPU at baseline while you try to push harder. What I saw is the TCP-over-TCP head-of-line-blocking signature — WireGuard's UDP-shaped retries fighting the outer TLS-over-TCP retransmit, buffers ballooning (12 MB → 520 MB in two minutes), throughput collapsing. - Your own result is the cleanest refutation of the credit theory: you got the GOMAXPROCS=1 effect (same bandwidth, half the CPU) on a non-burstable dedicated AMD box. If burst credits were the cause it wouldn't show up there. It matches an strace I ran showing ~76% of relay time in the Go scheduler (futex + nanosleep) — inter-thread contention that GOMAXPROCS=1 removes by collapsing to a single OS thread. And your 200–230 Mbps single-flow actually answers an open question I had. My 31.8 Mbps matched the t3.nano peer network baseline exactly, so I couldn't tell whether I was measuring the relay or the peer NIC. Your number says it was the peer NIC — the relay clears 200+ on adequate hardware. So the headline isn't "the relay caps at 32 Mbps," it's "the relay burns a lot of CPU per Mbps and falls over under multi-flow." Which is your point, and I agree with it. So I'd push back on "results should be considered invalid": the CPU-isn't-the-bottleneck-at-multi-flow finding, the multi-flow collapse, and the GOMAXPROCS win are all hardware-independent, and you confirmed two of them on bare metal yourself. The one absolute number worth re-running on non-burstable peers is the single-flow ceiling — and you've basically handed me that already. GOMAXPROCS=1 is a free ~25% on a 2-vCPU relay until the data plane gets reworked, so worth setting regardless. Agree on per-peer metering — one peer saturating a shared relay is a real problem, and a sensible default cap would help. On Cloudflare Realtime TURN: agree it's cheaper if you can offload relaying to a third party. Some deployments can't (trust/sovereignty constraints), so a self-hosted relay still has a reason to exist — it just needs to stop being this expensive per Mbps.

saavagebueno commented

2026-08-05 01:29:47 -04:00

@Silex commented on GitHub (Jul 15, 2026):

Chiming in with data from an independent setup — we hit the same per-pair ceiling, and I think we found the main root cause, with a relatively simple fix.

Setup: two Linux peers (1 Gbit NICs, ~5 ms RTT between them through the relay), stock netbird 0.74.2, a dedicated relay VPS running the stock netbirdio/relay image, relay forced. Baseline: ~950 Mbit/s P2P between the same two peers, but only ~95–100 Mbit/s single-flow through the relay.

Root cause: nothing in the relayed data path ever sizes its UDP socket buffers — your strace observation ("zero setsockopt calls") is exactly it. Every socket runs at the OS default (net.core.rmem_default, typically ~208 KiB). The counterintuitive part: the drops that matter don't happen on the relay at all — they happen in the sending peer's own kernel, on the local UDP socket netbird uses to pull packets out of WireGuard. A single fast flow overruns that ~208 KiB buffer, the kernel drops, the tunnelled TCP sees unexplained loss, and its congestion window collapses to ~95 Mbit/s. We confirmed this by watching per-run netstat -su deltas on all three hosts: the UDP drop counters move on the sender — not on the relay, not on the receiver.

Fix + numbers (single iperf3 flow, cubic, receiver-side Mbit/s, completely stock sysctls):

variant	up	down
stock 0.74.2	97.6	96.5
+ in-code socket buffer sizing (client-only)	180	150

Retransmissions on the tunnelled TCP go from hundreds per 30 s run to zero. The change sizes the relay-path sockets from inside netbird (SO_RCVBUFFORCE/SO_SNDBUFFORCE when privileged — which the daemon typically is — with portable SetReadBuffer/SetWriteBuffer as fallback), so peers need no sysctl tuning.

PR to fix this issue and gain roughly 1.5-2x perfs is https://github.com/netbirdio/netbird/pull/6774

Other things we tried along the way, some of them also with huge gains (see other PRs)

Batch reads + frame coalescing (recvmmsg on the WG socket, packing several packets per relay frame on the WS transport): 180 → 288 Mbit/s on top of the buffer fix. Real gain, but it needs a relay protocol extension, so we split it out. Both parts are now up as follow-ups: #6775 covers the wire-compatible batching + queueing (the 180 → 227 part), and #6776 is the frame-coalescing protocol extension (227 → 288), opened as a draft to discuss capability negotiation first.
Striping one flow across N relay connections: no measurable gain — the serial per-packet stages at both ends of the pipeline stay serial, so extra lanes don't help a single flow.
BBR on the inner TCP: rides right up to the drop knee, which was useful as a diagnostic (it showed loss, not bandwidth, was the binding constraint), but it's not a fix.
On your question #1: recent versions have NB_RELAY_TRANSPORT=ws|quic (shared/relay/client/transport.go) to pick the transport explicitly. Interesting twist: once the buffers are fixed, WS consistently outperformed QUIC in our runs (227 vs 183 Mbit/s with batched reads) — QUIC datagrams are capped by path MTU so they can't carry coalesced frames, and quic-go's receive datagram queue drops silently under bursts.

@Silex commented on GitHub (Jul 15, 2026): Chiming in with data from an independent setup — we hit the same per-pair ceiling, and I think we found the main root cause, with a relatively simple fix. **Setup:** two Linux peers (1 Gbit NICs, ~5 ms RTT between them through the relay), stock netbird 0.74.2, a dedicated relay VPS running the stock `netbirdio/relay` image, relay forced. Baseline: ~950 Mbit/s P2P between the same two peers, but only ~95–100 Mbit/s single-flow through the relay. **Root cause:** nothing in the relayed data path ever sizes its UDP socket buffers — your strace observation ("zero setsockopt calls") is exactly it. Every socket runs at the OS default (`net.core.rmem_default`, typically ~208 KiB). The counterintuitive part: the drops that matter don't happen on the relay at all — they happen in the **sending peer's own kernel**, on the local UDP socket netbird uses to pull packets out of WireGuard. A single fast flow overruns that ~208 KiB buffer, the kernel drops, the tunnelled TCP sees unexplained loss, and its congestion window collapses to ~95 Mbit/s. We confirmed this by watching per-run `netstat -su` deltas on all three hosts: the UDP drop counters move on the sender — not on the relay, not on the receiver. **Fix + numbers** (single iperf3 flow, cubic, receiver-side Mbit/s, completely stock sysctls): | variant | up | down | |---|---|---| | stock 0.74.2 | 97.6 | 96.5 | | + in-code socket buffer sizing (client-only) | 180 | 150 | Retransmissions on the tunnelled TCP go from hundreds per 30 s run to zero. The change sizes the relay-path sockets from inside netbird (`SO_RCVBUFFORCE`/`SO_SNDBUFFORCE` when privileged — which the daemon typically is — with portable `SetReadBuffer`/`SetWriteBuffer` as fallback), so peers need no sysctl tuning. PR to fix this issue and gain roughly 1.5-2x perfs is https://github.com/netbirdio/netbird/pull/6774 **Other things we tried along the way, some of them also with huge gains (see other PRs)** - *Batch reads + frame coalescing* (recvmmsg on the WG socket, packing several packets per relay frame on the WS transport): 180 → 288 Mbit/s on top of the buffer fix. Real gain, but it needs a relay protocol extension, so we split it out. Both parts are now up as follow-ups: #6775 covers the wire-compatible batching + queueing (the 180 → 227 part), and #6776 is the frame-coalescing protocol extension (227 → 288), opened as a draft to discuss capability negotiation first. - *Striping one flow across N relay connections*: no measurable gain — the serial per-packet stages at both ends of the pipeline stay serial, so extra lanes don't help a single flow. - *BBR on the inner TCP*: rides right up to the drop knee, which was useful as a diagnostic (it showed loss, not bandwidth, was the binding constraint), but it's not a fix. - On your question #1: recent versions have `NB_RELAY_TRANSPORT=ws|quic` (`shared/relay/client/transport.go`) to pick the transport explicitly. Interesting twist: once the buffers are fixed, WS consistently *out*performed QUIC in our runs (227 vs 183 Mbit/s with batched reads) — QUIC datagrams are capped by path MTU so they can't carry coalesced frames, and quic-go's receive datagram queue drops silently under bursts.

Sign in to join this conversation.

Branches Tags

main

claude/agent-network-test-cases-p743vw

android/gui-integration

revert/component-types

feat-post_quantum_ml_kem

ice-stun-wg-demux

dependabot/npm_and_yarn/proxy/web/npm_and_yarn-b39864987c

agent-network-setup-poc

dependabot/go_modules/aws-sdk-8f849ebaed

dependabot/github_actions/actions-a940c7c866

dependabot/go_modules/otel-e34c790afd

dependabot/go_modules/testcontainers-de325c0dd6

dependabot/go_modules/wireguard-dbd6b95108

dependabot/go_modules/pion-5f703e1eca

dependabot/go_modules/gorm-2271c8195b

fix-login-needed-check

dependabot/go_modules/google.golang.org/grpc-1.82.1

fix/ui-gtk3-support

enterprise-traefik-and-migration-fixes

disambiguate_p2p_metrics

revert/component-types-hookup

embedded-vnc

feature/ios-ssh

docs/agent-network-docs-update

dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/s3-1.106.3

dependabot/go_modules/github.com/pion/stun/v3-3.1.5

fix-ssh-authorized-users-multi-rule

peer-acl-multi-source

reverse-proxy-crowdsec-appsec

reverse-proxy-allow-match-or

client-local-metrics

lazy-conn-per-peer

lazy-conn-rosenpass

dependabot/go_modules/github.com/gopacket/gopacket-1.7.0

dependabot/go_modules/goauthentik.io/api/v3-3.2026050.6

dependabot/go_modules/github.com/pires/go-proxyproto-0.15.0

dependabot/go_modules/github.com/jackc/pgx/v5-5.10.0

dependabot/go_modules/github.com/oapi-codegen/runtime-1.6.0

dependabot/go_modules/github.com/eko/gocache/lib/v4-4.2.4

dependabot/go_modules/github.com/pkg/sftp-1.13.11

dependabot/go_modules/github.com/coreos/go-oidc/v3-3.20.0

ssh-windows-privilege-check

fix/explicit-cors-handling

fix/remove-math-rand

test/gui-memory-leak-fix

fix/ui-status-dispatch

install-script-ui-dependencies

fix/grpc-get-network-map

fix/subscribe-status-coalesce

fix/tray-menu-item-leak

fix/windows-tray-race

feature/changeset

worktree-dns-route-qtype-fallthrough

mdm_integration

mlsmaycon-patch-2

feat/agent-network-ollama

proxy-tunnel-cache-ttl-env

coderabbitai/utg/1e5b0a5

grpc-acl

test/battery-drain

components-impl-drop-indexes-use-xids-no-resource-policy-map

fix/nmap-relevant-groups

e2e-guardrail-blocks-unselected-model

fix/lazyconn-cold-start-allowed-ips

vertex-guardrails-model-access-e2e

add-atomic-cache-ops

refactor/relay-foreign-cache

ci/trigger-release-tests

feature/kimi-3-agent-networks-dns-warmup

feature/dns-lazy-conn-warmup

daemon-ipc-acl

feature/ui-translation-key-parity-check

refactor/relay-foreign-cache-tests

fix/lazyconn-idle-keep-wg-peer

dmitri-propagate-auth-grant-types-on-combined

0.74.7-branch

diagnose-empty-vs-corrupt-state

windows-sleep-detector

fix/cli-up-wait-for-daemon

rp_key_persistency

feature/native-grpc

0.74.6-branch

0.74.6-branch-sync

0.74.5-branch-sync

0.74.4-branch

fix/remove-stale-peers-removal

fix/remove-stale-proxy-logic

fix/nsis-preserve-autostart-on-upgrade

refactor/peer-event-bus

dependabot/go_modules/goauthentik.io/api/v3-3.2026050.3

dependabot/go_modules/github.com/pkg/sftp-1.13.10

components-impl-drop-indexes

fix-reset-aggregation-window-flake

dependabot/go_modules/github.com/pion/dtls/v3-3.1.5

0.74.x

fix/relay-states-lock

update-process-pkg

update-gopsutil-v4

fix/relay_states_lock

increase-sysinfo-timeout

dependabot/go_modules/github.com/Azure/go-ntlmssp-0.1.1

dependabot/go_modules/github.com/eko/gocache/store/redis/v4-4.2.6

dependabot/go_modules/github.com/eko/gocache/lib/v4-4.2.3

fix/fail-to-create-upnp-port-mapping-on-opnsense-firewall

0.74.3-branch

fix/routeselector-atomic-exit-node

netmap_progressive_alignment

nmap/components-impl

dependabot/go_modules/github.com/jackc/pgx/v5-5.9.2

dependabot/go_modules/github.com/oapi-codegen/runtime-1.4.2

dependabot/go_modules/github.com/gopacket/gopacket-1.6.1

dependabot/go_modules/github.com/coreos/go-oidc/v3-3.19.0

dependabot/go_modules/github.com/pires/go-proxyproto-0.12.0

fix/signal-watchdog-sync-stop

docs/agent-network

test/affected-logic

fix/revert-ice-filter

refactor/simplify-affected-peers

pascal-filter-policies-by-direction

claude/lock-contention-peer-connect-g8t6au

dmitri-filter-policies-by-direction

refactor/migrate-profiles-to-go

profile-bindings-ios

fix/skip-restart-unchanged-route

fix/mgmt-cache-async-resolve

refactor/wails-update-105

client_lifetime_serialization_refactor

fix/browser-ssh-2

fix/ipv6-and-netstack-accept-loop

fix/browser-ssh

profile-id-name-test

refactor/mgmt-bootstrap

feat/getting-started-unified-wizard

socket-grpc-permissions

fix/mysql-index-migration

windows-dns-firewall

tests/enable-race-on-tests

ui-refactor-gtk3

feature/affected-peers-grpc

profile-id

lazyconn-first-packet-fix-v2

claude/focused-gates-VMTgb

ui-tray-linux-leftclick

fix/ctx-enrichment

daemon-owner

feature/android-client-ssh

worktree-accept-ra-forwarding

nmap/combined-deploy

task/align_protobuff_toolset

feature/session-extend

add-json-yaml-flags

refactor/ephemeral-cleanup

claude/webtransport-relay-wasm-mUjY9

claude/vnc-udp-feasibility-6KB1U

fix/wgport-config

e2e-windows-dns-combined

fix/login-cmd-root-flags

feat/reseller-openapi-spec

github-issue-resolver

add-steamos-support

fix-darwin-uninstaller

flutter-test

ci/freebsd-pkg-bootstrap

cached-serial-check-on-sync

fix-mgmt-cache-bypass-overlay

revert-easyjson-5938

revert-ice-5820

revert-firewalld-5928

refactor/permissions-manager

revert-dns-5935-systemd-resolved

revert-dns-5935-5945

revert-dns-5945-mgmt-cache

feature/log-most-busy-peers

prototype/ui-wails

coderabbitai/utg/8ae8f20

feature/use-peer-fqdn-on-https

release/0.68.3

add-slack-channel

claude/rdp-token-passthrough-eNcqW

transparent-proxy

fix/macos-stale-route-eexist

crowdsec-selfhosted

fix/remove-otel-units

entire/checkpoints/v1

fix/getting-started

feat/static-connectors-combined-server

feature/use-local-keys-embedded

feature/fleetdm

set-env-only-if-not-fork

feature/expose-has-channel

fix/connection-status-race

fix/filter-cgnat-cni-ice-candidates

feature/check-cert-locker-before-acme

test/proxy-fixes

test/proxy-mtu

prototype/ui-tauri

test/proxy-speed

fix-reused-ports

feat/migrate-to-embedded-idp

feature/add-serial-to-proxy-merged

deploy/proxy-serial

test/connection

feature/disable-legacy-port

feature/flag-to-disable-legacy-port

test/perftest

fix/http-redirect

poc-token-command

dn-reverse-proxy

prototype/reverse-proxy-rename

prototype/reverse-proxy-logs-pagination

feature/client-metrics

prototype/reverse-proxy-clusters

debug-dns-route

fix/win-dns-batch

add-extra-route-logs

job-stream-notify-disconnection-eof

deploy/secrets-manager

trigger-proxy-update

bug/update-ios-client-code-build-tags

sync-client-netmap-serial

log/conn-disconn

nmap/compaction-deploy

ci-win-test

feature/disk-encryption-check

wasm-debug

swap-dns-prio

fix/dex-config

feature/migrate-auto-groups-to-table

nmap/compaction

dex-nocgo-stub

feature/exclude-terraform-from-rate-limiting

test-freebsd

retries-refactor

coderabbitai/docstrings/b7e98ac

feat/integrate-zitadel

bug/ios-hanging-reconection

zitadel-idp

feat/network-map-serial

refactor/get-account-no-users

feat/auto-upgrade

feature/report-high-pat-id

feature/temporary-access-for-resource

fix/nmap-fwrules

dont-restart-dns

prototype/ui

update-gomobile

go-dns-for-ice

wasm-ldflags

test-ldflags

wasmbuild-test

feature/networks-s2s

vk/compare-nmaps

dbg/bothmaps

reorder-dns-shutdown

fix/relay-reconnection-race

fix/nmap-exitnodes

vk/debug/nmap-both

move-licensed-code

feat/better-daemon-connection-lost-message

feat/auto-update-2

test/timings

refactor/getaccount-raw

tests/nmap-getaccount

refactor/nmap

refactor/nmap-limit-buffer

feature/detect-mac-wakeup

feature/extract-modules

quick-setings

feat/sync-limiter

feature/store-cache-impl

fix-install-version

feature/store-metrics

feature/metrics-on-store

feature/use-gorm-cache

loadtest-signal

unsymmetrical-squash

refactor/reducate-signaling

test/update-reduce

feature/store-cache

feature/remote-debug

cli-ws-proxy-backend-addr

feat/mgmt-map-serial

snyk-fix-d9d0081a4c7f9137bdb59d0d50a141a2

snyk-fix-7415cea5a11acd66753540ca2c598c63

job-yml-update

feature/android-allow-selecting-routes

fix/up-sequence

fix/dns-hash-update

snyk-fix-967adae9863f17f108ce8948d9117b8d

log/getaccount-by-peer

signal-suppressor

dns-exit-node

feature/auto-updates

feature/cache-srv-key

merged-fixes

fix/missed-offers-and-debug

debug-and-fixes

poc-wasm-clean-backend-s2s

test/remote-debug

debug-api

fix/remove-gpo-if-empty

fix/test-freebsd

fix/mysql-setup

fix/remove-logout-btn

handle-existing-domain-user

chore/unify-domain-validation

snyk-fix-c5fafc8a50ce1f29046e25a1fc346185

feat/profile-edit-btn

snyk-fix-a54966211e18d4cf67e5a2757cc006d1

log-short-id

feat/logout-ephemeral

log-checks

batch-wg-ops

nb-interface-default

feat/aws-integration

add/race-test

feature/relay-feature-versioning

fix/systemd-service-logs

poc/preprocessed-map

add-account-onboarding

bind-ipv6

fix/merge-main

logs/peerlogs-addpeer

feature/net-297-network-migration

feature/support-skip-auto-apply-exit-node-routes

set-cmd

set-command-with-cursor

feature/limit-update-channel

stop-using-locking-share

feature/poc-lazy-detection

feature/net-248-removal-of-sync-mutex-locks

test/multiple-peer-logging

preresolve

add-ns-punnycode-support

apply-routes-early

windows-search-domains

fix/connecting-route-filter

feature/management/rest-client/impersonate

debug-local-records

resource-fields-snake-case

test/grpc-rate-limit

traffic-correlation-policy

feature/rest-client-options

feat/events-metrics

feature/buf-cli

test/add-ratelimiter

test/remove-write-lock-on-add-peer

fix/add-peer-semaphore

feature/users-roles-endpoint

mlsmaycon-patch-1

debug-user-role

chore/primary-key-on-networks

feature/update-account-peers-buffer-startup

remove-ubuntu2004-runners

refactor/permissions-no-pat-allowed

ref/logrus-factory

use-conntrack-zone

deploy/permissions-account

feature/lazy-connection-idle

ref/improve-test-cov

restore-pr-3440

test/increase-grpc-timeouts

feat/buffer-account-peers-update

test/networkmapgeneration-changes

feature/base-manager

feature/flow-receiver

chore/benchmark-with-large-runner

refactor/handshake-initiator

client/ui-update-systray-icons

userspace-router

wgwatcher-test

output-if-key-already-exists

fix/relay-reconnection

feature/port-forwarding-client-codecleaning

detached2

test/callbacks-nil-iceconninfo

refactor/optimize-peer-expiration

enable-udp-port-for-docker-template

fix/relay-update

feature/apply-posture-netmap

fix/group-update-existing-resource

conntrack-stats

upgrade-okta-sdk

multi-price

test/conn-stat

set-min-parallel-tests-for-management

dns-interceptor

debug-dns

router-dns

add-static-system-info

debug-0.29.4

debug-0.33.0

account-refactoring

relay/2800_quic

route-get-account-refactoring

test/seed-random-routes

feature/get-account-refactoring

test/reconnect-race-condition

refactor/get-account-usage

feature/add-session-id-to-update-channel

improve-ipv4conn

fix/async-pion-event-handling

debug

add-offload

feature/validate-group-association-debug

fix/limit-conn-for-sqlite

test/engine-iface

test/transaction-for-jwt-sync

fix/engine-stop-in-foreground

feature/add-mysql-support

test-migration

refactor/header-size-values

relay/eliminate-gob

test/signal-dispatcher-with-relay

relay/debug

validate-icon

feature/ipv6-support

use-pre-expanded-peers-map

feature/use-signal-dispatcher

validate/peer-status

add-read-write-times

fix/sync-peer-race

feature/relay-status

netmap

evaluate/network-map-hash

fix/lower-dns-resolve-interval-on-fail

feature/relay

fix/go-mod-version

upgrade-nftables

synology-userspace-mode

fix/use-ip-for-default-routes-on-darwin

fix/proxy_close

enable-release-workflow-on-pr

deploy/peer-performance

feature/permanent-turn

feature/permanent-turn-proxy

deploy/posture-check-sqlite

feature/optimize_sqlite_save

debug-ios-behavior

fix/delete-route-only-after-adding

tshoot/windows-logger

remove-new-routing

refactor/eliminate-repo-dependency

add-arm-to-ci

refactor-demo-account-object

test/abc2

test/abc

send-ssh-rosenpass-config-meta

refactor-demo

ensure-schedule-never-runs-non-positive

feature/peer-validator-groupmgm

feature/peer-validator-fix

fix/include-active-dashboard-users

fix/handle-canceling-schedule

fix/geo-download

debug-google-workspace

yury/resolve-ip-to-location

feature/extend-sysinfo

sqlite-async-peer-status

yury/add-postgresql-store

fix/route

test-build

posture-checks-poc

debug-keycloak-idp

poc/netstack

for-pascal-tmp

peer-logout-management

manual-peer-logout

detached

chore/refactor-management

test/dns-bind

fix/enforce-acl-for-containers

yury/use-sync-map-in-updatechannel

fix/events-key-handling

filter-cache-on-load-account

fix/user-expiration

handle-user-context-cancellation

nb-client-k8s-statefulset

fake-addr

fix/iptables_in_docker

ebpf-debug

update-getting-started-flow-use-postgres

fix/peer_list_notification

feature/device-authentication-with-client-secret

feature/keep_alive

feat-groups-from-jwt

separate_proxy_from_wgconfig

fix/wg_conn

wg_conn_fix

wg_bind_parallel_processing

fix-rollback-get-acls

proxy_cfg_cleanup

performance-improvement-rego

update-lock-log-level

feat-client-side-acl

refactor/move_grpcserver_logic_to_account_manager

feature/event-storage

feature/update-idp-redeeming-invite

feature/api-peer-info

return-groupminimum-setupkey

feature/interface-bind

documentation_enhancement

fix-peer-registration

ssh

users_cache

pass-client-caller

client_caller_type

revert-283-feat-fix-windows-installer

periodic-peer-updates

ebpf

braginini/wasm

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: DYNR/netbird#11462