[GH-ISSUE #5497] Self Hosted Netbird Server behind NGINX Reverse Proxy - Error / Warnings #11310

Open
opened 2026-08-05 01:29:16 -04:00 by saavagebueno · 0 comments
Owner

Originally created by @CreativeWolf on GitHub (Mar 4, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5497

Errors / Warnings and stalled Connections

Netbird is running on IP 10.30.1.208 - 8 Core CPU, 4 GB RAM

NGINX Reverse Proxy is running on 10.30.1.168 - 6 Core, 4 GB RAM

It was all fine until I noticed dashboard timing out upon further investigation, found NGINX exhausting 4 CPU cores multiple times I went ahead and increased the NGINX Core as well as Netbird Server's Core but that doesn't seem to fully fix the issue:

Here's NGINX error.log

2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443"
2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443"
2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443"
2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443"
2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443"
2026/03/04 13:29:19 [error] 547023#547023: *21114666 upstream prematurely closed connection while sending to client, client: 2xx.1xx.xxx.88, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443"
2026/03/04 13:30:41 [error] 547020#547020: *19301094 upstream prematurely closed connection while sending to client, client: 2xx.2xx.9x.1xx, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443"
2026/03/04 13:33:01 [error] 547025#547025: *21583035 upstream prematurely closed connection while sending to client, client: 6x.xx.14x.7x, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443"
2026/03/04 13:33:25 [error] 547022#547022: *21690268 upstream rejected request with error 8 while reading response header from upstream, client: 6x.xx.14x.7x, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443"

Here's docker log from docker logs netbird-signal

2026-03-04T13:08:29.391+05:30 INFO ./caller_not_available:0: 2026/03/04 13:08:29 WARNING: [core] [Server #1] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2026-03-04T13:08:29.813+05:30 INFO ./caller_not_available:0: 2026/03/04 13:08:29 WARNING: [core] [Server #1] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"

2026/03/04 13:06:33 [error] 547025#547025: *13349656 connect() failed (111: Connection refused) while connecting to upstream, client: 49.43.227.47, server: netbird.domain.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.domain.com:443"
2026/03/04 13:06:33 [error] 547025#547025: *15026360 connect() failed (111: Connection refused) while connecting to upstream, client: 103.168.80.150, server: netbird.domain.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.sandhata.com:443"
2026/03/04 13:06:33 [error] 547022#547022: *16100060 connect() failed (111: Connection refused) while connecting to upstream, client: 10.30.0.18, server: netbird.sandhata.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.sandhata.com:443"

NetBird version

netbird version v0.66.1

Is any other VPN software installed?

No

Number of expected peers
400

Number of peers Enrolled
269

Number of peers Active
175

I'll be more than happy to provide more information as required.

Appreciate any inputs to fix this issue.

Thanks!

Originally created by @CreativeWolf on GitHub (Mar 4, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5497 **Errors / Warnings and stalled Connections** Netbird is running on IP 10.30.1.208 - 8 Core CPU, 4 GB RAM NGINX Reverse Proxy is running on 10.30.1.168 - 6 Core, 4 GB RAM It was all fine until I noticed dashboard timing out upon further investigation, found NGINX exhausting 4 CPU cores multiple times I went ahead and increased the NGINX Core as well as Netbird Server's Core but that doesn't seem to fully fix the issue: Here's NGINX error.log 2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443" 2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443" 2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443" 2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443" 2026/03/04 13:27:15 [error] 547023#547023: *20839439 upstream rejected request with error 8 while reading response header from upstream, client: 10.30.0.239, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443" 2026/03/04 13:29:19 [error] 547023#547023: *21114666 upstream prematurely closed connection while sending to client, client: 2xx.1xx.xxx.88, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443" 2026/03/04 13:30:41 [error] 547020#547020: *19301094 upstream prematurely closed connection while sending to client, client: 2xx.2xx.9x.1xx, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443" 2026/03/04 13:33:01 [error] 547025#547025: *21583035 upstream prematurely closed connection while sending to client, client: 6x.xx.14x.7x, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443" 2026/03/04 13:33:25 [error] 547022#547022: *21690268 upstream rejected request with error 8 while reading response header from upstream, client: 6x.xx.14x.7x, server: netbird.example.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.example.com:443" Here's docker log from docker logs netbird-signal 2026-03-04T13:08:29.391+05:30 INFO ./caller_not_available:0: 2026/03/04 13:08:29 WARNING: [core] [Server #1] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing" 2026-03-04T13:08:29.813+05:30 INFO ./caller_not_available:0: 2026/03/04 13:08:29 WARNING: [core] [Server #1] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing" 2026/03/04 13:06:33 [error] 547025#547025: *13349656 connect() failed (111: Connection refused) while connecting to upstream, client: 49.43.227.47, server: netbird.domain.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.domain.com:443" 2026/03/04 13:06:33 [error] 547025#547025: *15026360 connect() failed (111: Connection refused) while connecting to upstream, client: 103.168.80.150, server: netbird.domain.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.sandhata.com:443" 2026/03/04 13:06:33 [error] 547022#547022: *16100060 connect() failed (111: Connection refused) while connecting to upstream, client: 10.30.0.18, server: netbird.sandhata.com, request: "POST /signalexchange.SignalExchange/Send HTTP/2.0", upstream: "grpc://10.30.1.208:10000", host: "netbird.sandhata.com:443" **NetBird version** `netbird version v0.66.1` **Is any other VPN software installed?** No **Number of expected peers** `400` **Number of peers Enrolled** `269` **Number of peers Active** `175` I'll be more than happy to provide more information as required. Appreciate any inputs to fix this issue. Thanks!
saavagebueno added the triage-needed label 2026-08-05 01:29:16 -04:00
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11310