[GH-ISSUE #1357] Netbird behind Cloudflare: stream terminated by RST_STREAM #2277

Closed
opened 2026-08-05 00:46:15 -04:00 by saavagebueno · 10 comments
Owner

Originally created by @xiaolei0125 on GitHub (Dec 6, 2023).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/1357

Describe the problem
I use Cloudflare proxy my netbird management and signal service, But netbird client got disconnect and reconnect every 100 second.

this caused by cloudflare 524 timeoout: https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/#error-524-a-timeout-occurred.

Error 524 indicates that Cloudflare successfully connected to the origin web server, but the origin did not provide an HTTP response before the default 100 second connection timed out.

Expected behavior
Add grpc keep alive for management and signal service, adapt to a more complex network environment

NetBird status -d output:
Daemon version: 0.24.3
CLI version: 0.24.3
Management: Connected to https://nb.xxx.com:8443/
Signal: Connected to https://nb.xxx.com:8443/
FQDN: m1.nb.iot
NetBird IP: 100.64.0.3/16
Interface type: Kernel
Peers count: 3/4 Connected

Additional context
WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR
WARN signal/client/grpc.go:151: disconnected from the Signal Exchange due to an error: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 524 (); transport: received unexpected content-type "text/plain; charset=UTF-8"

Related issue
#771
#651

Originally created by @xiaolei0125 on GitHub (Dec 6, 2023). Original GitHub issue: https://github.com/netbirdio/netbird/issues/1357 **Describe the problem** I use Cloudflare proxy my netbird management and signal service, But netbird client got disconnect and reconnect every 100 second. this caused by cloudflare 524 timeoout: https://developers.cloudflare.com/support/troubleshooting/cloudflare-errors/troubleshooting-cloudflare-5xx-errors/#error-524-a-timeout-occurred. > Error 524 indicates that Cloudflare successfully connected to the origin web server, but the origin did not provide an HTTP response before the default 100 second connection timed out. **Expected behavior** Add grpc keep alive for management and signal service, adapt to a more complex network environment **NetBird status -d output:** Daemon version: 0.24.3 CLI version: 0.24.3 Management: Connected to https://nb.xxx.com:8443/ Signal: Connected to https://nb.xxx.com:8443/ FQDN: m1.nb.iot NetBird IP: 100.64.0.3/16 Interface type: Kernel Peers count: 3/4 Connected **Additional context** ` WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR` `WARN signal/client/grpc.go:151: disconnected from the Signal Exchange due to an error: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 524 (); transport: received unexpected content-type "text/plain; charset=UTF-8"` **Related issue** #771 #651
saavagebueno added the bug label 2026-08-05 00:46:16 -04:00
Author
Owner

@xiaolei0125 commented on GitHub (Mar 5, 2024):

Any update?

<!-- gh-comment-id:1978414867 --> @xiaolei0125 commented on GitHub (Mar 5, 2024): Any update?
Author
Owner

@horzadome commented on GitHub (Apr 3, 2024):

I've got this issue too, but it's not really Netbird's problem. The cause of this is Cloudflare's opinionated config which is 100s for their non-enterprise customers and up to 6000s for enterprise.

On the other hand, I'm guessing that implementing a keepalive on management and signal services could improve Netbird experience for normal operation too (non-Cloudflare). That's because corporate proxies and routers also have their own performance-oriented inactive connection timeouts just like Cloudflare, so this could be an overall performance win.

<!-- gh-comment-id:2033711404 --> @horzadome commented on GitHub (Apr 3, 2024): I've got this issue too, but it's not really Netbird's problem. The cause of this is Cloudflare's opinionated config which is 100s for their non-enterprise customers and up to 6000s for enterprise. On the other hand, I'm guessing that implementing a keepalive on management and signal services could improve Netbird experience for normal operation too (non-Cloudflare). That's because corporate proxies and routers also have their own performance-oriented inactive connection timeouts just like Cloudflare, so this could be an overall performance win.
Author
Owner

@toasterlolz commented on GitHub (May 13, 2024):

Any workround for using with cloudflare tunnel? I tried increasing the keep alive time through the Origin configurations, still I am getting the same error!

<!-- gh-comment-id:2106667996 --> @toasterlolz commented on GitHub (May 13, 2024): Any workround for using with cloudflare tunnel? I tried increasing the keep alive time through the Origin configurations, still I am getting the same error!
Author
Owner

@interna1error commented on GitHub (May 17, 2024):

+1 on this one. It’s not only cloudflare that has such kind of inactivity timeouts. For example, ingress-nginx also terminates inactive gRPC connections, and as of today it requires messing with the configuration-snippet to get netbird to work properly. Implementing app-level keepalive would definitely improve user experience for more sophisticated deployments. Looking forward to seeing this implemented!

<!-- gh-comment-id:2116668123 --> @interna1error commented on GitHub (May 17, 2024): +1 on this one. It’s not only cloudflare that has such kind of inactivity timeouts. For example, ingress-nginx also terminates inactive gRPC connections, and as of today it requires messing with the configuration-snippet to get netbird to work properly. Implementing app-level keepalive would definitely improve user experience for more sophisticated deployments. Looking forward to seeing this implemented!
Author
Owner

@JerboaGobi commented on GitHub (Jun 1, 2024):

+1 on this one. It’s not only cloudflare that has such kind of inactivity timeouts. For example, ingress-nginx also terminates inactive gRPC connections, and as of today it requires messing with the configuration-snippet to get netbird to work properly. Implementing app-level keepalive would definitely improve user experience for more sophisticated deployments. Looking forward to seeing this implemented!

Traefik also implements a default readTimeout setting of 60 seconds on its entrypoints. As I understand it is security control to limit DDoS attacks. Choosing to disable readTimeouts is a security risk that each organization would have to accept in order to run Netbird behind a proxy. Furthermore, Cloudflare customers can not disable read timeouts, with a max available setting of 6000s for their enterprise customers.

Edit: To anyone interested you can use Traefik's "reusePort" function on the entry point declarations to duplicate the entry point and then specify specific read timeout settings that only applies per entry point.

<!-- gh-comment-id:2143479188 --> @JerboaGobi commented on GitHub (Jun 1, 2024): > +1 on this one. It’s not only cloudflare that has such kind of inactivity timeouts. For example, ingress-nginx also terminates inactive gRPC connections, and as of today it requires messing with the configuration-snippet to get netbird to work properly. Implementing app-level keepalive would definitely improve user experience for more sophisticated deployments. Looking forward to seeing this implemented! Traefik also implements a default readTimeout setting of 60 seconds on its entrypoints. As I understand it is security control to limit DDoS attacks. Choosing to disable readTimeouts is a security risk that each organization would have to accept in order to run Netbird behind a proxy. Furthermore, Cloudflare customers can not disable read timeouts, with a max available setting of 6000s for their enterprise customers. Edit: To anyone interested you can use Traefik's "reusePort" function on the entry point declarations to duplicate the entry point and then specify specific read timeout settings that only applies per entry point.
Author
Owner

@Silex commented on GitHub (Feb 27, 2025):

@JerboaGobi can you help me a bit here? Whenever I try reusePort then all my routing fails, so I tried to use this:

# dynamic.yml
http:
  serversTransports:
    signalTransport:
      forwardingTimeouts:
        responseHeaderTimeout: '1h'

  routers:
    netbird-signal:
      rule: "Host(`netbird.domain.com`) && PathPrefix(`/signalexchange.SignalExchange/`)"
      service: netbird-signal
      entryPoints: https
      tls:
        certResolver: letsencrypt

  services:
    netbird-signal:
      loadBalancer:
        serversTransport: signalTransport
        servers:
          - url: "h2c://signal:10000"

But I'm not sure which timeouts to configure... can you share how you did it?

For the record here's what I tried:

# traefik.yml

entryPoints:
  https:
    address: ":443"
    reusePort: true # would not start without this line

  https-signal:
    address: ":443"
    reusePort: true
    transport:
      respondingTimeouts:
        readTimeout: "1h"

But then all routing failed...

<!-- gh-comment-id:2688219329 --> @Silex commented on GitHub (Feb 27, 2025): @JerboaGobi can you help me a bit here? Whenever I try `reusePort` then all my routing fails, so I tried to use this: ``` yaml # dynamic.yml http: serversTransports: signalTransport: forwardingTimeouts: responseHeaderTimeout: '1h' routers: netbird-signal: rule: "Host(`netbird.domain.com`) && PathPrefix(`/signalexchange.SignalExchange/`)" service: netbird-signal entryPoints: https tls: certResolver: letsencrypt services: netbird-signal: loadBalancer: serversTransport: signalTransport servers: - url: "h2c://signal:10000" ``` But I'm not sure which timeouts to configure... can you share how you did it? For the record here's what I tried: ``` yaml # traefik.yml entryPoints: https: address: ":443" reusePort: true # would not start without this line https-signal: address: ":443" reusePort: true transport: respondingTimeouts: readTimeout: "1h" ``` But then all routing failed...
Author
Owner

@JerboaGobi commented on GitHub (Feb 27, 2025):

@JerboaGobi can you help me a bit here? Whenever I try reusePort then all my routing fails, so I tried to use this:

dynamic.yml

http:
serversTransports:
signalTransport:
forwardingTimeouts:
responseHeaderTimeout: '1h'

routers:
netbird-signal:
rule: "Host(netbird.domain.com) && PathPrefix(/signalexchange.SignalExchange/)"
service: netbird-signal
entryPoints: https
tls:
certResolver: letsencrypt

services:
netbird-signal:
loadBalancer:
serversTransport: signalTransport
servers:
- url: "h2c://signal:10000"

But I'm not sure which timeouts to configure... can you share how you did it?

For the record here's what I tried:

traefik.yml

entryPoints:
https:
address: ":443"
reusePort: true # would not start without this line

https-signal:
address: ":443"
reusePort: true
transport:
respondingTimeouts:
readTimeout: "1h"

But then all routing failed...

I unfortunately don't have a config example for you. After discussing with the devs more we came to the conclusion that this issue doesn't directly impact the clients in any tangible manner. That's to say I've since rolled back the changes related to reusePort.

<!-- gh-comment-id:2688699953 --> @JerboaGobi commented on GitHub (Feb 27, 2025): > [@JerboaGobi](https://github.com/JerboaGobi) can you help me a bit here? Whenever I try `reusePort` then all my routing fails, so I tried to use this: > > # dynamic.yml > http: > serversTransports: > signalTransport: > forwardingTimeouts: > responseHeaderTimeout: '1h' > > routers: > netbird-signal: > rule: "Host(`netbird.domain.com`) && PathPrefix(`/signalexchange.SignalExchange/`)" > service: netbird-signal > entryPoints: https > tls: > certResolver: letsencrypt > > services: > netbird-signal: > loadBalancer: > serversTransport: signalTransport > servers: > - url: "h2c://signal:10000" > > But I'm not sure which timeouts to configure... can you share how you did it? > > For the record here's what I tried: > > # traefik.yml > > entryPoints: > https: > address: ":443" > reusePort: true # would not start without this line > > https-signal: > address: ":443" > reusePort: true > transport: > respondingTimeouts: > readTimeout: "1h" > > But then all routing failed... I unfortunately don't have a config example for you. After discussing with the devs more we came to the conclusion that this issue doesn't directly impact the clients in any tangible manner. That's to say I've since rolled back the changes related to reusePort.
Author
Owner

@Silex commented on GitHub (Feb 27, 2025):

No worries, thanks.

<!-- gh-comment-id:2689065852 --> @Silex commented on GitHub (Feb 27, 2025): No worries, thanks.
Author
Owner

@PizzaLovingNerd commented on GitHub (May 28, 2026):

Closing this as not a NetBird bug, this is Cloudflare doing Cloudflare things.

When Management and Signal gRPC streams sit idle, Cloudflare's orange-cloud proxy enforces a hard ~100s timeout (524). NetBird already uses gRPC keepalive (HTTP/2 PINGs), but Cloudflare counts DATA frame activity, not PINGs, so the stream still looks idle and gets cut.

Workarounds:

  • Don't proxy NetBird control-plane traffic through Cloudflare's HTTP proxy (DNS-only / grey cloud, or route gRPC around it)
  • Use a reverse proxy you can actually tune (Nginx, Traefik, Caddy) — see External Reverse Proxy docs
  • For partial progress on related proxy timeout issues, track #5184 / #5185
<!-- gh-comment-id:4569306291 --> @PizzaLovingNerd commented on GitHub (May 28, 2026): Closing this as not a NetBird bug, this is Cloudflare doing Cloudflare things. When Management and Signal gRPC streams sit idle, Cloudflare's orange-cloud proxy enforces a hard ~100s timeout (524). NetBird already uses gRPC keepalive (HTTP/2 PINGs), but Cloudflare counts DATA frame activity, not PINGs, so the stream still looks idle and gets cut. Workarounds: - Don't proxy NetBird control-plane traffic through Cloudflare's HTTP proxy (DNS-only / grey cloud, or route gRPC around it) - Use a reverse proxy you can actually tune (Nginx, Traefik, Caddy) — see [External Reverse Proxy docs](https://docs.netbird.io/selfhosted/external-reverse-proxy) - For partial progress on related proxy timeout issues, track [#5184](https://github.com/netbirdio/netbird/issues/5184) / [#5185](https://github.com/netbirdio/netbird/pull/5185)
Author
Owner

@HammyHavoc commented on GitHub (May 28, 2026):

Closing this as not a NetBird bug, this is Cloudflare doing Cloudflare things.

When Management and Signal gRPC streams sit idle, Cloudflare's orange-cloud proxy enforces a hard ~100s timeout (524). NetBird already uses gRPC keepalive (HTTP/2 PINGs), but Cloudflare counts DATA frame activity, not PINGs, so the stream still looks idle and gets cut.

Workarounds:

  • Don't proxy NetBird control-plane traffic through Cloudflare's HTTP proxy (DNS-only / grey cloud, or route gRPC around it)
  • Use a reverse proxy you can actually tune (Nginx, Traefik, Caddy) — see External Reverse Proxy docs
  • For partial progress on related proxy timeout issues, track #5184 / #5185

The recently added proxy feature on NetBird is excellent BTW. Kudos to anyone who worked on that, the results are absolutely fantastic.

<!-- gh-comment-id:4569389253 --> @HammyHavoc commented on GitHub (May 28, 2026): > Closing this as not a NetBird bug, this is Cloudflare doing Cloudflare things. > > When Management and Signal gRPC streams sit idle, Cloudflare's orange-cloud proxy enforces a hard ~100s timeout (524). NetBird already uses gRPC keepalive (HTTP/2 PINGs), but Cloudflare counts DATA frame activity, not PINGs, so the stream still looks idle and gets cut. > > Workarounds: > > - Don't proxy NetBird control-plane traffic through Cloudflare's HTTP proxy (DNS-only / grey cloud, or route gRPC around it) > - Use a reverse proxy you can actually tune (Nginx, Traefik, Caddy) — see [External Reverse Proxy docs](https://docs.netbird.io/selfhosted/external-reverse-proxy) > - For partial progress on related proxy timeout issues, track [#5184](https://github.com/netbirdio/netbird/issues/5184) / [#5185](https://github.com/netbirdio/netbird/pull/5185) The recently added proxy feature on NetBird is excellent BTW. Kudos to anyone who worked on that, the results are absolutely fantastic.
Sign in to join this conversation.
No Label bug
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#2277