[GH-ISSUE #5184] Management gRPC /management.ManagementService/Job intermittently times out (~every 2 minutes) with 504 Gateway Timeout, causing client disconnects / frequent “Network map updated” events #10040

Open
opened 2026-08-05 01:24:25 -04:00 by saavagebueno · 20 comments
Owner

Originally created by @maxpain on GitHub (Jan 26, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5184

Describe the problem

In a self-hosted NetBird control-plane setup, the client intermittently disconnects from Management roughly every ~2 minutes when calling the gRPC endpoint /management.ManagementService/Job. At the same time the client logs an event “Network map updated”. When relay was unavailable, this behavior caused frequent/continuous loss of connectivity inside the VPN. After fixing relay, the VPN data-plane is stable, but the periodic Management 504 / gRPC transport failure still occurs and looks abnormal.

Client error at the moment of failure:

Management: Disconnected, reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type "text/plain; charset=UTF-8"

On the management side, logs show:

2026-01-26T15:54:10Z WARN [context: GRPC, requestID: d5rorphfs40s73bt9h20, accountID: REDACTED] management/internals/shared/grpc/server.go:354: recv job response error: rpc error: code = Canceled desc = context canceled

This suggests the gRPC stream/long-poll is being interrupted (possibly by reverse proxy / LB / HTTP2 settings / timeouts), because the client receives an HTTP 504 text/plain instead of a proper gRPC response.

To Reproduce
1. Use a self-hosted NetBird control plane (Management + Signal) behind HTTPS on :443.
2. Connect a macOS client to the network.
3. Run watch -n 2 'netbird status --detail' (or keep the client running and observe logs/status for several minutes).
4. Observe that approximately every ~2 minutes the Management connection briefly drops with:
• unexpected HTTP status code received from server: 504 (Gateway Timeout)
• received unexpected content-type "text/plain; charset=UTF-8"
5. At the same moment, the client emits “Network map updated”.

Expected behavior

Management gRPC calls (including /management.ManagementService/Job) should remain stable without periodic 504 Gateway Timeout responses. The client should not show recurring “Management: Disconnected” events or receive non-gRPC HTTP responses (text/plain) on a gRPC endpoint.

Are you using NetBird Cloud?

Self-hosted NetBird control plane.

NetBird version

Client:
• OS: darwin/arm64
• Daemon version: 0.64.1
• CLI version: 0.64.1

Server:
• OS: Ubuntu 24.04
• Daemon version: 0.64.1

Is any other VPN software installed?

No

Additional context

This is a self-hosted NetBird control plane, but Cloudflare is placed in between clients and the Management/Signal endpoints (clients connect to https://vpn.example.com:443 via Cloudflare, not directly to the origin).

Also, traefik is used on between the management and CloudFlare:

"POST /management.ManagementService/Job HTTP/2.0" 499 21 "-" "-" 54 "netbird-management@docker" "h2c://172.18.0.6:33073" 125017ms

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @maxpain on GitHub (Jan 26, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5184 **Describe the problem** In a self-hosted NetBird control-plane setup, the client intermittently disconnects from Management roughly every ~2 minutes when calling the gRPC endpoint /management.ManagementService/Job. At the same time the client logs an event “Network map updated”. When relay was unavailable, this behavior caused frequent/continuous loss of connectivity inside the VPN. After fixing relay, the VPN data-plane is stable, but the periodic Management 504 / gRPC transport failure still occurs and looks abnormal. Client error at the moment of failure: ``` Management: Disconnected, reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type "text/plain; charset=UTF-8" ``` On the management side, logs show: ``` 2026-01-26T15:54:10Z WARN [context: GRPC, requestID: d5rorphfs40s73bt9h20, accountID: REDACTED] management/internals/shared/grpc/server.go:354: recv job response error: rpc error: code = Canceled desc = context canceled ``` This suggests the gRPC stream/long-poll is being interrupted (possibly by reverse proxy / LB / HTTP2 settings / timeouts), because the client receives an HTTP 504 text/plain instead of a proper gRPC response. **To Reproduce** 1. Use a self-hosted NetBird control plane (Management + Signal) behind HTTPS on :443. 2. Connect a macOS client to the network. 3. Run watch -n 2 'netbird status --detail' (or keep the client running and observe logs/status for several minutes). 4. Observe that approximately every ~2 minutes the Management connection briefly drops with: • unexpected HTTP status code received from server: 504 (Gateway Timeout) • received unexpected content-type "text/plain; charset=UTF-8" 5. At the same moment, the client emits “Network map updated”. **Expected behavior** Management gRPC calls (including /management.ManagementService/Job) should remain stable without periodic 504 Gateway Timeout responses. The client should not show recurring “Management: Disconnected” events or receive non-gRPC HTTP responses (text/plain) on a gRPC endpoint. **Are you using NetBird Cloud?** Self-hosted NetBird control plane. **NetBird version** Client: • OS: darwin/arm64 • Daemon version: 0.64.1 • CLI version: 0.64.1 Server: • OS: Ubuntu 24.04 • Daemon version: 0.64.1 **Is any other VPN software installed?** No **Additional context** This is a self-hosted NetBird control plane, but **Cloudflare is placed in between clients and the Management/Signal endpoints** (clients connect to https://vpn.example.com:443 via Cloudflare, not directly to the origin). Also, traefik is used on between the management and CloudFlare: ``` "POST /management.ManagementService/Job HTTP/2.0" 499 21 "-" "-" 54 "netbird-management@docker" "h2c://172.18.0.6:33073" 125017ms ``` **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2026-08-05 01:24:25 -04:00
Author
Owner

@maxpain commented on GitHub (Jan 26, 2026):

Root Cause

The /management.ManagementService/Job streaming endpoint has no application-level keepalive mechanism. When there are no pending jobs, the gRPC stream sits completely idle with no data flowing.

While the server has gRPC keepalive configured (HTTP/2 PING frames), reverse proxies like Traefik, Nginx, and Cloudflare measure idle time based on HTTP/2 DATA frames, not PING frames. This causes timeouts:

Traefik: 504 after 60s (default readTimeout)
Cloudflare: Connection closed after ~100-120s (hard limit)

Suggested Fix

Add a periodic heartbeat (e.g., every 30 seconds) to sendJobsLoop that sends an empty JobRequest to keep the stream alive.

The client already handles empty messages gracefully (logs "received unknown or empty job request, skipping" and continues), so no client changes are needed.

Workaround

Self-hosted users can set Traefik's readTimeout to 0s, but this doesn't help with Cloudflare's hard limit — the code fix is necessary.

<!-- gh-comment-id:3800580722 --> @maxpain commented on GitHub (Jan 26, 2026): ## Root Cause The `/management.ManagementService/Job` streaming endpoint has no application-level keepalive mechanism. When there are no pending jobs, the gRPC stream sits completely idle with no data flowing. While the server has gRPC keepalive configured (HTTP/2 PING frames), reverse proxies like Traefik, Nginx, and Cloudflare measure idle time based on HTTP/2 DATA frames, not PING frames. This causes timeouts: Traefik: 504 after 60s (default readTimeout) Cloudflare: Connection closed after ~100-120s (hard limit) ## Suggested Fix Add a periodic heartbeat (e.g., every 30 seconds) to `sendJobsLoop` that sends an empty `JobRequest` to keep the stream alive. The client already handles empty messages gracefully (logs "received unknown or empty job request, skipping" and continues), so no client changes are needed. ## Workaround Self-hosted users can set Traefik's readTimeout to 0s, but this doesn't help with Cloudflare's hard limit — the code fix is necessary.
Author
Owner

@mlsmaycon commented on GitHub (Jan 26, 2026):

@maxpain can you please share your debug bundle?

You can generate one with:

netbird debug bundle --upload-bundle --anonymize
<!-- gh-comment-id:3800641815 --> @mlsmaycon commented on GitHub (Jan 26, 2026): @maxpain can you please share your debug bundle? You can generate one with: ``` netbird debug bundle --upload-bundle --anonymize ```
Author
Owner

@maxpain commented on GitHub (Jan 26, 2026):

@mlsmaycon --anonymize doesn't work as expected, netbirdConfig.signal.uri, peerConfig.fqdn, remotePeers.fqdn, Routes.NetID, and a lot of other fields are not replaced in the network_map.json file.

I'm not sure if I can send this because there is a high risk of confidential data leakage.

<!-- gh-comment-id:3800871966 --> @maxpain commented on GitHub (Jan 26, 2026): @mlsmaycon `--anonymize` doesn't work as expected, `netbirdConfig.signal.uri`, `peerConfig.fqdn`, `remotePeers.fqdn`, `Routes.NetID`, and a lot of other fields are not replaced in the `network_map.json` file. I'm not sure if I can send this because there is a high risk of confidential data leakage.
Author
Owner

@mlsmaycon commented on GitHub (Jan 26, 2026):

@mlsmaycon --anonymize doesn't work as expected, netbirdConfig.signal.uri, peerConfig.fqdn, remotePeers.fqdn, Routes.NetID, and a lot of other fields are not replaced in the network_map.json file.

I'm not sure if I can send this because there is a high risk of confidential data leakage.

Yes, it only focuses on public domains and IPs. We will need to check it, but this info won't be public if you are using the upload option. It should be accessible only to our development team.

<!-- gh-comment-id:3800935864 --> @mlsmaycon commented on GitHub (Jan 26, 2026): > [@mlsmaycon](https://github.com/mlsmaycon) `--anonymize` doesn't work as expected, `netbirdConfig.signal.uri`, `peerConfig.fqdn`, `remotePeers.fqdn`, `Routes.NetID`, and a lot of other fields are not replaced in the `network_map.json` file. > > I'm not sure if I can send this because there is a high risk of confidential data leakage. Yes, it only focuses on public domains and IPs. We will need to check it, but this info won't be public if you are using the upload option. It should be accessible only to our development team.
Author
Owner

@maxpain commented on GitHub (Jan 26, 2026):

@mlsmaycon
Upload file key:
f98118189f3e0eb811235eac2c17b72288e030ea127f97667dec22c4c40501c3/2ef3638c-b28f-4d06-89aa-937d1e9e15ca

<!-- gh-comment-id:3803074713 --> @maxpain commented on GitHub (Jan 26, 2026): @mlsmaycon Upload file key: f98118189f3e0eb811235eac2c17b72288e030ea127f97667dec22c4c40501c3/2ef3638c-b28f-4d06-89aa-937d1e9e15ca
Author
Owner

@Paulpatou commented on GitHub (Jan 30, 2026):

I'm having the same problem, no Cloudflare but Traefik yes
Here is my key for uploaded file:
c0d5d15b400b33c8d3ef4263d6b2904f74529ca53681bb2b5677e25714b62644/c262e7bb-cdac-4951-94a1-4f77dcfb5f37

Self-hosted
Client:
• OS: Ubuntu 24.04
• Daemon version: 0.64.3
• CLI version: 0.64.3

Server:
• OS: Ubuntu 24.04
• Daemon version: 0.64.3

<!-- gh-comment-id:3822469759 --> @Paulpatou commented on GitHub (Jan 30, 2026): I'm having the same problem, no Cloudflare but Traefik yes Here is my key for uploaded file: c0d5d15b400b33c8d3ef4263d6b2904f74529ca53681bb2b5677e25714b62644/c262e7bb-cdac-4951-94a1-4f77dcfb5f37 Self-hosted Client: • OS: Ubuntu 24.04 • Daemon version: 0.64.3 • CLI version: 0.64.3 Server: • OS: Ubuntu 24.04 • Daemon version: 0.64.3
Author
Owner

@pappz commented on GitHub (Jan 30, 2026):

Thank you for the logs. I will check why this can happen. Could you please send me the server-side logs from the same time period as the debug bundle?

<!-- gh-comment-id:3824316691 --> @pappz commented on GitHub (Jan 30, 2026): Thank you for the logs. I will check why this can happen. Could you please send me the server-side logs from the same time period as the debug bundle?
Author
Owner

@pappz commented on GitHub (Jan 30, 2026):

@maxpain From me looks like, your issue does not appear to be related only to the Job stream. It looks like a general issue with your proxy. I suggest checking the proxy configuration. Do you have a load balancer set up?

This relevant for the sync stream

2026-01-26T20:33:20+03:00 WARN shared/management/client/grpc.go:333: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

This is relevant for jobs stream

2026-01-26T20:42:20+03:00 WARN shared/management/client/grpc.go:202: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 524 (); transport: received unexpected content-type "text/plain; charset=UTF-8"

This is relevant for Signaling stream:

2026-01-26T20:18:06+03:00 ERRO shared/signal/client/grpc.go:416: Stream receive error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

This also a bad sign for the health check:

2026-01-26T20:07:37+03:00 ERRO shared/management/client/grpc.go:444: failed while getting Management Service public key: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/html"
<!-- gh-comment-id:3824363109 --> @pappz commented on GitHub (Jan 30, 2026): @maxpain From me looks like, your issue does not appear to be related only to the Job stream. It looks like a general issue with your proxy. I suggest checking the proxy configuration. Do you have a load balancer set up? This relevant for the sync stream ``` 2026-01-26T20:33:20+03:00 WARN shared/management/client/grpc.go:333: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR ``` This is relevant for jobs stream ``` 2026-01-26T20:42:20+03:00 WARN shared/management/client/grpc.go:202: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 524 (); transport: received unexpected content-type "text/plain; charset=UTF-8" ``` This is relevant for Signaling stream: ``` 2026-01-26T20:18:06+03:00 ERRO shared/signal/client/grpc.go:416: Stream receive error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR ``` This also a bad sign for the health check: ``` 2026-01-26T20:07:37+03:00 ERRO shared/management/client/grpc.go:444: failed while getting Management Service public key: rpc error: code = Unimplemented desc = unexpected HTTP status code received from server: 404 (Not Found); transport: received unexpected content-type "text/html" ```
Author
Owner

@maxpain commented on GitHub (Jan 30, 2026):

@maxpain From me looks like, your issue does not appear to be related only to the Job stream. It looks like a general issue with your proxy. I suggest checking the proxy configuration. Do you have a load balancer set up?

Hello. Please check my MR, I managed to fix that bug.

<!-- gh-comment-id:3824636818 --> @maxpain commented on GitHub (Jan 30, 2026): > [@maxpain](https://github.com/maxpain) From me looks like, your issue does not appear to be related only to the Job stream. It looks like a general issue with your proxy. I suggest checking the proxy configuration. Do you have a load balancer set up? Hello. Please check [my MR](https://github.com/netbirdio/netbird/pull/5185), I managed to fix that bug.
Author
Owner

@pappz commented on GitHub (Jan 31, 2026):

@maxpain yes, I saw it and I understand it fixes your current issue, but keeping a gRPC connection alive is not an application-layer responsibility.

We have three gRPC-related streams: Signal and Management (network map sync, jobs). All of these depend on the load balancer or reverse proxy. If any of them is unstable, and based on your logs it looks like they are, you likely have issues with all of them. In that case, it’s only a matter of time before you face new symptoms.

Anyway, I will double check how we handle connectivity errors around the job stream. The sync and job streams depend on the same Management server instance. When either of them disconnects, we automatically disconnect the other stream as well, because we can assume they come from the same Management server connection.

And please answer for my question, "do you have a load balancer set up"?

<!-- gh-comment-id:3827925821 --> @pappz commented on GitHub (Jan 31, 2026): @maxpain yes, I saw it and I understand it fixes your current issue, but keeping a gRPC connection alive is not an application-layer responsibility. We have three gRPC-related streams: Signal and Management (network map sync, jobs). All of these depend on the load balancer or reverse proxy. If any of them is unstable, and based on your logs it looks like they are, you likely have issues with all of them. In that case, it’s only a matter of time before you face new symptoms. Anyway, I will double check how we handle connectivity errors around the job stream. The sync and job streams depend on the same Management server instance. When either of them disconnects, we automatically disconnect the other stream as well, because we can assume they come from the same Management server connection. And please answer for my question, "do you have a load balancer set up"?
Author
Owner

@maxpain commented on GitHub (Jan 31, 2026):

And please answer for my question, "do you have a load balancer set up"?

Yes.
Here is my schema:
NetBird Client → CloudFlare → Traefik → Management
While the server has gRPC keepalive configured (HTTP/2 PING frames), reverse proxies like Cloudflare measure idle time based on HTTP/2 DATA frames, not PING frames.
The problem is that CloudFlare has a hard limit on request timeout (100-120s) and I can't disable it

<!-- gh-comment-id:3828152961 --> @maxpain commented on GitHub (Jan 31, 2026): > And please answer for my question, "do you have a load balancer set up"? Yes. Here is my schema: NetBird Client → CloudFlare → Traefik → Management While the server has gRPC keepalive configured (HTTP/2 PING frames), reverse proxies like Cloudflare measure idle time based on HTTP/2 DATA frames, not PING frames. The problem is that CloudFlare has a hard limit on request timeout (100-120s) and I can't disable it
Author
Owner

@Paulpatou commented on GitHub (Feb 2, 2026):

Thank you for the logs. I will check why this can happen. Could you please send me the server-side logs from the same time period as the debug bundle?

I no longer have the logs from the same time, but they are always the same. Here they are:


netbird-management  | 2026-01-30T10:28:32Z WARN [context: GRPC, requestID: d5u8f54p69qs73act1ag, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:28:34Z WARN [context: GRPC, requestID: d5u8f5kp69qs73act1dg, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:29:46Z WARN [requestID: d5u8fnkp69qs73act2j0, accountID: d5st0a4p69qs73euk3t0, context: GRPC] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:29:47Z WARN [context: GRPC, requestID: d5u8fnsp69qs73act2kg, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:30:12Z WARN [requestID: d5u8fu4p69qs73act340, accountID: d5st0a4p69qs73euk3t0, context: GRPC] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:31:00Z WARN [context: GRPC, requestID: d5u8ga4p69qs73act480, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:31:05Z WARN [context: GRPC, requestID: d5u8gbcp69qs73act4d0, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:31:13Z WARN [context: GRPC, requestID: d5u8gdcp69qs73act4j0, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:32:09Z WARN [context: GRPC, requestID: d5u8grcp69qs73act5pg, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:32:13Z WARN [requestID: d5u8gscp69qs73act5tg, accountID: d5st0a4p69qs73euk3t0, context: GRPC] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-management  | 2026-01-30T10:32:18Z WARN [context: GRPC, requestID: d5u8gtkp69qs73act61g, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled

<!-- gh-comment-id:3833397477 --> @Paulpatou commented on GitHub (Feb 2, 2026): > Thank you for the logs. I will check why this can happen. Could you please send me the server-side logs from the same time period as the debug bundle? I no longer have the logs from the same time, but they are always the same. Here they are: ``` netbird-management | 2026-01-30T10:28:32Z WARN [context: GRPC, requestID: d5u8f54p69qs73act1ag, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:28:34Z WARN [context: GRPC, requestID: d5u8f5kp69qs73act1dg, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:29:46Z WARN [requestID: d5u8fnkp69qs73act2j0, accountID: d5st0a4p69qs73euk3t0, context: GRPC] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:29:47Z WARN [context: GRPC, requestID: d5u8fnsp69qs73act2kg, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:30:12Z WARN [requestID: d5u8fu4p69qs73act340, accountID: d5st0a4p69qs73euk3t0, context: GRPC] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:31:00Z WARN [context: GRPC, requestID: d5u8ga4p69qs73act480, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:31:05Z WARN [context: GRPC, requestID: d5u8gbcp69qs73act4d0, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:31:13Z WARN [context: GRPC, requestID: d5u8gdcp69qs73act4j0, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:32:09Z WARN [context: GRPC, requestID: d5u8grcp69qs73act5pg, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:32:13Z WARN [requestID: d5u8gscp69qs73act5tg, accountID: d5st0a4p69qs73euk3t0, context: GRPC] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled netbird-management | 2026-01-30T10:32:18Z WARN [context: GRPC, requestID: d5u8gtkp69qs73act61g, accountID: d5st0a4p69qs73euk3t0] management/internals/shared/grpc/server.go:359: recv job response error: rpc error: code = Canceled desc = context canceled ```
Author
Owner

@pappz commented on GitHub (Feb 2, 2026):

@maxpain could you create a debug bundle using your branch? I’d like to see how it operates in your backend infrastructure. Please enable the debug log level, let it run for 30–40 minutes, and then create the debug bundle.

<!-- gh-comment-id:3834582683 --> @pappz commented on GitHub (Feb 2, 2026): @maxpain could you create a debug bundle using your branch? I’d like to see how it operates in your backend infrastructure. Please enable the debug log level, let it run for 30–40 minutes, and then create the debug bundle.
Author
Owner

@pappz commented on GitHub (Feb 12, 2026):

Could you run a test with these settings in Traefik?

  - "--entrypoints.websecure.transport.respondingTimeouts.readTimeout=0s"
  - "--entrypoints.websecure.transport.respondingTimeouts.writeTimeout=0s"
<!-- gh-comment-id:3892170599 --> @pappz commented on GitHub (Feb 12, 2026): Could you run a test with these settings in Traefik? ``` - "--entrypoints.websecure.transport.respondingTimeouts.readTimeout=0s" - "--entrypoints.websecure.transport.respondingTimeouts.writeTimeout=0s" ```
Author
Owner

@maxpain commented on GitHub (Feb 12, 2026):

Could you run a test with these settings in Traefik?

  - "--entrypoints.websecure.transport.respondingTimeouts.readTimeout=0s"
  - "--entrypoints.websecure.transport.respondingTimeouts.writeTimeout=0s"

I already have this.

providers:
  docker:
    exposedByDefault: false
  file:
    directory: /etc/traefik/dynamic
    watch: true

accessLog:
  format: common

entryPoints:
  websecure:
    address: ":443"
    asDefault: true
    http:
      tls: {}
    transport:
      respondingTimeouts:
        idleTimeout: "0s"
        readTimeout: "0s"
<!-- gh-comment-id:3892304024 --> @maxpain commented on GitHub (Feb 12, 2026): > Could you run a test with these settings in Traefik? > > ``` > - "--entrypoints.websecure.transport.respondingTimeouts.readTimeout=0s" > - "--entrypoints.websecure.transport.respondingTimeouts.writeTimeout=0s" > ``` I already have this. ```yaml providers: docker: exposedByDefault: false file: directory: /etc/traefik/dynamic watch: true accessLog: format: common entryPoints: websecure: address: ":443" asDefault: true http: tls: {} transport: respondingTimeouts: idleTimeout: "0s" readTimeout: "0s" ```
Author
Owner

@maxpain commented on GitHub (Feb 12, 2026):

Just added writeTimeout=0s, nothing changed.
As I said before, we can't do anything on the Traefik side.

The problem is that CloudFlare has a hard limit on request timeout (100-120s) and I can't disable it

<!-- gh-comment-id:3892316212 --> @maxpain commented on GitHub (Feb 12, 2026): Just added `writeTimeout=0s`, nothing changed. As I said before, we can't do anything on the Traefik side. > The problem is that CloudFlare has a hard limit on request timeout (100-120s) and I can't disable it
Author
Owner

@ghazyami commented on GitHub (Feb 15, 2026):

I am experiencing the same.
I am using traefik as reverse proxy directly, no cloudflare.
The symptom that got me to investigate in the first place is that the android client after connecting shows connected for about a minute, then switches to connecting however the number of connected peers remains as it is, I am also able to access other peers and network resources even tho it's showing connecting.

After trying disabling timeout by setting the bellow on traefik configs, I no longer see this behavior on android client, and also do not see the log message on management container logs.

idleTimeout: "0s"
readTimeout: "0s"
writeTimeout: "0s"
<!-- gh-comment-id:3904554927 --> @ghazyami commented on GitHub (Feb 15, 2026): I am experiencing the same. I am using traefik as reverse proxy directly, no cloudflare. The symptom that got me to investigate in the first place is that the android client after connecting shows `connected` for about a minute, then switches to `connecting` however the number of connected peers remains as it is, I am also able to access other peers and network resources even tho it's showing `connecting`. After trying disabling timeout by setting the bellow on traefik configs, I no longer see this behavior on android client, and also do not see the log message on `management` container logs. ``` idleTimeout: "0s" readTimeout: "0s" writeTimeout: "0s" ```
Author
Owner

@JimmyKmi commented on GitHub (Mar 6, 2026):

I am experiencing the same. I am using traefik as reverse proxy directly, no cloudflare. The symptom that got me to investigate in the first place is that the android client after connecting shows connected for about a minute, then switches to connecting however the number of connected peers remains as it is, I am also able to access other peers and network resources even tho it's showing connecting.

After trying disabling timeout by setting the bellow on traefik configs, I no longer see this behavior on android client, and also do not see the log message on management container logs.

idleTimeout: "0s"
readTimeout: "0s"
writeTimeout: "0s"

I also experienced this issue. After reviewing the relevant GitHub issue, the openclaw tool suggested adding the following configuration to Traefik to disable timeouts:

traefik:
  command:
    # ... other configurations ...
    # Key: Disable timeouts
    - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=0"
    - "--entryPoints.websecure.transport.respondingTimeouts.writeTimeout=0"
    - "--entryPoints.websecure.transport.respondingTimeouts.idleTimeout=0"

Upon implementing this configuration, the problem was solved immediately, which was very effective.

<!-- gh-comment-id:4014234894 --> @JimmyKmi commented on GitHub (Mar 6, 2026): > I am experiencing the same. I am using traefik as reverse proxy directly, no cloudflare. The symptom that got me to investigate in the first place is that the android client after connecting shows `connected` for about a minute, then switches to `connecting` however the number of connected peers remains as it is, I am also able to access other peers and network resources even tho it's showing `connecting`. > > After trying disabling timeout by setting the bellow on traefik configs, I no longer see this behavior on android client, and also do not see the log message on `management` container logs. > > ``` > idleTimeout: "0s" > readTimeout: "0s" > writeTimeout: "0s" > ``` I also experienced this issue. After reviewing the relevant GitHub issue, the openclaw tool suggested adding the following configuration to Traefik to disable timeouts: ```yaml traefik: command: # ... other configurations ... # Key: Disable timeouts - "--entryPoints.websecure.transport.respondingTimeouts.readTimeout=0" - "--entryPoints.websecure.transport.respondingTimeouts.writeTimeout=0" - "--entryPoints.websecure.transport.respondingTimeouts.idleTimeout=0" ``` Upon implementing this configuration, the problem was solved immediately, which was very effective.
Author
Owner

@krishne35 commented on GitHub (Mar 17, 2026):

transport: respondingTimeouts: idleTimeout: "0s" readTimeout: "0s"

Setting these parameters on traefik won't have any security risk implementations?
As it's the only current working way to keep netbird proxy to keep working?

<!-- gh-comment-id:4077529379 --> @krishne35 commented on GitHub (Mar 17, 2026): `transport: respondingTimeouts: idleTimeout: "0s" readTimeout: "0s"` Setting these parameters on traefik won't have any security risk implementations? As it's the only current working way to keep netbird proxy to keep working?
Author
Owner

@alexmge commented on GitHub (Mar 30, 2026):

Just a quick note to bring back this issue into light.
I have the same setup as @maxpain and have the exact same issue, frequent disconnections from the Management and Signal endpoints occuring every minute or so.

The setup is Client -> Cloudflare -> Traefik -> Netbird server

Netbird server logs (v0.66.4) :

netbird-server     | 2026-03-30T12:42:22.002Z WARN [context: GRPC, requestID: d756usiqsq0s73aoq71g, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:42:39.596Z WARN [accountID: d6ulgqaqsq0s73aistog, context: GRPC, requestID: d756v0qqsq0s73aoq780] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:43:40.652Z WARN [context: GRPC, requestID: d756vg2qsq0s73aoq7sg, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:43:43.028Z WARN [accountID: d6ulgqaqsq0s73aistog, context: GRPC, requestID: d756vgqqsq0s73aoq7u0] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:44:57.756Z WARN [context: GRPC, requestID: d75703aqsq0s73aoq8g0, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:45:01.001Z WARN [context: GRPC, requestID: d75704aqsq0s73aoq8hg, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:46:16.397Z WARN [context: GRPC, requestID: d7570n2qsq0s73aoq940, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:46:41.037Z WARN [accountID: d6ulgqaqsq0s73aistog, context: GRPC, requestID: d7570taqsq0s73aoq9fg] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:47:34.050Z WARN [context: GRPC, requestID: d7571aiqsq0s73aoq9s0, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:47:53.854Z WARN [context: GRPC, requestID: d7571faqsq0s73aoqa2g, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:48:37.460Z WARN [context: GRPC, requestID: d7571qaqsq0s73aoqaeg, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:49:42.479Z WARN [context: GRPC, requestID: d7572aiqsq0s73aoqb50, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
netbird-server     | 2026-03-30T12:49:48.738Z WARN [context: GRPC, requestID: d7572c2qsq0s73aoqb7g, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled
<!-- gh-comment-id:4154883003 --> @alexmge commented on GitHub (Mar 30, 2026): Just a quick note to bring back this issue into light. I have the same setup as @maxpain and have the exact same issue, frequent disconnections from the Management and Signal endpoints occuring every minute or so. The setup is `Client -> Cloudflare -> Traefik -> Netbird server` Netbird server logs (v0.66.4) : ```log netbird-server | 2026-03-30T12:42:22.002Z WARN [context: GRPC, requestID: d756usiqsq0s73aoq71g, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:42:39.596Z WARN [accountID: d6ulgqaqsq0s73aistog, context: GRPC, requestID: d756v0qqsq0s73aoq780] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:43:40.652Z WARN [context: GRPC, requestID: d756vg2qsq0s73aoq7sg, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:43:43.028Z WARN [accountID: d6ulgqaqsq0s73aistog, context: GRPC, requestID: d756vgqqsq0s73aoq7u0] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:44:57.756Z WARN [context: GRPC, requestID: d75703aqsq0s73aoq8g0, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:45:01.001Z WARN [context: GRPC, requestID: d75704aqsq0s73aoq8hg, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:46:16.397Z WARN [context: GRPC, requestID: d7570n2qsq0s73aoq940, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:46:41.037Z WARN [accountID: d6ulgqaqsq0s73aistog, context: GRPC, requestID: d7570taqsq0s73aoq9fg] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:47:34.050Z WARN [context: GRPC, requestID: d7571aiqsq0s73aoq9s0, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:47:53.854Z WARN [context: GRPC, requestID: d7571faqsq0s73aoqa2g, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:48:37.460Z WARN [context: GRPC, requestID: d7571qaqsq0s73aoqaeg, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:49:42.479Z WARN [context: GRPC, requestID: d7572aiqsq0s73aoqb50, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled netbird-server | 2026-03-30T12:49:48.738Z WARN [context: GRPC, requestID: d7572c2qsq0s73aoqb7g, accountID: d6ulgqaqsq0s73aistog] management/internals/shared/grpc/server.go:369: recv job response error: rpc error: code = Canceled desc = context canceled ```
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#10040