SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-05-10 02:02:26 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Netbird behind nginx: stream terminated by RST_STREAM #266

New Issue
Open
opened 2025-11-20 05:08:48 -05:00 by saavagebueno · 11 comments
saavagebueno commented 2025-11-20 05:08:48 -05:00
Owner
Copy Link

Originally created by @ykorzikowski on GitHub (Jan 11, 2023).

Hey there,

I am successfully running netbird since couple of months. Some clients cant speak with each other. I am investigating this issue right now and found, that I have errors regarding the signal-service in my client-logs (all clients, also the working ones):

time="2023-01-11T09:50:08+01:00" level=warning msg="disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR" file="grpc.go:144"
time="2023-01-11T09:50:23+01:00" level=info msg="connected to the Signal Service stream" file="grpc.go:136"
time="2023-01-11T09:51:08+01:00" level=warning msg="disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR" file="grpc.go:144"
time="2023-01-11T09:51:23+01:00" level=info msg="connected to the Signal Service stream" file="grpc.go:136"

server{
    listen *:80;
    listen [::]:80;
    server_name  ganymede.redacted.com;

    location / {
        return 301 https://$server_name$request_uri;
    }
}

# Dashboard

server {
    include security.conf;
    listen *:443 ssl http2;
    listen [::]:443 ssl http2;
    server_name ganymede.redacted.com;
    ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem;

    access_log /var/log/nginx/ganymede.redacted.com/netbird.log;
    error_log /var/log/nginx/ganymede.redacted.com/err/netbird.log;

    location / {
        proxy_pass http://10.10.11.199:8080;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Scheme $scheme;
	proxy_set_header        X-Forwarded-Proto https;
        proxy_set_header        X-Forwarded-Host ganymede.redacted.com;
    }
}

# MGMT HTTPS API

server {
    include security.conf;
    listen *:4443 ssl http2;
    listen [::]:4443 ssl http2;
    server_name ganymede.redacted.com;
    ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem;

    access_log /var/log/nginx/ganymede.redacted.com/netbird-api.log;
    error_log /var/log/nginx/ganymede.redacted.com/err/netbird-api.log;

    location / {
        proxy_pass http://10.10.11.199:8081;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Scheme $scheme;
	proxy_set_header        X-Forwarded-Proto https;
        proxy_set_header        X-Forwarded-Host ganymede.redacted.com;
    }
}

# MGMT GRPC API

server {
    include security.conf;
    listen *:33073 ssl http2;
    listen [::]:33073 ssl http2;
    server_name ganymede.redacted.com;
    ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem;

    access_log /var/log/nginx/ganymede.redacted.com/netbird-grpc.log;
    error_log /var/log/nginx/ganymede.redacted.com/err/netbird-grpc.log;

    location / {
        grpc_pass grpc://10.10.11.199:33074;
        grpc_read_timeout 3600s;
    }
}

# Signal GRPC API

server {
    include security.conf;
    listen *:10000 ssl http2;
    listen [::]:10000 ssl http2;
    server_name ganymede.redacted.com;
    ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem;

    access_log /var/log/nginx/ganymede.redacted.com/netbird-signal.log;
    error_log /var/log/nginx/ganymede.redacted.com/err/netbird-signal.log;

    location / {
        grpc_pass grpc://10.10.11.199:10001;
        grpc_read_timeout 3600s;
	grpc_ssl_verify off;
    }
}

Is there a recommendation how to run the netbird service? Like my config or everythin on port 443 like https://github.com/netbirdio/netbird/issues/536 ?

Thank you for any help :)

Originally created by @ykorzikowski on GitHub (Jan 11, 2023). Hey there, I am successfully running netbird since couple of months. Some clients cant speak with each other. I am investigating this issue right now and found, that I have errors regarding the signal-service in my client-logs (all clients, also the working ones): ``` time="2023-01-11T09:50:08+01:00" level=warning msg="disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR" file="grpc.go:144" time="2023-01-11T09:50:23+01:00" level=info msg="connected to the Signal Service stream" file="grpc.go:136" time="2023-01-11T09:51:08+01:00" level=warning msg="disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR" file="grpc.go:144" time="2023-01-11T09:51:23+01:00" level=info msg="connected to the Signal Service stream" file="grpc.go:136" ``` ``` server{ listen *:80; listen [::]:80; server_name ganymede.redacted.com; location / { return 301 https://$server_name$request_uri; } } # Dashboard server { include security.conf; listen *:443 ssl http2; listen [::]:443 ssl http2; server_name ganymede.redacted.com; ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem; access_log /var/log/nginx/ganymede.redacted.com/netbird.log; error_log /var/log/nginx/ganymede.redacted.com/err/netbird.log; location / { proxy_pass http://10.10.11.199:8080; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Scheme $scheme; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-Host ganymede.redacted.com; } } # MGMT HTTPS API server { include security.conf; listen *:4443 ssl http2; listen [::]:4443 ssl http2; server_name ganymede.redacted.com; ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem; access_log /var/log/nginx/ganymede.redacted.com/netbird-api.log; error_log /var/log/nginx/ganymede.redacted.com/err/netbird-api.log; location / { proxy_pass http://10.10.11.199:8081; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Scheme $scheme; proxy_set_header X-Forwarded-Proto https; proxy_set_header X-Forwarded-Host ganymede.redacted.com; } } # MGMT GRPC API server { include security.conf; listen *:33073 ssl http2; listen [::]:33073 ssl http2; server_name ganymede.redacted.com; ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem; access_log /var/log/nginx/ganymede.redacted.com/netbird-grpc.log; error_log /var/log/nginx/ganymede.redacted.com/err/netbird-grpc.log; location / { grpc_pass grpc://10.10.11.199:33074; grpc_read_timeout 3600s; } } # Signal GRPC API server { include security.conf; listen *:10000 ssl http2; listen [::]:10000 ssl http2; server_name ganymede.redacted.com; ssl_certificate /etc/letsencrypt/live/ganymede.redacted.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ganymede.redacted.com/privkey.pem; access_log /var/log/nginx/ganymede.redacted.com/netbird-signal.log; error_log /var/log/nginx/ganymede.redacted.com/err/netbird-signal.log; location / { grpc_pass grpc://10.10.11.199:10001; grpc_read_timeout 3600s; grpc_ssl_verify off; } } ``` Is there a recommendation how to run the netbird service? Like my config or everythin on port 443 like https://github.com/netbirdio/netbird/issues/536 ? Thank you for any help :)
saavagebueno added the bugclientmanagement-service labels 2025-11-20 05:08:48 -05:00
saavagebueno commented 2025-11-20 05:08:49 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Jun 16, 2023):

This issue is caused by grpc and its native keepalive not being supported by revere-proxy. We are implementing a application keepalive tin #771

@mlsmaycon commented on GitHub (Jun 16, 2023): This issue is caused by grpc and its native keepalive not being supported by revere-proxy. We are implementing a application keepalive tin #771
saavagebueno commented 2025-11-20 05:08:49 -05:00
Author
Owner
Copy Link

@ykorzikowski commented on GitHub (Jun 22, 2023):

Just for reference: Found this: https://github.com/camunda-community-hub/zeebe-client-node-js/issues/101

@ykorzikowski commented on GitHub (Jun 22, 2023): Just for reference: Found this: https://github.com/camunda-community-hub/zeebe-client-node-js/issues/101
saavagebueno commented 2025-11-20 05:08:49 -05:00
Author
Owner
Copy Link

@mlsmaycon commented on GitHub (Jul 14, 2023):

@ykorzikowski, we notice that running Nginx v1.25.1 solved the issue. Can you test it?

@mlsmaycon commented on GitHub (Jul 14, 2023): @ykorzikowski, we notice that running Nginx v1.25.1 solved the issue. Can you test it?
saavagebueno commented 2025-11-20 05:08:49 -05:00
Author
Owner
Copy Link

@ykorzikowski commented on GitHub (Jul 25, 2023):

Hellooo,

I dont know. What helped is setting grpc_read_timeout 3600s; to 1 hour.

I am still using nginx version: nginx/1.18.0 and did not notice this issue since tweaking my config with above parameter.

@ykorzikowski commented on GitHub (Jul 25, 2023): Hellooo, I dont know. What helped is setting `grpc_read_timeout 3600s;` to 1 hour. I am still using `nginx version: nginx/1.18.0` and did not notice this issue since tweaking my config with above parameter.
saavagebueno commented 2025-11-20 05:08:49 -05:00
Author
Owner
Copy Link

@devopskupryk commented on GitHub (Sep 27, 2023):

Hello,

I'm trying to deploy the netbird management into my k8s cluster v.1.25.12 with the nginx ingress controller 1.19.10

Have specified the annotations in the netbird management and netbird signal ingresses:

      nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
      nginx.ingress.kubernetes.io/configuration-snippet: |
        allow all;
        grpc_read_timeout 3600s;
        grpc_send_timeout 3600s;
        grpc_socket_keepalive on;

Peers are registered and the netbird client connected but vpn doesn't work:

$ netbird status
Daemon version: 0.22.7
CLI version: 0.22.7
Management: Connected
Signal: Connected
FQDN: test.hidden.tech
NetBird IP: 100.77.11.161/16
Interface type: Kernel
Peers count: 0/5 Connected

sudo tail -n 30 /var/log/netbird/client.log
2023-09-27T12:08:47+03:00 WARN client/server/server.go:226: canceling previous waiting execution
2023-09-27T12:09:34+03:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2023-09-27T12:09:35+03:00 INFO client/internal/wgproxy/proxy_ebpf.go:79: local wg proxy listening on: 3128
2023-09-27T12:09:35+03:00 INFO iface/tun_linux.go:15: create tun interface with kernel WireGuard support: wt0
2023-09-27T12:09:50+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2023-09-27T12:09:50+03:00 INFO client/internal/connect.go:179: Netbird engine started, my IP is: 100.77.11.161/16
2023-09-27T12:09:50+03:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2023-09-27T12:09:50+03:00 INFO client/internal/dns/systemd_linux.go:135: adding 1 search domains and 0 match domains. Search list: [vpn.stage.heddin.tech] , Match list: []
2023-09-27T12:09:50+03:00 INFO client/internal/acl/manager.go:67: ACL rules processed in: 1.365082ms, total rules count: 2
2023-09-27T12:10:35+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
2023-09-27T12:10:50+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2023-09-27T12:11:35+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
2023-09-27T12:11:48+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2023-09-27T12:12:01+03:00 INFO client/internal/dns/systemd_linux.go:135: adding 1 search domains and 0 match domains. Search list: [vpn.stage.heddin.tech] , Match list: []
2023-09-27T12:12:01+03:00 INFO client/internal/acl/manager.go:67: ACL rules processed in: 461.906µs, total rules count: 2
2023-09-27T12:12:36+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR
2023-09-27T12:12:40+03:00 WARN signal/client/grpc.go:151: disconnected from the Signal Exchange due to an error: rpc error: code = Unavailable desc = connection error: desc = "error reading from server: read tcp 192.168.1.100:59684->hidden:443: read: connection timed out"
2023-09-27T12:12:41+03:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within `timeout`

How should I additionally configure my nginx ingress controller and\or ingress resources to solve this issue?

PS. I can't add additionaly:
grpc_pass grpc://$service_name:$service_port;
because nginx config already has:
grpc_pass grpc://$upstream_balancer;
and I get error when try:

Error: cannot patch "netbird-management" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: 
 nginx: [emerg] "grpc_pass" directive is duplicate in /tmp/nginx/nginx-cfg1131325741:4731
 nginx: configuration file /tmp/nginx/nginx-cfg1131325741 test failed
@devopskupryk commented on GitHub (Sep 27, 2023): Hello, I'm trying to deploy the netbird management into my k8s cluster v.1.25.12 with the nginx ingress controller 1.19.10 Have specified the annotations in the netbird management and netbird signal ingresses: ``` nginx.ingress.kubernetes.io/backend-protocol: "GRPC" nginx.ingress.kubernetes.io/configuration-snippet: | allow all; grpc_read_timeout 3600s; grpc_send_timeout 3600s; grpc_socket_keepalive on; ``` Peers are registered and the netbird client connected but vpn doesn't work: ``` $ netbird status Daemon version: 0.22.7 CLI version: 0.22.7 Management: Connected Signal: Connected FQDN: test.hidden.tech NetBird IP: 100.77.11.161/16 Interface type: Kernel Peers count: 0/5 Connected ``` ``` sudo tail -n 30 /var/log/netbird/client.log 2023-09-27T12:08:47+03:00 WARN client/server/server.go:226: canceling previous waiting execution 2023-09-27T12:09:34+03:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service 2023-09-27T12:09:35+03:00 INFO client/internal/wgproxy/proxy_ebpf.go:79: local wg proxy listening on: 3128 2023-09-27T12:09:35+03:00 INFO iface/tun_linux.go:15: create tun interface with kernel WireGuard support: wt0 2023-09-27T12:09:50+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream 2023-09-27T12:09:50+03:00 INFO client/internal/connect.go:179: Netbird engine started, my IP is: 100.77.11.161/16 2023-09-27T12:09:50+03:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2023-09-27T12:09:50+03:00 INFO client/internal/dns/systemd_linux.go:135: adding 1 search domains and 0 match domains. Search list: [vpn.stage.heddin.tech] , Match list: [] 2023-09-27T12:09:50+03:00 INFO client/internal/acl/manager.go:67: ACL rules processed in: 1.365082ms, total rules count: 2 2023-09-27T12:10:35+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR 2023-09-27T12:10:50+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream 2023-09-27T12:11:35+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR 2023-09-27T12:11:48+03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream 2023-09-27T12:12:01+03:00 INFO client/internal/dns/systemd_linux.go:135: adding 1 search domains and 0 match domains. Search list: [vpn.stage.heddin.tech] , Match list: [] 2023-09-27T12:12:01+03:00 INFO client/internal/acl/manager.go:67: ACL rules processed in: 461.906µs, total rules count: 2 2023-09-27T12:12:36+03:00 WARN signal/client/grpc.go:170: disconnected from the Signal service but will retry silently. Reason: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: PROTOCOL_ERROR 2023-09-27T12:12:40+03:00 WARN signal/client/grpc.go:151: disconnected from the Signal Exchange due to an error: rpc error: code = Unavailable desc = connection error: desc = "error reading from server: read tcp 192.168.1.100:59684->hidden:443: read: connection timed out" 2023-09-27T12:12:41+03:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = keepalive ping failed to receive ACK within `timeout` ``` How should I additionally configure my nginx ingress controller and\or ingress resources to solve this issue? PS. I can't add additionaly: `grpc_pass grpc://$service_name:$service_port;` because nginx config already has: `grpc_pass grpc://$upstream_balancer;` and I get error when try: ``` Error: cannot patch "netbird-management" with kind Ingress: admission webhook "validate.nginx.ingress.kubernetes.io" denied the request: nginx: [emerg] "grpc_pass" directive is duplicate in /tmp/nginx/nginx-cfg1131325741:4731 nginx: configuration file /tmp/nginx/nginx-cfg1131325741 test failed ```
saavagebueno commented 2025-11-20 05:08:49 -05:00
Author
Owner
Copy Link

@devopskupryk commented on GitHub (Oct 3, 2023):

Any news?

@devopskupryk commented on GitHub (Oct 3, 2023): Any news?
saavagebueno commented 2025-11-20 05:08:49 -05:00
Author
Owner
Copy Link

@ykorzikowski commented on GitHub (Oct 4, 2023):

Can you post some nginx access / error logs? Maybe this is some ingress configuration issue and noting regarding netbird.

@ykorzikowski commented on GitHub (Oct 4, 2023): Can you post some nginx access / error logs? Maybe this is some ingress configuration issue and noting regarding netbird.
saavagebueno commented 2025-11-20 05:08:50 -05:00
Author
Owner
Copy Link

@devopskupryk commented on GitHub (Oct 6, 2023):

Nginx ingress controllers' logs don't contain any error:

~ % stern nginx -i netbird
+ nginx-ingress-ingress-nginx-controller-ssvsd › controller
+ nginx-ingress-ingress-nginx-controller-kc929 › controller
+ nginx-ingress-ingress-nginx-controller-znnks › controller
+ nginx-ingress-ingress-nginx-controller-dsphr › controller
+ nginx-ingress-ingress-nginx-controller-qlglz › controller
+ nginx-ingress-ingress-nginx-controller-6jqht › controller
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:22 +0000] "POST /management.ManagementService/GetServerKey HTTP/2.0" 200 61 "-" "grpc-go/1.55.0" 118 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 91 0.004 200 d93bf7b778f62315eddbce6d13e36540
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:22 +0000] "POST /management.ManagementService/GetPKCEAuthorizationFlow HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 149 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 67 0.000 200 55a5a3f501b501a66b96cd41b7a64dec
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:27 +0000] "POST /management.ManagementService/GetServerKey HTTP/2.0" 200 61 "-" "grpc-go/1.55.0" 118 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 91 0.004 200 2f1504ba52578c47c42aa625646c24b5
nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:27 +0000] "POST /management.ManagementService/Login HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 352 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 88 0.000 200 3739e538f1cc0de4a30661be8e30c8e8
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:56 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 254 0.012 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.008 200 320af0b7fa14deb408df7a9655d6894e
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:56 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 c9b0464002cf4a0c99090ae3e4dd965b
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:57 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 8ce398045d021141db56fe2d7ab0a40b
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:36 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 89850818ca10896a4cd86d91ca4b0111
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:36 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 97ec32beb45da1830ee71e7062260acf
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:40 +0000] "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 154 60.000 [netbird-netbird-signal-80] [] 10.244.3.157:80 187 60.000 200 33cd1401fdfed21bbafd556ef35317db
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:06 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.006 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.008 200 5a77a24d9402d063a6a4ecc4a2c99929
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:15 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.007 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 1d285753adebe4f1647259ea2d60020c
nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:15 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 9fbd16a3312440bcc847bc6f2c93be4e
@devopskupryk commented on GitHub (Oct 6, 2023): Nginx ingress controllers' logs don't contain any error: ``` ~ % stern nginx -i netbird + nginx-ingress-ingress-nginx-controller-ssvsd › controller + nginx-ingress-ingress-nginx-controller-kc929 › controller + nginx-ingress-ingress-nginx-controller-znnks › controller + nginx-ingress-ingress-nginx-controller-dsphr › controller + nginx-ingress-ingress-nginx-controller-qlglz › controller + nginx-ingress-ingress-nginx-controller-6jqht › controller nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:22 +0000] "POST /management.ManagementService/GetServerKey HTTP/2.0" 200 61 "-" "grpc-go/1.55.0" 118 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 91 0.004 200 d93bf7b778f62315eddbce6d13e36540 nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:22 +0000] "POST /management.ManagementService/GetPKCEAuthorizationFlow HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 149 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 67 0.000 200 55a5a3f501b501a66b96cd41b7a64dec nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:27 +0000] "POST /management.ManagementService/GetServerKey HTTP/2.0" 200 61 "-" "grpc-go/1.55.0" 118 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 91 0.004 200 2f1504ba52578c47c42aa625646c24b5 nginx-ingress-ingress-nginx-controller-qlglz controller ip-hidden - - [06/Oct/2023:07:43:27 +0000] "POST /management.ManagementService/Login HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 352 0.002 [netbird-netbird-management-80] [] 10.244.3.224:80 88 0.000 200 3739e538f1cc0de4a30661be8e30c8e8 nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:56 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 254 0.012 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.008 200 320af0b7fa14deb408df7a9655d6894e nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:56 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 c9b0464002cf4a0c99090ae3e4dd965b nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:43:57 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 8ce398045d021141db56fe2d7ab0a40b nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:36 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 89850818ca10896a4cd86d91ca4b0111 nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:36 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 97ec32beb45da1830ee71e7062260acf nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:44:40 +0000] "POST /signalexchange.SignalExchange/ConnectStream HTTP/2.0" 200 0 "-" "grpc-go/1.55.0" 154 60.000 [netbird-netbird-signal-80] [] 10.244.3.157:80 187 60.000 200 33cd1401fdfed21bbafd556ef35317db nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:06 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.006 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.008 200 5a77a24d9402d063a6a4ecc4a2c99929 nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:15 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.007 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.004 200 1d285753adebe4f1647259ea2d60020c nginx-ingress-ingress-nginx-controller-dsphr controller ip-hidden - - [06/Oct/2023:07:45:15 +0000] "POST /signalexchange.SignalExchange/Send HTTP/2.0" 200 5 "-" "grpc-go/1.55.0" 217 0.002 [netbird-netbird-signal-80] [] 10.244.3.157:80 47 0.000 200 9fbd16a3312440bcc847bc6f2c93be4e ```
saavagebueno commented 2025-11-20 05:08:50 -05:00
Author
Owner
Copy Link

@devopskupryk commented on GitHub (Oct 11, 2023):

Any news?

@devopskupryk commented on GitHub (Oct 11, 2023): Any news?
saavagebueno commented 2025-11-20 05:08:50 -05:00
Author
Owner
Copy Link

@fede843 commented on GitHub (Apr 16, 2025):

I still see this. We are on nginx 1.27.2.
As it is mentioned here, increasing grpc_read_timeout reduce the frequency of this disconnection, but not sure if it might have a negative impact somewhere else. I've seen setting in the forums from 300s to 1d.

@fede843 commented on GitHub (Apr 16, 2025): I still see this. We are on nginx 1.27.2. As it is mentioned here, increasing grpc_read_timeout reduce the frequency of this disconnection, but not sure if it might have a negative impact somewhere else. I've seen setting in the forums from 300s to 1d.
saavagebueno commented 2025-11-20 05:08:50 -05:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Apr 17, 2025):

I still see this. We are on nginx 1.27.2. As it is mentioned here, increasing grpc_read_timeout reduce the frequency of this disconnection, but not sure if it might have a negative impact somewhere else. I've seen setting in the forums from 300s to 1d.

Another user seems to be working around it by setting a 7d timeout at https://github.com/netbirdio/netbird/issues/955#issuecomment-1602405962

@nazarewk commented on GitHub (Apr 17, 2025): > I still see this. We are on nginx 1.27.2. As it is mentioned here, increasing grpc_read_timeout reduce the frequency of this disconnection, but not sure if it might have a negative impact somewhere else. I've seen setting in the forums from 300s to 1d. Another user seems to be working around it by setting a `7d` timeout at https://github.com/netbirdio/netbird/issues/955#issuecomment-1602405962
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label bug client management-service
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#266
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?