Most of my network went down 6 hours ago #693

Closed
opened 2025-11-20 05:16:04 -05:00 by saavagebueno · 1 comment
Owner

Originally created by @tarasglek on GitHub (Mar 10, 2024).

Describe the problem

Just noticed I couldn't connect from taras-air to b550 machine

Luckily I have local access to that machine. Logs looked bad, but netbird up brought b550 back. Rest of my network is not so easy to recover as I use netbird as exclusive way to ssh into machines on other networks.

2024-03-10T02:06:56+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T02:07:24+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T02:07:38+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8"
2024-03-10T02:08:04+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T02:08:06+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T02:08:06+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 363.571µs, total rules count: 2
2024-03-10T02:08:25+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T02:08:53+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T02:09:01+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T02:09:01+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 740.047µs, total rules count: 2
2024-03-10T02:09:19+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T02:09:47+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T02:09:59+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T02:09:59+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 549.95µs, total rules count: 2
2024-03-10T02:33:51+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:11798
2024-03-10T03:04:59+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:49845
2024-03-10T03:30:20+02:00 INFO client/internal/peer/conn.go:358: connected to peer L8RkcRaDlvr/xUX7WlwOz0YoSWMnIZIitA5WVViB3iI=, endpoint address: 172.16.1.103:51820
2024-03-10T03:38:38+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:6916
2024-03-10T03:43:41+02:00 INFO client/internal/peer/conn.go:358: connected to peer L8RkcRaDlvr/xUX7WlwOz0YoSWMnIZIitA5WVViB3iI=, endpoint address: 172.16.1.103:51820
2024-03-10T03:49:22+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820
2024-03-10T03:49:39+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820
2024-03-10T03:54:09+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T03:54:35+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T03:54:39+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T03:54:39+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 559.689µs, total rules count: 2
2024-03-10T04:11:57+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:35853
2024-03-10T04:12:18+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820
2024-03-10T04:28:06+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820
2024-03-10T04:45:00+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:45535
2024-03-10T04:56:10+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820
2024-03-10T05:16:44+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:26858
2024-03-10T05:28:21+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820
2024-03-10T05:34:56+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:35:21+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:35:25+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:35:25+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 775.333µs, total rules count: 2
2024-03-10T05:35:48+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:36:14+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:36:27+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:36:27+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 544.039µs, total rules count: 2
2024-03-10T05:36:52+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:37:17+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:37:18+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:37:18+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 756.789µs, total rules count: 2
2024-03-10T05:38:00+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:38:27+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:38:28+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:38:28+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 329.668µs, total rules count: 2
2024-03-10T05:39:04+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:39:29+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:39:37+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:39:37+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 570.059µs, total rules count: 2
2024-03-10T05:40:05+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:40:30+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:40:51+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:40:51+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 605.645µs, total rules count: 2
2024-03-10T05:41:01+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:41:28+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:41:40+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:41:40+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 821.55µs, total rules count: 2
2024-03-10T05:41:47+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:42:18+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:42:38+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8"
2024-03-10T05:43:15+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:43:33+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8"
2024-03-10T05:44:07+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:44:28+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8"
2024-03-10T05:45:02+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:45:14+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8"
2024-03-10T05:45:41+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:45:53+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:45:53+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 829.504µs, total rules count: 2
2024-03-10T05:46:03+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820
2024-03-10T05:46:17+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:46:43+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:46:44+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:46:44+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 688.46µs, total rules count: 2
2024-03-10T05:47:24+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers
2024-03-10T05:48:00+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:48:02+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:30471
2024-03-10T05:48:22+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8"
2024-03-10T05:48:48+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-03-10T05:49:09+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted
2024-03-10T05:49:09+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 763.271µs, total rules count: 2
image

Expected behavior

p2p network should self-healing

Are you using NetBird Cloud?

yes under taras@glek.net

NetBird version

various. b550 had the newest client.

Originally created by @tarasglek on GitHub (Mar 10, 2024). **Describe the problem** Just noticed I couldn't connect from taras-air to b550 machine Luckily I have local access to that machine. Logs looked bad, but `netbird up` brought b550 back. Rest of my network is not so easy to recover as I use netbird as exclusive way to ssh into machines on other networks. ``` 2024-03-10T02:06:56+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T02:07:24+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T02:07:38+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8" 2024-03-10T02:08:04+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T02:08:06+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T02:08:06+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 363.571µs, total rules count: 2 2024-03-10T02:08:25+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T02:08:53+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T02:09:01+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T02:09:01+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 740.047µs, total rules count: 2 2024-03-10T02:09:19+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T02:09:47+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T02:09:59+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T02:09:59+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 549.95µs, total rules count: 2 2024-03-10T02:33:51+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:11798 2024-03-10T03:04:59+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:49845 2024-03-10T03:30:20+02:00 INFO client/internal/peer/conn.go:358: connected to peer L8RkcRaDlvr/xUX7WlwOz0YoSWMnIZIitA5WVViB3iI=, endpoint address: 172.16.1.103:51820 2024-03-10T03:38:38+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:6916 2024-03-10T03:43:41+02:00 INFO client/internal/peer/conn.go:358: connected to peer L8RkcRaDlvr/xUX7WlwOz0YoSWMnIZIitA5WVViB3iI=, endpoint address: 172.16.1.103:51820 2024-03-10T03:49:22+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820 2024-03-10T03:49:39+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820 2024-03-10T03:54:09+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T03:54:35+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T03:54:39+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T03:54:39+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 559.689µs, total rules count: 2 2024-03-10T04:11:57+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:35853 2024-03-10T04:12:18+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820 2024-03-10T04:28:06+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820 2024-03-10T04:45:00+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:45535 2024-03-10T04:56:10+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820 2024-03-10T05:16:44+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:26858 2024-03-10T05:28:21+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820 2024-03-10T05:34:56+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:35:21+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:35:25+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:35:25+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 775.333µs, total rules count: 2 2024-03-10T05:35:48+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:36:14+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:36:27+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:36:27+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 544.039µs, total rules count: 2 2024-03-10T05:36:52+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:37:17+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:37:18+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:37:18+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 756.789µs, total rules count: 2 2024-03-10T05:38:00+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:38:27+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:38:28+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:38:28+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 329.668µs, total rules count: 2 2024-03-10T05:39:04+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:39:29+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:39:37+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:39:37+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 570.059µs, total rules count: 2 2024-03-10T05:40:05+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:40:30+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:40:51+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:40:51+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 605.645µs, total rules count: 2 2024-03-10T05:41:01+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:41:28+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:41:40+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:41:40+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 821.55µs, total rules count: 2 2024-03-10T05:41:47+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:42:18+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:42:38+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8" 2024-03-10T05:43:15+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:43:33+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8" 2024-03-10T05:44:07+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:44:28+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8" 2024-03-10T05:45:02+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:45:14+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8" 2024-03-10T05:45:41+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:45:53+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:45:53+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 829.504µs, total rules count: 2 2024-03-10T05:46:03+02:00 INFO client/internal/peer/conn.go:358: connected to peer fgIc3VwixSXLdfGND7QNlphpAAZ3TaWIJXbP0rtYKGk=, endpoint address: 192.168.33.139:51820 2024-03-10T05:46:17+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:46:43+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:46:44+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:46:44+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 688.46µs, total rules count: 2 2024-03-10T05:47:24+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Internal desc = server closed the stream without sending trailers 2024-03-10T05:48:00+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:48:02+02:00 INFO client/internal/peer/conn.go:358: connected to peer zsF+oyjxYcsfjWzj++EiL3RC4k589/7qYvyQJ/1QcSU=, endpoint address: 194.44.26.50:30471 2024-03-10T05:48:22+02:00 WARN management/client/grpc.go:158: disconnected from the Management service but will retry silently. Reason: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type "text/html; charset=UTF-8" 2024-03-10T05:48:48+02:00 INFO management/client/grpc.go:143: connected to the Management Service stream 2024-03-10T05:49:09+02:00 WARN client/internal/engine.go:495: running SSH server is not permitted 2024-03-10T05:49:09+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 763.271µs, total rules count: 2 ``` <img width="728" alt="image" src="https://github.com/netbirdio/netbird/assets/857083/038baec5-3b28-492e-b47c-d5b7b5547167"> **Expected behavior** p2p network should self-healing **Are you using NetBird Cloud?** yes under taras@glek.net **NetBird version** various. b550 had the newest client.
saavagebueno added the triage-needed label 2025-11-20 05:16:04 -05:00
Author
Owner

@mlsmaycon commented on GitHub (Mar 10, 2024):

Hello @tarasglek sorry about the issue.

We notified the cloud users of the issue earlier via email and slack, but we missed a ticket here.

We faced an issue around the time of the reported incident. You can see more details here: https://github.com/netbirdio/netbird/issues/1689

@mlsmaycon commented on GitHub (Mar 10, 2024): Hello @tarasglek sorry about the issue. We notified the cloud users of the issue earlier via email and slack, but we missed a ticket here. We faced an issue around the time of the reported incident. You can see more details here: https://github.com/netbirdio/netbird/issues/1689
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#693