SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-07-01 19:52:38 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Relay server bug - Connections not closed after EOF exception #1410

New Issue
Open
opened 2025-11-20 05:29:50 -05:00 by saavagebueno · 0 comments
saavagebueno commented 2025-11-20 05:29:50 -05:00
Owner
Copy Link

Originally created by @mathiash98 on GitHub (Nov 12, 2024).

Originally assigned to: @pappz on GitHub.

Solved by #2879

Potential Relay server bug - Saturating number of connections on EOF websocket error: Hopefully I just misunderstand something:

Scenario:

Many peers with high ping and low internet speed

Background:

Hosting relay server behind nginx reverse proxy

Reproduction

  1. lsof count inside docker container before error: sudo docker exec netbird-relay-0-1 lsof | wc -> 800
  2. Whenever Relay server get error code relay/server/peer.go:61: failed to read message: failed to get reader: failed to read frame header: EOF it will return the Work code in b4d7605147/relay/server/peer.go (L48-L64) Which in turn result in removing deleting the peer from store in Relay.go file b4d7605147/relay/server/relay.go (L134-L139) and logs relay/server/relay.go:137: relay connection closed
  3. The client will also get an error code when sending the data and the same peer will then reconnect right after (I have not found the code for this yet) leading to log message: relay/server/relay.go:129: peer connected from: 172.23.0.1:39226
  4. The lsof has now increased by one sudo docker exec netbird-relay-0-1 lsof | wc -> 801
  5. When the lsof of the docker container reaches 1000 it will stop accepting new connections and the docker container must be restarted

Potential fix? -> Run p.conn.Close() inside b4d7605147/relay/server/peer.go (L48-L64) whenever an error is met?

Full log example showcasing that the peer_id is same:

netbird-relay-0-1  | 2024-11-11T12:54:32Z ERRO [peer_id: sha-Pfq9J4xZuOoCYsq6OMQCUZ7j0GOemtq/28tDkCvWGJc=] relay/server/peer.go:61: failed to read message: failed to get reader: failed to read frame header: EOF
netbird-relay-0-1  | 2024-11-11T12:54:32Z DEBG [peer_id: sha-Pfq9J4xZuOoCYsq6OMQCUZ7j0GOemtq/28tDkCvWGJc=] relay/server/relay.go:137: relay connection closed
netbird-relay-0-1  | 2024-11-11T12:54:32Z DEBG [peer_id: sha-0Uu8KZZs8rTiypKGr9PaxkKtTFWqnttiYaCZRwMFuHE=] relay/server/peer.go:196: peer not found: sha-Pfq9J4xZuOoCYsq6OMQCUZ7j0GOemtq/28tDkCvWGJc=
netbird-relay-0-1  | 2024-11-11T12:54:34Z INFO [peer_id: sha-Pfq9J4xZuOoCYsq6OMQCUZ7j0GOemtq/28tDkCvWGJc=] relay/server/relay.go:129: peer connected from: 172.23.0.1:43988

Can mention that I also observe that whenever a client is gracefully shut down or healtcheck timeout leads to a reduction of connected sockets.

Originally created by @mathiash98 on GitHub (Nov 12, 2024). Originally assigned to: @pappz on GitHub. Solved by #2879 Potential Relay server bug - Saturating number of connections on EOF websocket error: Hopefully I just misunderstand something: ## Scenario: Many peers with high ping and low internet speed ## Background: Hosting relay server behind nginx reverse proxy ## Reproduction 1. lsof count inside docker container before error: `sudo docker exec netbird-relay-0-1 lsof | wc` -> 800 2. Whenever Relay server get error code `relay/server/peer.go:61: failed to read message: failed to get reader: failed to read frame header: EOF` it will return the Work code in https://github.com/netbirdio/netbird/blob/b4d7605147e6ddbd22c214b42ef43267bc78ce80/relay/server/peer.go#L48-L64 Which in turn result in removing deleting the peer from store in `Relay.go` file https://github.com/netbirdio/netbird/blob/b4d7605147e6ddbd22c214b42ef43267bc78ce80/relay/server/relay.go#L134-L139 and logs `relay/server/relay.go:137: relay connection closed` 3. The client will also get an error code when sending the data and the same peer will then reconnect right after (I have not found the code for this yet) leading to log message: `relay/server/relay.go:129: peer connected from: 172.23.0.1:39226` 4. The lsof has now increased by one `sudo docker exec netbird-relay-0-1 lsof | wc` -> 801 5. When the lsof of the docker container reaches 1000 it will stop accepting new connections and the docker container must be restarted Potential fix? -> Run `p.conn.Close()` inside https://github.com/netbirdio/netbird/blob/b4d7605147e6ddbd22c214b42ef43267bc78ce80/relay/server/peer.go#L48-L64 whenever an error is met? Full log example showcasing that the peer_id is same: ``` netbird-relay-0-1 | 2024-11-11T12:54:32Z ERRO [peer_id: sha-Pfq9J4xZuOoCYsq6OMQCUZ7j0GOemtq/28tDkCvWGJc=] relay/server/peer.go:61: failed to read message: failed to get reader: failed to read frame header: EOF netbird-relay-0-1 | 2024-11-11T12:54:32Z DEBG [peer_id: sha-Pfq9J4xZuOoCYsq6OMQCUZ7j0GOemtq/28tDkCvWGJc=] relay/server/relay.go:137: relay connection closed netbird-relay-0-1 | 2024-11-11T12:54:32Z DEBG [peer_id: sha-0Uu8KZZs8rTiypKGr9PaxkKtTFWqnttiYaCZRwMFuHE=] relay/server/peer.go:196: peer not found: sha-Pfq9J4xZuOoCYsq6OMQCUZ7j0GOemtq/28tDkCvWGJc= netbird-relay-0-1 | 2024-11-11T12:54:34Z INFO [peer_id: sha-Pfq9J4xZuOoCYsq6OMQCUZ7j0GOemtq/28tDkCvWGJc=] relay/server/relay.go:129: peer connected from: 172.23.0.1:43988 ``` Can mention that I also observe that whenever a client is gracefully shut down or healtcheck timeout leads to a reduction of connected sockets.
saavagebueno added the server label 2025-11-20 05:29:50 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label server
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1410
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?