SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-07-27 16:32:34 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

[Important notification] NetBird Cloud Service Interruption 05/Oct/2025 #2344

New Issue
Open
opened 2025-11-20 07:08:16 -05:00 by saavagebueno · 0 comments
saavagebueno commented 2025-11-20 07:08:16 -05:00
Owner
Copy Link

Originally created by @mlsmaycon on GitHub (Oct 6, 2025).

🛠️ NetBird Post-Mortem Report

Summary

On Sunday, October 5th, around 11:00 AM CEST, we received alerts indicating panic events in the management service logs. During the investigation, we identified that the service was entering a restart loop and was unable to recover. The root cause was traced to an issue with the SentinelOne EDR integration.

Root Cause

The SentinelOne API, during scheduled maintenance on their side, returned an unexpected HTML response instead of the expected JSON-formatted error object. This response was not handled gracefully by our management service, resulting in a runtime panic due to a failed type assertion during error parsing.

As a consequence, the management service restarted every minute. NetBird clients attempted to reconnect at the same interval, but were unable to establish a connection, leading to repeated log entries and degraded service availability.

Impact

All NetBird clients connected to the affected management instance were impacted. Clients were unable to communicate with the management service during the incident window, resulting in temporary loss of access to configuration updates and control-plane operations.

Mitigation and Resolution

Temporary Fix

  • Isolated and disabled SentinelOne integration instances that were returning invalid responses due to maintenance, halting the panic loop.

Permanent Fixes Implemented

  • Improved the API response parser to validate the format before attempting to parse, preventing nil pointer dereference errors.
  • Updated logging mechanisms to fallback to standard logging in case the integration logger is unavailable or uninitialized.

Preventive Actions

To reduce the risk of similar issues in the future, we are implementing the following:

  • Robust Integration Error Handling:
    Extend test coverage for all third-party integrations to validate error-handling logic, including cases of unexpected response formats.

  • Panic Handling:
    Introduce package-level panic recovery handlers to catch and log unexpected errors without crashing the service.

Originally created by @mlsmaycon on GitHub (Oct 6, 2025). # 🛠️ NetBird Post-Mortem Report ## Summary On Sunday, October 5th, around 11:00 AM CEST, we received alerts indicating panic events in the management service logs. During the investigation, we identified that the service was entering a restart loop and was unable to recover. The root cause was traced to an issue with the SentinelOne EDR integration. ## Root Cause The SentinelOne API, during scheduled maintenance on their side, returned an unexpected HTML response instead of the expected JSON-formatted error object. This response was not handled gracefully by our management service, resulting in a runtime panic due to a failed type assertion during error parsing. As a consequence, the management service restarted every minute. NetBird clients attempted to reconnect at the same interval, but were unable to establish a connection, leading to repeated log entries and degraded service availability. ## Impact All NetBird clients connected to the affected management instance were impacted. Clients were unable to communicate with the management service during the incident window, resulting in temporary loss of access to configuration updates and control-plane operations. ## Mitigation and Resolution ### Temporary Fix - Isolated and disabled SentinelOne integration instances that were returning invalid responses due to maintenance, halting the panic loop. ### Permanent Fixes Implemented - Improved the API response parser to validate the format before attempting to parse, preventing nil pointer dereference errors. - Updated logging mechanisms to fallback to standard logging in case the integration logger is unavailable or uninitialized. ## Preventive Actions To reduce the risk of similar issues in the future, we are implementing the following: - **Robust Integration Error Handling:** Extend test coverage for all third-party integrations to validate error-handling logic, including cases of unexpected response formats. - **Panic Handling:** Introduce package-level panic recovery handlers to catch and log unexpected errors without crashing the service.
saavagebueno added the cloudstatus labels 2025-11-20 07:08:16 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label cloud status
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#2344
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?