mirror of
https://github.com/netbirdio/netbird.git
synced 2026-05-15 20:51:57 -04:00
Pings disappear from some peers to others after one week #1577
Closed
opened 2025-11-20 05:33:08 -05:00 by saavagebueno
·
25 comments
No Branch/Tag Specified
main
fix/wireguard-port-zero
windows-dns-firewall
ui-refactor
fix/wgport-config
feature/refactor-clusters
fix/rosenpass
drop-candidateviaroutes-filter
e2e-windows-dns-combined
refactor-combined
wasm-websocket-dial
feature/affected-peers
dependabot/go_modules/github.com/Azure/go-ntlmssp-0.1.1
debug-logs
reduce-embed-wg-pool
dependabot/go_modules/github.com/jackc/pgx/v5-5.9.2
fix/login-cmd-root-flags
feat/reseller-openapi-spec
github-issue-resolver
add-steamos-support
fix-darwin-uninstaller
flutter-test
dependabot/npm_and_yarn/proxy/web/postcss-8.5.12
ci/freebsd-pkg-bootstrap
cached-serial-check-on-sync
fix-mgmt-cache-bypass-overlay
revert-easyjson-5938
revert-ice-5820
revert-firewalld-5928
refactor/permissions-manager
wasm-js-func-release
revert-dns-5935-systemd-resolved
revert-dns-5935-5945
revert-dns-5945-mgmt-cache
feature/log-most-busy-peers
prototype/ui-wails
vnc-server
coderabbitai/utg/8ae8f20
feature/use-peer-fqdn-on-https
dependabot/go_modules/golang.org/x/image-0.38.0
feature/metrics-push-management-control
release/0.68.3
dependabot/go_modules/github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream-1.7.8
dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/s3-1.97.3
add-slack-channel
claude/rdp-token-passthrough-eNcqW
transparent-proxy
fix/macos-stale-route-eexist
crowdsec-selfhosted
fix/remove-otel-units
entire/checkpoints/v1
dependabot/go_modules/github.com/go-jose/go-jose/v4-4.1.4
fix/getting-started
feat/static-connectors-combined-server
feature/use-local-keys-embedded
feature/fleetdm
set-env-only-if-not-fork
feature/expose-has-channel
fix/connection-status-race
fix/filter-cgnat-cni-ice-candidates
feature/check-cert-locker-before-acme
test/proxy-fixes
test/proxy-mtu
prototype/ui-tauri
test/proxy-speed
fix-reused-ports
feat/migrate-to-embedded-idp
feature/add-serial-to-proxy-merged
deploy/proxy-serial
test/connection
feature/disable-legacy-port
feature/flag-to-disable-legacy-port
test/perftest
dependabot/go_modules/github.com/pion/dtls/v3-3.0.11
fix/http-redirect
poc-token-command
dn-reverse-proxy
prototype/reverse-proxy-rename
prototype/reverse-proxy-logs-pagination
feature/client-metrics
prototype/reverse-proxy-clusters
debug-dns-route
fix/win-dns-batch
add-extra-route-logs
job-stream-notify-disconnection-eof
deploy/secrets-manager
trigger-proxy-update
bug/update-ios-client-code-build-tags
sync-client-netmap-serial
log/conn-disconn
nmap/compaction-deploy
ci-win-test
feature/disk-encryption-check
wasm-debug
swap-dns-prio
fix/dex-config
feature/migrate-auto-groups-to-table
dependabot/go_modules/github.com/quic-go/quic-go-0.57.0
nmap/compaction
dex-nocgo-stub
feature/exclude-terraform-from-rate-limiting
test-freebsd
retries-refactor
coderabbitai/docstrings/b7e98ac
feat/integrate-zitadel
bug/ios-hanging-reconection
zitadel-idp
feat/network-map-serial
refactor/get-account-no-users
feat/auto-upgrade
feature/report-high-pat-id
feature/temporary-access-for-resource
fix/nmap-fwrules
dont-restart-dns
prototype/ui
update-gomobile
go-dns-for-ice
wasm-ldflags
test-ldflags
wasmbuild-test
feature/networks-s2s
vk/compare-nmaps
dbg/bothmaps
feature/changeset
reorder-dns-shutdown
fix/relay-reconnection-race
fix/nmap-exitnodes
vk/debug/nmap-both
move-licensed-code
feat/better-daemon-connection-lost-message
feat/auto-update-2
test/timings
refactor/getaccount-raw
tests/nmap-getaccount
refactor/nmap
refactor/nmap-limit-buffer
feature/detect-mac-wakeup
feature/extract-modules
quick-setings
feat/sync-limiter
feature/store-cache-impl
fix-install-version
feature/store-metrics
feature/metrics-on-store
feature/use-gorm-cache
loadtest-signal
unsymmetrical-squash
refactor/reducate-signaling
test/update-reduce
feature/store-cache
feature/remote-debug
cli-ws-proxy-backend-addr
feat/mgmt-map-serial
snyk-fix-d9d0081a4c7f9137bdb59d0d50a141a2
snyk-fix-7415cea5a11acd66753540ca2c598c63
job-yml-update
feature/android-allow-selecting-routes
fix/up-sequence
fix/dns-hash-update
snyk-fix-967adae9863f17f108ce8948d9117b8d
log/getaccount-by-peer
signal-suppressor
dns-exit-node
feature/auto-updates
feature/cache-srv-key
merged-fixes
fix/missed-offers-and-debug
debug-and-fixes
poc-wasm-clean-backend-s2s
test/remote-debug
debug-api
dependabot/go_modules/github.com/docker/docker-28.0.0incompatible
fix/remove-gpo-if-empty
fix/test-freebsd
fix/mysql-setup
fix/remove-logout-btn
handle-existing-domain-user
chore/unify-domain-validation
snyk-fix-c5fafc8a50ce1f29046e25a1fc346185
feat/profile-edit-btn
snyk-fix-a54966211e18d4cf67e5a2757cc006d1
log-short-id
feat/logout-ephemeral
log-checks
batch-wg-ops
nb-interface-default
feat/aws-integration
add/race-test
feature/relay-feature-versioning
fix/systemd-service-logs
poc/preprocessed-map
add-account-onboarding
bind-ipv6
fix/merge-main
logs/peerlogs-addpeer
feature/net-297-network-migration
feature/support-skip-auto-apply-exit-node-routes
set-cmd
set-command-with-cursor
feature/limit-update-channel
stop-using-locking-share
feature/poc-lazy-detection
feature/net-248-removal-of-sync-mutex-locks
test/multiple-peer-logging
preresolve
add-ns-punnycode-support
apply-routes-early
windows-search-domains
fix/connecting-route-filter
feature/management/rest-client/impersonate
debug-local-records
resource-fields-snake-case
test/grpc-rate-limit
traffic-correlation-policy
feature/rest-client-options
feat/events-metrics
feature/buf-cli
test/add-ratelimiter
test/remove-write-lock-on-add-peer
fix/add-peer-semaphore
feature/users-roles-endpoint
mlsmaycon-patch-1
debug-user-role
chore/primary-key-on-networks
feature/update-account-peers-buffer-startup
remove-ubuntu2004-runners
refactor/permissions-no-pat-allowed
ref/logrus-factory
use-conntrack-zone
deploy/permissions-account
feature/lazy-connection-idle
ref/improve-test-cov
restore-pr-3440
test/increase-grpc-timeouts
feat/buffer-account-peers-update
test/networkmapgeneration-changes
feature/base-manager
feature/flow-receiver
chore/benchmark-with-large-runner
refactor/handshake-initiator
client/ui-update-systray-icons
userspace-router
wgwatcher-test
output-if-key-already-exists
fix/relay-reconnection
feature/port-forwarding-client-codecleaning
detached2
test/callbacks-nil-iceconninfo
refactor/optimize-peer-expiration
enable-udp-port-for-docker-template
fix/relay-update
feature/apply-posture-netmap
fix/group-update-existing-resource
conntrack-stats
upgrade-okta-sdk
multi-price
test/conn-stat
set-min-parallel-tests-for-management
dns-interceptor
debug-dns
router-dns
add-static-system-info
debug-0.29.4
debug-0.33.0
account-refactoring
relay/2800_quic
route-get-account-refactoring
test/seed-random-routes
feature/get-account-refactoring
test/reconnect-race-condition
refactor/get-account-usage
feature/add-session-id-to-update-channel
improve-ipv4conn
fix/async-pion-event-handling
debug
add-offload
feature/validate-group-association-debug
fix/limit-conn-for-sqlite
test/engine-iface
test/transaction-for-jwt-sync
fix/engine-stop-in-foreground
feature/add-mysql-support
test-migration
refactor/header-size-values
relay/eliminate-gob
test/signal-dispatcher-with-relay
relay/debug
validate-icon
feature/ipv6-support
use-pre-expanded-peers-map
feature/use-signal-dispatcher
validate/peer-status
add-read-write-times
fix/sync-peer-race
feature/relay-status
netmap
evaluate/network-map-hash
fix/lower-dns-resolve-interval-on-fail
feature/relay
fix/go-mod-version
upgrade-nftables
synology-userspace-mode
fix/use-ip-for-default-routes-on-darwin
fix/proxy_close
enable-release-workflow-on-pr
deploy/peer-performance
feature/permanent-turn
feature/permanent-turn-proxy
deploy/posture-check-sqlite
feature/optimize_sqlite_save
debug-ios-behavior
fix/delete-route-only-after-adding
tshoot/windows-logger
remove-new-routing
refactor/eliminate-repo-dependency
add-arm-to-ci
refactor-demo-account-object
test/abc2
test/abc
send-ssh-rosenpass-config-meta
refactor-demo
ensure-schedule-never-runs-non-positive
feature/peer-validator-groupmgm
feature/peer-validator-fix
fix/include-active-dashboard-users
fix/handle-canceling-schedule
fix/geo-download
debug-google-workspace
yury/resolve-ip-to-location
feature/extend-sysinfo
sqlite-async-peer-status
yury/add-postgresql-store
fix/route
test-build
posture-checks-poc
debug-keycloak-idp
poc/netstack
for-pascal-tmp
peer-logout-management
manual-peer-logout
detached
chore/refactor-management
test/dns-bind
fix/enforce-acl-for-containers
yury/use-sync-map-in-updatechannel
fix/events-key-handling
filter-cache-on-load-account
fix/user-expiration
handle-user-context-cancellation
nb-client-k8s-statefulset
fake-addr
fix/iptables_in_docker
ebpf-debug
update-getting-started-flow-use-postgres
fix/peer_list_notification
feature/device-authentication-with-client-secret
feature/keep_alive
feat-groups-from-jwt
separate_proxy_from_wgconfig
fix/wg_conn
wg_conn_fix
wg_bind_parallel_processing
fix-rollback-get-acls
proxy_cfg_cleanup
performance-improvement-rego
update-lock-log-level
feat-client-side-acl
refactor/move_grpcserver_logic_to_account_manager
feature/event-storage
feature/update-idp-redeeming-invite
feature/api-peer-info
return-groupminimum-setupkey
feature/interface-bind
documentation_enhancement
fix-peer-registration
ssh
users_cache
pass-client-caller
client_caller_type
revert-283-feat-fix-windows-installer
periodic-peer-updates
ebpf
braginini/wasm
v0.71.1
v0.71.0
v0.70.5
v0.70.4
v0.70.3
v0.70.2
v0.70.1
v0.70.0
v0.69.0
v0.68.3
v0.68.2
v0.68.1
v0.68.0
v0.67.4
v0.67.3
v0.67.2
v0.67.1
v0.67.0
v0.66.4
v0.66.3
v0.66.2
v0.66.1
v0.66.0
v0.65.3
v0.65.2
v0.65.1
v0.65.0
v0.64.6
v0.64.5
v0.64.4
v0.64.3
v0.64.2
v0.64.1
v0.64.0
v0.63.0
v0.62.3
v0.62.2
v0.62.1
v0.62.0
v0.61.2
v0.61.1
v0.61.0
v0.60.9
v0.60.8
v0.60.7
v0.60.6
v0.60.5
v0.60.4
v0.60.3
v0.60.2
v0.60.1
v0.60.0
v0.59.13
v0.59.12
v0.59.11
v0.59.10
v0.59.9
v0.59.8
v0.59.7
v0.59.6
v0.59.5
v0.59.4
v0.59.3
v0.59.2
v0.59.1
v0.59.0
v0.58.2
v0.58.1
v0.58.0
v0.57.1
v0.57.0
v0.56.1
v0.56.0
v0.55.1
v0.55.0
v0.54.2
v0.54.1
v0.54.0
v0.53.0
v0.52.2
v0.52.1
v0.52.0
v0.51.2
v0.51.1
v0.51.0
v0.50.3
v0.50.2
v0.50.1
v0.50.0
v0.49.0
v0.48.0-dev2
v0.48.0
v0.47.2
v0.47.1
v0.47.0
v0.46.0
v0.45.3
v0.45.2
v0.45.1
v0.45.0
v0.44.0
v0.43.3
v0.43.2
v0.43.1
v0.43.0
v0.42.0
v0.41.3
v0.41.2
v0.41.1
v0.41.0
v0.40.1
v0.40.0
v0.39.2
v0.39.1
v0.39.0
v0.38.2
v0.38.1
v0.38.0
v0.37.2
v0.37.1
v0.37.0
v0.36.7
v0.36.6
v0.36.5
v0.36.4
v0.36.3
v0.36.2
v0.36.1
v0.36.0
v0.35.2
v0.35.1
v0.35.0
v0.34.1
v0.34.0
v0.33.0
v0.32.0
v0.31.1
v0.31.0
v0.30.3
v0.30.2
v0.30.1
v0.30.0
v0.29.4
v0.29.3
0.29.3
v0.29.2
v0.29.1
v0.29.0
v0.28.9
v0.28.8
v0.28.7
v0.28.6
v0.28.5
v0.28.4
v0.28.3
v0.28.2
v0.28.1
v0.28.0
v0.27.10
v0.27.9
v0.27.8
v0.27.7
v0.27.6
v0.27.5
v0.27.4
v0.27.3
v0.27.2
v0.27.1
v0.27.0
v0.26.7
v0.26.6
v0.26.5
v0.26.4
v0.26.3
v0.26.2
v0.26.1
v0.26.0
v0.25.9
v0.25.8
v0.25.7
v0.25.6
v0.25.5
v0.25.4
v0.25.3
v0.25.2
v0.25.1
v0.25.0
v0.24.4
v0.24.3
v0.24.2
v0.24.1
v0.24.0
v0.23.9
v0.23.8
v0.23.7
v0.23.6
v0.23.5
v0.23.4
v0.23.3
v0.23.2
v0.23.1
v0.23.0
v0.22.7
v0.22.6
v0.22.5
v0.22.4
v0.22.3
v0.22.2
v0.22.1
v0.22.0
v0.21.11
v0.21.10
v0.21.9
v0.21.8
v0.21.7
v0.21.6
v0.21.5
v0.21.4
v0.21.3
v0.21.2
v0.21.1
v0.21.0
v0.20.8
v0.20.7
v0.20.6
v0.20.5
v0.20.4
v0.20.3
v0.20.2
v0.20.1
v0.20.0
v0.19.0
v0.18.1
v0.18.0
v0.17.0
v0.16.0
v0.15.3
v0.15.2
v0.15.1
v0.15.0
v0.14.6
v0.14.5
v0.14.4
v0.14.3
v0.14.2
v0.14.1
v0.14.0
v0.13.0
v0.12.0
v0.11.6
v0.11.5
v0.11.4
v0.11.3
v0.11.2
v0.11.1
v0.11.0
v0.10.10
v0.10.9
v0.10.8
v0.10.7
v0.10.6
v0.10.5
v0.10.4
v0.10.3
v0.10.2
v0.10.1
v0.10.0
v0.9.8
v0.9.7
v0.9.6
v0.9.5
v0.9.4
v0.9.3
v0.9.2
v0.9.1
v0.9.0
v0.8.12
v0.8.11
v0.8.10
v0.8.9
v0.8.8
v0.8.7
v0.8.6
v0.8.5
v0.8.4
v0.8.3
v0.8.2
v0.8.1
v0.8.0
v0.7.1
v0.7.0
v0.6.4
v0.6.3
v0.6.2
v0.6.1
v0.6.0
v0.5.11
v0.5.10
v0.5.1
v0.5.0
v0.4.0
v0.3.5
v0.3.4
v0.3.3
v0.3.2
v0.3.1
v0.3.0
v0.2.3
v0.2.2-beta.1
v0.2.1-beta.5
v0.2.0-beta.5
v0.2.0-beta.4
v0.2.0-beta.3
v0.2.0-beta.2
v0.2.0-beta.1
v0.1.0-beta.3
v0.1.0-beta.2
v0.1.0-beta.1
v0.1.0-rc.2
v0.1.0-rc-1
v0.0.8-hotfix-1
v0.0.8
v0.0.7
v0.0.6
v0.0.5
v0.0.4
v0.0.3
v0.0.2
v0.0.1
v0.0.0
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
Mirrored from GitHub Pull Request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
saavagebueno
Clear assignees
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: SVI/netbird#1577
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @netandreus on GitHub (Jan 28, 2025).
Originally assigned to: @pappz on GitHub.
Describe the problem
I have 2 peers:
supporting HA-routes to multiple vlans.
Also I have:
My problem is that after a week these nodes (node-2 and node-3) losing connection to ELK node, and ELK node losing connection to node-2 and node-3, I can't ping them by netbird IP addresses. But in the same time I can ping other peers from ELK peer and ping other than ELK peers from node-2 and node-3.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Stable connection between peers.
Are you using NetBird Cloud?
No, I use self-hosted netbird.
NetBird version
0.36.3
NetBird status -dA output:
When I go by ssh to node-3 I see this:
And same from node-2:
Workaround
After I run:
WireGuard connection re-established.
Related issues
Can you please fix this?
@pappz commented on GitHub (Jan 28, 2025):
Hello @netandreus
Could you reproduce the issue with verbose logging enabled and send me the logs?
@pappz commented on GitHub (Jan 28, 2025):
@netandreus Could you send me the public keys of the node-2 and node-3? In the connection mechanism has some logic that depends from the public keys between the peers. Maybe if we know which lane of the algorithm is running on your elk side and on the node side then we can go nearer to the root cause of the issue.
@netandreus commented on GitHub (Jan 28, 2025):
@pappz sure, how can I fetch the public keys of my nodes?
@pappz commented on GitHub (Jan 28, 2025):
The
netbird status -dcommand will print out. Looking for the "Public key" expression, just like in your example in your original report:@netandreus commented on GitHub (Jan 28, 2025):
@pappz here they are:
node-2:
node-3:
@pappz commented on GitHub (Jan 28, 2025):
Great, thank you!
Do you know which restart solved the issue:
@netandreus commented on GitHub (Jan 28, 2025):
@pappz It does not matter, what I do at the ELK side, only restarting the Netbird agent (
netbird stop && netbird start) at the node side affects the situation.@netandreus commented on GitHub (Jan 30, 2025):
Good morning, @pappz ! Is there something from my side, that could help you?
@pappz commented on GitHub (Jan 30, 2025):
Hello @netandreus,
Thank you for the information. I found a potential bug and I have prepared a fix for it. You can track the changes here. Nevertheless, I am not sure this is the root cause of your issue because I haven't been able to reproduce it on my machine.
How easy is it to reproduce the issue? Could you enable verbose logging on your agent and collect the logs?
@netandreus commented on GitHub (Jan 30, 2025):
Thank you for your efforts, @pappz !
I can only disable cron restart and wait for one week. What exact should I do if (or when) it will occurs?
@pappz commented on GitHub (Jan 30, 2025):
With these commands, you can set the logging level:
netbird debug log level debugor
netbird debug log level verboseThe verbose level may log too much data to your disk over a week, so choose the debug level based on your preference. When an issue occurs, please collect the log files and send them to me.
@pappz commented on GitHub (Jan 30, 2025):
@netandreus
While you ping the unreachable server what error message do you see in the ping output? Is it something like this, or just a simple
Destination Host Unreachable.@pappz commented on GitHub (Jan 30, 2025):
I am working on another logic that can better manage the possible anomalies. Is it an option for you to do tests with a custom build with the patches?
@netandreus commented on GitHub (Jan 31, 2025):
@pappz yes, I can. I should deploy custom build on both nodes? And how can I rollback if somethings go wrong? How can I collect logs?
@pappz commented on GitHub (Jan 31, 2025):
@netandreus
I prepared the test version. Here is the package for Linux. If you are using a different OS, I will send you different artifacts.
Default installation path is
/usr/bin/netbird. Create a backup for easy rollback:Don't forget to set the proper debug level!
Logs are in /var/log/netbird. Clean them before testing for easier handling.
If you are testing with previous machines (node-2, node-3), no need to update ELK peer.
And do not forget, this is just a test version, be careful to use it in production env.
I hope this fix will solve your issue but meantime I will dig deeper into this topic.
@netandreus commented on GitHub (Feb 3, 2025):
Good morning, @pappz !
Can you please update the link, I can't download test version.
@pappz commented on GitHub (Feb 3, 2025):
Strange. Here is the updated link.
@pappz commented on GitHub (Feb 3, 2025):
@netandreus could you send to me a debug bundle? You can generate it with this command:
netbird debug bundle -SI would like to get a better picture of your network-related settings. This package contains all the necessary information.
@netandreus commented on GitHub (Feb 3, 2025):
@pappz Sure, but I can`t downloaad it. May be some persmssions issue from your side?
When I click to the file - I can only copy name. Maybe you need my google account or something from my side?
@pappz commented on GitHub (Feb 3, 2025):
The file that I uploaded is a ZIP archive. I think you opened it by the browser. If you download the full zip and manage it on your machine it would be easier.
@netandreus commented on GitHub (Feb 3, 2025):
Done.
@netandreus could you send to me a debug bundle? You can generate it with this command: netbird debug bundle -SDone.
You can find all files (both logs when error occures on stable version and debug bundle for test version) here:
https://drive.google.com/drive/folders/1sRO8GprHSPS5LYgUqWg5iP2wCTHm7PHa?usp=drive_link
I'm deployed test version on the both nodes.
node-2

node-3

Then I restarted netbird on elk node:
And see this one for node-2 at elk
and for node-3 at elk:
@netandreus commented on GitHub (Feb 3, 2025):
@pappz and now I can't ping neither node-3 nor node-2 from elk.
@pappz commented on GitHub (Feb 3, 2025):
@netandreus
Thank you for the logs. Can we schedule a call where we can better discover the situation together? It would be faster to resolve this issue than here.
@netandreus commented on GitHub (Feb 3, 2025):
@pappz sure, we can schedule a call for tomorrow 2025-02-04 from 10:00 GMT+4. I can give you access to Anydesk / ssh to these nodes. Please find me on Telegram - https://t.me/netandreus
@pappz commented on GitHub (Feb 4, 2025):
Thank you. I think you need to accept my messages. https://t.me/pzolinb