[GH-ISSUE #5715] CLI: "netbird up" hangs in a backoff retry loop when using an invalid setup key #11900

Open
opened 2026-08-05 01:31:32 -04:00 by saavagebueno · 1 comment
Owner

Originally created by @WoZ on GitHub (Mar 27, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5715

Describe the problem

When executing netbird up with an invalid setup key, the CLI does not exit with an error. Instead, it gets stuck in a retry loop (takes too long with automation). The daemon receives a NotFound or PermissionDenied error from the Management Service but treats it as a transient network error rather than a fatal authentication failure. This forces the user to manually interrupt the process and run netbird down to stop the background loop.

To Reproduce

Steps to reproduce the behavior:

  1. Run netbird up --hostname test-node --setup-key INVALID-KEY-XXXX
  2. Notice that the command hangs and does not return control to the terminal.
  3. Check the client logs.
  4. See continuous backoff warnings: operation failed, retrying in ... rpc error: code = NotFound desc = couldn't add peer: setup key is invalid.

Expected behavior

The CLI and daemon should recognize that an invalid setup key is a permanent error. The retry loop should be aborted immediately, and the netbird up command should exit, displaying a clear authentication failure message to the user (since an invalid key won't magically become valid after a few minutes of retries).

Are you using NetBird Cloud?

Applicable to both (the issue is in the client's internal retry logic).

NetBird version

0.67.1

Is any other VPN software installed?

No

Debug output

Here is the log snippet showing the infinite retry loop:

2026-03-27T10:30:36.774Z ERRO shared/management/client/grpc.go:575: failed to login to Management Service: rpc error: code = NotFound desc = couldn't add peer: setup key is invalid
2026-03-27T10:30:36.774Z ERRO client/internal/auth/auth.go:327: failed registering peer rpc error: code = NotFound desc = couldn't add peer: setup key is invalid
2026-03-27T10:30:36.774Z WARN client/internal/auth/auth.go:462: operation failed, retrying in 11.849974516s: rpc error: code = NotFound desc = couldn't add peer: setup key is invalid
2026-03-27T10:30:48.654Z ERRO shared/management/client/grpc.go:575: failed to login to Management Service: rpc error: code = PermissionDenied desc = no peer auth method provided, please use a setup key or interactive SSO login
...
2026-03-27T10:30:48.680Z WARN client/internal/auth/auth.go:462: operation failed, retrying in 11.499352542s: rpc error: code = NotFound desc = couldn't add peer: setup key is invalid

Screenshots

N/A

Additional context

It looks like the backoff implementation in client/internal/auth/auth.go (around line 462 based on the logs) is missing a check for fatal gRPC status codes.

Wrapping gRPC errors like codes.NotFound and codes.PermissionDenied in backoff.Permanent(err) during the peer registration/login phase would easily solve this issue and significantly improve the UX.

Have you tried these troubleshooting steps?

N/A

Originally created by @WoZ on GitHub (Mar 27, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5715 **Describe the problem** When executing `netbird up` with an invalid setup key, the CLI does not exit with an error. Instead, it gets stuck in a retry loop (takes too long with automation). The daemon receives a `NotFound` or `PermissionDenied` error from the Management Service but treats it as a transient network error rather than a fatal authentication failure. This forces the user to manually interrupt the process and run `netbird down` to stop the background loop. **To Reproduce** Steps to reproduce the behavior: 1. Run `netbird up --hostname test-node --setup-key INVALID-KEY-XXXX` 2. Notice that the command hangs and does not return control to the terminal. 3. Check the client logs. 4. See continuous backoff warnings: `operation failed, retrying in ... rpc error: code = NotFound desc = couldn't add peer: setup key is invalid`. **Expected behavior** The CLI and daemon should recognize that an invalid setup key is a permanent error. The retry loop should be aborted immediately, and the `netbird up` command should exit, displaying a clear authentication failure message to the user (since an invalid key won't magically become valid after a few minutes of retries). **Are you using NetBird Cloud?** Applicable to both (the issue is in the client's internal retry logic). **NetBird version** `0.67.1` **Is any other VPN software installed?** No **Debug output** Here is the log snippet showing the infinite retry loop: ```text 2026-03-27T10:30:36.774Z ERRO shared/management/client/grpc.go:575: failed to login to Management Service: rpc error: code = NotFound desc = couldn't add peer: setup key is invalid 2026-03-27T10:30:36.774Z ERRO client/internal/auth/auth.go:327: failed registering peer rpc error: code = NotFound desc = couldn't add peer: setup key is invalid 2026-03-27T10:30:36.774Z WARN client/internal/auth/auth.go:462: operation failed, retrying in 11.849974516s: rpc error: code = NotFound desc = couldn't add peer: setup key is invalid 2026-03-27T10:30:48.654Z ERRO shared/management/client/grpc.go:575: failed to login to Management Service: rpc error: code = PermissionDenied desc = no peer auth method provided, please use a setup key or interactive SSO login ... 2026-03-27T10:30:48.680Z WARN client/internal/auth/auth.go:462: operation failed, retrying in 11.499352542s: rpc error: code = NotFound desc = couldn't add peer: setup key is invalid ``` **Screenshots** N/A **Additional context** It looks like the backoff implementation in `client/internal/auth/auth.go` (around line 462 based on the logs) is missing a check for fatal gRPC status codes. Wrapping gRPC errors like `codes.NotFound` and `codes.PermissionDenied` in `backoff.Permanent(err)` during the peer registration/login phase would easily solve this issue and significantly improve the UX. **Have you tried these troubleshooting steps?** N/A
saavagebueno added the triage-needed label 2026-08-05 01:31:32 -04:00
Author
Owner

@arbnorme commented on GitHub (Apr 20, 2026):

We get almost exactly the same error with a fresh installation of Hetzner VMs

:~# cat /var/log/netbird/client.log | tail -50
2026-04-20T18:47:18.717Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:47:18.717Z WARN client/internal/auth/auth.go:419: connection error detected, attempting reconnection: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:47:18.738Z WARN client/internal/auth/auth.go:438: operation failed, retrying in 5.369663318s: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:47:34.122Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:47:34.122Z WARN client/internal/auth/auth.go:419: connection error detected, attempting reconnection: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:47:34.143Z WARN client/internal/auth/auth.go:438: operation failed, retrying in 3.708250653s: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:47:39.975Z INFO client/cmd/root.go:208: shutdown signal received
2026-04-20T18:47:41.985Z INFO client/cmd/service_controller.go:100: stopped NetBird service
2026-04-20T18:48:06.426Z INFO client/cmd/service_controller.go:27: starting NetBird service
2026-04-20T18:48:06.448Z INFO client/server/server.go:969: active profile config existed: true, err <nil>
2026-04-20T18:48:06.458Z INFO client/cmd/service_controller.go:74: started daemon server: /var/run/netbird.sock
2026-04-20T18:48:06.458Z INFO client/internal/connect.go:156: starting NetBird client version 0.69.0 on linux/amd64
2026-04-20T18:48:06.460Z INFO client/net/env_linux.go:70: system supports advanced routing
2026-04-20T18:48:16.812Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:48:28.303Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:48:40.319Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:48:51.562Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:49:03.289Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:49:23.623Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:50:12.204Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:50:18.558Z INFO client/server/server.go:885: service is down
2026-04-20T18:50:18.558Z INFO client/cmd/root.go:208: shutdown signal received
2026-04-20T18:50:23.558Z WARN client/server/server.go:847: timeout waiting for client goroutine to finish, proceeding anyway
2026-04-20T18:50:25.560Z INFO client/cmd/service_controller.go:100: stopped NetBird service
2026-04-20T18:50:25.620Z INFO client/cmd/service_controller.go:27: starting NetBird service
2026-04-20T18:50:25.621Z INFO client/server/server.go:969: active profile config existed: true, err <nil>
2026-04-20T18:50:25.621Z INFO client/cmd/service_controller.go:74: started daemon server: /var/run/netbird.sock
2026-04-20T18:50:25.621Z INFO client/internal/connect.go:156: starting NetBird client version 0.69.0 on linux/amd64
2026-04-20T18:50:25.622Z INFO client/net/env_linux.go:70: system supports advanced routing
2026-04-20T18:50:35.735Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:50:46.159Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:50:59.565Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:51:12.272Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:51:30.067Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:51:44.163Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:52:06.224Z INFO ./caller_not_available:0: 2026/04/20 18:52:06 WARNING: [core] [Server #1] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing"
2026-04-20T18:52:14.060Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:52:35.133Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:52:57.461Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:53:20.206Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:53:57.780Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:54:26.746Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:54:58.031Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:55:35.667Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:56:01.135Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:56:19.705Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:56:37.676Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:56:54.975Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:57:06.993Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
2026-04-20T18:57:20.663Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded
<!-- gh-comment-id:4283555833 --> @arbnorme commented on GitHub (Apr 20, 2026): We get almost exactly the same error with a fresh installation of Hetzner VMs ```bash :~# cat /var/log/netbird/client.log | tail -50 2026-04-20T18:47:18.717Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:47:18.717Z WARN client/internal/auth/auth.go:419: connection error detected, attempting reconnection: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:47:18.738Z WARN client/internal/auth/auth.go:438: operation failed, retrying in 5.369663318s: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:47:34.122Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:47:34.122Z WARN client/internal/auth/auth.go:419: connection error detected, attempting reconnection: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:47:34.143Z WARN client/internal/auth/auth.go:438: operation failed, retrying in 3.708250653s: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:47:39.975Z INFO client/cmd/root.go:208: shutdown signal received 2026-04-20T18:47:41.985Z INFO client/cmd/service_controller.go:100: stopped NetBird service 2026-04-20T18:48:06.426Z INFO client/cmd/service_controller.go:27: starting NetBird service 2026-04-20T18:48:06.448Z INFO client/server/server.go:969: active profile config existed: true, err <nil> 2026-04-20T18:48:06.458Z INFO client/cmd/service_controller.go:74: started daemon server: /var/run/netbird.sock 2026-04-20T18:48:06.458Z INFO client/internal/connect.go:156: starting NetBird client version 0.69.0 on linux/amd64 2026-04-20T18:48:06.460Z INFO client/net/env_linux.go:70: system supports advanced routing 2026-04-20T18:48:16.812Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:48:28.303Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:48:40.319Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:48:51.562Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:49:03.289Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:49:23.623Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:50:12.204Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:50:18.558Z INFO client/server/server.go:885: service is down 2026-04-20T18:50:18.558Z INFO client/cmd/root.go:208: shutdown signal received 2026-04-20T18:50:23.558Z WARN client/server/server.go:847: timeout waiting for client goroutine to finish, proceeding anyway 2026-04-20T18:50:25.560Z INFO client/cmd/service_controller.go:100: stopped NetBird service 2026-04-20T18:50:25.620Z INFO client/cmd/service_controller.go:27: starting NetBird service 2026-04-20T18:50:25.621Z INFO client/server/server.go:969: active profile config existed: true, err <nil> 2026-04-20T18:50:25.621Z INFO client/cmd/service_controller.go:74: started daemon server: /var/run/netbird.sock 2026-04-20T18:50:25.621Z INFO client/internal/connect.go:156: starting NetBird client version 0.69.0 on linux/amd64 2026-04-20T18:50:25.622Z INFO client/net/env_linux.go:70: system supports advanced routing 2026-04-20T18:50:35.735Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:50:46.159Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:50:59.565Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:51:12.272Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:51:30.067Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:51:44.163Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:52:06.224Z INFO ./caller_not_available:0: 2026/04/20 18:52:06 WARNING: [core] [Server #1] grpc: Server.processUnaryRPC failed to write status: connection error: desc = "transport is closing" 2026-04-20T18:52:14.060Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:52:35.133Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:52:57.461Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:53:20.206Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:53:57.780Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:54:26.746Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:54:58.031Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:55:35.667Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:56:01.135Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:56:19.705Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:56:37.676Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:56:54.975Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:57:06.993Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded 2026-04-20T18:57:20.663Z ERRO shared/management/client/grpc.go:587: failed to login to Management Service: rpc error: code = DeadlineExceeded desc = context deadline exceeded ```
Sign in to join this conversation.
No Label triage-needed
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#11900