[GH-ISSUE #5646] Ghost proxy peers stuck in "Connecting" — not visible in management API or dashboard, cannot be removed #11584

New Issue

Originally created by @SuperKali on GitHub (Mar 21, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5646

Description

After creating and later removing reverse proxy entries from the NetBird dashboard (web UI), multiple ghost "proxy" peers remain visible in netbird status --detail permanently stuck in "Connecting" status.

These proxy peers:

Are not visible in the management dashboard (Peers section)
Are not returned by the management API (GET /api/peers)
Cannot be removed — no CLI command, API endpoint, or dashboard option exists to delete them
Persist across service restarts (systemctl restart netbird)
Inflate the peer count (showing 6/16 Connected instead of 6/9)

Environment

NetBird version: 0.66.4 (daemon + CLI)
Deployment: Self-hosted (Zitadel IDP)
OS: Linux aarch64 (Ubuntu, kernel 6.8.0)
Interface type: Kernel

Steps to reproduce

Set up a self-hosted NetBird instance
Create reverse proxy entries from the dashboard web UI targeting services on various peers
Later remove/delete those reverse proxy entries from the dashboard
Observe that netbird status --detail still shows proxy peers that were associated with the deleted proxy configurations

Current behavior

netbird status --detail shows 7 ghost proxy peers with hostnames matching the pattern:
proxy---.

Example output:
proxy-XXXXXX-8-9.example.tun:
NetBird IP: 100.71.8.9
Status: Connecting
Connection type: -
ICE candidate (Local/Remote): -/-
Last connection update: 6 days ago
Last WireGuard handshake: -
Transfer status (received/sent) 0 B/0 B

Key observations:

7 proxy peers total: 6 stuck in "Connecting", 1 "Connected"
The proxy peer IDs do not match any peer ID returned by GET /api/peers (17 real peers vs 7 phantom proxies)
All stuck proxies show Latency: 0s, no WireGuard handshake, and 0 B transferred
Last connection update frozen at 6+ days ago for most
After systemctl restart netbird, the proxy peers reappear immediately (re-injected via network map from management/signal server)
No proxy-related API endpoints available (/api/proxies, /api/reverse-proxies, /api/exposed-services all return 404)

Expected behavior

When a reverse proxy entry is deleted from the dashboard, the associated proxy peer should be cleaned up from all clients' network maps
Ghost proxy peers should not persist indefinitely
There should be an API endpoint to list/manage reverse proxy peers (currently no endpoint exists)

Additional context

The reverse proxy entries were created and managed entirely through the dashboard UI, not via netbird expose CLI
No proxy-related entries found in journalctl -u netbird
The proxy peers appear to be injected into the network map by the management server but are not exposed through the standard peers API, making them impossible to manage or remove
Possibly related to #5421 (stale proxy entries preventing peer removal)

Originally created by @SuperKali on GitHub (Mar 21, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5646 ## Description After creating and later removing reverse proxy entries from the **NetBird dashboard** (web UI), multiple ghost "proxy" peers remain visible in `netbird status --detail` permanently stuck in **"Connecting"** status. These proxy peers: 1. Are **not visible** in the management dashboard (Peers section) 2. Are **not returned** by the management API (`GET /api/peers`) 3. **Cannot be removed** — no CLI command, API endpoint, or dashboard option exists to delete them 4. **Persist across service restarts** (`systemctl restart netbird`) 5. Inflate the peer count (showing 6/16 Connected instead of 6/9) ## Environment - **NetBird version:** 0.66.4 (daemon + CLI) - **Deployment:** Self-hosted (Zitadel IDP) - **OS:** Linux aarch64 (Ubuntu, kernel 6.8.0) - **Interface type:** Kernel ## Steps to reproduce 1. Set up a self-hosted NetBird instance 2. Create reverse proxy entries from the **dashboard web UI** targeting services on various peers 3. Later remove/delete those reverse proxy entries from the dashboard 4. Observe that `netbird status --detail` still shows proxy peers that were associated with the deleted proxy configurations ## Current behavior `netbird status --detail` shows 7 ghost proxy peers with hostnames matching the pattern: proxy---. Example output: proxy-XXXXXX-8-9.example.tun: NetBird IP: 100.71.8.9 Status: Connecting Connection type: - ICE candidate (Local/Remote): -/- Last connection update: 6 days ago Last WireGuard handshake: - Transfer status (received/sent) 0 B/0 B **Key observations:** - 7 proxy peers total: **6 stuck in "Connecting"**, 1 "Connected" - The proxy peer IDs do **not match** any peer ID returned by `GET /api/peers` (17 real peers vs 7 phantom proxies) - All stuck proxies show `Latency: 0s`, no WireGuard handshake, and 0 B transferred - `Last connection update` frozen at 6+ days ago for most - After `systemctl restart netbird`, the proxy peers reappear immediately (re-injected via network map from management/signal server) - No proxy-related API endpoints available (`/api/proxies`, `/api/reverse-proxies`, `/api/exposed-services` all return 404) ## Expected behavior - When a reverse proxy entry is deleted from the dashboard, the associated proxy peer should be cleaned up from all clients' network maps - Ghost proxy peers should not persist indefinitely - There should be an API endpoint to list/manage reverse proxy peers (currently no endpoint exists) ## Additional context - The reverse proxy entries were created and managed entirely through the **dashboard UI**, not via `netbird expose` CLI - No `proxy`-related entries found in `journalctl -u netbird` - The proxy peers appear to be injected into the network map by the management server but are not exposed through the standard peers API, making them impossible to manage or remove - Possibly related to #5421 (stale proxy entries preventing peer removal)

saavagebueno added the triage-needed label 2026-08-05 01:30:06 -04:00

saavagebueno commented

@heymoe commented on GitHub (Mar 22, 2026):

This is likely related to #5421 that you referenced and #5398 which I believe was the first issue to report this bug. It was fixed in v0.66.0 but It's unclear if the fix retroactively clears the ghosts entries but that seems to be a no in your case.

I personally ran into this issue and before the fix and I was able to manually delete the ghost entries by removing them from the targets table in the store.db sqlite3 database Netbird server / management uses.

The general steps to remove the ghost entries manually would look something like:

get access to the store.db file
use sqlite3 to "edit" the contents of the store.db (example command: sqlite3 store.db). Install sqlite3 if needed
view a list of existing targets: select * from targets; . Might be good idea to change the output mode to table or column first: .mode column so you can see what each piece of data is.
remove the ghosts: delete from targets where id=###; (replace ### with the ghost target id number)
repeat 4 until all ghosts are gone
exit sqlite3: .quit

The tricky part might be step 1 depending on how familiar you are with docker and linux. Some options might be: convert the docker volumes to bind mounts OR exec a shell into the netbird-server (or management) container. I'm sure there might be other options but those are the ones off the top of my head. I personally went the bind mount way just for easier access and backups of all the Netbird data in general.

Definitely make a backup of your working deployment before you do any of this.

@heymoe commented on GitHub (Mar 22, 2026): This is likely related to #5421 that you referenced and #5398 which I believe was the first issue to report this bug. It was fixed in v0.66.0 but It's unclear if the fix retroactively clears the ghosts entries but that seems to be a no in your case. I personally ran into this issue and before the fix and I was able to manually delete the ghost entries by removing them from the targets table in the store.db sqlite3 database Netbird server / management uses. The general steps to remove the ghost entries manually would look something like: 1) get access to the store.db file 2) use sqlite3 to "edit" the contents of the store.db (example command: `sqlite3 store.db`). Install sqlite3 if needed 3) view a list of existing targets: `select * from targets;` . Might be good idea to change the output mode to table or column first: `.mode column` so you can see what each piece of data is. 4) remove the ghosts: `delete from targets where id=###;` (replace ### with the ghost target id number) 5) repeat 4 until all ghosts are gone 6) exit sqlite3: `.quit` The tricky part might be step 1 depending on how familiar you are with docker and linux. Some options might be: convert the docker volumes to bind mounts OR exec a shell into the netbird-server (or management) container. I'm sure there might be other options but those are the ones off the top of my head. I personally went the bind mount way just for easier access and backups of all the Netbird data in general. Definitely make a backup of your working deployment before you do any of this.

saavagebueno commented

@SuperKali commented on GitHub (Mar 23, 2026):

Confirmed working on v0.66.4 (self-hosted, Docker, Linux aarch64).

A few clarifications that might help others:

The ghost peers are in the peers table, not targets — the table suggested above didn't have a name column in my case. The correct query is:

select id, name from peers where name like 'proxy-%';

To identify which ones are actually ghosts vs legitimate, cross-reference with netbird status --detail (ghosts show Connecting, 0 B transferred, no WireGuard handshake) and with the services table — only one proxy peer per active service should exist:

select id, name, enabled from services;

Then delete only the ghosts, keeping any connected one:

delete from peers where name like 'proxy-%' and id != '<id_to_keep>';

After docker restart netbird-management and systemctl restart netbird on the clients, the ghost peers are gone and the peer count is correct. The management server recreates legitimate proxy peers automatically.

@SuperKali commented on GitHub (Mar 23, 2026): Confirmed working on v0.66.4 (self-hosted, Docker, Linux aarch64). A few clarifications that might help others: **The ghost peers are in the `peers` table, not `targets`** — the table suggested above didn't have a `name` column in my case. The correct query is: ```sql select id, name from peers where name like 'proxy-%'; ``` To identify which ones are actually ghosts vs legitimate, cross-reference with `netbird status --detail` (ghosts show `Connecting`, 0 B transferred, no WireGuard handshake) and with the `services` table — only one proxy peer per active service should exist: ```sql select id, name, enabled from services; ``` Then delete only the ghosts, keeping any connected one: ```sql delete from peers where name like 'proxy-%' and id != '<id_to_keep>'; ``` After `docker restart netbird-management` and `systemctl restart netbird` on the clients, the ghost peers are gone and the peer count is correct. The management server recreates legitimate proxy peers automatically.

saavagebueno commented

@SuperKali commented on GitHub (Mar 30, 2026):

The problem still persists; I don't know how to solve it once and for all.

@SuperKali commented on GitHub (Mar 30, 2026): The problem still persists; I don't know how to solve it once and for all.

saavagebueno commented

@SuperKali commented on GitHub (Mar 30, 2026):

I dug into the source code and found the exact reason why ghost proxy peers accumulate.

The bug

CreateProxyPeer in management/internals/shared/grpc/proxy.go always calls peersManager.CreateProxyPeer() without checking for or cleaning up pre-existing proxy peers for the same cluster. Every time the proxy restarts it gets a new timestamped ID (e.g. netbird-proxy-20260329150431) and registers a brand-new record in the peers table — the old ones are never deleted.

There is a cleanupStaleProxies goroutine that runs every 5 minutes, but it only calls proxyManager.CleanupStale() which updates the proxies table. The peers table is never touched:

// proxy.go — runs every 5 min, cleans `proxies` table only
func (s *ProxyServiceServer) cleanupStaleProxies(ctx context.Context) {
    ticker := time.NewTicker(5 * time.Minute)
    // ...
    s.proxyManager.CleanupStale(ctx, 1*time.Hour) // never touches `peers`
}

// CreateProxyPeer — always inserts a new peer, never deletes old ones
func (s *ProxyServiceServer) CreateProxyPeer(...) {
    // ...
    err := s.peersManager.CreateProxyPeer(ctx, accountID, key, cluster)
    // no cleanup of previous proxy peers for the same cluster
}

This means orphaned peers in the peers table accumulate indefinitely with a stale peer_status_connected = 1, because their status is only updated via WireGuard heartbeats — which obviously never come from a peer that no longer exists.

Confirmed in the database

Cross-referencing peers and proxies makes the ghost peers immediately identifiable:

-- Ghost peers: last_seen is older than the active proxy's connected_at
SELECT p.name, p.peer_status_connected, p.peer_status_last_seen
FROM peers p, proxies px
WHERE p.name LIKE 'proxy-%'
  AND px.status = 'connected'
  AND p.peer_status_last_seen < px.connected_at
ORDER BY p.peer_status_last_seen;

In my case this returned 3 ghost peers with last_seen 6+ days old, while the active proxy (proxies.status = connected) had connected_at from a recent restart.

Suggested fix

Two possible approaches:

Option A — clean up in CreateProxyPeer before inserting the new peer:

func (s *ProxyServiceServer) CreateProxyPeer(ctx, accountID, key, cluster) {
    // Delete stale proxy peers for this cluster before creating a new one
    if err := s.peersManager.DeleteStaleProxyPeers(ctx, accountID, cluster); err != nil {
        log.Warnf("failed to cleanup stale proxy peers for cluster %s: %v", cluster, err)
    }
    err := s.peersManager.CreateProxyPeer(ctx, accountID, key, cluster)
    // ...
}

Option B — extend CleanupStale to also remove orphaned proxy peers from the peers table when no active proxy exists for their cluster.

Workaround (until fixed)

DELETE FROM peers
WHERE name LIKE 'proxy-%'
  AND peer_status_last_seen < (
    SELECT connected_at FROM proxies
    WHERE status = 'connected'
    ORDER BY connected_at DESC LIMIT 1
  );

Followed by docker restart netbird-management. The management server will recreate legitimate proxy peers automatically on the next proxy heartbeat.

@SuperKali commented on GitHub (Mar 30, 2026): I dug into the source code and found the exact reason why ghost proxy peers accumulate. ### The bug `CreateProxyPeer` in `management/internals/shared/grpc/proxy.go` always calls `peersManager.CreateProxyPeer()` without checking for or cleaning up pre-existing proxy peers for the same cluster. Every time the proxy restarts it gets a new timestamped ID (e.g. `netbird-proxy-20260329150431`) and registers a brand-new record in the `peers` table — the old ones are never deleted. There is a `cleanupStaleProxies` goroutine that runs every 5 minutes, but it only calls `proxyManager.CleanupStale()` which updates the `proxies` table. The `peers` table is never touched: ```go // proxy.go — runs every 5 min, cleans `proxies` table only func (s *ProxyServiceServer) cleanupStaleProxies(ctx context.Context) { ticker := time.NewTicker(5 * time.Minute) // ... s.proxyManager.CleanupStale(ctx, 1*time.Hour) // never touches `peers` } // CreateProxyPeer — always inserts a new peer, never deletes old ones func (s *ProxyServiceServer) CreateProxyPeer(...) { // ... err := s.peersManager.CreateProxyPeer(ctx, accountID, key, cluster) // no cleanup of previous proxy peers for the same cluster } ``` This means orphaned peers in the `peers` table accumulate indefinitely with a stale `peer_status_connected = 1`, because their status is only updated via WireGuard heartbeats — which obviously never come from a peer that no longer exists. ### Confirmed in the database Cross-referencing `peers` and `proxies` makes the ghost peers immediately identifiable: ```sql -- Ghost peers: last_seen is older than the active proxy's connected_at SELECT p.name, p.peer_status_connected, p.peer_status_last_seen FROM peers p, proxies px WHERE p.name LIKE 'proxy-%' AND px.status = 'connected' AND p.peer_status_last_seen < px.connected_at ORDER BY p.peer_status_last_seen; ``` In my case this returned 3 ghost peers with `last_seen` 6+ days old, while the active proxy (`proxies.status = connected`) had `connected_at` from a recent restart. ### Suggested fix Two possible approaches: **Option A** — clean up in `CreateProxyPeer` before inserting the new peer: ```go func (s *ProxyServiceServer) CreateProxyPeer(ctx, accountID, key, cluster) { // Delete stale proxy peers for this cluster before creating a new one if err := s.peersManager.DeleteStaleProxyPeers(ctx, accountID, cluster); err != nil { log.Warnf("failed to cleanup stale proxy peers for cluster %s: %v", cluster, err) } err := s.peersManager.CreateProxyPeer(ctx, accountID, key, cluster) // ... } ``` **Option B** — extend `CleanupStale` to also remove orphaned proxy peers from the `peers` table when no active proxy exists for their cluster. ### Workaround (until fixed) ```sql DELETE FROM peers WHERE name LIKE 'proxy-%' AND peer_status_last_seen < ( SELECT connected_at FROM proxies WHERE status = 'connected' ORDER BY connected_at DESC LIMIT 1 ); ``` Followed by `docker restart netbird-management`. The management server will recreate legitimate proxy peers automatically on the next proxy heartbeat.

saavagebueno commented

@pascal-fischer commented on GitHub (Mar 31, 2026):

So the idea is that the peers are added as ephemeral peers. You are correct that on every reboot the proxy will register new peers. The idea is that those are short lived and are being cleaned by the normal ephemeral peer cleanup (which is deleting them after they are disconnected for more than 10 min).

Next time you have some ghost proxy peers can you look into the DB and check for ephemeral, peer_status_connected and peer_status_last_seen

@pascal-fischer commented on GitHub (Mar 31, 2026): So the idea is that the peers are added as ephemeral peers. You are correct that on every reboot the proxy will register new peers. The idea is that those are short lived and are being cleaned by the normal ephemeral peer cleanup (which is deleting them after they are disconnected for more than 10 min). Next time you have some ghost proxy peers can you look into the DB and check for `ephemeral`, `peer_status_connected` and `peer_status_last_seen`

saavagebueno commented

@SuperKali commented on GitHub (Apr 2, 2026):

@pascal-fischer Confirmed in DB:

sqlite3 /var/lib/docker/volumes/netbird_netbird_management/_data/store.db \
  "SELECT name, ephemeral, peer_status_connected, peer_status_last_seen FROM peers WHERE name LIKE 'proxy-%' ORDER BY peer_status_last_seen DESC;"

proxy-d770f5jm9lgs73cvovb0|1|1|2026-04-02 06:07:19  ← active proxy
proxy-d74jv5bm9lgs73efpghg|1|1|2026-03-30 09:46:37  ← ghost, 3 days old

Both ephemeral=1, both peer_status_connected=1.
The ghost peer never got flipped to connected=0, so the 10-min ephemeral cleanup never triggered.

@SuperKali commented on GitHub (Apr 2, 2026): @pascal-fischer Confirmed in DB: ```bash sqlite3 /var/lib/docker/volumes/netbird_netbird_management/_data/store.db \ "SELECT name, ephemeral, peer_status_connected, peer_status_last_seen FROM peers WHERE name LIKE 'proxy-%' ORDER BY peer_status_last_seen DESC;" ``` ```bash proxy-d770f5jm9lgs73cvovb0|1|1|2026-04-02 06:07:19 ← active proxy proxy-d74jv5bm9lgs73efpghg|1|1|2026-03-30 09:46:37 ← ghost, 3 days old ``` Both ephemeral=1, both peer_status_connected=1. The ghost peer never got flipped to connected=0, so the 10-min ephemeral cleanup never triggered.

saavagebueno commented

@pascal-fischer commented on GitHub (Apr 7, 2026):

Hi, great. So we have an issue with the status field that we are aware of. This can also happen with regular peers as there is a race condition. We want to move to a heartbeat approach to recover the connected states. This is something we planned and will be implementing in the near future. I can not give you an exact date yet. Meanwhile cleaning in the database is the only option I fear

@pascal-fischer commented on GitHub (Apr 7, 2026): Hi, great. So we have an issue with the status field that we are aware of. This can also happen with regular peers as there is a race condition. We want to move to a heartbeat approach to recover the connected states. This is something we planned and will be implementing in the near future. I can not give you an exact date yet. Meanwhile cleaning in the database is the only option I fear

saavagebueno commented

@SuperKali commented on GitHub (Apr 7, 2026):

The interesting thing is that you know about it and that it hasn't been forgotten; it's a real problem that needs to be solved as soon as possible.

@SuperKali commented on GitHub (Apr 7, 2026): The interesting thing is that you know about it and that it hasn't been forgotten; it's a real problem that needs to be solved as soon as possible.

saavagebueno commented

@arjones852 commented on GitHub (Apr 30, 2026):

This is still an issue. Can confirm as a new user that started at version 0.69, I have 6 ghost peers in my list that started me down the rabbit hole of what are these and why are they there.

@arjones852 commented on GitHub (Apr 30, 2026): This is still an issue. Can confirm as a new user that started at version 0.69, I have 6 ghost peers in my list that started me down the rabbit hole of what are these and why are they there.

saavagebueno commented

@SuperKali commented on GitHub (Apr 30, 2026):

The issue is still present with latest version on my side

@SuperKali commented on GitHub (Apr 30, 2026): The issue is still present with latest version on my side

saavagebueno commented

@towerplease commented on GitHub (May 30, 2026):

Can confirm this issue on the latest version (0.71.4)

@towerplease commented on GitHub (May 30, 2026): Can confirm this issue on the latest version (0.71.4)

saavagebueno commented

@dapru commented on GitHub (Jun 8, 2026):

Still an issue with the current version (0.72.2). Noticed it due to connection problems with android clients

@dapru commented on GitHub (Jun 8, 2026): Still an issue with the current version (0.72.2). Noticed it due to connection problems with android clients

saavagebueno commented

@salsugair commented on GitHub (Jun 21, 2026):

Can confirm this issue on the latest version (0.73.1)

@salsugair commented on GitHub (Jun 21, 2026): Can confirm this issue on the latest version (0.73.1)

saavagebueno commented

@Xury46 commented on GitHub (Jun 21, 2026):

I ran into this exact issue as well, I tried editing the store.db but it's encrypted and I decided it wasn't work potentially bricking my network. I would love to have a better way to clean this up.

@Xury46 commented on GitHub (Jun 21, 2026): I ran into this exact issue as well, I tried editing the store.db but it's encrypted and I decided it wasn't work potentially bricking my network. I would love to have a better way to clean this up.

saavagebueno commented

@SuperKali commented on GitHub (Jun 21, 2026):

The issue still persists, hope in a fix soon, proxy table is growing on my side.

@SuperKali commented on GitHub (Jun 21, 2026): The issue still persists, hope in a fix soon, proxy table is growing on my side.

saavagebueno commented

@garrettlaman commented on GitHub (Jun 26, 2026):

This issue seems to cause DNS resolution problems on Android when connecting to Netbird-Only Access services. When an Android client resolves a Netbird-Only Access proxy host, the Netbird resolver returns overlay IP addresses for the ghost proxies in addition to the active proxy.

When connecting to proxied services, apps are using the ghost overlay IP addresses instead of the active proxy IP, effectively breaking proxy functionality on Android.

In my case I'm up to 25 ghost proxies, so each DNS request is returning 25 stale IPs and the 1 active proxy IP, which is never used.

@garrettlaman commented on GitHub (Jun 26, 2026): This issue seems to cause DNS resolution problems on Android when connecting to Netbird-Only Access services. When an Android client resolves a Netbird-Only Access proxy host, the Netbird resolver returns overlay IP addresses for the ghost proxies in addition to the active proxy. When connecting to proxied services, apps are using the ghost overlay IP addresses instead of the active proxy IP, effectively breaking proxy functionality on Android. In my case I'm up to 25 ghost proxies, so each DNS request is returning 25 stale IPs and the 1 active proxy IP, which is never used.

saavagebueno commented

@rwtallant13 commented on GitHub (Jun 27, 2026):

Same issue here and I believe the DNS issues go beyond just on Android. On my Linux system my netbird-only access sites do not load since the DNS returns a unconnected proxy IP.

@rwtallant13 commented on GitHub (Jun 27, 2026): Same issue here and I believe the DNS issues go beyond just on Android. On my Linux system my netbird-only access sites do not load since the DNS returns a unconnected proxy IP.

saavagebueno commented

@Aurel004 commented on GitHub (Jul 1, 2026):

Got 70+ proxy-peers for me, each time Netbird restarts, it creates a new peer

@Aurel004 commented on GitHub (Jul 1, 2026): Got 70+ proxy-peers for me, each time Netbird restarts, it creates a new peer

saavagebueno commented