mirror of https://github.com/netbirdio/netbird.git synced 2026-03-31 06:24:18 -04:00

Files

Zoltan Papp 91f0d5cefd [client] Feature/client metrics (#5512 )

* Add client metrics

* Add client metrics system with OpenTelemetry and VictoriaMetrics support

Implements a comprehensive client metrics system to track peer connection
stages and performance. The system supports multiple backend implementations
(OpenTelemetry, VictoriaMetrics, and no-op) and tracks detailed connection
stage durations from creation through WireGuard handshake.

Key changes:
- Add metrics package with pluggable backend implementations
- Implement OpenTelemetry metrics backend
- Implement VictoriaMetrics metrics backend
- Add no-op metrics implementation for disabled state
- Track connection stages: creation, semaphore, signaling, connection ready, and WireGuard handshake
- Move WireGuard watcher functionality to conn.go
- Refactor engine to integrate metrics tracking
- Add metrics export endpoint in debug server

* Add signaling metrics tracking for initial and reconnection attempts

* Reset connection stage timestamps during reconnections to exclude unnecessary metrics tracking

* Delete otel lib from client

* Update unit tests

* Invoke callback on handshake success in WireGuard watcher

* Add Netbird version tracking to client metrics

Integrate Netbird version into VictoriaMetrics backend and metrics labels. Update `ClientMetrics` constructor and metric name formatting to include version information.

* Add sync duration tracking to client metrics

Introduce `RecordSyncDuration` for measuring sync message processing time. Update all metrics implementations (VictoriaMetrics, no-op) to support the new method. Refactor `ClientMetrics` to use `AgentInfo` for static agent data.

* Remove no-op metrics implementation and simplify ClientMetrics constructor

Eliminate unused `noopMetrics` and refactor `ClientMetrics` to always use the VictoriaMetrics implementation. Update associated logic to reflect these changes.

* Add total duration tracking for connection attempts

Calculate total duration for both initial connections and reconnections, accounting for different timestamp scenarios. Update `Export` method to include Prometheus HELP comments.

* Add metrics push support to VictoriaMetrics integration

* [client] anchor connection metrics to first signal received

* Remove creation_to_semaphore connection stage metric

The semaphore queuing stage (Created → SemaphoreAcquired) is no longer
tracked. Connection metrics now start from SignalingReceived. Updated
docs and Grafana dashboard accordingly.

* [client] Add remote push config for metrics with version-based eligibility

Introduce remoteconfig.Manager that fetches a remote JSON config to control
metrics push interval and restrict pushing to a specific agent version
range. When NB_METRICS_INTERVAL is set, remote config is bypassed
entirely for local override.

* [client] Add WASM-compatible NewClientMetrics implementation

Replace NewClientMetrics in metrics.go with a WASM-specific stub in metrics_js.go, returning nil for compatibility with JS builds. Simplify method usage for WASM targets.

* Add missing file

* Update default case in DeploymentType.String to return "unknown" instead of "selfhosted"

* [client] Rework metrics to use timestamped samples instead of histograms

Replace cumulative Prometheus histograms with timestamped point-in-time
samples that are pushed once and cleared. This fixes metrics for sparse
events (connections/syncs that happen once at startup) where rate() and
increase() produced incorrect or empty results.

Changes:
- Switch from VictoriaMetrics histogram library to raw Prometheus text
  format with explicit millisecond timestamps
- Reset samples after successful push (no resending stale data)
- Rename connection_to_handshake → connection_to_wg_handshake
- Add netbird_peer_connection_count metric for ICE vs Relay tracking
- Simplify dashboard: point-based scatter plots, donut pie chart
- Add maxStalenessInterval=1m to VictoriaMetrics to prevent forward-fill
- Fix deployment_type Unknown returning "selfhosted" instead of "unknown"
- Fix inverted shouldPush condition in push.go

* [client] Add InfluxDB metrics backend alongside VictoriaMetrics

Add influxdb.go with timestamped line protocol export for sparse
one-shot events. Restore victoria.go to use proper Prometheus
histograms. Update Grafana dashboards, add InfluxDB datasource,
and update docs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* [client] Fix metrics issues and update dev docker setup

- Fix StopPush not clearing push state, preventing restart
- Fix race condition reading currentConnPriority without lock in recordConnectionMetrics
- Fix stale comment referencing old metrics server URL
- Update docker-compose for InfluxDB: add scoped tokens, .env config, init scripts
- Rename docker-compose.victoria.yml to docker-compose.yml

* [client] Add anonymised peer tracking to pushed metrics

Introduce peer_id and connection_pair_id tags to InfluxDB metrics.
Public keys are hashed (truncated SHA-256) for anonymisation. The
connection pair ID is deterministic regardless of which side computes
it, enabling deduplication of reconnections in the ICE vs Relay
dashboard. Also pin Grafana to v11.6.0 for file-based provisioning
and fix datasource UID references.

* Remove unused dependencies from go.mod and go.sum

* Refactor InfluxDB ingest pipeline: extract validation logic

- Move line validation logic to `validateLine` and `validateField` helper functions.
- Improve error handling with structured validation and clearer separation of concerns.
- Add stderr redirection for error messages in `create-tokens.sh`.

* Set non-root user in Dockerfile for Ingest service

* Fix Windows CI: command line too long

* Remove Victoria metrics

* Add hashed peer ID as Authorization header in metrics push

* Revert influxdb in docker compose

* Enable gzip compression and authorization validation for metrics push and ingest

* Reducate code of complexity

* Update debug documentation to include metrics.txt description

* Increase `maxBodySize` limit to 50 MB and update gzip reader wrapping logic

* Refactor deployment type detection to use URL parsing for improved accuracy

* Update readme

* Throttle remote config retries on fetch failure

* Preserve first WG handshake timestamp, ignore rekeys

* Skip adding empty metrics.txt to debug bundle in debug mode

* Update default metrics server URL to https://ingest.netbird.io

* Atomic metrics export-and-reset to prevent sample loss between Export and Reset calls

* Fix doc

* Refactor Push configuration to improve clarity and enforce minimum push interval

* Remove `minPushInterval` and update push interval validation logic

* Revert ExportAndReset, it is acceptable data loss

* Fix metrics review issues: rename env var, remove stale infra, add tests

- Rename NB_METRICS_ENABLED to NB_METRICS_PUSH_ENABLED to clarify that
  collection is always active (for debug bundles) and only push is opt-in
- Change default config URL from staging to production (ingest.netbird.io)
- Delete broken Prometheus dashboard (used non-existent metric names)
- Delete unused VictoriaMetrics datasource config
- Replace committed .env with .env.example containing placeholder values
- Wire Grafana admin credentials through env vars in docker-compose
- Make metricsStages a pointer to prevent reset-vs-write race on reconnect
- Fix typed-nil interface in debug bundle path (GetClientMetrics)
- Use deterministic field order in InfluxDB Export (sorted keys)
- Replace Authorization header with X-Peer-ID for metrics push
- Fix ingest server timeout to use time.Second instead of float
- Fix gzip double-close, stale comments, trim log levels
- Add tests for influxdb.go and MetricsStages

* Add login duration metric, ingest tag validation, and duration bounds

- Add netbird_login measurement recording login/auth duration to management
  server, with success/failure result tag
- Validate InfluxDB tags against per-measurement allowlists in ingest server
  to prevent arbitrary tag injection
- Cap all duration fields (*_seconds) at 300s instead of only total_seconds
- Add ingest server tests for tag/field validation, bounds, and auth

* Add arch tag to all metrics

* Fix Grafana dashboard: add arch to drop columns, add login panels

* Validate NB_METRICS_SERVER_URL is an absolute HTTP(S) URL

* Address review comments: fix README wording, update stale comments

* Clarify env var precedence does not bypass remote config eligibility

* Remove accidentally committed pprof files

---------

Co-authored-by: Viktor Liu <viktor@netbird.io>

2026-03-22 12:45:41 +01:00

7.0 KiB

Raw Blame History

Client Metrics

Internal documentation for the NetBird client metrics system.

Overview

Client metrics track connection performance and sync durations using InfluxDB line protocol (influxdb.go). Each event is pushed once then cleared.

Metrics collection is always active (for debug bundles). Push to backend is:

Disabled by default (opt-in via NB_METRICS_PUSH_ENABLED=true)
Managed at daemon layer (survives engine restarts)

Architecture

Layer Separation

Daemon Layer (connect.go)
  ├─ Creates ClientMetrics instance once
  ├─ Starts/stops push lifecycle
  └─ Updates AgentInfo on profile switch
      │
      ▼
Engine Layer (engine.go)
  └─ Records metrics via ClientMetrics methods

Ingest Server

Clients do not talk to InfluxDB directly. An ingest server sits between clients and InfluxDB:

Client ──POST──▶ Ingest Server (:8087) ──▶ InfluxDB (internal)
                  │
                  ├─ Validates line protocol
                  ├─ Allowlists measurements, fields, and tags
                  ├─ Rejects out-of-bound values
                  └─ Serves remote config at /config

No secret/token-based client auth — the ingest server holds the InfluxDB token server-side. Clients must send a hashed peer ID via X-Peer-ID header.
InfluxDB is not exposed — only accessible within the docker network
Source: ingest/main.go

Metrics Collected

Connection Stage Timing

Measurement: netbird_peer_connection

Field	Timestamps	Description
`signaling_to_connection_seconds`	`SignalingReceived → ConnectionReady`	ICE/relay negotiation time after the first signal is received from the remote peer
`connection_to_wg_handshake_seconds`	`ConnectionReady → WgHandshakeSuccess`	WireGuard cryptographic handshake latency once the transport layer is ready
`total_seconds`	`SignalingReceived → WgHandshakeSuccess`	End-to-end connection time anchored at the first received signal

Tags:

deployment_type: "cloud" | "selfhosted" | "unknown"
connection_type: "ice" | "relay"
attempt_type: "initial" | "reconnection"
version: NetBird version string
os: Operating system (linux, darwin, windows, android, ios, etc.)
arch: CPU architecture (amd64, arm64, etc.)

Note: SignalingReceived is set when the first offer or answer arrives from the remote peer (in both initial and reconnection paths). It excludes the potentially unbounded wait for the remote peer to come online.

Sync Duration

Measurement: netbird_sync

Field	Description
`duration_seconds`	Time to process a sync message from management server

Tags:

deployment_type: "cloud" | "selfhosted" | "unknown"
version: NetBird version string
os: Operating system (linux, darwin, windows, android, ios, etc.)
arch: CPU architecture (amd64, arm64, etc.)

Measurement: netbird_login

Field	Description
`duration_seconds`	Time to complete the login/auth exchange with management server

Tags:

deployment_type: "cloud" | "selfhosted" | "unknown"
result: "success" | "failure"
version: NetBird version string
os: Operating system (linux, darwin, windows, android, ios, etc.)
arch: CPU architecture (amd64, arm64, etc.)

Buffer Limits

The InfluxDB backend limits in-memory sample storage to prevent unbounded growth when pushes fail:

Max age: Samples older than 5 days are dropped
Max size: Estimated buffer size capped at 5 MB (~20k samples)

Configuration

Client Environment Variables

Variable	Default	Description
`NB_METRICS_PUSH_ENABLED`	`false`	Enable metrics push to backend
`NB_METRICS_SERVER_URL`	(from remote config)	Ingest server URL (e.g., `https://ingest.netbird.io`)
`NB_METRICS_INTERVAL`	(from remote config)	Push interval (e.g., "1m", "30m", "4h")
`NB_METRICS_FORCE_SENDING`	`false`	Skip remote config, push unconditionally
`NB_METRICS_CONFIG_URL`	`https://ingest.netbird.io/config`	Remote push config URL

NB_METRICS_SERVER_URL and NB_METRICS_INTERVAL override their respective values but do not bypass remote config eligibility checks (version range). Use NB_METRICS_FORCE_SENDING=true to skip all remote config gating.

Ingest Server Environment Variables

Variable	Default	Description
`INGEST_LISTEN_ADDR`	`:8087`	Listen address
`INFLUXDB_URL`	`http://influxdb:8086/api/v2/write?org=netbird&bucket=metrics&precision=ns`	InfluxDB write endpoint
`INFLUXDB_TOKEN`	(required)	InfluxDB auth token (server-side only)
`CONFIG_METRICS_SERVER_URL`	(empty — disables /config)	`server_url` in the remote config JSON (the URL clients push metrics to)
`CONFIG_VERSION_SINCE`	`0.0.0`	Minimum client version to push metrics
`CONFIG_VERSION_UNTIL`	`99.99.99`	Maximum client version to push metrics
`CONFIG_PERIOD_MINUTES`	`5`	Push interval in minutes

The ingest server serves a remote config JSON at GET /config when CONFIG_METRICS_SERVER_URL is set. Clients can use NB_METRICS_CONFIG_URL=http://<ingest>/config to fetch it.

Configuration Precedence

For URL and Interval, the precedence is:

Environment variable - NB_METRICS_SERVER_URL / NB_METRICS_INTERVAL
Remote config - fetched from NB_METRICS_CONFIG_URL
Default - 5 minute interval, URL from remote config

Push Behavior

StartPush() spawns background goroutine with timer
First push happens immediately on startup
Periodically: push() → Export() → HTTP POST to ingest server
On failure: log error, continue (non-blocking)
On success: Reset() clears pushed samples
StopPush() cancels context and waits for goroutine

Samples are collected with exact timestamps, pushed once, then cleared. No data is resent.

Local Development Setup

1. Configure and Start Services

# From this directory (client/internal/metrics/infra)
cp .env.example .env
# Edit .env to set INFLUXDB_ADMIN_PASSWORD, INFLUXDB_ADMIN_TOKEN, and GRAFANA_ADMIN_PASSWORD
docker compose up -d

This starts:

Ingest server on http://localhost:8087 — accepts client metrics (requires X-Peer-ID header, no secret/token auth)
InfluxDB — internal only, not exposed to host
Grafana on http://localhost:3001

2. Configure Client

export NB_METRICS_PUSH_ENABLED=true
export NB_METRICS_FORCE_SENDING=true
export NB_METRICS_SERVER_URL=http://localhost:8087
export NB_METRICS_INTERVAL=1m

3. Run Client

cd ../../../..
go run ./client/ up

4. View in Grafana

InfluxDB dashboard: http://localhost:3001/d/netbird-influxdb-metrics

5. Verify Data

# Query via InfluxDB (using admin token from .env)
docker compose exec influxdb influx query \
  'from(bucket: "metrics") |> range(start: -1h)' \
  --org netbird

# Check ingest server health
curl http://localhost:8087/health

7.0 KiB Raw Blame History

Client Metrics

Overview

Architecture

Layer Separation

Ingest Server

Metrics Collected

Connection Stage Timing

Sync Duration

Login Duration

Buffer Limits

Configuration

Client Environment Variables

Ingest Server Environment Variables

Configuration Precedence

Push Behavior

Local Development Setup

1. Configure and Start Services

2. Configure Client

3. Run Client

4. View in Grafana

5. Verify Data

7.0 KiB

Raw Blame History