[PR #6010] [CLOSED] [proxy,management] per-service ACME with DNS-01 multi-provider support #28898

Open
opened 2026-08-05 08:07:07 -04:00 by saavagebueno · 0 comments
Owner

📋 Pull Request Information

Original PR: https://github.com/netbirdio/netbird/pull/6010
Author: @TechHutTV
Created: 4/28/2026
Status: Closed

Base: mainHead: brandon/dns-lego-feature


📝 Commits (10+)

  • 5649037 [proxy,management] CertBackend abstraction + per-service DNS-01 fields (Wave 1)
  • 372f972 Per-service ACME backend routing
  • f651732 Proxy fetches encrypted credentials at issuance
  • e426560 Four DNS providers via pluggable adapter registry
  • 10b118e Added DNS provider record writer for auto-configure
  • 2e296aa Plumbed Private bool through proto, struct, validation, OpenAPI, proxy log
  • 3aad17a Auto-managed DNS records for private services
  • 5e5d5dc Proxy now blocks public access to private services
  • 91dfcb4 Cleaned wave refs; added logging, error polish, security packet
  • 14058e5 Regenerate codegen + add management/idp/base62 to proxy Dockerfile

📊 Changes

76 files changed (+9769 additions, -786 deletions)

View changed files

📝 go.mod (+46 -36)
📝 go.sum (+102 -77)
management/internals/modules/credentials/credential.go (+44 -0)
management/internals/modules/credentials/manager/manager.go (+239 -0)
management/internals/modules/credentials/manager/manager_test.go (+349 -0)
management/internals/modules/credentials/providertypes/types.go (+28 -0)
management/internals/modules/credentials/recordwriter/apex.go (+62 -0)
management/internals/modules/credentials/recordwriter/cloudflare.go (+269 -0)
management/internals/modules/credentials/recordwriter/cloudflare_test.go (+231 -0)
management/internals/modules/credentials/recordwriter/digitalocean.go (+346 -0)
management/internals/modules/credentials/recordwriter/digitalocean_test.go (+255 -0)
management/internals/modules/credentials/recordwriter/recordwriter.go (+105 -0)
management/internals/modules/credentials/recordwriter/rfc2136.go (+271 -0)
management/internals/modules/credentials/recordwriter/rfc2136_test.go (+593 -0)
management/internals/modules/credentials/recordwriter/route53.go (+310 -0)
management/internals/modules/credentials/recordwriter/route53_test.go (+299 -0)
management/internals/modules/credentials/secretpayload/payload.go (+55 -0)
management/internals/modules/credentials/secretpayload/payload_test.go (+59 -0)
📝 management/internals/modules/reverseproxy/domain/domain.go (+29 -1)
📝 management/internals/modules/reverseproxy/domain/manager/api.go (+29 -1)

...and 56 more files

📄 Description

Summary (proof of concept for testing)

Adds DNS-01 ACME issuance to the reverse-proxy alongside the existing tls-alpn-01 / http-01 flows, with per-service routing so each service decides which challenge type and DNS provider to use. DNS provider credentials are stored encrypted on the management server and fetched by the proxy at issuance time over a new gRPC method.

Out of the box, four DNS providers are supported: Cloudflare, AWS Route 53, DigitalOcean, and RFC 2136 (BIND, PowerDNS, Knot, NSD). Adding a new provider is a single registry entry on the proxy plus a backend adapter.

Existing services keep working unchanged — they fall back to the proxy's globally-configured default challenge type, which remains tls-alpn-01.

The end-to-end DNS-01 path was first verified on a throwaway spike branch (Cloudflare → Let's Encrypt staging on a real domain) before being lifted into this PR in four reviewable commits.

Pairs with the dashboard PR on brandon/dns-lego-ui (TechHutTV/dashboard).

Changes

Proxy — ACME refactor

  • proxy/internal/acme/backend.go — new CertBackend interface (GetCertificate, ReadCertFromDisk, DeleteCert) so different challenge types can plug in.
  • proxy/internal/acme/autocert_backend.go — wraps the existing autocert.Manager and satisfies CertBackend; handles tls-alpn-01 and http-01 exactly as before.
  • proxy/internal/acme/lego_backend.go — new Lego-based backend for dns-01. Asymmetric semantics vs. autocert: GetCertificate is load-from-disk only; Issue(ctx) performs the actual DNS-01 dance and writes to disk. Manager prefetches certificates for dns-01 services so TLS handshakes don't block on issuance.
  • proxy/internal/acme/manager.go — refactored from embedding autocert.Manager to holding a multi-backend map keyed by challenge type, with a configurable default. Locker, domain state, wildcard handling, notifier, and metrics stay at the orchestrator level.
  • proxy/internal/acme/legoclient/ — new package: an account+client cache around go-acme/lego/v4, plus per-provider adapters and a registry.
    • provider_cloudflare.go, provider_route53.go, provider_digitalocean.go, provider_rfc2136.go
    • providers.go — central registry; new providers register themselves here.
  • proxy/server.go — wires the multi-backend manager. The challenge-type env var (NB_PROXY_ACME_CHALLENGE_TYPE) becomes the default; per-service config overrides it.

Management — credentials store and resolver

  • management/server/credentials.go — encrypted credential record. Multi-field secrets (Route 53, RFC 2136) are JSON-encoded into the existing EncryptedSecret column with a legacy single-string fallback so older records keep working.
  • management/server/http/handlers/credentials/credentials_handler.go — new HTTP CRUD: POST /api/credentials, GET /api/credentials, GET /api/credentials/{id}, PUT /api/credentials/{id}, DELETE /api/credentials/{id}. Secrets are never returned in responses; PUT accepts a partial update so the secret can be rotated independently of the name.
  • management/server/store/sql_store.go — store methods for credentials (account-scoped). Includes a GetByRefWithSecret helper used by the resolver to decrypt at fetch time.
  • management/internals/shared/grpc/proxy.go — implements the new ResolveCredential RPC. Audit-logs every resolve.
  • management/server/activity/codes.go — new audit codes for credential create / update / delete / resolve.
  • management/server/permissions/modules/module.go — new permission module entry; access is currently gated alongside the existing services module pending a dedicated dashboard permission.
  • AutoMigrate handles the new credentials table on management startup.

Per-service config

  • shared/management/proto/proxy_service.protoProxyMapping gains optional fields challenge_type (13), dns_provider (14), dns_credentials_ref (15). New ResolveCredential RPC + request/response messages.
  • shared/management/http/api/openapi.yml — the same three fields on the reverse-proxy service schema; full credentials API surface; companion Go types regenerated into types.gen.go.
  • management/modules/reverseproxy/service/service.go — per-service fields plumbed through GORM AutoMigrate.
  • management/modules/reverseproxy/service/manager/manager.go — new validateACMEConfig enforces internal consistency (e.g., dns-01 requires dns_provider; non-dns-01 challenge types reject dns_provider / dns_credentials_ref).

Tests

  • proxy/internal/acme/manager_test.go — multi-backend routing, prefetch, locker behavior, ReadCertFromDisk short-circuit.
  • proxy/internal/acme/lego_backend_test.go — backend semantics including the load-from-disk-only GetCertificate contract.
  • proxy/internal/acme/locker_test.go — same-domain serialization and different-domain parallelism.
  • proxy/internal/acme/legoclient/providers_test.go — registry + per-provider adapter shape (no live ACME calls).
  • management/server/http/handlers/credentials/credentials_handler_test.go — full CRUD coverage including the JSON-encoded secret round-trip and the legacy single-string fallback.
  • management/modules/reverseproxy/service/manager/manager_test.govalidateACMEConfig table tests covering allowed and rejected combinations.

Out of scope (intentional)

  • Custom Provider escape hatch — Lego's NewDNSChallengeProviderByName is env-var-only and doesn't compose cleanly with per-service in-memory credentials. Deferred until there's demand for a provider not in the registry.
  • Pebble-based integration tests in CI — exercised manually via the spike; CI integration is its own hardening pass.
  • memguard-style in-process secret hardening — the encryption-at-rest model is in place; in-memory hardening is a separate review.
  • Status surfacing in the dashboard services list — covered by a follow-up dashboard slice.

How to test

  1. Apply this branch to a management server and proxy in the same environment; restart both. AutoMigrate creates the credentials table and adds the three new columns to reverse_proxies.
  2. Add a Cloudflare credential via POST /api/credentials (or via the dashboard PR's new DNS Credentials page).
  3. Create a reverse-proxy service with challenge_type: "dns-01", dns_provider: "cloudflare", dns_credentials_ref: <id from step 2>, and a domain in that Cloudflare zone.
  4. Tail the proxy log — you should see a TXT-record write to Cloudflare, then a successful Let's Encrypt issuance, then curl https://your-domain returns the configured backend.
  5. Repeat with Route 53 (multi-field credential), DigitalOcean (single token, exercises the auth_token adapter path), and RFC 2136 (TSIG path) against your own zones to exercise each adapter end-to-end.
  6. Edit a credential's secret via PUT /api/credentials/{id} with secret_fields set — confirm the next renewal picks up the new secret without restarting either service.
  7. Delete a credential currently referenced by a service — the management API should reject (409) so a live service can't be silently broken.

Verification

  • go build ./... — clean.
  • go test ./proxy/... ./management/... ./shared/... — passes locally. One pre-existing Docker-dependent store test is skipped in environments without Docker; not introduced by this PR.
  • Spike branch verified the Cloudflare DNS-01 path end-to-end against Let's Encrypt staging on a real domain prior to landing.

Stack

Checklist

  • Is it a bug fix
  • Is a typo/documentation fix
  • Is a feature enhancement
  • It is a refactor
  • Created tests that fail without the change (if possible)

By submitting this pull request, you confirm that you have read and agree to the terms of the Contributor License Agreement.

Documentation

Select exactly one:

  • I added/updated documentation for this change
  • Documentation is not needed for this change (explain why)

Docs PR URL (required if "docs added" is checked)

Paste the PR link from https://github.com/netbirdio/docs here:

https://github.com/netbirdio/docs/pull/__


🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/netbirdio/netbird/pull/6010 **Author:** [@TechHutTV](https://github.com/TechHutTV) **Created:** 4/28/2026 **Status:** ❌ Closed **Base:** `main` ← **Head:** `brandon/dns-lego-feature` --- ### 📝 Commits (10+) - [`5649037`](https://github.com/netbirdio/netbird/commit/5649037447594bfe6918684560c4395c18efe0ef) [proxy,management] CertBackend abstraction + per-service DNS-01 fields (Wave 1) - [`372f972`](https://github.com/netbirdio/netbird/commit/372f97233b6628803429f970c72f59fc65cb25a3) Per-service ACME backend routing - [`f651732`](https://github.com/netbirdio/netbird/commit/f65173297fa910f9771cb507eb7fb1e20a25d9fa) Proxy fetches encrypted credentials at issuance - [`e426560`](https://github.com/netbirdio/netbird/commit/e426560c47eae53f92f9812af6e251490561a08b) Four DNS providers via pluggable adapter registry - [`10b118e`](https://github.com/netbirdio/netbird/commit/10b118e21bf9b1ed185a01515b070981389660cf) Added DNS provider record writer for auto-configure - [`2e296aa`](https://github.com/netbirdio/netbird/commit/2e296aad7af8992d5248773dd4e3fdd2b506488d) Plumbed Private bool through proto, struct, validation, OpenAPI, proxy log - [`3aad17a`](https://github.com/netbirdio/netbird/commit/3aad17a599ae726c28d9e3c7d1b19fab8d80ec20) Auto-managed DNS records for private services - [`5e5d5dc`](https://github.com/netbirdio/netbird/commit/5e5d5dcf33c2adddb1b498e6140f51aac5b923b3) Proxy now blocks public access to private services - [`91dfcb4`](https://github.com/netbirdio/netbird/commit/91dfcb4b145717e34f5fbc8b72020dda87c92ea2) Cleaned wave refs; added logging, error polish, security packet - [`14058e5`](https://github.com/netbirdio/netbird/commit/14058e5d653b0540daec2b5525bd11f1963a56b4) Regenerate codegen + add management/idp/base62 to proxy Dockerfile ### 📊 Changes **76 files changed** (+9769 additions, -786 deletions) <details> <summary>View changed files</summary> 📝 `go.mod` (+46 -36) 📝 `go.sum` (+102 -77) ➕ `management/internals/modules/credentials/credential.go` (+44 -0) ➕ `management/internals/modules/credentials/manager/manager.go` (+239 -0) ➕ `management/internals/modules/credentials/manager/manager_test.go` (+349 -0) ➕ `management/internals/modules/credentials/providertypes/types.go` (+28 -0) ➕ `management/internals/modules/credentials/recordwriter/apex.go` (+62 -0) ➕ `management/internals/modules/credentials/recordwriter/cloudflare.go` (+269 -0) ➕ `management/internals/modules/credentials/recordwriter/cloudflare_test.go` (+231 -0) ➕ `management/internals/modules/credentials/recordwriter/digitalocean.go` (+346 -0) ➕ `management/internals/modules/credentials/recordwriter/digitalocean_test.go` (+255 -0) ➕ `management/internals/modules/credentials/recordwriter/recordwriter.go` (+105 -0) ➕ `management/internals/modules/credentials/recordwriter/rfc2136.go` (+271 -0) ➕ `management/internals/modules/credentials/recordwriter/rfc2136_test.go` (+593 -0) ➕ `management/internals/modules/credentials/recordwriter/route53.go` (+310 -0) ➕ `management/internals/modules/credentials/recordwriter/route53_test.go` (+299 -0) ➕ `management/internals/modules/credentials/secretpayload/payload.go` (+55 -0) ➕ `management/internals/modules/credentials/secretpayload/payload_test.go` (+59 -0) 📝 `management/internals/modules/reverseproxy/domain/domain.go` (+29 -1) 📝 `management/internals/modules/reverseproxy/domain/manager/api.go` (+29 -1) _...and 56 more files_ </details> ### 📄 Description ## Summary (proof of concept for testing) Adds DNS-01 ACME issuance to the reverse-proxy alongside the existing `tls-alpn-01` / `http-01` flows, with per-service routing so each service decides which challenge type and DNS provider to use. DNS provider credentials are stored encrypted on the management server and fetched by the proxy at issuance time over a new gRPC method. Out of the box, four DNS providers are supported: **Cloudflare**, **AWS Route 53**, **DigitalOcean**, and **RFC 2136** (BIND, PowerDNS, Knot, NSD). Adding a new provider is a single registry entry on the proxy plus a backend adapter. Existing services keep working unchanged — they fall back to the proxy's globally-configured default challenge type, which remains `tls-alpn-01`. The end-to-end DNS-01 path was first verified on a throwaway spike branch (Cloudflare → Let's Encrypt staging on a real domain) before being lifted into this PR in four reviewable commits. Pairs with the dashboard PR on `brandon/dns-lego-ui` (TechHutTV/dashboard). ## Changes ### Proxy — ACME refactor - `proxy/internal/acme/backend.go` — new `CertBackend` interface (`GetCertificate`, `ReadCertFromDisk`, `DeleteCert`) so different challenge types can plug in. - `proxy/internal/acme/autocert_backend.go` — wraps the existing `autocert.Manager` and satisfies `CertBackend`; handles `tls-alpn-01` and `http-01` exactly as before. - `proxy/internal/acme/lego_backend.go` — new Lego-based backend for `dns-01`. Asymmetric semantics vs. autocert: `GetCertificate` is load-from-disk only; `Issue(ctx)` performs the actual DNS-01 dance and writes to disk. Manager prefetches certificates for `dns-01` services so TLS handshakes don't block on issuance. - `proxy/internal/acme/manager.go` — refactored from embedding `autocert.Manager` to holding a multi-backend map keyed by challenge type, with a configurable default. Locker, domain state, wildcard handling, notifier, and metrics stay at the orchestrator level. - `proxy/internal/acme/legoclient/` — new package: an account+client cache around `go-acme/lego/v4`, plus per-provider adapters and a registry. - `provider_cloudflare.go`, `provider_route53.go`, `provider_digitalocean.go`, `provider_rfc2136.go` - `providers.go` — central registry; new providers register themselves here. - `proxy/server.go` — wires the multi-backend manager. The challenge-type env var (`NB_PROXY_ACME_CHALLENGE_TYPE`) becomes the default; per-service config overrides it. ### Management — credentials store and resolver - `management/server/credentials.go` — encrypted credential record. Multi-field secrets (Route 53, RFC 2136) are JSON-encoded into the existing `EncryptedSecret` column with a legacy single-string fallback so older records keep working. - `management/server/http/handlers/credentials/credentials_handler.go` — new HTTP CRUD: `POST /api/credentials`, `GET /api/credentials`, `GET /api/credentials/{id}`, `PUT /api/credentials/{id}`, `DELETE /api/credentials/{id}`. Secrets are never returned in responses; PUT accepts a partial update so the secret can be rotated independently of the name. - `management/server/store/sql_store.go` — store methods for credentials (account-scoped). Includes a `GetByRefWithSecret` helper used by the resolver to decrypt at fetch time. - `management/internals/shared/grpc/proxy.go` — implements the new `ResolveCredential` RPC. Audit-logs every resolve. - `management/server/activity/codes.go` — new audit codes for credential create / update / delete / resolve. - `management/server/permissions/modules/module.go` — new permission module entry; access is currently gated alongside the existing services module pending a dedicated dashboard permission. - AutoMigrate handles the new `credentials` table on management startup. ### Per-service config - `shared/management/proto/proxy_service.proto` — `ProxyMapping` gains optional fields `challenge_type` (13), `dns_provider` (14), `dns_credentials_ref` (15). New `ResolveCredential` RPC + request/response messages. - `shared/management/http/api/openapi.yml` — the same three fields on the reverse-proxy service schema; full credentials API surface; companion Go types regenerated into `types.gen.go`. - `management/modules/reverseproxy/service/service.go` — per-service fields plumbed through GORM AutoMigrate. - `management/modules/reverseproxy/service/manager/manager.go` — new `validateACMEConfig` enforces internal consistency (e.g., `dns-01` requires `dns_provider`; non-dns-01 challenge types reject `dns_provider` / `dns_credentials_ref`). ### Tests - `proxy/internal/acme/manager_test.go` — multi-backend routing, prefetch, locker behavior, ReadCertFromDisk short-circuit. - `proxy/internal/acme/lego_backend_test.go` — backend semantics including the load-from-disk-only `GetCertificate` contract. - `proxy/internal/acme/locker_test.go` — same-domain serialization and different-domain parallelism. - `proxy/internal/acme/legoclient/providers_test.go` — registry + per-provider adapter shape (no live ACME calls). - `management/server/http/handlers/credentials/credentials_handler_test.go` — full CRUD coverage including the JSON-encoded secret round-trip and the legacy single-string fallback. - `management/modules/reverseproxy/service/manager/manager_test.go` — `validateACMEConfig` table tests covering allowed and rejected combinations. ## Out of scope (intentional) - **Custom Provider escape hatch** — Lego's `NewDNSChallengeProviderByName` is env-var-only and doesn't compose cleanly with per-service in-memory credentials. Deferred until there's demand for a provider not in the registry. - **Pebble-based integration tests** in CI — exercised manually via the spike; CI integration is its own hardening pass. - **memguard-style in-process secret hardening** — the encryption-at-rest model is in place; in-memory hardening is a separate review. - **Status surfacing** in the dashboard services list — covered by a follow-up dashboard slice. ## How to test 1. Apply this branch to a management server and proxy in the same environment; restart both. AutoMigrate creates the `credentials` table and adds the three new columns to `reverse_proxies`. 2. Add a Cloudflare credential via `POST /api/credentials` (or via the dashboard PR's new DNS Credentials page). 3. Create a reverse-proxy service with `challenge_type: "dns-01"`, `dns_provider: "cloudflare"`, `dns_credentials_ref: <id from step 2>`, and a domain in that Cloudflare zone. 4. Tail the proxy log — you should see a TXT-record write to Cloudflare, then a successful Let's Encrypt issuance, then `curl https://your-domain` returns the configured backend. 5. Repeat with Route 53 (multi-field credential), DigitalOcean (single token, exercises the `auth_token` adapter path), and RFC 2136 (TSIG path) against your own zones to exercise each adapter end-to-end. 6. Edit a credential's secret via `PUT /api/credentials/{id}` with `secret_fields` set — confirm the next renewal picks up the new secret without restarting either service. 7. Delete a credential currently referenced by a service — the management API should reject (409) so a live service can't be silently broken. ## Verification - `go build ./...` — clean. - `go test ./proxy/... ./management/... ./shared/...` — passes locally. One pre-existing Docker-dependent store test is skipped in environments without Docker; not introduced by this PR. - Spike branch verified the Cloudflare DNS-01 path end-to-end against Let's Encrypt staging on a real domain prior to landing. ## Issue ticket number and link ## Stack <!-- branch-stack --> ### Checklist - [ ] Is it a bug fix - [ ] Is a typo/documentation fix - [X] Is a feature enhancement - [X] It is a refactor - [ ] Created tests that fail without the change (if possible) > By submitting this pull request, you confirm that you have read and agree to the terms of the [Contributor License Agreement](https://github.com/netbirdio/netbird/blob/main/CONTRIBUTOR_LICENSE_AGREEMENT.md). ## Documentation Select exactly one: - [ ] I added/updated documentation for this change - [ ] Documentation is **not needed** for this change (explain why) ### Docs PR URL (required if "docs added" is checked) Paste the PR link from https://github.com/netbirdio/docs here: https://github.com/netbirdio/docs/pull/__ --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
saavagebueno added the pull-request label 2026-08-05 08:07:07 -04:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#28898