SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-05-16 05:01:57 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

upgrading an older selfhosted netbird installation with zitadel/cockroachdb to zitadel/postgresql .... success #2101

New Issue
Closed
opened 2025-11-20 07:03:48 -05:00 by saavagebueno · 0 comments
saavagebueno commented 2025-11-20 07:03:48 -05:00
Owner
Copy Link

Originally created by @ssachse on GitHub (Jul 22, 2025).

<head></head>

Hello everyone,

If you are migrating ZITADEL from CockroachDB to PostgreSQL and hit the same road‑blocks I did, this concise guide (plus full config files) might save you some hours.

ZITADEL – Migration from CockroachDB to PostgreSQL (Staging → Production)

TL;DR

  1. Reset & start the Postgres container (zdb) with a fresh volume.

  2. Init and setup --for‑mirror the empty target DB.

  3. Mirror from Cockroach to Postgres using --system --replace.

  4. Run setup --for‑mirror again to recreate projections & permissions.

  5. Launch ZITADEL in Postgres mode (start-from-init --tlsMode external).

  6. Wait until GET /debug/ready returns 200, then shut down Cockroach.

  7. Remove the CRDB service from docker-compose.yml when everything is green.

1  Files & Directories

1.1  Environment files

zitadel-pg.env and zdb.env hold credentials, TLS flags, and first‑instance machine‑user settings (see complete listing in §5).

1.2  Config files (./configs)

  • pg.yaml – connection details for init & setup.

  • mirror.yaml – defines source (Cockroach with TLS) and destination (Postgres without TLS).
    Increase EventBulkSize (e.g. 10 000) to speed up the mirror.

1.3  docker-compose.yml

Stack containing Caddy reverse‑proxy, NetBird components, CRDB, Postgres, and two ZITADEL containers (zitadel, zitadel-cli).
Use ghcr.io/zitadel/zitadel:v3.3.0 or newer for the CLI – older versions include a mirror bug.

2  Step‑by‑Step Migration

2.1  Reset the Postgres volume

# stop container & delete volume
docker compose down zdb
docker volume rm root_netbird_zdb_data
# start fresh instance
docker compose up -d zdb

2.2  Initialise the empty target DB

# create schema
docker compose run --rm zitadel-cli init --config /cfg/pg.yaml
# prepare for mirror
docker compose run --rm zitadel-cli setup --for-mirror \
  --config /cfg/pg.yaml --masterkeyFromEnv

2.3  Run the mirror job

# copies assets, keys, auth_requests, events & projections
docker compose run --rm zitadel-cli mirror --system --replace \
  --config /cfg/mirror.yaml --masterkeyFromEnv

Disk full? If Cockroach stops with insufficient remaining capacity, delete the EMERGENCY_BALLAST file or free space, then rerun the mirror.

2.4  Run setup again (projections & permissions)

docker compose run --rm zitadel-cli setup --for-mirror \
  --config /cfg/pg.yaml --masterkeyFromEnv

2.5  Start ZITADEL on Postgres

docker compose up -d zitadel

Wait until:

curl -f http://localhost:8080/debug/ready

returns HTTP 200.

2.6  Verify counts

docker compose run --rm zitadel-cli mirror verify --system \
  --config /cfg/mirror.yaml --masterkeyFromEnv

Small negative diffs for projections.*, auth.users2, and keys4* are expected.

2.7  Decommission CockroachDB

docker compose down crdb
curl -f http://localhost:8080/debug/ready  # should still be 200

Then remove the crdb service from your compose file.

3  Troubleshooting

Issue | Cause | Solution -- | -- | -- store X has insufficient remaining capacity | Cockroach node < 5 % free | Delete unused data or EMERGENCY_BALLAST, add storage, repeat mirror Mirror fails with older CLI | Bug fixed in ≥ v3.3 | Use ghcr.io/zitadel/zitadel:v3.3+ for zitadel-cli /debug/ready returns 503 | ZITADEL not fully started or DB unreachable | Check container logs, rerun setup --for-mirror

4  Useful Endpoints

  • Ready: GET /debug/ready

  • Health: GET /debug/healthz

  • Metrics: GET /debug/metrics

5  Complete Reference Files

5.1  zitadel-pg.env

ZITADEL_LOG_LEVEL=debug
ZITADEL_MASTERKEY=--EDITED--
MASTERKEY=--EDITED--
ZITADEL_SPOOLER_ENABLED=true
ZITADEL_DATABASE_POSTGRES_HOST=zdb
ZITADEL_DATABASE_POSTGRES_PORT=5432
ZITADEL_DATABASE_POSTGRES_DATABASE=zitadel
ZITADEL_DATABASE_POSTGRES_USER_USERNAME=zitadel
ZITADEL_DATABASE_POSTGRES_USER_PASSWORD=--EDITED--
ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE=disable
ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME=root
ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD=--EDITED--
ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE=disable
ZITADEL_EXTERNALSECURE=true
ZITADEL_TLS_ENABLED="false"
ZITADEL_EXTERNALPORT=443
ZITADEL_EXTERNALDOMAIN=--EDITED--
ZITADEL_FIRSTINSTANCE_PATPATH=/machinekey/zitadel-admin-sa.token
ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_USERNAME=zitadel-admin-sa
ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_NAME=Admin
ZITADEL_FIRSTINSTANCE_ORG_MACHINE_PAT_SCOPES=openid
ZITADEL_FIRSTINSTANCE_ORG_MACHINE_PAT_EXPIRATIONDATE=2025-08-11T19:08:35Z

5.2  zdb.env

POSTGRES_USER=root
POSTGRES_PASSWORD=--EDITED--

5.3  config/pg.yaml

Database:
  postgres:
    Host: zdb
    Port: 5432
    Database: zitadel
    User:
      Username: zitadel
      Password: --EDITED--
      SSL:
        Mode: disable
    Admin:
      Username: root
      Password: --EDITED--
      SSL:
        Mode: disable
FirstInstance:
  Org:
    Human:
      Username: admin@localhost
      Password: --EDITED--

5.4  config/mirror.yaml

Source:
  cockroach:
    Host: crdb
    Port: 26257
    Database: zitadel
    User:
      Username: zitadel_user
      SSL:
        Mode: verify-full
        RootCert: /crdb-certs/ca.crt
        Cert: /crdb-certs/client.zitadel_user.crt
        Key: /crdb-certs/client.zitadel_user.key
Destination:
  postgres:
    Host: zdb
    Port: 5432
    Database: zitadel
    User:
      Username: zitadel
      Password: --EDITED--
      SSL:
        Mode: disable
EventBulkSize: 10000
MaxAuthRequestAge: 720h
Projections:
  ConcurrentInstances: 7
  EventBulkLimit: 1000
Auth:
  Spooler:
    BulkLimit: 1000
Admin:
  Spooler:
    BulkLimit: 10
Log:
  Level: info

5.5  docker-compose.yml (relevant services)

services:
  caddy:
    image: caddy
    restart: unless-stopped
    networks: [ netbird ]
    ports:
      - "443:443"
      - "443:443/udp"
      - "80:80"
      - "8080:8080"
    volumes:
      - netbird_caddy_data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile

zitadel:

restart: always

networks: [ netbird ]

image: ghcr.io/zitadel/zitadel:v2.64.1

command: start-from-init --masterkeyFromEnv --tlsMode external

env_file:

- ./zitadel-pg.env

depends_on:

zdb:

condition: service_healthy

volumes:

- ./machinekey:/machinekey

- netbird_zitadel_certs:/crdb-certs:ro


zitadel-cli:

image: ghcr.io/zitadel/zitadel:v3.3.2

command: /bin/true

networks: [ netbird ]

env_file:

- ./zitadel-pg.env

volumes:

- netbird_zitadel_certs:/crdb-certs:ro

- ./machinekey:/machinekey:ro

- ./configs:/cfg


crdb:

image: cockroachdb/cockroach:v25.1.2

command: start-single-node --advertise-addr crdb --external-io-dir=/cockroach/nodelocal

restart: always

networks: [ netbird ]

volumes:

- netbird_cr


Originally created by @ssachse on GitHub (Jul 22, 2025). <head></head><p style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Hello everyone,</p><p style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">If you are migrating<span class="Apple-converted-space"> </span><strong>ZITADEL</strong><span class="Apple-converted-space"> </span>from<span class="Apple-converted-space"> </span><strong>CockroachDB</strong><span class="Apple-converted-space"> </span>to<span class="Apple-converted-space"> </span><strong>PostgreSQL</strong><span class="Apple-converted-space"> </span>and hit the same road‑blocks I did, this concise guide (plus full config files) might save you some hours.</p><hr style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><h2 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">ZITADEL – Migration from CockroachDB to PostgreSQL (Staging → Production)</h2><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">TL;DR</h3><ol style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><li><p><strong>Reset &amp; start the Postgres container</strong><span class="Apple-converted-space"> </span>(<code inline="">zdb</code>) with a fresh volume.</p></li><li><p><strong>Init</strong><span class="Apple-converted-space"> </span>and<span class="Apple-converted-space"> </span><strong>setup --for‑mirror</strong><span class="Apple-converted-space"> </span>the empty target DB.</p></li><li><p><strong>Mirror</strong><span class="Apple-converted-space"> </span>from Cockroach to Postgres using<span class="Apple-converted-space"> </span><code inline="">--system --replace</code>.</p></li><li><p><strong>Run setup --for‑mirror</strong><span class="Apple-converted-space"> </span>again to recreate projections &amp; permissions.</p></li><li><p>Launch ZITADEL in<span class="Apple-converted-space"> </span><strong>Postgres mode</strong><span class="Apple-converted-space"> </span>(<code inline="">start-from-init --tlsMode external</code>).</p></li><li><p>Wait until<span class="Apple-converted-space"> </span><code inline="">GET /debug/ready</code><span class="Apple-converted-space"> </span>returns 200, then shut down Cockroach.</p></li><li><p>Remove the CRDB service from<span class="Apple-converted-space"> </span><code inline="">docker-compose.yml</code><span class="Apple-converted-space"> </span>when everything is green.</p></li></ol><hr style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><h2 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">1  Files &amp; Directories</h2><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">1.1  Environment files</h3><p style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code inline="">zitadel-pg.env</code><span class="Apple-converted-space"> </span>and<span class="Apple-converted-space"> </span><code inline="">zdb.env</code><span class="Apple-converted-space"> </span>hold credentials, TLS flags, and first‑instance machine‑user settings (see complete listing in §5).</p><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">1.2  Config files (<code inline="">./configs</code>)</h3><ul style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><li><p><strong>pg.yaml</strong><span class="Apple-converted-space"> </span>– connection details for<span class="Apple-converted-space"> </span><code inline="">init</code><span class="Apple-converted-space"> </span>&amp;<span class="Apple-converted-space"> </span><code inline="">setup</code>.</p></li><li><p><strong>mirror.yaml</strong><span class="Apple-converted-space"> </span>– defines<span class="Apple-converted-space"> </span><em>source</em><span class="Apple-converted-space"> </span>(Cockroach with TLS) and<span class="Apple-converted-space"> </span><em>destination</em><span class="Apple-converted-space"> </span>(Postgres without TLS).<br>Increase<span class="Apple-converted-space"> </span><code inline="">EventBulkSize</code><span class="Apple-converted-space"> </span>(e.g. 10 000) to speed up the mirror.</p></li></ul><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">1.3  <code inline="">docker-compose.yml</code></h3><p style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Stack containing Caddy reverse‑proxy, NetBird components, CRDB, Postgres, and two ZITADEL containers (<code inline="">zitadel</code>,<span class="Apple-converted-space"> </span><code inline="">zitadel-cli</code>).<br>Use<span class="Apple-converted-space"> </span><strong><code inline="">ghcr.io/zitadel/zitadel:v3.3.0</code><span class="Apple-converted-space"> </span>or newer</strong><span class="Apple-converted-space"> </span>for the CLI – older versions include a mirror bug.</p><hr style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><h2 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">2  Step‑by‑Step Migration</h2><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">2.1  Reset the Postgres volume</h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-bash"># stop container &amp; delete volume docker compose down zdb docker volume rm root_netbird_zdb_data # start fresh instance docker compose up -d zdb </code></pre><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">2.2  Initialise the empty target DB</h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-bash"># create schema docker compose run --rm zitadel-cli init --config /cfg/pg.yaml # prepare for mirror docker compose run --rm zitadel-cli setup --for-mirror \ --config /cfg/pg.yaml --masterkeyFromEnv </code></pre><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">2.3  Run the mirror job</h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-bash"># copies assets, keys, auth_requests, events &amp; projections docker compose run --rm zitadel-cli mirror --system --replace \ --config /cfg/mirror.yaml --masterkeyFromEnv </code></pre><blockquote style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><p><strong>Disk full?</strong><span class="Apple-converted-space"> </span>If Cockroach stops with<span class="Apple-converted-space"> </span><code inline="">insufficient remaining capacity</code>, delete the<span class="Apple-converted-space"> </span><code inline="">EMERGENCY_BALLAST</code><span class="Apple-converted-space"> </span>file or free space, then rerun the mirror.</p></blockquote><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">2.4  Run setup again (projections &amp; permissions)</h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-bash">docker compose run --rm zitadel-cli setup --for-mirror \ --config /cfg/pg.yaml --masterkeyFromEnv </code></pre><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">2.5  Start ZITADEL on Postgres</h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-bash">docker compose up -d zitadel </code></pre><p style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Wait until:</p><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-bash">curl -f http://localhost:8080/debug/ready </code></pre><p style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">returns<span class="Apple-converted-space"> </span><strong>HTTP 200</strong>.</p><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">2.6  Verify counts</h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-bash">docker compose run --rm zitadel-cli mirror verify --system \ --config /cfg/mirror.yaml --masterkeyFromEnv </code></pre><p style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Small negative diffs for<span class="Apple-converted-space"> </span><code inline="">projections.*</code>,<span class="Apple-converted-space"> </span><code inline="">auth.users2</code>, and<span class="Apple-converted-space"> </span><code inline="">keys4*</code><span class="Apple-converted-space"> </span>are expected.</p><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">2.7  Decommission CockroachDB</h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-bash">docker compose down crdb curl -f http://localhost:8080/debug/ready # should still be 200 </code></pre><p style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">Then remove the<span class="Apple-converted-space"> </span><strong>crdb</strong><span class="Apple-converted-space"> </span>service from your compose file.</p><hr style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><h2 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">3  Troubleshooting</h2> Issue | Cause | Solution -- | -- | -- store X has insufficient remaining capacity | Cockroach node < 5 % free | Delete unused data or EMERGENCY_BALLAST, add storage, repeat mirror Mirror fails with older CLI | Bug fixed in ≥ v3.3 | Use ghcr.io/zitadel/zitadel:v3.3+ for zitadel-cli /debug/ready returns 503 | ZITADEL not fully started or DB unreachable | Check container logs, rerun setup --for-mirror <hr style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><h2 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">4  Useful Endpoints</h2><ul style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><li><p><strong>Ready:</strong><span class="Apple-converted-space"> </span><code inline="">GET /debug/ready</code></p></li><li><p><strong>Health:</strong><span class="Apple-converted-space"> </span><code inline="">GET /debug/healthz</code></p></li><li><p><strong>Metrics:</strong><span class="Apple-converted-space"> </span><code inline="">GET /debug/metrics</code></p></li></ul><hr style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><h2 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">5  Complete Reference Files</h2><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">5.1  <code inline="">zitadel-pg.env</code></h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-env">ZITADEL_LOG_LEVEL=debug ZITADEL_MASTERKEY=--EDITED-- MASTERKEY=--EDITED-- ZITADEL_SPOOLER_ENABLED=true ZITADEL_DATABASE_POSTGRES_HOST=zdb ZITADEL_DATABASE_POSTGRES_PORT=5432 ZITADEL_DATABASE_POSTGRES_DATABASE=zitadel ZITADEL_DATABASE_POSTGRES_USER_USERNAME=zitadel ZITADEL_DATABASE_POSTGRES_USER_PASSWORD=--EDITED-- ZITADEL_DATABASE_POSTGRES_USER_SSL_MODE=disable ZITADEL_DATABASE_POSTGRES_ADMIN_USERNAME=root ZITADEL_DATABASE_POSTGRES_ADMIN_PASSWORD=--EDITED-- ZITADEL_DATABASE_POSTGRES_ADMIN_SSL_MODE=disable ZITADEL_EXTERNALSECURE=true ZITADEL_TLS_ENABLED="false" ZITADEL_EXTERNALPORT=443 ZITADEL_EXTERNALDOMAIN=--EDITED-- ZITADEL_FIRSTINSTANCE_PATPATH=/machinekey/zitadel-admin-sa.token ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_USERNAME=zitadel-admin-sa ZITADEL_FIRSTINSTANCE_ORG_MACHINE_MACHINE_NAME=Admin ZITADEL_FIRSTINSTANCE_ORG_MACHINE_PAT_SCOPES=openid ZITADEL_FIRSTINSTANCE_ORG_MACHINE_PAT_EXPIRATIONDATE=2025-08-11T19:08:35Z </code></pre><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">5.2  <code inline="">zdb.env</code></h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-env">POSTGRES_USER=root POSTGRES_PASSWORD=--EDITED-- </code></pre><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">5.3  <code inline="">config/pg.yaml</code></h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-yaml">Database: postgres: Host: zdb Port: 5432 Database: zitadel User: Username: zitadel Password: --EDITED-- SSL: Mode: disable Admin: Username: root Password: --EDITED-- SSL: Mode: disable FirstInstance: Org: Human: Username: admin@localhost Password: --EDITED-- </code></pre><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">5.4  <code inline="">config/mirror.yaml</code></h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-yaml">Source: cockroach: Host: crdb Port: 26257 Database: zitadel User: Username: zitadel_user SSL: Mode: verify-full RootCert: /crdb-certs/ca.crt Cert: /crdb-certs/client.zitadel_user.crt Key: /crdb-certs/client.zitadel_user.key Destination: postgres: Host: zdb Port: 5432 Database: zitadel User: Username: zitadel Password: --EDITED-- SSL: Mode: disable EventBulkSize: 10000 MaxAuthRequestAge: 720h Projections: ConcurrentInstances: 7 EventBulkLimit: 1000 Auth: Spooler: BulkLimit: 1000 Admin: Spooler: BulkLimit: 10 Log: Level: info </code></pre><h3 style="font-style: normal; font-variant-caps: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);">5.5  <code inline="">docker-compose.yml</code><span class="Apple-converted-space"> </span>(relevant services)</h3><pre style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><code class="language-yaml">services: caddy: image: caddy restart: unless-stopped networks: [ netbird ] ports: - "443:443" - "443:443/udp" - "80:80" - "8080:8080" volumes: - netbird_caddy_data:/data - ./Caddyfile:/etc/caddy/Caddyfile zitadel: restart: always networks: [ netbird ] image: ghcr.io/zitadel/zitadel:v2.64.1 command: start-from-init --masterkeyFromEnv --tlsMode external env_file: - ./zitadel-pg.env depends_on: zdb: condition: service_healthy volumes: - ./machinekey:/machinekey - netbird_zitadel_certs:/crdb-certs:ro zitadel-cli: image: ghcr.io/zitadel/zitadel:v3.3.2 command: /bin/true networks: [ netbird ] env_file: - ./zitadel-pg.env volumes: - netbird_zitadel_certs:/crdb-certs:ro - ./machinekey:/machinekey:ro - ./configs:/cfg crdb: image: cockroachdb/cockroach:v25.1.2 command: start-single-node --advertise-addr crdb --external-io-dir=/cockroach/nodelocal restart: always networks: [ netbird ] volumes: - netbird_cr </code></pre><br class="Apple-interchange-newline" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><br class="Apple-interchange-newline">
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#2101
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?