[GH-ISSUE #2654] Docker for Management and Caddy is in restart loop #5737

Closed
opened 2026-08-05 01:04:52 -04:00 by saavagebueno · 8 comments
Owner

Originally created by @EKV89 on GitHub (Sep 26, 2024).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/2654

Hi!

My caddy and management container are stuck in a reboot loop, please see the following docker logs:

docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
12945bbfbfc7 ghcr.io/zitadel/zitadel:v2.54.3 "/app/zitadel start-…" 20 minutes ago Up 8 minutes netbird-docker_zitadel_1
f3c70c38bf70 netbirdio/signal:latest "/go/bin/netbird-sig…" 20 minutes ago Up 8 minutes netbird-docker_signal_1
42489d51fb73 postgres:16-alpine "docker-entrypoint.s…" 20 minutes ago Up 8 minutes (healthy) 5432/tcp netbird-docker_zdb_1
1d0758c2999e netbirdio/management:latest "/go/bin/netbird-mgm…" 20 minutes ago Restarting (1) 53 seconds ago netbird-docker_management_1
68ab3492acb7 coturn/coturn "docker-entrypoint.s…" 20 minutes ago Up 8 minutes netbird-docker_coturn_1
bd787e56fdff netbirdio/dashboard:latest "/usr/bin/supervisor…" 20 minutes ago Up 8 minutes 80/tcp, 443/tcp netbird-docker_dashboard_1
69e2d7310909 caddy "caddy run --config …" 20 minutes ago Restarting (1) 55 seconds ago netbird-docker_caddy_1
e694ce7a9927 netbirdio/relay:latest "/go/bin/netbird-rel…" 20 minutes ago Up 8 minutes 0.0.0.0:33080->33080/tcp, :::33080->33080/tcp netbird-docker_relay_1

### Docker logs from Management container:

2024-09-26T07:55:16Z INFO [context: SYSTEM] management/cmd/management.go:497: loading OIDC configuration from the provided IDP configuration endpoint https://abc.xyz/.well-known/openid-configuration
Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://abc.xyz/.well-known/openid-configuration Get "https://abc.xyz/.well-known/openid-configuration": dial tcp 1.2.3.4:443: connect: connection refused

### Docker logs from Caddy container:

streams":[{"dial":"dashboard:80"}]}]},{}],"automatic_https":{"disable":true}}}}}
{"level":"info","ts":1727337375.2716815,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"debug","ts":1727337375.2719836,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1727337375.2720249,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1727337375.2721002,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1727337375.2721803,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1727337375.2722132,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["abc.xyz"]}
{"level":"info","ts":1727337375.2725453,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000430e80"}
{"level":"error","ts":1727337375.2725632,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
{"level":"info","ts":1727337375.2726264,"logger":"tls","msg":"finished cleaning storage units"}
Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [abc.xyz]: automate: manage [abc.xyz]: abc.xyz: caching certificate: decoding certificate metadata: unexpected end of JSON input
{"level":"info","ts":1727337435.8905706,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1727337435.8943582,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1727337435.8943884,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1727337435.8991036,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1727337435.8994842,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1727337435.899514,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":1727337435.899587,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00026fe00"}
{"level":"warn","ts":1727337435.8996568,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"debug","ts":1727337435.8996904,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"deferred":true,"delete":["Server"],"set":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=3600; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"]}}}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.management.v1.ManagementService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.system.v1.SystemService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/.well-known/openid-configuration"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"signal:10000"}]}],"match":[{"path":["/signalexchange.SignalExchange/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.admin.v1.AdminService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"management:80"}]}],"match":[{"path":["/management.ManagementService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.auth.v1.AuthService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/management/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/system/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/assets/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/admin/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/oauth/v2/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/auth/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/oidc/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/saml/v2/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/openapi/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/device/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/debug/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/device"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"management:80"}]}],"match":[{"path":["/api/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/ui/"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"dashboard:80"}]}],"match":[{"path":["/"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":80"],"routes":[{},{"handle":[{"handler":"headers","response":{"deferred":true,"delete":["Server"],"set":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=3600; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"]}}}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"signal:10000"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"management:80"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"management:80"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"dashboard:80"}]}]},{}],"automatic_https":{"disable":true}}}}}
{"level":"info","ts":1727337435.9027612,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"debug","ts":1727337435.9031258,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1727337435.903154,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1727337435.9032362,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1727337435.9032514,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1727337435.9032586,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["abc.xyz"]}
{"level":"info","ts":1727337435.9037006,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00026fe00"}
{"level":"error","ts":1727337435.9036887,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
{"level":"info","ts":1727337435.9037304,"logger":"tls","msg":"finished cleaning storage units"}
Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [abc.xyz]: automate: manage [abc.xyz]: abc.xyz: caching certificate: decoding certificate metadata: unexpected end of JSON input
{"level":"info","ts":1727337496.635272,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
{"level":"info","ts":1727337496.6408808,"msg":"adapted config to JSON","adapter":"caddyfile"}
{"level":"warn","ts":1727337496.6409295,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
{"level":"info","ts":1727337496.646059,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1727337496.6467438,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":1727337496.6467912,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":1727337496.6470478,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80}
{"level":"info","ts":1727337496.6477964,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00051f980"}
{"level":"debug","ts":1727337496.647155,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"deferred":true,"delete":["Server"],"set":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=3600; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"]}}}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.management.v1.ManagementService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.system.v1.SystemService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/.well-known/openid-configuration"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"signal:10000"}]}],"match":[{"path":["/signalexchange.SignalExchange/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.admin.v1.AdminService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"management:80"}]}],"match":[{"path":["/management.ManagementService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.auth.v1.AuthService/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/management/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/system/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/assets/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/admin/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/oauth/v2/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/auth/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/oidc/v1/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/saml/v2/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/openapi/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/device/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/debug/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/device"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"management:80"}]}],"match":[{"path":["/api/"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/ui/"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"dashboard:80"}]}],"match":[{"path":["/"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":80"],"routes":[{},{"handle":[{"handler":"headers","response":{"deferred":true,"delete":["Server"],"set":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=3600; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"]}}}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"signal:10000"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"management:80"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"management:80"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"dashboard:80"}]}]},{}],"automatic_https":{"disable":true}}}}}
{"level":"info","ts":1727337496.6526024,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
{"level":"debug","ts":1727337496.6529436,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
{"level":"info","ts":1727337496.6529741,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"debug","ts":1727337496.6531074,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
{"level":"info","ts":1727337496.6531997,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"info","ts":1727337496.6532428,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["abc.xyz"]}
{"level":"error","ts":1727337496.6536767,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
{"level":"info","ts":1727337496.6537178,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1727337496.6537852,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00051f980"}
Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [abc.xyz]: automate: manage [abc.xyz]: abc.xyz: caching certificate: decoding certificate metadata: unexpected end of JSON input

Originally created by @EKV89 on GitHub (Sep 26, 2024). Original GitHub issue: https://github.com/netbirdio/netbird/issues/2654 Hi! My caddy and management container are stuck in a reboot loop, please see the following docker logs: docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 12945bbfbfc7 ghcr.io/zitadel/zitadel:v2.54.3 "/app/zitadel start-…" 20 minutes ago Up 8 minutes netbird-docker_zitadel_1 f3c70c38bf70 netbirdio/signal:latest "/go/bin/netbird-sig…" 20 minutes ago Up 8 minutes netbird-docker_signal_1 42489d51fb73 postgres:16-alpine "docker-entrypoint.s…" 20 minutes ago Up 8 minutes (healthy) 5432/tcp netbird-docker_zdb_1 **1d0758c2999e netbirdio/management:latest "/go/bin/netbird-mgm…" 20 minutes ago Restarting (1) 53 seconds ago netbird-docker_management_1** 68ab3492acb7 coturn/coturn "docker-entrypoint.s…" 20 minutes ago Up 8 minutes netbird-docker_coturn_1 bd787e56fdff netbirdio/dashboard:latest "/usr/bin/supervisor…" 20 minutes ago Up 8 minutes 80/tcp, 443/tcp netbird-docker_dashboard_1 **69e2d7310909 caddy "caddy run --config …" 20 minutes ago Restarting (1) 55 seconds ago netbird-docker_caddy_1** e694ce7a9927 netbirdio/relay:latest "/go/bin/netbird-rel…" 20 minutes ago Up 8 minutes 0.0.0.0:33080->33080/tcp, :::33080->33080/tcp netbird-docker_relay_1 **### Docker logs from Management container:** 2024-09-26T07:55:16Z INFO [context: SYSTEM] management/cmd/management.go:497: loading OIDC configuration from the provided IDP configuration endpoint https://abc.xyz/.well-known/openid-configuration Error: failed reading provided config file: /etc/netbird/management.json: failed fetching OIDC configuration from endpoint https://abc.xyz/.well-known/openid-configuration Get "https://abc.xyz/.well-known/openid-configuration": dial tcp 1.2.3.4:443: connect: connection refused **### Docker logs from Caddy container:** streams":[{"dial":"dashboard:80"}]}]},{}],"automatic_https":{"disable":true}}}}} {"level":"info","ts":1727337375.2716815,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"} {"level":"debug","ts":1727337375.2719836,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true} {"level":"info","ts":1727337375.2720249,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]} {"level":"debug","ts":1727337375.2721002,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false} {"level":"info","ts":1727337375.2721803,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]} {"level":"info","ts":1727337375.2722132,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["abc.xyz"]} {"level":"info","ts":1727337375.2725453,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000430e80"} {"level":"error","ts":1727337375.2725632,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"} {"level":"info","ts":1727337375.2726264,"logger":"tls","msg":"finished cleaning storage units"} Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [abc.xyz]: automate: manage [abc.xyz]: abc.xyz: caching certificate: decoding certificate metadata: unexpected end of JSON input {"level":"info","ts":1727337435.8905706,"msg":"using config from file","file":"/etc/caddy/Caddyfile"} {"level":"info","ts":1727337435.8943582,"msg":"adapted config to JSON","adapter":"caddyfile"} {"level":"warn","ts":1727337435.8943884,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2} {"level":"info","ts":1727337435.8991036,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]} {"level":"info","ts":1727337435.8994842,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443} {"level":"info","ts":1727337435.899514,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"} {"level":"info","ts":1727337435.899587,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00026fe00"} {"level":"warn","ts":1727337435.8996568,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80} {"level":"debug","ts":1727337435.8996904,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"deferred":true,"delete":["Server"],"set":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=3600; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"]}}}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.management.v1.ManagementService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.system.v1.SystemService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/.well-known/openid-configuration"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"signal:10000"}]}],"match":[{"path":["/signalexchange.SignalExchange/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.admin.v1.AdminService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"management:80"}]}],"match":[{"path":["/management.ManagementService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.auth.v1.AuthService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/management/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/system/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/assets/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/admin/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/oauth/v2/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/auth/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/oidc/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/saml/v2/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/openapi/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/device/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/debug/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/device"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"management:80"}]}],"match":[{"path":["/api/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/ui/*"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"dashboard:80"}]}],"match":[{"path":["/*"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":80"],"routes":[{},{"handle":[{"handler":"headers","response":{"deferred":true,"delete":["Server"],"set":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=3600; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"]}}}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"signal:10000"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"management:80"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"management:80"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"dashboard:80"}]}]},{}],"automatic_https":{"disable":true}}}}} {"level":"info","ts":1727337435.9027612,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"} {"level":"debug","ts":1727337435.9031258,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true} {"level":"info","ts":1727337435.903154,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]} {"level":"debug","ts":1727337435.9032362,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false} {"level":"info","ts":1727337435.9032514,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]} {"level":"info","ts":1727337435.9032586,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["abc.xyz"]} {"level":"info","ts":1727337435.9037006,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00026fe00"} {"level":"error","ts":1727337435.9036887,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"} {"level":"info","ts":1727337435.9037304,"logger":"tls","msg":"finished cleaning storage units"} Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [abc.xyz]: automate: manage [abc.xyz]: abc.xyz: caching certificate: decoding certificate metadata: unexpected end of JSON input {"level":"info","ts":1727337496.635272,"msg":"using config from file","file":"/etc/caddy/Caddyfile"} {"level":"info","ts":1727337496.6408808,"msg":"adapted config to JSON","adapter":"caddyfile"} {"level":"warn","ts":1727337496.6409295,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2} {"level":"info","ts":1727337496.646059,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]} {"level":"info","ts":1727337496.6467438,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443} {"level":"info","ts":1727337496.6467912,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"} {"level":"warn","ts":1727337496.6470478,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80} {"level":"info","ts":1727337496.6477964,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00051f980"} {"level":"debug","ts":1727337496.647155,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"headers","response":{"deferred":true,"delete":["Server"],"set":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=3600; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"]}}}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.management.v1.ManagementService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.system.v1.SystemService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/.well-known/openid-configuration"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"signal:10000"}]}],"match":[{"path":["/signalexchange.SignalExchange/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.admin.v1.AdminService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"management:80"}]}],"match":[{"path":["/management.ManagementService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/zitadel.auth.v1.AuthService/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/management/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/system/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/assets/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/admin/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/oauth/v2/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/auth/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/oidc/v1/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/saml/v2/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/openapi/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/device/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/debug/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/device"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"management:80"}]}],"match":[{"path":["/api/*"]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}],"match":[{"path":["/ui/*"]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"dashboard:80"}]}],"match":[{"path":["/*"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":80"],"routes":[{},{"handle":[{"handler":"headers","response":{"deferred":true,"delete":["Server"],"set":{"Referrer-Policy":["strict-origin-when-cross-origin"],"Strict-Transport-Security":["max-age=3600; includeSubDomains; preload"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Xss-Protection":["1; mode=block"]}}}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"signal:10000"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"management:80"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"management:80"}]}]},{"handle":[{"handler":"reverse_proxy","transport":{"protocol":"http","versions":["h2c","2"]},"upstreams":[{"dial":"zitadel:8080"}]}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"dashboard:80"}]}]},{}],"automatic_https":{"disable":true}}}}} {"level":"info","ts":1727337496.6526024,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"} {"level":"debug","ts":1727337496.6529436,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true} {"level":"info","ts":1727337496.6529741,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]} {"level":"debug","ts":1727337496.6531074,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false} {"level":"info","ts":1727337496.6531997,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]} {"level":"info","ts":1727337496.6532428,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["abc.xyz"]} {"level":"error","ts":1727337496.6536767,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"} {"level":"info","ts":1727337496.6537178,"logger":"tls","msg":"finished cleaning storage units"} {"level":"info","ts":1727337496.6537852,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00051f980"} Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [abc.xyz]: automate: manage [abc.xyz]: abc.xyz: caching certificate: decoding certificate metadata: unexpected end of JSON input
saavagebueno added the waiting-feedbacktriage-neededdockerself-hosting labels 2026-08-05 01:04:52 -04:00
Author
Owner

@mlsmaycon commented on GitHub (Sep 26, 2024):

@EKV89 have you checked why this error is happening?

Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [[abc.xyz](http://abc.xyz/)]: automate: manage [[abc.xyz](http://abc.xyz/)]: [abc.xyz](http://abc.xyz/): caching certificate: decoding certificate metadata: unexpected end of JSON input

Maybe something got corrupted in the Caddyfile?

<!-- gh-comment-id:2376232137 --> @mlsmaycon commented on GitHub (Sep 26, 2024): @EKV89 have you checked why this error is happening? ``` Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [[abc.xyz](http://abc.xyz/)]: automate: manage [[abc.xyz](http://abc.xyz/)]: [abc.xyz](http://abc.xyz/): caching certificate: decoding certificate metadata: unexpected end of JSON input ``` Maybe something got corrupted in the Caddyfile?
Author
Owner

@EKV89 commented on GitHub (Sep 26, 2024):

Hi @mlsmaycon !

This is the output of my Caddyfile, how does it look to you:

{
debug
servers :80,:443 {
protocols h1 h2c
}
}

(security_headers) {
header * {
# enable HSTS
# https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#strict-transport-security-hsts
# NOTE: Read carefully how this header works before using it.
# If the HSTS header is misconfigured or if there is a problem with
# the SSL/TLS certificate being used, legitimate users might be unable
# to access the website. For example, if the HSTS header is set to a
# very long duration and the SSL/TLS certificate expires or is revoked,
# legitimate users might be unable to access the website until
# the HSTS header duration has expired.
# The recommended value for the max-age is 2 year (63072000 seconds).
# But we are using 1 hour (3600 seconds) for testing purposes
# and ensure that the website is working properly before setting
# to two years.

    Strict-Transport-Security "max-age=3600; includeSubDomains; preload"

    # disable clients from sniffing the media type
    # https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#x-content-type-options
    X-Content-Type-Options "nosniff"

    # clickjacking protection
    # https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#x-frame-options
    X-Frame-Options "DENY"

    # xss protection
    # https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#x-xss-protection
    X-XSS-Protection "1; mode=block"

    # Remove -Server header, which is an information leak
    # Remove Caddy from Headers
    -Server

    # keep referrer data off of HTTP connections
    # https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#referrer-policy
    Referrer-Policy strict-origin-when-cross-origin
}

}

:80, abc.xyz:443 {
import security_headers
# Signal
reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000
# Management
reverse_proxy /api/* management:80
reverse_proxy /management.ManagementService/* h2c://management:80
# Zitadel
reverse_proxy /zitadel.admin.v1.AdminService/* h2c://zitadel:8080
reverse_proxy /admin/v1/* h2c://zitadel:8080
reverse_proxy /zitadel.auth.v1.AuthService/* h2c://zitadel:8080
reverse_proxy /auth/v1/* h2c://zitadel:8080
reverse_proxy /zitadel.management.v1.ManagementService/* h2c://zitadel:8080
reverse_proxy /management/v1/* h2c://zitadel:8080
reverse_proxy /zitadel.system.v1.SystemService/* h2c://zitadel:8080
reverse_proxy /system/v1/* h2c://zitadel:8080
reverse_proxy /assets/v1/* h2c://zitadel:8080
reverse_proxy /ui/* h2c://zitadel:8080
reverse_proxy /oidc/v1/* h2c://zitadel:8080
reverse_proxy /saml/v2/* h2c://zitadel:8080
reverse_proxy /oauth/v2/* h2c://zitadel:8080
reverse_proxy /.well-known/openid-configuration h2c://zitadel:8080
reverse_proxy /openapi/* h2c://zitadel:8080
reverse_proxy /debug/* h2c://zitadel:8080
reverse_proxy /device/* h2c://zitadel:8080
reverse_proxy /device h2c://zitadel:8080
# Dashboard
reverse_proxy /* dashboard:80
}

<!-- gh-comment-id:2376265790 --> @EKV89 commented on GitHub (Sep 26, 2024): Hi @mlsmaycon ! This is the output of my Caddyfile, how does it look to you: { debug servers :80,:443 { protocols h1 h2c } } (security_headers) { header * { # enable HSTS # https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#strict-transport-security-hsts # NOTE: Read carefully how this header works before using it. # If the HSTS header is misconfigured or if there is a problem with # the SSL/TLS certificate being used, legitimate users might be unable # to access the website. For example, if the HSTS header is set to a # very long duration and the SSL/TLS certificate expires or is revoked, # legitimate users might be unable to access the website until # the HSTS header duration has expired. # The recommended value for the max-age is 2 year (63072000 seconds). # But we are using 1 hour (3600 seconds) for testing purposes # and ensure that the website is working properly before setting # to two years. Strict-Transport-Security "max-age=3600; includeSubDomains; preload" # disable clients from sniffing the media type # https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#x-content-type-options X-Content-Type-Options "nosniff" # clickjacking protection # https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#x-frame-options X-Frame-Options "DENY" # xss protection # https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#x-xss-protection X-XSS-Protection "1; mode=block" # Remove -Server header, which is an information leak # Remove Caddy from Headers -Server # keep referrer data off of HTTP connections # https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html#referrer-policy Referrer-Policy strict-origin-when-cross-origin } } :80, abc.xyz:443 { import security_headers # Signal reverse_proxy /signalexchange.SignalExchange/* h2c://signal:10000 # Management reverse_proxy /api/* management:80 reverse_proxy /management.ManagementService/* h2c://management:80 # Zitadel reverse_proxy /zitadel.admin.v1.AdminService/* h2c://zitadel:8080 reverse_proxy /admin/v1/* h2c://zitadel:8080 reverse_proxy /zitadel.auth.v1.AuthService/* h2c://zitadel:8080 reverse_proxy /auth/v1/* h2c://zitadel:8080 reverse_proxy /zitadel.management.v1.ManagementService/* h2c://zitadel:8080 reverse_proxy /management/v1/* h2c://zitadel:8080 reverse_proxy /zitadel.system.v1.SystemService/* h2c://zitadel:8080 reverse_proxy /system/v1/* h2c://zitadel:8080 reverse_proxy /assets/v1/* h2c://zitadel:8080 reverse_proxy /ui/* h2c://zitadel:8080 reverse_proxy /oidc/v1/* h2c://zitadel:8080 reverse_proxy /saml/v2/* h2c://zitadel:8080 reverse_proxy /oauth/v2/* h2c://zitadel:8080 reverse_proxy /.well-known/openid-configuration h2c://zitadel:8080 reverse_proxy /openapi/* h2c://zitadel:8080 reverse_proxy /debug/* h2c://zitadel:8080 reverse_proxy /device/* h2c://zitadel:8080 reverse_proxy /device h2c://zitadel:8080 # Dashboard reverse_proxy /* dashboard:80 }
Author
Owner

@mlsmaycon commented on GitHub (Sep 26, 2024):

It worked fine for me. Can you test the following docker-compose locally?

caddy-test.zip

It won't listen to any call, but is just to test against your container version

<!-- gh-comment-id:2376319573 --> @mlsmaycon commented on GitHub (Sep 26, 2024): It worked fine for me. Can you test the following docker-compose locally? [caddy-test.zip](https://github.com/user-attachments/files/17145485/caddy-test.zip) It won't listen to any call, but is just to test against your container version
Author
Owner

@EKV89 commented on GitHub (Sep 27, 2024):

I made some progress, i got it up and running by doing this in docker-compose.yml

volumes:
  - netbird_caddy_data:/**data**

And to:

volumes:
  - netbird_caddy_data:/**data1**

It seams like the old data-mount point somehow got corrupted and didn't want to work.

<!-- gh-comment-id:2378604444 --> @EKV89 commented on GitHub (Sep 27, 2024): I made some progress, i got it up and running by doing this in docker-compose.yml volumes: - netbird_caddy_data:/**data** And to: volumes: - netbird_caddy_data:/**data1** It seams like the old data-mount point somehow got corrupted and didn't want to work.
Author
Owner

@ndziuba commented on GitHub (Oct 1, 2024):

Had exacly the same problem when the disk of the vm because of logs and a initial small disk size was out of space.
A simple docker volume rm of the caddy volume fixes this issue and redownloads the certs.

<!-- gh-comment-id:2385600026 --> @ndziuba commented on GitHub (Oct 1, 2024): Had exacly the same problem when the disk of the vm because of logs and a initial small disk size was out of space. A simple `docker volume rm` of the caddy volume fixes this issue and redownloads the certs.
Author
Owner

@hrfried commented on GitHub (Oct 16, 2024):

Had exacly the same problem when the disk of the vm because of logs and a initial small disk size was out of space. A simple docker volume rm of the caddy volume fixes this issue and redownloads the certs.

Can confirm this fixed it for me too. Docker logs ballooned to 30 gigs on a cheap VM and I didn't notice because I was moving and not using my personal netbird instance for a couple weeks. Truncated the json log (i.e. `truncate -s 0 </path/to/docker/volume>/-json.log) to size 0 because the disk was completely full (even though log size was set to top out at 500 megs in the compose—strange) and hit this.

Bringing down containers and removing the relevant caddy docker volume, then bringing containers back up fixed it right away.

<!-- gh-comment-id:2418307425 --> @hrfried commented on GitHub (Oct 16, 2024): > Had exacly the same problem when the disk of the vm because of logs and a initial small disk size was out of space. A simple `docker volume rm` of the caddy volume fixes this issue and redownloads the certs. Can confirm this fixed it for me too. Docker logs ballooned to 30 gigs on a cheap VM and I didn't notice because I was moving and not using my personal netbird instance for a couple weeks. Truncated the json log (i.e. `truncate -s 0 </path/to/docker/volume>/<LONG-GUID>-json.log) to size 0 because the disk was completely full (even though log size was set to top out at 500 megs in the compose—strange) and hit this. Bringing down containers and removing the relevant caddy docker volume, then bringing containers back up fixed it right away.
Author
Owner

@nazarewk commented on GitHub (Apr 28, 2025):

Hello @EKV89,

We're currently reviewing our open issues and would like to verify if this problem still exists in the latest NetBird version.

Could you please confirm if the issue is still there?

We may close this issue temporarily if we don't hear back from you within 2 weeks, but feel free to reopen it with updated information.

Thanks for your contribution to improving the project!

<!-- gh-comment-id:2835664224 --> @nazarewk commented on GitHub (Apr 28, 2025): Hello @EKV89, We're currently reviewing our open issues and would like to verify if this problem still exists in the [latest NetBird version](https://github.com/netbirdio/netbird/releases). Could you please confirm if the issue is still there? We may close this issue temporarily if we don't hear back from you within **2 weeks**, but feel free to reopen it with updated information. Thanks for your contribution to improving the project!
Author
Owner

@mlsmaycon commented on GitHub (Jun 1, 2025):

closing issue due to no recent feedback. Feel free to open a new one if the issue persist or reopen if this was a feature request.

<!-- gh-comment-id:2927891574 --> @mlsmaycon commented on GitHub (Jun 1, 2025): closing issue due to no recent feedback. Feel free to open a new one if the issue persist or reopen if this was a feature request.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#5737