Waiting for zitadel to be ready loop, with quickstart script #1391

@mlsmaycon commented on GitHub (Nov 5, 2024):

The error message suggests checking the firewall, can you confirm that you've followed the requirements section of the guide and checked the Caddy container logs?

Unable to connect to Zitadel for more than 45s, please check the output above, your firewall 
rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates

@mlsmaycon commented on GitHub (Nov 5, 2024): The error message suggests checking the firewall, can you confirm that you've followed the requirements section of the [guide](https://docs.netbird.io/selfhosted/selfhosted-quickstart#requirements) and checked the Caddy container logs? ```shell Unable to connect to Zitadel for more than 45s, please check the output above, your firewall rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates ```

saavagebueno commented

@luc-caspar commented on GitHub (Nov 5, 2024):

@mlsmaycon Thank you for your answer.
On the machine itself, no firewall is running. Instead, I am relying on AWS security group to punch holes where necessary. This is the list of currently open ports:

As for the caddy logs, they are the same as the ones I provided in the original post.

@luc-caspar commented on GitHub (Nov 5, 2024): @mlsmaycon Thank you for your answer. On the machine itself, no firewall is running. Instead, I am relying on AWS security group to punch holes where necessary. This is the list of currently open ports: ![image](https://github.com/user-attachments/assets/0186fbeb-31f2-4578-b9a1-bdb4c2145fc2) As for the caddy logs, they are the same as the ones I provided in the original post.

saavagebueno commented

@yblis commented on GitHub (Nov 9, 2024):

Hello, I have the same problem on a new VPN, even with the firewall disabled.

@yblis commented on GitHub (Nov 9, 2024): Hello, I have the same problem on a new VPN, even with the firewall disabled.

saavagebueno commented

@mikebakke commented on GitHub (Nov 19, 2024):

This seems pretty much the issue I have. Granted I'm running behind haproxy but all that is doing is sending the raw 443 traffic to the vm with the quickstart script. All other required ports are forwarded direct from my router to the instance. I see caddy generate a certificate for my domain but then there's a loop waiting for Zitadel and messages which suggest there's no certificate for my haproxy nodes' IP address.

@mikebakke commented on GitHub (Nov 19, 2024): This seems pretty much the issue I have. Granted I'm running behind haproxy but all that is doing is sending the raw 443 traffic to the vm with the quickstart script. All other required ports are forwarded direct from my router to the instance. I see caddy generate a certificate for my domain but then there's a loop waiting for Zitadel and messages which suggest there's no certificate for my haproxy nodes' IP address.

saavagebueno commented

@TasdidurRahman commented on GitHub (Dec 12, 2024):

facing the same issue! @luc-caspar could you solve it?

@TasdidurRahman commented on GitHub (Dec 12, 2024): facing the same issue! @luc-caspar could you solve it?

saavagebueno commented

@luc-caspar commented on GitHub (Dec 12, 2024):

@TasdidurRahman Unfortunately, I have not had the time to investigate further. My current solution was simply to keep the netbird v0.25 instance that is running perfectly and not upgrade it.

@luc-caspar commented on GitHub (Dec 12, 2024): @TasdidurRahman Unfortunately, I have not had the time to investigate further. My current solution was simply to keep the netbird v0.25 instance that is running perfectly and not upgrade it.

saavagebueno commented

@NiklasDor commented on GitHub (Dec 15, 2024):

same error with 0.34.1

@NiklasDor commented on GitHub (Dec 15, 2024): same error with 0.34.1

saavagebueno commented

@NiklasDor commented on GitHub (Dec 21, 2024):

Problem only occurs on Ubuntu, just did a install on debian without any problems

@NiklasDor commented on GitHub (Dec 21, 2024): Problem only occurs on Ubuntu, just did a install on debian without any problems

saavagebueno commented

@davidchi2020 commented on GitHub (Dec 24, 2024):

same error on centos9 with 0.35

@davidchi2020 commented on GitHub (Dec 24, 2024): same error on centos9 with 0.35

saavagebueno commented

@bitinerant commented on GitHub (Jan 1, 2025):

In my particular case, the relevant error was dial tcp: lookup dashboard on 127.0.0.11:53: server misbehaving buried in the output of docker logs ubuntu_caddy_1, indicating that Caddy can't resolve local DNS names. (I'm not sure I would have ever found this without ChatGPT.) You can test if this is a problem on your system via docker run --rm busybox ping dashboard.

@bitinerant commented on GitHub (Jan 1, 2025): In my particular case, the relevant error was `dial tcp: lookup dashboard on 127.0.0.11:53: server misbehaving` buried in the output of `docker logs ubuntu_caddy_1`, indicating that Caddy can't resolve local DNS names. (I'm not sure I would have ever found this without ChatGPT.) You can test if this is a problem on your system via `docker run --rm busybox ping dashboard`.

saavagebueno commented

@gmckeown commented on GitHub (Feb 6, 2025):

@bitinerant What did you do to resolve this issue?

@gmckeown commented on GitHub (Feb 6, 2025): @bitinerant What did you do to resolve this issue?

saavagebueno commented

@georgetarlas commented on GitHub (Feb 6, 2025):

In my case was simple. From an initial misconfiguration, puclic IP was not accessible by lets encrypt that runs during installation, so I become banned by lest encrypt.

{"level":"error","ts":1738874957.3955424,"logger":"tls.obtain","msg":"will retry","error":"[vpn.XXX.gr] Obtain: [vpn.XXX.gr] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2025-02-07 03:32:26 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":5,"retrying_in":600,"elapsed":627.088738971,"max_duration":2592000}

@georgetarlas commented on GitHub (Feb 6, 2025): In my case was simple. From an initial misconfiguration, puclic IP was not accessible by lets encrypt that runs during installation, so I become banned by lest encrypt. {"level":"error","ts":1738874957.3955424,"logger":"tls.obtain","msg":"will retry","error":"[vpn.XXX.gr] Obtain: [vpn.XXX.gr] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2025-02-07 03:32:26 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":5,"retrying_in":600,"elapsed":627.088738971,"max_duration":2592000}

saavagebueno commented

@ashayh commented on GitHub (Feb 7, 2025):

I am self hosting Netbird on Fedora in a home network for accessing Home assistant. I faced the same problem as OP but was able to fix using:

Created a DNS A record for my home ISP WAN address.
Opened all required ports for Netbird from the home router to the VM running Fedora.
Needed to turn Selinux off so that Caddy could read the mounted config file in the docker volume.
For adding a Android client with a gmail address, I had to sign up for a free SMTP provider, and add it to Zitadel via https://netbird-hostname/ui/console/instance?id=smtpprovider This allowed me to send a verification email and add a Android client as a peer.

Finally, for accessing Home assistant from outside the home network in Android:

Install the Netbird addon in HA. Change the config for the addon and give a friendly name like xxxxx.
Make sure the HA instance shows up as a peer.
Change the URL for home assistant in the Android app to http://xxxxx.netbird.selfhosted/ in all places.

@ashayh commented on GitHub (Feb 7, 2025): I am self hosting Netbird on Fedora in a home network for accessing Home assistant. I faced the same problem as OP but was able to fix using: 1. Created a DNS A record for my home ISP WAN address. 2. Opened all required ports for Netbird from the home router to the VM running Fedora. 4. Needed to turn Selinux off so that Caddy could read the mounted config file in the docker volume. For adding a Android client with a gmail address, I had to sign up for a free SMTP provider, and add it to Zitadel via https://netbird-hostname/ui/console/instance?id=smtpprovider This allowed me to send a verification email and add a Android client as a peer. Finally, for accessing Home assistant from outside the home network in Android: 1. Install the Netbird addon in HA. Change the config for the addon and give a friendly name like xxxxx. 2. Make sure the HA instance shows up as a peer. 3. Change the URL for home assistant in the Android app to http://xxxxx.netbird.selfhosted/ in all places.

saavagebueno commented

@rafaelfmuniz commented on GitHub (Feb 10, 2025):

I'm also stuck in the same loop

@rafaelfmuniz commented on GitHub (Feb 10, 2025): I'm also stuck in the same loop

saavagebueno commented

@userdiv444 commented on GitHub (Mar 6, 2025):

Using Debian 12.9, and Docker version 27.5.1, script v0.31.1 (latestst) = stuck as well.
Help would be appreciated.

@userdiv444 commented on GitHub (Mar 6, 2025): Using Debian 12.9, and Docker version 27.5.1, script v0.31.1 (latestst) = stuck as well. Help would be appreciated.

saavagebueno commented

@b4rgut commented on GitHub (Mar 12, 2025):

I had a problem with Ubuntu 24.04.2 LTS and netbird v0.38.0. I fixed it by adding information about the local IP of the server and the netbird domain to /etc/hosts: 192.168.1.x netbird.example.com .

The installation was successful, but the dashboard was not responding. I looked at the docker logs netbird-dashboard-1 and saw the error socket() [::]:80 failed (97: Address family not supported by protocol), which says that Nginx is trying to listen to IPv6 addresses (::):80, but it is disabled in the system or container settings IPv6 support.

I have indeed disabled IPv6. When I turned it on, everything worked.

@b4rgut commented on GitHub (Mar 12, 2025): I had a problem with Ubuntu 24.04.2 LTS and netbird v0.38.0. I fixed it by adding information about the local IP of the server and the netbird domain to /etc/hosts: `192.168.1.x netbird.example.com `. The installation was successful, but the dashboard was not responding. I looked at the `docker logs netbird-dashboard-1` and saw the error `socket() [::]:80 failed (97: Address family not supported by protocol)`, which says that Nginx is trying to listen to IPv6 addresses `(::):80`, but it is disabled in the system or container settings IPv6 support. I have indeed disabled IPv6. When I turned it on, everything worked.

saavagebueno commented

@YapWC commented on GitHub (Mar 23, 2025):

curl: (22) The requested URL returned error: 403

Unable to connect to Zitadel for more than 45s, please check the output above, your firewall rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates

If anyone if facing issue with 403 error and cf-mitigated: challenge then it is something to do with your CloudFlare proxy. I switched the proxy off on my CloudFlare and everything works.

For more context I am not sure why the above error only appears on Hetzner Europe Server, I tried with Hetzner Singapore Server and it worked fine without any error with the exact configuration and setup all along. Maybe CloudFlare is flagging the Public IP Address of my Europe Server as suspicious (something to do with IP Reputation) and therefore denying the request.

Anyways hope this help for those that is on CloudFlare.

Update:
So it is true that my Public IP Address of the EU Server was posed with Managed Challenge by CloudFlare Proxy when it tries to access Zitadel or NetBird via API as well. In this case instead of turning off the Proxy there is another solution.

Go to your CloudFlare DNS -> Security -> Events. You will see which IP address is being blocked/Challanged, then just copy the IP Address go to WAF -> Tools and add the IP address, Allow this Website. This is another solution for those who would like to have the proxy enabled all the time.

@YapWC commented on GitHub (Mar 23, 2025): ``` curl: (22) The requested URL returned error: 403 Unable to connect to Zitadel for more than 45s, please check the output above, your firewall rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates ``` If anyone if facing issue with `403 error` and `cf-mitigated: challenge` then it is something to do with your CloudFlare proxy. I switched the proxy off on my CloudFlare and everything works. For more context I am not sure why the above error only appears on Hetzner Europe Server, I tried with Hetzner Singapore Server and it worked fine without any error with the exact configuration and setup all along. Maybe CloudFlare is flagging the Public IP Address of my Europe Server as suspicious (something to do with IP Reputation) and therefore denying the request. Anyways hope this help for those that is on CloudFlare. Update: So it is true that my Public IP Address of the EU Server was posed with Managed Challenge by CloudFlare Proxy when it tries to access Zitadel or NetBird via API as well. In this case instead of turning off the Proxy there is another solution. Go to your CloudFlare DNS -> Security -> Events. You will see which IP address is being blocked/Challanged, then just copy the IP Address go to WAF -> Tools and add the IP address, Allow this Website. This is another solution for those who would like to have the proxy enabled all the time.

saavagebueno commented

@Builder-DE-TH commented on GitHub (Apr 27, 2025):

This is still broken nearly 6 months later. Just tried again with v0.43.0 on a Proxmox LXC behind NPMplus and got the same error.

@Builder-DE-TH commented on GitHub (Apr 27, 2025): This is still broken nearly 6 months later. Just tried again with v0.43.0 on a Proxmox LXC behind NPMplus and got the same error.

saavagebueno commented

@ntk148v commented on GitHub (May 8, 2025):

Hit the same issue, Caddy logs returns:


{"level":"debug","ts":1746688657.6179042,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"kiennt.io.vn"}
{"level":"debug","ts":1746688657.617912,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.io.vn"}
{"level":"debug","ts":1746688657.6179156,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.vn"}
{"level":"debug","ts":1746688657.6179192,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}

@ntk148v commented on GitHub (May 8, 2025): Hit the same issue, Caddy logs returns: ```shell {"level":"debug","ts":1746688657.6179042,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"kiennt.io.vn"} {"level":"debug","ts":1746688657.617912,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.io.vn"} {"level":"debug","ts":1746688657.6179156,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.vn"} {"level":"debug","ts":1746688657.6179192,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"} ```

saavagebueno commented

@ntk148v commented on GitHub (May 16, 2025):

Hi folks, I have created a PR for this. Hope this helps, at least in my case, everything is working well now.

@ntk148v commented on GitHub (May 16, 2025): Hi folks, I have created a PR for this. Hope this helps, at least in my case, everything is working well now.

saavagebueno commented

@aaluopy commented on GitHub (Jul 2, 2025):

same error on RockyLinux9 with 0.49.0

[root@aluopy netbird]# export NETBIRD_DOMAIN=xxx.xxx.com; curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started-with-zitadel.sh | bash
Use Postgres as default Zitadel database.
For using CockroachDB please the environment variable 'export ZITADEL_DATABASE=cockroach'.
Rendering initial files...

Starting Zitadel IDP for user management


WARN[0000] /apps/netbird/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion 
[+] Running 7/7
 ✔ Network netbird_netbird                 Created                                                                                                                                                                                      0.1s 
 ✔ Volume "netbird_netbird_zitadel_certs"  Created                                                                                                                                                                                      0.0s 
 ✔ Volume "netbird_netbird_zdb_data"       Created                                                                                                                                                                                      0.0s 
 ✔ Volume "netbird_netbird_caddy_data"     Created                                                                                                                                                                                      0.0s 
 ✔ Container netbird-caddy-1               Started                                                                                                                                                                                      0.4s 
 ✔ Container netbird-zdb-1                 Healthy                                                                                                                                                                                      5.9s 
 ✔ Container netbird-zitadel-1             Started                                                                                                                                                                                      6.0s 

Initializing Zitadel with NetBird's applications

Waiting for Zitadel's PAT to be created  . . . . done
Reading Zitadel PAT
Waiting for Zitadel to become ready  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying x.x.x.x:443...
* Connected to xxx.xxx.com (x.x.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS header, Unknown (21):
{ [5 bytes data]
* TLSv1.3 (IN), TLS alert, internal error (592):
{ [2 bytes data]
* error:0A000438:SSL routines::tlsv1 alert internal error
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

Unable to connect to Zitadel for more than 45s, please check the output above, your firewall rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates
 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

@aaluopy commented on GitHub (Jul 2, 2025): same error on RockyLinux9 with 0.49.0 ``` [root@aluopy netbird]# export NETBIRD_DOMAIN=xxx.xxx.com; curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started-with-zitadel.sh | bash Use Postgres as default Zitadel database. For using CockroachDB please the environment variable 'export ZITADEL_DATABASE=cockroach'. Rendering initial files... Starting Zitadel IDP for user management WARN[0000] /apps/netbird/docker-compose.yml: the attribute `version` is obsolete, it will be ignored, please remove it to avoid potential confusion [+] Running 7/7 ✔ Network netbird_netbird Created 0.1s ✔ Volume "netbird_netbird_zitadel_certs" Created 0.0s ✔ Volume "netbird_netbird_zdb_data" Created 0.0s ✔ Volume "netbird_netbird_caddy_data" Created 0.0s ✔ Container netbird-caddy-1 Started 0.4s ✔ Container netbird-zdb-1 Healthy 5.9s ✔ Container netbird-zitadel-1 Started 6.0s Initializing Zitadel with NetBird's applications Waiting for Zitadel's PAT to be created . . . . done Reading Zitadel PAT Waiting for Zitadel to become ready . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying x.x.x.x:443... * Connected to xxx.xxx.com (x.x.x.x) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * CAfile: /etc/pki/tls/certs/ca-bundle.crt * TLSv1.0 (OUT), TLS header, Certificate Status (22): } [5 bytes data] * TLSv1.3 (OUT), TLS handshake, Client hello (1): } [512 bytes data] * TLSv1.2 (IN), TLS header, Unknown (21): { [5 bytes data] * TLSv1.3 (IN), TLS alert, internal error (592): { [2 bytes data] * error:0A000438:SSL routines::tlsv1 alert internal error 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 * Closing connection 0 curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error Unable to connect to Zitadel for more than 45s, please check the output above, your firewall rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

saavagebueno commented

@mlsmaycon commented on GitHub (Jul 2, 2025):

@aaluopy

Unable to connect to Zitadel for more than 45s, please check the output above, your firewall rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates

Can you share what did you check from this message?

@mlsmaycon commented on GitHub (Jul 2, 2025): @aaluopy ``` Unable to connect to Zitadel for more than 45s, please check the output above, your firewall rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates ``` Can you share what did you check from this message?

saavagebueno commented

@aaluopy commented on GitHub (Jul 4, 2025):

@aaluopy

Unable to connect to Zitadel for more than 45s, please check the output above, your firewall rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates

Can you share what did you check from this message?

Sorry, forgot to update my troubleshooting status. I figured out the issue—it's because my domain isn't ICP-registered. The Caddy logs showed Let's Encrypt attempting HTTP-01 validation by accessing my domain, but the response received was the ICP filing interception page from https://dnspod.qcloud.com/static/webblock.html?d=xxx.mydomain.cn.

{"log":"{\"level\":\"error\",\"ts\":1751640755.798765,\"msg\":\"challenge failed\",\"identifier\":\"xxx.mydomain.cn\",\"challenge_type\":\"http-01\",\"problem\":{\"type\":\"urn:ietf:params:acme:error:unauthorized\",\"title\":\"\",\"detail
\":\"x.x.x.x: Invalid response from https://dnspod.qcloud.com/static/webblock.html?d=xxx.mydomain.cn: \\\"\u003c!DOCTYPE html\u003e\\\\n\u003chtml\u003e\\\\n\\\\n\u003chead\u003e\\\\n  \\\\n  \u003cmeta http-equiv=\\\\\\\"X-UA-Compat
ible\\\\\\\" content=\\\\\\\"IE=edge\\\\\\\" /\u003e\\\\n\u003cmeta http-equiv=\\\\\\\"Content-Type\\\\\\\" cont\\\"\",\"instance\":\"\",\"subproblems\":null},\"stacktrace\":\"github.com/mholt/acmez/v3.(*Client).pollAuthorization\\n\\tgi
thub.com/mholt/acmez/v3@v3.1.2/client.go:557\\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\\n\\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\\n\\tgithub.com/mholt/acmez/v3
@v3.1.2/client.go:136\\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\\n\\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\\n\\tgithub.com/caddyserver/certmagic@v
0.23.0/acmeissuer.go:382\\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\\n\\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2
\\n\\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\\ngithub.com/caddyserver/certmagic.doWithRetry\\n\\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\\n\\tgithub.co
m/caddyserver/certmagic@v0.23.0/config.go:700\\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\\n\\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\\n\\tgith
ub.com/caddyserver/certmagic@v0.23.0/config.go:415\\ngithub.com/caddyserver/certmagic.(*jobManager).worker\\n\\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73\"}\n","stream":"stderr","time":"2025-07-04T14:52:35.798919079Z"}

@aaluopy commented on GitHub (Jul 4, 2025): > [@aaluopy](https://github.com/aaluopy) > > ``` > Unable to connect to Zitadel for more than 45s, please check the output above, your firewall rules and the caddy container logs to confirm if there are any issues provisioning TLS certificates > ``` > > Can you share what did you check from this message? Sorry, forgot to update my troubleshooting status. I figured out the issue—it's because my domain isn't ICP-registered. The Caddy logs showed Let's Encrypt attempting HTTP-01 validation by accessing my domain, but the response received was the ICP filing interception page from https://dnspod.qcloud.com/static/webblock.html?d=xxx.mydomain.cn. ``` {"log":"{\"level\":\"error\",\"ts\":1751640755.798765,\"msg\":\"challenge failed\",\"identifier\":\"xxx.mydomain.cn\",\"challenge_type\":\"http-01\",\"problem\":{\"type\":\"urn:ietf:params:acme:error:unauthorized\",\"title\":\"\",\"detail \":\"x.x.x.x: Invalid response from https://dnspod.qcloud.com/static/webblock.html?d=xxx.mydomain.cn: \\\"\u003c!DOCTYPE html\u003e\\\\n\u003chtml\u003e\\\\n\\\\n\u003chead\u003e\\\\n \\\\n \u003cmeta http-equiv=\\\\\\\"X-UA-Compat ible\\\\\\\" content=\\\\\\\"IE=edge\\\\\\\" /\u003e\\\\n\u003cmeta http-equiv=\\\\\\\"Content-Type\\\\\\\" cont\\\"\",\"instance\":\"\",\"subproblems\":null},\"stacktrace\":\"github.com/mholt/acmez/v3.(*Client).pollAuthorization\\n\\tgi thub.com/mholt/acmez/v3@v3.1.2/client.go:557\\ngithub.com/mholt/acmez/v3.(*Client).solveChallenges\\n\\tgithub.com/mholt/acmez/v3@v3.1.2/client.go:378\\ngithub.com/mholt/acmez/v3.(*Client).ObtainCertificate\\n\\tgithub.com/mholt/acmez/v3 @v3.1.2/client.go:136\\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).doIssue\\n\\tgithub.com/caddyserver/certmagic@v0.23.0/acmeissuer.go:489\\ngithub.com/caddyserver/certmagic.(*ACMEIssuer).Issue\\n\\tgithub.com/caddyserver/certmagic@v 0.23.0/acmeissuer.go:382\\ngithub.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue\\n\\tgithub.com/caddyserver/caddy/v2@v2.10.0/modules/caddytls/acmeissuer.go:288\\ngithub.com/caddyserver/certmagic.(*Config).obtainCert.func2 \\n\\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:626\\ngithub.com/caddyserver/certmagic.doWithRetry\\n\\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:104\\ngithub.com/caddyserver/certmagic.(*Config).obtainCert\\n\\tgithub.co m/caddyserver/certmagic@v0.23.0/config.go:700\\ngithub.com/caddyserver/certmagic.(*Config).ObtainCertAsync\\n\\tgithub.com/caddyserver/certmagic@v0.23.0/config.go:505\\ngithub.com/caddyserver/certmagic.(*Config).manageOne.func1\\n\\tgith ub.com/caddyserver/certmagic@v0.23.0/config.go:415\\ngithub.com/caddyserver/certmagic.(*jobManager).worker\\n\\tgithub.com/caddyserver/certmagic@v0.23.0/async.go:73\"}\n","stream":"stderr","time":"2025-07-04T14:52:35.798919079Z"} ```

saavagebueno commented

@Raito00 commented on GitHub (Oct 9, 2025):

when try latest:

export NETBIRD_DOMAIN=netbird.example.com; curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started-with-zitadel.sh | bash

I have the same problem on
Proxmox Debian 12 LXC

-- Unable to connect to Zitadel for more than 45s,

@Raito00 commented on GitHub (Oct 9, 2025): when try latest: ``` export NETBIRD_DOMAIN=netbird.example.com; curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started-with-zitadel.sh | bash ``` I have the same problem on Proxmox Debian 12 LXC -- Unable to connect to Zitadel for more than 45s,

saavagebueno commented

@Mi-Siu commented on GitHub (Oct 10, 2025):

when try latest:
export NETBIRD_DOMAIN=netbird.example.com; curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started-with-zitadel.sh | bash
I have the same problem on Proxmox Debian 12 LXC

-- Unable to connect to Zitadel for more than 45s,

Found somewhere that adding to /etc/hosts domain (the same you are exporting as NETBIRD_DOMAIN) with your LOCAL machine IP might be helpful, and... for me it worked, script continues (also changed DNS instead of my local to 8.8.8.8 in /etc/resolv.conf - don't know if that matters).

@Mi-Siu commented on GitHub (Oct 10, 2025): > when try latest: > > ``` > export NETBIRD_DOMAIN=netbird.example.com; curl -fsSL https://github.com/netbirdio/netbird/releases/latest/download/getting-started-with-zitadel.sh | bash > ``` > > I have the same problem on Proxmox Debian 12 LXC > > -- Unable to connect to Zitadel for more than 45s, Found somewhere that adding to /etc/hosts domain (the same you are exporting as NETBIRD_DOMAIN) with your LOCAL machine IP might be helpful, and... for me it worked, script continues (also changed DNS instead of my local to 8.8.8.8 in /etc/resolv.conf - don't know if that matters).

saavagebueno commented

@Tacioandrade commented on GitHub (Nov 5, 2025):

I'm having the same problem! I initially installed it on a Debian 13 VM running on a Proxmox behind a NAT, it didn't work. I thought it was because ports 80 and 443 were behind a NAT and the application couldn't generate the HTTPS certificates via Certbot, so I hired a VPS from Hetzner to test it and the problem persists.

I've already tried:
1 - Setting the DNS configuration for Netbird in the /etc/hosts file (both 127.0.0.1 and the WAN IP)
2 - Running the script as root instead of the normal user

In addition, the "netbird-zitadel-1" container also loops, outputting this:

time="2025-11-05T22:03:11Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.smtp_configs5 time="2025-11-05T22:03:11Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.debug_events time="2025-11-05T22:03:11Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.notifications time="2025-11-05T22:03:12Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.instance_domains time="2025-11-05T22:03:12Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.login_policies5 time="2025-11-05T22:03:12Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.security_policies2 time="2025-11-05T22:03:13Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.privacy_policies4 time="2025-11-05T22:03:14Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.milestones time="2025-11-05T22:03:15Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.password_complexity_policies2 time="2025-11-05T22:03:15Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.project_roles4 time="2025-11-05T22:03:16Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.notifications time="2025-11-05T22:03:16Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.custom_texts2 time="2025-11-05T22:03:16Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.flow_triggers3 time="2025-11-05T22:03:17Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.instance_members4 time="2025-11-05T22:03:17Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.authn_keys2 time="2025-11-05T22:03:17Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.lockout_policies3 ^Croot@netbird:~# docker logs -f netbird-zitadel-1

@Tacioandrade commented on GitHub (Nov 5, 2025): I'm having the same problem! I initially installed it on a Debian 13 VM running on a Proxmox behind a NAT, it didn't work. I thought it was because ports 80 and 443 were behind a NAT and the application couldn't generate the HTTPS certificates via Certbot, so I hired a VPS from Hetzner to test it and the problem persists. I've already tried: 1 - Setting the DNS configuration for Netbird in the /etc/hosts file (both 127.0.0.1 and the WAN IP) 2 - Running the script as root instead of the normal user In addition, the "netbird-zitadel-1" container also loops, outputting this: `time="2025-11-05T22:03:11Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.smtp_configs5 time="2025-11-05T22:03:11Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.debug_events time="2025-11-05T22:03:11Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.notifications time="2025-11-05T22:03:12Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.instance_domains time="2025-11-05T22:03:12Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.login_policies5 time="2025-11-05T22:03:12Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.security_policies2 time="2025-11-05T22:03:13Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.privacy_policies4 time="2025-11-05T22:03:14Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.milestones time="2025-11-05T22:03:15Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.password_complexity_policies2 time="2025-11-05T22:03:15Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.project_roles4 time="2025-11-05T22:03:16Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.notifications time="2025-11-05T22:03:16Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.custom_texts2 time="2025-11-05T22:03:16Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.flow_triggers3 time="2025-11-05T22:03:17Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.instance_members4 time="2025-11-05T22:03:17Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.authn_keys2 time="2025-11-05T22:03:17Z" level=debug msg="trigger iteration" caller="/home/runner/work/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:415" iteration=0 projection=projections.lockout_policies3 ^Croot@netbird:~# docker logs -f netbird-zitadel-1`

saavagebueno commented

@mikebakke commented on GitHub (Nov 5, 2025):

@Tacioandrade how long did you leave it "looping"? I've just looked at my working instance and I seem to have a lot of those messages in my log. My recollection is it took a minute or 2 to finally setup and output the creds.

I started with a proxmox VM which failed to gen certs but then like you bought a VPS and it did run - I don't believe I modified the quick setup - just ran it. I've just checked on the VPS and I don't see any changes in my configs to get it to work.

Since then I have managed to do it on a vm in proxmox by adding the L4 module to my existing caddy which just forwards any raw packets for 80 and 443 to the VM.

@mikebakke commented on GitHub (Nov 5, 2025): @Tacioandrade how long did you leave it "looping"? I've just looked at my working instance and I seem to have a *lot* of those messages in my log. My recollection is it took a minute or 2 to finally setup and output the creds. I started with a proxmox VM which failed to gen certs but then like you bought a VPS and it did run - I don't believe I modified the quick setup - just ran it. I've just checked on the VPS and I don't see any changes in my configs to get it to work. Since then I have managed to do it on a vm in proxmox by adding the L4 module to my existing caddy which just forwards any raw packets for 80 and 443 to the VM.

saavagebueno commented