GCP request is missing required authentication credential #389

Closed
opened 2025-11-20 04:23:27 -05:00 by saavagebueno · 11 comments
Owner

Originally created by @linchenqian on GitHub (Apr 30, 2024).

TLDR: using GCP provider is unable to update due to 401 error "CREDENTIALS_MISSING"

  1. Is this urgent: Yes

  2. DNS provider(s) you use: GCP

  3. Program version:

    Running version 2.6.0 built on 2024-02-03T18:57:00Z (commit c338c28ce3ab3aa9948f0fe133307b75fe6d34d2)

  4. What are you using to run the container: docker in Mac / windows binary(exe file) / container in TrueNas

  5. Extra information (optional)

Logs:

2024-04-30T00:56:09-04:00 INFO ipv4 address of <mydomain.com> is 108.xxx.xxx.xxx and your ipv4 address  is 108.xx.xxx.xx
2024-04-30T00:56:09-04:00 INFO Updating record [domain: <mydomain.com> | host: @ | provider: gcp | ip: ipv4] to use 108.xx.xxx.xx
2024-04-30T00:56:09-04:00 ERROR getting record resource set: googleapi: Error 401: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "method": "cloud.dns.api.v1.ResourceRecordSetsService.Get",
      "service": "dns.googleapis.com"
    },
    "reason": "CREDENTIALS_MISSING"
  }
]

More details:
Reason: required, Message: Login Required.

Configuration file:

{
  "settings": [
    {
      "provider": "gcp",
      "project": "my-project",
      "zone": "my-zone",
      "credentials": {
        "type": "service_account",
        "project_id": "myproject",
        "private_key_id": "abcd123456",
        "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgI..........MqGX/\n-----END PRIVATE KEY-----\n",
        "client_email": "ddns-updater-xxxxxx@xxxxxxx.iam.gserviceaccount.com",
        "client_id": "10163123456789",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/......",
        "universe_domain": "googleapis.com"
      },
      "domain": "mydomain.com",
      "host": "@",
      "ip_version": "ipv4",
      "ipv6_suffix": ""
    }
  ]
}

Host OS: Windows10/MacOS/Debian12

Originally created by @linchenqian on GitHub (Apr 30, 2024). <!-- YOU CAN CHAT THERE EVENTUALLY: https://github.com/qdm12/ddns-updater/discussions --> **TLDR**: using GCP provider is unable to update due to 401 error "CREDENTIALS_MISSING" 1. Is this urgent: Yes 2. DNS provider(s) you use: GCP 3. Program version: <!-- See the line at the top of your logs --> `Running version 2.6.0 built on 2024-02-03T18:57:00Z (commit c338c28ce3ab3aa9948f0fe133307b75fe6d34d2)` 4. What are you using to run the container: docker in Mac / windows binary(exe file) / container in TrueNas 6. Extra information (optional) Logs: ```log 2024-04-30T00:56:09-04:00 INFO ipv4 address of <mydomain.com> is 108.xxx.xxx.xxx and your ipv4 address is 108.xx.xxx.xx 2024-04-30T00:56:09-04:00 INFO Updating record [domain: <mydomain.com> | host: @ | provider: gcp | ip: ipv4] to use 108.xx.xxx.xx 2024-04-30T00:56:09-04:00 ERROR getting record resource set: googleapi: Error 401: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project. Details: [ { "@type": "type.googleapis.com/google.rpc.ErrorInfo", "domain": "googleapis.com", "metadata": { "method": "cloud.dns.api.v1.ResourceRecordSetsService.Get", "service": "dns.googleapis.com" }, "reason": "CREDENTIALS_MISSING" } ] More details: Reason: required, Message: Login Required. ``` Configuration file: ```json { "settings": [ { "provider": "gcp", "project": "my-project", "zone": "my-zone", "credentials": { "type": "service_account", "project_id": "myproject", "private_key_id": "abcd123456", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvgI..........MqGX/\n-----END PRIVATE KEY-----\n", "client_email": "ddns-updater-xxxxxx@xxxxxxx.iam.gserviceaccount.com", "client_id": "10163123456789", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/......", "universe_domain": "googleapis.com" }, "domain": "mydomain.com", "host": "@", "ip_version": "ipv4", "ipv6_suffix": "" } ] } ``` Host OS: Windows10/MacOS/Debian12
Author
Owner

@qdm12 commented on GitHub (Apr 30, 2024):

Update your image, Running version latest built on 2020-03-13T01:30:06Z is 4 years old.

@qdm12 commented on GitHub (Apr 30, 2024): Update your image, `Running version latest built on 2020-03-13T01:30:06Z` is 4 years old.
Author
Owner

@linchenqian commented on GitHub (Apr 30, 2024):

Sorry forgot to update that line in the template, it's actually 2.6.0 built on 2024-02-03

Running version 2.6.0 built on 2024-02-03T18:57:00Z (commit c338c28ce3ab3aa9948f0fe133307b75fe6d34d2)
@linchenqian commented on GitHub (Apr 30, 2024): Sorry forgot to update that line in the template, it's actually 2.6.0 built on 2024-02-03 ``` Running version 2.6.0 built on 2024-02-03T18:57:00Z (commit c338c28ce3ab3aa9948f0fe133307b75fe6d34d2) ```
Author
Owner

@linchenqian commented on GitHub (Apr 30, 2024):

can you check again please?

@linchenqian commented on GitHub (Apr 30, 2024): can you check again please?
Author
Owner

@qdm12 commented on GitHub (May 1, 2024):

Can you try with the latest image (pull it)? I recently upgraded the gcp dependency, maybe this solves it?

@qdm12 commented on GitHub (May 1, 2024): Can you try with the latest image (pull it)? I recently upgraded the gcp dependency, maybe this solves it?
Author
Owner

@linchenqian commented on GitHub (May 3, 2024):

updated to the latest version, still having same issue:

Full logs:

========================================
========================================
============= ddns-updater =============
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================

Running version latest built on 2024-05-03T07:56:04.164Z (commit b131c3d)

🔧 Need help? https://github.com/qdm12/ddns-updater/discussions/new
🐛 Bug? https://github.com/qdm12/ddns-updater/issues/new
✨ New feature? https://github.com/qdm12/ddns-updater/issues/new
☕ Discussion? https://github.com/qdm12/ddns-updater/discussions/new
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2024-05-03T09:16:19Z INFO Settings summary:
├── HTTP client
|   └── Timeout: 10s
├── Update
|   ├── Period: 5m0s
|   └── Cooldown: 5m0s
├── Public IP fetching
|   ├── HTTP enabled: yes
|   ├── HTTP IP providers
|   |   └── all
|   ├── HTTP IPv4 providers
|   |   └── all
|   ├── HTTP IPv6 providers
|   |   └── all
|   ├── DNS enabled: yes
|   ├── DNS timeout: 3s
|   └── DNS over TLS providers
|       └── all
├── Resolver: use Go default resolver
├── Server
|   ├── Listening address: :8000
|   └── Root URL: /
├── Health
|   └── Server listening address: 127.0.0.1:9999
├── Paths
|   └── Data directory: /updater/data
├── Backup: disabled
└── Logger
    ├── Level: info
    └── Caller: hidden
2024-05-03T09:16:19Z INFO reading JSON config from file /updater/data/config.json
2024-05-03T09:16:19Z INFO Found single setting to update record
2024-05-03T09:16:19Z INFO Reading history from database: domain xxxx.xxx host @ ipv4
2024-05-03T09:16:19Z INFO [backup] disabled
2024-05-03T09:16:19Z INFO [healthcheck server] listening on 127.0.0.1:9999
2024-05-03T09:16:19Z INFO [http server] listening on :8000
2024-05-03T09:16:20Z INFO obtaining ipv4 address succeeded after 1 failed try
2024-05-03T09:16:20Z INFO ipv4 address of xxxx.xxx is 108.123.111.111 and your ipv4 address  is 108.123.222.233
2024-05-03T09:16:20Z INFO Updating record [domain: xxxx.xxx | host: @ | provider: gcp | ip: ipv4] to use 108.123.222.233
2024-05-03T09:16:20Z ERROR getting record resource set: googleapi: Error 401: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "method": "cloud.dns.api.v1.ResourceRecordSetsService.Get",
      "service": "dns.googleapis.com"
    },
    "reason": "CREDENTIALS_MISSING"
  }
]

More details:
Reason: required, Message: Login Required.
@linchenqian commented on GitHub (May 3, 2024): updated to the latest version, still having same issue: Full logs: ``` ======================================== ======================================== ============= ddns-updater ============= ======================================== =========== Made with ❤️ by ============ ======= https://github.com/qdm12 ======= ======================================== ======================================== Running version latest built on 2024-05-03T07:56:04.164Z (commit b131c3d) 🔧 Need help? https://github.com/qdm12/ddns-updater/discussions/new 🐛 Bug? https://github.com/qdm12/ddns-updater/issues/new ✨ New feature? https://github.com/qdm12/ddns-updater/issues/new ☕ Discussion? https://github.com/qdm12/ddns-updater/discussions/new 💻 Email? quentin.mcgaw@gmail.com 💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12 2024-05-03T09:16:19Z INFO Settings summary: ├── HTTP client | └── Timeout: 10s ├── Update | ├── Period: 5m0s | └── Cooldown: 5m0s ├── Public IP fetching | ├── HTTP enabled: yes | ├── HTTP IP providers | | └── all | ├── HTTP IPv4 providers | | └── all | ├── HTTP IPv6 providers | | └── all | ├── DNS enabled: yes | ├── DNS timeout: 3s | └── DNS over TLS providers | └── all ├── Resolver: use Go default resolver ├── Server | ├── Listening address: :8000 | └── Root URL: / ├── Health | └── Server listening address: 127.0.0.1:9999 ├── Paths | └── Data directory: /updater/data ├── Backup: disabled └── Logger ├── Level: info └── Caller: hidden 2024-05-03T09:16:19Z INFO reading JSON config from file /updater/data/config.json 2024-05-03T09:16:19Z INFO Found single setting to update record 2024-05-03T09:16:19Z INFO Reading history from database: domain xxxx.xxx host @ ipv4 2024-05-03T09:16:19Z INFO [backup] disabled 2024-05-03T09:16:19Z INFO [healthcheck server] listening on 127.0.0.1:9999 2024-05-03T09:16:19Z INFO [http server] listening on :8000 2024-05-03T09:16:20Z INFO obtaining ipv4 address succeeded after 1 failed try 2024-05-03T09:16:20Z INFO ipv4 address of xxxx.xxx is 108.123.111.111 and your ipv4 address is 108.123.222.233 2024-05-03T09:16:20Z INFO Updating record [domain: xxxx.xxx | host: @ | provider: gcp | ip: ipv4] to use 108.123.222.233 2024-05-03T09:16:20Z ERROR getting record resource set: googleapi: Error 401: Request is missing required authentication credential. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project. Details: [ { "@type": "type.googleapis.com/google.rpc.ErrorInfo", "domain": "googleapis.com", "metadata": { "method": "cloud.dns.api.v1.ResourceRecordSetsService.Get", "service": "dns.googleapis.com" }, "reason": "CREDENTIALS_MISSING" } ] More details: Reason: required, Message: Login Required. ```
Author
Owner

@qdm12 commented on GitHub (May 3, 2024):

Last questions before I jump in verifying the code, was it ever working before for you? Have you tried with another program to verify your credentials are correct? Thanks!

@qdm12 commented on GitHub (May 3, 2024): Last questions before I jump in verifying the code, was it ever working before for you? Have you tried with another program to verify your credentials are correct? Thanks!
Author
Owner

@linchenqian commented on GitHub (May 4, 2024):

was it ever working before for you?

No, this is first time I use ddns-updater. I tried on multiple platform, all result in same error.

Have you tried with another program to verify your credentials are correct?

Yes, I have verified the credentials with gcloud cli

root@14d144a2678b:/home# cat creds.json 
{
                "type": "service_account",
                "project_id": ......
            }
root@14d144a2678b:/home# gcloud auth activate-service-account --key-file ./creds.json 
Activated service account credentials for: [ddns-updater-xxxxxxx@xxxx-xxx.iam.gserviceaccount.com]
root@14d144a2678b:/home# gcloud auth print-access-token
ya29.c.c0AY_VpZgwgNS......
root@14d144a2678b:/home# gcloud dns record-sets describe xxxx.xxx. --zone=my-zone --type=A --project=my-project
NAME        TYPE  TTL  DATA
xxxx.xxx.  A     300  100.0: 108.xx.xxx.xx

and also I tried copy part of the source code from update.go to a new program for testing, and it was able to pass the point where the error was thrown (getResourceRecordSet). and get result from google cloud dns. here is the code of the test program:

package main

import (
	"encoding/json"
	"errors"
	"fmt"
	"net/http"
	"github.com/qdm12/ddns-updater/internal/provider/constants"
	ddnserrors "github.com/qdm12/ddns-updater/internal/provider/errors"
	"github.com/qdm12/goshutdown"
	clouddns "google.golang.org/api/dns/v1"
	"google.golang.org/api/googleapi"
	"google.golang.org/api/option"
)

func main() {
	// Raw JSON string
	raw_creds := `{
		"type": "service_account",
		"project_id": ......
	}`

	// Assign the raw JSON string to a variable of type json.RawMessage
	credentials := json.RawMessage(raw_creds)

	_, ctx, _ := goshutdown.NewGoRoutineHandler("runner")
	ddnsService, err := clouddns.NewService(ctx, option.WithCredentialsJSON(credentials))
	if err != nil {
		fmt.Printf("Error creating DNS service: %s\n", err)
		return
	}

	rrSetsService := clouddns.NewResourceRecordSetsService(ddnsService)

	fqdn := fmt.Sprintf("%s.%s.", "", "xxx.xxx")
	recordType := constants.A
	recordResourceSet, err := getResourceRecordSet(rrSetsService, fqdn, recordType)
	if err != nil {
		fmt.Printf("Error getting resource record set: %s\n", err)
		return
	}

	for _, rrdata := range recordResourceSet.Rrdatas {
		if rrdata == "108.222.111.002" {
			// already up to date
			fmt.Print("ip ")
		}
	}
}

func getResourceRecordSet(rrSetsService *clouddns.ResourceRecordSetsService,
	fqdn, recordType string) (resourceRecordSet *clouddns.ResourceRecordSet, err error) {
	resourceRecordSet, err = rrSetsService.Get("my-project", "my-zone", fqdn, recordType).Do()
	if err != nil {
		googleAPIError := new(googleapi.Error)
		if errors.As(err, &googleAPIError) && googleAPIError.Code == http.StatusNotFound {
			return nil, fmt.Errorf("%w: %w", ddnserrors.ErrRecordResourceSetNotFound, err)
		}
		return nil, err
	}

	return resourceRecordSet, nil
}

@linchenqian commented on GitHub (May 4, 2024): > was it ever working before for you? No, this is first time I use ddns-updater. I tried on multiple platform, all result in same error. > Have you tried with another program to verify your credentials are correct? Yes, I have verified the credentials with gcloud cli ``` root@14d144a2678b:/home# cat creds.json { "type": "service_account", "project_id": ...... } root@14d144a2678b:/home# gcloud auth activate-service-account --key-file ./creds.json Activated service account credentials for: [ddns-updater-xxxxxxx@xxxx-xxx.iam.gserviceaccount.com] root@14d144a2678b:/home# gcloud auth print-access-token ya29.c.c0AY_VpZgwgNS...... root@14d144a2678b:/home# gcloud dns record-sets describe xxxx.xxx. --zone=my-zone --type=A --project=my-project NAME TYPE TTL DATA xxxx.xxx. A 300 100.0: 108.xx.xxx.xx ``` and also I tried copy part of the source code from `update.go` to a new program for testing, and it was able to pass the point where the error was thrown (`getResourceRecordSet`). and get result from google cloud dns. here is the code of the test program: ```go package main import ( "encoding/json" "errors" "fmt" "net/http" "github.com/qdm12/ddns-updater/internal/provider/constants" ddnserrors "github.com/qdm12/ddns-updater/internal/provider/errors" "github.com/qdm12/goshutdown" clouddns "google.golang.org/api/dns/v1" "google.golang.org/api/googleapi" "google.golang.org/api/option" ) func main() { // Raw JSON string raw_creds := `{ "type": "service_account", "project_id": ...... }` // Assign the raw JSON string to a variable of type json.RawMessage credentials := json.RawMessage(raw_creds) _, ctx, _ := goshutdown.NewGoRoutineHandler("runner") ddnsService, err := clouddns.NewService(ctx, option.WithCredentialsJSON(credentials)) if err != nil { fmt.Printf("Error creating DNS service: %s\n", err) return } rrSetsService := clouddns.NewResourceRecordSetsService(ddnsService) fqdn := fmt.Sprintf("%s.%s.", "", "xxx.xxx") recordType := constants.A recordResourceSet, err := getResourceRecordSet(rrSetsService, fqdn, recordType) if err != nil { fmt.Printf("Error getting resource record set: %s\n", err) return } for _, rrdata := range recordResourceSet.Rrdatas { if rrdata == "108.222.111.002" { // already up to date fmt.Print("ip ") } } } func getResourceRecordSet(rrSetsService *clouddns.ResourceRecordSetsService, fqdn, recordType string) (resourceRecordSet *clouddns.ResourceRecordSet, err error) { resourceRecordSet, err = rrSetsService.Get("my-project", "my-zone", fqdn, recordType).Do() if err != nil { googleAPIError := new(googleapi.Error) if errors.As(err, &googleAPIError) && googleAPIError.Code == http.StatusNotFound { return nil, fmt.Errorf("%w: %w", ddnserrors.ErrRecordResourceSetNotFound, err) } return nil, err } return resourceRecordSet, nil } ```
Author
Owner

@qdm12 commented on GitHub (May 5, 2024):

Hmm strange, your code looks rather similar to the one in ddns-updater, maybe it's just the json credentials not being parsed/read correctly. I've done two things:

  • ddns-updater now checks the JSON of credentials is well formed and has the "type" field present (commit 093e8154f3)
  • I pushed a 'debug' docker image (see PR #710) with image name qmcgaw/ddns-updater:pr-710 which logs the JSON credentials, can you please try it and report if the JSON credentials are well formed?

Thanks!

@qdm12 commented on GitHub (May 5, 2024): Hmm strange, your code looks rather similar to the one in ddns-updater, maybe it's just the json credentials not being parsed/read correctly. I've done two things: - ddns-updater now checks the JSON of credentials is well formed and has the `"type"` field present (commit 093e8154f3cc090ab217ec6c95632e7154a9fb3e) - I pushed a 'debug' docker image (see PR #710) with image name `qmcgaw/ddns-updater:pr-710` which logs the JSON credentials, can you please try it and report if the JSON credentials are well formed? Thanks!
Author
Owner

@qdm12 commented on GitHub (Jun 14, 2024):

If you modify the line

ddnsService, err := clouddns.NewService(ctx, option.WithCredentialsJSON(credentials))

to

ddnsService, err := clouddns.NewService(ctx, option.WithCredentialsJSON(credentials), option.WithHTTPClient(http.DefaultClient))

Does it still work?

I'm digging in the google sdk to hopefully throw it out, and it seems like plugging in your own client just does not care about authentication; so that could be why it doesn't work??

@qdm12 commented on GitHub (Jun 14, 2024): If you modify the line ```go ddnsService, err := clouddns.NewService(ctx, option.WithCredentialsJSON(credentials)) ``` to ```go ddnsService, err := clouddns.NewService(ctx, option.WithCredentialsJSON(credentials), option.WithHTTPClient(http.DefaultClient)) ``` Does it still work? I'm digging in the google sdk to hopefully throw it out, and it seems like plugging in your own client just does not care about authentication; so that could be why it doesn't work??
Author
Owner

@qdm12 commented on GitHub (Jun 15, 2024):

Hi @linchenqian I just pushed a84a1d664a which removes the dependency on the SDK, and it should work (with the custom http client, oauth2, the credentials.json, everything, hopefully!). Please try it with docker and the latest image (docker pull qmcgaw/ddns-updater). Thanks!

@qdm12 commented on GitHub (Jun 15, 2024): Hi @linchenqian I just pushed a84a1d664a02b09b9313cb13e1bf8a1acbbb73b1 which removes the dependency on the SDK, and it *should* work (with the custom http client, oauth2, the credentials.json, everything, hopefully!). Please try it with docker and the latest image (`docker pull qmcgaw/ddns-updater`). Thanks!
Author
Owner

@qdm12 commented on GitHub (Sep 20, 2024):

Closing due to inactivity.

@qdm12 commented on GitHub (Sep 20, 2024): Closing due to inactivity.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/ddns-updater#389