[GH-ISSUE #5086] gRPC client sets :authority header with port, breaking GKE Gateway compatibility #10666

Open
opened 2026-08-05 01:26:49 -04:00 by saavagebueno · 2 comments
Owner

Originally created by @eliminyro on GitHub (Jan 10, 2026).
Original GitHub issue: https://github.com/netbirdio/netbird/issues/5086

Description

When connecting to a NetBird management server behind a GKE Gateway (Google Kubernetes Engine Gateway API), native clients fail with:

retrying Login to the Management service in 1.102599784s due to error rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key

Extended debugging reveals the underlying gRPC error:

GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info sudo netbird up -F -l debug --management-url https://nb-api.example.com
2026-01-10T17:07:08+01:00 DEBG client/net/dialer_dial.go:20: Dialing tcp nb-api.example.com:443
2026-01-10T17:07:09+01:00 DEBG client/internal/login.go:65: connected to the Management service https://nb-api.example.com:443
2026-01-10T17:07:09+01:00 ERRO shared/management/client/grpc.go:282: failed while getting Management Service public key: rpc error: code = Unimplemented desc = fault filter abort
2026-01-10T17:07:09+01:00 ERRO client/internal/login.go:111: failed while getting Management Service public key: failed while getting Management Service public key
2026-01-10T17:07:09+01:00 WARN client/cmd/root.go:248: retrying Login to the Management service in 963.390646ms due to error failed while getting Management Service public key

Root Cause

The NetBird gRPC client sends the :authority HTTP/2 pseudo-header with the port included (e.g., nb-api.example.com:443), but GKE Gateway's HTTPRoute only matches hostnames without ports (e.g., nb-api.example.com). This causes the gateway to reject the request with "fault filter abort".

Evidence

Testing with grpcurl reproduces the issue:

# Fails - sends :authority with port
$ grpcurl -proto management.proto nb-api.example.com:443 management.ManagementService/GetServerKey
ERROR: Code: Unimplemented, Message: fault filter abort

# Works - explicit authority without port
$ grpcurl -authority nb-api.example.com -proto management.proto nb-api.example.com:443 management.ManagementService/GetServerKey
{
  "key": "...",
  "expiresAt": ...
}

Additionally, curl (which sends :authority without port) works correctly:

$ curl --http2 -X POST -H "Content-Type: application/grpc" https://nb-api.example.com/management.ManagementService/GetServerKey
# Returns grpc-status: 2 (expected for empty request body)

Direct connection via kubectl port-forward (bypassing the gateway) works perfectly, confirming the management server itself is correctly configured.

Environment

  • Deployment: Self-hosted NetBird on GKE
  • Gateway: GKE Gateway API with gke-l7-regional-external-managed gateway class
  • NetBird version: latest (netbirdio/management:latest)
  • Client: Native macOS client
  • Protocol: gRPC over HTTPS (TLS termination at gateway, re-encryption to backend)

Expected Behavior

The gRPC client should send the :authority header without the port when connecting to standard HTTPS port 443, matching the behavior of other HTTP/2 clients like curl.

Originally created by @eliminyro on GitHub (Jan 10, 2026). Original GitHub issue: https://github.com/netbirdio/netbird/issues/5086 ## Description When connecting to a NetBird management server behind a GKE Gateway (Google Kubernetes Engine Gateway API), native clients fail with: ``` retrying Login to the Management service in 1.102599784s due to error rpc error: code = Unknown desc = getting device authorization flow info failed with error: failed while getting Management Service public key ``` Extended debugging reveals the underlying gRPC error: ```bash GRPC_GO_LOG_VERBOSITY_LEVEL=99 GRPC_GO_LOG_SEVERITY_LEVEL=info sudo netbird up -F -l debug --management-url https://nb-api.example.com ``` ``` 2026-01-10T17:07:08+01:00 DEBG client/net/dialer_dial.go:20: Dialing tcp nb-api.example.com:443 2026-01-10T17:07:09+01:00 DEBG client/internal/login.go:65: connected to the Management service https://nb-api.example.com:443 2026-01-10T17:07:09+01:00 ERRO shared/management/client/grpc.go:282: failed while getting Management Service public key: rpc error: code = Unimplemented desc = fault filter abort 2026-01-10T17:07:09+01:00 ERRO client/internal/login.go:111: failed while getting Management Service public key: failed while getting Management Service public key 2026-01-10T17:07:09+01:00 WARN client/cmd/root.go:248: retrying Login to the Management service in 963.390646ms due to error failed while getting Management Service public key ``` ## Root Cause The NetBird gRPC client sends the `:authority` HTTP/2 pseudo-header with the port included (e.g., `nb-api.example.com:443`), but GKE Gateway's HTTPRoute only matches hostnames without ports (e.g., `nb-api.example.com`). This causes the gateway to reject the request with "fault filter abort". ## Evidence Testing with `grpcurl` reproduces the issue: ```bash # Fails - sends :authority with port $ grpcurl -proto management.proto nb-api.example.com:443 management.ManagementService/GetServerKey ERROR: Code: Unimplemented, Message: fault filter abort # Works - explicit authority without port $ grpcurl -authority nb-api.example.com -proto management.proto nb-api.example.com:443 management.ManagementService/GetServerKey { "key": "...", "expiresAt": ... } ``` Additionally, `curl` (which sends `:authority` without port) works correctly: ```bash $ curl --http2 -X POST -H "Content-Type: application/grpc" https://nb-api.example.com/management.ManagementService/GetServerKey # Returns grpc-status: 2 (expected for empty request body) ``` Direct connection via `kubectl port-forward` (bypassing the gateway) works perfectly, confirming the management server itself is correctly configured. ## Environment - **Deployment**: Self-hosted NetBird on GKE - **Gateway**: GKE Gateway API with `gke-l7-regional-external-managed` gateway class - **NetBird version**: latest (netbirdio/management:latest) - **Client**: Native macOS client - **Protocol**: gRPC over HTTPS (TLS termination at gateway, re-encryption to backend) ## Expected Behavior The gRPC client should send the `:authority` header without the port when connecting to standard HTTPS port 443, matching the behavior of other HTTP/2 clients like `curl`.
Author
Owner

@eliminyro commented on GitHub (Jan 10, 2026):

Validated Root Cause

I confirmed the issue by patching the client to set the :authority header without the port. After rebuilding and testing:

Before patch:

ERRO shared/management/client/grpc.go:282: failed while getting Management Service public key: rpc error: code = Unimplemented desc = fault filter abort

After patch:
Client connects successfully and proceeds to OAuth authentication.

How I tested

Applied this patch to client/grpc/dialer.go:

--- a/client/grpc/dialer.go
+++ b/client/grpc/dialer.go
@@ -4,6 +4,7 @@ import (
 	"context"
 	"crypto/tls"
 	"crypto/x509"
+	"net"
 	"fmt"
 	"runtime"
 	"time"
@@ -44,11 +45,18 @@ func CreateConnection(ctx context.Context, addr string, tlsEnabled bool, compone
 	connCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
 	defer cancel()

+	// Extract host without port for the :authority header to match HTTP conventions
+	// and ensure compatibility with L7 load balancers that route based on hostname.
+	authority := addr
+	if host, _, err := net.SplitHostPort(addr); err == nil {
+		authority = host
+	}
+
 	conn, err := grpc.DialContext(
 		connCtx,
 		addr,
 		transportOption,
 		WithCustomDialer(tlsEnabled, component),
 		grpc.WithBlock(),
+		grpc.WithAuthority(authority),
 		grpc.WithKeepaliveParams(keepalive.ClientParameters{

This matches what grpcurl -authority <host> does, which also works against my setup.

I will leave the actual implementation approach to you, as there may be edge cases or considerations I am not aware of.

<!-- gh-comment-id:3733195237 --> @eliminyro commented on GitHub (Jan 10, 2026): ## Validated Root Cause I confirmed the issue by patching the client to set the `:authority` header without the port. After rebuilding and testing: **Before patch:** ``` ERRO shared/management/client/grpc.go:282: failed while getting Management Service public key: rpc error: code = Unimplemented desc = fault filter abort ``` **After patch:** Client connects successfully and proceeds to OAuth authentication. ### How I tested Applied this patch to `client/grpc/dialer.go`: ```diff --- a/client/grpc/dialer.go +++ b/client/grpc/dialer.go @@ -4,6 +4,7 @@ import ( "context" "crypto/tls" "crypto/x509" + "net" "fmt" "runtime" "time" @@ -44,11 +45,18 @@ func CreateConnection(ctx context.Context, addr string, tlsEnabled bool, compone connCtx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() + // Extract host without port for the :authority header to match HTTP conventions + // and ensure compatibility with L7 load balancers that route based on hostname. + authority := addr + if host, _, err := net.SplitHostPort(addr); err == nil { + authority = host + } + conn, err := grpc.DialContext( connCtx, addr, transportOption, WithCustomDialer(tlsEnabled, component), grpc.WithBlock(), + grpc.WithAuthority(authority), grpc.WithKeepaliveParams(keepalive.ClientParameters{ ``` This matches what `grpcurl -authority <host>` does, which also works against my setup. I will leave the actual implementation approach to you, as there may be edge cases or considerations I am not aware of.
Author
Owner

@iamkhalidbashir commented on GitHub (Feb 28, 2026):

+1 this
do you have client builds @eliminyro ?

<!-- gh-comment-id:3978760213 --> @iamkhalidbashir commented on GitHub (Feb 28, 2026): +1 this do you have client builds @eliminyro ?
Sign in to join this conversation.
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: DYNR/netbird#10666