SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-08-04 19:55:09 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

Failed to add for key 169.254.169.254/32: route not allowed since 0.46.0 #1985

New Issue
Closed
opened 2025-11-20 06:10:41 -05:00 by saavagebueno · 4 comments
saavagebueno commented 2025-11-20 06:10:41 -05:00
Owner
Copy Link

Originally created by @Kamaradeivanov on GitHub (Jun 19, 2025).

Describe the problem

Hello, i'm unable to access private DNS records (from GCP) since client version 0.46.0. After checking debug logs, I encountered following errors :

2025-06-19T09:54:16+02:00 ERRO client/internal/routemanager/manager.go:464: No active handler found for route gcp-google-metadata-server|169.254.169.254/32
2025-06-19T09:54:16+02:00 ERRO client/internal/engine.go:1025: failed to update routes: 1 error occurred:
	* update system routes: 1 error occurred:
	* add route 169.254.169.254/32: failed to add for key 169.254.169.254/32: route not allowed

I'm not sure, but this method added 2 weeks ago could be in cause.

23b5d45b68/client/internal/routemanager/systemops/systemops.go (L61)

To Reproduce

Steps to reproduce the behavior:

  1. Install client in version 0.45.3
  2. On control plane, create a resource to add 169.254.169.254/32 to a network with a routing peers setup in GCP environment, create a policy to allow this route from your client
  3. Connect the client
  4. Run route show, a route is present to send traffic to the wiregard interface
  5. Install a client in version 0.46 or upper
  6. There is no more route for 169.254.169.254
  7. Private DNS records are unreachable

Expected behavior

I should have a route that allows 169.254.169.254/32 to be accessible via the wiregard interface, but that's not the case. Rolling back clients to version 0.45.3 solve the issue.

Are you using NetBird Cloud?

No, i'm on a self-host NetBird's control plane.

NetBird version

netbird version 0.46.0 an upper (tested on 0.47.2 also)

Is any other VPN software installed?

No other VPN

Debug output

Peers detail:
 netbird-peers-7bdc558fd8-6j9zk.netbird.selfhosted:
  NetBird IP: 100.XX.XX.XX
  Public key: iO6maextXqGWGxoJWQsoqQFHBTSjuiJ4ttjM14IuG3w=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://vpn.anon-FZtWu.domain:443/relay
  Last connection update: 25 minutes, 13 seconds ago
  Last WireGuard handshake: 2 minutes, 10 seconds ago
  Transfer status (received/sent) 1.5 KiB/3.3 KiB
  Quantum resistance: false
  Networks: -
  Latency: 0s

 netbirds-peers.netbird.selfhosted:
  NetBird IP: 100.XX.XX.XX
  Public key: iwRyUoEVxg56PqDxgans722kQ4X0Zv18p56FKbCe9iY=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://vpn.anon-FZtWu.domain:443/relay
  Last connection update: 25 minutes, 13 seconds ago
  Last WireGuard handshake: 2 minutes, 10 seconds ago
  Transfer status (received/sent) 1.5 KiB/3.3 KiB
  Quantum resistance: false
  Networks: -
  Latency: 0s

 netbird-peers-7bdc558fd8-7p4m5.netbird.selfhosted:
  NetBird IP: 100.XX.XX.XX
  Public key: GN28grvhldRYZzbyL2VyehN/HxZ/HNW4iZPDL897Q2s=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://vpn.anon-FZtWu.domain:443/relay
  Last connection update: 25 minutes, 13 seconds ago
  Last WireGuard handshake: 2 minutes, 10 seconds ago
  Transfer status (received/sent) 1.5 KiB/3.3 KiB
  Quantum resistance: false
  Networks: 10.XX.XX.XX/12
  Latency: 0s

 netbird-peers-7bdc558fd8-knvth.netbird.selfhosted:
  NetBird IP: 100.XX.XX.XX
  Public key: LaI+4rbH5aohKJ/pZ9WxDgvk0DEhaeMe5IpUmPMA0Cw=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://vpn.anon-FZtWu.domain:443/relay
  Last connection update: 25 minutes, 13 seconds ago
  Last WireGuard handshake: 2 minutes, 10 seconds ago
  Transfer status (received/sent) 1.5 KiB/3.3 KiB
  Quantum resistance: false
  Networks: -
  Latency: 0s

 netbird.netbird.selfhosted:
  NetBird IP: 100.XX.XX.XX
  Public key: rnCYvzZXjtBKhJwV8LV8tYzTe31rLHITzMo8wieBsGo=
  Status: Connected
  -- detail --
  Connection type: Relayed
  ICE candidate (Local/Remote): -/-
  ICE candidate endpoints (Local/Remote): -/-
  Relay server address: rels://vpn.anon-FZtWu.domain:443/relay
  Last connection update: 25 minutes, 13 seconds ago
  Last WireGuard handshake: 1 minute, 57 seconds ago
  Transfer status (received/sent) 41.1 MiB/3.5 MiB
  Quantum resistance: false
  Networks: 10.XX.XX.XX/10, 10.XX.XX.XX/14, 10.XX.XX.XX/12
  Latency: 0s

Events:
  [WARNING] DNS (f06f7e48-e309-41b5-be2f-42c191dcc46f)
    Message: All upstream servers failed (probe failed)
    Time: 6 minutes, 37 seconds ago
    Metadata: upstreams: 169.254.169.254:53
  [WARNING] DNS (704966b6-2a14-4f8c-892a-d55cdd7935b1)
    Message: All upstream servers failed (probe failed)
    Time: 6 minutes, 37 seconds ago
    Metadata: upstreams: 169.254.169.254:53
  [WARNING] DNS (e2ef6ca5-8212-4cc8-8b1a-e02567f97f41)
    Message: All upstream servers failed (probe failed)
    Time: 6 minutes, 37 seconds ago
    Metadata: upstreams: 169.254.169.254:53
  [WARNING] DNS (31d64be0-2ce9-4c78-aeda-bacb40e216c9)
    Message: All upstream servers failed (probe failed)
    Time: 6 minutes, 37 seconds ago
    Metadata: upstreams: 169.254.169.254:53
  [INFO] SYSTEM (fb6165cf-cbb8-4831-8351-ae75394a7416)
    Message: Network map updated
    Time: 6 minutes, 37 seconds ago
  [WARNING] DNS (f8bd7b92-8f9d-44fb-8eb0-79a87188965e)
    Message: All upstream servers failed (probe failed)
    Time: 5 minutes, 1 seconds ago
    Metadata: upstreams: 169.254.169.254:53
  [WARNING] DNS (1023a698-aade-435a-a2e5-15fd3814f060)
    Message: All upstream servers failed (probe failed)
    Time: 5 minutes, 1 seconds ago
    Metadata: upstreams: 169.254.169.254:53
  [WARNING] DNS (9cd76b2c-ea29-4a46-8fbc-8c283d080315)
    Message: All upstream servers failed (probe failed)
    Time: 5 minutes, 1 seconds ago
    Metadata: upstreams: 169.254.169.254:53
  [WARNING] DNS (b0b9c907-3489-4e7b-af45-4ed257b07246)
    Message: All upstream servers failed (probe failed)
    Time: 5 minutes, 1 seconds ago
    Metadata: upstreams: 169.254.169.254:53
  [INFO] SYSTEM (fb107350-c6b5-49e5-9ebc-60ac1aa02989)
    Message: Network map updated
    Time: 5 minutes, 1 seconds ago
OS: windows/amd64
Daemon version: 0.47.2
CLI version: 0.47.2
Management: Connected to https://vpn.anon-FZtWu.domain:443
Signal: Connected to https://vpn.anon-FZtWu.domain:443
Relays:
  [rels://vpn.anon-FZtWu.domain:443/relay] is Available
Nameservers:
  [169.254.169.254:53] for [anon-FZtWu.domain, anon-3dEZe.domain, anon-Bs5zY.domain, anon-f9Yfr.domain] is Unavailabl
e, reason: 1 error occurred:
        * with udp: dial udp 169.254.169.254:53: connect: A socket operation was attempted to an unreachable network.
FQDN: ivan-desktop.netbird.selfhosted
NetBird IP: 100.XX.XX.XX/16
Interface type: Userspace
Quantum resistance: false
Lazy connection: false
Networks: -
Forwarding rules: 0
Peers count: 5/5 Connected

Screenshots

No screenshot

Additional context

No more context

Have you tried these troubleshooting steps?

  • Reviewed client troubleshooting (if applicable)
  • Checked for newer NetBird versions
  • Searched for similar issues on GitHub (including closed ones)
  • Restarted the NetBird client
  • Disabled other VPN software
  • Checked firewall settings
Originally created by @Kamaradeivanov on GitHub (Jun 19, 2025). **Describe the problem** Hello, i'm unable to access private DNS records (from GCP) since client version 0.46.0. After checking debug logs, I encountered following errors : ``` 2025-06-19T09:54:16+02:00 ERRO client/internal/routemanager/manager.go:464: No active handler found for route gcp-google-metadata-server|169.254.169.254/32 2025-06-19T09:54:16+02:00 ERRO client/internal/engine.go:1025: failed to update routes: 1 error occurred: * update system routes: 1 error occurred: * add route 169.254.169.254/32: failed to add for key 169.254.169.254/32: route not allowed ``` I'm not sure, but this method added 2 weeks ago could be in cause. https://github.com/netbirdio/netbird/blob/23b5d45b68c4caeb8bb493f58aed08aad665aa4d/client/internal/routemanager/systemops/systemops.go#L61 **To Reproduce** Steps to reproduce the behavior: 1. Install client in version 0.45.3 2. On control plane, create a resource to add 169.254.169.254/32 to a network with a routing peers setup in GCP environment, create a policy to allow this route from your client 3. Connect the client 4. Run `route show`, a route is present to send traffic to the wiregard interface 5. Install a client in version 0.46 or upper 6. There is no more route for 169.254.169.254 7. Private DNS records are unreachable **Expected behavior** I should have a route that allows 169.254.169.254/32 to be accessible via the wiregard interface, but that's not the case. Rolling back clients to version 0.45.3 solve the issue. **Are you using NetBird Cloud?** No, i'm on a self-host NetBird's control plane. **NetBird version** `netbird version 0.46.0 an upper (tested on 0.47.2 also)` **Is any other VPN software installed?** No other VPN **Debug output** ``` Peers detail: netbird-peers-7bdc558fd8-6j9zk.netbird.selfhosted: NetBird IP: 100.XX.XX.XX Public key: iO6maextXqGWGxoJWQsoqQFHBTSjuiJ4ttjM14IuG3w= Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://vpn.anon-FZtWu.domain:443/relay Last connection update: 25 minutes, 13 seconds ago Last WireGuard handshake: 2 minutes, 10 seconds ago Transfer status (received/sent) 1.5 KiB/3.3 KiB Quantum resistance: false Networks: - Latency: 0s netbirds-peers.netbird.selfhosted: NetBird IP: 100.XX.XX.XX Public key: iwRyUoEVxg56PqDxgans722kQ4X0Zv18p56FKbCe9iY= Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://vpn.anon-FZtWu.domain:443/relay Last connection update: 25 minutes, 13 seconds ago Last WireGuard handshake: 2 minutes, 10 seconds ago Transfer status (received/sent) 1.5 KiB/3.3 KiB Quantum resistance: false Networks: - Latency: 0s netbird-peers-7bdc558fd8-7p4m5.netbird.selfhosted: NetBird IP: 100.XX.XX.XX Public key: GN28grvhldRYZzbyL2VyehN/HxZ/HNW4iZPDL897Q2s= Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://vpn.anon-FZtWu.domain:443/relay Last connection update: 25 minutes, 13 seconds ago Last WireGuard handshake: 2 minutes, 10 seconds ago Transfer status (received/sent) 1.5 KiB/3.3 KiB Quantum resistance: false Networks: 10.XX.XX.XX/12 Latency: 0s netbird-peers-7bdc558fd8-knvth.netbird.selfhosted: NetBird IP: 100.XX.XX.XX Public key: LaI+4rbH5aohKJ/pZ9WxDgvk0DEhaeMe5IpUmPMA0Cw= Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://vpn.anon-FZtWu.domain:443/relay Last connection update: 25 minutes, 13 seconds ago Last WireGuard handshake: 2 minutes, 10 seconds ago Transfer status (received/sent) 1.5 KiB/3.3 KiB Quantum resistance: false Networks: - Latency: 0s netbird.netbird.selfhosted: NetBird IP: 100.XX.XX.XX Public key: rnCYvzZXjtBKhJwV8LV8tYzTe31rLHITzMo8wieBsGo= Status: Connected -- detail -- Connection type: Relayed ICE candidate (Local/Remote): -/- ICE candidate endpoints (Local/Remote): -/- Relay server address: rels://vpn.anon-FZtWu.domain:443/relay Last connection update: 25 minutes, 13 seconds ago Last WireGuard handshake: 1 minute, 57 seconds ago Transfer status (received/sent) 41.1 MiB/3.5 MiB Quantum resistance: false Networks: 10.XX.XX.XX/10, 10.XX.XX.XX/14, 10.XX.XX.XX/12 Latency: 0s Events: [WARNING] DNS (f06f7e48-e309-41b5-be2f-42c191dcc46f) Message: All upstream servers failed (probe failed) Time: 6 minutes, 37 seconds ago Metadata: upstreams: 169.254.169.254:53 [WARNING] DNS (704966b6-2a14-4f8c-892a-d55cdd7935b1) Message: All upstream servers failed (probe failed) Time: 6 minutes, 37 seconds ago Metadata: upstreams: 169.254.169.254:53 [WARNING] DNS (e2ef6ca5-8212-4cc8-8b1a-e02567f97f41) Message: All upstream servers failed (probe failed) Time: 6 minutes, 37 seconds ago Metadata: upstreams: 169.254.169.254:53 [WARNING] DNS (31d64be0-2ce9-4c78-aeda-bacb40e216c9) Message: All upstream servers failed (probe failed) Time: 6 minutes, 37 seconds ago Metadata: upstreams: 169.254.169.254:53 [INFO] SYSTEM (fb6165cf-cbb8-4831-8351-ae75394a7416) Message: Network map updated Time: 6 minutes, 37 seconds ago [WARNING] DNS (f8bd7b92-8f9d-44fb-8eb0-79a87188965e) Message: All upstream servers failed (probe failed) Time: 5 minutes, 1 seconds ago Metadata: upstreams: 169.254.169.254:53 [WARNING] DNS (1023a698-aade-435a-a2e5-15fd3814f060) Message: All upstream servers failed (probe failed) Time: 5 minutes, 1 seconds ago Metadata: upstreams: 169.254.169.254:53 [WARNING] DNS (9cd76b2c-ea29-4a46-8fbc-8c283d080315) Message: All upstream servers failed (probe failed) Time: 5 minutes, 1 seconds ago Metadata: upstreams: 169.254.169.254:53 [WARNING] DNS (b0b9c907-3489-4e7b-af45-4ed257b07246) Message: All upstream servers failed (probe failed) Time: 5 minutes, 1 seconds ago Metadata: upstreams: 169.254.169.254:53 [INFO] SYSTEM (fb107350-c6b5-49e5-9ebc-60ac1aa02989) Message: Network map updated Time: 5 minutes, 1 seconds ago OS: windows/amd64 Daemon version: 0.47.2 CLI version: 0.47.2 Management: Connected to https://vpn.anon-FZtWu.domain:443 Signal: Connected to https://vpn.anon-FZtWu.domain:443 Relays: [rels://vpn.anon-FZtWu.domain:443/relay] is Available Nameservers: [169.254.169.254:53] for [anon-FZtWu.domain, anon-3dEZe.domain, anon-Bs5zY.domain, anon-f9Yfr.domain] is Unavailabl e, reason: 1 error occurred: * with udp: dial udp 169.254.169.254:53: connect: A socket operation was attempted to an unreachable network. FQDN: ivan-desktop.netbird.selfhosted NetBird IP: 100.XX.XX.XX/16 Interface type: Userspace Quantum resistance: false Lazy connection: false Networks: - Forwarding rules: 0 Peers count: 5/5 Connected ``` **Screenshots** No screenshot **Additional context** No more context **Have you tried these troubleshooting steps?** - [x] Reviewed [client troubleshooting](https://docs.netbird.io/how-to/troubleshooting-client) (if applicable) - [x] Checked for newer NetBird versions - [x] Searched for similar issues on GitHub (including closed ones) - [x] Restarted the NetBird client - [x] Disabled other VPN software - [x] Checked firewall settings
saavagebueno added the triage-needed label 2025-11-20 06:10:41 -05:00
saavagebueno commented 2025-11-20 06:10:42 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Jun 19, 2025):

I understand that this is a special address used in cloud environments, but it is part of the link-local subnet 169.254.0.0/16 and not supposed to be routed! Do you have any alternative address to access the DNS there?

@lixmal commented on GitHub (Jun 19, 2025): I understand that this is a special address used in cloud environments, but it is part of the link-local subnet `169.254.0.0/16` and **not supposed to be routed**! Do you have any alternative address to access the DNS there?
saavagebueno commented 2025-11-20 06:10:43 -05:00
Author
Owner
Copy Link

@Kamaradeivanov commented on GitHub (Jun 19, 2025):

Most cloud providers use the 169.254.169.254/32 address to allow access to the Cloud-Init Metadata Server, as you can see here: e9016aecea/client/system/detect_cloud.

Below is the resolvectl status result from a Nebird peer on GCP:

Global
         Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub
Current DNS Server: 169.254.169.254
       DNS Servers: 169.254.169.254
        DNS Domain: local

Link 2 (ens4)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 169.254.169.254
       DNS Servers: 169.254.169.254
        DNS Domain: c.project-name.internal google.internal

Can we expect an exception for routing the address 169.254.169.254?

Can we expect an exception for routing the address 169.254.169.254? Am I the only one using Netbird and private DNS zone records from hyperscalers?

I suppose I could set up an intermediate DNS server with DNSMaq on my GCP peer, but that would cost me the maintenance of another tool.

@Kamaradeivanov commented on GitHub (Jun 19, 2025): Most cloud providers use the 169.254.169.254/32 address to allow access to the Cloud-Init Metadata Server, as you can see here: https://github.com/netbirdio/netbird/tree/e9016aeceaa85be61f317c9bc3adfec7ff4558d0/client/system/detect_cloud. Below is the resolvectl status result from a Nebird peer on GCP: ``` Global Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported resolv.conf mode: stub Current DNS Server: 169.254.169.254 DNS Servers: 169.254.169.254 DNS Domain: local Link 2 (ens4) Current Scopes: DNS Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Current DNS Server: 169.254.169.254 DNS Servers: 169.254.169.254 DNS Domain: c.project-name.internal google.internal ``` Can we expect an exception for routing the address 169.254.169.254? Can we expect an exception for routing the address 169.254.169.254? Am I the only one using Netbird and private DNS zone records from hyperscalers? I suppose I could set up an intermediate DNS server with DNSMaq on my GCP peer, but that would cost me the maintenance of another tool.
saavagebueno commented 2025-11-20 06:10:43 -05:00
Author
Owner
Copy Link

@nazarewk commented on GitHub (Jun 19, 2025):

I might be a little too pedantic here, but routing cloud machine's metadata outside the cloud sounds like a pretty serious credential leak/data exfiltration risk. If you really do want to expose pieces of it, I'd advise at least spinning up a local DNS forwarded and/or a very strict reverse proxy to whatever you want to access in the metadata server, then exposing that local mechanism's local address over the NetBird instead of metadata endpoints directly.

If you do want to just use the DNS, you could set it up as domain (possibly wildcard) based Network Resource instead.

@nazarewk commented on GitHub (Jun 19, 2025): I might be a little too pedantic here, but routing cloud machine's metadata outside the cloud sounds like a pretty serious credential leak/data exfiltration risk. If you really do want to expose pieces of it, I'd advise at least spinning up a local DNS forwarded and/or a very strict reverse proxy to whatever you want to access in the metadata server, then exposing that local mechanism's **local** address over the NetBird instead of metadata endpoints directly. If you do want to just use the DNS, you could set it up as domain (possibly wildcard) based Network Resource instead.
saavagebueno commented 2025-11-20 06:10:43 -05:00
Author
Owner
Copy Link

@Kamaradeivanov commented on GitHub (Jun 19, 2025):

Actually, I only allow access to the metadata server on port 53 UDP via policy, so the risk is probably mitigated. However, I could be wrong, and this could be a major security vulnerability.

In any case, I was so focused on the Nameserver configuration on Netbird that I forgot that we could set up a wildcard domain directly on the network resource. It works perfectly! Thank you for your time.

@Kamaradeivanov commented on GitHub (Jun 19, 2025): Actually, I only allow access to the metadata server on port 53 UDP via policy, so the risk is probably mitigated. However, I could be wrong, and this could be a major security vulnerability. In any case, I was so focused on the Nameserver configuration on Netbird that I forgot that we could set up a wildcard domain directly on the network resource. It works perfectly! Thank you for your time.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label triage-needed
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1985
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?