SVI/netbird
1
0
Fork 0
mirror of https://github.com/netbirdio/netbird.git synced 2026-06-10 01:12:51 -04:00
Code Issues 3.0k Packages Projects Releases 10 Wiki Activity

DNS response not routed back to client via Netbird Agent in docker with VPC access #1502

New Issue
Closed
opened 2025-11-20 05:31:48 -05:00 by saavagebueno · 7 comments
saavagebueno commented 2025-11-20 05:31:48 -05:00
Owner
Copy Link

Originally created by @hidemire on GitHub (Dec 17, 2024).

Describe the problem

When using Netbird to set up VPN access to a VPC network (10.10.0.0/16), DNS queries initiated from the client (e.g., 100.103.197.210) are sent to the DNS server inside the VPC (10.10.0.2) via a Netbird agent deployed within an ECS task on EC2. However, the DNS responses do not reach the client.

image

Expected behavior

The DNS response from 10.10.0.2 should reach the originating client (100.103.197.210) via the Netbird agent.

Are you using NetBird Cloud? - Yes

NetBird version

agent - 0.34.1
macos client - 0.34.1

NetBird status -dA output (client) 
Peers detail:
 e35c8306ac1c.netbird.cloud:
  NetBird IP: 100.103.237.110
  Public key: ZU9MOESxT0cRY54EoXRcJwEDACPv4SY42UPDM2uv8zg=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 192.168.64.1:51820/198.51.100.0:12013
  Relay server address: rels://streamline-de-fra1-1.relay.netbird.io:443
  Last connection update: 22 minutes, 40 seconds ago
  Last WireGuard handshake: 1 minute, 33 seconds ago
  Transfer status (received/sent) 3.7 KiB/4.1 KiB
  Quantum resistance: false
  Routes: 10.10.0.0/16
  Latency: 127.600875ms

OS: darwin/arm64
Daemon version: 0.34.1
CLI version: 0.34.1
Management: Connected to https://api.netbird.io:443
Signal: Connected to https://signal.netbird.io:443
Relays:
  [stun:stun.netbird.io:5555] is Available
  [turns:turn.netbird.io:443?transport=tcp] is Available
  [rels://streamline-de-fra1-1.relay.netbird.io:443] is Available
Nameservers:
FQDN: w22031.netbird.cloud
NetBird IP: 100.103.197.210/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 1/1 Connected
tcpdump -i any -vvv -n port 53 (agent)

The client executes the command: dig @10.10.0.2 dagster.dev.dagster.local

tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
21:42:27.709905 wt0   In  IP (tos 0x0, ttl 64, id 62910, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
21:42:27.709940 eth0  Out IP (tos 0x0, ttl 63, id 62910, offset 0, flags [none], proto UDP (17), length 82)
  172.17.0.2.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
21:42:27.712150 eth0  In  IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98)
  10.10.0.2.53 > 172.17.0.2.59642: [udp sum ok] 19638 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [1m] A 10.10.87.201 ar: . OPT UDPsize=4096 (70)
21:42:32.619758 wt0   In  IP (tos 0x0, ttl 64, id 52575, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
21:42:32.619772 eth0  Out IP (tos 0x0, ttl 63, id 52575, offset 0, flags [none], proto UDP (17), length 82)
  172.17.0.2.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
21:42:32.619976 eth0  In  IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98)
  10.10.0.2.53 > 172.17.0.2.59642: [udp sum ok] 19638 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [55s] A 10.10.87.201 ar: . OPT UDPsize=4096 (70)
21:42:37.663749 wt0   In  IP (tos 0x0, ttl 64, id 26739, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
21:42:37.663761 eth0  Out IP (tos 0x0, ttl 63, id 26739, offset 0, flags [none], proto UDP (17), length 82)
  172.17.0.2.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
21:42:37.663993 eth0  In  IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98)
  10.10.0.2.53 > 172.17.0.2.59642: [udp sum ok] 19638 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [50s] A 10.10.87.201 ar: . OPT UDPsize=4096 (70)
iptables-save (agent) 
# Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 17 21:43:44 2024
*mangle
:PREROUTING ACCEPT [3279:3135055]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:NETBIRD-RT-PRE - [0:0]
-A PREROUTING -i wt0 -m addrtype --dst-type LOCAL -j MARK --set-xmark 0x1bd01/0xffffffff
-A PREROUTING -j NETBIRD-RT-PRE
-A NETBIRD-RT-PRE -d 10.10.0.0/16 -i wt0 -m conntrack --ctstate NEW -j MARK --set-xmark 0x1bd11/0xffffffff
-A NETBIRD-RT-PRE -s 10.10.0.0/16 ! -i wt0 -m conntrack --ctstate NEW -j MARK --set-xmark 0x1bd12/0xffffffff
COMMIT
# Completed on Tue Dec 17 21:43:44 2024
# Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 17 21:43:44 2024
*filter
:INPUT ACCEPT [3265:3133907]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2814:283965]
:NETBIRD-ACL-INPUT - [0:0]
:NETBIRD-ACL-OUTPUT - [0:0]
:NETBIRD-RT-FWD - [0:0]
-A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wt0 -j NETBIRD-ACL-INPUT
-A INPUT -i wt0 -j DROP
-A FORWARD -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m mark --mark 0x1bd01 -j NETBIRD-ACL-INPUT
-A FORWARD -i wt0 -j NETBIRD-RT-FWD
-A FORWARD -i wt0 -j DROP
-A OUTPUT -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT ! -d 100.103.0.0/16 -o wt0 -j ACCEPT
-A OUTPUT -o wt0 -j NETBIRD-ACL-OUTPUT
-A OUTPUT -o wt0 -j DROP
-A NETBIRD-ACL-INPUT -j ACCEPT
-A NETBIRD-ACL-OUTPUT -j ACCEPT
-A NETBIRD-RT-FWD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A NETBIRD-RT-FWD -d 10.10.0.0/16 -j ACCEPT
COMMIT
# Completed on Tue Dec 17 21:43:44 2024
# Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 17 21:43:44 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [48:3742]
:NETBIRD-RT-NAT - [0:0]
-A POSTROUTING -j NETBIRD-RT-NAT
-A NETBIRD-RT-NAT ! -o lo -m mark --mark 0x1bd11 -j MASQUERADE
-A NETBIRD-RT-NAT -o wt0 -m mark --mark 0x1bd12 -j MASQUERADE
COMMIT
# Completed on Tue Dec 17 21:43:44 2024
nft list ruleset (agent) 
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
      chain NETBIRD-RT-FWD {
              xt match "conntrack" counter packets 8 bytes 656 accept
              ip daddr 10.10.0.0/16 counter packets 4 bytes 328 accept
      }

      chain NETBIRD-ACL-INPUT {
              counter packets 0 bytes 0 accept
      }

      chain NETBIRD-ACL-OUTPUT {
              counter packets 0 bytes 0 accept
      }

      chain INPUT {
              type filter hook input priority filter; policy accept;
              iifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept
              iifname "wt0" counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT
              iifname "wt0" counter packets 0 bytes 0 drop
      }

      chain OUTPUT {
              type filter hook output priority filter; policy accept;
              oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept
              ip daddr != 100.103.0.0/16 oifname "wt0" counter packets 0 bytes 0 accept
              oifname "wt0" counter packets 0 bytes 0 jump NETBIRD-ACL-OUTPUT
              oifname "wt0" counter packets 0 bytes 0 drop
      }

      chain FORWARD {
              type filter hook forward priority filter; policy accept;
              oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept
              meta mark 0x0001bd01 counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT
              iifname "wt0" counter packets 14 bytes 1148 jump NETBIRD-RT-FWD
              iifname "wt0" counter packets 0 bytes 0 drop
      }
}
# Warning: table ip mangle is managed by iptables-nft, do not touch!
table ip mangle {
      chain NETBIRD-RT-PRE {
              ip daddr 10.10.0.0/16 iifname "wt0" xt match "conntrack" counter packets 4 bytes 328 xt target "MARK"
              ip saddr 10.10.0.0/16 iifname != "wt0" xt match "conntrack" counter packets 0 bytes 0 xt target "MARK"
      }

      chain PREROUTING {
              type filter hook prerouting priority mangle; policy accept;
              iifname "wt0" xt match "addrtype" counter packets 0 bytes 0 xt target "MARK"
              counter packets 3580 bytes 4062979 jump NETBIRD-RT-PRE
      }
}
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
      chain NETBIRD-RT-NAT {
              oifname != "lo" meta mark 0x0001bd11 counter packets 6 bytes 492 xt target "MASQUERADE"
              oifname "wt0" meta mark 0x0001bd12 counter packets 0 bytes 0 xt target "MASQUERADE"
      }

      chain POSTROUTING {
              type nat hook postrouting priority srcnat; policy accept;
              counter packets 64 bytes 4916 jump NETBIRD-RT-NAT
      }
}
ip route (agent) 
default via 172.17.0.1 dev eth0
100.103.0.0/16 dev wt0 scope link  src 100.103.237.110
172.17.0.0/16 dev eth0 scope link  src 172.17.0.2
ECS container definitions 
{
"containerDefinitions": [
  {
    "name": "agent",
    "image": "netbirdio/netbird:latest",
    "cpu": 0,
    "portMappings": [],
    "essential": true,
    "environment": [],
    "mountPoints": [
      {
        "sourceVolume": "netbird-client",
        "containerPath": "/etc/netbird"
      }
    ],
    "volumesFrom": [],
    "linuxParameters": {
      "capabilities": {
        "add": ["NET_ADMIN", "SYS_ADMIN", "SYS_RESOURCE"],
        "drop": []
      }
    },
    "secrets": [
      {
        "name": "NB_SETUP_KEY",
        "valueFrom": "REDACTED"
      }
    ],
    "user": "root",
    "privileged": true,
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": "redacted",
        "awslogs-region": "us-east-1",
        "awslogs-stream-prefix": "agent"
      }
    },
    "systemControls": []
  }
]
}
docker container inspect agent-container 
[
{
  "Id": "017bd520a7fc54ab502a1f607c0954ef261b0b8d13dfb4ed864c2e13cf5a9bc5",
  "Created": "2024-12-17T21:23:50.424260748Z",
  "Path": "/usr/local/bin/netbird",
  "Args": ["up"],
  "State": {
    "Status": "running",
    "Running": true,
    "Paused": false,
    "Restarting": false,
    "OOMKilled": false,
    "Dead": false,
    "Pid": 1709,
    "ExitCode": 0,
    "Error": "",
    "StartedAt": "2024-12-17T21:23:53.711975634Z",
    "FinishedAt": "0001-01-01T00:00:00Z"
  },
  "Image": "sha256:bf0d5410a49f95db023add30a5c1da4260e3017fb769595ecf90ce65f527a7af",
  "ResolvConfPath": "/var/lib/docker/containers/017bd520a7fc54ab502a1f607c0954ef261b0b8d13dfb4ed864c2e13cf5a9bc5/resolv.conf",
  "HostnamePath": "/var/lib/docker/containers/017bd520a7fc54ab502a1f607c0954ef261b0b8d13dfb4ed864c2e13cf5a9bc5/hostname",
  "HostsPath": "/var/lib/docker/containers/017bd520a7fc54ab502a1f607c0954ef261b0b8d13dfb4ed864c2e13cf5a9bc5/hosts",
  "LogPath": "",
  "Name": "/ecs-m-dev-netbird-netbird-agent-10-agent-be8ba0ba90849983ce01",
  "RestartCount": 0,
  "Driver": "overlay2",
  "Platform": "linux",
  "MountLabel": "",
  "ProcessLabel": "",
  "AppArmorProfile": "",
  "HostConfig": {
    "Binds": [
      "ecs-m-dev-netbird-netbird-agent-10-netbird-client-94a993e3f4f9a3ad4100:/etc/netbird",
      "redacted",
      "redacted",
      "redacted",
      "redacted",
      "redacted",
      "redacted",
      "redacted"
    ],
    "ContainerIDFile": "",
    "NetworkMode": "bridge",
    "PortBindings": {},
    "RestartPolicy": {
      "Name": "no",
      "MaximumRetryCount": 0
    },
    "AutoRemove": false,
    "VolumeDriver": "",
    "VolumesFrom": [],
    "ConsoleSize": [0, 0],
    "CapAdd": ["NET_ADMIN", "SYS_ADMIN", "SYS_RESOURCE"],
    "CapDrop": [],
    "CgroupnsMode": "host",
    "Dns": null,
    "DnsOptions": null,
    "DnsSearch": null,
    "ExtraHosts": null,
    "GroupAdd": null,
    "IpcMode": "shareable",
    "Cgroup": "",
    "Links": null,
    "OomScoreAdj": 0,
    "PidMode": "",
    "Privileged": true,
    "PublishAllPorts": false,
    "ReadonlyRootfs": false,
    "SecurityOpt": ["label=disable"],
    "UTSMode": "",
    "UsernsMode": "",
    "ShmSize": 67108864,
    "Runtime": "runc",
    "Isolation": "",
    "CpuShares": 2,
    "Memory": 0,
    "NanoCpus": 0,
    "CgroupParent": "/ecs/5011fa3ce88d471baa4c0c3fe1303f44",
    "BlkioWeight": 0,
    "BlkioWeightDevice": null,
    "BlkioDeviceReadBps": null,
    "BlkioDeviceWriteBps": null,
    "BlkioDeviceReadIOps": null,
    "BlkioDeviceWriteIOps": null,
    "CpuPeriod": 0,
    "CpuQuota": 0,
    "CpuRealtimePeriod": 0,
    "CpuRealtimeRuntime": 0,
    "CpusetCpus": "",
    "CpusetMems": "",
    "Devices": null,
    "DeviceCgroupRules": null,
    "DeviceRequests": null,
    "MemoryReservation": 0,
    "MemorySwap": 0,
    "MemorySwappiness": null,
    "OomKillDisable": false,
    "PidsLimit": null,
    "Ulimits": [
      {
        "Name": "nofile",
        "Hard": 65536,
        "Soft": 32768
      }
    ],
    "CpuCount": 0,
    "CpuPercent": 0,
    "IOMaximumIOps": 0,
    "IOMaximumBandwidth": 0,
    "MaskedPaths": null,
    "ReadonlyPaths": null
  },
  "GraphDriver": {
    "Data": {
      "LowerDir": "/var/lib/docker/overlay2/39d39dd007d86dcf145ed874c03d976bff0360557acf8638e3d6b432d6335791-init/diff:/var/lib/docker/overlay2/a32fa2ed08396f5c190b57116bfba5c9c8ee272b5134ad0e1cdbd9b1bcda2457/diff:/var/lib/docker/overlay2/9f661ea3baff676629379664f0f56a489acf37412ee98b1626996fc3147cc992/diff:/var/lib/docker/overlay2/650112fbe79d9a86b45e3e8dcbb90d40f37aa0f26693f5c8b7764ffe560b1de6/diff",
      "MergedDir": "/var/lib/docker/overlay2/39d39dd007d86dcf145ed874c03d976bff0360557acf8638e3d6b432d6335791/merged",
      "UpperDir": "/var/lib/docker/overlay2/39d39dd007d86dcf145ed874c03d976bff0360557acf8638e3d6b432d6335791/diff",
      "WorkDir": "/var/lib/docker/overlay2/39d39dd007d86dcf145ed874c03d976bff0360557acf8638e3d6b432d6335791/work"
    },
    "Name": "overlay2"
  },
  "Mounts": [
    {
      "Type": "bind",
      "Source": "/var/lib/ecs/deps/execute-command/bin/3.3.859.0/ssm-session-worker",
      "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/ssm-session-worker",
      "Mode": "ro",
      "RW": false,
      "Propagation": "rprivate"
    },
    {
      "Type": "bind",
      "Source": "/var/lib/ecs/deps/execute-command/config/amazon-ssm-agent-Orvj12YkCf4DKDu1cHTOVj7smDviWx1T4Kg3Q_IdNYA=.json",
      "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/configuration/amazon-ssm-agent.json",
      "Mode": "ro",
      "RW": false,
      "Propagation": "rprivate"
    },
    {
      "Type": "bind",
      "Source": "/var/lib/ecs/deps/execute-command/config/seelog-gEZ-TIvHAyOLfMC5wiWRofgDMlDzaCZ6zcswnAoop84=.xml",
      "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/configuration/seelog.xml",
      "Mode": "ro",
      "RW": false,
      "Propagation": "rprivate"
    },
    {
      "Type": "bind",
      "Source": "/var/lib/ecs/deps/execute-command/certs/tls-ca-bundle.pem",
      "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/certs/amazon-ssm-agent.crt",
      "Mode": "ro",
      "RW": false,
      "Propagation": "rprivate"
    },
    {
      "Type": "bind",
      "Source": "/var/log/ecs/exec/5011fa3ce88d471baa4c0c3fe1303f44/agent",
      "Destination": "/var/log/amazon/ssm",
      "Mode": "",
      "RW": true,
      "Propagation": "rprivate"
    },
    {
      "Type": "volume",
      "Name": "ecs-m-dev-netbird-netbird-agent-10-netbird-client-94a993e3f4f9a3ad4100",
      "Source": "",
      "Destination": "/etc/netbird",
      "Driver": "amazon-ecs-volume-plugin",
      "Mode": "",
      "RW": true,
      "Propagation": ""
    },
    {
      "Type": "bind",
      "Source": "/var/lib/ecs/deps/execute-command/bin/3.3.859.0/amazon-ssm-agent",
      "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/amazon-ssm-agent",
      "Mode": "ro",
      "RW": false,
      "Propagation": "rprivate"
    },
    {
      "Type": "bind",
      "Source": "/var/lib/ecs/deps/execute-command/bin/3.3.859.0/ssm-agent-worker",
      "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/ssm-agent-worker",
      "Mode": "ro",
      "RW": false,
      "Propagation": "rprivate"
    }
  ],
  "Config": {
    "Hostname": "017bd520a7fc",
    "Domainname": "",
    "User": "root",
    "AttachStdin": false,
    "AttachStdout": false,
    "AttachStderr": false,
    "Tty": false,
    "OpenStdin": false,
    "StdinOnce": false,
    "Env": [
      "NB_SETUP_KEY=REDACTED",
      "AWS_EXECUTION_ENV=AWS_ECS_EC2",
      "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=",
      "ECS_CONTAINER_METADATA_URI=",
      "ECS_CONTAINER_METADATA_URI_V4=",
      "ECS_AGENT_URI=",
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
      "NB_FOREGROUND_MODE=true"
    ],
    "Cmd": null,
    "Image": "netbirdio/netbird:latest@sha256:c484d8665fa9f6250a3f19e6e7f3756ae853218039042d3d79b79cb651bfc0f9",
    "Volumes": null,
    "WorkingDir": "/",
    "Entrypoint": ["/usr/local/bin/netbird", "up"],
    "OnBuild": null,
    "Labels": {
      "com.amazonaws.ecs.cluster": "m-dev-netbird",
      "com.amazonaws.ecs.container-name": "agent",
      "com.amazonaws.ecs.task-arn": "REDACTED",
      "com.amazonaws.ecs.task-definition-family": "m-dev-netbird-netbird-agent",
      "com.amazonaws.ecs.task-definition-version": "10",
      "maintainer": "dev@netbird.io",
      "org.opencontainers.image.created": "2024-12-05T15:08:45Z",
      "org.opencontainers.image.revision": "ff330e644e2bd9c6658fdc1daae0798886e32adb",
      "org.opencontainers.image.title": "netbird",
      "org.opencontainers.image.version": "0.34.1"
    }
  },
  "NetworkSettings": {
    "Bridge": "",
    "SandboxID": "39d6942086c3d0928e41e435c81082feb318d3e241959e28c83745d579b61893",
    "SandboxKey": "/var/run/docker/netns/39d6942086c3",
    "Ports": {},
    "HairpinMode": false,
    "LinkLocalIPv6Address": "",
    "LinkLocalIPv6PrefixLen": 0,
    "SecondaryIPAddresses": null,
    "SecondaryIPv6Addresses": null,
    "EndpointID": "8d036f24774293bb0581e3e880b0c81b89b094e74cca707fa2050b5a346fb335",
    "Gateway": "172.17.0.1",
    "GlobalIPv6Address": "",
    "GlobalIPv6PrefixLen": 0,
    "IPAddress": "172.17.0.2",
    "IPPrefixLen": 16,
    "IPv6Gateway": "",
    "MacAddress": "02:42:ac:11:00:02",
    "Networks": {
      "bridge": {
        "IPAMConfig": null,
        "Links": null,
        "Aliases": null,
        "MacAddress": "02:42:ac:11:00:02",
        "NetworkID": "d3617a47f0d897a952fa9d822903ae82b4ef19f1b30b959cf49486a43d35dcb1",
        "EndpointID": "8d036f24774293bb0581e3e880b0c81b89b094e74cca707fa2050b5a346fb335",
        "Gateway": "172.17.0.1",
        "IPAddress": "172.17.0.2",
        "IPPrefixLen": 16,
        "IPv6Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "DriverOpts": null,
        "DNSNames": null
      }
    }
  }
}
]
Originally created by @hidemire on GitHub (Dec 17, 2024). **Describe the problem** When using Netbird to set up VPN access to a VPC network (`10.10.0.0/16`), DNS queries initiated from the client (e.g., `100.103.197.210`) are sent to the DNS server inside the VPC (`10.10.0.2`) via a Netbird agent deployed within an ECS task on EC2. However, the DNS responses do not reach the client. ![image](https://github.com/user-attachments/assets/fb4aba98-12c7-4e62-9225-628d8d910080) **Expected behavior** The DNS response from `10.10.0.2` should reach the originating client (`100.103.197.210`) via the Netbird agent. **Are you using NetBird Cloud?** - Yes **NetBird version** agent - `0.34.1` macos client - `0.34.1` <details> <summary> <b>NetBird status -dA output (client)</b> </summary> ``` Peers detail: e35c8306ac1c.netbird.cloud: NetBird IP: 100.103.237.110 Public key: ZU9MOESxT0cRY54EoXRcJwEDACPv4SY42UPDM2uv8zg= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): host/prflx ICE candidate endpoints (Local/Remote): 192.168.64.1:51820/198.51.100.0:12013 Relay server address: rels://streamline-de-fra1-1.relay.netbird.io:443 Last connection update: 22 minutes, 40 seconds ago Last WireGuard handshake: 1 minute, 33 seconds ago Transfer status (received/sent) 3.7 KiB/4.1 KiB Quantum resistance: false Routes: 10.10.0.0/16 Latency: 127.600875ms OS: darwin/arm64 Daemon version: 0.34.1 CLI version: 0.34.1 Management: Connected to https://api.netbird.io:443 Signal: Connected to https://signal.netbird.io:443 Relays: [stun:stun.netbird.io:5555] is Available [turns:turn.netbird.io:443?transport=tcp] is Available [rels://streamline-de-fra1-1.relay.netbird.io:443] is Available Nameservers: FQDN: w22031.netbird.cloud NetBird IP: 100.103.197.210/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 1/1 Connected ``` </details> <details> <summary> <b>tcpdump -i any -vvv -n port 53 (agent)</b> </summary> The client executes the command: `dig @10.10.0.2 dagster.dev.dagster.local` ``` tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 21:42:27.709905 wt0 In IP (tos 0x0, ttl 64, id 62910, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 21:42:27.709940 eth0 Out IP (tos 0x0, ttl 63, id 62910, offset 0, flags [none], proto UDP (17), length 82) 172.17.0.2.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 21:42:27.712150 eth0 In IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98) 10.10.0.2.53 > 172.17.0.2.59642: [udp sum ok] 19638 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [1m] A 10.10.87.201 ar: . OPT UDPsize=4096 (70) 21:42:32.619758 wt0 In IP (tos 0x0, ttl 64, id 52575, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 21:42:32.619772 eth0 Out IP (tos 0x0, ttl 63, id 52575, offset 0, flags [none], proto UDP (17), length 82) 172.17.0.2.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 21:42:32.619976 eth0 In IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98) 10.10.0.2.53 > 172.17.0.2.59642: [udp sum ok] 19638 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [55s] A 10.10.87.201 ar: . OPT UDPsize=4096 (70) 21:42:37.663749 wt0 In IP (tos 0x0, ttl 64, id 26739, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 21:42:37.663761 eth0 Out IP (tos 0x0, ttl 63, id 26739, offset 0, flags [none], proto UDP (17), length 82) 172.17.0.2.59642 > 10.10.0.2.53: [udp sum ok] 19638+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 21:42:37.663993 eth0 In IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98) 10.10.0.2.53 > 172.17.0.2.59642: [udp sum ok] 19638 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [50s] A 10.10.87.201 ar: . OPT UDPsize=4096 (70) ``` </details> <details> <summary> <b>iptables-save (agent)</b> </summary> ``` # Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 17 21:43:44 2024 *mangle :PREROUTING ACCEPT [3279:3135055] :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :NETBIRD-RT-PRE - [0:0] -A PREROUTING -i wt0 -m addrtype --dst-type LOCAL -j MARK --set-xmark 0x1bd01/0xffffffff -A PREROUTING -j NETBIRD-RT-PRE -A NETBIRD-RT-PRE -d 10.10.0.0/16 -i wt0 -m conntrack --ctstate NEW -j MARK --set-xmark 0x1bd11/0xffffffff -A NETBIRD-RT-PRE -s 10.10.0.0/16 ! -i wt0 -m conntrack --ctstate NEW -j MARK --set-xmark 0x1bd12/0xffffffff COMMIT # Completed on Tue Dec 17 21:43:44 2024 # Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 17 21:43:44 2024 *filter :INPUT ACCEPT [3265:3133907] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [2814:283965] :NETBIRD-ACL-INPUT - [0:0] :NETBIRD-ACL-OUTPUT - [0:0] :NETBIRD-RT-FWD - [0:0] -A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A INPUT -i wt0 -j NETBIRD-ACL-INPUT -A INPUT -i wt0 -j DROP -A FORWARD -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -m mark --mark 0x1bd01 -j NETBIRD-ACL-INPUT -A FORWARD -i wt0 -j NETBIRD-RT-FWD -A FORWARD -i wt0 -j DROP -A OUTPUT -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A OUTPUT ! -d 100.103.0.0/16 -o wt0 -j ACCEPT -A OUTPUT -o wt0 -j NETBIRD-ACL-OUTPUT -A OUTPUT -o wt0 -j DROP -A NETBIRD-ACL-INPUT -j ACCEPT -A NETBIRD-ACL-OUTPUT -j ACCEPT -A NETBIRD-RT-FWD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A NETBIRD-RT-FWD -d 10.10.0.0/16 -j ACCEPT COMMIT # Completed on Tue Dec 17 21:43:44 2024 # Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 17 21:43:44 2024 *nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [48:3742] :NETBIRD-RT-NAT - [0:0] -A POSTROUTING -j NETBIRD-RT-NAT -A NETBIRD-RT-NAT ! -o lo -m mark --mark 0x1bd11 -j MASQUERADE -A NETBIRD-RT-NAT -o wt0 -m mark --mark 0x1bd12 -j MASQUERADE COMMIT # Completed on Tue Dec 17 21:43:44 2024 ``` </details> <details> <summary> <b>nft list ruleset (agent)</b> </summary> ``` # Warning: table ip filter is managed by iptables-nft, do not touch! table ip filter { chain NETBIRD-RT-FWD { xt match "conntrack" counter packets 8 bytes 656 accept ip daddr 10.10.0.0/16 counter packets 4 bytes 328 accept } chain NETBIRD-ACL-INPUT { counter packets 0 bytes 0 accept } chain NETBIRD-ACL-OUTPUT { counter packets 0 bytes 0 accept } chain INPUT { type filter hook input priority filter; policy accept; iifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept iifname "wt0" counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT iifname "wt0" counter packets 0 bytes 0 drop } chain OUTPUT { type filter hook output priority filter; policy accept; oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept ip daddr != 100.103.0.0/16 oifname "wt0" counter packets 0 bytes 0 accept oifname "wt0" counter packets 0 bytes 0 jump NETBIRD-ACL-OUTPUT oifname "wt0" counter packets 0 bytes 0 drop } chain FORWARD { type filter hook forward priority filter; policy accept; oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept meta mark 0x0001bd01 counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT iifname "wt0" counter packets 14 bytes 1148 jump NETBIRD-RT-FWD iifname "wt0" counter packets 0 bytes 0 drop } } # Warning: table ip mangle is managed by iptables-nft, do not touch! table ip mangle { chain NETBIRD-RT-PRE { ip daddr 10.10.0.0/16 iifname "wt0" xt match "conntrack" counter packets 4 bytes 328 xt target "MARK" ip saddr 10.10.0.0/16 iifname != "wt0" xt match "conntrack" counter packets 0 bytes 0 xt target "MARK" } chain PREROUTING { type filter hook prerouting priority mangle; policy accept; iifname "wt0" xt match "addrtype" counter packets 0 bytes 0 xt target "MARK" counter packets 3580 bytes 4062979 jump NETBIRD-RT-PRE } } # Warning: table ip nat is managed by iptables-nft, do not touch! table ip nat { chain NETBIRD-RT-NAT { oifname != "lo" meta mark 0x0001bd11 counter packets 6 bytes 492 xt target "MASQUERADE" oifname "wt0" meta mark 0x0001bd12 counter packets 0 bytes 0 xt target "MASQUERADE" } chain POSTROUTING { type nat hook postrouting priority srcnat; policy accept; counter packets 64 bytes 4916 jump NETBIRD-RT-NAT } } ``` </details> <details> <summary> <b>ip route (agent)</b> </summary> ``` default via 172.17.0.1 dev eth0 100.103.0.0/16 dev wt0 scope link src 100.103.237.110 172.17.0.0/16 dev eth0 scope link src 172.17.0.2 ``` </details> <details> <summary> <b>ECS container definitions</b> </summary> ``` { "containerDefinitions": [ { "name": "agent", "image": "netbirdio/netbird:latest", "cpu": 0, "portMappings": [], "essential": true, "environment": [], "mountPoints": [ { "sourceVolume": "netbird-client", "containerPath": "/etc/netbird" } ], "volumesFrom": [], "linuxParameters": { "capabilities": { "add": ["NET_ADMIN", "SYS_ADMIN", "SYS_RESOURCE"], "drop": [] } }, "secrets": [ { "name": "NB_SETUP_KEY", "valueFrom": "REDACTED" } ], "user": "root", "privileged": true, "logConfiguration": { "logDriver": "awslogs", "options": { "awslogs-group": "redacted", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "agent" } }, "systemControls": [] } ] } ``` </details> <details> <summary> <b>docker container inspect agent-container</b> </summary> ``` [ { "Id": "017bd520a7fc54ab502a1f607c0954ef261b0b8d13dfb4ed864c2e13cf5a9bc5", "Created": "2024-12-17T21:23:50.424260748Z", "Path": "/usr/local/bin/netbird", "Args": ["up"], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 1709, "ExitCode": 0, "Error": "", "StartedAt": "2024-12-17T21:23:53.711975634Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:bf0d5410a49f95db023add30a5c1da4260e3017fb769595ecf90ce65f527a7af", "ResolvConfPath": "/var/lib/docker/containers/017bd520a7fc54ab502a1f607c0954ef261b0b8d13dfb4ed864c2e13cf5a9bc5/resolv.conf", "HostnamePath": "/var/lib/docker/containers/017bd520a7fc54ab502a1f607c0954ef261b0b8d13dfb4ed864c2e13cf5a9bc5/hostname", "HostsPath": "/var/lib/docker/containers/017bd520a7fc54ab502a1f607c0954ef261b0b8d13dfb4ed864c2e13cf5a9bc5/hosts", "LogPath": "", "Name": "/ecs-m-dev-netbird-netbird-agent-10-agent-be8ba0ba90849983ce01", "RestartCount": 0, "Driver": "overlay2", "Platform": "linux", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "HostConfig": { "Binds": [ "ecs-m-dev-netbird-netbird-agent-10-netbird-client-94a993e3f4f9a3ad4100:/etc/netbird", "redacted", "redacted", "redacted", "redacted", "redacted", "redacted", "redacted" ], "ContainerIDFile": "", "NetworkMode": "bridge", "PortBindings": {}, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": [], "ConsoleSize": [0, 0], "CapAdd": ["NET_ADMIN", "SYS_ADMIN", "SYS_RESOURCE"], "CapDrop": [], "CgroupnsMode": "host", "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "shareable", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": true, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": ["label=disable"], "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "Runtime": "runc", "Isolation": "", "CpuShares": 2, "Memory": 0, "NanoCpus": 0, "CgroupParent": "/ecs/5011fa3ce88d471baa4c0c3fe1303f44", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": null, "DeviceCgroupRules": null, "DeviceRequests": null, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": false, "PidsLimit": null, "Ulimits": [ { "Name": "nofile", "Hard": 65536, "Soft": 32768 } ], "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "GraphDriver": { "Data": { "LowerDir": "/var/lib/docker/overlay2/39d39dd007d86dcf145ed874c03d976bff0360557acf8638e3d6b432d6335791-init/diff:/var/lib/docker/overlay2/a32fa2ed08396f5c190b57116bfba5c9c8ee272b5134ad0e1cdbd9b1bcda2457/diff:/var/lib/docker/overlay2/9f661ea3baff676629379664f0f56a489acf37412ee98b1626996fc3147cc992/diff:/var/lib/docker/overlay2/650112fbe79d9a86b45e3e8dcbb90d40f37aa0f26693f5c8b7764ffe560b1de6/diff", "MergedDir": "/var/lib/docker/overlay2/39d39dd007d86dcf145ed874c03d976bff0360557acf8638e3d6b432d6335791/merged", "UpperDir": "/var/lib/docker/overlay2/39d39dd007d86dcf145ed874c03d976bff0360557acf8638e3d6b432d6335791/diff", "WorkDir": "/var/lib/docker/overlay2/39d39dd007d86dcf145ed874c03d976bff0360557acf8638e3d6b432d6335791/work" }, "Name": "overlay2" }, "Mounts": [ { "Type": "bind", "Source": "/var/lib/ecs/deps/execute-command/bin/3.3.859.0/ssm-session-worker", "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/ssm-session-worker", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/ecs/deps/execute-command/config/amazon-ssm-agent-Orvj12YkCf4DKDu1cHTOVj7smDviWx1T4Kg3Q_IdNYA=.json", "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/configuration/amazon-ssm-agent.json", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/ecs/deps/execute-command/config/seelog-gEZ-TIvHAyOLfMC5wiWRofgDMlDzaCZ6zcswnAoop84=.xml", "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/configuration/seelog.xml", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/ecs/deps/execute-command/certs/tls-ca-bundle.pem", "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/certs/amazon-ssm-agent.crt", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/log/ecs/exec/5011fa3ce88d471baa4c0c3fe1303f44/agent", "Destination": "/var/log/amazon/ssm", "Mode": "", "RW": true, "Propagation": "rprivate" }, { "Type": "volume", "Name": "ecs-m-dev-netbird-netbird-agent-10-netbird-client-94a993e3f4f9a3ad4100", "Source": "", "Destination": "/etc/netbird", "Driver": "amazon-ecs-volume-plugin", "Mode": "", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "/var/lib/ecs/deps/execute-command/bin/3.3.859.0/amazon-ssm-agent", "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/amazon-ssm-agent", "Mode": "ro", "RW": false, "Propagation": "rprivate" }, { "Type": "bind", "Source": "/var/lib/ecs/deps/execute-command/bin/3.3.859.0/ssm-agent-worker", "Destination": "/ecs-execute-command-daba431c-059a-4942-b1dd-3c5f80aa8bf3/ssm-agent-worker", "Mode": "ro", "RW": false, "Propagation": "rprivate" } ], "Config": { "Hostname": "017bd520a7fc", "Domainname": "", "User": "root", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "NB_SETUP_KEY=REDACTED", "AWS_EXECUTION_ENV=AWS_ECS_EC2", "AWS_CONTAINER_CREDENTIALS_RELATIVE_URI=", "ECS_CONTAINER_METADATA_URI=", "ECS_CONTAINER_METADATA_URI_V4=", "ECS_AGENT_URI=", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", "NB_FOREGROUND_MODE=true" ], "Cmd": null, "Image": "netbirdio/netbird:latest@sha256:c484d8665fa9f6250a3f19e6e7f3756ae853218039042d3d79b79cb651bfc0f9", "Volumes": null, "WorkingDir": "/", "Entrypoint": ["/usr/local/bin/netbird", "up"], "OnBuild": null, "Labels": { "com.amazonaws.ecs.cluster": "m-dev-netbird", "com.amazonaws.ecs.container-name": "agent", "com.amazonaws.ecs.task-arn": "REDACTED", "com.amazonaws.ecs.task-definition-family": "m-dev-netbird-netbird-agent", "com.amazonaws.ecs.task-definition-version": "10", "maintainer": "dev@netbird.io", "org.opencontainers.image.created": "2024-12-05T15:08:45Z", "org.opencontainers.image.revision": "ff330e644e2bd9c6658fdc1daae0798886e32adb", "org.opencontainers.image.title": "netbird", "org.opencontainers.image.version": "0.34.1" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "39d6942086c3d0928e41e435c81082feb318d3e241959e28c83745d579b61893", "SandboxKey": "/var/run/docker/netns/39d6942086c3", "Ports": {}, "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "8d036f24774293bb0581e3e880b0c81b89b094e74cca707fa2050b5a346fb335", "Gateway": "172.17.0.1", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "172.17.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "MacAddress": "02:42:ac:11:00:02", "Networks": { "bridge": { "IPAMConfig": null, "Links": null, "Aliases": null, "MacAddress": "02:42:ac:11:00:02", "NetworkID": "d3617a47f0d897a952fa9d822903ae82b4ef19f1b30b959cf49486a43d35dcb1", "EndpointID": "8d036f24774293bb0581e3e880b0c81b89b094e74cca707fa2050b5a346fb335", "Gateway": "172.17.0.1", "IPAddress": "172.17.0.2", "IPPrefixLen": 16, "IPv6Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "DriverOpts": null, "DNSNames": null } } } } ] ``` </details>
saavagebueno added the routes label 2025-11-20 05:31:48 -05:00
saavagebueno commented 2025-11-20 05:31:49 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Dec 18, 2024):

Hi @hidemire,

which OS/kernel does the router run?

can you provide conntrack -L | grep 10.10.0.2 after the request?

@lixmal commented on GitHub (Dec 18, 2024): Hi @hidemire, which OS/kernel does the router run? can you provide `conntrack -L | grep 10.10.0.2` after the request?
saavagebueno commented 2025-11-20 05:31:49 -05:00
Author
Owner
Copy Link

@hidemire commented on GitHub (Dec 18, 2024):

@lixmal

uname -a

Linux ip-10-10-47-152.ec2.internal 4.14.355-271.569.amzn2.aarch64 #1 SMP Tue Nov 5 10:11:28 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

cat /etc/os-release (EC2)

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"

cat /etc/os-release (agent container)

NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.20.3
PRETTY_NAME="Alpine Linux v3.20"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"

conntrack -L | grep 10.10.0.2

conntrack v1.4.8 (conntrack-tools): 7 flow entries have been shown.
udp      17 172 src=100.103.197.210 dst=10.10.0.2 sport=62889 dport=53 src=10.10.0.2 dst=172.17.0.2 sport=53 dport=62889 [ASSURED] mark=0 use=1
@hidemire commented on GitHub (Dec 18, 2024): @lixmal **uname -a** ```Linux ip-10-10-47-152.ec2.internal 4.14.355-271.569.amzn2.aarch64 #1 SMP Tue Nov 5 10:11:28 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux``` **cat /etc/os-release (EC2)** ``` NAME="Amazon Linux" VERSION="2" ID="amzn" ID_LIKE="centos rhel fedora" VERSION_ID="2" PRETTY_NAME="Amazon Linux 2" ANSI_COLOR="0;33" CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2" HOME_URL="https://amazonlinux.com/" SUPPORT_END="2025-06-30" ``` **cat /etc/os-release (agent container)** ``` NAME="Alpine Linux" ID=alpine VERSION_ID=3.20.3 PRETTY_NAME="Alpine Linux v3.20" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues" ``` **conntrack -L | grep 10.10.0.2** ``` conntrack v1.4.8 (conntrack-tools): 7 flow entries have been shown. udp 17 172 src=100.103.197.210 dst=10.10.0.2 sport=62889 dport=53 src=10.10.0.2 dst=172.17.0.2 sport=53 dport=62889 [ASSURED] mark=0 use=1 ```
saavagebueno commented 2025-11-20 05:31:49 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Dec 24, 2024):

Could you try client version 0.31.0 please? 0.31.1+ onward uses a slightly different way for NAT which might fail here.

@lixmal commented on GitHub (Dec 24, 2024): Could you try client version 0.31.0 please? 0.31.1+ onward uses a slightly different way for NAT which might fail here.
saavagebueno commented 2025-11-20 05:31:49 -05:00
Author
Owner
Copy Link

@hidemire commented on GitHub (Dec 24, 2024):

After changing the client and agent versions to 0.31.0, the tcpdump output has changed slightly, the request for 10.10.0.2.53 does not even occur, and conntrack shows UNREPLIED.

iptables-save (agent) 
# Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 24 11:20:00 2024
*mangle
:PREROUTING ACCEPT [2506:6226908]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -i wt0 -m addrtype --dst-type LOCAL -j MARK --set-xmark 0x1bd01/0xffffffff
COMMIT
# Completed on Tue Dec 24 11:20:00 2024
# Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 24 11:20:00 2024
*filter
:INPUT ACCEPT [2493:6225782]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2160:212046]
:NETBIRD-ACL-INPUT - [0:0]
:NETBIRD-ACL-OUTPUT - [0:0]
:NETBIRD-RT-FWD - [0:0]
-A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wt0 -j NETBIRD-ACL-INPUT
-A INPUT -i wt0 -j DROP
-A FORWARD -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m mark --mark 0x1bd01 -j NETBIRD-ACL-INPUT
-A FORWARD -i wt0 -j NETBIRD-RT-FWD
-A FORWARD -i wt0 -j DROP
-A OUTPUT -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT ! -d 100.103.0.0/16 -o wt0 -j ACCEPT
-A OUTPUT -o wt0 -j NETBIRD-ACL-OUTPUT
-A OUTPUT -o wt0 -j DROP
-A NETBIRD-ACL-INPUT -j ACCEPT
-A NETBIRD-ACL-OUTPUT -j ACCEPT
-A NETBIRD-RT-FWD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A NETBIRD-RT-FWD -d 10.10.0.0/16 -j ACCEPT
COMMIT
# Completed on Tue Dec 24 11:20:00 2024
# Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 24 11:20:00 2024
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [60:4494]
:NETBIRD-RT-NAT - [0:0]
-A POSTROUTING -j NETBIRD-RT-NAT
-A NETBIRD-RT-NAT -d 10.10.0.0/16 -i wt0 ! -o lo -j MASQUERADE
-A NETBIRD-RT-NAT -s 10.10.0.0/16 ! -i lo -o wt0 -j MASQUERADE
COMMIT
# Completed on Tue Dec 24 11:20:00 2024
nft list ruleset (agent) 
# Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
        chain NETBIRD-RT-FWD {
                xt match "conntrack" counter packets 0 bytes 0 accept
                ip daddr 10.10.0.0/16 counter packets 15 bytes 1230 accept
        }

        chain NETBIRD-ACL-INPUT {
                counter packets 0 bytes 0 accept
        }

        chain NETBIRD-ACL-OUTPUT {
                counter packets 0 bytes 0 accept
        }

        chain INPUT {
                type filter hook input priority filter; policy accept;
                iifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept
                iifname "wt0" counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT
                iifname "wt0" counter packets 0 bytes 0 drop
        }

        chain OUTPUT {
                type filter hook output priority filter; policy accept;
                oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept
                ip daddr != 100.103.0.0/16 oifname "wt0" counter packets 0 bytes 0 accept
                oifname "wt0" counter packets 0 bytes 0 jump NETBIRD-ACL-OUTPUT
                oifname "wt0" counter packets 0 bytes 0 drop
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept
                meta mark 0x0001bd01 counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT
                iifname "wt0" counter packets 15 bytes 1230 jump NETBIRD-RT-FWD
                iifname "wt0" counter packets 0 bytes 0 drop
        }
}
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
        chain NETBIRD-RT-NAT {
                ip daddr 10.10.0.0/16 iifname "wt0" oifname != "lo" counter packets 0 bytes 0 xt target "MASQUERADE"
                ip saddr 10.10.0.0/16 iifname != "lo" oifname "wt0" counter packets 0 bytes 0 xt target "MASQUERADE"
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                counter packets 62 bytes 4622 jump NETBIRD-RT-NAT
        }
}
# Warning: table ip mangle is managed by iptables-nft, do not touch!
table ip mangle {
        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
                iifname "wt0" xt match "addrtype" counter packets 0 bytes 0 xt target "MARK"
        }
}
tcpdump -i any -vvv -n port 53 (agent) 
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:14:23.217455 wt0   In  IP (tos 0x0, ttl 64, id 59026, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
11:14:23.217477 eth0  Out IP (tos 0x0, ttl 63, id 59026, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
11:14:28.222397 wt0   In  IP (tos 0x0, ttl 64, id 53419, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
11:14:28.222416 eth0  Out IP (tos 0x0, ttl 63, id 53419, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
11:14:33.226637 wt0   In  IP (tos 0x0, ttl 64, id 3217, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
11:14:33.226653 eth0  Out IP (tos 0x0, ttl 63, id 3217, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
conntrack -L | grep 10.10.0.2 
conntrack v1.4.8 (conntrack-tools): 5 flow entries have been shown.
udp      17 26 src=100.103.197.210 dst=10.10.0.2 sport=53659 dport=53 [UNREPLIED] src=10.10.0.2 dst=100.103.197.210 sport=53 dport=53659 mark=0 use=1
NetBird status -dA output (client) 
Peers detail:
 e35c8306ac1c.netbird.cloud:
  NetBird IP: 100.103.237.110
  Public key: ZU9MOESxT0cRY54EoXRcJwEDACPv4SY42UPDM2uv8zg=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:19103
  Relay server address: rels://streamline-de-fra1-2.relay.netbird.io:443
  Last connection update: 4 minutes, 1 seconds ago
  Last WireGuard handshake: 1 minute, 55 seconds ago
  Transfer status (received/sent) 1.3 KiB/1.8 KiB
  Quantum resistance: false
  Routes: 10.10.0.0/16
  Latency: 128.034458ms

OS: darwin/arm64
Daemon version: 0.31.0
CLI version: 0.31.0
Management: Connected to https://api.netbird.io:443
Signal: Connected to https://signal.netbird.io:443
Relays:
  [stun:stun.netbird.io:5555] is Available
  [turns:turn.netbird.io:443?transport=tcp] is Available
  [rels://streamline-de-fra1-2.relay.netbird.io:443] is Available
Nameservers:
FQDN: w22031.netbird.cloud
NetBird IP: 100.103.197.210/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 1/1 Connected

For version 0.35.0, the tcpdump and conntrack output is the same as for 0.34.1.
I also tried replacing the arm64 instance with a x64 one, but it didn't help.

@hidemire commented on GitHub (Dec 24, 2024): After changing the client and agent versions to 0.31.0, the tcpdump output has changed slightly, the request for 10.10.0.2.53 does not even occur, and conntrack shows UNREPLIED. <details> <summary> <b>iptables-save (agent)</b> </summary> ``` # Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 24 11:20:00 2024 *mangle :PREROUTING ACCEPT [2506:6226908] :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] -A PREROUTING -i wt0 -m addrtype --dst-type LOCAL -j MARK --set-xmark 0x1bd01/0xffffffff COMMIT # Completed on Tue Dec 24 11:20:00 2024 # Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 24 11:20:00 2024 *filter :INPUT ACCEPT [2493:6225782] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [2160:212046] :NETBIRD-ACL-INPUT - [0:0] :NETBIRD-ACL-OUTPUT - [0:0] :NETBIRD-RT-FWD - [0:0] -A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A INPUT -i wt0 -j NETBIRD-ACL-INPUT -A INPUT -i wt0 -j DROP -A FORWARD -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -m mark --mark 0x1bd01 -j NETBIRD-ACL-INPUT -A FORWARD -i wt0 -j NETBIRD-RT-FWD -A FORWARD -i wt0 -j DROP -A OUTPUT -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A OUTPUT ! -d 100.103.0.0/16 -o wt0 -j ACCEPT -A OUTPUT -o wt0 -j NETBIRD-ACL-OUTPUT -A OUTPUT -o wt0 -j DROP -A NETBIRD-ACL-INPUT -j ACCEPT -A NETBIRD-ACL-OUTPUT -j ACCEPT -A NETBIRD-RT-FWD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A NETBIRD-RT-FWD -d 10.10.0.0/16 -j ACCEPT COMMIT # Completed on Tue Dec 24 11:20:00 2024 # Generated by iptables-save v1.8.10 (nf_tables) on Tue Dec 24 11:20:00 2024 *nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [60:4494] :NETBIRD-RT-NAT - [0:0] -A POSTROUTING -j NETBIRD-RT-NAT -A NETBIRD-RT-NAT -d 10.10.0.0/16 -i wt0 ! -o lo -j MASQUERADE -A NETBIRD-RT-NAT -s 10.10.0.0/16 ! -i lo -o wt0 -j MASQUERADE COMMIT # Completed on Tue Dec 24 11:20:00 2024 ``` </details> <details> <summary> <b>nft list ruleset (agent)</b> </summary> ``` # Warning: table ip filter is managed by iptables-nft, do not touch! table ip filter { chain NETBIRD-RT-FWD { xt match "conntrack" counter packets 0 bytes 0 accept ip daddr 10.10.0.0/16 counter packets 15 bytes 1230 accept } chain NETBIRD-ACL-INPUT { counter packets 0 bytes 0 accept } chain NETBIRD-ACL-OUTPUT { counter packets 0 bytes 0 accept } chain INPUT { type filter hook input priority filter; policy accept; iifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept iifname "wt0" counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT iifname "wt0" counter packets 0 bytes 0 drop } chain OUTPUT { type filter hook output priority filter; policy accept; oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept ip daddr != 100.103.0.0/16 oifname "wt0" counter packets 0 bytes 0 accept oifname "wt0" counter packets 0 bytes 0 jump NETBIRD-ACL-OUTPUT oifname "wt0" counter packets 0 bytes 0 drop } chain FORWARD { type filter hook forward priority filter; policy accept; oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept meta mark 0x0001bd01 counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT iifname "wt0" counter packets 15 bytes 1230 jump NETBIRD-RT-FWD iifname "wt0" counter packets 0 bytes 0 drop } } # Warning: table ip nat is managed by iptables-nft, do not touch! table ip nat { chain NETBIRD-RT-NAT { ip daddr 10.10.0.0/16 iifname "wt0" oifname != "lo" counter packets 0 bytes 0 xt target "MASQUERADE" ip saddr 10.10.0.0/16 iifname != "lo" oifname "wt0" counter packets 0 bytes 0 xt target "MASQUERADE" } chain POSTROUTING { type nat hook postrouting priority srcnat; policy accept; counter packets 62 bytes 4622 jump NETBIRD-RT-NAT } } # Warning: table ip mangle is managed by iptables-nft, do not touch! table ip mangle { chain PREROUTING { type filter hook prerouting priority mangle; policy accept; iifname "wt0" xt match "addrtype" counter packets 0 bytes 0 xt target "MARK" } } ``` </details> <details> <summary> <b>tcpdump -i any -vvv -n port 53 (agent)</b> </summary> ``` tcpdump: data link type LINUX_SLL2 tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 11:14:23.217455 wt0 In IP (tos 0x0, ttl 64, id 59026, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 11:14:23.217477 eth0 Out IP (tos 0x0, ttl 63, id 59026, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 11:14:28.222397 wt0 In IP (tos 0x0, ttl 64, id 53419, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 11:14:28.222416 eth0 Out IP (tos 0x0, ttl 63, id 53419, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 11:14:33.226637 wt0 In IP (tos 0x0, ttl 64, id 3217, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 11:14:33.226653 eth0 Out IP (tos 0x0, ttl 63, id 3217, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.52493 > 10.10.0.2.53: [udp sum ok] 1700+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) ``` </details> <details> <summary> <b>conntrack -L | grep 10.10.0.2</b> </summary> ``` conntrack v1.4.8 (conntrack-tools): 5 flow entries have been shown. udp 17 26 src=100.103.197.210 dst=10.10.0.2 sport=53659 dport=53 [UNREPLIED] src=10.10.0.2 dst=100.103.197.210 sport=53 dport=53659 mark=0 use=1 ``` </details> <details> <summary> <b>NetBird status -dA output (client)</b> </summary> ``` Peers detail: e35c8306ac1c.netbird.cloud: NetBird IP: 100.103.237.110 Public key: ZU9MOESxT0cRY54EoXRcJwEDACPv4SY42UPDM2uv8zg= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/prflx ICE candidate endpoints (Local/Remote): 198.51.100.0:51820/198.51.100.1:19103 Relay server address: rels://streamline-de-fra1-2.relay.netbird.io:443 Last connection update: 4 minutes, 1 seconds ago Last WireGuard handshake: 1 minute, 55 seconds ago Transfer status (received/sent) 1.3 KiB/1.8 KiB Quantum resistance: false Routes: 10.10.0.0/16 Latency: 128.034458ms OS: darwin/arm64 Daemon version: 0.31.0 CLI version: 0.31.0 Management: Connected to https://api.netbird.io:443 Signal: Connected to https://signal.netbird.io:443 Relays: [stun:stun.netbird.io:5555] is Available [turns:turn.netbird.io:443?transport=tcp] is Available [rels://streamline-de-fra1-2.relay.netbird.io:443] is Available Nameservers: FQDN: w22031.netbird.cloud NetBird IP: 100.103.197.210/16 Interface type: Userspace Quantum resistance: false Routes: - Peers count: 1/1 Connected ``` </details> For version 0.35.0, the tcpdump and conntrack output is the same as for 0.34.1. I also tried replacing the arm64 instance with a x64 one, but it didn't help.
saavagebueno commented 2025-11-20 05:31:49 -05:00
Author
Owner
Copy Link

@hidemire commented on GitHub (Jan 25, 2025):

Hello. We tried version 0.36.3, the problem is still present. The tcpdump logs have changed since the last time.
Can we provide more information to better understand this problem and solve it?

iptables-save (agent) 
 Generated by iptables-save v1.8.10 (nf_tables) on Sat Jan 25 17:27:12 2025
*mangle
:PREROUTING ACCEPT [4480:7621456]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:NETBIRD-RT-PRE - [0:0]
-A PREROUTING -i wt0 -m addrtype --dst-type LOCAL -j MARK --set-xmark 0x1bd01/0xffffffff
-A PREROUTING -j NETBIRD-RT-PRE
-A NETBIRD-RT-PRE -d 10.10.0.0/16 -i wt0 -m conntrack --ctstate NEW -j MARK --set-xmark 0x1bd11/0xffffffff
-A NETBIRD-RT-PRE -s 10.10.0.0/16 ! -i wt0 -m conntrack --ctstate NEW -j MARK --set-xmark 0x1bd12/0xffffffff
COMMIT
# Completed on Sat Jan 25 17:27:12 2025
# Generated by iptables-save v1.8.10 (nf_tables) on Sat Jan 25 17:27:12 2025
*filter
:INPUT ACCEPT [4389:7614490]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:NETBIRD-ACL-INPUT - [0:0]
:NETBIRD-RT-FWD - [0:0]
-A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i wt0 -j NETBIRD-ACL-INPUT
-A INPUT -i wt0 -j DROP
-A FORWARD -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -m mark --mark 0x1bd01 -j NETBIRD-ACL-INPUT
-A FORWARD -i wt0 -j NETBIRD-RT-FWD
-A FORWARD -i wt0 -j DROP
-A NETBIRD-ACL-INPUT -j ACCEPT
-A NETBIRD-RT-FWD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A NETBIRD-RT-FWD -s 100.103.197.210/32 -d 10.10.0.0/16 -j ACCEPT
COMMIT
# Completed on Sat Jan 25 17:27:12 2025
# Generated by iptables-save v1.8.10 (nf_tables) on Sat Jan 25 17:27:12 2025
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [135:14978]
:NETBIRD-RT-NAT - [0:0]
-A POSTROUTING -j NETBIRD-RT-NAT
-A NETBIRD-RT-NAT ! -o lo -m mark --mark 0x1bd11 -j MASQUERADE
-A NETBIRD-RT-NAT -o wt0 -m mark --mark 0x1bd12 -j MASQUERADE
COMMIT
# Completed on Sat Jan 25 17:27:12 2025
nft list ruleset (agent) 
 Warning: table ip filter is managed by iptables-nft, do not touch!
table ip filter {
        chain NETBIRD-RT-FWD {
                xt match "conntrack" counter packets 19 bytes 1200 accept
                ip saddr 100.103.197.210 ip daddr 10.10.0.0/16 counter packets 16 bytes 1098 accept
        }

        chain NETBIRD-ACL-INPUT {
                counter packets 2 bytes 168 accept
        }

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept
                meta mark 0x0001bd01 counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT
                iifname "wt0" counter packets 59 bytes 4278 jump NETBIRD-RT-FWD
                iifname "wt0" counter packets 24 bytes 1980 drop
        }

        chain INPUT {
                type filter hook input priority filter; policy accept;
                iifname "wt0" xt match "conntrack" counter packets 30 bytes 2520 accept
                iifname "wt0" counter packets 2 bytes 168 jump NETBIRD-ACL-INPUT
                iifname "wt0" counter packets 0 bytes 0 drop
        }
}
# Warning: table ip mangle is managed by iptables-nft, do not touch!
table ip mangle {
        chain NETBIRD-RT-PRE {
                ip daddr 10.10.0.0/16 iifname "wt0" xt match "conntrack" counter packets 16 bytes 1098 xt target "MARK"
                ip saddr 10.10.0.0/16 iifname != "wt0" xt match "conntrack" counter packets 0 bytes 0 xt target "MARK"
        }

        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
                iifname "wt0" xt match "addrtype" counter packets 32 bytes 2688 xt target "MARK"
                counter packets 4543 bytes 7627142 jump NETBIRD-RT-PRE
        }
}
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
        chain NETBIRD-RT-NAT {
                oifname != "lo" meta mark 0x0001bd11 counter packets 14 bytes 934 xt target "MASQUERADE"
                oifname "wt0" meta mark 0x0001bd12 counter packets 0 bytes 0 xt target "MASQUERADE"
        }

        chain POSTROUTING {
                type nat hook postrouting priority srcnat; policy accept;
                counter packets 149 bytes 15912 jump NETBIRD-RT-NAT
        }
}
tcpdump -i any -vvv -n port 53 (agent) 
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
17:28:10.367797 wt0   In  IP (tos 0x0, ttl 64, id 2072, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.53036 > 10.10.0.2.53: [udp sum ok] 25294+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
17:28:10.367825 eth0  Out IP (tos 0x0, ttl 63, id 2072, offset 0, flags [none], proto UDP (17), length 82)
  172.17.0.2.53036 > 10.10.0.2.53: [udp sum ok] 25294+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
17:28:10.369652 eth0  In  IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98)
  10.10.0.2.53 > 172.17.0.2.53036: [udp sum ok] 25294 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [1m] A 10.10.106.26 ar: . OPT UDPsize=4096 (70)
17:28:15.368045 wt0   In  IP (tos 0x0, ttl 64, id 2965, offset 0, flags [none], proto UDP (17), length 82)
  100.103.197.210.53036 > 10.10.0.2.53: [udp sum ok] 25294+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
17:28:15.368059 eth0  Out IP (tos 0x0, ttl 63, id 2965, offset 0, flags [none], proto UDP (17), length 82)
  172.17.0.2.53036 > 10.10.0.2.53: [udp sum ok] 25294+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
17:28:15.368245 eth0  In  IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98)
  10.10.0.2.53 > 172.17.0.2.53036: [udp sum ok] 25294 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [55s] A 10.10.106.26 ar: . OPT UDPsize=4096 (70)
conntrack -L | grep 10.10.0.2 
conntrack v1.4.8 (conntrack-tools): 7 flow entries have been shown.
udp      17 128 src=100.103.197.210 dst=10.10.0.2 sport=53036 dport=53 src=10.10.0.2 dst=172.17.0.2 sport=53 dport=53036 [ASSURED] mark=0 use=1
NetBird status -dA output (client) 
Peers detail:
 e35c8306ac1c.netbird.cloud:
  NetBird IP: 100.103.237.110
  Public key: ZU9MOESxT0cRY54EoXRcJwEDACPv4SY42UPDM2uv8zg=
  Status: Connected
  -- detail --
  Connection type: P2P
  ICE candidate (Local/Remote): srflx/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:38532/198.51.100.1:34969
  Relay server address: rels://streamline-fi-hel1-0.relay.netbird.io:443
  Last connection update: 9 minutes, 22 seconds ago
  Last WireGuard handshake: 1 minute, 32 seconds ago
  Transfer status (received/sent) 2.9 KiB/9.7 KiB
  Quantum resistance: false
  Routes: 10.10.0.0/16
  Networks: 10.10.0.0/16
  Latency: 128.814084ms

OS: darwin/arm64
Daemon version: 0.36.3
CLI version: 0.36.3
Management: Connected to https://api.netbird.io:443
Signal: Connected to https://signal.netbird.io:443
Relays: 
  [stun:stun.netbird.io:5555] is Available
  [turns:turn.netbird.io:443?transport=tcp] is Available
  [rels://streamline-fi-hel1-0.relay.netbird.io:443] is Available
Nameservers: 
FQDN: w22031.netbird.cloud
NetBird IP: 100.103.197.210/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Networks: -
Peers count: 1/1 Connected

P.S. We also tried to run netbird outside of docker using curl -fsSL https://pkgs.netbird.io/install.sh | sh and netbird up but also without success.

@hidemire commented on GitHub (Jan 25, 2025): Hello. We tried version 0.36.3, the problem is still present. The tcpdump logs have changed since the last time. Can we provide more information to better understand this problem and solve it? <details> <summary> <b>iptables-save (agent)</b> </summary> ``` Generated by iptables-save v1.8.10 (nf_tables) on Sat Jan 25 17:27:12 2025 *mangle :PREROUTING ACCEPT [4480:7621456] :INPUT ACCEPT [0:0] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [0:0] :NETBIRD-RT-PRE - [0:0] -A PREROUTING -i wt0 -m addrtype --dst-type LOCAL -j MARK --set-xmark 0x1bd01/0xffffffff -A PREROUTING -j NETBIRD-RT-PRE -A NETBIRD-RT-PRE -d 10.10.0.0/16 -i wt0 -m conntrack --ctstate NEW -j MARK --set-xmark 0x1bd11/0xffffffff -A NETBIRD-RT-PRE -s 10.10.0.0/16 ! -i wt0 -m conntrack --ctstate NEW -j MARK --set-xmark 0x1bd12/0xffffffff COMMIT # Completed on Sat Jan 25 17:27:12 2025 # Generated by iptables-save v1.8.10 (nf_tables) on Sat Jan 25 17:27:12 2025 *filter :INPUT ACCEPT [4389:7614490] :FORWARD ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :NETBIRD-ACL-INPUT - [0:0] :NETBIRD-RT-FWD - [0:0] -A INPUT -i wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A INPUT -i wt0 -j NETBIRD-ACL-INPUT -A INPUT -i wt0 -j DROP -A FORWARD -o wt0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A FORWARD -m mark --mark 0x1bd01 -j NETBIRD-ACL-INPUT -A FORWARD -i wt0 -j NETBIRD-RT-FWD -A FORWARD -i wt0 -j DROP -A NETBIRD-ACL-INPUT -j ACCEPT -A NETBIRD-RT-FWD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A NETBIRD-RT-FWD -s 100.103.197.210/32 -d 10.10.0.0/16 -j ACCEPT COMMIT # Completed on Sat Jan 25 17:27:12 2025 # Generated by iptables-save v1.8.10 (nf_tables) on Sat Jan 25 17:27:12 2025 *nat :PREROUTING ACCEPT [0:0] :INPUT ACCEPT [0:0] :OUTPUT ACCEPT [0:0] :POSTROUTING ACCEPT [135:14978] :NETBIRD-RT-NAT - [0:0] -A POSTROUTING -j NETBIRD-RT-NAT -A NETBIRD-RT-NAT ! -o lo -m mark --mark 0x1bd11 -j MASQUERADE -A NETBIRD-RT-NAT -o wt0 -m mark --mark 0x1bd12 -j MASQUERADE COMMIT # Completed on Sat Jan 25 17:27:12 2025 ``` </details> <details> <summary> <b>nft list ruleset (agent)</b> </summary> ``` Warning: table ip filter is managed by iptables-nft, do not touch! table ip filter { chain NETBIRD-RT-FWD { xt match "conntrack" counter packets 19 bytes 1200 accept ip saddr 100.103.197.210 ip daddr 10.10.0.0/16 counter packets 16 bytes 1098 accept } chain NETBIRD-ACL-INPUT { counter packets 2 bytes 168 accept } chain FORWARD { type filter hook forward priority filter; policy accept; oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept meta mark 0x0001bd01 counter packets 0 bytes 0 jump NETBIRD-ACL-INPUT iifname "wt0" counter packets 59 bytes 4278 jump NETBIRD-RT-FWD iifname "wt0" counter packets 24 bytes 1980 drop } chain INPUT { type filter hook input priority filter; policy accept; iifname "wt0" xt match "conntrack" counter packets 30 bytes 2520 accept iifname "wt0" counter packets 2 bytes 168 jump NETBIRD-ACL-INPUT iifname "wt0" counter packets 0 bytes 0 drop } } # Warning: table ip mangle is managed by iptables-nft, do not touch! table ip mangle { chain NETBIRD-RT-PRE { ip daddr 10.10.0.0/16 iifname "wt0" xt match "conntrack" counter packets 16 bytes 1098 xt target "MARK" ip saddr 10.10.0.0/16 iifname != "wt0" xt match "conntrack" counter packets 0 bytes 0 xt target "MARK" } chain PREROUTING { type filter hook prerouting priority mangle; policy accept; iifname "wt0" xt match "addrtype" counter packets 32 bytes 2688 xt target "MARK" counter packets 4543 bytes 7627142 jump NETBIRD-RT-PRE } } # Warning: table ip nat is managed by iptables-nft, do not touch! table ip nat { chain NETBIRD-RT-NAT { oifname != "lo" meta mark 0x0001bd11 counter packets 14 bytes 934 xt target "MASQUERADE" oifname "wt0" meta mark 0x0001bd12 counter packets 0 bytes 0 xt target "MASQUERADE" } chain POSTROUTING { type nat hook postrouting priority srcnat; policy accept; counter packets 149 bytes 15912 jump NETBIRD-RT-NAT } } ``` </details> <details> <summary> <b>tcpdump -i any -vvv -n port 53 (agent)</b> </summary> ``` tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes 17:28:10.367797 wt0 In IP (tos 0x0, ttl 64, id 2072, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.53036 > 10.10.0.2.53: [udp sum ok] 25294+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 17:28:10.367825 eth0 Out IP (tos 0x0, ttl 63, id 2072, offset 0, flags [none], proto UDP (17), length 82) 172.17.0.2.53036 > 10.10.0.2.53: [udp sum ok] 25294+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 17:28:10.369652 eth0 In IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98) 10.10.0.2.53 > 172.17.0.2.53036: [udp sum ok] 25294 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [1m] A 10.10.106.26 ar: . OPT UDPsize=4096 (70) 17:28:15.368045 wt0 In IP (tos 0x0, ttl 64, id 2965, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.53036 > 10.10.0.2.53: [udp sum ok] 25294+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 17:28:15.368059 eth0 Out IP (tos 0x0, ttl 63, id 2965, offset 0, flags [none], proto UDP (17), length 82) 172.17.0.2.53036 > 10.10.0.2.53: [udp sum ok] 25294+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 17:28:15.368245 eth0 In IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98) 10.10.0.2.53 > 172.17.0.2.53036: [udp sum ok] 25294 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [55s] A 10.10.106.26 ar: . OPT UDPsize=4096 (70) ``` </details> <details> <summary> <b>conntrack -L | grep 10.10.0.2</b> </summary> ``` conntrack v1.4.8 (conntrack-tools): 7 flow entries have been shown. udp 17 128 src=100.103.197.210 dst=10.10.0.2 sport=53036 dport=53 src=10.10.0.2 dst=172.17.0.2 sport=53 dport=53036 [ASSURED] mark=0 use=1 ``` </details> <details> <summary> <b>NetBird status -dA output (client)</b> </summary> ``` Peers detail: e35c8306ac1c.netbird.cloud: NetBird IP: 100.103.237.110 Public key: ZU9MOESxT0cRY54EoXRcJwEDACPv4SY42UPDM2uv8zg= Status: Connected -- detail -- Connection type: P2P ICE candidate (Local/Remote): srflx/prflx ICE candidate endpoints (Local/Remote): 198.51.100.0:38532/198.51.100.1:34969 Relay server address: rels://streamline-fi-hel1-0.relay.netbird.io:443 Last connection update: 9 minutes, 22 seconds ago Last WireGuard handshake: 1 minute, 32 seconds ago Transfer status (received/sent) 2.9 KiB/9.7 KiB Quantum resistance: false Routes: 10.10.0.0/16 Networks: 10.10.0.0/16 Latency: 128.814084ms OS: darwin/arm64 Daemon version: 0.36.3 CLI version: 0.36.3 Management: Connected to https://api.netbird.io:443 Signal: Connected to https://signal.netbird.io:443 Relays: [stun:stun.netbird.io:5555] is Available [turns:turn.netbird.io:443?transport=tcp] is Available [rels://streamline-fi-hel1-0.relay.netbird.io:443] is Available Nameservers: FQDN: w22031.netbird.cloud NetBird IP: 100.103.197.210/16 Interface type: Userspace Quantum resistance: false Routes: - Networks: - Peers count: 1/1 Connected ``` </details> P.S. We also tried to run netbird outside of docker using `curl -fsSL https://pkgs.netbird.io/install.sh | sh` and `netbird up` but also without success.
saavagebueno commented 2025-11-20 05:31:49 -05:00
Author
Owner
Copy Link

@lixmal commented on GitHub (Jan 25, 2025):

Hi @hidemire,

I honestly don't know what could be the issue, everything seems to be in order, except that this counter doesn't go up:

        chain FORWARD {
                type filter hook forward priority filter; policy accept;
                oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept

Have you tried a different Linux image (especially different kernel)?

Another option you could try is switching the iptables backend:

update-alternatives --config iptables

then select iptables-legacy

Also extend your tcpdump for icmp in case there are any unreachables sent out.

@lixmal commented on GitHub (Jan 25, 2025): Hi @hidemire, I honestly don't know what could be the issue, everything seems to be in order, except that this counter doesn't go up: ``` chain FORWARD { type filter hook forward priority filter; policy accept; oifname "wt0" xt match "conntrack" counter packets 0 bytes 0 accept ``` Have you tried a different Linux image (especially different kernel)? Another option you could try is switching the iptables backend: ``` update-alternatives --config iptables ``` then select iptables-legacy Also extend your tcpdump for `icmp` in case there are any `unreachable`s sent out.
saavagebueno commented 2025-11-20 05:31:49 -05:00
Author
Owner
Copy Link

@hidemire commented on GitHub (Jan 28, 2025):

Hi, @lixmal. Thanks for your reply.

After extending tcpdump to include icmp, we saw the unreachable s, but could not figure out what caused it.

tcpdump -i any -vvv -n '(port 53 or icmp)' (agent) 
14:47:29.775276 wt0   In  IP (tos 0x0, ttl 64, id 45159, offset 0, flags [none], proto UDP (17), length 82)
    100.103.197.210.52211 > 10.10.0.2.53: [udp sum ok] 30370+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
14:47:29.775294 eth0  Out IP (tos 0x0, ttl 63, id 45159, offset 0, flags [none], proto UDP (17), length 82)
    172.17.0.2.52211 > 10.10.0.2.53: [udp sum ok] 30370+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54)
14:47:29.775588 eth0  In  IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98)
    10.10.0.2.53 > 172.17.0.2.52211: [udp sum ok] 30370 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [50s] A 10.10.106.26 ar: . OPT UDPsize=4096 (70)
14:47:29.775603 eth0  Out IP (tos 0xc0, ttl 255, id 14512, offset 0, flags [none], proto ICMP (1), length 126)
    172.17.0.2 > 10.10.0.2: ICMP 172.17.0.2 udp port 52211 unreachable, length 106
        IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98)
    10.10.0.2.53 > 172.17.0.2.52211: [udp sum ok] 30370 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [50s] A 10.10.106.26 ar: . OPT UDPsize=4096 (70)

We tried the other version of the kernel (5.10.230-223.885.amzn2.aarch64) and everything worked.

I'm closing the issue, but if you have any additional questions, I'll be happy to answer them.

@hidemire commented on GitHub (Jan 28, 2025): Hi, @lixmal. Thanks for your reply. After extending `tcpdump` to include `icmp`, we saw the `unreachable` s, but could not figure out what caused it. <details> <summary> <b>tcpdump -i any -vvv -n '(port 53 or icmp)' (agent)</b> </summary> ``` 14:47:29.775276 wt0 In IP (tos 0x0, ttl 64, id 45159, offset 0, flags [none], proto UDP (17), length 82) 100.103.197.210.52211 > 10.10.0.2.53: [udp sum ok] 30370+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 14:47:29.775294 eth0 Out IP (tos 0x0, ttl 63, id 45159, offset 0, flags [none], proto UDP (17), length 82) 172.17.0.2.52211 > 10.10.0.2.53: [udp sum ok] 30370+ [1au] A? dagster.dev.dagster.local. ar: . OPT UDPsize=4096 (54) 14:47:29.775588 eth0 In IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98) 10.10.0.2.53 > 172.17.0.2.52211: [udp sum ok] 30370 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [50s] A 10.10.106.26 ar: . OPT UDPsize=4096 (70) 14:47:29.775603 eth0 Out IP (tos 0xc0, ttl 255, id 14512, offset 0, flags [none], proto ICMP (1), length 126) 172.17.0.2 > 10.10.0.2: ICMP 172.17.0.2 udp port 52211 unreachable, length 106 IP (tos 0x0, ttl 254, id 0, offset 0, flags [DF], proto UDP (17), length 98) 10.10.0.2.53 > 172.17.0.2.52211: [udp sum ok] 30370 q: A? dagster.dev.dagster.local. 1/0/1 dagster.dev.dagster.local. [50s] A 10.10.106.26 ar: . OPT UDPsize=4096 (70) ``` </details> We tried the other version of the kernel (`5.10.230-223.885.amzn2.aarch64`) and everything worked. I'm closing the issue, but if you have any additional questions, I'll be happy to answer them.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
2021 Q4
2022 Q1
2022 Q1
accessibility
acl
agent
agent
Android
Android
api
authentik
automation
azure
battery-usage
bug
cache
client
client-ui
cloud
cloud-only
cloudflare
community
compatibility
config-idp
config-issue
connection
contribution
coturn
cross-vpn
dashboard
data-usage
distribution
dns
docker
documentation
duplicate
enhancement
enhancement
event-stream
feature-request
freebsd
getting-started
go
good first issue
gui
help wanted
home-assistant
idp
inconsistency
integration
integrations
ios
ipv6
jwt
k8s
keycloak
linux
login
macos
management-service
missing-docs
mobile
moved-internal
needs-review
netbird-ui
networking
new-platform
nginx
notification
okta
openwrt
packaging
peer-management
peer-management
peer-management
performance
postgres
posture-checks
psk
pull-request
Mirrored from GitHub Pull Request
 question
refactor
relay
release
rfc
routes
security
security-related
self-hosting
server
signal
sleep-issue
ssh
ssl
status
store
synology
system-compatibility-issue
test-suite
third-party-integration
triage
triage-needed
troubleshooting
UX
waiting-feedback
windows
wontfix
zitadel
No Label routes
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
saavagebueno
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SVI/netbird#1502
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?