DNS customizations not applied within kubernetes when vm-type=vz.

[plobsing]

Description

Containers run under Kubernetes cannot resolve host.docker.internal nor custom addresses specified using the --dns-host flag. This only happens within Kubernetes; similar containers running at the Docker layer have no problems resolving these addresses. This only happens when --vm-type=vz is specified; DNS resolution works as expected when --vm-type=qemu is used.

Version

colima version v0.6.7
git commit: ba1be00
WARN[0000] cannot detect current running executable: error looking up 'colima-Darwin-arm64' in PATH: exec: "colima-Darwin-arm64": executable file not found in $PATH
WARN[0000] falling back to first CLI argument

runtime: docker
arch: aarch64
client: v23.0.5
server: v24.0.7

kubernetes
Client Version: v1.26.13
Kustomize Version: v4.5.7
Server Version: v1.28.3+k3s2
limactl version 0.19.1
qemu-img version 8.2.0
Copyright (c) 2003-2023 Fabrice Bellard and the QEMU Project developers

Operating System

• macOS Intel <= 13 (Ventura)
• macOS Intel >= 14 (Sonoma)
• Apple Silicon <= 13 (Ventura)
• Apple Silicon >= 14 (Sonoma)
• Linux

Output of colima status

WARN[0000] cannot detect current running executable: error looking up 'colima-Darwin-arm64' in PATH: exec: "colima-Darwin-arm64": executable file not found in $PATH
WARN[0000] falling back to first CLI argument
INFO[0000] colima is running using macOS Virtualization.Framework
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] address: 192.168.107.2
INFO[0000] socket: unix:///Users/peter/.colima/default/docker.sock
INFO[0000] kubernetes: enabled

Reproduction Steps

1. Start from a fresh state: run colima delete to clear any lingering state. Maybe not necessary, but better to be sure.
2. Start Colima specifying a VZ VM. Optionally include additional DNS hosts.

colima start --arch aarch64 --vm-type=vz --vz-rosetta --mount-type=virtiofs --memory=16 --with-kubernetes --network-address --dns-host 'foo.bar=192.168.5.2'

3. Pull an Alpine image for testing

docker pull alpine:3.19

4. Create a dummy pod we can kubectl-exec into based on the Alpine image

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: alpine
  namespace: default
spec:
  containers:
  - image: alpine:3.19
    command:
      - /bin/sh
      - "-c"
      - "sleep 60m"
    imagePullPolicy: Never
    name: alpine
  restartPolicy: Always
EOF

5. Kubectl-exec into the pod and try resolving addresses.

❮ kube exec -it alpine -- nslookup host.docker.internal
Server:		10.43.0.10
Address:	10.43.0.10:53

** server can't find host.docker.internal: NXDOMAIN

** server can't find host.docker.internal: NXDOMAIN

command terminated with exit code 1
❯ kube exec -it alpine -- nslookup foo.bar
Server:		10.43.0.10
Address:	10.43.0.10:53

Non-authoritative answer:

Non-authoritative answer:

❯ kube exec -it alpine -- nslookup github.com
Server:		10.43.0.10
Address:	10.43.0.10:53

Non-authoritative answer:

Non-authoritative answer:
Name:	github.com
Address: 192.30.255.113

host.docker.internal fails with NXDOMAIN; our injected foo.bar comes back with no results; resolution of ordinary domains such as github.com work as normal.

6. Restart VM; repeat DNS requests using Alpine Docker container. Restart seems to be needed otherwise Docker resolutions sometimes fail too for a short while (cacheing maybe?).

❮ docker run -it alpine:3.19 nslookup host.docker.internal                 
Server:		192.168.5.2
Address:	192.168.5.2:53

Non-authoritative answer:

Non-authoritative answer:
Name:	host.docker.internal
Address: 192.168.5.2

❯ docker run -it alpine:3.19 nslookup foo.bar
Server:		192.168.5.2
Address:	192.168.5.2:53

Non-authoritative answer:

Non-authoritative answer:
Name:	foo.bar
Address: 192.168.5.2

❯ docker run -it alpine:3.19 nslookup github.com
Server:		192.168.5.2
Address:	192.168.5.2:53

Non-authoritative answer:

Non-authoritative answer:
Name:	github.com
Address: 192.30.255.113

Resolutions seem to be working at the Docker layer. Our pod is still around, we can try the resolutions there again, but they still fail.

7. Run the above steps again, but under QEMU. All resolutions succeed.

colima start --arch aarch64 --vm-type=qemu --memory=16 --with-kubernetes --network-address --dns-host 'foo.bar=192.168.5.2'

Expected behaviour

My expectations had been:

• resolution works the same for all containers, regardless of VM type, regardless of whether run under Kubernetes or not
• host.docker.internal resolves to 192.168.5.2
• custom DNS results specified using --dns-host apply to all containers. In the above case foo.bar returns a result of 192.168.5.2.

Additional context

No response

[maxbronnikov10]

+

[plobsing]

FYI, I recently figured out that this was caused by a default template that I had created using a very old version of Colima that no longer works in newer versions. The DNS issue was resolved for me by deleting the offending template file

rm ~/.colima/_templates/default.yaml

I've filed #1192 to improve usability of the templates system.