Running Docker in Docker on GKE Autopilot with gVisor

3 min read

Docker in Docker is an interesting workload to run on GKE in Autopilot mode, as for Autopilot you’re limited to userland programs, and Docker in Docker likes to run as root. This is a bit of a legacy from the days when docker was actually running on the host (now it’s containerd), so it “made sense” to just punch out of the container into the host’s docker environment to execute there, since, why not. Of course, this presents some security and supportability issues, as now that process is root. Fortunately there is a way to run Docker in Docker completely in userland, in a way compatible with GKE Autopilot. Here’s my test.

The solution is gVisor which allows you to provide Docker in Docker an execution environment that looks like it has full access, without actually giving it full access. Best of both worlds. The gVisor project has documented how to use Docker in Docker on GKE Autopilot, so I gave that a spin.

Updating the cluster to support docker-in-docker with gVisor

Before I could get it to work, I needed to make two changes to my Autopilot cluster:

Enable NET_ADMIN (which is disabled by default, as it grants elevated security permissions)
Update the cluster to 1.32 (which enables containers using gVisor to request SYS_ADMIN permissions)

You can easily update a cluster to enable NET_ADMIN as follows. This param can also be added during creation.

CLUSTER_NAME=autopilot-cluster-1
LOCATION=us-central1

gcloud container clusters update $CLUSTER_NAME --workload-policies=allow-net-admin --location $LOCATION

I then updated my cluster in the UI to the latest version of 1.32.

Building the builder container

Now to run the example. First we need to build our docker builder container. gVisor ships with a basic example that configures docker with the right network setup.

First, clone the gVisor project:

git clone https://github.com/google/gvisor.git
cd gvisor

Build and push the sample. Create your own Artifact Registry Docker repo, and copy the path to set REPO with your own path.

REPO=us-west1-docker.pkg.dev/gke-autopilot-test/dind-test
IMAGE=$REPO/docker-in-gvisor:latest

docker build -t $IMAGE images/basic/docker
docker push $IMAGE

echo "$IMAGE pushed".

I put a copy in docker.io/wdenniss/docker-in-gvisor:latest so you don’t have to build it yourself.

Running docker-in-docker on Autopilot

Now we deploy our new container as a Pod. Replace the container image with your own if you built it in the previous step.

apiVersion: v1
kind: Pod
metadata:
  name: docker-in-gvisor
spec:
  runtimeClassName: gvisor
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - "8.8.8.8"
      - "8.8.4.4"
  containers:
  - name: docker-in-gvisor
    image: docker.io/wdenniss/docker-in-gvisor:latest
    securityContext:
      capabilities:
        add: [AUDIT_WRITE,CHOWN,DAC_OVERRIDE,FOWNER,FSETID,KILL,MKNOD,NET_BIND_SERVICE,NET_RAW,SETFCAP,SETGID,SETPCAP,SETUID,SYS_CHROOT,SYS_PTRACE,NET_ADMIN,SYS_ADMIN]
    volumeMounts:
      - name: docker
        mountPath: /var/lib/docker
  volumes:
  - name: docker
    emptyDir: {}

docker-in-gvisor.yaml

Note: it’s important to specify the nameservers here, as Docker in Docker gets configured with these nameservers. By default, GKE uses internal nameservers which Docker in Docker can’t resolve. By setting external nameservers here, we ensure Docker in Docker can resolve names.

NOTE: for production, you probably don’t want to override the Pod’s nameservers, and instead configure the builder container to have whatever DNS configuration you like. I took the easy route here to get the gVisor demo docker container working without modification.

kubectl create -f https://raw.githubusercontent.com/WilliamDenniss/autopilot-examples/refs/heads/master/docker-in-gvisor/docker-in-gvisor.yaml

kubectl exec -it docker-in-gvisor -- bash

Copying the commands from gVisor’s example to create a Dockerfile and build it, we get (you can just copy/paste this block directly):

mkdir whalesay && cd whalesay
cat > Dockerfile <<EOF
FROM ubuntu
RUN apt-get update && apt-get install -y cowsay curl
RUN mkdir -p /usr/share/cowsay/cows/
RUN curl -o /usr/share/cowsay/cows/docker.cow https://raw.githubusercontent.com/docker/whalesay/master/docker.cow
ENTRYPOINT ["/usr/games/cowsay", "-f", "docker.cow"]
EOF

Build this container in our Docker in Docker environment:

docker build -t whalesay .

And now, run it:

docker run -it --rm whalesay "Containers do not contain, but gVisor-s do!"

You should see something like this:

root@docker-in-gvisor:/whalesay# docker run -it --rm whalesay "Containers do not contain, but gVisor-s do!"
 _________________________________________
/ Containers do not contain, but gVisor-s \
\ do!                                     /
 -----------------------------------------
    \
     \
      \
                    ##         .
              ## ## ##        ==
           ## ## ## ## ##    ===
       /"""""""""""""""""\___/ ===
      {                       /  ===-
       \______ O           __/
         \    \         __/
          \____\_______/

root@docker-in-gvisor:/whalesay#

As you can see, it is possible to use docker in GKE Autopilot via gvisor!

To clean up:

kubectl delete pod/docker-in-gvisor

Conclusion

At the end of the day, is all this worth it? I asked Gemini’s deep research to compare the pro’s and cons, here’s the conclusion:

gVisor offers substantial and fundamental security advantages over running Docker-in-Docker or Docker-out-of-Docker using privileged mode or host socket mounting within a Kubernetes cluster. By introducing a user-space kernel and significantly reducing the direct attack surface of the host kernel, gVisor provides a robust additional layer of isolation specifically designed to mitigate container escapes and kernel exploits. This makes it a far superior choice for running untrusted or security-sensitive workloads.

Privileged DinD/DooD, conversely, prioritize enabling Docker functionality within a containerized environment at the expense of security. The requirement for privileged access or direct control over the host Docker daemon fundamentally undermines container isolation principles, creating significant risks. While potentially necessary for certain legacy CI/CD workflows, these methods should be treated as high-risk configurations and replaced with safer alternatives like Kaniko, Buildah, or carefully evaluated sandboxed runtimes like gVisor whenever possible. The performance and compatibility overhead of gVisor represents a trade-off for its security benefits, requiring workload-specific evaluation, but for scenarios demanding strong isolation, it provides a demonstrably more secure foundation than privileged DinD/DooD.

So that’s Docker in Docker, in gVisor. I hope you found it useful.