# SSD Ephemeral Storage on GKE (including Autopilot) ![](Ephemeral-Storage.png) Do you need to provision a whole bunch of ephemeral storage to your Autopilot Pods? For example, as part of a data processing pipeline? In the past with Kubernetes, you might have used [`emptyDir`](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) as a way to allocate a bunch of storage (taken from the node’s boot disk) to your containers. This however requires that you carefully plan your node’s boot disks to ensure the disk has enough total storage to handle the Pods you plan to schedule, and may result in overprovisioning (i.e. unused storage capacity) if the Pods that end up being scheduled don’t use it all. Lots of inflexible, upfront planning combined with potential wastage 🤮. Update: this is now [documented in GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/generic-ephemeral-volumes). Autopilot, thankfully, eliminates node-based planning—but then, how can you secure a huge ephemeral volume for your Pod’s scratch space if you can’t choose your node’s boot disk size? Fortunately, Kubernetes 1.23 introduced the [generic ephemeral volume](https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes) mount, which makes this a breeze 🎉. Ephemeral volumes allow us to mount a persistent volume into the container that persists for the life of the Pod. Being an attached volume, you can allocate huge amounts of storage (up to 64Ti on Google Cloud), independent of the node boot disk, or what other pods on the node are doing! You can tune the storage class for this volume as well, which in Google Cloud gives you a few different options, like the high-performance SSD (“pd-ssd”) and the mid-ranged Balanced SSD (“pd-balanced”). In this example, I will demonstrate how you would mount a 1TiB SSD ephemeral volume into a Pod running on Autopilot. In Autopilot, everything is configured automatically, and is ready for use. If you use GKE Standard and created your cluster on an older version (prior to 1.20), you can follow [these steps](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver#enabling_the_on_an_existing_cluster) to enable the driver. Newer versions of GKE Standard also have this enabled by default. First, we need to define our storage class, this is where we specify the type of PD that we want. The different options are [defined here](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver#create_a_storageclass), in our case let’s use `pd-ssd` which offers the highest performance. ```yaml {hl_lines=[8, 9]} apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: ssd provisioner: pd.csi.storage.gke.io volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true parameters: type: pd-ssd ``` [storage-class-ssd.yaml](https://github.com/WilliamDenniss/autopilot-examples/blob/master/ephemeral-volume/storage-class-ssd.yaml) ```yaml {hl_lines=[25, 26, 27, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41]} apiVersion: apps/v1 kind: Deployment metadata: name: ephemeral-example spec: replicas: 1 selector: matchLabels: pod: example-pod strategy: type: Recreate template: metadata: labels: pod: example-pod spec: containers: - name: busybox-container image: busybox resources: requests: cpu: 500m memory: 2Gi ephemeral-storage: 2Gi volumeMounts: - mountPath: "/scratch" name: scratch-volume command: [ "sleep", "1000000" ] volumes: - name: scratch-volume ephemeral: volumeClaimTemplate: metadata: labels: type: scratch-volume spec: accessModes: [ "ReadWriteOnce" ] storageClassName: "ssd" resources: requests: storage: 1Ti ``` [ephemeral-example.yaml](https://github.com/WilliamDenniss/autopilot-examples/blob/master/ephemeral-volume/ephemeral-example.yaml) Create both resources: ```shell kubectl create -f https://raw.githubusercontent.com/WilliamDenniss/autopilot-examples/master/ephemeral-volume/storage-class-ssd.yaml kubectl create -f https://raw.githubusercontent.com/WilliamDenniss/autopilot-examples/master/ephemeral-volume/ephemeral-example.yaml ``` To verify that things worked, let’s shell into the pod, and run `df -h`. As you can see, we have a 1TiB disk mounted at `/scratch`. ```shell {hl_lines=[7]} $ kubectl exec -it deploy/ephemeral-example -- sh / # df -h Filesystem Size Used Available Use% Mounted on overlay 94.3G 3.6G 90.6G 4% / tmpfs 64.0M 0 64.0M 0% /dev tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup /dev/sdb 1006.9G 28.0K 1006.8G 0% /scratch /dev/sda1 94.3G 3.6G 90.6G 4% /etc/hosts /dev/sda1 94.3G 3.6G 90.6G 4% /dev/termination-log /dev/sda1 94.3G 3.6G 90.6G 4% /etc/hostname /dev/sda1 94.3G 3.6G 90.6G 4% /etc/resolv.conf shm 64.0M 0 64.0M 0% /dev/shm tmpfs 2.0G 12.0K 2.0G 0% /var/run/secrets/kubernetes.io/serviceaccount tmpfs 1.9G 0 1.9G 0% /proc/acpi tmpfs 64.0M 0 64.0M 0% /proc/kcore tmpfs 64.0M 0 64.0M 0% /proc/keys tmpfs 64.0M 0 64.0M 0% /proc/timer_list tmpfs 1.9G 0 1.9G 0% /proc/scsi tmpfs 1.9G 0 1.9G 0% /sys/firmware ``` Now let’s compare the write performance. In theory, our `/scratch` volume should be faster since it’s based on the SSD PD. As we can see from this simple test, we get roughly 2x faster writes on the SSD ephemeral volume writing 40x 50MB worth of data. Nice! ```shell {hl_lines=[1, 4, 6, 9]} # dd if=/dev/zero of=/tmp/test1.img bs=50M count=40 oflag=direct 40+0 records in 40+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 17.4605 s, 120 MB/s # dd if=/dev/zero of=/scratch/test1.img bs=50M count=40 oflag=direct 40+0 records in 40+0 records out 2097152000 bytes (2.1 GB, 2.0 GiB) copied, 8.29781 s, 253 MB/s ``` Finally, for fun let’s scale it up and then delete to see our ephemeral resources. As you can see, these disks only stick around for the duration of the Pod (which is the whole point). ```shell $ kubectl scale deploy/ephemeral-example --replicas 10 deployment.apps/ephemeral-example scaled $ kubectl get pods NAME READY STATUS RESTARTS AGE ephemeral-example-649fbddfb8-8rq4s 1/1 Running 0 5m45s ephemeral-example-649fbddfb8-d2f2n 1/1 Running 0 2m37s ephemeral-example-649fbddfb8-g8xqs 1/1 Running 0 2m37s ephemeral-example-649fbddfb8-kkvn7 1/1 Running 0 2m37s ephemeral-example-649fbddfb8-mt4jb 1/1 Running 0 2m37s ephemeral-example-649fbddfb8-npxn7 1/1 Running 0 2m37s ephemeral-example-649fbddfb8-pwcsn 1/1 Running 0 2m37s ephemeral-example-649fbddfb8-s9cv7 1/1 Running 0 2m37s ephemeral-example-649fbddfb8-t4ppr 1/1 Running 0 2m37s ephemeral-example-649fbddfb8-wlw8v 1/1 Running 0 2m37s $ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE ephemeral-example-5c7f74dd84-5slbb-scratch-volume Bound pvc-c2eeb65e-1608-4131-b87f-9c00303a05e0 1Ti RWO ssd 22m ephemeral-example-5c7f74dd84-c972t-scratch-volume Bound pvc-48c2e8e4-cbcd-439a-b2dd-d2b824b5c392 1Ti RWO ssd 17m ephemeral-example-5c7f74dd84-dr4ng-scratch-volume Bound pvc-ecf724d1-1095-4213-8d57-3067f9cea5f6 1Ti RWO ssd 14m ephemeral-example-649fbddfb8-8rq4s-scratch-volume Bound pvc-435e3b92-e925-4bfc-99c6-be55a2665f57 1Ti RWO ssd 5m47s ephemeral-example-649fbddfb8-d2f2n-scratch-volume Bound pvc-f359fdef-ae60-46dd-8d45-b7b578650daa 1Ti RWO ssd 2m40s ephemeral-example-649fbddfb8-g8xqs-scratch-volume Bound pvc-888aff7d-16ba-42c5-91e6-16ffa5647e21 1Ti RWO ssd 2m40s ephemeral-example-649fbddfb8-kkvn7-scratch-volume Bound pvc-26afd18b-f039-41b3-9927-81aa39bd7d1c 1Ti RWO ssd 2m40s ephemeral-example-649fbddfb8-mt4jb-scratch-volume Bound pvc-893208c5-4677-4b42-9fba-d987157c0e04 1Ti RWO ssd 2m40s ephemeral-example-649fbddfb8-npxn7-scratch-volume Bound pvc-e9fc41b3-818b-42df-9418-994222298695 1Ti RWO ssd 2m40s ephemeral-example-649fbddfb8-pwcsn-scratch-volume Bound pvc-8487d270-beb8-4842-b107-dff8f0862d70 1Ti RWO ssd 2m40s ephemeral-example-649fbddfb8-s9cv7-scratch-volume Bound pvc-c37dd9ce-33b3-4010-ae5f-938e4c033d13 1Ti RWO ssd 2m40s ephemeral-example-649fbddfb8-t4ppr-scratch-volume Bound pvc-ba856fac-6ea3-4cf7-98b5-2aa599870d48 1Ti RWO ssd 2m40s ephemeral-example-649fbddfb8-wlw8v-scratch-volume Bound pvc-98ef6aac-93b4-4495-a359-2fc717729a09 1Ti RWO ssd 2m40s $ kubectl delete deploy/ephemeral-example deployment.apps "ephemeral-example" deleted $ kubectl get pvc No resources found in default namespace. ```