Update: this information is now available in the official docs.
If you want to know what version of your GPU drivers are active on GKE, here’s a one-liner:
kubectl logs -l k8s-app=nvidia-gpu-device-plugin \
-c "nvidia-gpu-device-plugin" --tail=-1 -n kube-system \
| grep Driver
What this command does is get all the logs of Pods with the label k8s-app=nvidia-gpu-device-plugin (there are several different DaemonSets that can install the drivers depending on the size of the node, but they all share this label). It will print the logs for the container named nvidia-gpu-device-plugin, and uses --tail=-1 to output all log messages. The final grep command isolates just the line we care about.
Here’s an example on my cluster:
$ kubectl logs -l k8s-app=nvidia-gpu-device-plugin -c "nvidia-gpu-device-plugin" --tail=-1 -n kube-system | grep Driver
I1206 18:37:08.251742 5851 metrics.go:144] nvml initialized successfully. Driver version: 535.104.12
I1206 18:37:05.043525 5686 metrics.go:144] nvml initialized successfully. Driver version: 535.104.12
$ kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3-gke.1203000