William Denniss

Deploying an ADK agent for Gemini Enterprise: Agent Runtime or Cloud Run?

Wed, 02 Sep 2026 18:44:54 -0700

Comment: After iterating on the API Contract post, I asked Codex 5.6 Sol Ultra to form an opinion about when to use Agent Runtime and when to use Cloud Run in the context of a GE app connected agent, and then write a post. I specifically instructed it to avoid the cloudpickle deployment options (which only work on Agent Runtime), to focus on the other trade-offs. This is the resulting post, presented unmodified. I think it captures the nuances well and presents a balanced view. –William

Running stateful ADK agents on Substrate and GKE

Tue, 01 Sep 2026 21:56:54 -0700

Agent Substrate can run stateful Google ADK agents across a shared pool of GKE worker Pods. After an agent responds, the harness can suspend its whole process—including its conversation, in-memory working state, and files—and Substrate can restore it on whichever compatible worker is free for the next request.

The neat part is that this doesn’t require cold-starting ADK again. A Python ADK agent can easily take ten seconds or more to initialize, but Substrate’s golden snapshot pays that cost just once per template. Each actor then resumes from its own latest snapshot instead of rebuilding its process and session. This also makes it stateful. It picks up where it left off, while its previous worker is free to run another agent. It’s really performant, and great for agents that can benefit from a persistant state.

Serving the Agent Platform API contract

Mon, 13 Jul 2026 21:41:56 +0000

With Agent Platform’s bring your own container (BYOC) deployment option, you need to serve the standard API contract yourself. In The Agent Platform API contract: what your agent must expose, and why, I covered why the contract exists and walked through the thirteen methods that AdkApp exposes. Now let’s put a serving layer around them and call the deployed agent end to end.

On the managed paths (the pickle, stub, and source options from the deploy post), the platform builds the serving layer for you — AdkApp already has all thirteen methods, and the platform either introspects the list or you declare it. You never see the plumbing.

The Agent Platform API contract: what your agent must expose, and why

Mon, 13 Jul 2026 21:41:56 +0000

Agent Runtime can host a container with arbitrary HTTP endpoints. To use it through the Agent Platform Python SDK or the Google Cloud console playground — or through Gemini Enterprise’s native ADK integration — the container must also implement the operations those integrations expect. These routes and request formats are the HTTP side of the Agent Platform runtime contract. Agent Platform builds the serving layer for object and source deployments; with a custom Dockerfile or prebuilt container, you serve it yourself. I covered those choices in 5 ways to deploy agents on Agent Platform. This post explains why the contract exists and what an ADK agent exposes through it.

5 ways to deploy agents on Agent Platform

Wed, 01 Jul 2026 17:52:38 +0000

So you’ve built an ADK agent—in my case the Trading Agent from my post series, an agent that reads S&P 500 news, scores the sentiment, and places paper trades on Alpaca—and now you want it running on Agent Platform’s managed runtime instead of on your laptop. It turns out there are actually five different ways to package your agent in runtime, each with some trade offs. This post explores those options and when you might want to use each one.

About

Fri, 19 Jun 2026 12:00:00 -0700

Hi, I’m William. I like to build things. This is my blog where I mostly post about my professional work.

You can learn more about what I’ve created on LinkedIn, and what I’m doing now on Twitter (and, this blog!). If you want to get in touch, best way is to DM me on one of those two networks 🙂

OAuth Device Flow

Fri, 19 Jun 2026 18:01:56 +0000

https://datatracker.ietf.org/doc/html/rfc8628

Security Event Token (SET)

Fri, 19 Jun 2026 18:01:26 +0000

https://datatracker.ietf.org/doc/html/rfc8417

AppAuth

Thu, 18 Jun 2026 16:57:11 +0000

See: http://appauth.io/

Videos

Wed, 17 Jun 2026 19:40:26 +0000

Google Next 2025

Transforming your Business with AI: The Google Kubernetes Engine (GKE) Advantage. Spotlight session highlighting major updates. Autopilot content starts at XYZ.

How Shopify runs Black Friday, Cyber Monday on GKE. Highlights usage of CCC (at XYZ), and Autopilot’s new container-optimized compute platform (at ZYC).

Launches

Wed, 17 Jun 2026 19:36:56 +0000

In addition to the self-published blog posts on this site, here is a selection of launch posts relevant to my work, published on Google blog channels, and elsewhere.

GKE Autopilot

Meet the new GKE: Extending Autopilot to all qualifying clusters, September 2025

eBook: Kubernetes for Developers, A clear and practical beginner’s guide to Kubernetes

GKE under the hood: Container-optimized compute delivers fast autoscaling for Autopilot, August 2025

Happy birthday, GKE! Let’s celebrate with new features and better pricing, August 2025

Deterministic memories

Mon, 15 Jun 2026 18:37:26 +0000

The Trading agent we deployed earlier will automatically have session persistence when deployed on Agent Platform. You can test this by asking it a day later “What trades did you execute” from the same session that you executed the trades from. The full conversation history is preserved, even while the agent is suspended (you’re not paying for it to run continuously).

You can also inspect the saved session data from the console.

Creating an Agent Platform resource identifier

Tue, 09 Jun 2026 13:54:14 -0700

To use Agent Platform services like Memory Bank, and Sessions from other environments, you need to first create a resource that is currently known as an “agent engine”.

Note: if you are familiar with the Python ADK libraries, you can follow these instructions instead, to create an instance via the SDK.

Here’s a single API call to create a resource identifier that you can use right away:

# 1. Set your variables
PROJECT_ID=$(gcloud config get-value project)
LOCATION="us-west1" # Replace with your target region
RESOURCE_DISPLAY_NAME="My Memory Bank"

# 2. Execute the REST API request
curl -X POST \
  -H "Authorization: Bearer $(gcloud auth print-access-token)" \
  -H "Content-Type: application/json" \
  "https://${LOCATION}-aiplatform.googleapis.com/v1/projects/${PROJECT_ID}/locations/${LOCATION}/reasoningEngines" \
  -d "{
    \"displayName\": \"${RESOURCE_DISPLAY_NAME}\"
  }"

The creation returns immediately with an operation id, and executes async, you can query the API with the operation, or look up the Deployments UI to get the resource.

Agent memories with ADK

Mon, 01 Jun 2026 16:16:44 -0700

ADK Agents will automatically have session persistence when deployed on Agent Platform. You can test this by asking follow up questions in the running sessions like “What trades did you execute?”. The full conversation history is preserved, even while the agent is suspended (you’re not paying for it to run continuously), and is visibile in the console.

When you start a new session however, that context is gone. It’s common to want some level of memory and personalization for an agent. For example, if the user says “I never want to buy ___ stock”, ideally we would remember this and factor that in to trading decisions.

Deploy an ADK agent to Agent Platform Runtime

Fri, 15 May 2026 11:03:02 -0700

In my previous post, we created an agent with ADK and ran it locally. The next step is to deploy it to Agent Platform runtime.

You can deploy to the runtime using a Python script that imports your agent, and configures the remote resource, and performs the deployment. This approach serializes your live object graph which is simple, but not suitable for production. We’ll cover production hardening later.

Create the following script in the root project folder (the one containing trading_agent from the previous example). There’s nothing to edit in it: the project ID is read from your environment (via Application Default Credentials), and the staging bucket name is derived from it (gs://${PROJECT_ID}-trading-agent), with the bucket created on first run if it doesn’t exist. Just point gcloud at the right project first:

Building a stock-trading agent with ADK

Wed, 22 Apr 2026 11:17:03 -0700

The Agent Development Kit (ADK) is Google’s open-source Python framework for building AI agents. You write the agent’s logic and the tools it can call, and ADK handles orchestrating everything — with evaluation and deployment built in too.

I wanted a slightly more interesting app than Hello World to exercise more parts of the platform, so I figured I’d write a little stock trading agent that applies a sentiment momentum trading strategy. We’ll use ADK and the Gemini API to execute trades on Alpaca, chosen for its REST API and free paper-trading account.

Upgrading a GKE cluster to use Autopilot Compute Class

Fri, 12 Dec 2025 15:41:36 -0700

Autopilot is now available in GKE clusters via Compute Class. See the official announcement, and documentation.

Versions with this feature are now available for clusters in the Regular release channel. Here’s how to upgrade early and get these features today.

Per the docs, the feature is in version 1.34.1-gke.1829001 or later. To upgrade we need to find a valid version for our control plane (master) equal or later to this version and upgrade to it.

Programmable Spend-based CUD Purchase

Wed, 15 Oct 2025 12:00:00 -0700

Did you know it’s possible to purchase spend-based CUDs programmatically, including (for now) the legacy Autopilot CUD and the new Compute flexible CUD?

Let’s take it for a spin.

Firstly, you’ll need the procurement API enabled and IAM permissions setup. Follow the steps in Before you begin in the official doc.

The gcloud CLI will need to be authenticated in order to generate the needed access token (gcloud auth print-access-token should print out an auth token)

Using C4A and C4D with Compute Class

Sun, 17 Aug 2025 12:40:09 -0700

I’ve been following Arm for a while, so was glad that the C4A VM was GA’d earlier this year. These machines run Google silicon Axion chip. It’s also great to see broad geographical availability including regions like Tokyo. Feels like Arm is finally ready for prime time.

C4D running 5th Gen AMD EPYC processors (Turin) was even more recently released. Let’s try them both out!

Arm on GKE

Here’s how to deploy an Arm workload on GKE using Compute Class. I’ll be using Autopilot mode.

Simulate a zonal failure on Autopilot

Fri, 30 May 2025 22:58:52 -0700

GKE’s Autopilot mode is designed to be safer to operate and prevents many issues that can lead to sudden outages like misconfigured firewalls. However, if you’re trying to actually simulate a failure like when performing a zonal failure simulation, Autopilot’s safeguards can sometimes get in the way, blocking some of the potential ways to run such a test.

It is possibile still though to simulate the loss of compute capacity in a zone and see how your load balancing and workload will respond. Given that this is one of the key attributes in your control (e.g. number of replicas, zonal spread topology, etc), I believe this is a useful test. We can achieve this through kubectl cordon, kubectl delete pods and a little scripting.

From an Unsupported macOS to Supported Windows

Sun, 11 May 2025 12:00:00 -0700

I had some old MacBooks, one upgradable to Big Sur (no longer supported), one upgradable to Ventura (supported, but soon to be obsolete). My family needed some windows machines, and remarkably I was able to move from unsupported and old macOS to completely supported Windows 11. Who knows how many more years of support these old machines can get, but presumably till Windows 11 EOL which is impressive.

Steps

Move/backup/delete all data from the Macbook and any existing Bootcamp installs (your MacBook will be completely wiped during this process)
Update to the very latest macOS (so that your firmware is up to date; once you do this you can’t update your firmware again which is fine because it likely won’t get any more updates anyway)
Download your Windows Support Software (very important). Boot Camp -> Action -> Download Windows Support.
Get an 8GB thumb drive and create a Windows 11 installable boot
Copy the Windows Support Software onto the same thumb drive, there should be room. Else, use another.
Boot into the Windows 11 installer. (Did you save your Windows Support Software? Go back and do this if you didn’t).
You need to add some bypass options for the TPM check. Shift F10. regedit.
- Navigate to HKEY_LOCAL_MACHINE\SYSTEM\Setup
- Edit -> New -> Key, named LabConfig
- Open that key, create two DWORD (32-bit) values:
  - BypassTPMCheck
  - BypassSecureBootCheck.
- Modify both DWORDs and set the value to 1
- BypassCPUCheck and BypassRAMCheck are also options, but I didn’t need them for my macbooks (and it was good to know that with the check still active, they apparently pass the minimum bar for those.
Start installer
Format the disk. There is no need to install drivers at this step, it works without them. This is the part that wipes all data on the disk.
- Delete Disk 0, Partition 2 (and all partitions on Disk 0, except Partition 1). Should eventually read “Unallocated space”. You may need to hit “Refresh” a few times.
- Create a partition on the unallocated space. If you get an error (I did), just refresh, it seemed to actually work. Then format the new partition (the largest one on Disk 0)
Select the newly created partition to install Windows 11
Install
When prompted to install the Wifi driver, you can’t. Don’t try to load drivers from your thumb drive, they don’t work. Instead bypass the Wifi requirement, press Shift+F10 and run OOBE\BYPASSNRO in the command prompt.
Once windows is setup, then run bootcamp Setup.exe (which will install your Wifi and other drivers)

You now have a pure Windows 11 machine!

Running Docker in Docker on GKE Autopilot with gVisor

Mon, 05 May 2025 18:43:04 -0700

Docker in Docker is an interesting workload to run on GKE in Autopilot mode, as for Autopilot you’re limited to userland programs, and Docker in Docker likes to run as root. This is a bit of a legacy from the days when docker was actually running on the host (now it’s containerd), so it “made sense” to just punch out of the container into the host’s docker environment to execute there, since, why not. Of course, this presents some security and supportability issues, as now that process is root. Fortunately there is a way to run Docker in Docker completely in userland, in a way compatible with GKE Autopilot. Here’s my test.

Running DeepSeek open reasoning models on GKE

Wed, 26 Feb 2025 15:52:00 -0700

DeepSeek’s R1 open model launch caused quite a stir with one of the first open reasoning models. Here’s how to run a demo of it locally on GKE!

We can use an Nvidia L4 (or A100 40GB) to run the 8B Llama distilled model, or a A100 80GB to run the 14 and 32B Quen distilled models. I’ll be pairing this with a sample Gradio application that can stream responses, and handles the “</think>” block (unique to the reasoning models) by adding a horizontal line to the output to delineate where the thinking ends, and the final response begins. In your own application of course, you may choose to hide the “</think>” block entirely, or display it in a different manner, like the current LLM chat apps do.

Kubernetes History Inspector

Thu, 30 Jan 2025 14:56:30 -0700

GKE users get access to an awesome new tool this week: the Kubernetes History Inspector. This product, released as open source, parses Kubernetes and GKE logs to generate a timeline with all events in the cluster.

Kubernetes is a complicated system with multiple objects, and various automated processes. A single Deployment results in at least 3 objects: the Deployment, ReplicaSet (one for each app version), and Pods (one for each replica). Nodes come and go. Pods may fail and be re-created. Horizontal Pod Autoscaling might add and remove Pods, etc, etc. Kubernetes History Inspector enables you to view all the interactions of this system with a single pane of glass. Brilliant.

In-app browsers and RFC 8252

Fri, 15 Nov 2024 16:57:47 +0000

In 2016 I wrote RFC 8252 seeking to codify a new best practice on using the system browser (or in-app browser tabs) to perform OAuth flows, rather than a built-in WebView.

A little history: you don’t see this embedded WebView OAuth pattern much any more, but when I joined the Identity team at Google in 2014, it was everywhere, and there was a popular OSS client library which helped developers do it this way. With the WebView pattern, the user would login to their identity provider in a view inside the app during the OAuth flow (as documented in section 9 of RFC 6749). This had some problems including security (the containing app can take the session authentication token, which is typically more privileged than the OAuth token), and usability (the user has to login again every time), a full list of issues can be found in section 8.12 of RFC 8252. My work on 8252 was to codify a new approach by using the system browser. As for that old client library? Together with my colleagues, I created a better one to provide a reference implementation of 8252. In 2016, Google formally required developers to adopt this modern approach for better security and usability.

Getting resource recommendations from VPA

Wed, 06 Nov 2024 12:07:06 -0700

Setting appropriate resource requirements in Pods is one of the most critical tasks of a developer deploying in Kubernetes. In section 5.2 of my book, I discuss how you can use kubectl top pods to analyze and set resource values. If you’re fortunate enough to use GKE where the VPA component is offered out of the box, there is another option for analyzing usage which makes this even easier and more accurate: running VPA in advisory mode.

Scaling container resources automatically with VPA

Tue, 05 Nov 2024 12:47:51 -0700

Vertical Pod Autoscaling is a system that measures Pod utilization and attempts to set the right resource requests. For example, if the Pod is constantly using more CPU, VPA will increase the CPU requests. Contrasting Horizontal Pod Autoscaling (HPA) which can create more replicas as your Pods use more resources, VPA changes the resources of each replica.

In the past, GKE in Autopilot mode had a 250 milliCPU resource increment (meaning valid Pod sizes were 250m, 500m, 750m, etc). Now that Autopilot supports burstable QoS and fine-grained resource increments, VPA should be even more useful, so let’s give it a spin.

Using Node-based pricing on GKE Autopilot

Tue, 03 Sep 2024 11:19:49 -0700

New this year, Autopilot now has two pricing models: the original Pod-based model, and the new node-based option. The pricing page does a pretty good job of explaining the difference (at least I hope it explains it well, as I wrote it), and how best to utilize each option, but here’s a quick recap anyway:

The Pod model is great for when you don’t want to think about or worry about bin-packing of nodes—allow GKE to take of everything. It uses an all-inclusive pricing model where you don’t need to be concerned with underutilized nodes, or odd-shaped workloads. The node model is useful when you have specific hardware requirements (like a particular GPU, or CPU), or have large workloads that bin-pack well and you just want to buy some VMs. The node model is billed by the node at Compute Engine prices, with a small added premium. The node model can work out cheaper, provided you fill the nodes. The most optimal way to use Autopilot is typically with a mix of both strategies, taking advantage of whichever pricing model works better for the workload in question.

Serving Stable Diffusion with RayServe on GKE Autopilot

Tue, 03 Sep 2024 11:12:25 -0700

Now that you’ve tried out Stable Diffusion on GKE Autopilot via the WebUI, you might be wondering how you’d go about adding stable diffusion as a proper micro-service that other components of your application can call. One popular way is via Ray. Let’s try this tutorial: Serve a StableDiffusion text-to-image model on Kubernetes, on GKE Autopilot. Here goes:

Create an Autopilot Cluster

You’ll want version 1.28 or later for this, to get the newer CUDA drivers. This will do the trick:

Clearing macOS Desktop Clutter

Tue, 23 Apr 2024 12:00:00 -0700

Every year I create a folder on my macOS desktop with the current year, like 2024 and move all the desktop items into it (so it contains a folder named 2023 and so on). The reason? To clear my desktop, as for some reason macOS defaults to it for saving various files.

This two-liner is better. Simply turns off the display of the files on your desktop. Perfect 👌

defaults write com.apple.finder CreateDesktop false
killall Finder

And to change the default screenshot location:

Stable Diffusion WebUI on GKE Autopilot

Thu, 15 Feb 2024 15:30:20 -0700

I recently set out to run Stable Diffusion on GKE in Autopilot mode, building a container from scratch using the AUTOMATIC1111‘s webui. This is likely not how you’d host a stable diffusion service for production (which would make for a good topic of another blog post), but it’s a fun way to try out the tech.

My first key learning was to start with a Google Deep Learning Container which provides a useful base image with CUDA 12 support and is based on Debian. This provides the perfect operating environment for running a CUDA 12 application on Google Cloud. I first attempted to create my own image using Debian, but Stable Diffusion couldn’t find the GPU—better just to use the Google ones that are preconfigured with everything I need!

LLM Model Serving on Autopilot

Thu, 21 Dec 2023 00:10:16 -0700

When building your business using LLMs as a key component, you may wish to be a master of your own domain and run your own model. Running your own LLM protects you from changes like pricing increases or API availability with third-party services, guarantees the privacy of your data (no data needs to leave your VPC), and lets you experiment with advanced topics like fine-tuning. GKE, and in particular Autopilot mode is a great place to run your own LLM, as it takes care of much of the compute orchestration for you, allowing you to focus your energy on providing the LLM API for your business.

Cordoning nodes in GKE Autopilot

Wed, 20 Dec 2023 20:03:27 -0700

Occasionally you may wish to remove and replace a node in GKE Autopilot.

That can be done with

kubectl cordon $NODE_NAME

If you want to immediate drain the Pods and put them in the pending state, you can further drain the node:

kubectl drain $NODE_NAME --delete-emptydir-data --ignore-daemonsets

What about if you need to do all nodes? Be aware that this is pretty disruptive, and lot more aggressive than if you leave Autopilot to handle the upgrade itself (which is recommended), but you can apply this technique to the cluster as a whole like so:

CUDA 12 on GKE Autopilot

Wed, 20 Dec 2023 00:29:00 -0700

Per the NVIDIA docs, CUDA 12 applications require driver 525.60.04+. This driver is available as part of GKE 1.28. To upgrade an existing cluster to the latest version of 1.28:

VERSION="1.28"
REGION="us-central1"
CLUSTER_NAME="autopilot-cluster-1"
gcloud container clusters upgrade $CLUSTER_NAME \
    --region $REGION \
    --master --cluster-version $VERSION

This upgrades the control plane, and schedules the nodes to follow, which generally completes within a day or two (depending on how many nodes you have).

Using Image Streaming with DockerHub

Wed, 20 Dec 2023 00:23:58 -0700

Image streaming is a really great way to speed up workload scaling on GKE. Take for example the deep learning image from Google. In my testing, the container is created in just 20s, instead of 3m50s. While there is slightly higher latency on reads while the image streams, the 3m30s head start is going to well make up for that.

If you use GKE in Autopilot mode, and Artifact Registry for your images, then you’re already using image streaming (to verify, look for the ImageStreaming event with kubectl get events, if it’s not there, then check your settings such as the API enablement).

Finding the NVIDIA Driver Version on GKE

Tue, 19 Dec 2023 18:56:01 -0700

Update: this information is now available in the official docs.

If you want to know what version of your GPU drivers are active on GKE, here’s a one-liner:

kubectl logs -l k8s-app=nvidia-gpu-device-plugin \
    -c "nvidia-gpu-device-plugin" --tail=-1 -n kube-system \
    | grep Driver

What this command does is get all the logs of Pods with the label k8s-app=nvidia-gpu-device-plugin (there are several different DaemonSets that can install the drivers depending on the size of the node, but they all share this label). It will print the logs for the container named nvidia-gpu-device-plugin, and uses --tail=-1 to output all log messages. The final grep command isolates just the line we care about.

Testing Namespace Isolation using NetworkPolicy

Mon, 06 Nov 2023 11:17:26 -0700

In my post Isolating namespaces with NetworkPolicy, I presented a pattern for isolating namespaces from each other in a cluster using NetworkPolicy. Let’s put those concepts in to practice.

Trying it out

Cluster requirements

This example is designed to run on GKE, using DPv2 and CloudDNS. This is the default for Autopilot, so the following is enough to get you a working environment:

gcloud container clusters create-auto autopilot-cluster-3 \
--region us-west1

For the node-based version of GKE, enable both options. On the command line, add --enable-dataplane-v2 --cluster-dns=clouddns

Isolating namespaces with NetworkPolicy

Mon, 06 Nov 2023 11:16:16 -0700

Kubernetes allows for multi-tenancy within the cluster using namespaces. You can host multiple different applications each in their own namespace that at times can feel almost like a separate cluster. If you configure RBAC you can grant users and service accounts access to specific namespaces and even control what objects they can modify within.

However, by default every Pod can communicate with every other Pod, cluster wide. This is actually a really nice design property of Kubernetes, which allows different teams to publish their own services, but it doesn’t give you any network isolation at all. While a team may want to allow internal access of a particular service from other namespaces, they may not want all their internal services exposed.

Editing IP Masquerading rules for GKE Autopilot

Thu, 05 Oct 2023 10:30:00 -0700

IPs are masqueraded by default in Autopilot to use the node IP for egress traffic. which is just a fancy way of saying, the Pod traffic looks like it comes from the node’s IP.

This is handy when using non-RFC1918 ranges in GKE to avoid IP exhaustion, since your Node IP range is typically in RFC 1918 space which is what other services may be expecting. However, GKE does not masquerade traffic to internal addresses (like 10.0.0.0/8) by default. If you have a service that is expecting IPs in the RFC 1918 range, you may need to adjust the masquerade rules to include those destinations.

GKE Network Planning (2023)

Wed, 04 Oct 2023 05:30:22 -0700

GKE operates on a flat VPC structure. That means that every Node and Pod has an identity within your VPC, and their IPs are not re-used. This is convenient, as Pods are addressable within the VPC, but unless you create multiple VPCs to isolate resources, you can end up using a lot of IPs very quickly. Fortunately, the system is pretty flexible and there are some steps you can take to optimize.

Adding Pod IP ranges to GKE clusters

Mon, 02 Oct 2023 22:40:44 -0700

Did you know that you can now add Pod IP ranges to GKE clusters? Pods use a lot of IPs, which in the past forced you to compromise. Do you allocate a lot of IPs to the cluster allowing for growth while reserving a big group of IPs, or do you allocate just a little to conserve IPs but risking the need to recreate the cluster if you expand.

BYO Service Mesh on GKE Autopilot

Mon, 25 Sep 2023 16:53:05 -0700

Did you know you can now run any service mesh on Autopilot? That’s right! Even though Autopilot is a fully managed Kubernetes platform which by nature means you don’t have full root access to the node, that doesn’t mean you’re limited in what service mesh you can install.

How does it work? Service Meshes require the NET_ADMIN Linux permission. While NET_ADMIN isn’t a super high privileged permission, it has occasionally been the source of CVEs, so to keep things safe, the permission is off by default. For those who want to run their own service mesh though, now you can use it. Just add --workload-policies=allow-net-admin when creating a cluster (CLI only for now), or at any time update the cluster as follows:

Mapping Machine Types to Autopilot Compute Classes

Fri, 12 May 2023 02:03:15 -0700

Autopilot offers a slightly more abstract view of compute infrastructure than GKE’s Standard mode. The idea is to focus more on the different use-cases than the minutiae of individual machine configurations. However, if you’re coming from GKE Standard or GCE, and know those machine types well, you might want to understand the mapping between the two.

Here’s a quick reference for the resulting machine type from an Autopilot nodeSelector.

Multi-Cluster Services on GKE

Fri, 05 May 2023 04:29:27 -0700

Connect internal services from multiple clusters together in one logical namespace. Easily connect services running in Autopilot to Standard and vice versa, share services between teams running on their own services, and back an internal service by replicas in multiple clusters for cross-regional availability. All with the Multi-cluster Service support in GKE.

For this demo, let’s create a service in a GKE Autopilot cluster, and access it from a GKE Standard cluster.

Strict Pod Co-location

Thu, 30 Mar 2023 02:44:22 -0700

Pod affinity is a useful technique in Kubernetes for expressing a requirement that a pod, say with the “reader” role, is co-located with another pod, say with the “writer’ role. You can express that requirement by adding something like the following to the reader pod.

      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: pod
                operator: In
                values:
                - writer-pod
            topologyKey: "kubernetes.io/hostname"

The catch is, if there is no space on the node, and nothing that can be evicted, this reader pod may never schedule.

Reducing GKE Log Ingestion

Fri, 10 Mar 2023 20:23:43 -0700

[Update (2023-12-20): You can now turn off workload logging in Autopilot. That is the recommended approach if you want to remove all workload logs. To disable workload logs for Autopilot (for example, if you use a third-party logging agent like DataDog), pass this value at cluster creation:

gcloud container clusters create-auto $CLUSTER_NAME \
    --logging=SYSTEM

Or update

gcloud container clusters update $CLUSTER_NAME \
    --logging=SYSTEM

The remainder of this blog post is as originally posted, and shows you how to fine-tune the logs that that you keep]

HA 3-zone Deployments with PodSpreadTopology on Autopilot

Thu, 01 Dec 2022 04:59:08 -0700

PodSpreadTopology is a way to get Kubernetes to spread out your pods across a failure domain, typically nodes or zones. Kubernetes platforms typically have some default spread built in, although it may not be as aggressive as you want (meaning, it might be more tolerant of imbalanced spread).

Here’s an example Deployment with a PodSpreadToplogy that will result in an even spread over all zones (I also have a writeup on this topic which you can preview in my book).

SSD Ephemeral Storage on GKE (including Autopilot)

Mon, 28 Nov 2022 17:13:41 -0700

Do you need to provision a whole bunch of ephemeral storage to your Autopilot Pods? For example, as part of a data processing pipeline? In the past with Kubernetes, you might have used emptyDir as a way to allocate a bunch of storage (taken from the node’s boot disk) to your containers. This however requires that you carefully plan your node’s boot disks to ensure the disk has enough total storage to handle the Pods you plan to schedule, and may result in overprovisioning (i.e. unused storage capacity) if the Pods that end up being scheduled don’t use it all. Lots of inflexible, upfront planning combined with potential wastage 🤮.

TensorFlow on GKE Autopilot with GPU acceleration

Mon, 03 Oct 2022 16:16:57 -0700

Last week, GKE announced GPU support for Autopilot. Here’s a fun way to try it out: a TensorFlow-enabled Jupyter Notebook with GPU-acceleration! We can even add state, so you can save your work between sessions. Autopilot makes all this really, really easy, as you can configure everything as a Kubernetes object.

Update: this post is now on the Google Cloud blog.

Setup

First, create a GKE Autopilot cluster running 1.24 (1.24.2-gke.1800+ to be exact). Be sure you’re in one of the regions with GPUs (the pricing table helpfully shows which region.

Migrating an IP-based service in GKE

Tue, 23 Aug 2022 22:06:00 -0700

Let’s say you want to migrate a service in GKE from one cluster to another (including between Standard and Autopilot clusters), and keep the same external IP while you do. DNS might be the ideal way to update your service address, for whatever reason you need to keep the IP the same. Fortunately, it is possible to preserve the IP and assign it to a new service in a different cluster in Kubernetes.

Running GKE Autopilot at Scale

Sun, 21 Aug 2022 00:07:09 -0700

If you like what you see with Autopilot (now the recommended way to use GKE), and want to run it at scale, there are a few things to know when setting up the cluster, and scaling your workload.

Cluster Creation

To run a large Autopilot cluster, it’s advisable to do two things: 1) create the cluster with private networking (while not compulsory, public IPs are in short supply, and you may run into quota issues with public networking at large scale). 2) provision a large enough CIDR range for your nodes as this can’t be changed later (/13 to really future proof the cluster).

Provisioning one-off spare capacity for GKE Autopilot

Sun, 21 Aug 2022 00:03:31 -0700

I previously documented how to add spare capacity to an Autopilot Kubernetes cluster, whereby you create a placeholder Deployment to provision some scheduling headroom. This works to constantly give you a certain amount of headroom, so for example if you have a 2vCPU placeholder (a.k.a. balloon) Deployment, and use that capacity it will get rescheduled. A useful concept to add some rapid scaling capabilities to Autopilot.

Update: GKE now has an official guide for provisioning spare capacity.

Kubernetes for Developers: Reader Discount

Sat, 20 Aug 2022 23:41:14 -0700

Readers of my blog can score a special 35% discount for my book Kubernetes for Developers! Simply enter this code at checkout: au35den

High-Performance Compute on Autopilot

Wed, 27 Jul 2022 04:47:07 -0700

This week, Autopilot announced support for the Scale-Out Compute Class, for both x86 and Arm architectures. The point of this compute class is to give you cores for better single-threaded performance, and improved price/performance for “scale-out” workloads — basically for when you are saturating the CPU, and/or need faster single-threaded performance (e.g. remote compilation, etc).

To use, simply add “compute-class: Scale-Out” to your workloads. They can be Arm, or x86 — but pay attention to the available regions. Also note that you need to be using a very new version of Autopilot (see this blog for a CLI command to get you a Scale-Out qualified cluster).

Building Arm Images with Cloud Build

Fri, 15 Jul 2022 18:51:53 -0700

This week’s big news in Google Cloud was the addition to Arm across a wide range of products, including GCE VMs, and GKE (both Standard and Autopilot). In an earlier post, I covered how to get an Arm-ready Autopilot cluster on day 1.

The recommended way to build images for Arm is with buildx. This is also how you can build multi-arch Docker images (ones that can run on both Arm and x86), which is a good idea to do for the time being due to the limited availability of Arm at launch.

Arm on Autopilot

Wed, 13 Jul 2022 18:27:14 -0700

Arm was made available in Preview on Google Cloud, and GKE Autopilot today! As this is an early stage Preview, there’s a few details to pay attention to if you want to try it out, like the version, regions and quota. I put together this quickstart for trying out Arm in Autopilot today.

Arm nodes are available in 3 regions, so be sure to use one from this list:

us-central
europe-west4
asia-southeast1

You’ll also need to create a cluster with a version greater than 1.24.1-gke.1400. This version isn’t the default today, even in the Rapid channel, so you’ll need to create a cluster at a specific version. The following command works at the time of writing, but in the future (once the patch version has been retired), you’ll need to specify a currently valid version newer than that one.

Minimizing Pod Disruption on Autopilot

Tue, 14 Jun 2022 07:58:06 -0700

There are 3 common reasons why a Pod may be terminated on Autopilot: node upgrades, a cluster scale-down, and a node repair. PDBs and graceful termination periods modify the disruption to pods when these events happen, and maintenance windows and exclusions control when upgrade events can occur.

Upgrade

gracefulTerminationPeriod: limited to one hour
PDB: is respected for up to one hour
Maintenance windows: respected
Unmanaged Pod: no impact (won’t prevent upgrade)

Note: under the hood, Autopilot performs upgrades using a node pool upgrade with surge enabled, the documentation of that behavior applies. Unmanaged Pods are those not managed by a higher-order workload controller like Deployment, Job, StatefulSet.

Building GKE Autopilot

Wed, 08 Jun 2022 22:33:26 -0700

Building a Nodeless Kubernetes Platform. Photo Credit: Kaslin Fields

Last month gave a presentation at KubeCon Europe in Valencia on “Building a Nodeless Kubernetes Platform”. In it, I shared the details about the creation of GKE Autopilot including some key decisions that we made, how the product was implemented, and why I believe that the design leads to an ideal fully managed platform.

Transferring a Google Fi eSIM on iOS

Fri, 03 Jun 2022 12:00:00 -0700

Google Fi, despite being originally designed for Android supports iOS, and works really well there. I’ve been using it for years. The only drawback is the lack of 5G (update: it now has 5G!), but for international travel I don’t think there is anything better. The Fi eSIM option is useful too, especially since Fi is the line I never want to remove (since it works everywhere). I found the transfer of eSIM a bit confusing though, so wanted to share the steps that I’ve used a few times now, and found works every time (no pSIM required!):

Preferring Spot in GKE Autopilot

Thu, 26 May 2022 03:41:56 -0700

Spot Pods are a great way to save money on Autopilot, currently 70% off the regular price. The catch is two-fold:

Your workload can be disrupted
There may not always be spot capacity available

For workload disruption, this is simply a judgement call. You should only run workloads that can accept disruption (abrupt termination). If you have a batch job that would lose hours of work, it’s not a good fit. Generally StatefulSet shouldn’t be run on spot compute, as most stateful workloads are less tolerant to disruption.

Separating Workloads in Autopilot

Thu, 26 May 2022 02:14:20 -0700

Autopilot while being operationally nodeless, still creates nodes for your workloads behind the scenes. Sometimes it may be desirable as an operator to separate your workloads so that certain workloads are scheduled on their separates nodes, a technique known as workload separation. One example I heard recently was a cluster that primarily processes large batch jobs. In addition to these spikey workloads that cause a lot of scale-up and scale-downs (something that Autopilot takes care of for you), there are a few “stable” workloads that need to run consistently and are better to have placed by themselves.

Developing in Rails via Docker

Sat, 09 Apr 2022 20:23:40 -0700

One of the neat things about Docker is that you don’t need to install any dev tools on your local machine.

Sometimes though you’ll need to run commands in the dev environment without your app being built (say if there is an error building the app, or you want to run a command like rails new). The way I do this is to have a separate Dockerfile and docker-compose config that configures the dev environment, rather than the app.

Kubernetes Nodes and Autopilot

Tue, 14 Dec 2021 20:20:57 -0700

One of the key design decisions of GKE Autopilot is the fact that we kept the same semantic meaning of the Kubernetes node object.

It’s “nodeless” in the sense that you don’t need to care about, or plan for nodes—they are provisioned and managed automatically based on your PodSpec. However, the node object still exists and has the same meaning. Each node is still backed by a single VM under the hood.

Testing Knative on GKE Autopilot

Mon, 13 Dec 2021 08:24:38 -0700

Ahmet’s blog Did we market Knative wrong? got me interested to try out Knative again. I’ll confess that I found the original Istio requirement a bit of a heavy one just to run serverless, and it looked like the project has matured a lot since the early days. Version 1.0.0 was also just released, so it seems like pretty good timing to give it a go.

Naturally, I wanted to see if it worked on Autopilot. Autopilot is a “nodeless” Kubernetes platform, and Knative offers a serverless layer for Kubernetes, so in theory this can be a pretty good match. I was a bit worried that Knative might require root access to nodes, and thus but up against some Autopilot restrictions. The good news is that despite some complications, it does work!

Install kn on Cloud Shell

Mon, 13 Dec 2021 06:34:23 -0700

Want to run the Knative CLI tool kn in Google Cloud’s Cloud Shell? It’s easy:

Download the latest release from github.

linux-amd64 is the platform you want for CloudShell. Here I download version 1.3.0:

wget "https://github.com/knative/client/releases/download/knative-v1.3.0/kn-linux-amd64"

Then, move it into place, and add it to your PATH:

chmod +x kn-linux-amd64
mkdir -p .local/bin
mv kn-linux-amd64 .local/bin/kn
echo 'export PATH=".local/bin:$PATH"' >> .bashrc
. .bashrc

[Updated 2024-02-15 with version 1.3.0]

Understanding Kubernetes Node Resource Allocation

Wed, 15 Sep 2021 19:08:37 -0700

The question often comes up: where do my node resources go in Kubernetes? In this post, I share how to use kubectl describe node to look at resource usage by different categories (system, user, etc).

The way I think about Kubernetes node resource usage is to assign resource allocations into 4 categories:

OS and Kubernetes overhead
System pods
User pods
Empty space

The OS & Kubernetes overhead includes the Linux kernel, the kubelet, and for memory, an allocation known as the “eviction threshold” which allows the kubelet a chance to catch Pods that are going over their memory allocation and take action according to their Kubernetes priorities before the kernel OOM handling kicks in which is a blunter instrument.

Creating an Autopilot cluster at a specific version

Mon, 16 Aug 2021 23:02:40 -0700

Sometimes you may wish to create or update a GKE Autopilot cluster with a specific version. For example, the big news this week is that mutating webhooks are supported in Autopilot (from version 1.21.3-gke.900). Rather than waiting for your desired version to be the default in your cluster’s release channel, you can update ahead of time.

Through the UI, you can choose the Autopilot release channel but each release channel actually has a few different versions. “Rapid” is the obvious release channel to use for testing new features, but you can be even more bleeding-edge than the default version in the Rapid channel by specifying the version.

Choosing the right network size for Autopilot

Tue, 03 Aug 2021 22:57:32 -0700

Update (2023-09): Autopilot now supports the ability to add additional Pod IP ranges, so you no longer need to plan this up front. Read the official docs, and my test run.

One of the most important decisions you can make for your Autopilot cluster is selecting the right network size. Too small and you’ll constraint your potential growth. Too large and you might run out of IPs if you have a lot of cluster. You can’t change the Pod address size after creation, so it’s important to get it right.

Using GKE Autopilot in specific zones

Wed, 14 Apr 2021 23:02:33 -0700

GKE Autopilot is deployed using the regional cluster architecture. This has a number of advantages such as giving you 3 master nodes for high availability of the control plane, and the ability to spread pods among zones for high availability of your workloads. But sometimes this may be more than what you need, and zonal pattern would work better.

Update: there is now an official doc on this topic.

If you’re dealing with zonal resources, say for example you want to mount a persistent disk into a Pod, where the disk is a zonal disk — you’ll want to target your zone explicitly.

GKE Autopilot: how to know if Pending pods will be scheduled

Fri, 12 Mar 2021 22:15:55 -0700

GKE Autopilot is pretty magical. You create a cluster just by picking a region and giving it a name, schedule Kubernetes workloads and the compute resources are provisioned automatically.

While Kubernetes is provisioning resources, your Pods will be in the Pending state. This is all well and good, except… there are other reasons that your Pods can be pending.

NAME READY STATUS RESTARTS AGE
example-deploy-84576c8598-7jbqc 1/1 Pending 0 34s
example-deploy-84576c8598-7mbfh 1/1 Pending 0 33s
example-deploy-84576c8598-bhfqt 1/1 Pending 0 34s

Pods that are Pending. Will they be provisioned, or won’t they?

Configuring phpMyAdmin with CloudSQL, RDS, and Docker

Thu, 04 Mar 2021 07:36:39 -0700

I still have a place in my heart for phpMyAdmin, that tool that I was introduced when developing PHP on my shared hosting provider many moons ago. It’s just a great way to inspect your MariaDB/MySQL databases.

These days I don’t install any web applications (or databases) on my local machine, I use Docker for everything, and Docker Compose in particular, to get repeatable environments.

Here’s what I use for phpMyAdmin:

Provisioning spare capacity in GKE Autopilot with placeholder balloon pods

Wed, 24 Feb 2021 06:37:05 -0700

Autopilot is a new mode of operation for Google Kubernetes Engine (GKE) where compute capacity is dynamically provisioned based on your pod’s requirements. Among other innovations, it essentially functions as a fully automatic cluster autoscaler.

Update: GKE now has an official guide for provisioning spare capacity.

When you deploy a new pod in this environment, sometimes they’ll be existing spare capacity on the cluster and it will start booting right away, other times more compute capacity (nodes) will need to be added. Autopilot is quick to respond when more resources are needed, provisioning new capacity in around 60 to 80 seconds, after which the pod will boot.