# Running stateful ADK agents on Substrate and GKE [Agent Substrate](https://github.com/agent-substrate/substrate) can run stateful Google ADK agents across a shared pool of GKE worker Pods. After an agent responds, the harness can suspend its whole process—including its conversation, in-memory working state, and files—and Substrate can restore it on whichever compatible worker is free for the next request. The neat part is that this doesn't require cold-starting ADK again. A Python ADK agent can easily take ten seconds or more to initialize, but Substrate's golden snapshot pays that cost just once per template. Each actor then resumes from its own latest snapshot instead of rebuilding its process and session. This also makes it stateful. It picks up where it left off, while its previous worker is free to run another agent. It's really performant, and great for agents that can benefit from a persistant state. I wanted to try this with something a little more agent-like than a counter, so I put two existing Google ADK agents on Substrate: one that researches [Hacker News](https://github.com/WilliamDenniss/hnscout), and another, [QuakeAgent](https://github.com/WilliamDenniss/quakeagent), that analyzes the USGS earthquake feed. In the run below I create seven actors from each agent, send every actor a prompt and a follow-up, and run all fourteen workflows against a pool of just five workers. The complete code is in [WilliamDenniss/substrate-adk-demo](https://github.com/WilliamDenniss/substrate-adk-demo). Here's how it works and how to run it on GKE. Note: Substrate is in early development. Its own README says it isn't ready for production use and doesn't promise API compatibility yet. This is a demo on a disposable cluster, not a production recommendation. ## Actors and workers The central idea in Substrate is that an agent and the compute currently running it are two different things. An **actor** is the long-lived logical instance. It has an identity, lifecycle state, and its own snapshots. A **worker** is a warm Kubernetes Pod that can run one actor at a time. The actor may be running on a worker now, suspended in object storage a moment later, and restored onto a different worker when its next request arrives. That separation is what makes multiplexing possible. Agent processes often spend a lot of time waiting for the next user turn, yet a normal deployment keeps their CPU and memory allocation around anyway. Substrate lets a larger set of actors share a smaller pool of warm workers. In this demo, any of the five workers can run either kind of agent, and all fourteen actors take turns using them. ![A running Actor inside a Substrate Worker Pod, with suspended Actors stored as snapshots](substrate-actor-worker-lifecycle.svg) *Figure: A Kubernetes Pod backs one warm Worker. The Worker hosts one running Actor at a time, while suspended Actors occupy no Worker and can resume from their snapshots.* Substrate's router provides a stable address for each actor: ``` ..actors.resources.substrate.ate.dev ``` If the actor is suspended, the router asks the control plane to restore it onto an eligible free worker and then forwards the request. If every worker is busy, the router can *park* the request and retry until capacity becomes available. The caller waits rather than immediately receiving a `503`. An **atespace** is part of an actor's identity and provides an isolation boundary. It isn't a Kubernetes namespace, even though this demo happens to use the same name, `ate-demo-python-adk`, for both. ## Actor templates and golden snapshots Actors are created from an **ActorTemplate**. This is roughly the actor equivalent of a Pod template: it describes the container image, command, environment, resource requirements, sandbox, worker selection, health check, and snapshot policy. Templates are immutable because their definition is tied to a particular initial snapshot. To change an agent version, create a new template rather than editing the old one in place. Creating a template also creates a **golden snapshot**. Substrate temporarily boots the workload, waits for it to become ready, then takes a full checkpoint. New actors from that template can start from the shared checkpoint rather than repeating the complete cold-start path. Container initialization and Python imports have already happened once. Pretty neat. The golden snapshot isn't the actor's evolving state. Once an actor has run, its most recent suspend creates a separate **last snapshot** for that specific actor, and that is what its next resume restores. The mental model is: 1. ActorTemplate creation boots a temporary golden actor and captures the golden snapshot. 2. A new actor starts from that shared golden snapshot. 3. Suspending the actor writes its own latest snapshot and releases its worker. 4. The next request restores that actor-specific snapshot onto any compatible free worker. ![A timeline showing one shared golden snapshot followed by independent per-Actor snapshots after each turn](substrate-snapshot-timeline.svg) *Figure: The template's golden snapshot is captured once after boot. Each Actor starts there, then advances its own latest snapshot every time a turn completes and the Actor suspends.* This is both a big part of why Substrate is so performant, and what gives Substrate agent actors state. A Python ADK agent can easily take ten seconds or more to become ready: it has to start the interpreter, import its dependencies, initialize the agent stack, and build a fairly large in-memory object graph before it can serve a request. The golden snapshot moves that cost from every new actor to once per template. Restoring an actor rehydrates the already-initialized process, so it can accept work without replaying the complete boot sequence. Actor-specific snapshots extend the same advantage across turns: the agent resumes from its post-turn state rather than restarting and reconstructing its session. Since it is creating a snapshot after each turn, it also makes the actor stateful. You can write artifacts to disk, and even memory, and they'll be there on the next turn. This makes it incredibly useful for agents from ones built on frameworks like ADK to coding harnesses. ![A five-turn timeline showing one Actor retaining in-memory data and a report.md artifact while moving between two Substrate Workers](substrate-single-actor-five-turns.svg) *Figure: One Actor runs five turns across Worker A and Worker B. Each full snapshot carries its in-memory working context and the `report.md` artifact, so both kinds of state survive every suspension and restore.* ## The demo The concrete setup has two ActorTemplates and one five-worker pool. Seven actors come from the Hacker News template and another seven come from the earthquake template, but any free worker can run an actor from either one. A worker isn't a permanent home for one agent image—the template and snapshot tell it what to restore next. Each actor receives an initial prompt and a follow-up. The demo runner suspends the actor after each response, freeing that worker for someone else. When the follow-up arrives, Substrate restores the actor's latest snapshot and the same ADK conversation continues. With fourteen actors sharing five workers, the router also has to park requests while all five workers are occupied. Here's a video of an example run. You can see the requests interleaved as actors load, process a turn, and suspend again. What we saw in the video were five workers serving fourteen actors, with each actor making two turns. Here's a visual of how those turns interleave: ![A timeline showing seven Hacker News Actors and seven Quake Actors multiplexed across five Substrate Workers](substrate-two-agent-multiplexing.svg) *Figure: Two different agents share one WorkerPool. Seven Hacker News Actors and seven Quake Actors contribute 28 turns; only five run at once, excess requests are parked, an Actor's next turn can resume on a different Worker, and all five Workers end free.* ## Run it yourself The rest of the post is the hands-on part. You'll create a disposable GKE cluster, install Substrate with two timeout changes for long-running agent requests, and deploy the two ADK agents. ### Check out both repositories Start with the Substrate and demo repositories next to each other. The demo scripts look for Substrate at `../substrate` by default: ```shell mkdir substrate-adk-on-gke cd substrate-adk-on-gke git clone https://github.com/agent-substrate/substrate.git git clone https://github.com/WilliamDenniss/substrate-adk-demo.git ``` Substrate moves quickly, so use this same checkout for the setup tool, installation, and `kubectl-ate` CLI. I initially had newer demo manifests talking to an older installed control plane; errors about `spec.ateomImage` or missing actor-template references are a sign of that version skew. For a disposable demo, a clean install from the current checkout is much simpler than trying to migrate the old resources. Warning: Substrate is moving quickly with many breaking changes. In fact, in the time it took me to create this demo, there was a change that broke this demo. The current code is based on Substrate commit `f936206dd544d2dcfd190fac61d1fd2fa0a933c0`. If you check out Substrate `HEAD`, you'll likely need to point your coding agent at this demo to refactor it, but the core principles should remain valid. ### Install Substrate on GKE Follow the [GKE quickstart](https://github.com/agent-substrate/substrate#gke-quickstart-development) to install Substrate on a GKE cluster. Configure the development environment, then let the setup tool create the GKE cluster, GCS bucket and IAM bindings: ```shell cd substrate cp hack/ate-dev-env.sh.example .ate-dev-env.sh # Edit .ate-dev-env.sh for your project, cluster, bucket and KO_DOCKER_REPO. source .ate-dev-env.sh gcloud auth application-default login --project="${PROJECT_ID}" go run ./tools/setup-gcp bootstrap ``` The setup tool creates the cluster with the Kubernetes beta certificate APIs Substrate requires. Those APIs must be enabled when the GKE cluster is created, so I would use the supplied bootstrap path for a first test rather than trying to retrofit an existing cluster. `KO_DOCKER_REPO` must point to a container repository you can push to. The setup tool grants the required image-pull permissions, but it doesn't create the repository itself. This is where `ko` publishes Substrate's control-plane and gVisor worker images so GKE can pull them. If you're using a new Artifact Registry repository, create it once and configure Docker authentication before installing Substrate. This example matches the `KO_DOCKER_REPO` value used later in the demo: ```shell gcloud artifacts repositories create ate-images \ --repository-format=docker \ --location="${GCE_REGION}" \ --project="${PROJECT_ID}" gcloud auth configure-docker "${GCE_REGION}-docker.pkg.dev" ``` #### Two timeout changes for LLM requests I made two behavioral changes to `manifests/ate-install/atenet-router.yaml` to work better with this demo (allowing for longer LLM response times and more queuing). I recommend applying them before installing Substrate: 1. Increase `--parked-request-budget` from its five-second default to one minute. An LLM turn can occupy all five workers for much longer than five seconds, so queued actors need more time for another actor to finish and suspend. 2. Enable a five-minute `--route-timeout`. The normal end-to-end route timeout is too short for a model response that holds the HTTP request open throughout generation. The relevant diff is: ```diff spec: template: spec: containers: - name: atenet-router args: + - "--parked-request-budget=1m" # Long-running model calls hold the route open until generation ends. - # - "--route-timeout=5m" + - "--route-timeout=5m" ``` These are demo values. A production value should come from your observed model latency and the amount of queuing you're willing to accept. A longer queue can absorb a short capacity shortage, but it also means clients wait longer before they receive an error. With those changes in place, install the system from the same checkout: ```shell ./hack/install-ate.sh --deploy-ate-system ``` ### How the demo is wired Both containers run ADK's standard API server. The ActorTemplates override the images' command with the equivalent of: ```dockerfile CMD ["adk", "api_server", "--host=0.0.0.0", "--port=80", "/app"] ``` Port 80 is Substrate's default HTTP ingress port for actors. The absolute `/app` is also deliberate: the current runtime starts the process with `/` as its working directory, so using `.` makes ADK search the wrong directory and the eventual `/run` call returns `404`. The news scout agent container image exposes the `hn_signal_scout` ADK app, while the earthquake agent container image exposes `quake_agent`. The two images are already published as `docker.io/wdenniss/hnscout:latest` and `docker.io/wdenniss/quakeagent:latest`; the Artifact Registry repository configured during setup is for Substrate's images, not these agent images. (Hopefully once Substrate is a little more mature there will be released artifacts you can use directly). Both templates are configured to use `Full` snapshots, which capture process memory, the changed root filesystem, and durable-volume data. That's important for ADK because its in-process session service and local artifacts survive the suspend/resume cycle. Substrate also supports `Data` snapshots for applications that only need durable volume contents and can cold boot the process again. The GCS bucket configured during setup stores the golden and per-actor snapshots.
Golden and per-actor snapshots stored in the configured GCS bucket
*Image: Golden and per-actor snapshots stored in the configured GCS bucket* Both agents call Gemini through Gemini Enterprise Agent Platform (formerly known as Vertex AI). Their Google client libraries use Application Default Credentials (ADC), which on GKE resolves to short-lived credentials supplied through Workload Identity Federation for GKE. There is no `GOOGLE_API_KEY` in the templates. Instead, the workload proves its identity and receives short-lived access tokens rather than carrying a reusable bearer secret. On Google Cloud, IAM and Workload Identity can be configured as part of deployment, so the application obtains credentials automatically. The demo grants the shared Substrate egress identity permission to call Gemini Enterprise Agent Platform. Matching labels let every worker run an actor from either template. Substrate automatically resumes an actor when traffic arrives, but it doesn't decide when an arbitrary HTTP application is idle. Completing an ADK response therefore doesn't automatically suspend the actor. The demo runner explicitly suspends each actor after every model response; a real agent harness needs to do the same, or implement its own idle policy. The counter demo also suspends explicitly—this isn't an automatic side effect of finishing an HTTP request. ### Configure and run the agents Move into the demo checkout and create its local environment file: ```shell cd ../substrate-adk-demo cp .env.example .env ``` Set these values in `.env`, using the same project, bucket and image repository you configured for Substrate: ```shell GOOGLE_CLOUD_PROJECT=your-project-id GOOGLE_CLOUD_LOCATION=global BUCKET_NAME=your-substrate-snapshot-bucket KO_DOCKER_REPO=us-west1-docker.pkg.dev/your-project-id/ate-images ``` Now grant the demo's model permission and validate the configuration: ```shell make grant-auth make validate ``` The `make` targets are just short names for shell scripts; Substrate doesn't depend on Make. I find `make actor-demo` easier to remember, but `./scripts/actor-demo.sh` does the same thing. Keep the router private and port-forward it in another terminal: ```shell kubectl port-forward -n ate-system service/atenet-router 8000:80 ``` With that terminal running, the concrete demo lifecycle is four commands. Run them one at a time; after `make deploy` is a good time to open the monitoring terminals in the next section. ```shell make deploy make actor-demo ACTORS_PER_AGENT=7 make actor-cleanup make cleanup ``` `make deploy` creates the five-worker pool, creates both ActorTemplates, and waits for both golden snapshots. It doesn't create any ordinary actors, so all five workers are idle when deployment finishes. The wait for a golden snapshot is real work—the agent container has to boot, pass its health check and be checkpointed—so it can take a little while. If the template enters a failed state, inspect that error rather than waiting indefinitely. The second command creates seven Hacker News actors and seven earthquake actors, then submits all fourteen two-turn workflows concurrently. The five-worker pool runs five actors at a time while the router parks the excess requests. Each actor receives a different first prompt and a follow-up that depends on the first result. `make actor-cleanup` removes those fourteen numbered actors but keeps the templates and worker pool. Finally, `make cleanup` removes the remaining demo resources. This gives you a useful pause between the third and fourth commands to confirm that selective actor cleanup really did leave the infrastructure alone. If you omit `ACTORS_PER_AGENT`, the starting value is five per agent (ten actors total). You can also limit how many workflows the client submits at once: ```shell make actor-demo ACTORS_PER_AGENT=7 MAX_CONCURRENCY=8 ``` `MAX_CONCURRENCY` doesn't change the WorkerPool. If client concurrency is no higher than worker capacity, there won't be excess requests for the router to park. If you want a smaller check after `make deploy`, run `make smoke`. It creates or reuses one actor from each template, sends one real model prompt to each, prints the results, then suspends both actors. ### What to watch This demo is more interesting with a few terminals open before you run `make actor-demo`. First, install `kubectl-ate` as a kubectl plugin if you haven't already: ```shell cd ../substrate go install ./cmd/kubectl-ate cd ../substrate-adk-demo ``` If `kubectl` says `unknown command "ate"`, the resulting `kubectl-ate` binary isn't on your `PATH`. You can call `kubectl-ate` directly, or add your Go binary directory to `PATH`. Now watch the physical Pods, logical actors, and workers: ```shell watch -d 'kubectl get pods -n ate-demo-python-adk' watch -d 'kubectl ate get actors -a ate-demo-python-adk' watch -d 'kubectl ate get workers' ``` On macOS, `watch` is available from Homebrew (`brew install watch`). As the demo runs, only five workers can be assigned at once. Actor states move between `RUNNING` and `SUSPENDED`, and a later resume may place an actor on a different worker. Once the demo finishes, every numbered actor should be suspended and all five workers should be free. To see request parking, expose the router's status port in another terminal: ```shell kubectl -n ate-system port-forward deployment/atenet-router 4040:4040 ``` Then watch the active parked-request count while `make actor-demo` is running: 
watch -d "curl -fsS 'http://127.0.0.1:4040/statusz?format=json' | jq -c '.parking'"
The useful field is `.parking.active`. It should rise while all workers are occupied and fall as actors finish, suspend, and release capacity. If you look only after the run, it will quite correctly be zero. The demo runner prints the actual model responses. In my run, one earthquake actor found the deepest event, suspended, then handled this follow-up after it was restored: ``` Prompt: Convert the depth you just reported from kilometers to miles and show the calculation. Result: 64.644 km × 0.621371 = 40.167 miles ``` The exact earthquake and Hacker News results will of course change. The useful part is that the follow-up understands *the depth you just reported* even though the process was snapshotted and the worker released between turns. For the container's stdout and stderr while an actor is active, use the actor-aware log command: ```shell kubectl ate logs actors quakeagent-4 -a ate-demo-python-adk -f ``` That follows the actor rather than a particular Pod. A suspended actor has no active Pod to query, so for this short demo the labeled API responses printed by `make actor-demo` are usually the easiest view. Substrate also adds actor, atespace, template, and container metadata to structured logs for a centralized logging backend; the [observability guide](https://github.com/agent-substrate/substrate/blob/main/docs/observability.md) covers that model in more detail. You can also scale the warm worker capacity using the normal Kubernetes scale subresource: ```shell kubectl scale workerpool/python-adk \ -n ate-demo-python-adk \ --replicas=8 kubectl rollout status deployment/python-adk -n ate-demo-python-adk ``` Run the demo again and you'll see less parking because eight actors can be active at once. A later `make deploy` reapplies the demo manifest's default of five workers. ### Cleanup To remove only the numbered actors created by `make actor-demo`, while keeping the templates, pool, and any suspended smoke-test actors: ```shell make actor-cleanup ``` To remove the full demo and revoke its Gemini Enterprise Agent Platform permission: ```shell make cleanup make revoke-auth ``` The cleanup deliberately retains the snapshot objects in GCS, so delete that prefix or tear down the disposable project when you're done. The shared `atenet-egress` identity used here is also intentionally demo-only: every actor routed through it can obtain the same Google Cloud credentials. A production design needs actor-aware credential isolation, and the unauthenticated ADK API server should never be exposed publicly as configured here.