Agent Substrate can run stateful Google ADK agents across a shared pool of GKE worker Pods. After an agent responds, the harness can suspend its whole process—including its conversation, in-memory working state, and files—and Substrate can restore it on whichever compatible worker is free for the next request.

The neat part is that this doesn’t require cold-starting ADK again. A Python ADK agent can easily take ten seconds or more to initialize, but Substrate’s golden snapshot pays that cost just once per template. Each actor then resumes from its own latest snapshot instead of rebuilding its process and session. That also makes the agent stateful: it picks up where it left off, while its previous worker is free to run another agent.

Actors and workers

The central idea in Substrate is that an agent and the compute currently running it are two different things.

An actor is the long-lived logical instance. It has an identity, lifecycle state, and its own snapshots. A worker is a warm Kubernetes Pod that can run one actor at a time. The actor may be running on a worker now, suspended in object storage a moment later, and restored onto a different worker when its next request arrives.

That separation is what makes multiplexing possible. Agent processes often spend a lot of time waiting for the next user turn, yet a normal deployment keeps their CPU and memory allocation around anyway. Substrate lets a larger set of actors share a smaller pool of warm workers.

Figure: A Kubernetes Pod backs one warm Worker. The Worker hosts one running Actor at a time, while suspended Actors occupy no Worker and can resume from their snapshots.

Substrate’s router provides a stable address for each actor:

<actor>.<atespace>.actors.resources.substrate.ate.dev

If the actor is suspended, the router asks the control plane to restore it onto an eligible free worker and then forwards the request. If every worker is busy, the router can park the request and retry until capacity becomes available. The caller waits rather than immediately receiving a 503.

An atespace is part of an actor’s identity and provides an isolation boundary. It isn’t a Kubernetes namespace, even though this demo happens to use the same name, ate-demo-python-adk, for both.

Actor templates and golden snapshots

Actors are created from an ActorTemplate. This is roughly the actor equivalent of a Pod template: it describes the container image, command, environment, resource requirements, sandbox, worker selection, health check, and snapshot policy. Templates are immutable because their definition is tied to a particular initial snapshot. To change an agent version, create a new template rather than editing the old one in place.

Creating a template also creates a golden snapshot. Substrate temporarily boots the workload, waits for it to become ready, then takes a full checkpoint. New actors from that template can start from the shared checkpoint rather than repeating the complete cold-start path. Container initialization and Python imports have already happened once. Pretty neat.

The golden snapshot isn’t the actor’s evolving state. Once an actor has run, its most recent suspend creates a separate last snapshot for that specific actor, and that is what its next resume restores. The mental model is:

1. ActorTemplate creation boots a temporary golden actor and captures the golden snapshot.
2. A new actor starts from that shared golden snapshot.
3. Suspending the actor writes its own latest snapshot and releases its worker.
4. The next request restores that actor-specific snapshot onto any compatible free worker.

Figure: The template’s golden snapshot is captured once after boot. Each Actor starts there, then advances its own latest snapshot every time a turn completes and the Actor suspends.

This is both a big part of why Substrate is performant, and what gives its actors state. A Python ADK agent can easily take ten seconds or more to become ready: it has to start the interpreter, import its dependencies, initialize the agent stack, and build a fairly large in-memory object graph before it can serve a request. The golden snapshot moves that cost from every new actor to once per template. Restoring an actor rehydrates the already-initialized process, so it can accept work without replaying the complete boot sequence. Actor-specific snapshots extend the same advantage across turns: the agent resumes from its post-turn state rather than restarting and reconstructing its session.

Since Substrate creates a snapshot after each turn, the actor can retain files and in-memory state for the next request. That is useful for agents built on frameworks like ADK, and for coding harnesses too.

Figure: One Actor runs five turns across Worker A and Worker B. Each full snapshot carries its in-memory working context and the report.md artifact, so both kinds of state survive every suspension and restore.

The demo

I wanted to try this with something a little more agent-like than a counter, so I put two existing Google ADK agents on Substrate: one that researches Hacker News, and another, QuakeAgent, that analyzes the USGS earthquake feed. In the run below I create seven actors from each agent, send every actor a prompt and a follow-up, and run all fourteen workflows against a pool of just five workers. The complete code is in WilliamDenniss/substrate-adk-demo.

The concrete setup has two ActorTemplates and one five-worker pool. Seven actors come from the Hacker News template and another seven come from the earthquake template, but any free worker can run an actor from either one. A worker isn’t a permanent home for one agent image—the template and snapshot tell it what to restore next.

Each actor receives an initial prompt and a follow-up. The demo runner suspends the actor after each response, freeing that worker for someone else. When the follow-up arrives, Substrate restores the actor’s latest snapshot and the same ADK conversation continues. With fourteen actors sharing five workers, the router also has to park requests while all five workers are occupied.

Here’s a video of an example run. You can see the requests interleaved as actors load, process a turn, and suspend again.

What we saw in the video were five workers serving fourteen actors, with each actor making two turns. Here’s a visual of how those turns interleave:

Figure: Two different agents share one WorkerPool. Seven Hacker News Actors and seven Quake Actors contribute 28 turns; only five run at once, excess requests are parked, an Actor’s next turn can resume on a different Worker, and all five Workers end free.

Want to run it yourself? Next up, I’ll show how I put this demo together on GKE.