# Running stateful agents on Agent Substrate
[Agent Substrate](https://github.com/agent-substrate/substrate) can run stateful
Google ADK agents across a shared pool of GKE worker Pods. After an agent
responds, the harness can suspend its whole process—including its conversation,
in-memory working state, and files—and Substrate can restore it on whichever
compatible worker is free for the next request.
The neat part is that this doesn't require cold-starting ADK again. A Python ADK
agent can easily take ten seconds or more to initialize, but Substrate's golden
snapshot pays that cost just once per template. Each actor then resumes from its
own latest snapshot instead of rebuilding its process and session. That also
makes the agent stateful: it picks up where it left off, while its previous
worker is free to run another agent.
## Actors and workers
The central idea in Substrate is that an agent and the compute currently
running it are two different things.
An **actor** is the long-lived logical instance. It has an identity, lifecycle
state, and its own snapshots. A **worker** is a warm Kubernetes Pod that can run
one actor at a time. The actor may be running on a worker now, suspended in
object storage a moment later, and restored onto a different worker when its
next request arrives.
That separation is what makes multiplexing possible. Agent processes often
spend a lot of time waiting for the next user turn, yet a normal deployment
keeps their CPU and memory allocation around anyway. Substrate lets a larger
set of actors share a smaller pool of warm workers.
Scroll horizontally to explore. Open full-size diagram
*Figure: A Kubernetes Pod backs one warm Worker. The Worker hosts one running Actor at a time, while suspended Actors occupy no Worker and can resume from their snapshots.*
Substrate's router provides a stable address for each actor:
```
..actors.resources.substrate.ate.dev
```
If the actor is suspended, the router asks the control plane to restore it onto
an eligible free worker and then forwards the request. If every worker is busy,
the router can *park* the request and retry until capacity becomes available.
The caller waits rather than immediately receiving a `503`.
An **atespace** is part of an actor's identity and provides an isolation
boundary. It isn't a Kubernetes namespace, even though this demo happens to use
the same name, `ate-demo-python-adk`, for both.
## Actor templates and golden snapshots
Actors are created from an **ActorTemplate**. This is roughly the actor
equivalent of a Pod template: it describes the container image, command,
environment, resource requirements, sandbox, worker selection, health check,
and snapshot policy. Templates are immutable because their definition is tied
to a particular initial snapshot. To change an agent version, create a new
template rather than editing the old one in place.
Creating a template also creates a **golden snapshot**. Substrate temporarily
boots the workload, waits for it to become ready, then takes a full checkpoint.
New actors from that template can start from the shared checkpoint rather than
repeating the complete cold-start path. Container initialization and Python
imports have already happened once. Pretty neat.
The golden snapshot isn't the actor's evolving state. Once an actor has run,
its most recent suspend creates a separate **last snapshot** for that specific
actor, and that is what its next resume restores. The mental model is:
1. ActorTemplate creation boots a temporary golden actor and captures the
golden snapshot.
2. A new actor starts from that shared golden snapshot.
3. Suspending the actor writes its own latest snapshot and releases its worker.
4. The next request restores that actor-specific snapshot onto any compatible
free worker.
Scroll horizontally to explore. Open full-size diagram
*Figure: The template's golden snapshot is captured once after boot. Each Actor starts there, then advances its own latest snapshot every time a turn completes and the Actor suspends.*
This is both a big part of why Substrate is performant, and what gives its
actors state. A Python ADK agent can easily take ten seconds or more to become
ready: it has to start the interpreter, import its dependencies, initialize the
agent stack, and build a fairly large in-memory object graph before it can serve
a request. The golden snapshot moves that cost from every new actor to once per
template. Restoring an actor rehydrates the already-initialized process, so it
can accept work without replaying the complete boot sequence. Actor-specific
snapshots extend the same advantage across turns: the agent resumes from its
post-turn state rather than restarting and reconstructing its session.
Since Substrate creates a snapshot after each turn, the actor can retain files
and in-memory state for the next request. That is useful for agents built on
frameworks like ADK, and for coding harnesses too.
Scroll horizontally to explore. Open full-size diagram
*Figure: One Actor runs five turns across Worker A and Worker B. Each full snapshot carries its in-memory working context and the `report.md` artifact, so both kinds of state survive every suspension and restore.*
## The demo
I wanted to try this with something a little more agent-like than a counter, so
I put two existing Google ADK agents on Substrate: one that researches [Hacker
News](https://github.com/WilliamDenniss/hnscout), and another,
[QuakeAgent](https://github.com/WilliamDenniss/quakeagent), that analyzes the
USGS earthquake feed. In the run below I create seven actors from each agent,
send every actor a prompt and a follow-up, and run all fourteen workflows against
a pool of just five workers. The complete code is in
[WilliamDenniss/substrate-adk-demo](https://github.com/WilliamDenniss/substrate-adk-demo).
Note: Substrate is in early development. Its own README says it isn't ready for
production use and doesn't promise API compatibility yet. This is a demo on a
disposable cluster, not a production recommendation.
The concrete setup has two ActorTemplates and one five-worker pool. Seven actors
come from the Hacker News template and another seven come from the earthquake
template, but any free worker can run an actor from either one. A worker isn't a
permanent home for one agent image—the template and snapshot tell it what to
restore next.
Each actor receives an initial prompt and a follow-up. The demo runner suspends
the actor after each response, freeing that worker for someone else. When the
follow-up arrives, Substrate restores the actor's latest snapshot and the same
ADK conversation continues. With fourteen actors sharing five workers, the
router also has to park requests while all five workers are occupied.
Here's a video of an example run. You can see the requests interleaved as actors
load, process a turn, and suspend again.
What we saw in the video were five workers serving fourteen actors, with each
actor making two turns. Here's a visual of how those turns interleave:
Scroll horizontally to explore. Open full-size diagram
*Figure: Two different agents share one WorkerPool. Seven Hacker News Actors and seven Quake Actors contribute 28 turns; only five run at once, excess requests are parked, an Actor's next turn can resume on a different Worker, and all five Workers end free.*
Want to run it yourself? [Next up, I'll show how I put this demo together on
GKE.](/agent-substrate-demo/)
