Cloud Intelligence™Cloud Intelligence™

Cloud Intelligence™

Your Agents Are Idle. Your Kubernetes Bill Isn’t.

Kubernetes has no native concept of a sleeping Pod. Agent Substrate multiplexes thousands of stateful agents onto a shared worker pool, without the cold-start penalty.

This page is also available in Deutsch, Español, Français, Italiano, 日本語, and Português.

Sep 24, 20268 min read
Chimbu Chinnadurai

About Chimbu Chinnadurai

Senior Cloud Architect II

I've probably debugged a Kubernetes issue in more time zones than I care to count. Based in London, I help engineering teams across EMEA get their clusters to behave — and actually understand why they misbehaved in the first place.

I write, speak, and guest on podcasts about all things cloud-native. Away from the terminal: I enjoy cooking almost as much as simplifying overly complex systems.

My personal page

Interactive agents, personal assistants, coding agents, and tool-calling sandboxes spend most of their runtime waiting on a human or an external trigger. They're not doing any actual compute in that window. On Kubernetes, that idle time still costs real resources, capacity that could otherwise be serving active work.

Standard Kubernetes ties every workload to a dedicated Pod. Pod scheduling throughput and startup latency set a hard ceiling, and Kubernetes has no native concept of hibernating a Pod. Park millions of idle agents as live Pods and you'll exhaust Pod limits and control-plane memory long before you exhaust actual compute. The usual workaround, tearing Pods down and reconstructing agent state from external storage on every wakeup, works. But it means every team rebuilds the same state-externalization plumbing and pays full cold-start cost on every resume.

Agent Substrate takes a different approach: suspend idle agents, snapshot their RAM and local files, and restore that state onto an available sandbox in under a second the moment the agent needs to act again.

Note: Agent Substrate is in early development and not ready for production use. The APIs are expected to change, and backward compatibility is not guaranteed.

What Agent Substrate is

Agent Substrate is a secure-by-default agent execution runtime built to run millions of sandboxes at 10x the density of standard container runtimes, with sub-500ms resume and 500+ suspend/resume activations per second. Isolation comes from gVisor by default, with an optional micro-VM sandbox class (Kata Containers plus Cloud Hypervisor) for workloads that need a harder boundary.

The core idea is multiplexing. Substrate maps a large number of actors, your agent instances, onto a much smaller pool of workers, the sandboxes that actually execute them, on the assumption that agent-like workloads spend most of their time idle.

media

A few terms worth knowing before anything else clicks into place:

  • Actor: one running instance of an agent
  • ActorTemplate: the immutable blueprint (container image, environment, snapshot policy) used to create actors of a given version
  • Worker: the sandbox pod where an active actor executes
  • WorkerPool: a fleet of pre-warmed workers ready to receive an actor
  • Atespace: a namespace-like grouping that actors and templates live in

The reason this can run so much denser than one-Pod-per-agent is that Substrate takes the Kubernetes control plane out of the critical path. Kubernetes still provisions and manages the underlying Pods, Substrate just doesn't route every suspend and resume cycle through Pod scheduling. Its own control plane handles that directly: ateapi for actor and worker lifecycle, atecontroller reconciling WorkerPools, atenet for DNS and routing, atelet running as a node-level DaemonSet.

Agent Substrate works with any framework that produces standard OCI containers. The project lists integration patterns for the Agent Development Kit, LangChain, and Claude Code. It isn't an agent framework or SDK itself, but the substrate underneath one.

Why it's worth the complexity

Untrusted code, whether AI-generated or a one-off tool call, runs inside kernel and network isolation without exposing the rest of the cluster. An actor's working memory and files persist across suspend cycles, so it resumes exactly where it paused instead of starting over. Restore is fast enough, a fraction of a second, that response latency stays acceptable even for bursty, human-facing agents. And because a shared pool of warm workers serves many actors, you're not paying for compute that idle actors aren't using.

That combination fits a specific set of workloads well. Background assistants that hold context over days but are only active in short bursts get most of their savings from suspending between those bursts. Disposable sandboxes for running untrusted LLM-generated code benefit from the sub-second restore: spin one up, run the task, release it, without a full cold boot each time. Coding agents that hold a real-time back-and-forth with a developer keep their terminal and filesystem state intact between prompts, while the cluster only spends compute on developers who are actually active.

How a request actually flows through it

When an actor is created from a template, Substrate spins up a temporary "golden pod," runs your container's startup logic once, and takes a golden snapshot the moment it's initialized. Every subsequent actor of that template resumes from that snapshot rather than booting cold, which is why expensive initialization (loading a model, opening baseline connections) belongs in your entry point rather than somewhere it'd be repeated on every wakeup.

media

One caveat worth flagging: Substrate doesn't currently detect idleness on its own. Suspension is triggered by an explicit API call from whatever system is orchestrating the actors, not by Substrate watching activity and deciding for itself.

Configuring it: WorkerPool and ActorTemplate

Substrate resources are Kubernetes CRDs under the ate.dev/v1alpha1 API group, so they fit into a normal GitOps flow. Two resources do most of the work.

A WorkerPool defines the physical capacity: how many warm pods to keep standing by, which sandbox runtime they use, and any node placement rules.

apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
name: research-agent-pool
namespace: ate-demo
spec:
replicas: 10
ateomImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor
# sandboxClass defaults to gvisor; set to microvm for a Kata + Cloud
# Hypervisor pool if your workload needs stronger isolation.

An ActorTemplate defines the workload itself: its container, where snapshots get stored, and which worker pools it's allowed to run on.

apiVersion: ate.dev/v1alpha1
kind: ActorTemplate
metadata:
name: research-agent
namespace: ate-demo
spec:
pauseImage: "gcr.io/gke-release/pause@sha256:<digest>"
containers:
- name: agent
image: registry.example.com/research-agent@sha256:<digest>
readyz:
httpGet:
path: /readyz
port: 8080
sandboxClass: gvisor
workerSelector:
matchLabels:
workload: research-agent
snapshotsConfig:
location: gs://my-bucket/snapshots/research-agent/

A few details worth knowing before you write your own. Container images must be pinned by digest, since changing the image invalidates existing snapshots. The readyz probe is optional but worth setting, it gates both cold boot and restore on your workload actually being ready to serve traffic, not just having started. And each new version of your agent should get its own ActorTemplate (research-agent-v2, and so on), since Substrate treats a template as an immutable state root rather than something you mutate in place.

Once the template reaches Ready, you create an actor logically with the CLI:

Terminal window
kubectl ate create atespace demo
kubectl ate create actor my-agent-1 -a demo --template=research-agent

and it resumes on any free worker in the referenced pool the moment a request arrives.

Cloud bill shouldn't be a mystery

One platform for AI and Cloud optimization.

Where to go for the details

The project's own documentation is the right place to configure this for real, rather than relying on a blog post to stay current with a system whose APIs are explicitly still moving:

  • README and quickstart, including a local kind-based setup that needs nothing beyond Go, kubectl, and Docker
  • API Configuration Guide for the full WorkerPool, ActorTemplate, and SandboxConfig field reference
  • Architecture guide for how the control plane, node supervisor, and networking stack fit together
  • Demos directory for worked examples: a stateful counter, a sandboxed shell environment, a Claude Code multiplexing demo, and an autoscaled worker pool driven by an HPA

Running it on GKE specifically

Everything above works on any Kubernetes cluster. If your infrastructure happens to be GKE, Google has built a guided on-ramp for it too. As of September 11, 2026, Agent Substrate on GKE is open to all Google Cloud customers for evaluation and non-production use, with production support gated behind an allowlist under a limited GA program.

The fastest path is a single installer in the ai-on-gke/substrate-gke repo. It walks you through provisioning or targeting a GKE cluster, then enables the configurations Substrate needs and deploys the control plane onto it.

Terminal window
curl -sSL https://raw.githubusercontent.com/ai-on-gke/substrate-gke/main/install.sh | bash

media

Google's own Agent Substrate on GKE documentation and install guide cover the rest.

The cost-attribution problem this creates

Multiplexing thousands of actors onto a shared WorkerPool solves the idle-compute problem. It creates a different one: standard cost attribution stops working. Tag-based FinOps tooling assumes a reasonably stable mapping between a Pod and the team or customer it serves. Once actors are being suspended, resumed onto whichever worker happens to be free, and sharing a pool with hundreds of other actors, that mapping is gone. The compute line item on a shared WorkerPool also tells you nothing about the LLM token spend each actor is individually driving through whatever model or gateway it calls.

DoiT Attribute™ is built for that specific gap. It reads gateway traffic at runtime with an eBPF sensor rather than relying on tags, and traces each inference call back to the agent, feature, or customer that triggered it, separating human traffic from non-human (agent) traffic in the process. For a platform running large numbers of Substrate actors against shared LLM gateways, that's the difference between knowing your aggregate AI spend and knowing which actor, customer, or feature is actually driving it.

Closing remarks

Agent Substrate is solving a real problem: agent workloads sit idle most of the time, and neither Kubernetes nor cloud billing accounts for that. It's also genuinely early. The project's own docs describe it as not ready for production use, and the APIs are expected to keep changing; idle detection itself is something you still have to wire up yourself, not something Substrate does for you.

If you're running it against shared LLM gateways at any real volume, knowing what each actor actually costs is the next problem you'll hit.

Book a demo of Attribute →