Manuals / Agent Substrate 101 / Part 7 · Integrations
7.1

kagent 1.0 on Substrate

kagent makes each Session one actor, resumes it for every turn, and suspends it after, so a turn pays half a second beside the model and keeps only files.

You install kagent 1.0 and send your agent a message. The answer comes back in about a second. Then you run kubectl get pods, and you see one worker pod and no pod for your agent. The agent is an actor, and kagent suspended it again before you looked.

When you finish this section, you can install kagent 1.0, write the three objects that make an agent, and predict what one turn costs. You can also say what a kagent snapshot keeps and which hosts the agent can reach.

We ran kagent v1.0.0. kagent v1.0.1 came out on the day of our run, 2026-11-12, and we did not test it. We ran it twice. In the first run the model was a mock server in the cluster. Those times exclude model time and show the cost of Substrate alone. In the second run the model was a small open model on CPU. The subsection on a turn with a real model reports it, and every other number in this section is from the first run.

The install, as we ran it

The documented CLI path worked with no fix. We used a kind cluster on Kubernetes 1.37 and one command. We found no document of kagent v1.0.0 that requires that Kubernetes version, and we tested no other.

StepSeconds
kind create cluster --name kagent --image kindest/node:v1.37.026.1
download the kagent CLI v1.0.05.5
kagent --config kagent-config.yaml install --namespace kagent216.1

We measured each step once, on a second x86 server guest (16 vCPU, 31 GB). The install command took 216 s and ended with exit code 0. Raw runs

The install command reads one small file.

yamlthe installer configuration that we usedFull log
install:
  kagent:
    valuesFiles: [kagent-values.yaml]
The complete file. The values file that it names holds one setting, grpcReflection true under config.server.

kagent does not use a Substrate that you installed before. It installs its own distribution of Substrate, from images named ghcr.io/kagent-dev/substrate/*:v1.0.0. The install made six Helm releases: kagent-crds, substrate-crds, substrate-podcert, substrate-atelet, substrate, and kagent. In this distribution, atenet-router and atenet-egress each run one agentgateway v1.7.0 container. It has no Envoy.

The Substrate charts are not in the kagent repository. The installer pulls them from the chart registry of the kagent project. The kagent code builds against the kagent copy of the Substrate repository at v1.0.0. The installer also sets four Substrate values that change how the install behaves.

What the kagent installer setsEffect
The preview features PerTemplateSnapshotLocation and GoldenEgressPolicyBoth are on. A plain ate-setup install has them off.
Two bootstrap owners: the identity of the kagent controller, and ate-clientThe kagent controller owns the whole install, as kubectl ate does.
The ingress allowlist of the router: the SPIFFE ID of the kagent controllerOnly the kagent controller can send requests to actors. The atenet-client identity of an ate-setup install is not in the list.
A namespace policy of the credential provider: atespace kagent to namespace kagentActors in the atespace kagent can get Secrets of the namespace kagent.

The install created one WorkerPool, kagent-default, with one worker pod. It created no default Harness. Advice that tells you to select a Harness named kagent-default describes a later build. At v1.0.0 you write the Harness.

The three objects you write

An agent at v1.0.0 is three Kubernetes objects in the group api.kagent.dev/v1alpha3. YAML with the group kagent.dev is older than 1.0 and does not apply.

yamlthe Harness, the AgentTemplate, and the Agent of our runFull log
…
apiVersion: api.kagent.dev/v1alpha3
kind: Harness
metadata:
  name: lab-harness
  namespace: kagent
spec:
  kagent: {}
  substrate:
    workerPoolRef:
      name: kagent-default
    snapshotPolicy:
      location: s3://ate-snapshots/kagent/
---
apiVersion: api.kagent.dev/v1alpha3
kind: AgentTemplate
metadata:
  name: lab-template
  namespace: kagent
spec:
  description: Lab agent that runs shell commands for the Substrate manual experiments
  modelConfig:
    name: mock-model
  systemPrompt: You are a lab agent. Use the bash tool when the user asks you to run a command.
---
apiVersion: api.kagent.dev/v1alpha3
kind: Agent
metadata:
  name: lab-agent
  namespace: kagent
spec:
  template:
    ref:
      name: lab-template
  harnessRef:
    name: lab-harness
The ModelConfig above these three objects is cut. It names the mock model server.

Harness: the runtime that executes the agent, with the WorkerPool and the snapshot location that it uses. kagent: {} selects the engine of kagent. The other types are codex, claude, and byo. The codex and claude harnesses are planned for the next edition.

AgentTemplate: the model, the prompt, and the tools. It has no Substrate field.

Agent: the pair of one AgentTemplate and one Harness. kagent compiles each revision of that pair into one ActorTemplate through the Substrate API. Ours was lab-agent-7f643ac37f02.

The Agent was Ready about 24 s after we applied it. The golden snapshot build used 9.84 s of that time. Substrate has no Actor CRD and no ActorTemplate CRD, so kubectl get shows neither object. Use kubectl ate.

One Session is one Actor

Session: one conversation with an Agent. kagent creates one actor for it, named session-<uuid>, in the atespace kagent. The atespace has the name of the Kubernetes namespace of the Agent.

The actor is created in ACTOR_STATE_SUSPENDED, and kagent does not resume it. The first message does. A Session create took 0.39 s at the median over 20 creates.

Fig. 7.1one recorded kagent turn, from the message to the answersequence
One recorded kagent turn resumes the Session actor, runs, and suspends it again Five lifelines, each under a small drawing of its party: the client as a terminal, the kagent controller as a server with the kagent mark, atenet-router as a gateway with the agentgateway mark, ate-api-server as a server with the Agent Substrate mark, and the actor named session-uuid as three layers. Nine numbered exchanges show one recorded turn, each with its time from the moment the client sent the message. The router asks for ResumeActor and the actor is RUNNING 0.41 s later. The turn runs. kagent calls SuspendActor, the client has its answer after 1.26 s, and the actor is SUSPENDED after 1.36 s. TIME client kagent CLI kagent controller :8083 atenet-router agentgateway ate-api-server ate-system actor session-<uuid> 1 +0.00 s message A2A, with the Session id 2 +0.32 s ate-target-actor mTLS, port 443 3 +0.47 s ResumeActor the router asks 4 +0.88 s RUNNING resumed in 0.41 s 5 the turn gRPC to port 80 of the actor 6 +1.07 s TASK_STATE_COMPLETED the turn ran 0.19 s 7 +1.23 s SuspendActor synchronous, files only 8 +1.26 s answer 1.26 s at the client 9 +1.36 s SUSPENDED suspended in 0.13 s One recorded turn of 98. Over all of them the resume took 0.380 s at the median, and the suspend took 0.142 s. The model was a mock server.
Every turn resumes the Session actor through the router, and kagent calls SuspendActor as soon as the task completes. Read top to bottom. The drawing above each lifeline is its party, with the mark of its project. The left column is the time of each step in one recorded turn, counted from the message. Amber arrows are state changes. The model was a mock server.

The log of ate-api-server shows the cycle. These lines are one Session: the create, then two turns that were 4 s apart.

outputone Session in the state log, create and two turnsFull log
2026-11-12T10:20:35.633350007Z	session-01a1cf46-0786-748f-ad16-cc3bb0e1f2e1	create	suspended
2026-11-12T10:20:36.067645235Z	session-01a1cf46-0786-748f-ad16-cc3bb0e1f2e1	resume	resuming
2026-11-12T10:20:36.485975028Z	session-01a1cf46-0786-748f-ad16-cc3bb0e1f2e1	resume	running
2026-11-12T10:20:36.862693911Z	session-01a1cf46-0786-748f-ad16-cc3bb0e1f2e1	suspend	suspending
2026-11-12T10:20:36.995534634Z	session-01a1cf46-0786-748f-ad16-cc3bb0e1f2e1	suspend	suspended
2026-11-12T10:20:41.387075902Z	session-01a1cf46-0786-748f-ad16-cc3bb0e1f2e1	resume	resuming
2026-11-12T10:20:41.796382831Z	session-01a1cf46-0786-748f-ad16-cc3bb0e1f2e1	resume	running
2026-11-12T10:20:42.143125306Z	session-01a1cf46-0786-748f-ad16-cc3bb0e1f2e1	suspend	suspending
2026-11-12T10:20:42.270402046Z	session-01a1cf46-0786-748f-ad16-cc3bb0e1f2e1	suspend	suspended
The columns are time, actor, operation, and new state.

All 98 turns of our run had this shape: resume, run, suspend. Substrate does not suspend an idle actor, as SuspendActor and ResumeActor shows. kagent does that work for you, after every turn.

What one turn costs

Fig. 7.2every resume, suspend, and turn of the kagent runchart
A kagent turn costs the same with a pause and with no pause Every run of our kagent test with a mock model, as one dot each, with the median as a line. Resume 0.380 s over 98 turns, suspend 0.142 s over 99 suspends, a turn 4 seconds after the last one 0.726 s, a turn with no pause 0.738 s, and the first turn of a new Session 0.605 s. 0 0.25 0.5 0.75 1 seconds for each run, with a mock model STATE CHANGES INSIDE A TURN THE WHOLE TURN, IN THE CLUSTER 0.380 s resume 98 runs 0.142 s suspend 99 runs 0.726 s turn, 4 s apart 20 runs 0.738 s turn, no pause 20 runs 0.605 s first turn 10 runs no warm turn: 0.012 s apart one dot: one run, sorted from the fastest to the slowest. black line: the median.
A turn that follows the last one with no pause costs the same as a turn 4 s later, because every turn resumes the actor. Each dot is one run, and each column is sorted from the fastest run to the slowest. The black line is the median. Amber columns are the two state changes inside a turn. Plum columns are the whole turn as kagent reports it.

We measured 98 turns on the same second x86 server guest (16 vCPU, 31 GB), with a mock model. The resume took 0.380 s at the median, from 0.216 to 0.542 s. The suspend took 0.142 s at the median over 99 suspends, from 0.120 to 0.398 s. A whole turn took 0.726 s in the cluster and 1.224 s at the client, over 20 turns spaced 4 s apart. Raw runs

No warm turn exists. Twenty turns with no pause took 0.738 s at the median, against 0.726 s for the spaced turns. The uptime inside the sandbox read 0.6 to 0.9 s on every turn. So you pay about 0.52 s of resume and suspend on each turn, and a real model adds its own time to that. The next subsection measures both.

A files-only resume is this fast only because the kagent harness starts fast. Every such resume is a new start of the image. We ran a Python agent at the same fidelity on the same guest, on the cluster of the Part 9 demos. It needed 4.5 to 5.1 s for each turn after a suspend. Snapshot fidelity compares the cases.

One of the 98 turns, turn 3 of the first session, took 6.56 s, with a gap of 5.5 s before the resume. We did not find the cause.

A turn with a real model

The mock model answers at once, so it hides the largest part of a real turn. We ran the same cluster again with a real model. It was qwen2.5, an open model, in two sizes. The model server was Ollama 0.40.16 in a container on the guest, with 12 CPUs, one request at a time, and no GPU.

ModelSizeTokens per second, 5 requestsCorrect tool call, 10 requests
qwen2.5:3b1.9 GBmedian 7.1, from 6.9 to 7.310 of 10
qwen2.5:1.5b986 MBmedian 13.9, from 13.1 to 14.39 of 10

This is a small model on slow hardware, and we chose it because both graphics cards of the server were in use by another team. A model on a GPU or a hosted model gives other model times. Read the model seconds below as the time of our machine. The Substrate seconds do not depend on the model.

Each turn asked the agent to run one command with its bash tool and to report one number. So a turn is two model requests: one that returns the tool call, and one that reads the result.

Fig. 7.3every kagent turn with a small real model on CPUchart
Every kagent turn with a small real model on CPU All 50 turns of the kagent run with a small open model on CPU, one bar each, in the order of the run. The plum part of a bar is the time of the model requests and the amber part is the rest of the turn, which holds the resume and the suspend. With the 3b model, a turn in one session took a median of 17.1 s and the model was 93 percent of it. With the 1.5b model the median was 11.0 s. 48 turns were clean. One turn with the 1.5b model did not end: the client gave up after 300 seconds, and the next turn waited 190 seconds behind it. 0 5 10 15 20 seconds for each turn at the client MODEL REQUESTS THE REST OF THE TURN NO ANSWER, OR WAITED 17.1 s 3b, one session 20 of 20 clean 14.4 s 3b, new 5 of 5 clean 11.0 s 1.5b, one session 20 of 20 clean 1.5b, new 3 of 5 clean 300 s with no answer, then 190 s one bar: one turn, in run order. black line: the median of a series with no failed turn. resume plus suspend: about 0.7 s of each turn. qwen2.5 on 12 CPUs, no GPU.
The model was about nine tenths of every clean turn, and one turn of 50 did not end. Each bar is one turn, in the order of the run. The plum part is the time of the model requests of the turn. The amber part is the rest, which holds the resume and the suspend. A rose bar is a turn that did not end, or a turn that waited behind it. Both are cut at the top of the chart.

We measured 50 turns on the same second x86 server guest (16 vCPU, 31 GB). Each model ran 20 turns in one Session and the first turn of 5 new Sessions. Raw runs

SeriesTurnsCompletedTurn at the clientModel requestsResumeSuspendModel share
3b, one Session2020median 17.1 s, from 11.8 to 19.215.9 s0.46 s0.24 s0.93
3b, first turn of a new Session55median 14.4 s, from 13.4 to 16.413.0 s0.45 s0.17 s0.92
1.5b, one Session2020median 11.0 s, from 10.0 to 19.79.8 s0.47 s0.23 s0.89
1.5b, first turn of a new Session5412.2, 13.2, and 13.4 s for the three clean turns7.2 to 12.3 s0.46 s0.16 ssee below

Every one of the 49 completed turns again had the shape resume, run, suspend. The resume and the suspend took about 0.7 s together, as with the mock model. The turn grew from 1.2 s at the client to 11 to 17 s, and the model was the difference. The resume and suspend columns are from the log of ate-api-server. For the last turn of each long series, the saved log held no resume record, so those two medians have 19 values.

Three turns of the 1.5b series sent three model requests in place of two. Two more turns of that series had about 5 s that neither the model nor Substrate accounts for. We did not find the cause.

One turn did not end

In the third new Session of the 1.5b series, the model did not stop. It had produced 2,800 tokens when we restarted the model server, 7 minutes and 40 s after the request began. The kagent CLI gave up after 300.13 s with context deadline exceeded and exit code 1.

Nothing ended the turn before the client did. The Session actor was ACTOR_STATE_RUNNING 193 s into the turn, and it held its worker CPU for all that time. The next turn waited behind the same model request, because the server took one request at a time. That turn completed after 189.8 s.

This is one event with one small model, and a larger model is less likely to run away. The cost of the event does not depend on the model. An actor that waits for a model is RUNNING, and kagent suspends it only when the turn completes. Nothing bounds a turn by default. Two settings of the ModelConfig do, and we tested both in a later run.

Two settings that bound a turn

The ModelConfig of kagent v1.0.0 has three fields for this under spec.openAI: maxTokens, maxCompletionTokens, and timeout. The timeout is in seconds, and it is the timeout of the HTTP client that calls the model. The two token fields exclude each other. The Agent, the AgentTemplate, and the Harness have no field for the seconds of a turn or for the number of model calls. The kagent CLI has --timeout, with a default of 5 minutes.

yamltwo ModelConfig objects, one with each boundFull log
apiVersion: api.kagent.dev/v1alpha3
kind: ModelConfig
metadata: {name: w-tok, namespace: kagent}
spec:
  provider: OpenAI
  model: qwen2.5:1.5b
  apiKeySecret: kagent-openai
  apiKeySecretKey: OPENAI_API_KEY
  stream: true
  openAI:
    baseUrl: http://ollama.kagent.svc.cluster.local:11434/v1
    maxTokens: 128
…
apiVersion: api.kagent.dev/v1alpha3
kind: ModelConfig
metadata: {name: w-time, namespace: kagent}
spec:
  provider: OpenAI
  model: qwen2.5:1.5b
  apiKeySecret: kagent-openai
  apiKeySecretKey: OPENAI_API_KEY
  stream: true
  openAI:
    baseUrl: http://ollama.kagent.svc.cluster.local:11434/v1
    timeout: 20
The AgentTemplate and Agent objects that use each ModelConfig are cut.

We first tried to get the runaway back, and it did not come. The first turns of 10 new Sessions on the 1.5b agent with no bound all completed, in 5.9 to 11.5 s. So we forced a long answer with a prompt that asks for the numbers 1 to 3000, one on each line. Sent to the model server directly, that prompt was still producing text when our request ended at 75 s. An earlier prompt asked for one word 5000 times, and the model stopped by itself in 7 of 7 turns.

We measured each row below on the same second x86 server guest (16 vCPU, 31 GB), after a restart of the guest. Every turn was the first turn of a new Session with the count prompt and the 1.5b model. Raw runs

SettingTurnsResultTurn at the clientThe actor and the model afterwards
maxTokens: 1283TASK_STATE_COMPLETED, with 128 output tokens and an answer that ends at 4615.6, 19.0, and 19.4 sACTOR_STATE_SUSPENDED at the end of each turn
timeout: 203TASK_STATE_FAILED, and the CLI ended with exit code 120.8, 20.9, and 21.2 sACTOR_STATE_SUSPENDED 5 s later, with the model server at 0% CPU
no bound, and --timeout 60s at the CLI1the CLI ended at 60.12 s with context deadline exceedednot endedACTOR_STATE_RUNNING in all 8 samples over 240 s, with the model server at about 1,200% CPU

With timeout: 20, the task failed with this text: llm error response (code STREAM_ERROR): "context deadline exceeded (Client.Timeout or context cancellation while reading body)". The model server logged each of those three requests at 19.9 s. The Session was still usable. The next turn in one of those Sessions completed in 17.1 s, in one try.

Neither bound broke a normal tool turn. With maxTokens: 128, 5 of 5 tool turns completed, in 5.5 to 8.0 s. With timeout: 20, 5 of 5 completed, in 5.8 to 12.1 s. On our slow model server, a tool turn of 12.1 s is close to a bound of 20 s. Choose the seconds from the speed of your model.

The row with no bound shows what a client timeout does not do. After the CLI gave up, the model request ran on for 10 minutes and 10 s, until we restarted the model server. A turn on a new Session waited behind it and reached the CLI default at 300.13 s. A second turn on the first Session returned resource conflict after 0.13 s. The timeout of the client stops the client. It does not stop the model call, and it does not suspend the actor.

We did not see a runaway happen by itself while a bound was set. That case and maxCompletionTokens are planned for the next edition.

Our position: set timeout on every ModelConfig, because it also ends a model call that stalls. Add maxTokens when a cut answer is better for your users than a failed turn. Do not count the client timeout as a bound. Then count how many Sessions can wait for the model at one time, because each one holds 1 CPU of a worker. The subsection on parallel turns below has that arithmetic.

What the snapshot keeps

kagent asks Substrate for the files-only fidelity, SNAPSHOT_FIDELITY_VOLUMES, and it suspends in the synchronous mode. The snapshot is in the object store before the turn ends. It holds the directory /data and nothing else.

In turn 1 our agent wrote one file under /data, wrote one file under /tmp, and started a background sleep. Turn 2 looked for all three.

outputwhat turn 2 found after one suspend and one resumeFull log
note=hello-from-turn1
scratch=cat: can't open '/tmp/scratch.txt': No such file or directory
bg=
uptime=0.74
PID   ELAPSED COMMAND
    1  0:01   /app
…
The rest of the output, the environment and the mounts, is cut.

The file under /data survived. The file under /tmp and the background process did not, and PID 1 was a new process. Write to /data everything that the next turn needs. Snapshot fidelity compares this mode with a memory snapshot.

The three egress rules that kagent writes

kagent writes one egress policy for each actor through the Substrate API. Our cluster had no Kubernetes EgressPolicy object at all.

yamlthe egress policy of one Session actorFull log
…
rules:
- http:
    effects:
      replaceHeaders:
      - actorJwt:
          audiences:
          - kagent-runtime
          expirationSeconds: "300"
        header: authorization
        prefix: 'Bearer '
    hostnames:
    - kagent-controller.kagent.svc.cluster.local
    ports:
      numbers:
      - 8083
- http:
    effects:
      replaceHeaders:
      - credentialUri: ate-secret://k8s.io/default/kagent/kagent-openai/OPENAI_API_KEY
        header: authorization
        prefix: 'Bearer '
    hostnames:
    - mock-llm.kagent.svc.cluster.local
    ports:
      numbers:
      - 8080
- https:
    hostnames:
    - api.openai.com
    ports:
      numbers:
      - 443
The CLI warning and the metadata above the rules are cut.
  1. The kagent controller on port 8083. The gateway replaces the authorization header with an Actor JWT for the audience kagent-runtime.
  2. The model host of the ModelConfig. The gateway replaces the authorization header with the value of a Kubernetes Secret. Credential injection explains the mechanism.
  3. api.openai.com on port 443. kagent adds this rule for every OpenAI ModelConfig, also when the model is at another host.

The actor never had the key. Its environment held OPENAI_API_KEY=kagent-credential-injected, and the mock model received the real value on all 10 calls.

A rotated key is slower here than on Envoy. The gateway, agentgateway v1.7.0, keeps each fetched credential for 300 seconds. So a new value of the Secret can take up to 5 minutes to reach the model calls. We measured that delay on the agentgateway dataplane of upstream Substrate, and not in the kagent cluster. Three changes of a Secret reached the upstream after 280.5, 200.9, and 100.6 s, and no request failed. Credential injection has that run.

Everything else was denied. A request to example.com failed in DNS. A request to the IP address 1.1.1.1 returned 403. The short name mock-llm.kagent also returned 403, and only the full name mock-llm.kagent.svc.cluster.local returned 200. Give kagent full DNS names for model and tool hosts.

Parallel turns on one worker

The worker pod of the install has no resource limits, so it reports the whole node: 16,000 millicores. Each actor reserves 1 CPU. The worker therefore ran at most 16 actors at one time.

We measured this on the same x86 server guest with one worker. Eight turns that started together all completed, in 2.47 to 3.57 s. Twenty-four turns that started together all completed, in 4.25 to 7.85 s. The worker showed 16 running actors, and the other 8 turns waited. Sizing a pool gives the arithmetic. Raw runs

Fig. 7.48 and 24 kagent turns at one time on one workerchart
Parallel kagent turns on one worker that holds 16 actors Seconds at the client for every turn of two tests on one worker pod, sorted from the fastest to the slowest. Eight turns that started together took 2.47 to 3.57 seconds. Of 24 turns that started together, the first 16 took 4.25 to 5.45 seconds and the last 8 took 6.45 to 7.85 seconds, because the worker ran at most 16 actors. One turn alone took 1.22 seconds. 0 2 4 6 8 1 8 8 turns at one time seconds at the client, for each turn 0 2 4 6 8 1 8 16 24 24 turns at one time, sorted by time 16 actors run: 16 of 16 CPU reserved 8 turns wait one turn alone: 1.22 s 3.57 s 5.45 s 7.85 s one dot: one turn, with a mock model. All 32 turns completed.
Sixteen of 24 parallel turns ended within 5.45 s, and the other eight waited for a free CPU and ended up to 2.4 s later. Each dot is one turn, sorted by its time at the client. The violet band holds the turns that waited while the worker ran 16 actors. The dashed line is one turn alone.

The CLI port-forward collision

The kagent CLI opens its own port-forward to port 8083 for each call. Five of eight CLI calls that we started together failed.

outputa kagent CLI call that lost the portFull log
Error: connect to kagent gateway: kubectl port-forward exited before the server became ready (kubectl: Unable to listen on port 8083: … bind: address already in use …
The second listener error and two more lines are cut.

Start one kubectl port-forward to the controller yourself, and point every CLI call at it. With a standing port-forward, a Session create took 0.17 s at the median in place of 0.39 s.

Agent Substrate 1.0.0 · b34fbee · 2026-11-05 · Edition 2026.11