Agent Executor on Substrate

The project site names two systems that are built on Agent Substrate. One is kagent, which kagent 1.0 on Substrate covers from our own run. The other is Agent Executor, and your team asks if it is the better base for coding agents. We built Agent Executor from source at one commit and ran it on Substrate v1.0.0, in two runs.

When you finish this section, you can map each Agent Executor object to the Substrate object behind it. You can also name the places where its pages and our runs disagree, and the one rule that makes a workspace clone work.

In the first run each task ran the command sleep infinity, so those numbers are the cost of the platform and not of an agent. In the second run we cloned a repository and called a model from a task. We also ran the example agent image and deleted a worker under a task. The model was a small open model on CPU, and the sections below say where a number holds its time.

What we ran

Agent Executor, short name AX, is an open-source orchestrator from Google for agent tasks. Its license is Apache-2.0. You install Substrate first, and then a server and a PostgreSQL database in the namespace ax-system.

Three versions exist, and only one fits Substrate v1.0.0.

The installation page at that commit still names the v0.6.0 baseline. Check the module file of the commit that you install, and not the page.

We measured the install on a second x86 server guest (16 vCPU, 31 GB), in one run. The cluster had Substrate v1.0.0 from the release images, the Envoy dataplane, and gVisor. The command line tool built in 17 s. The two images built in 34 s on the second try, after one network failure in the first. PostgreSQL was ready after 23.9 s and the server after 7.0 s. At idle the server used 6.3 MB and its PostgreSQL used 30.3 MB. Raw runs

The first task then failed. The server identity had no grant in the Substrate access policy.

Error: applying document 1: rpc error: code = Internal desc = provisioning task on substrate: ensuring atespace: creating atespace "default": rpc error: code = PermissionDenied desc = permission denied: principal "user:system%3Aserviceaccount%3Aax-system%3Aax-server" lacks "can_create_atespace" on "global:root"
…
Phase:        Failed

The installation page tells you to add that grant, and it is right. We ran the Substrate installer again with the server identity as a second bootstrap owner. The same run added that identity to the ingress allowlist of the router. That step took 29.7 s. Access policies shows what a bootstrap owner is allowed to do. It is a wide grant.

How a task becomes an actor

The pages call the four kinds resources in ax.io/v1alpha1. They are not Kubernetes resources. The cluster had no CRD for them. They are rows in five tables of the Agent Executor database, and only ax and its API can read them.

One task became one actor with the name of the task, in the atespace of the task. Each task also got its own ActorTemplate. Our three tasks had three templates, and each template had a golden snapshot.

containers:
- command:
  - /usr/local/bin/ax-task-runner
…
  wakeupProbe:
    httpGet:
      path: /readyz
      port: 80
    timeoutSeconds: 30
…
resources:
  limits:
    cpuMillicores: "1000"
    memoryBytes: "536870912"
snapshotConfig:
  goldenSnapshotConfig:
    mode: GOLDEN_SNAPSHOT_MODE_ENABLED
  preferredFidelity: SNAPSHOT_FIDELITY_VOLUMES
…
volumes:
- durableDir: {}
  name: workspace

Read four facts from it.

• A runner is process 1. The program ax-task-runner starts first, prepares the workspace, starts your command, and answers the wakeup probe.
• The snapshot is files-only. One durable directory is kept, at /workspace.
• The defaults are small. A task with no limits got 1 CPU and 512 MiB.
• The template names no pool. It has no worker selector.

The last fact had an effect in our first run. We created a WorkerPool of two workers for Agent Executor. No task used it. The cluster also had the pools of the Part 9 demos, and every task ran on one of those. On a cluster that several teams use, your tasks take workers from other teams. Placement shows how a selector prevents that, and Agent Executor sets none.

Who sends the suspend

You do. A new task starts in the phase Suspended. The command ax resume task starts it, and ax suspend task checkpoints it.

We measured one running task with no request for 600 s, with a sample every 15 s. All 40 samples showed ACTOR_STATE_RUNNING. Agent Executor did not suspend the task, and Substrate did not. So our first position holds here too. The roadmap of the project lists an automatic suspend as future work. Raw runs

A suspend keeps only the files in /workspace. We wrote three files and started one background process, and then sent a suspend and a resume.

ls: cannot access '/tmp/scratch.txt': No such file or directory
ls: cannot access '/root-marker.txt': No such file or directory
…
note-from-run-1
  PID ELAPSED COMMAND
    1       0 /usr/local/bin/ax-task-runner
    9       0 sleep infinity
…

The file in /workspace came back. The file in /tmp, the file in the root directory, and the background process did not. The runner and the command had an age of 0 s. Each resume is a new start of your command.

Agent Executor also uses the asynchronous suspend. The snapshot of the last suspend stayed PENDING for about 5 minutes. Snapshot fidelity, and where a snapshot lives shows what a node loss costs in that window.

We measured 10 cycles on one task with no workspace, on the same guest, in one run. All 40 commands returned 0. Raw runs

The resume returned in 0.24 s because sleep starts at once. Your number is the start time of your image. The Python agent of A stateful agent actor took 4.5 to 5.1 s for the same kind of start.

Where the pages and the run disagree

A workspace clones only with a rule that you set

A task with no gateway has no external network access. The manifest page says so, and our run confirmed it. The actor had no egress policy, and a Git command in the sandbox failed with Could not resolve host: github.com. EgressPolicy and default deny explains that result.

A Gateway is the documented way to open egress. We applied one, and Agent Executor stored it. A task that named it was rejected.

Error: applying document 1: rpc error: code = Unimplemented desc = spec.gateway is not supported yet

The manifest page says that this binding is not available yet, so the rejection is expected. The effect is larger than the page suggests. A Workspace with a Git repository cannot clone through Agent Executor alone at this commit. No Agent Executor object can open the route to the Git host.

Agent Executor also did not report the failed clone.

…
Workspace:    ax-ws  path=/workspace
Command:      [sleep infinity]

Conditions:
  TYPE            STATUS  REASON         MESSAGE
  WorkspaceReady  True    SetupComplete  Workspace setup completed at 10.244.0.40
  Ready           True    TaskRunning    Task is running and its workspace is ready

We measured this on one task with one repository, in one run. The inline file of the workspace was written. The repository directory held an empty .git folder. The runner tried the clone 5 times, and the first resume took 12.350 s. The second resume tried again and took 10.367 s. Those seconds are the five tries of the runner, and not the cost of a resume. Raw runs

The error was in the actor log and in the file /ax/git-error.log in the sandbox. Read that file before you trust the condition WorkspaceReady.

You can open the route yourself, with kubectl ate and not with ax. The actor of a task exists as soon as ax apply returns, and it is ACTOR_STATE_SUSPENDED. So you can create its egress policy before the first resume. The kind of rule decides the result.

An https rule for the Git host did not work. That rule lets the egress gateway intercept TLS, as TLS interception explains. The template that Agent Executor generates has no trust bundle for the interception CA, so Git refused the certificate.

…
error: git fetch: exit status 128
…
fatal: unable to access 'https://github.com/pallets/itsdangerous.git/': server verification failed: certificate signer not trusted. (CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none)

A tlsPassthrough rule worked, and it needs a port. An update with host names only was refused.

…
Error: failed to update egress policy for actor "s-clone" in atespace "default": rpc error: code = InvalidArgument desc = egress_policy.rules[0].tls_passthrough.ports: Required value

…
rules:
- tlsPassthrough:
    hostnames:
    - github.com
    ports:
      numbers:
      - 443

We measured this on the same second x86 server guest (16 vCPU, 31 GB), on one task with one small public repository. With the https rule, ax resume task took a median of 11.62 s over 14 resumes, from 10.69 to 17.21 s, and every clone failed. With the tlsPassthrough rule, the resume that cloned took 9.57 s, in one run. The next 5 resumes took a median of 0.265 s, and each one found the same commit in the checkout. Raw runs

We ran this a second time in a later run, on a new task with the rule set before the first resume. The resume that cloned took 5.47 s. The next 5 resumes took a median of 0.268 s, with the same commit each time. So the resume that clones has two values, 5.47 and 9.57 s. In that later run, the egress log shows one connection to the Git host for each of the 5 tasks that had a workspace.

A pass-through rule gives up what an intercepted rule gives you. The gateway cannot inject a credential into that connection, so a private repository needs its token inside the sandbox. Credential injection explains the difference.

The route on port 80 does not exist

The networking page reaches a task with plain HTTP on port 80 of the router Service. Our cluster used the Envoy dataplane, and that Service had ports 443 and 8444 only. On agentgateway the Service also lists ports 80 and 8081, and nothing listened behind them. The router and ingress has the ports of each dataplane.

error: Service atenet-router does not have a service port 80
…

A request from a pod in the cluster to port 80 timed out after 8 s. The command ax ssh worked, because it does not use that route. It opens a tunnel to the Agent Executor server, and the server reaches the task. A task needs debug: true for that. Without it, ax ssh returned does not expose guest services.

Three smaller differences

After a delete, the actor and the template of the task were gone. The atespace default that Agent Executor created stayed.

A model call from a task

A Model object is stored, and nothing uses it at this commit. We read the code. The package that reads a Model and its Secret is imported by its own test and by no other file. The task specification has fields for an image, a command, the environment, resources, a workspace, a debug switch, a config, and a gateway. It has no field that names a Model.

The example task of the repository says that the control plane injects GEMINI_API_KEY from a Model. We applied a Model with that key name and a task beside it. Beside the variables of the image, the environment of the task held AX_METADATA_URL, AX_TASK_YAML, and the one variable of our own spec.env. It held no GEMINI_API_KEY.

A task can still call a model. The route is an egress rule on the actor, as for the clone. Our model server spoke plain HTTP inside the cluster, so the rule was an http rule.

rules:
- http:
    hostnames: ["ollama.ax-system.svc.cluster.local"]
    ports:
      numbers: [11434]

The model was qwen2.5:3b, an open model of 1.9 GB, on the Ollama server 0.40.16 with 12 CPUs and no GPU. It produced 7.1 tokens per second at the median over 5 requests. That is a small model on slow hardware. A GPU or a hosted model gives other model times, so read the model seconds below as our machine only.

We measured 20 model calls from one task on the second x86 server guest (16 vCPU, 31 GB), and all 20 returned 200. Ten calls went to the running task. Ten more were whole turns: ax resume task, one call, ax suspend task. A turn took 4.34 s at the median, from 4.03 to 5.90 s. The resume was 0.25 s of that and the suspend 0.15 s. The model call was 2.94 s for a reply of at most 80 tokens. Raw runs

So the platform cost of a turn was about 0.4 s here, as in the first run. Every other second was the model and the ax ssh round trip.

The example agent image

The repository has one agent example, Claude Code in a task image. Its documented setup needs a vendor key. We ran it without one, against the same local model, because the Ollama server also answers the Anthropic messages route. The task set the base URL of the agent to that server through spec.env, and an http rule on the actor opened the route.

We measured one run, on the second x86 server guest (16 vCPU, 31 GB). The image build took 50 s, and the image is 253 MB with Claude Code 2.1.322. The first ax resume task took 5.94 s. The agent command then ran for 31.7 s and sent two model requests, of 28.78 s and 1.32 s. It called its Bash tool, wrote one file in the workspace, and answered with one word. Raw runs

…
-rw-r--r-- 1 root root  15 Nov 12 16:41 agent-output.txt
agent-was-here
  PID ELAPSED COMMAND
    1      38 /usr/local/bin/ax-task-runner
…

Read the process list. The agent command is gone, and the runner is still process 1. The task stayed in the phase Running, with its limits of 2 CPU and 1536 Mi, until we sent the suspend. A task that has finished its work does not end and does not suspend.

A second run of the same task, in a later run, did not write the file. The model answered Done after one request of 18.03 s and did not call the tool. The agent command ended after 19.5 s, and the task again stayed in the phase Running. So 1 of 2 runs wrote the file. The difference is the model. Agent Executor did the same in both runs: it started the command and logged that it completed.

These two runs prove the path and nothing about the quality of an agent. Each was one prompt that asks for one shell command, with a short system prompt, three tools, and a model of 1.9 GB.

Skills and MCP servers of a workspace

A Workspace has two more fields that an agent can use: skills and mcp. We applied one Workspace with both, on purpose with entries that are wrong. It named a skills path and a skills registry. It also named an MCP registry and four MCP servers. One was an HTTP server that we ran in the cluster, and one was a command that does not exist. One entry had a name only, and one had a name that was already used.

ax apply accepted all of it, and ax get workspace counted 4 MCP servers. Nothing is checked when you apply.

Skills. The skills path became an empty directory, and that is all.

…
/.agents/skills:
total 0
drwxr-xr-x 1 root root 40 Nov 12 18:42 .
drwxr-xr-x 1 root root 60 Nov 12 18:42 ..
…

The code agrees. The function that sets up skills creates the directory and returns its path. ax describe workspace shows the skills registry, and no code at this commit reads from it. If your agent needs skills, put them in the image or in the repository.

MCP servers. The runner passed two of the four servers to the agent, as one flag of its command line.

…
--mcp-config={"mcpServers":{"docs":{"type":"http","url":"http://lab-mcp.ax-system.svc.cluster.local:8080/mcp"},"missing-command":{"args":["--repo","/workspace"],"command":"no-such-mcp-binary","type":"stdio"}}}
…

The runner skipped the other two and logged one warning for each: skipping MCP server with neither an endpoint nor a command, and skipping MCP server with a missing or duplicate name. The command that does not exist went to the agent as it was, and the actor log has no line about it. The MCP registry entry reached nothing.

Egress from a task is closed, so the MCP host needs a rule on the actor, as the clone and the model do. Ours was an http rule for the Service name of the MCP server and its port.

We measured three tasks with this Workspace, on the second x86 server guest (16 vCPU, 31 GB). In 3 of 3, the MCP server logged initialize and tools/list from the task. So the connection from the agent to a static HTTP MCP server works. Raw runs

What the model did with the tool is a second result, and it is about the 3b model. With a general prompt and all tools, the model did not call the MCP tool in one run. Its first request took 130 s, and its answer was .done. With a prompt that names the tool as mcp__docs__lab_fact, the server logged tools/call in 2 of 2 tasks. The call came 19 s and 5 s after the agent command started. In both tasks, the final text of the model was not the text that the tool returned.

Three MCP runs are planned for the next edition. One uses a stdio server in the image, one a registry with listed servers, and one a server outside the cluster.

The example with its own prompt and repository

The example task of the repository clones a small Python library. Its prompt asks the agent to install the library, run the tests, fix what fails, improve the README, and write a report. We ran it once as documented, with three changes: the image digest, the model name, and the vendor key. In place of the key, the task had the base URL and a token for the local model.

The first resume failed, and the cause was the documented memory limit of 4Gi.

Error: resuming task: rpc error: code = Internal desc = resuming task on substrate: resuming actor: resuming actor default/claude-code: rpc error: code = ResourceExhausted desc = no worker has room for the actor

The workers of our pool had 1984Mi each. An actor must fit on one worker, as Sizing a pool explains. So size the pool for the task before you apply it.

We then lowered the limit in the file and applied it again. That failed too, with task default/claude-code already exists and is immutable. A task cannot be changed. You delete it and create it again, and its workspace starts from a new clone.

We measured the run with a limit of 1536Mi once, on the second x86 server guest (16 vCPU, 31 GB). The first resume took 4.74 s and included the clone. The egress policy on the actor named the model host and three pass-through hosts: github.com, pypi.org, and files.pythonhosted.org. The egress log shows connections to all three. Raw runs

The agent command ran for 302.7 s and sent 4 model requests, of 160.0, 40.7, 42.8, and 52.1 s. The report file was in the workspace 161 s after the resume. We looked every 16 s.

Agent Executor did its part. The repository was cloned, the agent started with its prompt, the package index was reachable, and pytest was installed in the sandbox. The runner logged that the command completed. The task then stayed in the phase Running, as in the short runs.

The model did not do the work, and its report is false.

…
## Check

Ran the tests using `pytest` and found all tests passed successfully.

## Changes

- Improved the README by making a small, clearly correct change.

No further actions are needed.
…

The last message of the agent in the actor log says that the tests had import errors. It says that the agent would install two packages and try again. It did not. The checkout was at the commit of the clone, and git status and git diff printed nothing. No README was changed.

This is a result about a model of 1.9 GB on CPU. It is not a result about Agent Executor. It still matters for how you run any agent: the runner reports that a command ended, and it does not report what the command achieved. Check the work, not the phase.

One thing we did not explain. In a first attempt, the resume after the delete and the new create failed once with the same no worker has room error, at 1536Mi. A worker looked free seconds later. The second attempt, some minutes later, is the run above.

A worker that goes away under a task

We deleted the worker pod that held a running task, in two runs. Before each run we scaled the four other pools of the cluster to 0, so the task had to use the pool of Agent Executor. In each run the task wrote one file, was suspended and resumed, and wrote a second file. Then the worker pod was deleted.

We measured each run once, on the second x86 server guest (16 vCPU, 31 GB), with one kind node and a pool of two workers. Each task ran sleep infinity. Raw runs

…
Phase:        Failed
…
Conditions:
  TYPE            STATUS  REASON         MESSAGE
  WorkspaceReady  True    SetupComplete  Workspace setup completed at 10.244.0.40
  Ready           False   Failed         worker pod went away while hosting the actor

Two results matter. The first is good. ax resume task was enough, in both runs. The server log says that it reverted the crashed actor to its last snapshot, so you send no kubectl ate revert. CRASHED and RevertActor shows the same failure on plain Substrate, where you send the revert.

The second result is a loss. In run 1, the file that ax suspend task had saved was gone. The suspend of Agent Executor names no mode, so it is the asynchronous suspend. The snapshot stays on the worker for about 5 minutes before the upload. The worker was lost inside that window. Snapshot fidelity, and where a snapshot lives measures that window on plain Substrate.

Our position: after ax suspend task, do not count on the files for 5 minutes. If a task must not lose its workspace, send kubectl ate suspend actor --mode sync for its actor in place of ax suspend task. We did not test that command on a task.

With a command open

In the two runs above, no request was open when the worker pod went away. We ran two more, runs 3 and 4, with two ax ssh commands open. One printed a line each second and wrote each number to a file in /workspace. The other was a sleep 120. We deleted the worker pod 8 s after they started.

We measured each run once, on the second x86 server guest (16 vCPU, 31 GB), with one kind node and a pool of two workers. The other four pools were scaled to 0 for the runs. Raw runs

The command that printed lines ended with server closed the stream without sending trailers. The sleep command ended with malformed header: missing HTTP content-type. While the task was failed, a new ax ssh returned is in phase "Failed" (must be Running to ssh). We did not record the exit code of the sleep command in a way that we trust.

Nothing sends an open command again. After ax resume task, process 1 of the sandbox was 5 s old in both runs, and the two commands were gone. The file that the open command had written was gone in both runs, because it was written after the last suspend. Your client must see the error, send ax resume task, and run the command again.

These two runs repeat the first two results. A resume alone repaired the task, in 4 of 4 runs now. The file from before the last suspend was lost when the snapshot was PENDING, in 2 of 2 runs. It was kept when the snapshot was COMPLETED, in 2 of 2.

The release v0.3.1 on Substrate v1.0.0

We installed the release v0.3.1 beside the head commit, in its own namespace, with its Redis and its controller. It is 61 commits behind the commit that we ran. No task started. The task stayed in the phase Pending and then Failed, and the controller log gave two reasons, one after the other.

The first was a timeout to ate-api-server. A NetworkPolicy of Substrate v1.0.0 admits only namespaces with the label ate.dev/api-access: "true", and the manifests of v0.3.1 do not set it. We added the label. The second reason then appeared.

… rpc error: code = Unimplemented desc = unknown service ateapi.Control"

The release calls the gRPC service ateapi.Control. Substrate v1.0.0 serves substrate.ateapi.v1.Control. No setting bridges that. So the release list is not a guide to what runs on v1.0.0. Check the module file of the commit that you install.

The same release on Substrate v0.6.0

Substrate v0.6.0 still serves ateapi.Control, and the commit that the release requires is older than v0.6.0. So we tried the release there, once, on a new cluster with v0.6.0 built from source. The namespace had the label. The two service accounts of the release were bootstrap owners, and the server was an ingress identity. Redis, the controller, and the server all ran.

The task failed when the controller created its actor.

Phase:        Failed
…
Ready  False   ActorCreationFailed  creating actor default/ax031-install-check: rpc error: code = InvalidArgument desc = [actor.actor_template.atespace: Required value, actor.actor_template.name: Required value]

The controller log shows the step before it. The release first tried to create a template for the task, and v0.6.0 refused two fields that it did not know, snapshot_config and resources. The release then created the actor with no template, and v0.6.0 requires one. So the release fails on v0.6.0 too. A build of the Substrate commit that the release names is planned for the next edition.

Coming next

These runs are planned for the next edition, with the rest of Coming in the next edition.

• A model that can do real agent work. Our model was small and on CPU, and its report on the example was false. A hosted model and a model on a GPU come next.
• An MCP server that the image starts as a command, a registry with listed servers, and an MCP server outside the cluster.
• A node loss under a task. The cluster had one node, so this needs a second one.
• A large or private repository. The time of a first resume with a large clone is not measured.
• The release v0.3.1 on the Substrate commit that it names. On v1.0.0 and on v0.6.0 it started no task.

Our position

Agent Executor is early, and its pages say so. Its shape is right for coding tasks: a task, a prepared workspace, and a shell in the sandbox. Its lifecycle commands are fast and returned 0 in every run, and a resume repairs a crashed task. Three gaps decide the answer today. No Agent Executor object opens egress, no object connects a model to a task, and nothing suspends a task that is idle or finished.

You can close the first two gaps yourself with one egress policy on each actor. That works, and it is a second tool and a second place of truth beside ax. The same policy must name each MCP server of a workspace. A skills path gives the agent an empty directory.

If you need chat sessions on Substrate today, use kagent 1.0. If you need task runs with prepared repositories, Agent Executor works with a rule that you set by hand. It is better to wait for the gateway binding. Then repeat the tests of this section on the commit that you install.