Manuals / Agent Substrate 101 / Part 2 · Start

kubectl ate

kubectl ate talks to ate-api-server and reads Postgres, and without --token-file every caller acts as one shared identity.

You type kubectl get actors on a cluster that runs Agent Substrate, and Kubernetes answers that it has no such resource. The actor exists. You asked the wrong server. Actors, templates, and workers are rows in the Substrate database, and one plugin reads them.

When you finish this section, you can install kubectl ate and say how each command reaches the control plane. You can also explain the warning that it prints on every call.

Install the plugin

Plugin: kubectl-ate is one binary. When it is on your PATH, kubectl runs it for every command that starts with kubectl ate.

The release publishes the binary for Linux and macOS, on amd64 and arm64. There is no Windows build. Download the file and its checksum, check it, and install it:

bash

BASE=https://github.com/agent-substrate/substrate/releases/download/v1.0.0
curl -LO $BASE/kubectl-ate-linux-amd64
curl -LO $BASE/kubectl-ate-linux-amd64.sha256
shasum -a 256 -c kubectl-ate-linux-amd64.sha256
mkdir -p ~/.local/bin
install -m 0755 kubectl-ate-linux-amd64 ~/.local/bin/kubectl-ate

On a Mac with Apple silicon, the file is kubectl-ate-darwin-arm64. If you have the checkout and Go, you can build the plugin from the tag:

bash

go install ./cmd/kubectl-ate

kubectl finds the plugin only on your PATH. Check that ~/.local/bin is on it, or the directory $(go env GOPATH)/bin when you use go install.

We measured the build once on an x86 server guest with 16 vCPU and 23 GB. go install took 5 s after the install of the control plane, because the Go modules were already on disk. In our runs we always built the plugin from the tag. We checked that both release files exist, and we did not time the download. Raw runs

Check the version before you go on:

bash

kubectl ate --version

How a command reaches the control plane

kubectl ate does not ask the Kubernetes API for actors. It uses your kubeconfig to open a port-forward to the Service api in the namespace ate-system. Without --token-file, it also uses the kubeconfig to request a token for the shared ate-client ServiceAccount. Then it sends a gRPC request through that port-forward to ate-api-server, and ate-api-server answers from Postgres. Figure 2.2 shows that path beside the path of a plain kubectl command.

Fig. 2.2where kubectl ate and kubectl read fromflow

kubectl ate reads atespaces, templates, actors, and workers from Postgres through ate-api-server, and kubectl reads only three custom resources from etcd. The top row is every kubectl ate command except proxy. The bottom row is plain kubectl. The dashed box is the identity that the top row uses by default.

Two facts follow from the figure.

First, each command opens its own port-forward. So kubectl ate needs a working Kubernetes API server, and your account needs the right to port-forward into ate-system. With kube-apiserver down, a command fails like this:

outputa kubectl ate command while kube-apiserver is downFull log

Error: failed to connect to ate-api-server: getting api service: Get "https://127.0.0.1:44015/api/v1/namespaces/ate-system/services/api": EOF

Actors kept serving requests during that outage. Only the CLI and the proxy failed. Section 2.3 has the numbers.

Second, the flag --endpoint skips the port-forward. A program inside the cluster passes the address of ate-api-server and dials it directly. Its pod must be in a namespace with the label ate.dev/api-access=true. We did not set that label ourselves, and we did see the network rule behind it in the kagent run. A pod in default could not reach the API, and a pod in kagent could. Section 5.1 has that probe.

The shared identity warning

Every command in our recordings starts with the same line:

outputthe first actor, created without a token fileFull log

Warning: no --token-file given; acting as the shared identity system:serviceaccount:ate-system:ate-client rather than as yourself. Pass --token-file to act as yourself.
ATESPACE           NAME           TEMPLATE                   STATE                   WORKER POD   WORKER IPS   VERSION   AGE
ate-demo-counter   my-counter-1   ate-demo-counter/counter   ACTOR_STATE_SUSPENDED   <none>                    1         0s

Shared identity: the ServiceAccount ate-client in ate-system. When you pass no token, the plugin asks Kubernetes for a token of that ServiceAccount and sends it to ate-api-server.

The warning is correct, and you must not ignore it on a shared cluster. Every person and every script that can port-forward into ate-system is the same caller to Substrate. An access policy cannot tell them apart, because it sees one name.

On a kind cluster on your own machine, the shared identity is fine. On a cluster with more than one team, give each caller a token and pass it with --token-file. Section 5.1 shows which token to use, and section 5.2 limits what a caller can do in an atespace.

Our scripts removed the line from their output with grep -v '^Warning: no --token-file'. Do that in a script that parses a table. Do not do it for people.

The commands of the first day

An actor is addressed by its atespace and its name, so most commands need -a with an atespace. No default atespace exists.

Command	What it does
`kubectl ate get actors -a ate-demo-counter`	lists the actors of one atespace
`kubectl ate get actors -A`	lists the actors of all atespaces
`kubectl ate get actor my-counter-1 -a ate-demo-counter -o yaml`	prints one actor with its snapshots
`kubectl ate create actor my-counter-1 -a ate-demo-counter --template counter`	creates an actor from a template
`kubectl ate suspend actor my-counter-1 -a ate-demo-counter`	writes a snapshot and frees the worker
`kubectl ate resume actor my-counter-1 -a ate-demo-counter`	restores the actor onto a worker
`kubectl ate revert actor my-counter-1 -a ate-demo-counter`	returns a crashed actor to its last durable snapshot
`kubectl ate logs actor my-counter-1 -a ate-demo-counter`	prints the logs of the actor on its worker
`kubectl ate get workers`	lists the workers with their used CPU and memory
`kubectl ate proxy actor my-counter-1 -a ate-demo-counter`	forwards a local port to the actor

-o json and -o yaml print the full object. Use them when you debug, because the table hides the snapshots. This is the new actor as JSON:

jsonthe status of a new actorFull log

  "status": {
    "lastAssignedGeneration": 1,
    "snapshots": [
      {
…
        "durableSnapshot": {
…
          "state": "SNAPSHOT_STORAGE_STATE_COMPLETED"
        },
        "fidelity": "SNAPSHOT_FIDELITY_MEMORY",
        "generation": 1,
        "owner": "SNAPSHOT_OWNER_TAG",
…
    "state": "ACTOR_STATE_SUSPENDED"

Cut from the actor object. The metadata and the sandbox version attributes are cut.

The actor has never run, and it already has one snapshot. The owner is a tag, and the tag belongs to the template. Section 3.1 explains that snapshot. The command reference lists every command and flag.

Three mistakes we made

-A on workers. A worker belongs to a Kubernetes namespace and not to an atespace. The flag that works for actors fails for workers:

output-A is not a flag of get workerFull log

Error: unknown flag "-A"
…
  -a, --atespace string        filter worker pods hosting actors in a specific atespace
…
  -n, --namespace string       scope output to a specific Kubernetes namespace

The rest of the usage text is cut.

Run kubectl ate get workers with no flag to see all workers.

kubectl get for an actor. kubectl get actor and kubectl get actortemplate return nothing, because those objects are not custom resources. kubectl get workerpool does work, because a WorkerPool is one.

A flag from an old guide. Guides for v0.x create an actor with --template-ref. At v1.0.0 the flag is --template. Section 8.4 lists the other changes.

Agent Substrate 1.0.0 · b34fbee · 2026-11-05 · Edition 2026.11