Manuals / Agent Substrate 101 / Part 6 · Network
6.1

The router and ingress

Every request to an actor passes through atenet-router, which demands a client certificate with an allowed SPIFFE ID and calls ResumeActor before it forwards one byte.

Your actor answers through kubectl ate proxy. Then a service in another namespace calls the router directly, and the call fails in the TLS handshake with no HTTP status to read. Nothing is wrong with the actor. The router refused the caller before it looked at the request.

When you finish this section, you can address an actor through the router and name the identity that the router accepts. You can also read the three ways that it refuses a caller on each dataplane.

One Service for every actor

An actor has no Service and no hostname. Every actor in the cluster is reached through one Service, atenet-router.ate-system.svc, and the caller names the target in a header.

Routing header: the HTTP header ate-target-actor, with the value <atespace>/<actor>, such as ate-demo-counter/my-counter-1. The Host header stays free for your application.

The Service has two ports that matter. Port 443 carries plain HTTPS requests to port 80 of the actor. Port 8444 accepts an HTTP CONNECT over TLS, for an actor that listens on another port. The routing header goes on the outer CONNECT, and the port goes in its authority. Our probe sent CONNECT actor:80 and got the same answer as on port 443.

Envoy dataplaneagentgateway dataplane
Containers in the router podatenet-router and envoyone agentgateway
Listeners for callers8443 and 84448443 and 8444
Service ports443 and 8444443, 8444, 80, 8081, and 15020
Plain HTTP on 80 or 8081no such Service portconnection refused

The agentgateway Service lists ports 80 and 8081, and the NetworkPolicy admits them. No process listens on them. A plain HTTP request to either port failed in 3 ms with a refused connection. Do not read the Service as proof of an open port.

outputthe router Service on the agentgateway dataplaneFull log
atenet-router             ClusterIP   10.96.5.247     <none>        443/TCP,8444/TCP,15020/TCP,80/TCP,8081/TCP …
One line. The columns for the external IP, the age, and the selector are cut.

A later probe tried every port from two pods. One was in another namespace and had no certificate. The other was in ate-system and held the allowed identity.

PortEnvoy dataplaneagentgateway dataplane
Service 443, container 8443TLS. The connection is closed without a client certificate. With the allowed identity and the routing header: 200. Without the header: 404, invalid actor referenceTLS. Without a client certificate: the alert certificate required. With the allowed identity and the routing header: 200. Without the header: 404, ingress failed
Service 8444TLS. The connection is closed without a client certificate. A POST with the identity gets 404TLS, with the same alert. A POST with the identity gets 405, method not allowed, because the listener accepts CONNECT only
Service 80, container 8080not in the Service, timeoutin the Service, connection refused, no listener
Service 8081not in the Service, timeoutin the Service, connection refused, no listener
Service 15020, statsnot in the Service, timeoutopen with no TLS and no credentials. /metrics returns 200 to a pod in another namespace
Pod port 15021timeout404 on plain HTTP, the readiness listener
Pod port 15000timeouttimeout

We measured each port once on each dataplane, on a second x86 server guest (16 vCPU, 23 GB), with a client timeout of 4 s. Ports 443 and 8444 need a client certificate on both dataplanes. With agentgateway the Service has five ports and the container declares six, and the rendered configuration binds 8443 and 8444 for callers. Ports 80 and 8081 are dead entries in the Service. With Envoy the Service has two ports. Raw runs

outputthree more probes on the agentgateway dataplaneFull log
stats 15020 /metrics from another namespace: 200|13394|58|# HELP agentgateway_substrate_request_parking_active Number of requests waiting for a Substrate actor to become routable.|
…
service 8444 with allowed client cert, POST with routing header: 405 exit=0 err=|method not allowed
service 443 with allowed client cert, no routing header: 404|ingress failed
Three lines of the file. The first is from a pod in another namespace, and the other two used the allowed client certificate. The fields after the status are the size in bytes, the number of metric families, and the first line of the body.

Security note: the stats port of agentgateway needs no credentials. Any pod that can reach the Service can read /metrics on port 15020, in plain HTTP. Our pod in another namespace got 13,394 bytes with 58 metric families, and the names include request parking for actors. We found no actor name and no header value in that output, and we did not read the content of every label. Until you have read your own output, treat the port as readable by every tenant of the cluster. A NetworkPolicy for port 15020 of the router pods closes it.

Use the short name of the Service. The router certificate is valid for atenet-router.ate-system.svc. The longer name atenet-router.ate-system.svc.cluster.local failed the certificate name check on both dataplanes.

One request, step by step

The router keeps no table of running actors. For each request it asks ate-api-server where the actor is, with the same ResumeActor call that wakes a suspended actor. The answer carries the state, the worker IP, and the worker identity. Then the router opens a mutual TLS tunnel to that worker and forwards the request.

Fig. 6.1one request through the routersequence
One request through atenet-router to a running actor Five lifelines: a caller pod with the identity atenet-client, atenet-router, ate-api-server, Postgres, and the worker pod that runs my-counter-1. Seven numbered exchanges: the caller sends POST over mutual TLS with the ate-target-actor header, the router calls ResumeActor, ate-api-server reads the actor row, and it returns ACTOR_STATE_RUNNING with the worker IP and SPIFFE ID. That call took 9.2 milliseconds. The router opens an mTLS CONNECT to the worker, the actor answers 200, and the caller has the 200 after 37 milliseconds. caller pod atenet-client atenet-router Service port 443 ate-api-server control plane Postgres actor rows worker pod my-counter-1 9.2 ms of the 37 ms 1 POST / over mTLS ate-target-actor header 2 ResumeActor for every request 3 read actor row 4 ACTOR_STATE_RUNNING worker IP and SPIFFE ID 5 mTLS CONNECT to the worker then actor port 80 6 200 7 200 37 ms at the caller
The router calls ResumeActor for every request, also when the actor already runs, and only then opens a tunnel to the worker. Read from top to bottom. Step 2 happens for each request, which puts ate-api-server and Postgres on the path of each one. The bracket is the time from step 2 to step 4 for this request, from the router log.

The router log shows the call for an actor that was already ACTOR_STATE_RUNNING. Our prober sent one request per second, and each one produced this pair.

outputthe router asks for a running actorFull log
{"time":"2026-11-12T11:04:04.204261373Z","level":"INFO","msg":"ResumeActor","actor":{"type":"*ateapi.Actor","atespace":"lab-e","name":"e-run"},…
{"time":"2026-11-12T11:04:04.208776711Z","level":"INFO","msg":"ResumeActor result","actor":{"type":"*ateapi.Actor","atespace":"lab-e","name":"e-run"},"state":"ACTOR_STATE_RUNNING","workerIP":"10.244.0.36","workerSPIFFEID":"spiffe://cluster.local/ns/lab-e/sa/ateworker",…
Two lines for one request. The trace fields at the end of each line are cut.

This design has a cost on the warm path. A running actor is only as available as ate-api-server and Postgres. We scaled ate-api-server to zero for 60 seconds. The running actor returned 503 with the body actor lab-e/e-run unavailable for 60 requests in a row. Section 8.2 has the full timeline.

We measured the allowed path on a second x86 server guest (16 vCPU, 23 GB, Envoy dataplane), one request each. A request on port 443 to a running actor took 0.037 s. The router log gives 9.2 ms of that to the ResumeActor call. The same request through CONNECT on port 8444 took 0.032 s. On the agentgateway dataplane the two requests took 0.035 s and 0.029 s. Raw runs

When no worker has room, the router holds the request and tries again. Section 4.4 covers that wait.

Who the router lets in

The router serves TLS only. It requires a client certificate from the pod identity signer of the cluster. Then it compares the SPIFFE ID in that certificate with a fixed list.

Ingress allowlist: the list of SPIFFE IDs that the router accepts, set at install with --ingress-allowed-spiffe-ids. The flag has no default. An identity on the list can reach every actor in every atespace.

That last sentence is the one to remember. The router authenticates the caller. It does not check that this caller is allowed to reach this actor. Put a service of your own in front of it when callers must see only their own actors. Section 5.1 covers the client side.

We sent the same request from three pods. The first had no client certificate. The second had a valid certificate for spiffe://cluster.local/ns/probe/sa/default, which is not on the list. The third had the allowed identity.

CallerEnvoy dataplaneagentgateway dataplane
No client certificateconnection cut, Broken pipe in curlTLS alert certificate required
Valid certificate, identity not on the listTLS alert certificate unknownconnection reset
Allowed identity200 on 443 and through 8444200 on 443 and through 8444
Allowed identity, actor does not exist404, actor ate-demo-counter/does-not-exist not found404, ingress failed
Allowed identity, no routing header404, invalid actor reference404, ingress failed
Fig. 6.2what a caller of the router sees on each dataplanedecision
What a caller of atenet-router sees on each dataplane Five rows in two groups, with the outcome on the Envoy dataplane and on the agentgateway dataplane. In the first group the router refuses the caller in the TLS layer and the caller gets no HTTP status: with no client certificate, Envoy cuts the connection and agentgateway sends the alert certificate required. With an identity that is not on the list, Envoy sends the alert certificate unknown and agentgateway resets the connection. In the second group the caller has the allowed identity atenet-client and gets an HTTP answer: 200 from my-counter-1 in 0.037 seconds on Envoy and 0.035 seconds on agentgateway, 404 for an actor that does not exist, and 404 for a request with no routing header. THE REQUEST ENVOY DATAPLANE AGENTGATEWAY DATAPLANE Refused in the TLS layer: no HTTP status no client certificate a pod with no certificate connection cut curl: Broken pipe TLS alert certificate required identity not on the list ns/probe/sa/default TLS alert certificate unknown connection reset reset by peer, no alert Allowed identity atenet-client: an HTTP answer actor my-counter-1 ACTOR_STATE_RUNNING 200 in 0.037 s 0.032 s through 8444 200 in 0.035 s 0.029 s through 8444 actor does-not-exist no such actor in the atespace 404 in 0.024 s body: actor not found 404 in 0.022 s body: ingress failed no routing header ate-target-actor is absent 404 in 0.018 s body: invalid actor reference 404 in 0.018 s body: ingress failed
Both dataplanes refuse a caller with no certificate or the wrong SPIFFE ID in the TLS layer, so the caller gets no HTTP status. Read each row from the request to its outcome on each dataplane. A dashed rose box is a refusal with no HTTP status. A solid rose box is a 404 with a body. A green box is a request that reached the actor, with the time at the caller.
outputno client certificate, agentgateway dataplaneFull log
* TLSv1.3 (IN), TLS alert, unknown (628):
curl: (56) OpenSSL SSL_read: OpenSSL/3.3.2: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0
exit=56
Three lines of curl output. The TLS setup lines before them are cut.

No refusal carries an HTTP status, so a client library reports a transport error. When you debug a caller, look for the TLS alert first. The agentgateway error body is also less useful after authentication: it says ingress failed for an unknown actor and for a missing header.

One claim in older material is wrong for an installed cluster. The agentgateway router does authenticate callers. The installer adds the client certificate check and the allowlist to every agentgateway install that it makes. We also installed with an empty allowlist. The rule in the router configuration became source.spiffeId in [], and the router refused both callers that we tried.

The identity in the quickstart does not exist

The quickstart passes one identity to the installer: spiffe://cluster.local/ns/ate-system/sa/atenet-client. After the install, the namespace ate-system has no ServiceAccount with that name.

outputServiceAccounts in ate-system after the installFull log
NAME                      AGE
ate-api-server            3m33s
ate-client                3m31s
ate-controller            3m31s
ate-idp-server            3m32s
atelet                    3m30s
atenet-egress             3m20s
atenet-router             3m28s
default                   4m18s
k8s-credential-provider   3m21s
The whole list, about three minutes after the install finished.

kubectl ate proxy still works, because it does not run as a pod. It asks the cluster for a certificate with that identity, and it names a pod and a ServiceAccount that do not exist. A pod of yours cannot do that. To call the router from a pod with the quickstart identity, we first ran kubectl create serviceaccount atenet-client -n ate-system. Then we mounted a pod certificate from the pod identity signer.

Do not copy that into production. The quickstart identity is one name that every user of kubectl ate proxy shares. Give each calling service its own ServiceAccount, and list each SPIFFE ID in --ingress-allowed-spiffe-ids. Treat the right to request certificates in ate-system as the right to become any component there.

Agent Substrate 1.0.0 · b34fbee · 2026-11-05 · Edition 2026.11