The dataplane choice: Envoy or agentgateway

The installer has a flag, --atenet-dataplane, with two values and no advice. If you install kagent, the choice is already made for you: it runs agentgateway. Either way you now operate one of two proxies. They do not behave the same when a request is refused or a rule is removed.

When you finish this section, you can choose a dataplane for a cluster and give the reason. You can also say what happens to an open connection after you remove its rule.

One flag changes both gateways

Dataplane: the proxy software inside the pods atenet-router and atenet-egress. The flag --atenet-dataplane takes envoy, the default, or agentgateway. One value applies to both pods.

The Substrate API does not change with the flag. Actors, policies, and kubectl ate work the same. What changes is the code that enforces a policy.

On the Envoy dataplane, two pieces of Substrate code decide. A Substrate process beside Envoy authenticates the actor, decides each HTTP request, and supplies the rules. A Substrate module that Envoy loads into its own process decides TLS connections by server name. On the agentgateway dataplane, policy code inside agentgateway v1.7.0 makes it. Two implementations of one policy explain every difference in this section.

You can switch a running cluster. Run deploy ate-system again with the same flags and the new value.

We measured the switch from Envoy to agentgateway once, on a second x86 server guest (16 vCPU, 23 GB). The command took 32 s and exited with 0. Two running actors stayed in ACTOR_STATE_RUNNING on their workers through the switch. We did not send requests during those 32 seconds, so we cannot say how many would fail. Raw runs

The full comparison

We captured the same inventory and ran the same probes on each dataplane, on one cluster.

We measured idle memory with one sample for each container, soon after start, on a second x86 server guest (16 vCPU, 23 GB). The five Envoy dataplane containers held 25.6, 21.6, 22.8, 12.0, and 8.9 MB. The two agentgateway containers held 19.9 and 17.8 MB. In bursts of up to 300 resumes, the Envoy process of the router used 0.12 to 0.19 cores on average. It did not set the rate, as section 4.5 shows. We did not load agentgateway, and we did not compare the two dataplanes under load. Raw runs

Three rows need a comment.

The log row is a volume and a content matter. At debug level, agentgateway prints the headers of each request. In our run it masked authorization and it never printed an injected secret. It did print, in clear, a custom header value that the actor sent. Store those logs with the same care as request headers.

The securityContext row is as the installer wrote it. We did not try to add one to the agentgateway containers.

The last row favours agentgateway, and the next subsection puts it in context.

An open connection outlives its rule

You remove a host from a policy because you want the actor to stop talking to it. The gateway applies the new policy to new work only. This is a known behaviour of v1.0.0, and both dataplanes have it.

Two mechanisms are behind it. The gateway keeps the policy of an actor in a cache, 10 seconds by default on Envoy, so a change takes effect late. And a decision is not revisited: an http or https decision holds for the request, and a tlsPassthrough decision holds for the connection.

We measured each case once for each dataplane, on the second x86 server guest (16 vCPU, 23 GB), with a poll about every 2 seconds. Zero is the moment the update returned. Raw runs

We measured the last two rows in a later run on the same guest (16 vCPU, 23 GB), once for each dataplane and each protocol. One curl process in the actor sent 20 requests on one connection, one every 2 s. We removed the rule about 13 s after the start. The start time of each run has a resolution of 1 s, so each time in those rows can be up to 1 s later. Raw runs

HTTP/1.1 200 OK
Date: Thu, 12 Nov 2026 10:57:50 GMT
HTTP/1.1 200 OK
Date: Thu, 12 Nov 2026 10:58:13 GMT
end=10:58:17

The 39 s and the 18 s mark the ends of our test responses. In no run did the gateway end an open connection, so we have no upper limit to give you. An actor that keeps one pass-through connection open keeps its access.

A reused connection is a different case, and it does not keep a removed rule alive. Under an http or an https rule, both dataplanes check the policy for each request. The next request on the open connection got 403, over HTTP/1.1 and over HTTP/2. The delay was the same as for a new request: about 1 s on agentgateway, and 5 to 9 s on Envoy. A new connection after the removal failed earlier, because the name did not resolve.

200 http=1.1 new_conns=0 lport=50509 t=0.012208
200 http=1.1 new_conns=0 lport=50509 t=0.008836
403 http=1.1 new_conns=0 lport=50509 t=0.007677
403 http=1.1 new_conns=0 lport=50509 t=0.009808
…

So the rule is: a removal stops requests, and it does not stop a response that is streaming or a pass-through connection. Removing a rule is therefore not a way to cut off an actor now. For an http rule, remove the rule and then suspend the actor.

We measured that cut-off once on each dataplane, on the same guest (16 vCPU, 23 GB). The actor held one open HTTP response from a host that writes one chunk each second. An asynchronous suspend actor returned in 0.64 s on agentgateway and in 0.54 s on Envoy. The upstream saw the connection gone 2.2 s and 1.3 s after the suspend call began. It writes once each second, so the real close can be up to 1 s earlier. Raw runs

The connection does not come back. After a resume, no new chunk reached the actor. The curl processes were still in the process list of the actor, restored from memory with sockets that no longer work. An application that you suspend must be able to reconnect. We used an http rule in this test. A repeat with a tlsPassthrough connection is planned for the next edition.

Our verdict

Choose Envoy, the default, for a cluster where actors run code that you do not trust and policies have https rules. The reason is the CA key. Envoy keeps it away from the process that parses actor traffic, and section 6.4 explains what that key can do. Envoy also gives better error bodies at the router. It is the only dataplane that accepts the experimental hook for an extra egress filter.

Choose agentgateway in three cases. You run kagent, which installs it. You run small clusters where 53 MB for each cluster matters. Or you already operate agentgateway and want one proxy to know. Section 7.2 covers that last case.

If you choose agentgateway, do three things. Limit who can read Secrets in ate-system and who can open a shell in atenet-egress. Send the egress logs to a store with access rules. And set RUST_LOG=info on the egress Deployment.

We measured that setting with 100 requests at each level, on the same guest (16 vCPU, 23 GB), in one run. At debug, the installer default, the gateway wrote 4,380 lines and 1,590,006 bytes. In 200 of those lines it printed the value of a custom header that the actor sent. At info it wrote 100 lines and 46,300 bytes, one line for each request, and no header value. At warn it wrote no line. All requests returned 200 at every level. Raw runs

kubectl -n ate-system set env deploy/atenet-egress RUST_LOG=info

The command restarts the egress pod. The installer writes debug again at its next run, so the setting is yours to keep. The level of atenet-router, and whether info logs enough to debug a denied request, are checks for the next edition.

Do not choose on speed. One allowed request took 0.037 s on Envoy and 0.035 s on agentgateway in our probes, and one request each proves nothing about load.