TLS interception and the trust bundle

Your agent calls an API over HTTPS, and the policy allows the host. The call fails with a certificate error that you cannot reproduce from your laptop. The host is fine. The actor talked to atenet-egress, which answered with a certificate from a private CA, and the client in the actor does not trust that CA.

When you finish this section, you can choose between an https rule and a tlsPassthrough rule for a host. You can also give an actor the trust bundle, and say where the CA private key lives on each dataplane.

Two rule kinds, two certificates

Both rule kinds allow TLS to a host. They differ in who holds the session.

With an https rule, the gateway ends the TLS session of the actor. It presents a certificate that it created for the requested name, reads each request, and applies the policy to each one. Then it opens its own TLS session to the real host and verifies that host against public roots.

With a tlsPassthrough rule, the gateway reads only the server name in the handshake. If a rule lists that name and port, the gateway forwards the bytes and never decrypts them.

We allowed example.com with https and example.org with tlsPassthrough, and we printed the certificate that the actor received from each.

issuer=CN=U7ICLCR3N66DBMBSAQNS5JHTHI
subject=
notBefore=Nov 12 10:48:04 2026 GMT
notAfter=Nov 12 11:08:04 2026 GMT
X509v3 Subject Alternative Name: critical
DNS:example.com

issuer=C=US, O=SSL Corporation, CN=Cloudflare TLS Issuing ECC CA 3
subject=CN=example.org
notBefore=Sep 26 22:05:05 2026 GMT
notAfter=Dec 25 22:13:27 2026 GMT

Our rule: use https for every host that needs a credential, and for every host where you want a decision for each request. Use tlsPassthrough for a client that pins certificates or brings its own client certificate. Know what you give up with it. The gateway decides once, and the connection then carries whatever the actor sends.

Give the actor the trust bundle

Without the bundle, a client in the actor rejects the minted certificate. With the bundle, the same request works.

$ curl -sS -m 15 -o /dev/null -w 'code=%{http_code} verify=%{ssl_verify_result}\n' https://example.com/
exit=60
code=000 verify=20
…
stderr: curl: (60) SSL certificate problem: unable to get local issuer certificate
…
$ curl -sS -m 15 -o /dev/null -w 'code=%{http_code} verify=%{ssl_verify_result} http=%{http_version}\n' --cacert /run/ate/egress-trust-bundle/trust-bundle.pem https://example.com/
exit=0
code=200 verify=0 http=2

Trust bundle: a file that Substrate projects into the actor, with the interception CA of the cluster and a set of public roots. You ask for it in the ActorTemplate with a systemInfo volume.

volumes:
- name: system-info
  systemInfo:
    dataSources:
    - trustBundle:
        names:
        - egress-mitm.ate.dev
        - system-roots.ate.dev
        path: egress-trust-bundle/trust-bundle.pem

The first name is the interception CA. The second is the set of public roots, so one file serves intercepted hosts and pass-through hosts. Our bundle held 149 certificates in 222,672 bytes.

Most clients do not read that path on their own. Setting SSL_CERT_FILE to the path was enough for curl in our run. Other runtimes read other variables, such as REQUESTS_CA_BUNDLE for Python requests and NODE_EXTRA_CA_CERTS for Node.js. Set them in the env of the template, not in the agent prompt.

Do not copy the bundle into the container image. The CA can rotate, and the projected file follows the rotation. A copy in an image does not.

The leaf certificate differs between the dataplanes

Both dataplanes sign with the same CA. The certificates that they mint do not look alike.

On Envoy a separate process mints each leaf with the flag --leaf-cert-ttl=15m, and it dates the start 5 minutes into the past. On agentgateway the leaf is valid for 2,121 years.

For the client, both work. For you, the difference changes two things. First, the error text for a missing bundle differs, so search for both strings when you debug. Second, do not write a test that asserts a field of the leaf. A switch of dataplane changes every row of that table.

We prefer the short lifetime of the Envoy leaf. A leaf that stays valid for centuries makes the CA the only control. If the CA key is exposed, a rotation of the CA is the single way to stop a forged certificate.

Where the CA private key lives

The interception CA is the Secret egress-mitm-ca-pool in the namespace ate-system. Whoever holds its private key can present a trusted certificate for any host to every actor that trusts the bundle. So the place where that key is mounted matters more than any field of the leaf.

On the Envoy dataplane, the pod atenet-egress has three containers. The key is mounted into one of them, sdsmint, which serves a Unix socket and no network port. Envoy asks it for certificates over a Unix socket. The process that parses traffic from actors never holds the key.

sdsmint --uds-path=/var/run/sdsmint/sdsmint.sock --ca-pool-path=/run/ca-state/mitm-pool.json --leaf-cert-ttl=15m --log-level=info …

On the agentgateway dataplane, the pod has one container, and that container mounts the key.

tls:
  mode: dynamicCa
  cert: /run/egress-mitm/tls.crt
  key: /run/egress-mitm/tls.key

The same process accepts connections from every actor, parses their requests, and holds the signing key. In our install that container also had no securityContext. The envoy container drops all capabilities and runs as a user other than root.

We do not know of an attack that reads the key through agentgateway. Our point is about the size of the loss if one appears. On Envoy a memory disclosure bug in the proxy exposes traffic. On agentgateway the same class of bug exposes the CA. On either dataplane, limit who can read Secrets in ate-system, because the Secret is the same.

Which upstream certificates the gateway trusts

Interception has a second TLS connection, from the gateway to the real host. The actor trusts the gateway through the bundle. The gateway must trust the host by itself. On both dataplanes it reads the roots from one file in its own container, /etc/ssl/certs/ca-certificates.crt. The agentgateway configuration names that file as backendTLS.root, and the Envoy configuration names it as trusted_ca. Substrate v1.0.0 has no flag and no documented setting for one more upstream CA.

So a host with a certificate from a private CA fails under an https rule. We ran a gateway in the cluster with such a certificate, and an actor called it through an https rule.

We measured this once on each dataplane, on a second x86 server guest (16 vCPU, 23 GB), with 30 requests each. With the default install, 0 of 30 requests succeeded on either one. On agentgateway the actor got 503 with the body upstream connection failed, and the gateway logged invalid peer certificate: UnknownIssuer. On Envoy the actor got 503 with upstream connect error or disconnect/reset before headers. reset reason: remote connection failure. The Envoy log had CERTIFICATE_VERIFY_FAILED. Raw runs

The fix that worked is a mount over the root file. We put the public roots and our CA in one ConfigMap, 122 certificates in all, and mounted it in the proxy container of atenet-egress.

[
  {"op": "add", "path": "/spec/template/spec/volumes/-", "value": {"name": "extra-roots", "configMap": {"name": "egress-extra-roots"}}},
  {"op": "add", "path": "/spec/template/spec/containers/0/volumeMounts/-", "value": {"name": "extra-roots", "mountPath": "/etc/ssl/certs/ca-certificates.crt", "subPath": "ca-certificates.crt"}}
]

With the patch, the same requests returned 200 on both dataplanes, 30 of 30. The patch is outside the installer, and it does not last. After a later ate-setup deploy ate-system run, the proxy container had no such mount. Apply the patch again after each installer run, and test one request to the private host each time.

The actor never sees the private CA. It sees a leaf for the host name that the interception CA minted, and it verifies that leaf with the projected bundle.

What a new intercepted connection costs

We measured the same private host on the same guest (16 vCPU, 23 GB), with a request of 60 bytes. With a new connection for each request, the median was 618 ms on agentgateway and 614 ms on Envoy, with n=10 each. The TLS handshake with the interception endpoint took 606 and 600 ms of that. On one reused connection, the median was 10.0 and 11.9 ms, with n=30 each. An http rule to a host in the cluster cost 23 to 28 ms for a new connection. Raw runs

So interception is cheap for each request and costly for each connection, about 0.6 s on both dataplanes. A client that opens one HTTPS connection for each call pays it each time. Keep connections open, or use a client with a pool. We did not find which step inside the handshake takes the time. agentgateway with Substrate has the full table.