Sandbox classes: gVisor and microVM

A security reviewer asks one question about your agent platform: is each agent in a container or in a virtual machine? With Substrate the honest answer is neither one, and the choice is yours. An actor runs in a gVisor sandbox or in a small virtual machine. The choice is a field of the WorkerPool, and it is hard to change later.

When you finish this section, you can choose a class for a pool and check that a node can run it. You can also name the behaviours that differ between the two classes.

Most measurements in this manual come from the gvisor class. On the x86 server guest we also ran one test actor on both classes, at 0, 200, and 500 MiB of memory. This section reports that comparison, and it names the runs that come next.

Two classes, one field

Sandbox class: the kind of isolation that the workers of a pool give to each actor. The value is gvisor or microvm.

You set the class in spec.sandboxClasses[0].name of the WorkerPool. The set is closed. The API accepts only those two names, and a plain Linux container is not an option.

What the node must provide

A gVisor worker runs on any Linux node. That is why it runs on a laptop.

A microVM worker needs the device /dev/kvm. atelet offers the device to Kubernetes as the resource ate.dev/kvm, only on nodes that have it. The controller adds a request for that resource to every microVM worker pod. So a microVM pool on a cluster without KVM does not fail loudly. Its pods stay Pending.

The script that creates the kind cluster tells you which case you are in. On the Apple silicon laptop under Docker Desktop it printed this:

Probing for /dev/kvm in the Docker environment...
/dev/kvm not available: micro-VM support disabled (gVisor still works).

On the x86 server guest, which has nested KVM, the same script printed the other answer:

Probing for /dev/kvm in the Docker environment...
/dev/kvm found: micro-VM (kata + cloud-hypervisor) support will be enabled.

On that guest the install needed no further step for microvm. The kind script had passed /dev/kvm to the node, and the installer had created the SandboxConfig.

So on a Mac under Docker Desktop you have the gvisor class only. On a cloud VM, the instance type must allow nested virtualization. Check the node type before you plan for microvm.

The installer prepares both classes by default. It creates one SandboxConfig for each class and loads the guest kernel and the other microVM files into the object store. Pass --sandbox-classes=gvisor to skip the microVM part.

How the limits apply

The ActorTemplate declares one CPU limit and one memory limit for the actor. The two classes turn them into different things.

CPU. A gVisor actor with a limit of 500 or 1000 millicores sees 2 CPUs, and the node kernel slows it to its quota. A microVM actor with 500 millicores starts with 1 virtual CPU and gets half a CPU of node time. A program that sizes its thread pool from the CPU count behaves differently on each class.

Memory. On gvisor the limit is the memory that the sandbox can use. On microvm the limit is the RAM of the guest, and the virtual machine monitor needs more memory on top. The worker counts that overhead against its capacity, so the same pool holds fewer microVM actors than the limits alone suggest. In our run the overhead was 7.6 MiB for an actor with a limit of 456 MiB.

Containers. An actor can have more than one container. On gvisor one Sentry process runs all of them, and a limit on one container has no effect. On microvm each container has its own cgroup in the guest. A container that passes its memory limit is killed alone, and the other containers of the actor continue.

That last behaviour has a cost that others reported. In issue 3615, a container in a microVM actor was killed for memory, and the actor stayed ACTOR_STATE_RUNNING with no log line. Actor and its seven states shows the same silence for a process that exits on gvisor, which we did measure.

A snapshot pins the class

The class is not only a property of the pool. The first snapshot of an actor records it, and every later restore of that actor needs a worker of the same class. This is the snapshot record of our counter actor after a synchronous suspend.

    fidelity: SNAPSHOT_FIDELITY_MEMORY
    generation: 3
    locality: kind-control-plane
    owner: SNAPSHOT_OWNER_ACTOR
    sandboxRuntime:
      sandboxClass: gvisor
      version:
        attributes:
        - key: architecture
          value: amd64
        - key: cpu_features
          value: 8c0000002e500800008003f70000000f00400004219c01ab178bfbfffff83203
        - key: sandbox_bundle_hash
          value: sha256:88e9809119d7fa2204897a60f8bb155add102018b4b4947e7876c9bc447312fa

A microVM snapshot records different attributes. This is the same part of the record for the counter actor on a microVM pool.

    fidelity: SNAPSHOT_FIDELITY_MEMORY
    generation: 5
    locality: kind-control-plane
    owner: SNAPSHOT_OWNER_ACTOR
    sandboxRuntime:
      sandboxClass: microvm
      version:
        attributes:
        - key: architecture
          value: amd64
        - key: cpu_model
          value: "23:49"
        - key: cpu_vendor
          value: AuthenticAMD
        - key: sandbox_bundle_hash
          value: sha256:1d40aaaf66896731f227d4cfe129ab724721b592ff89c73df603e8e2372a67b8

A gVisor snapshot names the CPU features as a bit string. A microVM snapshot names the CPU vendor and one CPU model. Three consequences follow from those records:

• You cannot move an actor to the other class. A template that selects pools of both classes still gives each actor one class, the class of its golden snapshot.
• A memory snapshot is tied to more than the class. It records the CPU architecture, the CPU features or the CPU model, and the exact sandbox build. A worker restores memory only when its runtime is compatible.
• The result of a mismatch is a cold start. When no compatible worker has room, Substrate takes a snapshot from the object store and restores only its files. It uses another worker of the same class. The processes start again, and the actor loses what it held in memory.

The third point is the one to plan for. A pool that mixes node types with different CPUs, or a rollout of a new sandbox build, can turn a resume into a cold start. Placement shows where that decision is made.

What others reported about parity

The two classes do not behave the same at v1.0.0, and the project tracks the gaps in issue 3226. These reports come from the issue tracker. We ran none of them.

The two classes, measured

We ran the test actor of SuspendActor and ResumeActor on a microVM pool. The machine, the node, and the actor image were the ones behind the gVisor numbers. The actor holds a fixed amount of random memory.

We measured both columns on the x86 server guest (24 vCPU, 47 GB, nested KVM), with one kind node and the Envoy dataplane. Each cell is a median. Each size had 5 first calls, 10 resumes, 10 asynchronous suspends, and 5 synchronous suspends. The first call and the resume are timed at the client, through kubectl ate proxy. All 90 requests of the microVM run returned 200. A request to a running actor took 0.065 seconds on microvm and 0.064 seconds on gvisor. Raw runs

The table has four results:

• A small actor pays a fixed cost on microvm. At 0 MiB the resume was 0.10 seconds slower and the first call was 0.22 seconds slower. The restore starts a guest with its own kernel before the actor runs.
• Memory costs less on microvm. From 0 to 500 MiB, a resume grew by about 1 ms for each MiB on microvm and by about 2 ms on gvisor. The medians were 0.02 seconds apart at 200 MiB. Above that size microvm was the faster class.
• A large new actor starts much faster on microvm. At 500 MiB the five first calls took 1.26 to 2.96 seconds on microvm and 1.86 to 6.76 seconds on gvisor.
• An asynchronous suspend costs the same. The medians differ by 0.06 seconds or less at every size. A synchronous suspend of a small actor is slower on microvm, because its snapshot is larger.

What one actor costs on the node

We measured these values on the same x86 server guest, with one sample for each cell. Resident memory is the value that ps printed on the node. The snapshot size is the disk use of the snapshot directory of one actor. The gVisor booking is from the pool of the second x86 server guest. Raw runs

A microVM snapshot holds the whole guest, with its kernel and its agent. It was 59 to 70 MiB larger than the memory that the actor held. Add that amount for each actor when you size the node disk and the bucket in sizing a pool. The file memory-ranges is sparse. Its listed size is the RAM of the guest, and its disk use is what the guest touched.

The clock inside the actor

The test actor reports the age of its process in every reply. The two classes count that age differently.

We measured three suspend and resume cycles of the 0 MiB actor on each class, on the same guest. One cycle took 1.6 to 1.8 seconds. On microvm the age grew by 1.8 or 1.9 seconds in each cycle: 28.4, 30.3, and 32.1 seconds. On gvisor it grew by 0.2 or 0.3 seconds in each cycle: 0.5, 0.8, and 1.0 seconds. Raw runs

So a microVM actor sees the time that passed while it was suspended. A gVisor actor sees only the time that it ran. A timeout, a cache lifetime, or a token check inside the actor gives a different result on each class. We read the elapsed-time clock of the process only, and we did not compare the calendar time.

Coming next

All of these numbers come from one x86 machine with one CPU model. The next edition runs microvm in these cases:

• on arm64
• with the agentgateway dataplane
• with a resume from the object store
• with more than 500 MiB held
• on a pool whose nodes have different CPU models
• with the loss of a node

The reports from the issue tracker in the table above also stay reports. We ran none of them.

Our position

Choose by the memory that your actors hold. With small actors, start with gvisor. It runs on every node, it resumed a small actor 0.10 seconds faster, and its snapshot is as large as the memory and no larger. It is also the class that kagent installs.

Choose microvm in three cases. A reviewer requires hardware virtualization between tenants. You need a limit for each container. Or your actors hold more than about 200 MiB. In the third case microvm resumed as fast or faster on our machine, and a new 500 MiB actor started four times faster. You pay with a node type that has KVM and with 59 to 70 MiB of extra snapshot for each actor.

Then test your own workload. The reports above say that the two classes differ in the cases that hurt: memory pressure and many small files.

Neither class makes the worker pod a boundary. Both worker types run as root with SYS_ADMIN and NET_ADMIN, and the threat model of Substrate rates the two classes as equal.