The first time I got a microVM to start quickly, I measured the wrong thing.
The kernel entered fast. That felt great. It also said almost nothing about how long a user waited for useful work.
Before the workload could do anything, the host still had to find or build artifacts, verify them, create storage, start broker processes, establish an authenticated channel, apply policy, activate the guest, and wait for the first real operation.
“Boot time” had become a flattering label for one span inside a much longer path.
So I stopped using it as the main number.
Start the clock where the user starts waiting
A launch can be divided into phases:
flowchart LR
A[Request] --> P[Prepare and verify]
P --> K[Kernel entry]
K --> R[Agent ready]
R --> X[Signed activation]
X --> E[Environment ready]
E --> U[First useful RPC]
U --> C[Reaped and cleaned]
The number I care about is not only kernel entry.
It is closer to:
Each term can disappear from a benchmark without disappearing from the product.
That is how fast demos become slow systems.
Preparation is real work
A cold request may need to:
- resolve an OCI tag to a digest
- download layers
- build a guest image
- build or fetch a kernel
- generate dm-verity metadata
- verify signatures and hashes
- materialize an ext4 rootfs
- create a runtime overlay
- stage the guest agent
- prepare backend-specific state
A warm request may need none of those things.
Putting both into one “launch latency” bucket produces a number that is impossible to reason about.
MVM separates the lanes.
A prepared-cold launch means every required artifact is already local, verified, and compatible with the requested execution shape. An artifact miss is measured separately. A mount miss is separate. A warm claim is separate.
That makes the numbers less impressive in a slide and much more useful in engineering.
/bin/true is a probe, not a product
A tiny command is useful for measuring overhead. It is also easy to misuse.
If the benchmark stops when /bin/true exits, it may still avoid the path a real workload uses:
- authenticated service calls
- input delivery
- stdout collection
- policy checks
- network setup
- artifact writes
- exit capture
- audit emission
- cleanup
MVM keeps /bin/true as a launch probe, but the lifecycle contract continues to the first useful RPC.
That boundary is intentionally higher.
A user does not care that PID 1 exists. A user cares that the requested operation can cross the real channel and receive a result.
Security belongs inside the latency budget
There is a tempting way to hit a performance target: move security work out of the timed path.
Verify the image later. Start the VM before policy is ready. Reuse an authenticated channel. Add the audit event asynchronously. Restore a warm machine and trust that its old authority is still acceptable.
That can make a graph look better.
It also changes the system.
In MVM, the latency budget includes the parts that make the launch mean what it says:
- artifact verification
- signed-plan admission
- fresh boot identity
- broker binding
- policy application
- authenticated readiness
- secret resolution at the boundary
- evidence for the admitted launch
A 150 ms insecure path is not the faster version of a 220 ms secure path. It is a different product.
The target has to constrain the real path.
Warm is a different operation
Warm restore is often presented as a faster boot.
That wording hides the most important difference: warm restore reuses computation.
A clean parent has already paid for kernel boot, initialization, and selected warmup work. Claiming a child from it is a copy-on-write operation plus fresh identity and admission.
The expected latency should be lower because the operation is different.
flowchart TB
subgraph Cold["Prepared cold"]
C1[Fresh VMM] --> C2[Fresh guest boot]
C2 --> C3[Authenticate]
C3 --> C4[Activate]
C4 --> C5[First useful RPC]
end
subgraph Warm["Warm claim"]
W1[Clean parent] --> W2[CoW child]
W2 --> W3[Fresh identity]
W3 --> W4[Re-admit and reconnect]
W4 --> W5[First useful RPC]
end
MVM’s current engineering budgets treat these as separate lanes: prepared-cold has its own p50, p95, and p99 targets; warm claim has a much smaller target.
Those are targets, not a claim that every backend on every machine already meets them.
That wording matters. A budget tells the project where to go. A published result should say exactly where it was measured.
The artifact identity is part of the benchmark
A warm parent is only reusable when its immutable execution shape matches the child request.
That shape includes more than the kernel:
- kernel and initramfs digests
- rootfs lower layers
- verity metadata
- runtime overlay
- VMM and backend version
- guest-agent protocol
- CPU and memory shape
- device topology
- network-policy shape
- warmup profile
- readiness probe
Change one of those and the parent may no longer be valid.
This is why “warm hit rate” is not just a cache statistic. It reflects how precisely the system defines reuse.
A loose key creates unsafe reuse. An overly specific key destroys reuse. The template identity has to capture exactly what changes behavior and nothing that belongs to tenant authority.
Density changes the answer
A single VM on an idle laptop is not a fleet.
As concurrency increases, launch latency interacts with:
- resident memory
- page faults
- file cache
- scheduler contention
- storage bandwidth
- broker process count
- queue depth
- cleanup lag
A warm path that looks excellent at one instance may collapse at one hundred because each “lightweight” parent pins too much memory.
So the benchmark needs more than time.
For MVM, a useful warm sample includes:
- readiness latency
- first-command latency
- resident memory at readiness
- resident growth after the first command
- major and minor page-fault deltas where the platform exposes them
- contamination and capability checks
- backend, host, artifact, and sizing identity
A missing counter is marked unavailable. It is not reported as zero.
That sounds minor. It prevents a surprising amount of benchmark fiction.
p50 is not enough
Median latency tells you what a typical launch looks like.
Infrastructure is judged by the tail.
A cold path that usually hits 180 ms but periodically spends 1.8 seconds on filesystem work is not a 180 ms system. A warm pool that occasionally falls back to cold without saying so is not a warm pool.
MVM’s report gate uses multiple measured samples after warmups and keeps the lanes distinct. A warm-required operation should refuse if warm capability is unavailable rather than silently recording a cold fallback as warm success.
The rule is simple:
Do not let the label survive when the mechanism changed.
That applies to performance claims as much as security claims.
Cleanup has a latency too
Most benchmarks stop when the workload returns.
The host still has work to do:
- collect the exit status
- flush audit state
- stop gateways and brokers
- remove routes and firewall rules
- release leases
- close files
- delete or hand off ephemeral state
- reap the VMM process
If cleanup falls behind, the next launch inherits the damage.
Ports stay busy. memory remains charged. network rules survive. state directories look live. a warm pool loses capacity.
Reaping belongs in the lifecycle because the system is not ready for the next request until ownership is settled.
The number should carry its conditions
A useful performance statement has enough context to be falsifiable:
prepared-cold
backend=HVF
host=Apple M-series
cpu=2
memory=512 MiB
artifact=<digest>
samples=20
warmups=2
boundary=request -> first useful RPC
p50=...
p95=...
p99=...Without the lane, host, backend, artifact, and boundary, the number is marketing material.
Sometimes marketing material is appropriate. It should not be mistaken for a regression gate.
What I mean by fast now
I still want MVM to feel immediate.
The goal is not to make performance language cautious enough that nobody can be wrong. The goal is to make the work target the actual user-visible path.
Fast means:
- artifacts are already prepared when the lane says they are
- verification is complete
- policy is active
- the guest is authenticated
- the workload can perform its first real operation
- the runtime can cleanly own the next launch
Kernel entry is worth measuring. It helps explain where time goes.
It is not the result.
Two hundred milliseconds is not a boot time.
It is a budget for a complete, named path through the system.