The first fast microVM number I was proud of measured the wrong thing. I had a small number, a clean trace, and the wrong stopwatch.
The kernel entered quickly. The trace looked clean. The number was small enough to feel like progress.
Then I followed the request from the moment a user pressed Enter.
Before the workload could do anything useful, the host still had to find or build its artifacts, verify them, create storage, start broker processes, establish an authenticated channel, apply policy, activate the guest, deliver the environment, and wait for a real operation to cross the boundary and return.
I had measured one attractive span inside a much longer path and called it “boot time.”
The number was not false. The label was, and that was enough to send the optimization in the wrong direction.
Start the clock where the waiting starts
A launch is a sequence of transitions, not one event:
flowchart LR
A[Request] --> P[Prepare and verify]
P --> K[Kernel entry]
K --> R[Agent ready]
R --> X[Signed activation]
X --> E[Environment ready]
E --> U[First useful RPC]
U --> C[Reaped and cleaned]
Kernel entry is worth measuring because it helps explain where time goes. It is not the boundary the user experiences.
The more honest number is closer to:
Every term in that equation can be removed from a benchmark without being removed from the product. That is how a fast demo turns into a slow system while every individual chart remains technically accurate.
Cold work has to be named
A truly cold request may have to resolve an OCI tag, download layers, build a guest image or kernel, generate dm-verity metadata, verify signatures, materialize an ext4 root filesystem, create a runtime overlay, and stage backend-specific state.
A warm request may do almost none of that.
Putting both under one “launch latency” number made it impossible to tell whether we were improving virtualization or merely hitting the cache more often.
MVM now separates the lanes.
A prepared-cold launch means the required artifacts are already local, verified, and compatible with the requested execution shape. An artifact miss is measured separately. A mount miss is separate. A warm claim is its own operation.
This makes the benchmark table a little less dramatic and the engineering work much more useful. When a percentile moves, we can ask which part of the lifecycle moved with it.
/bin/true can lie politely
A tiny command is a good overhead probe. It is also a very easy place to stop the clock too early.
If a benchmark ends when /bin/true exits, it may never exercise the path a real workload depends on: authenticated service calls, input delivery, stdout collection, policy checks, network setup, artifact writes, audit emission, exit capture, and cleanup.
MVM still uses tiny commands to measure launch overhead. The lifecycle contract continues to the first useful RPC.
That boundary matters because a user does not care that PID 1 exists. They care that the operation they requested can cross the real channel and return a result.
I started treating “agent ready” the way I treat “server process started.” It is an internal milestone, not proof that the product is ready to serve.
Security does not happen outside the stopwatch
There is an easy way to improve a latency graph: move inconvenient security work out of the measured span.
Verify the image later. Start the VM while policy is still loading. Reuse an authenticated channel. Append the audit event asynchronously. Restore a warm machine and assume yesterday’s authority is still acceptable.
The graph gets faster because the system has changed.
MVM’s budget includes the work that makes a launch mean what it claims to mean:
- artifact verification
- signed-plan admission
- a fresh boot identity
- broker binding
- policy activation
- authenticated readiness
- secret resolution at the host boundary
- evidence for the admitted launch
A 150 ms path that skips those steps is not the optimized version of a 220 ms secure path. It is another product with a different contract.
Performance targets are only useful when they constrain the path we actually intend to ship.
Warm restore is not a faster cold boot
I used to hear warm restore described as “booting faster.” That hides the interesting part: it is reusing computation.
A clean parent has already paid for the kernel boot, initialization, and selected warmup work. Claiming a child is a copy-on-write operation followed by fresh identity, re-admission, and reconnection.
flowchart TB
subgraph Cold["Prepared cold"]
C1[Fresh VMM] --> C2[Fresh guest boot]
C2 --> C3[Authenticate]
C3 --> C4[Activate]
C4 --> C5[First useful RPC]
end
subgraph Warm["Warm claim"]
W1[Clean parent] --> W2[CoW child]
W2 --> W3[Fresh identity]
W3 --> W4[Re-admit and reconnect]
W4 --> W5[First useful RPC]
end
The warm target should be much lower because the operation is different. It still has to mint new authority rather than blindly resuming the parent’s live session.
MVM gives prepared-cold and warm-claim paths separate p50, p95, and p99 budgets. Those are engineering targets, not universal published results. A real result has to name the backend, host, artifact, machine shape, sample count, and measurement boundary.
The distinction between a target and a measurement is part of the honesty of the performance claim.
The cache key is part of the security model
A warm parent is reusable only when its immutable execution shape matches the child request.
That shape includes the kernel and initramfs digests, root filesystem layers, verity metadata, runtime overlay, backend and VMM version, guest-agent protocol, CPU and memory shape, device topology, network-policy shape, warmup profile, and readiness probe.
A loose key creates unsafe reuse. An excessively specific key destroys the hit rate. The identity has to include exactly what changes execution behavior while excluding tenant authority such as secrets, mutable state, and live sessions.
This is why warm hit rate is not just a cache statistic. It tells us whether we have defined reusable computation precisely enough.
Density eventually argues with latency
One VM on an idle laptop can make almost any launch path look good.
At higher concurrency, latency starts negotiating with resident memory, page faults, file cache pressure, scheduler contention, storage bandwidth, broker count, queue depth, and cleanup lag. A “lightweight” warm parent that pins too much memory can perform beautifully at one instance and collapse at one hundred.
A useful sample therefore carries more than elapsed time. MVM records readiness and first-command latency, resident memory at readiness, growth after the first command, page-fault deltas where the host exposes them, contamination checks, and the exact backend and artifact identity.
If a counter is unavailable, it is marked unavailable. It is not reported as zero.
That small distinction prevents the report from inventing evidence where the platform supplied none.
The tail tells the operational story
A median is comforting. Infrastructure is experienced in the tail.
A prepared-cold path that usually completes in 180 ms but occasionally spends 1.8 seconds materializing a filesystem is not meaningfully a 180 ms system. A warm pool that silently falls back to cold is not producing warm samples, no matter what label survives in the report.
The rule we use is simple:
When the mechanism changes, the label has to change with it.
A warm-required operation should refuse when warm capability is unavailable. It should not record a cold fallback as a successful warm claim.
That is the same discipline I want for security claims: the interface should not preserve a stronger word after the guarantee underneath it has changed.
The request is not over when the command exits
My early benchmarks also stopped too soon at the other end.
After the workload returns, the host still has to capture the exit status, flush audit state, stop gateways, remove routes and firewall rules, release leases, close files, hand off or delete ephemeral state, and reap the VMM process.
If that work accumulates, the next launch inherits the mess. Ports remain busy. Memory stays charged. Network rules survive. State directories look live. The warm pool quietly loses capacity.
Cleanup has latency because ownership is not settled until the runtime has reclaimed the resources from the previous request.
A system that starts quickly and leaks every launch is only borrowing its performance from the future.
What “fast” means to me now
A useful performance statement carries its conditions:
prepared-cold
backend=HVF
host=Apple M-series
cpu=2
memory=512 MiB
artifact=<digest>
samples=20
warmups=2
boundary=request -> first useful RPC
p50=...
p95=...
p99=...Move the stopwatch to the user’s wait
For the next launch benchmark, write the boundary before you run it:
start = user request
stop = first useful operation returnsThen split the trace into preparation, boot, authentication, activation, environment delivery, and cleanup. If one of those spans is excluded, say so in the name of the number.
Without the lane, host, backend, artifact, and boundary, the number may still be good marketing. It is not a regression gate.
I still want MVM to feel immediate. The lesson was not to bury every number under caveats. It was to aim the optimization at the path the user actually experiences.
Fast now means that the artifacts are prepared when the lane says they are, verification has completed, policy is active, the guest is authenticated, the requested operation can run, and the runtime can cleanly own the next request afterward.
Two hundred milliseconds is not a boot time. It is a budget for a complete, named path through the system—and every word in that name matters. I would rather publish a slower number that tells the truth about the wait than a faster one that only describes the nicest part of it.