The kernel entered fast. The trace looked clean. The number was small enough to feel like progress.
Then I followed the request from the moment a user pressed Enter.
Before the workload could do anything useful, the host still had to find or build its artifacts, verify them, create storage, start broker processes, establish an authenticated channel, apply policy, activate the guest, deliver the environment, and wait for a real operation to cross the boundary and return.
I had measured one attractive span inside a much longer path and called it “boot time.”
The number was never false. The label was, and that was enough to send the optimization in the wrong direction.
Start the clock where the waiting starts
A launch is a sequence of transitions, not one event:
flowchart LR
A[Request] --> P[Prepare and verify]
P --> K[Kernel entry]
K --> R[Agent ready]
R --> X[Signed activation]
X --> E[Environment ready]
E --> U[First useful RPC]
U --> C[Reaped and cleaned]
Kernel entry is worth measuring because it helps explain where time goes. It isn’t the boundary the user experiences.
The more honest number is closer to:
Every term in that equation can be removed from a benchmark without being removed from the product. That’s how a fast demo turns into a slow system while every individual chart stays technically accurate.
Name the cold work
A truly cold request may have to resolve an OCI tag, download layers, build a guest image or kernel, generate dm-verity metadata, verify signatures, materialize an ext4 root filesystem, create a runtime overlay, and stage backend-specific state.
A warm request may do almost none of that.
Putting both under one “launch latency” number made it impossible to tell whether we were improving virtualization or merely hitting the cache more often.
MVM now separates the lanes.
A prepared-cold launch means the required artifacts are already local, verified, and compatible with the requested execution shape. An artifact miss is measured separately. A mount miss is separate. A warm claim is its own operation.
This makes the benchmark table a little less dramatic and the engineering work much more useful. When a percentile moves, we can ask which part of the lifecycle moved with it.
/bin/true can lie politely
A tiny command is a good overhead probe. It’s also a very easy place to stop the clock too early.
If a benchmark ends when /bin/true exits, it may never exercise the path a real workload depends on: authenticated service calls, input delivery, stdout collection, policy checks, network setup, artifact writes, audit emission, exit capture, and cleanup.
MVM still uses tiny commands to measure launch overhead. The lifecycle contract continues to the first useful RPC.
That boundary matters because a user doesn’t care that PID 1 exists. They care that the operation they requested can cross the real channel and return a result.
I started treating “agent ready” the way I treat “server process started.” It’s an internal milestone, not proof that the product is ready to serve.
Security does not happen outside the stopwatch
There’s an easy way to improve a latency graph: move inconvenient security work out of the measured span.
Verify the image later. Start the VM while policy is still loading. Reuse an authenticated channel. Append the audit event asynchronously. Restore a warm machine and assume yesterday’s authority is still acceptable.
The graph gets faster because the system has changed.
MVM’s budget includes the work that makes a launch mean what it claims to mean:
- artifact verification
- signed-plan admission
- a fresh boot identity
- broker binding
- policy activation
- authenticated readiness
- secret resolution at the host boundary
- evidence for the admitted launch
A 150 ms path that skips those steps isn’t the optimized version of a 220 ms secure path. It’s another product with a different contract.
Performance targets are only useful when they constrain the path we actually intend to ship.
Warm restore is reusing computation
I used to hear warm restore described as “booting faster.” That hides the interesting part: it’s reusing computation.
A clean parent has already paid for the kernel boot, initialization, and selected warmup work. Claiming a child is a copy-on-write operation followed by fresh identity, re-admission, and reconnection.
flowchart TB
subgraph Cold["Prepared cold"]
C1[Fresh VMM] --> C2[Fresh guest boot]
C2 --> C3[Authenticate]
C3 --> C4[Activate]
C4 --> C5[First useful RPC]
end
subgraph Warm["Warm claim"]
W1[Clean parent] --> W2[CoW child]
W2 --> W3[Fresh identity]
W3 --> W4[Re-admit and reconnect]
W4 --> W5[First useful RPC]
end
The warm target should be much lower because the operation is different. It still has to mint new authority rather than blindly resuming the parent’s live session.
MVM gives prepared-cold and warm-claim paths separate p50, p95, and p99 budgets. Those are engineering targets, not universal published results. A real result has to name the backend, host, artifact, machine shape, sample count, and measurement boundary.
The distinction between a target and a measurement is part of the honesty of the performance claim.
What goes in the cache key
A warm parent is reusable only when its immutable execution shape matches the child request.
That shape includes the kernel and initramfs digests, root filesystem layers, verity metadata, runtime overlay, backend and VMM version, guest-agent protocol, CPU and memory shape, device topology, network-policy shape, warmup profile, and readiness probe.
A loose key creates unsafe reuse. An excessively specific key destroys the hit rate. The identity has to include exactly what changes execution behavior while excluding tenant authority such as secrets, mutable state, and live sessions.
That’s why warm hit rate isn’t just a cache statistic. It tells us whether we’ve defined reusable computation precisely enough.
Density eventually argues with latency
One VM on an idle laptop can make almost any launch path look good.
At higher concurrency, latency starts negotiating with resident memory, page faults, file cache pressure, scheduler contention, storage bandwidth, broker count, queue depth, and cleanup lag. A “lightweight” warm parent that pins too much memory can perform beautifully at one instance and collapse at one hundred.
A useful sample therefore carries more than elapsed time. MVM records readiness and first-command latency, resident memory at readiness, growth after the first command, page-fault deltas where the host exposes them, contamination checks, and the exact backend and artifact identity.
If a counter is unavailable, it’s marked unavailable. It’s not reported as zero.
That small distinction prevents the report from inventing evidence where the platform supplied none.
Infrastructure lives in the tail
A median is comforting. Infrastructure is experienced in the tail.
A prepared-cold path that usually completes in 180 ms but occasionally spends 1.8 seconds materializing a filesystem isn’t meaningfully a 180 ms system. A warm pool that silently falls back to cold isn’t producing warm samples, no matter what label survives in the report.
The rule we use is simple:
When the mechanism changes, the label has to change with it.
A warm-required operation should refuse when warm capability is unavailable. It shouldn’t record a cold fallback as a successful warm claim.
That’s the same discipline I want for security claims: the interface shouldn’t preserve a stronger word after the guarantee underneath it has changed.
The request is not over when the command exits
My early benchmarks also stopped too soon at the other end.
After the workload returns, the host still has to capture the exit status, flush audit state, stop gateways, remove routes and firewall rules, release leases, close files, hand off or delete ephemeral state, and reap the VMM process.
If that work accumulates, the next launch inherits the mess. Ports remain busy. Memory stays charged. Network rules survive. State directories look live. The warm pool quietly loses capacity.
Cleanup has latency because ownership isn’t settled until the runtime has reclaimed the resources from the previous request.
A system that starts quickly and leaks every launch is only borrowing its performance from the future.
What “fast” means to me now
A useful performance statement carries its conditions:
prepared-cold
backend=HVF
host=Apple M-series
cpu=2
memory=512 MiB
artifact=<digest>
samples=20
warmups=2
boundary=request -> first useful RPC
p50=...
p95=...
p99=...Move the stopwatch to the user’s wait
For the next launch benchmark, write the boundary before you run it:
start = user request
stop = first useful operation returnsThen split the trace into preparation, boot, authentication, activation, environment delivery, and cleanup. If one of those spans is excluded, say so in the name of the number.
Without the lane, host, backend, artifact, and boundary, the number may still be good marketing. It isn’t a regression gate.
I still want MVM to feel immediate, and I don’t want to bury every number under caveats either. The optimization should aim at the path the user actually experiences.
Fast now means the artifacts are prepared when the lane says they are, verification has completed, policy is active, the guest is authenticated, the requested operation can run, and the runtime can cleanly own the next request afterward.
Two hundred milliseconds is a budget for a complete, named path through the system, and the cleanup span has to be in that name too — it was the part my early benchmarks kept leaving out.
The rule I keep now is short: the stopwatch starts when the user presses Enter and stops when the operation they asked for comes back. Write that into the benchmark’s name before you run it, not after you like the number.