I Deleted the Code That Worked

A refactor spec of ours said the raw-packet network path had been deleted. It had not.

We had deleted it once, brought it back a few weeks later as an opt-in compatibility mode, and the sentence in the spec had quietly stopped being true. No test failed. The build was green. The document was simply wrong, and it was wrong in the specific direction that makes a reader trust you.

The networking was never the problem. We had two networking paths and one security claim — and a claim that is true for only one of them is worse than no claim at all, because the reader cannot tell which one they are trusting.

So we gave the guest a TUN

MVM’s design principle is that the guest never holds ambient authority. A workload microVM has no NIC. There is no TAP device on the host, no bridge, no NAT rule, no packet filter. Every byte the workload sends leaves over a narrow authenticated channel to a per-VM endpoint on the host, and the host is the party that opens the actual socket.

That is the path we care about. I call it Path A.

flowchart TB
    W[Workload] --> L[Guest loopback adapter]
    L --> V[Authenticated vsock session]
    V --> E[Per-VM endpoint]
    E --> P[Policy, DNS, secrets, budget, audit]
    P --> O[Host-originated socket]
    O --> X[External network]

Path A is not free. A great deal of software expects ordinary sockets — package managers, Python’s requests, a browser, a language runtime’s own HTTP client. Under Path A those programs have to be adapted to a loopback proxy, a SOCKS5 tunnel, a controlled DNS stub, or a typed connector. Rewriting all of that around a custom transport would turn the security boundary into a compatibility project, and a boundary that is too painful to use gets routed around.

So we built Path B. A point-to-point TUN interface in the guest named mvm0, running its own normal TCP/IP stack, with raw IPv4 and IPv6 frames tunneled back to a host-side forwarder.

flowchart TB
    W[Workload] --> A[Path A - guest loopback adapter]
    W --> B[Path B - guest mvm0 TUN]
    A --> V[Authenticated vsock session]
    V --> E1[Per-VM endpoint]
    E1 --> O1[Host-originated socket]
    B --> F[Raw IPv4 and IPv6 frames]
    F --> D[Host forwarder with smoltcp]
    D --> N[Host TUN and nftables]
    N --> O2[Host socket]
    O1 --> X[External network]
    O2 --> X[External network]

Path B worked. It was tested. It had measurements. It is the reason a lot of ordinary software could run without being rewritten, and I was proud of it.

And it is the reason we had a security problem.

Two implementations of one question

Path B was not a second copy of Path A. It was a second implementation of the same decision — “may this workload reach that destination right now?”

That decision had two implementations, and they had:

• different policy code
• different resource accounting
• different audit event shapes
• different security properties

We also had a security claim in the posture document, and this was the wording:

Claim 10. No untrusted workload reaches the network unless explicitly admitted by policy.

Claim 10 is enforced by a gate that denies by default. NetworkPolicy starts at deny-all. A workload with no network grant has no network capability at all, and the NIC-less guest has no route around that absence. A workload with a narrow grant still starts from the same deny-all gate, and only an admitted typed flow can cause the host to resolve a name, connect a socket, or bind a listener.

With two paths, that claim is true of one of them.

And nobody can detect the difference from the outside. If a workload goes out over Path A, the policy gate ran. If it goes out over Path B, some other code ran that implements a similar rule with different edge cases. A reader — including a reviewer, including me six months later — cannot tell which enforcement handled a given request. The audit trail records that a connection was admitted. It does not record which of two code paths admitted it.

This is the sentence from the decision record that I keep coming back to:

Two policy implementations drift. One of them is always the one nobody audited most recently.

Not always the one that is wrong. But always the one you are least sure about.

Which means the practical failure mode is not an exploit. It is that you stop being able to answer questions about your own system, and you do not notice that you stopped.

Let me make the arithmetic explicit, because I think it is the part that generalizes beyond networking. Suppose a single path’s implementation fails to enforce the policy with probability $p$, and there are $n$ independent paths. Then the probability that at least one path lets something through is:

$$\Pr[\text{policy missed}] = 1 - (1-p)^{n}$$

Here $p$ is the per-path miss probability and $n$ is the number of independent implementations. Two implementations of the same rule are close to independent in the way that matters, because they share no code and therefore share almost no bugs. Going from $n=1$ to $n=2$ roughly doubles the chance of a miss, from $p$ to $2p - p^2$.

That probability is the lesser problem. With $n=2$, a miss is no longer attributable. You cannot fix what you cannot localize.

So we deleted Path B

The decision was to keep exactly one external networking path for an untrusted workload, and delete the other.

flowchart TB
    W[Workload] --> L[Guest loopback adapter]
    L --> V[Authenticated FlowMux session on vsock 5253]
    V --> E[One per-VM network endpoint]
    E --> P[Policy, DNS, substitution, redaction, budget, audit]
    P --> O[Host-originated socket or host-owned listener]

The endpoint is flow-aware at L4 with selective L7, and that distinction matters — I will come back to it.

TCP and UDP payloads whose signed plan requests no content transformation are relayed as opaque bytes. The host does not parse them, does not reconstruct a guest TCP stack, and does not claim to have inspected them.

L7 parsing, host-side TLS origination, credential substitution, and redaction run only for an explicitly typed HTTP or connector flow whose signed plan requires them. If a plan requires a transformation and the flow shape is opaque, admission refuses rather than silently downgrading.

The deletion list is long enough to be worth reading once, because every entry is a thing that used to work:

NetworkMode, L3Vsock, HostVsockProxy, raw_ip_stack, the guest mvm0 TUN, mvm-net-agent, mvm-netd, NetworkControl, NetworkData, the host TUN, nftables forwarding, and the smoltcp datapath. That is more than 41,000 lines removed across the contract, the guest agent, the host gateway, the VMM lifecycle, packaging, Nix, the kernel requirement, CI, fuzz targets, and the live-product test suite.

The compatibility we kept is narrower, and all of it terminates inside the one endpoint: the loopback HTTP proxy, SOCKS5h, SOCKS5 UDP, the controlled DNS stub, the mediated ping helper, and the typed SDK connectors. A program that ignores those has no network route and fails closed. That is a deliberate ceiling, not an unfinished feature.

The constraint we wrote down

While deleting the second path, I ran into a constraint that is more interesting than the deletion itself.

The obvious way to make every flow transformable is to terminate all guest TLS at the host. The host presents a certificate, the guest trusts it, the host reads and rewrites the plaintext. Done that way, the host can substitute a credential into any request, redact any response, and inspect everything the workload does.

It is also a trap, and I want to be precise about why, because the reasons are independent of how well it is implemented.

It is not achievable in general. Certificate pinning defeats it. Mutual TLS defeats it. QUIC defeats it. Encrypted Client Hello defeats it. And each of those is common in exactly the workloads that care most. A mechanism that silently stops applying to a subset of traffic is worse than one that never claimed to apply, because from the outside the two are indistinguishable.

It inverts the trust story. Our posture is that the guest is untrusted and the host does not need to read guest payloads in order to enforce policy. A universal interception CA makes the host’s ability to enforce depend on its ability to decrypt everything the workload does. That is a strictly larger and more fragile capability than the one claim 10 actually needs.

It enlarges the blast radius of the endpoint. The endpoint already holds credentials in the clear for substitution. Give it a CA key the guest trusts for every destination, and a per-destination compromise becomes a universal one.

It makes honest claims impossible. “Every flow is inspected” would be false in practice while being the documented behavior. We would be shipping a guarantee we know is not true, and the only way anyone would find out is by discovering it during an incident.

So we wrote the constraint down instead of hiding it:

Arbitrary guest-originated TLS and host-side credential replacement are mutually exclusive.

A workload gets one or the other, and it is told plainly which one it got. If the signed plan asks for host-side substitution or redaction, the workload declares a typed transform flow and the host originates the TLS itself to a plan-bound destination. If the workload wants ordinary end-to-end TLS, it gets ordinary end-to-end TLS, and nobody pretends the bytes were inspected.

There is a related decision inside the same ADR that I think generalizes even further, and it is about secrets rather than TLS. Our claim is:

Claim 13. The managed substitution path hands the guest placeholders, never raw secret values.

The guest gets an opaque token the host minted. The real credential stays on the host and is substituted into the outbound request at the last hop. If the destination sends the value back — in an echo endpoint, in an error page that quotes the request, in an API response that lists the caller’s own key — the endpoint scrubs it out of the response before the guest sees it.

And here is the detail I did not expect to have to care about: the request path and the response path are separate problems. A secret that cannot be exfiltrated by reading the guest’s memory can still be exfiltrated by making the destination hand it back. Writing the value into a request is half the work. The other half is making sure it cannot come home.

Two escape hatches we turned down

Two proposals came up during the deletion, and both were reasonable.

Both were rejected.

Transparent connect interception. The idea was to keep a general IP stack working by intercepting connect() calls — via ptrace, or via seccomp user-notification — so the host could apply policy to connections made by programs that ignore the supported adapters. The mechanism exists and it works.

It was rejected because it requires weakening PR_SET_DUMPABLE and ptrace isolation, or reading workload memory. Our own note in the decision record says it plainly: the isolation is worth more than the compatibility.

Keep the raw-packet path as a dev and test escape hatch. This one looked innocent. It is a dev-only path, so it cannot be reached in production, right? The problem is that a real alternate network implementation retained for testing is a second production path with a different label. The label is what people read. Hermetic protocol doubles are allowed; a second network stack is not.

I care about this one specifically now, in a way I did not when we wrote the decision. In a system that runs AI-generated code, an escape hatch is not a convenience. It is a thing the workload can discover, and a workload that can find it does not need to be malicious to use it — a confused agent debugging its own network failure is enough. Any boundary that can be turned off by the thing it is protecting is not a boundary.

So it cannot come back

A decision in a document is a decision until someone adds a feature. So the result is enforced by the build, not by discipline.

xtask check-single-network-path pins every claim-bearing runner to the shared endpoint spawner, requires the one NetworkFlow channel, rejects the retired L3 and NIC symbols, and inventories every production workload connect and listener bind. A separate socket-owner gate permits outbound connects and ingress binds only inside the endpoint, with narrow enumerated exemptions for unrelated host infrastructure. xtask check-one-guest-protocol rejects a guest caller of the network-flow port that does not construct a FlowMux client.

The important part is the negative fixtures. Each gate is tested with a synthetic forbidden case and with its exact allowed cases, so a second path does not fail by accident of review — it fails CI. A signed-plan projection test exercises TCP, UDP, DNS, typed connectors, and ingress against the same policy object, budget owner, identity, VM resource, and audit sink, which is how we prove that “one path” is a property of the object graph and not a comment.

The public surface is closed too. raw_ip_stack was removed from the Rust IR, the Python and TypeScript SDKs, the generated schemas, fixtures, examples, CLI help, and public docs. A stale serialized declaration fails at the outer compatibility boundary with a message that names the supported alternatives, and the rejected mode is not representable in the admitted domain types at all:

// crates/mvm-contract/src/ir/workload.rs
if wire.raw_ip_stack.is_some() {
    return Err(
        "raw_ip_stack has been retired; use the guest loopback HTTP proxy, SOCKS5h/UDP, \
         controlled DNS, mediated ping, or a typed connector",
    );
}

You cannot pass the old flag. You cannot ask for it in a manifest. You cannot smuggle it through a schema. The only thing you can do is read an error that tells you what to do instead.

Who paid for it

Some workloads genuinely need an in-guest IP stack, and they lost their supported path. raw_ip_stack=true now fails with a migration error. The migration was staged rather than atomic — during it, no new production L3 workload was allowed to boot, so the tree never ran three live production paths at once.

That is a real cost, paid by people who did not cause the problem. A boundary that only costs the people who build it is not a real boundary; it is a hobby.

The bill is still unpaid

Here is the part I am least comfortable writing, which is why I am writing it.

Before deleting the old path, we built a performance harness and recorded labelled baselines for both implementations. The thresholds are explicit and published: opaque latency at most 5% slower, throughput at least 95%, peak RSS growth at most 10%, and transformed HTTP latency at most 10% slower — unless an owner-approved measured exception is recorded.

The current comparison against the recorded legacy L3 baseline truthfully reads passed: false. Twelve of thirty-two checks pass. The maximum latency ratio is 14.5× and the minimum throughput ratio is 0.325×. The failures concentrate in opaque TCP (ten), UDP (six), and transformed HTTP (four).

For one concrete case, on the same host, same backend, same 64-byte payload, thirty samples:

Metric	Legacy L3	FlowMux	Ratio
Connect latency p50	109 µs	275 µs	2.52×
Connect latency p95	182 µs	299 µs	1.64×
Request latency p50	48 µs	152 µs	3.17×
Throughput	730,009 B/s	265,033 B/s	0.36×

The measured diagnosis is that the cost sits in short-flow authenticated framing and session setup, and in the endpoint relay path — not in steady-state transfer. That diagnosis is a measured one. It is not an acceptance exception, and no exception has been approved. The failing evidence is retained in the tree, the thresholds were not edited, and the final performance decision is still open.

I want to be careful about what I am claiming here. I am not claiming the new path is faster, which was the original hypothesis — the host does not parse every IP packet and does not reconstruct guest TCP in smoltcp, so it should win on bulk transfer, and it does not win on the cases that matter. I am also not claiming the work is done. What I am claiming is that the security decision was made on security grounds, the cost is measured rather than estimated, and the cost is currently unpaid.

That combination is the honest one, and it is also the only one that survives contact with a reviewer.

Where the cost lands matters too. The regression is concentrated in per-flow setup, which is exactly the shape of an agent’s network use: many small calls, not one large transfer. If this runtime is going to carry agent workloads, the setup path is the path that has to get fast, and that is a concrete engineering target rather than a vague hope.

Why agents make this worse

The reason I care about this decision more than the networking details is that agents make the general problem worse, and I do not think that is obvious enough yet.

An AI agent is a program that decides on its own what to do next. It reads data it did not write, calls tools, generates code, drives a browser, talks to services, and sometimes creates more agents. The difference from an ordinary service is not that it is smarter. It is that the sequence of actions is not known in advance, so you cannot enumerate the operations up front and approve them one at a time.

That kills the usual control. A tool registry checked against a fixed task list works when someone knows what the task is. When the task is open-ended, the list of things the workload might do is open-ended too, and a registry of named tools stops being the boundary — the process that holds the registry still has network access, credentials, a writable filesystem, and the ability to launch more work.

So the control has to move down, underneath the task, into the machine. And once it is down there, the question that matters is not “is the policy good?” but “how many places can authority cross the boundary?”

For an agent that can browse, call an API, run code, and spawn another agent, that is at least four capabilities. Four places authority can enter. If each capability carries its own policy implementation, you have four independent chances to be wrong — back to the arithmetic above, with $n = 4$ — and, worse, you cannot tell from the outside which enforcement actually ran for a given request. Your audit log will record that a request was admitted. It will not record which of four code paths admitted it.

That is the reason the count of enforcement points is a security property and not a tidiness preference. It is also the reason I would push back hard on a well-meaning feature request that says “let’s just add a special path for tool calls” or “let’s add a debug mode that relaxes the policy.” Both feel small. Both multiply $n$.

There is a second reason, and it is about honesty rather than probability.

Recall the sentence in the posture document that I opened with. The mechanism was sound throughout; the description of it was not. And the document is what a reviewer reads to decide whether to trust the posture. A runtime whose documentation describes one architecture while the tree contains two is not merely untidy — it is misreporting its own security surface, and it will do so in the direction that makes itself look better.

For an AI system this matters more, not less. When the workload’s behavior is not fully knowable in advance, the operator’s ability to trust the system has to come from the description being accurate and from the description being the same shape as the implementation. “We inspect everything” that is false in practice is worse than “we inspect the flows we declared,” because the first one leaves the operator unable to reason about what happened.

For me, security-first means three specific things:

1. The shape of the system follows from the security model, not the other way around. We decide where authority may enter, and then features that do not fit inside that boundary get refused rather than accommodated. Every feature request is also a request to expand the blast radius; the discipline is to make that expansion explicit, bounded, and attributable — or to refuse it.
2. A bypass is not a convenience. Any escape hatch exists for developers and is discovered by workloads. In an agent runtime it is not a compatibility feature, it is an attack surface with a friendly name.
3. The description is part of the security property. If the document and the tree disagree, the document is the bug, and the fix is to change the document or delete the code so the document can be true again.

One door

Two implementations of one security control is not redundancy. It is two chances to be wrong, and no way to tell which one was in play.

I used to think a second path was a safety net. It took me longer than it should have to accept that a second path is a second opinion nobody can compare against. If I cannot point at the single place where a decision is made, I do not have a security model — I have a set of security mechanisms and a document describing them.

The habit I picked up is smaller than a rule. Before I merge anything that touches the boundary, I count the places a request can be admitted. If the count is not one, I don’t merge.

The policy being too strict is recoverable — I can loosen it and ship again. The number of doors is not. When I build the AI infrastructure layer, I want to be able to draw the boundary on one page and put my finger on the one place where a request is admitted or refused. If I cannot do that, I have been counting features instead.

---

Next: I Took the NIC Away