What Result Would Make Us Stop?

I had spent days running experiments whose results could not change the next decision.

The jobs were real. The metrics moved. The artifacts looked scientific. I could point at a dashboard and describe what had happened.

What we had not decided was what any possible result would make us do.

How much improvement would justify adopting the idea? What regression would close the direction? Which baseline mattered? What would count as evidence that the instrument itself was broken? Without those answers, every outcome could be described as “interesting” and followed by another run.

That is not a research loop. It is an observation loop with no exit.

The discipline that grew inside R⁴ came from wanting the repository to remember more than the experiments that survived. A substantive run should begin with a null baseline, a falsifier, and a pre-declared decision rule. Negative results should remain searchable. A measurement that cannot turn red should be treated as a broken instrument.

The point is not to make research bureaucratic. I want each experiment to be capable of ending an argument.

Decide what the result will change before seeing it

“Try a better routing geometry and see whether quality improves” sounds like a plan until the output arrives.

How much improvement counts? On which data? Is a quality gain still worthwhile if latency doubles? Does a flat result refute the idea, or only show that the measurement is too noisy? If the result is worse, does the direction close or does the team immediately adjust three parameters and run it again?

A useful experiment attaches an action to the hypothesis:

flowchart TD
    H[Hypothesis] --> M[Measurement]
    M --> B[Null baseline]
    M --> F[Falsifier]
    M --> E[Exit threshold]

    B --> R{Observed result}
    F --> R
    E --> R

    R -- clears adoption rule --> A[Adopt]
    R -- informative but below rule --> K[Record and keep open]
    R -- contradicts hypothesis --> C[Close direction]
    R -- instrument cannot distinguish --> I[Repair instrument]

Let $\Delta$ be improvement over the null and $\tau$ the minimum gain that would justify the complexity. A rule might be:

$$\text{adopt} \iff \Delta_{\text{lower confidence bound}} > \tau$$

The threshold does not have to be statistical alone. It can include memory, latency, implementation complexity, or operational risk. What matters is that it exists before the result begins persuading us to move it.

Graphs are very good at making a marginal result feel meaningful after the work has already been spent.

The most useful result sometimes closes the feature

R⁴ explored a standalone two-pass generation path. The first pass would produce anchors; the second would use those anchors to improve generation.

With externally supplied good anchors, the channel carried useful signal. When the system had to predict its own anchors, quality degraded. Confidence gating did not rescue it. A larger run reproduced the inversion.

flowchart LR
    E[External anchors] --> P[Two-pass path]
    P --> G[Positive gain]

    D[Self-drafted anchors] --> P2[Same proposed mechanism]
    P2 --> N[Negative result]
    N --> C[Standalone direction closed]
    G --> I[Anchored infill remains valid]

It would have been easy to delete the branch and retain only the general impression that “two-pass generation did not work.”

The actual result was more precise:

• the channel itself can carry signal
• self-generated anchors introduce diffuse drift
• low-confidence steps are not the only cause
• more scale did not remove the failure
• externally anchored infill remains a different, viable mechanism

That negative result separated two ideas that looked almost identical in prose.

Keeping it in the repository prevents a future contributor—or a future agent—from rediscovering the appealing standalone version under another name and repeating the same search.

One knob can hide two different mechanisms

Another experiment increased the subdivision of the code space.

The requested mechanical effect happened. More stages created more occupied keys and fewer records per key. The instrument confirmed that the system had received the extra resolution we asked for.

Quality went down.

The immediate explanation was that sparse keys must be bad. Later work complicated that story. Improving codebook fit also reduced records per key, while quality improved slightly.

The useful distinction became:

• fit changes which key receives the evidence
• resolution changes how many keys divide the evidence

flowchart TD
    E[Observed evidence] --> F[Improve fit]
    E --> R[Increase resolution]

    F --> K1[Evidence moves to a better key]
    R --> K2[Evidence splits across more keys]

    K1 --> P[Measured small improvement]
    K2 --> N[Measured regression]

The failed run did more than reject one parameter setting. It refined the causal model.

Without that record, a future optimization might treat “fewer records per key” as a universal warning and discard a useful fit improvement for the wrong reason.

Negative results are not merely tombstones. Sometimes they are the only artifact that explains why two apparently similar changes behave differently.

The instrument has to prove it can notice the mechanism

At one point, multiple experimental arms reported clean all-zero results.

The output looked reassuringly unambiguous. It was also suspicious. A mechanism expected to move a signal had apparently produced nothing across every path.

That led to one of the bluntest rules in the R⁴ research record:

An all-zero result across every arm is a harness bug until proven otherwise.

A test can pass because the mechanism is correct. It can also pass because the mechanism never ran.

flowchart TD
    T[Test or measurement] --> P{Plant a relevant defect}
    P -- result changes --> S[Sensitive instrument]
    P -- result unchanged --> B[Broken or vacuous instrument]
    B --> X[Do not interpret original result]

Before trusting a measurement, we should know that it responds when the relevant behavior is removed or altered. Disable the routing term: does the metric move? Bypass corruption detection: does verification fail? Skip the experimental code path: does the execution counter drop?

This is mutation testing applied to the research apparatus.

A dashboard should not be allowed to lend authority to a result until the instrument has shown that it can distinguish the world with the mechanism from the world without it.

Zero and missing are not the same observation

The all-zero problem exposed another modeling mistake.

A metric value of 0 means the instrument ran and observed none of the quantity. A missing value means the measurement did not happen or was unavailable.

Those states should not share a number.

$$m \in \mathbb{R} \cup \{\text{not measured}\}$$

stateDiagram-v2
    [*] --> Planned
    Planned --> Measured: instrument ran
    Planned --> Absent: skipped / unavailable
    Measured --> Zero: observed 0
    Measured --> Nonzero: observed value
    Absent --> [*]
    Zero --> [*]
    Nonzero --> [*]

When a dashboard replaces absence with zero, it invents evidence. When an exit-rule report prints “NOT MET” despite a missing prerequisite metric, it implies that the experiment ran and failed rather than admitting that no verdict is possible.

Typed absence sounds like a small data-model improvement. It protects the conclusion from the presentation layer.

Sometimes the experiment is measuring the wrong kind of thing

One of the more consequential R⁴ investigations found that the deployed retrieval path compared incompatible vector meanings.

The index stored a content vector. The query used a routing vector. Their shapes matched, so the code ran. Cosine similarity returned plausible numbers. The geometric signal was effectively noise.

Earlier sweeps around projection shape and lexical weighting looked inert because the geometric term they were trading against was dead.

When the query was rebuilt from the same content representation as the stored side, retrieval quality moved sharply. A lexical weight that had appeared irrelevant became meaningful—and removing it improved the corrected path.

flowchart LR
    Q1[Routing vector query] --> C[Compare to stored content vector]
    C --> Z[Chance-level geometry]
    Z --> T[Knob sweeps look inert]

    Q2[Content vector query] --> C2[Compare like with like]
    C2 --> S[Geometric signal returns]
    S --> R[Re-evaluate old conclusions]

The category error was not visible in the tensor shape. Everything type-checked at the dimensional level. The values simply represented different things.

That discovery did not make the earlier measurements fake. It changed what they could support. Some negative verdicts became superseded because they had been collected against an inert signal. Gates that expected unrelated rows to move were themselves wrong.

A research record needs room for that kind of revision. Keeping old data does not mean freezing its interpretation forever.

A working mechanism can still produce a bad product

Research repositories often collapse two questions:

1. Did the proposed system mechanism work?
2. Did it produce a useful product outcome?

R⁴ can successfully compile an artifact, execute an allocation-free integer hot path, reproduce bytes deterministically, emit a content-addressed witness, and serve the runtime through an API.

Open-ended generation quality may still be weak.

flowchart TD
    M[Mechanism works] --> C[Compiler/runtime correctness]
    M --> D[Determinism]
    M --> W[Witness generation]

    Q[Product quality] --> O[Out-of-distribution behavior]
    Q --> I[Instruction following]
    Q --> U[User usefulness]

    C -. does not imply .-> U
    D -. does not imply .-> U
    W -. does not imply .-> U

A strong systems result should not be inflated into a model-quality claim. A weak model result should not erase real compiler and runtime progress either.

The honest state may be:

The mechanism is implemented and measured. The product hypothesis is not yet supported.

That sentence gives the next experiment somewhere to begin. “Early but promising” usually does not.

Closed directions should remain part of the map

Deleting failed implementation code may be the right maintenance choice. Deleting why it failed is not.

A durable experiment record should retain the hypothesis, the commit that instantiated it, the dataset and artifact revisions, the null baseline, the pre-declared exit rule, raw results, the interpretation at the time, later revisions, and the action the result caused.

This does not require preserving every speculative thought. It does require preserving decisions that narrow the search space.

A closed direction is a guardrail. It tells the next person that the absence of a feature is not necessarily an unfinished task; it may be the result of evidence.

The same status should become machine-readable. A capability can be marked:

• verified
• implemented but not externally validated
• open
• refuted
• superseded

flowchart LR
    D[Capability dictionary] --> S[Status ledger]
    S --> W[Witness / measurement]
    W --> C[CI honesty gate]
    C --> P[Published docs and API]
    C -. mismatch .-> F[Fail build]

CI can then reject impossible combinations: an open claim presented as verified, a verified claim without a witness, a refuted path enabled by default, or documentation that names a capability the artifact does not support.

Prose can explain the result. The repository should prevent the status from silently drifting away from it.

Failure makes the next ambitious experiment cheaper

Recording negative results is often described as humility. I think of it as leverage.

Deep research is expensive because the search space is large. A refuted hypothesis, a repaired instrument, or a sharper causal distinction removes territory from that space. The next run becomes more decisive because the repository remembers where the project has already been.

The alternative is a codebase full of attractive mechanisms with no durable explanation for why half of them are disabled.

Write the stop rule before the experiment

For the next run, write these sentences before collecting data:

• If the result is good enough, we will ____.
• If the result is bad enough, we will ____.
• If the result is ambiguous, we will ____.
• If the instrument cannot distinguish those cases, we will repair ____.

Then leave the original answers beside the result. Do not quietly move the threshold because the graph turned out to be interesting.

The days we lost to experiments with no exit rule changed how I want research to look in code. A run should not exist only to generate numbers. It should be capable of changing the plan, closing a direction, or proving that the instrument needs repair.

A research repository should not be a highlight reel. It should be the map, including the roads that ended, because those are often the roads that save the next person the most time.