We lost days to experiments whose result could not change the next action.

The runs were real. The dashboards moved. The artifacts looked scientific. But we had not decided what evidence would make us keep the idea, change it, or kill it.

That is not research. It is expensive observation.

The discipline in R⁴ grew out of that frustration. Substantive experiments need a pre-declared exit rule, a null baseline, and a falsifier. Negative results stay in the repository. A measurement that cannot turn red is treated as a broken instrument.

The point is not bureaucracy. It is to stop the codebase from remembering only the ideas that survived.

A hypothesis needs an action attached to it

A common experiment plan looks like this:

Try a better routing geometry and see whether quality improves.

That is not specific enough.

How much improvement counts? On which dataset? Against which baseline? What happens if latency doubles? What result closes the idea? What result only means the instrument is too noisy?

A usable plan states the decision before the run:

flowchart TD
    H[Hypothesis] --> M[Measurement]
    M --> B[Null baseline]
    M --> F[Falsifier]
    M --> E[Exit threshold]

    B --> R{Observed result}
    F --> R
    E --> R

    R -- clears adoption rule --> A[Adopt]
    R -- informative but below rule --> K[Record and keep open]
    R -- contradicts hypothesis --> C[Close direction]
    R -- instrument cannot distinguish --> I[Repair instrument]

Let Δ\Delta be the measured improvement over the null and τ\tau the pre-declared threshold that would justify adoption.

The decision should be something like:

adopt  ⟺  Δlower confidence bound>τ\text{adopt} \iff \Delta_{\text{lower confidence bound}} > \tau

Not:

The graph looked promising, so we merged it.

The threshold does not have to be purely statistical. It can include latency, memory, implementation complexity, or blast radius. It needs to be written down before the result starts arguing for itself.

Negative results are part of the architecture

R⁴ has several directions that were measured and closed.

Standalone two-pass generation was refuted twice. The idea worked when good anchors were supplied externally. It degraded when the system had to predict its own anchors. Confidence gating did not rescue it. A larger run reproduced the inversion.

That result is more useful than an abandoned branch named experiment-final-2.

It says:

  • the channel can carry signal
  • self-generated anchors introduce diffuse drift
  • the failure is not isolated to low-confidence steps
  • more scale did not remove it
  • the standalone variant should not quietly return under a new name
flowchart LR
    E[External anchors] --> P[Two-pass path]
    P --> G[Positive gain]

    D[Self-drafted anchors] --> P2[Same proposed mechanism]
    P2 --> N[Negative result]
    N --> C[Standalone direction closed]
    G --> I[Anchored infill remains valid]

The failed experiment separates two mechanisms that looked similar in prose.

That is architectural knowledge.

“More capacity” is not one hypothesis

Another R⁴ experiment increased code-space subdivision.

The requested mechanical effect happened. More stages produced more occupied keys and fewer records per key. The instrument confirmed that the system had received the additional resolution the hypothesis asked for.

Quality went down.

The easy story would be “too few records per key is bad.” Later measurements complicated it. Improving codebook fit also reduced records per key, while quality increased slightly.

The useful distinction became:

  • fit changes which key evidence lands on
  • resolution changes how many keys divide the evidence

Those are not the same lever.

flowchart TD
    E[Observed evidence] --> F[Improve fit]
    E --> R[Increase resolution]

    F --> K1[Evidence moves to a better key]
    R --> K2[Evidence splits across more keys]

    K1 --> P[Measured small improvement]
    K2 --> N[Measured regression]

A negative result did more than reject a parameter value. It sharpened the causal model.

If the repository had kept only the successful codebook-fit change, future work might still waste time rediscovering that finer subdivision is not automatically better.

The instrument has to be able to fail

One of the strongest rules in the R⁴ research record is blunt:

An all-zero result across every arm is a harness bug until proven otherwise.

That rule exists because multiple instruments produced reassuringly clean zeroes while never exercising the behavior they claimed to measure.

A test can pass because the mechanism is correct.

It can also pass because the mechanism never ran.

The same problem appears in security tests, performance benchmarks, and conformance suites. A green result is only meaningful when the test has demonstrated sensitivity to the property.

flowchart TD
    T[Test or measurement] --> P{Plant a relevant defect}
    P -- result changes --> S[Sensitive instrument]
    P -- result unchanged --> B[Broken or vacuous instrument]
    B --> X[Do not interpret original result]

This is mutation testing applied to research apparatus.

If the routing term is removed, does the routing metric move? If the corruption check is bypassed, does verification fail? If the experimental code path is disabled, does the counter drop to zero?

The instrument should prove it can notice the thing before its result enters the research record.

Absence is not zero

A skipped measurement and a measured zero answer different questions.

  • 0 means the instrument ran and observed none of the quantity.
  • null means the quantity was not measured.

Collapsing those states makes dashboards look complete while hiding holes in the run.

Let mm be a metric value. The true type is not always a real number:

m∈R∪{not measured}m \in \mathbb{R} \cup \{\text{not measured}\}

Using zero as the second state invents evidence.

stateDiagram-v2
    [*] --> Planned
    Planned --> Measured: instrument ran
    Planned --> Absent: skipped / unavailable
    Measured --> Zero: observed 0
    Measured --> Nonzero: observed value
    Absent --> [*]
    Zero --> [*]
    Nonzero --> [*]

The same rule applies to verdicts. If a prerequisite measurement is absent, an exit-rule verdict should be suppressed—not printed as “NOT MET,” which implies the experiment ran and failed.

Typed absence sounds like a data-model detail. It protects the research conclusion.

Record the category error, not just the fix

A particularly useful R⁴ result found that the deployed retrieval path compared the wrong kinds of vectors.

The index stored a content vector. The query used a routing vector. The shapes were compatible, so the code ran. The cosine similarity was effectively noise.

Several earlier tuning experiments around projection shape and lexical weighting appeared negative or inert because the geometric term they traded against was dead.

Once the query was built from the same content representation as the stored side, retrieval quality moved sharply. A lexical weight that had looked irrelevant became meaningful, and removing it improved the corrected path.

flowchart LR
    Q1[Routing vector query] --> C[Compare to stored content vector]
    C --> Z[Chance-level geometry]
    Z --> T[Knob sweeps look inert]

    Q2[Content vector query] --> C2[Compare like with like]
    C2 --> S[Geometric signal returns]
    S --> R[Re-evaluate old conclusions]

The important record is not merely “fixed retrieval.”

It is:

  • the earlier measurements were real
  • their interpretation depended on a dead signal
  • some negative verdicts were superseded
  • the category error was kind, not shape
  • old gates that expected unrelated rows to move were themselves wrong

Research history needs revision without pretending the previous data never existed.

Mechanism success is not product success

R⁴’s compiler and runtime can work while generation quality remains weak.

Those are different claims.

A system may successfully:

  • compile a model into a new artifact
  • run an allocation-free integer hot path
  • reproduce artifact bytes deterministically
  • emit a content-addressed witness
  • serve an API

and still fail to produce useful open-ended text.

flowchart TD
    M[Mechanism works] --> C[Compiler/runtime correctness]
    M --> D[Determinism]
    M --> W[Witness generation]

    Q[Product quality] --> O[Out-of-distribution behavior]
    Q --> I[Instruction following]
    Q --> U[User usefulness]

    C -. does not imply .-> U
    D -. does not imply .-> U
    W -. does not imply .-> U

A research repository should make both sides visible.

Otherwise, a strong systems result gets inflated into an AI-quality claim, and a weak model result obscures real infrastructure progress.

The honest status can be:

The mechanism is implemented and measured. The product hypothesis is not yet supported.

That sentence is much more useful than “early but promising.”

Closed directions should remain searchable

Deleting failed code may be correct. Deleting the reason it failed is not.

A durable research record should retain:

  • the hypothesis
  • the implementation or commit that instantiated it
  • the dataset and artifact revisions
  • the null baseline
  • the exit rule
  • the raw result
  • the interpretation at the time
  • later revisions to that interpretation
  • the action the result caused

This does not require turning the repository into a lab notebook for every thought. It does require preserving decisions that constrain future work.

A closed direction is a guardrail for the next person—or the next agent—who sees the same attractive idea.

Research status should be machine-readable

Prose alone drifts.

If a public API exposes a capability, a model or ledger should say whether that capability is:

  • verified
  • implemented but not externally validated
  • open
  • refuted
  • superseded

CI can then enforce rules such as:

  • open claims cannot be presented as verified
  • every verified claim has a witness
  • every witness still exists
  • a refuted path is not enabled by default
  • an artifact’s declared capability matches its evaluation report
flowchart LR
    D[Capability dictionary] --> S[Status ledger]
    S --> W[Witness / measurement]
    W --> C[CI honesty gate]
    C --> P[Published docs and API]
    C -. mismatch .-> F[Fail build]

That is the same idea as schema validation for configuration. The repository should reject impossible status combinations before publishing them.

The point is to make ambition cheaper

Recording failure is sometimes described as humility.

It is more practical than that.

Deep research is expensive because the search space is large. Negative results reduce that space. A refuted hypothesis, a repaired instrument, and a narrowed causal model make the next experiment cheaper and more decisive.

The alternative is a codebase full of attractive mechanisms with no memory of why they were disabled.

A research repository should not be a highlight reel.

It should be the map of where the project has actually been.