Rigor & the evidence engine¶
A study's conclusion is computed from its evidence, not asserted. This chapter is about the machinery that makes that sentence true — the deterministic pipeline that reads a run, measures it, rolls the measurements up into a verdict, and writes the whole chain back into the study's own files so a human can audit every hop.
The evidence spine¶
An investigation is a structured research effort — a shared question, a set of studies, each with baselines, variants, runs, acceptance criteria, findings, and expert feedback. Traditionally, the connective tissue between "here is a run" and "here is what it means" is prose: an agent or a scientist writes a paragraph asserting that the model passed.
The evidence spine (the investigation spine) replaces that prose tissue with code. Information flows from sources to conclusions through deterministic Python functions that read structured data, compute results, and write them back into the project's YAML files. Nothing in this path is an LLM: it is ordinary, testable, reproducible Python. The AI's job is to author the inputs — the question, the tests, the mechanism prose — while the spine derives everything downstream.
You never hand-write status: pass. The verdict is a function
of the latest run's measured outcomes — which is exactly what stops a study from claiming a
result it never produced.
The parallel-slot convention¶
For every place a human (or an AI acting as author) records intent, there is a parallel slot the code fills with derived evidence. The two never share a field. Authored intent is preserved verbatim; computed evidence lands beside it; and the dashboard compares them.
| What it is | Authored slot (human intent) | Computed slot (code-derived) |
|---|---|---|
| Per-run test results | runs[].outcomes |
runs[].computed_outcomes |
| Study conclusion | executive.verdict |
.computed_acceptance |
| Gate state | pipeline_gate.gate_status |
.gate_evaluator |
| A finding | finding.statement / .summary |
finding.evidence / .expected / .provenance |
When the computed slot disagrees with the authored one, the reconciler raises a
diverges_from_authored flag — and that flag is the dashboard headline. A study whose
author wrote "passing" but whose code-computed acceptance says otherwise does not render as
passing; it renders as diverged, in red, with both values shown. Drift becomes impossible
to hide because the disagreement is a first-class, diffable fact.
Why the split earns its keep
Authored and computed values live in the same file but in different fields, so a single
git diff shows you both what a claim asserts and whether the evidence still supports it.
Auditability, drift detection, and self-documenting studies all fall out of this one
convention.
Where these live in code
All four coded slots are real fields in viva_superpowers, not just design vocabulary:
runs[].computed_outcomes (written by study_evaluator / study_outcomes),
executive.computed_acceptance (investigation_status), and
pipeline_gate.gate_evaluator (study_verdict.write_gate_evaluator) — the last two each
carrying the diverges_from_authored flag. The study_outcomes.sync() entry point drives
the reconciliation. Each write is a comment-preserving ruamel round-trip that only ever
fills the coded slot and never touches the authored one beside it.
Compute → Show → Act¶
The spine is layered into three planes. The bottom plane computes; the middle plane renders what was computed; the top plane turns the rendered state into the next scientific action.
1 · Compute¶
The eight-stage spine. Deterministic Python reads sources and runs, measures outcomes, rolls
them into verdicts and findings, and reconciles authored ↔ computed with one
study_outcomes.sync().
2 · Show¶
Spine presentation. Code-vs-authored chips, diverges_from_authored badges, per-readout
validation marks, and a verdict-annotated study graph — all rendered by the AI-free
Workbench.
3 · Act¶
The active layer — Propagate · Navigate · Guide. Expert feedback becomes a tracked action,
which writes a finding's next_action, which seeds a child study. The loop closes on itself.
The eight-stage spine¶
Every study's evidence flows through the same ordered stages. Each stage reads the one before it and writes a different part of the study file.
flowchart LR
I["01 Inputs<br/><small>sources · bands</small>"] --> D["02 Design<br/><small>readouts · tests</small>"]
D --> R["03 Runs<br/><small>runs.db</small>"]
R --> O["04 Outcomes<br/><small>measure · pass_if</small>"]
O --> V["05 Verdicts<br/><small>roll_up</small>"]
V --> F["06 Findings<br/><small>evidence slots</small>"]
F --> A["07 Acceptance<br/><small>investigation</small>"]
A --> Fb["08 Feedback<br/><small>next_action</small>"]
Fb -.->|seeds a child study| I
The reflexive loop at the end — feedback seeding the next study — is what makes the spine an engine rather than a report format: an investigation propagates, shows, and acts on its own knowledge.
The spine gates¶
Between stages sit gates: each is a deterministic verifier with a small, fixed result vocabulary. A gate never guesses; when it cannot resolve something it says so explicitly, in a bucket a human can act on.
| Gate | Question it answers | Result vocabulary |
|---|---|---|
| Observable | Does this readout resolve to a real emission in the model's structure? | ok · unresolved · not_in_structure · aspirational |
| Outcome | Did the run clear the test's bar? | PASS · FAIL · PARTIAL · BLOCKED |
| Citation | Is this acceptance band backed by a literature source? | cited · uncited |
| Parameter drift | Did the run honor the study's enforced parameters? | in-band · drifted |
| Study | Do this study's prerequisites and outcomes let it conclude? | passed · failed · needs_calibration · blocked |
| Investigation | Do the member studies' verdicts satisfy the acceptance criteria? | met · partial · unmet |
| Feedback | Has each expert item been resolved into an action? | open · actioned · closed |
These compose into a single gate trail — the audit path from a raw source to a resolved
expert note: source → band → readout validation → run → outcome → study verdict → finding →
investigation acceptance → feedback status.
The observable gate deserves special emphasis because it is the never-fabricate guard: a
readout marked not_in_structure means the study declared an observable the composite cannot
actually emit. The spine refuses to invent a number for it — it buckets it as unresolved and
tells you a human (or a model change) is needed.
The three rules¶
Every write the spine makes obeys three rules, and they recur across every design document.
Never clobber the human¶
Authored intent and machine evidence live in parallel fields. Code fills the absent
computed slot; it never overwrites what a person wrote. Your verdict stays exactly as you
typed it — beside, not under, the computed one.
Never guess¶
When something cannot be resolved, the spine buckets it (code · agent · needs_rerun)
rather than fabricating a verdict. A closed, eval-free DSL means a measurement either
computes honestly or reports that it could not.
Never destroy provenance¶
Every write is comment-preserving (ruamel round-trip) and lands as a git commit. Verdict, readiness, and open debts are reproducible, auditable, and diffable. The history is the evidence trail.
Behavior tests as machine-checkable specs¶
The unit that drives all of this is the behavior test: not prose, but a spec that names how to measure a result from a run and what passing means. Because the same definition drives both execution and the dashboard's pass/fail rendering, a test is required to have four properties:
- Executable — a
measure(a path into the run's structure, a reduction, a window) that a deterministic reader can evaluate againstruns.db, with no human in the loop. - Honest — a
gate_classthat distinguishes an acceptance criterion (a directional prior stated before the run) from a regression pin (a threshold set after seeing the data), so a study cannot quietly tune the model to pass a bar it drew after the fact. - Traceable —
requires_simulationandcitesbind the verdict to a specific run and to the literature, so the number and its justification travel together. - Renderable — one structured object drives execution and the pass/fail pill, so what you read on the dashboard is the same fact the grader computed.
A worked example: the DnaA-ATP fraction¶
Here is a real behavior test from the dnaa-replication line of work. It asks whether the
fraction of DnaA bound as DnaA-ATP stays inside its biologically expected band, in every
generation, over a multi-generation run.
behavior_tests:
- name: dnaa_atp_fraction_in_band
classification: primary
gate_class: acceptance_criterion
measure:
kind: generation_average
path: "MONOMER0-160[c] / (PD03831[c] + MONOMER0-160[c] + MONOMER0-4565[c])"
pass_if:
op: in_range_every_generation
low: 0.2
high: 0.5
requires_simulation: baseline
cites: [Boesen2024]
When the report card runs, the spine resolves each [c] token to a real store path, evaluates
the ratio with a safe AST resolver (no eval), averages it per generation, and checks the
band. It does not write pass: true; it writes a structured, per-generation measurement:
computed_outcomes:
dnaa_atp_fraction_in_band:
evaluated_by: code
result: PASS
operator: generation_average/in_range_every_generation
measured_value: { 1: 0.499, 2: 0.421, 3: 0.388, 4: 0.352, 5: 0.310, 6: 0.271, 7: 0.225 }
The verdict is now derived from those seven numbers. If generation 7 had dipped to 0.18,
result would flip to FAIL on its own — and if the authored executive.verdict still said
"passing," the diverges_from_authored flag would fire. The measurement chain that made this
possible runs structure → readout → self-describing store → RunReader → measure → verdict,
with the run reader living in the sibling emitters package so the dashboard never has to.
A behavior test is a machine-checkable spec, not prose.
Two studies that cite the same band and measure the same path will grade the same way, on any machine that can replay the run.
The lint backstop¶
The derive-on-read machinery keeps a single study honest. A second, independent
layer keeps the whole workspace honest: a static, fail-loud linter
(scripts/lint-workspace.py plus the pure checks in viva_genesis/audit_checks.py)
that runs before anything is trusted and exits non-zero on a hard failure. It is the
honest-bar backstop — it catches the ways a study or investigation can look finished
while quietly referring to something that isn't there.
The hard-fail checks that matter most:
| Check | Fails when… |
|---|---|
| YAML-parse | workspace.yaml (or an investigation.yaml) is missing or unparseable — a broken file can no longer silently skip the checks below |
| dangling-ref | a finding/behavior_test cites key is absent from papers.bib, or a prerequisites[].study doesn't resolve to a slug |
| dangling-edge-target | a parent_studies / parent_relations / builds_on / handoff edge points at a study or plan_id that no longer exists (added as PR #147) |
| gate-member-roster | a plan_gates[].member_studies entry isn't in the investigation's studies: roster |
| gate-matched-null | a gate authored passed isn't confirmed by the derived plan-progress (its matched-null check) |
| verdict-evidence | a study claims a pass but a primary behavior test has no PASS run outcome backing it |
| planned-but-passed | a planned baseline/arm also claims a result (gate passed / supported / Accepted) |
| artifact-missing | a ran/passed study's committed artifact file isn't on disk |
| datasets / references | a declared dataset or cited PDF is missing, or its sha256 doesn't match |
Be honest about the limits. The linter does not re-run simulations or re-derive any
number — it checks that recorded outcomes[].result tokens are PASS and that
artifact files exist, not that the values are right. Value-level concerns
(knife-edge margins, missing _units) are WARN-only heuristics, and per-study
JSON-schema / dashboard load_spec correctness is advisory here and enforced
dashboard-side.
Alongside it, the Workbench adds a live, graded assessment the linter can't give:
the deterministic L0–L5 reproducibility audit ladder
(below; an investigation grades as its weakest
member) and a composite contract audit (composite_audit.py: per-process contract
completeness and unit-compatible wiring at every shared store). The linter is a static
integrity gate over YAML, source, and files; the Workbench audit is a live
reproducibility-and-contract report. They are complementary.
The reproducibility audit ladder (L0–L5)¶
The spine gates above grade a study's evidence; a second, orthogonal ladder grades its
reproducibility — whether the study could be rebuilt and re-run from its own files at
all. The L0–L5 audit (study_audit.py, surfaced at /api/study-audit) walks six
ascending tiers, each a set of named deterministic checks. A study — or an investigation,
graded as its weakest member — earns a level only when every check below it passes.
| Level | Asks | Representative checks |
|---|---|---|
| L0 · Canonical | Is the spec even in canonical shape? | canonical-model-schema (no top-level baseline: / variants:), no-nested-study (study lives flat, not under investigations/**/) |
| L1 · Resolvable | Do its declarations resolve? | composite-resolves · params-are-generator-accepted · inputs-from-resolves |
| L2 · Keyable | Can every run be content-addressed? | node-keyable |
| L3 · Recorded | Did the runs leave outputs? | outputs-present |
| L4 · Graded | Is there a real report-card verdict? | report-card-verdict |
| L5 · Executable | Is the dependency graph a runnable DAG? | dag-acyclic · no-dangling-edges · topological-executable |
The ladder is tiered by severity. Hard-tier failures hard-gate --gate CI — an L0
no-nested-study or canonical-model-schema violation, or an L5 dangling edge, fails the
build rather than merely warning. That is what makes the canonical layout and the flat
studies/ convention enforced, not just recommended.
The audit's deeper purpose is to catch post-hoc rationalization — the ways a study can be dressed up to look settled after the fact:
- preregistration — acceptance thresholds must be pre-registered, not back-filled after seeing the data. A bar drawn to fit the result it was meant to test is not evidence.
- supporting-provenance — a finding's claim must trace to a resolvable
evidence.from_testand to realprovenance.run_ids, so a conclusion cannot cite runs that do not exist.
Where the spine gates ask "does the evidence support the verdict?", the audit ladder asks "could anyone else stand this study up again and get the same answer?" — and both must clear before a result is trustworthy.
Hardening a study¶
A study that looks done is not the same as a study whose conclusions survive scrutiny. Hardening closes the gap between the two through a small set of concrete, verifiable moves (the signature of hardening as an agent-applicable, human-verifiable transformation is covered in Studies).
Three moves carry most of the work — one standalone skill plus two of its sibling subcommands:
| Move | What it hardens |
|---|---|
/viva-harden-investigation |
Finds the one load-bearing gap that most weakens the headline claim — an unbacked scaffold, a knife-edge single-seed pass, an overclaimed verdict, a failing gate with a real divergence — and closes that one the right way for its kind, rather than pouring generic rigor onto an un-diagnosed failure. |
/viva-tests cite-bands |
Surfaces candidate evidence from expert PDFs for uncited acceptance bands and writes structured cites / calibration_anchor provenance into the study — so a threshold traces to a paper instead of being a magic number. Every write goes through the sanctioned POST /api/band-provenance path; the skill never fabricates a threshold. |
/viva-harden-investigation biology-forward |
Runs the deterministic populate step first (populate_finding_observations fills evidence.observed, expected.range, divergence_factor, provenance.run_ids from the computed outcomes and bands), then guides the author to write the biological interpretation over that scaffold. A divergence_factor > 0 forces the status to partial or contradicts — never confirms. Numbers first, mechanism prose second. |
Skill surface consolidated (#281)
At the current package HEAD the /viva-* surface was consolidated (21 → 17 skills), so
band-citing is now the cite-bands subcommand of /viva-tests and
biology-forward authoring is a mode of /viva-harden-investigation. Older installed plugin
builds may still expose /viva-cite-bands and /viva-biology-forward as standalone skills;
the behavior is the same either way.
Warning
Hardening never means "sprinkle more seeds everywhere." A failing report-card gate with a real divergence must be root-caused before it is patched — and a verdict of real and understood (no bug found) is itself a complete hardening. The point is to explain the signal, not to bury it under undifferentiated rigor.
Next: Working with AI agents