---
title: Benchmark methodology
---

This page covers what the harnesses compute, how a delta is judged against noise, when
two numbers compare, and where the numbers are biased. The rules live in code:
`bench/_gate.py` (shared gate), `bench/bootstrap.py`, `bench/judge_calibration.py`,
`bench/bench_namespaces.py` and `bench/failure_taxonomy.py`
([source entry points](#source-entry-points)).

## Recall and answer accuracy are separate axes

**Recall@K** asks whether a relevant memory landed in the top K. **Answer accuracy** asks
whether the synthesized answer is correct. One can saturate while the other stays low.
`longmemeval.py` prints recall@K and judged accuracy as separate metrics;
`coding_agent_qa.py` prints R@5 / R@10 / MRR apart from judged QA accuracy. A headline
from an embedder with no reader and no judge is a recall figure. Do not set it beside an
accuracy number.

## Task categories

Each harness keeps its source's category labels, so a result reads per category.

| Harness               | Categories                                                                                                                                                      |
| --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `locomo_qa.py`        | 1 multi-hop, 2 temporal, 3 open-domain, 4 single-hop, 5 adversarial. `overall_comparable` covers 1–4 (the published column); `overall_all` scores 5 as abstain. |
| `longmemeval.py`      | The dataset's `question_type` (`--question-type`); `--stale-check` labels wrong knowledge-update answers that returned the superseded value.                    |
| `memoryagentbench.py` | Accurate retrieval, test-time learning, long-range understanding, conflict resolution.                                                                          |
| `beam.py`             | Ten memory abilities, graded with the dataset's rubric judge; event ordering adds Kendall τ_b × F1.                                                             |
| `membench.py`         | Participation and observation scenarios at factual and reflective levels.                                                                                       |
| `code_intel_qa.py`    | Row `type`: `symbol_def`, `callers`, `conceptual`, `hub`, `history`, `test_set`, `abstain_probe`; plus `lane` and `facets`.                                     |
| `coding_agent_qa.py`  | Bug-fix landing, infra fix, refactor location, flaky test, perf leak, schema migration, multi-session causal, temporal, team convention.                        |
| `op_planning.py`      | Each task's `class`: `instruction_type` × `structure`.                                                                                                          |

## Ground truth

| Kind                            | Where                                                                                                | How it is built                                                                                                                    | Weakness                                                                       |
| ------------------------------- | ---------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------ |
| Substring gold keys             | `bench/datasets/synthetic_memory_qa.json`, `extraction_regressions.json`, `graph_regressions.json`   | Hand-written; a hit is any gold string in the returned `memory_key` or excerpt (case-folded, `_` as space)                         | Lenient on purpose; a verbose excerpt can match by accident                    |
| Published gold answers          | Third-party datasets ([datasets](/benchmarks/running#datasets))                                             | The dataset's answer and evidence labels                                                                                           | Inherits the dataset's label errors and saturation                             |
| Hand-labelled query → key pairs | `retrieval_recall.py`, `plane_ablation.py`, `hard_distractors.py` `--dataset`                        | A consumer labels queries against an existing corpus                                                                               | Small sets; labels stay out of the repository                                  |
| Labelled by use                 | `relied_on_set.py` → `bench/datasets/relied_on/`                                                     | A search whose returned keys an agent later relied on (`used` / `used_cited` events) inside a short window                         | Joined by time, not id; favours what the current ranking already surfaced      |
| Mined from merged PRs           | `bench/fixtures/code_intel_mined_*.jsonl`, `code_intel_history.jsonl`, `code_plane_ablation.py mine` | Query = issue or PR title; gold = changed `src/` files. Rows whose text names a gold path stem are dropped, and the share reported | The index contains every fix: absolutes are an upper bound, read paired deltas |
| Model-authored, model-verified  | `bench/fixtures/code_intel_oss.jsonl`                                                                | A third-party corpus whose queries and paths a model wrote and a model judge checked                                               | A miss may be the label's; `secondary` paths are never credited                |

Structural code gold (`symbol_def`, `callers`) is exact but was written from knowledge of
the upstream repository. After every fresh backfill, run once without gates, read the
mismatches and correct the gold: a wrong gold is a corpus bug.

Judge calibration builds its own probes (`build_probes`): an item's gold phrased as a
reader answers is a known-correct probe; another item's gold from the same category is a
known-wrong probe. Adversarial items are excluded.

## Judges

The reader and judge are part of the instrument, not the system under test.

- **Pinned per harness.** `longmemeval.py` and `coding_agent_qa.py` read with
  `gpt-4o-mini` and judge with `gpt-4o-2024-08-06` (the LongMemEval judge snapshot).
  `locomo_qa.py` reads with `gpt-4.1-mini` and judges with `gpt-4.1`.
  `BENCH_READER_MODEL` / `BENCH_JUDGE_MODEL` override them and print a comparability
  warning; `OPENAI_BASE_URL` points at a compatible gateway.
- **Grading Loom's answer.** `longmemeval.py --answerer loom-ask` and
  `locomo_qa.py --reader ask` grade `memory.ask`; `--ask-arms` compares `memory.ask`
  configurations on one ingest.
- **Rubrics are files.** `bench/rubrics/entailment.md` is the NLI prompt
  `dialogue_consistency.py` loads. `beam.py` uses the dataset's rubric judge.
- **Agreement.** `locomo_qa.py --second-judge-model` re-grades a sample and reports
  agreement and Cohen's kappa.

Lexical scores exist only to sit beside published lexical tables.
`longmemeval.py --lexical-scores` adds token F1, BLEU-1 and ROUGE-L on a **simplified**
answer (`--simplify-model`, default the judge; else a labelled heuristic). Without
simplification they grade verbosity: a wrong date can outscore a correct sentence
([reference values](/benchmarks/results#lexical-scoring-reference-values)). They never replace
the judge.

## Metrics

| Metric                     | Definition                                                                                                                                                  | Computed in                                      |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| recall@K                   | Share of questions with a gold hit in the top K                                                                                                             | `quality.py` `summarise`, `retrieval_recall.py`  |
| MRR                        | Mean of 1 / rank of the first gold hit (0 when none)                                                                                                        | same                                             |
| rank@1                     | Share of questions whose gold ranked first                                                                                                                  | same                                             |
| precision@K                | Gold hits among returned hits, averaged                                                                                                                     | `quality.py`                                     |
| `best_cos_rank`            | Rank of the highest-cosine hit; a gap from rank 1 means fusion put a weaker match first                                                                     | `retrieval_recall.py`                            |
| exact@1, caller-set F1     | Structural code exactness                                                                                                                                   | `code_intel_qa.py`                               |
| abstain precision / recall | Recall = probes abstained on / probes; precision = probes abstained on / all abstains                                                                       | `code_intel_qa.py`                               |
| `placebo@k`                | Per question, as many paths drawn (seeded) from the pool the system returned across all conceptual questions. Recall below it is pool coverage, not ranking | `code_intel_qa.py`                               |
| tokens per answer          | Mean / p90 `tokens_returned` per answered question                                                                                                          | `code_intel_qa.py`, acceptance benches           |
| judged accuracy            | Majority label CORRECT over repeats                                                                                                                         | `_gate.py`                                       |
| flip rate, discordance     | Within-arm share of repeated items whose labels disagreed; between-arm share of pairs whose majority labels differ                                          | `_gate.py` `flip_rate`, `GateResult.discordance` |
| failure stage              | `extraction`, `graph_resolution`, `retrieval_ranking`, `compression_loss`, `reader`, `not_verbatim`                                                         | `failure_taxonomy.py`                            |

The failure stage is computed per missed item, in pipeline order: not recalled is
`retrieval_ranking`; gold in a retrieved excerpt but answered wrong is `reader`; gold only
in the raw evidence behind a retrieved memory is `compression_loss`; gold nowhere
verbatim is `not_verbatim`. `lme_diagnose.py` dumps LongMemEval misses with these labels.
A `reader` regression under unchanged retrieval is the reader's.

Request failures are excluded from ratios, never counted as misses (`code_intel_qa.py`
prints `errors(excluded)`). `quality.py` aborts on any HTTP failure.

## Baselines

Loom compares against techniques and its own arms, under the same instrument on the same
ingest.

| Baseline                                                          | Where                                                                              |
| ----------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| Lexical-only, lexical plus query expansion, vector-only retrieval | `make bench-code-legs`: each a paired B arm against the hybrid default             |
| Summary-only memory                                               | `msc_dmr.py` `digest` arm beside `ask`, `search`, `search_recall`, `search_reader` |
| Store verbatim vs extract facts                                   | `memoryarena/` and `memorydata/` `verbatim` / `extract` modes                      |
| Long context, BM25, plain dense retrieval                         | `memoryarena/memory_server.py` arms                                                |
| Coding agent with the plugin off                                  | `code_plane_ablation.py` arm A                                                     |
| Bulk file reads                                                   | `code_shunt_tokens.py` "cold" arm                                                  |
| Brute-force nearest neighbours                                    | `vector_tenant_recall.py`                                                          |
| One strong or one cheap synthesis model                           | `route_replay.py` strong, cheap and cascade arms on frozen retrievals              |
| The source system's native ranking                                | `clickgap_migration_validate.py`: the export's `recall_count` order                |
| Per-plane ablation                                                | `plane_ablation.py` `execution_planes` masks                                       |

## Statistics

### Paired, clustered bootstrap

A ranking change ships on the **lower bound** of the paired delta, not the mean.
`paired_bootstrap` takes per-item deltas (B minus A), resamples whole clusters with
replacement (10,000 resamples, `seed=7`), and reads the 2.5th and 97.5th percentiles. A
gain counts only when the lower bound is above zero. Items about one mechanism move
together, so harnesses pass a cluster key: a conversation, a question type, or for code
the first two path segments (`cluster_of_path`). Too few clusters make the interval too
narrow; read `clusters` beside the bound. `code_plane_ablation.py` adds a paired Wilcoxon
signed-rank test over per-task means across seeds.

### Repeats and the majority label

The reader re-rolls even at temperature 0, and retrieval inside synthesis re-decomposes
the query. `--repeats N` asks and judges each item N times on the same retrieval and keeps
the **majority** label. A tie is WRONG; an item is ERROR only when every run errored.
Gate runs use three repeats (`GATE_REPEATS`). `identical_share` over the repeats'
`top_keys` shows whether a flip was the reader's or the retrieval's. Repeats cut label
noise ([simulation](/benchmarks/results#majority-voting-on-simulated-reader-noise)); only more
items cut sampling noise.

### The verdict

`paired_gate` runs the bootstrap over the arms' majority labels:

| Verdict      | Condition                                                                                                                                                        |
| ------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pass`       | the 95% lower bound clears `--gate-min-delta` (0 by default; a negative value is a non-inferiority gate)                                                         |
| `fail`       | the 95% upper bound is below the threshold                                                                                                                       |
| `unresolved` | the interval straddles it. Inside the null band (`--gate-band`, default `DEFAULT_BAND_PP` = 4.0 pp) that is the noise floor; outside it the run is under-powered |

Items unstable within either arm's repeats are also reported as a stable-only pair.

**Measure your own band.** The default came from one harness and one reader. Run the same
configuration as arms A and B with `--repeats 3 --gate`, and pass the result's wider side
(`band_from_null`) as `--gate-band` on the real comparison.

### How wide the interval is

Width depends on how many paired items the arms disagree on, not only on n. These rows are
`paired_bootstrap` on fixed deltas (+1 = B right and A wrong, −1 the reverse, 0 = agree):

| n                    | Wins / ties / losses             | Mean delta | 95% interval      | Reading                                        |
| -------------------- | -------------------------------- | ---------- | ----------------- | ---------------------------------------------- |
| 60                   | 1 / 59 / 0                       | +1.7 pp    | +0.0 .. +5.0 pp   | One question; the bound does not clear zero    |
| 30                   | 2 / 26 / 2                       | 0          | −13.3 .. +13.3 pp | A same-config pair: a 13-point swing is inside |
| 60                   | 5 / 50 / 5                       | 0          | −10.0 .. +10.0 pp |                                                |
| 164                  | 13 / 138 / 13                    | 0          | −6.1 .. +6.1 pp   |                                                |
| 500                  | 40 / 420 / 40                    | 0          | −3.6 .. +3.6 pp   |                                                |
| 60                   | 6 / 52 / 2                       | +6.7 pp    | −1.7 .. +16.7 pp  | A "7-point gain" that is unresolved            |
| 60                   | 8 / 50 / 2                       | +10.0 pp   | +0.0 .. +20.0 pp  | The bound only reaches zero: not a pass        |
| 500                  | 40 / 440 / 20                    | +4.0 pp    | +1.0 .. +7.0 pp   | A smaller gain that passes because n is larger |
| 60, 6 clusters of 10 | 8 / 50 / 2, wins in two clusters | +10.0 pp   | +0.0 .. +30.0 pp  | Correlated items widen the interval            |

```bash
.venv/bin/python - <<'EOF'
from bench.bootstrap import paired_bootstrap
def row(w, t, l, clusters=None):
    d = [1.0] * w + [-1.0] * l + [0.0] * t
    print(paired_bootstrap(d, clusters).line(unit="pp", scale=100))
row(1, 59, 0); row(2, 26, 2); row(5, 50, 5); row(13, 138, 13); row(40, 420, 40)
row(6, 52, 2); row(8, 50, 2); row(40, 440, 20)
row(8, 50, 2, [f"c{i // 10}" for i in range(60)])
EOF
```

### Set sizes

Small sets cannot rank close arms.

| Set                                    | Size                         | One item is                       |
| -------------------------------------- | ---------------------------- | --------------------------------- |
| `synthetic_memory_qa.json`             | 15 questions                 | 6.7 points: catches collapse only |
| `code_intel_qa.jsonl` conceptual slice | 36 of 60 questions           | 2.8 points                        |
| Algorithm-gate fixtures                | 19 extraction, 2 graph cases | a smoke test                      |
| `code.ask` cache pairs                 | 42 paraphrase, 43 different  | more than 2 points                |

### Judge calibration

`bench/judge_calibration.py` measures the judge's true- and false-positive rates on the
probes. Observed accuracy is `TPR·p + FPR·(1−p)`, so a paired delta shrinks by Youden's
`J = TPR − FPR`. With `--judge-calibration <json>` the gate reports `Δ/J` and widens the
interval by the calibration set's own variance (delta method). A judge with `J ≤ 0`
carries no information; inputs pass through unchanged.

## When two numbers compare

All of these must match. If one differs, label the comparison non-comparable.

| Must match                | Why                                                                                         |
| ------------------------- | ------------------------------------------------------------------------------------------- |
| Axis                      | Recall@K and judged accuracy are different metrics                                          |
| Reader and judge          | A stronger reader moves reasoning-heavy categories; a different judge moves labels          |
| Dataset split and slice   | LongMemEval oracle vs S, LoCoMo categories 1–4 vs 1–5, a lane vs the whole file             |
| Search mode               | The unpinned default is `LOOM_MEMORY_SEARCH_DEFAULT_MODE`; another mode is an ablation      |
| Embedder and server build | An embedder swap moves the whole similarity scale; `quality.py`'s `.meta.json` records both |
| Index revision (code)     | `code_intel_qa.py` names the `as_of.commit` that served the answers and the stale count     |
| Crediting                 | A `--credit` run reinforces what it reads                                                   |
| Accuracy denominator      | Leaving abstentions out flatters an arm that abstains more                                  |

**Abstentions.** The judged gate counts an abstention as wrong. `route_replay.py` reports
both: `accuracy` = correct / (correct + wrong), and `accuracy_abstain_wrong` = correct /
(correct + wrong + abstained), the gate's denominator. Errored asks are out of both. Quote
the one the verdict reads, and name it.

**Upper bounds and placebos.** Mined code lanes run against an index that contains every
fix (`fix_in_index` on each ablation task): quote paired deltas, not absolutes. Recall
that does not beat `placebo@5` / `placebo@10` is pool coverage. A tiny set makes the
placebo trivially strong. A set most arms score near the ceiling on cannot separate them;
read per-category results and the stable-only pair.

## Fake providers make scores meaningless

A checkout's `.env` runs `LOOM_EMBEDDING_MODEL=fake-ngram-3072`, CI runs `fake-ngram-1024`,
and both run `LOOM_LLM_PROVIDER=fake` (the released stack embeds with `local` and keeps the
fake LLM). The fake embedder (`server/embedding/fake.py`) hashes character
trigrams into buckets, so similarity tracks spelling, not meaning. The fake LLM
(`server/llm/fake.py`) returns shaped JSON keyed off an anchor phrase, so pipelines run
without extracting anything real. Under them:

- recall measures string overlap between question and stored text;
- extraction benches measure the fake's output, not the prompt;
- the similarity gates use their scalar defaults, because adaptive calibration skips fake
  embedders.

A green gate on fake embeddings is worse than no gate: it reports safety that was never
measured. So `make bench-quality-gate` and `make bench-code-intel` are not in CI
(enforcing them needs a paid provider secret), and CI runs only the pure scoring code
(`pytest bench`) and the gate functions' unit tests (`quality.gate`: the boundary passes,
0/0 fails). Set `LOOM_REQUIRE_REAL_PROVIDERS=true` on a bench server so a fake or
key-less provider makes `/v1/health` report `degraded` before you spend a run.

## Reproducibility controls

| Control                    | Mechanism                                                                                                                                                                          |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Deterministic resampling   | `paired_bootstrap` and `paired_gate` are deterministic under `seed`; the gate records `resamples` and `seed`                                                                       |
| Deterministic samples      | Sample flags take a seed (`musique.py --seed`, `judge_calibration.py`, mined-lane samples)                                                                                         |
| Recorded server instrument | `quality.py` writes a `.meta.json` with `embedding_model`, `embedding_dim`, `llm_model`, `llm_provider`, `build_commit` from `/v1/health`; `code_intel_qa.py` names `as_of.commit` |
| Pinned scope               | Search harnesses send `namespace_scope: "this"`, so rows from other readable namespaces never enter the window. Ad-hoc probes must too                                             |
| Evaluation reads           | `measurement_fields()` sends `auto_touch: false` / `reinforce: false`; `--credit` opts in; on a namespace read before, pass `reinforcement_weight: 0` or use fresh rows            |
| Fresh namespaces           | One throwaway namespace per item or conversation, erased at the end                                                                                                                |
| Warm-up                    | `search_latency.py --warm 1` discards the first call per query, so embedding-cache misses do not count as retrieval                                                                |
| Token lifetime             | Mint a token that outlives the run ([running](/benchmarks/running#2-mint-a-token-that-outlives-the-run))                                                                                  |

### Bench namespaces

Harnesses that seed data create throwaway namespaces (`lme-…`, `locomo-…`, `mab-…`,
`qabench-…`; new harnesses use `bench-`). `bench/bench_namespaces.py`:

- registers each before its first write with `traffic_class = "bench"`, `kind = "bench"`
  and `capture_generation_io = True` (default);
- runs the work inside `erase_on_exit`, which erases every minted name with one purging
  erasure when the run ends, fails or is interrupted. The body sets `external: false`, so
  no external backend is touched;
- with `--keep-namespaces`, keeps them for a later re-score (`--namespaces-json`) and
  registers them `kind = "memory"` so the idle collector leaves them.

Erasure needs an org-admin token. On a refusal the harness logs the `loom namespaces-gc`
command. The `namespace_gc` worker purges idle `kind = bench` namespaces after
`LOOM_NAMESPACE_GC_IDLE_HOURS`, and minted names start with a prefix in
`LOOM_NAMESPACE_BENCH_PREFIXES`, so aborted runs are found too.

### HTTP policy

Harnesses post through `bench/_http.py` (`post` / `apost` return the response,
`post_json` / `apost_json` the body):

| Response                                      | Policy                                                                                                         |
| --------------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| 429                                           | waits out `Retry-After` (1 s when absent, clamped to 0.2–60 s) and retries; a wait does not use the 5xx budget |
| 5xx                                           | retried with a `0.5 × 2^n` s backoff until `attempts` requests have been sent                                  |
| any other status ≥ 400, or the last 429 / 5xx | raises `httpx.HTTPStatusError` with the start of the body                                                      |
| transport error                               | propagates; a harness that rides out a restart keeps its own loop                                              |

## Sources of bias

| Source                   | Effect                                                                             | Mitigation in the code                                               |
| ------------------------ | ---------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| Author-written sets      | The people tuning the system wrote the synthetic set and chose its threshold on it | Keep it a smoke gate; decide rankings on public or use-labelled sets |
| Lenient substring gold   | A verbose excerpt can contain a gold word by accident                              | MRR, rank@1 and precision@K beside recall                            |
| Upper-bound code lanes   | The index already contains every fix                                               | Header warnings, `fix_in_index`; quote paired deltas only            |
| Model-authored gold      | The third-party code-search labels were written and checked by models              | `secondary` paths never credited; paired reading only                |
| Labels by use            | A relied-on set contains only what the current ranking surfaced                    | Paired sweep on the bootstrap bound; one source among several        |
| Judge leniency           | An LLM judge passes topical-but-wrong answers at some rate                         | `judge_calibration.py` TPR/FPR, `--second-judge-model`               |
| Judge and reader drift   | A provider revision behind a pinned name changes labels                            | Pinned snapshots where the dataset names one; run both arms together |
| Self-reinforcement       | Crediting reads reinforce what the next pass ranks                                 | Evaluation reads by default                                          |
| Excluded errors          | If failures cluster on hard items, the ratio rises                                 | Error counts printed; `quality.py` aborts on any failure             |
| Leakage in agent benches | An agent finds the fix through the network or git history                          | The ablation's exclusion ledger drops such cells, fail-closed        |
| Saturation               | Categories most systems ace cannot rank arms                                       | Per-category results; the stable-only pair                           |
| Harness parity           | External testbeds wrap every query in their own template for every method          | The adapters keep that parity                                        |

## Source entry points

- `bench/_gate.py`: `paired_gate`, `verdict_for`, `majority_label`, `flip_rate`,
  `JudgeCalibration`, `measurement_fields`, `add_gate_arguments`.
- `bench/bootstrap.py`: `paired_bootstrap`, `cluster_of_path`.
- `bench/judge_calibration.py`: `build_probes`.
- `bench/bench_namespaces.py`: `register_body`, `erase_on_exit`, `erase_on_exit_sync`.
- `bench/failure_taxonomy.py`: `classify_answer_failure`.
- `bench/answer_scoring.py`: `token_f1`, `bleu1`, `rouge_l`, `cohen_kappa`.
- Tests: `tests/unit/test_bench_gate.py`, `test_bench_bootstrap.py`,
  `test_bench_judge_calibration.py`, `test_bench_namespaces.py`,
  `test_bench_failure_taxonomy.py`, `test_answer_scoring.py`,
  `test_bench_gate_help_renders.py`.

## Related

- [Running](/benchmarks/running) and [results](/benchmarks/results).
- [Retrieval and ranking](/concepts/retrieval-and-ranking): what moves a ranking.
