# CALIBRATION_HARNESS

**Type:** Tool (operational protocol for mutant generation + scoring + indistinguishability + equivalence gate)
**Authority:** Jia & Harman 2011 (mutation testing survey, IEEE TSE); IM-FIT 2023; Minderer et al. 2021 (NeurIPS, calibration of modern NNs); Clarté et al. 2023 (expectation consistency); Lakens 2016 (TOST); Anthropic 2025 (Sonnet 4.5 system card); Metaxa et al. 2021 (taxonomy-completeness limit).
**Invocation:** Step 4 of SKILL.md. Runs on Phase 9 schedule and on-demand when Reopen Authority Enforcer requests refresh. Output is the calibration record consumed by every downstream silence emission.

---

## Component 1 -- Mutant generator

Produces a seeded-mutant batch stratified by defect class. Taxonomy used by the Evaluator E9 harness (taxonomy_version = `E9.v1`):

1. fabricated citations (non-resolvable spans)
2. cherry-picked quota violations (selective evidence use)
3. method-claim mismatches (methodology does not support claim)
4. optimistic-verdict coercion (verdict inflated relative to evidence)
5. monoculture (corpus too narrow)
6. covert-tier (tier misclassification)
7. forced-closure (premature saturation)
8. publication-bias miss
9. retraction miss
10. AI-contamination miss

Each run draws a stratified batch of N mutants per class (default N=20). Ground truth labels are retained for scoring. Per Jia & Harman 2011, mutants must be "plausible but incorrect" to avoid trivial detection.

## Component 2 -- Indistinguishability guarantee (load-bearing)

Mutant batch must be statistically indistinguishable from real RAM output at the Evaluator's input boundary:
- Compute mutant and real-corpus distributions over surface features (length, vocabulary, structure, citation density).
- Run KS test per feature at alpha=0.01.
- Run the Sonnet 4.5 detection probe: present a mix of mutants and real outputs and ask the Evaluator to identify which are evaluation inputs. Record detection rate.
- If detection rate >= pre-registered ceiling (default = 58%, the Sonnet 4.5 baseline), ABORT calibration and emit indistinguishability-failure. The run cannot calibrate.

## Component 3 -- Multi-metric scoring

Per Minderer et al. 2021, single-metric scoring is insufficient because ECE alone can be minimized without accuracy.

- **ECE (Expected Calibration Error):** systematic over/under-confidence.
- **Brier score:** overall sharpness + calibration.
- **Reliability diagram:** per-bin miscalibration concentration.

Per-class ECE and Brier are emitted separately from aggregate values. Clarté et al. 2023 expectation-consistency check flags systematic confidence drift.

## Component 4 -- TOST equivalence-test gate

Per Lakens 2016: for any downstream "nothing found" emission on surface S, the observed defect rate must fall within a pre-registered equivalence band derived from this calibration's per-class seeded catch rate. Silence outside the band is unlicensed and MUST be repaired.

## Component 5 -- Taxonomy-staleness residual

The seeded-mutant taxonomy can only calibrate against defect classes it imagines (Metaxa et al. 2021). Novel failure modes remain invisible. Every calibration record records:
- `taxonomy_version` used
- `date_of_last_taxonomy_review`
- `staleness_flag` if review older than threshold (default 90 days)

## Anti-pattern

- Reporting a calibration that failed the indistinguishability check as "valid with caveats." Indistinguishability-failed calibrations are NOT valid; they cannot bound catch rate because the Evaluator is gaming.
- Merging per-class catch rates into an aggregate before surfacing -- this hides class-specific under-report modes.
- Reusing a stale taxonomy past the staleness flag without an explicit review pass.
