Grading mechanistic explanations by simulation — replacement networks, transcripts and trained models

Artefacts of the study Grading mechanistic explanations by weights-blind simulation by Libor "Burny" Burian and Victor Levoso, developed in a mechanistic interpretability hackathon on automatically evaluating mechanistic interpretability explanations co-organized in the mechinterp.com community.

An LLM agent (Claude Haiku 5.5 in Claude Code; local Qwen models were tried and stopped at chance), offline except for the model endpoint and never shown the target's weights, reads an explanation of a one-layer modular-addition transformer (Nanda et al. 2023) and writes the weights of a replacement network. The replacement is graded against the target by output KL and by cross-validated hidden-state similarity. Code, preregistration, paper and all graded results: GitHub. Target models: BurnyCoder/grokking-modular-addition-runs.

Contents

Folder What
replacements/<run_id>/ one folder per graded agent run (116 so far, of which 15 are stage-0 development runs on the dev target, never reported as results): weights.npz (the replacement, 11 float64 arrays in the paper's layout), build_weights.py (the agent's script), transcript.md, events.ndjson.gz (full event stream), wire.log.gz (every request and response body between the sandbox and the model; no headers), result.json (grader output)
compiler/<target>/<explanation>.npz the agent-free reference compiler's build for every explanation of every target
trained/T-new/ the held-out target trained after the protocol freeze (paper recipe, seed 11): all 401 checkpoints, curves, metadata
trained/planted_T0/ the reference-compiled E3 model for T0 and its snapshots after 100, 1,000 and 10,000 further training steps
trained/students_T0/ three students distilled on T0's full output logits (20,000 steps each)

Run IDs read stage__target__explanation[-variant]__simulator__temperature__seed. Local user names and home paths in logs are replaced by user. Load a replacement with numpy.load("weights.npz"); the array names and shapes are in the repository's sandbox_template/ARCHITECTURE.md.

Runs

Stage Target Explanation Simulator T Variant N failed accuracy KL_out ↓ (mean ± sd) HS ↑ (mean ± sd)
ablations T0 E2 qwen3.5-4b 0.6 – 1 0 0.009 4.73 0.000
ablations T0 E2 qwen3.5-9b 0.6 – 1 0 0.009 7.63 0.000
ablations T0 E3 claude-haiku-5-5 default fmt-code 3 0 1.000 0.00175 ± 0.000396 0.897 ± 0.0301
ablations T0 E3 claude-haiku-5-5 default fmt-math 3 0 0.999 0.00322 ± 0.00379 0.915 ± 0.00311
ablations T0 E3 claude-haiku-5-5 default noselftest 3 0 1.000 0.00376 ± 0.00218 0.942 ± 0.0253
ablations T0 E3 qwen3.5-4b 0.6 – 1 0 0.009 4.73 0.000
ablations T0 E4 qwen3.5-4b 0.6 – 1 0 0.009 4.73 0.000
ablations T0 W-freq qwen3.5-4b 0.6 – 1 0 0.009 6.07 0.000
controls T0 E0 claude-haiku-5-5 default – 3 0 1.000 0.00455 ± 0.00431 0.130 ± 0.109
controls T0 W-freq claude-haiku-5-5 default – 3 0 1.000 0.00293 ± 0.00195 0.070 ± 0.0211
controls T0 W-alg claude-haiku-5-5 default – 3 0 1.000 0.024 ± 0.0171 0.909 ± 0.0268
controls T0 W-role claude-haiku-5-5 default – 3 0 1.000 0.00098 ± 0.00076 0.204 ± 0.06
controls T0 W-scramble claude-haiku-5-5 default – 3 0 0.998 0.0062 ± 0.00838 0.936 ± 0.0183
hyper T0 E3 claude-haiku-5-5 default effort-high 3 0 1.000 0.00102 ± 0.000717 0.936 ± 0.0344
hyper T0 E3 claude-haiku-5-5 default effort-low 3 0 0.853 0.661 ± 1.14 0.921 ± 0.0033
hyper T0 W-freq claude-haiku-5-5 default effort-high 3 0 1.000 0.00143 ± 0.000605 0.093 ± 0.00464
hyper T0 W-freq claude-haiku-5-5 default effort-low 3 0 0.923 0.344 ± 0.594 0.088 ± 0.0252
main T0 E1 claude-haiku-5-5 default – 3 0 1.000 0.0969 ± 0.152 0.046 ± 0.00177
main T0 E2 claude-haiku-5-5 default – 3 0 1.000 0.000212 ± 0.000178 0.894 ± 0.00542
main T0 E3 claude-haiku-5-5 default – 3 0 1.000 0.00972 ± 0.0115 0.929 ± 0.0124
main T0 E4 claude-haiku-5-5 default – 3 0 0.986 0.06 ± 0.0455 0.943 ± 0.0282
seed T-new E3 claude-haiku-5-5 default – 3 0 0.910 0.328 ± 0.237 0.910 ± 0.0613
seed T-new E4 claude-haiku-5-5 default – 3 0 0.997 0.0124 ± 0.0214 0.984 ± 0.000181
seed T-new W-freq claude-haiku-5-5 default – 3 0 0.907 0.432 ± 0.461 0.094 ± 0.0689
seed T-paper E1 claude-haiku-5-5 default – 3 0 0.998 0.621 ± 0.725 0.590 ± 0.34
seed T-paper E3 claude-haiku-5-5 default – 3 0 0.962 0.141 ± 0.207 0.940 ± 0.0251
seed T-paper E4 claude-haiku-5-5 default – 3 0 0.894 0.413 ± 0.18 0.937 ± 0.00801
seed T-paper W-freq claude-haiku-5-5 default – 3 0 0.998 0.00677 ± 0.0116 0.117 ± 0.0384
seed T-paper E-paper claude-haiku-5-5 default – 3 0 1.000 0.00115 ± 0.00196 0.950 ± 0.041
seed T1 E2 claude-haiku-5-5 default – 3 0 1.000 0.00127 ± 0.00161 0.957 ± 0.00305
seed T1 E3 claude-haiku-5-5 default – 3 0 0.908 0.384 ± 0.143 0.936 ± 0.0167
seed T1 E4 claude-haiku-5-5 default – 3 0 0.901 0.498 ± 0.245 0.960 ± 0.00558
seed T1 W-freq claude-haiku-5-5 default – 3 0 0.854 0.537 ± 0.136 0.051 ± 0.0247
seed T2 E2 claude-haiku-5-5 default – 3 0 1.000 3.41e-05 ± 1.17e-05 0.900 ± 0.0969
seed T2 E3 claude-haiku-5-5 default – 3 0 1.000 5.84e-05 ± 1.37e-05 0.962 ± 0.015
seed T2 E4 claude-haiku-5-5 default – 3 0 0.990 0.0487 ± 0.0548 0.951 ± 0.0146
seed T2 W-freq claude-haiku-5-5 default – 3 0 0.999 0.00563 ± 0.00588 0.098 ± 0.0113
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Collection including BurnyCoder/grading-explanations-by-simulation

Paper for BurnyCoder/grading-explanations-by-simulation