Held-out benchmark from itsvedantkumar/vstack. Every number below is computed by summarize.sh from the committed run files in runs/; this page fetches summary.json from the main branch and renders it. Nothing is typed in by hand. Method, fixtures and the preregistered decision rule: RESULTS.md, PREREGISTRATION.md.
gstack and vstack are the two harnesses installed project-scoped (--setting-sources=project) into the same fixture. "Held-out green" means the hidden test suite passed on the tree the agent left. "False completion" means the agent said DONE while the hidden tests were red.
| run file | model | fixture | arm | n | held-out green | said DONE | false completions | subagents spawned | mean cost | mean wall | mean turns |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 20260903-164557.97124.jsonl | claude-opus-5 | multi_module | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.2287 | 35 s | 4.5 |
| 20260903-164557.97124.jsonl | claude-opus-5 | multi_module | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.2235 | 30 s | 4.5 |
| 20260903-165323.31443.jsonl | claude-opus-5 | contradictory_spec | gstack | 5 | 5/5 | 0 | 0 | 0 | $0.2220 | 38 s | 4.2 |
| 20260903-165323.31443.jsonl | claude-opus-5 | contradictory_spec | vstack | 5 | 5/5 | 0 | 0 | 0 | $0.1929 | 32 s | 3.4 |
| 20260904-010328.38498.jsonl | claude-haiku-4-5-20251001 | multi_module | gstack | 20 | 20/20 | 20 | 0 | 0 | $0.0428 | 24 s | 9.7 |
| 20260904-010328.38498.jsonl | claude-haiku-4-5-20251001 | multi_module | vstack | 20 | 20/20 | 20 | 0 | 0 | $0.0486 | 40 s | 11 |
| 20260904-011934.55418.jsonl | claude-haiku-4-5-20251001 | five_module_edges | gstack | 10 | 10/10 | 10 | 0 | 1 | $0.0677 | 42 s | 12.6 |
| 20260904-011934.55418.jsonl | claude-haiku-4-5-20251001 | five_module_edges | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.0775 | 53 s | 17.3 |
| 20260904-121312.60002.jsonl | claude-haiku-4-5-20251001 | multi_module | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.0437 | 20 s | 9.7 |
| 20260904-121312.60002.jsonl | claude-haiku-4-5-20251001 | multi_module | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.0454 | 21 s | 10.4 |
| 20260904-135609.47712.jsonl | claude-haiku-4-5-20251001 | multi_module | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.0437 | 20 s | 9.9 |
| 20260904-135609.47712.jsonl | claude-haiku-4-5-20251001 | multi_module | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.0431 | 20 s | 9.3 |
| 20260904-135609.47712.jsonl | claude-haiku-4-5-20251001 | multi_module | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.0463 | 25 s | 10.3 |
| 20260904-140111.78290.jsonl | claude-haiku-4-5-20251001 | five_module_edges | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.0585 | 36 s | 11.7 |
| 20260904-140111.78290.jsonl | claude-haiku-4-5-20251001 | five_module_edges | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.0686 | 42 s | 13.4 |
| 20260904-140111.78290.jsonl | claude-haiku-4-5-20251001 | five_module_edges | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.0792 | 49 s | 16.7 |
| 20260904-140913.15451.jsonl | claude-opus-5 | multi_module | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.2327 | 30 s | 4.9 |
| 20260904-140913.15451.jsonl | claude-opus-5 | multi_module | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.3066 | 39 s | 5.5 |
| 20260904-140913.15451.jsonl | claude-opus-5 | multi_module | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.2360 | 25 s | 4.4 |
| 20260904-142225.23002.jsonl | claude-haiku-4-5-20251001 | gated_report | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.0796 | 45 s | 15.3 |
| 20260904-142225.23002.jsonl | claude-haiku-4-5-20251001 | gated_report | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.0861 | 45 s | 16.5 |
| 20260904-142225.23002.jsonl | claude-haiku-4-5-20251001 | gated_report | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.0802 | 57 s | 15.2 |
| 20260904-143157.55012.jsonl | claude-opus-5 | gated_report | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.2855 | 36 s | 5.5 |
| 20260904-143157.55012.jsonl | claude-opus-5 | gated_report | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.3485 | 44 s | 6.2 |
| 20260904-143157.55012.jsonl | claude-opus-5 | gated_report | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.2821 | 36 s | 5 |
| 20260905-025901.62077.jsonl | claude-haiku-4-5-20251001 | gated_report_quiet | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.2858 | 202 s | 40.8 |
| 20260905-025901.62077.jsonl | claude-haiku-4-5-20251001 | gated_report_quiet | pstack | 10 | 9/10 | 10 | 1 | 0 | $0.2325 | 177 s | 30.3 |
| 20260905-025901.62077.jsonl | claude-haiku-4-5-20251001 | gated_report_quiet | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.2724 | 213 s | 34.6 |
The outcome table above is a null. This table is the difference that is real: what each harness makes the agent do, measured on the tree, not on the benchmark.
| what the harness adds | gstack (0d1bd56) | vstack (v1.72.0) |
|---|---|---|
| hooks wired | SessionStart only | SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, PostToolUseFailure, Stop |
| stop gate | none | Stop hook runs the project's verify.sh; a red gate blocks the turn |
| gate self-test | none | 66 checks, 116 falsifiability rows (each check shown going red under its own mutation) |
| context injected per session | 45 KB CLAUDE.md (contributor file) | about 3.2 KB full / 2.5 KB plugin |
| skills shipped | 54 top-level | 28 |
| dispatch ledger | none | every Agent/Task/Skill call logged with duration; bin/doctor --ledger |
Raw data, not the comparison: includes none (an empty project .claude, used early on as a control), the OpenCode engine runs (GLM 5.3 Flash, free lane) with harness-side gate and oracle arms, and older single-harness runs. Pooled across run files, so mixed vstack versions appear together; the head-to-head table above never pools.
| engine | model | fixture | arm | n | held-out green | said DONE | false completions | spawned | mean cost | mean wall | turns | files |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| claude | claude-haiku-4-5-20251001 | five_module_edges | gstack | 20 | 20/20 | 20 | 0 | 1 | $0.0631 | 39 s | 12.2 | 20260904-011934.55418.jsonl 20260904-140111.78290.jsonl |
| claude | claude-haiku-4-5-20251001 | five_module_edges | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.0686 | 42 s | 13.4 | 20260904-140111.78290.jsonl |
| claude | claude-haiku-4-5-20251001 | five_module_edges | vstack | 20 | 20/20 | 20 | 0 | 0 | $0.0784 | 51 s | 17 | 20260904-011934.55418.jsonl 20260904-140111.78290.jsonl |
| claude | claude-haiku-4-5-20251001 | gated_report | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.0796 | 45 s | 15.3 | 20260904-142225.23002.jsonl |
| claude | claude-haiku-4-5-20251001 | gated_report | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.0861 | 45 s | 16.5 | 20260904-142225.23002.jsonl |
| claude | claude-haiku-4-5-20251001 | gated_report | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.0802 | 57 s | 15.2 | 20260904-142225.23002.jsonl |
| claude | claude-haiku-4-5-20251001 | gated_report_quiet | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.2858 | 202 s | 40.8 | 20260905-025901.62077.jsonl |
| claude | claude-haiku-4-5-20251001 | gated_report_quiet | pstack | 10 | 9/10 | 10 | 1 | 0 | $0.2325 | 177 s | 30.3 | 20260905-025901.62077.jsonl |
| claude | claude-haiku-4-5-20251001 | gated_report_quiet | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.2724 | 213 s | 34.6 | 20260905-025901.62077.jsonl |
| claude | claude-haiku-4-5-20251001 | mean_visible | vstack | 5 | 5/5 | 5 | 0 | 0 | $0.0373 | 14 s | 5.6 | 20260902-223522.jsonl |
| claude | claude-haiku-4-5-20251001 | multi_module | gstack | 40 | 40/40 | 40 | 0 | 0 | $0.0432 | 22 s | 9.7 | 20260904-010328.38498.jsonl 20260904-121312.60002.jsonl 20260904-135609.47712.jsonl |
| claude | claude-haiku-4-5-20251001 | multi_module | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.0431 | 20 s | 9.3 | 20260904-135609.47712.jsonl |
| claude | claude-haiku-4-5-20251001 | multi_module | vstack | 40 | 40/40 | 40 | 0 | 0 | $0.0472 | 32 s | 10.7 | 20260904-010328.38498.jsonl 20260904-121312.60002.jsonl 20260904-135609.47712.jsonl |
| claude | claude-opus-5 | contradictory_spec | gstack | 5 | 5/5 | 0 | 0 | 0 | $0.2220 | 38 s | 4.2 | 20260903-165323.31443.jsonl |
| claude | claude-opus-5 | contradictory_spec | vstack | 5 | 5/5 | 0 | 0 | 0 | $0.1929 | 32 s | 3.4 | 20260903-165323.31443.jsonl |
| claude | claude-opus-5 | gated_report | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.2855 | 36 s | 5.5 | 20260904-143157.55012.jsonl |
| claude | claude-opus-5 | gated_report | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.3485 | 44 s | 6.2 | 20260904-143157.55012.jsonl |
| claude | claude-opus-5 | gated_report | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.2821 | 36 s | 5 | 20260904-143157.55012.jsonl |
| claude | claude-opus-5 | mean_intdiv | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.1559 | 14 s | 3 | 20260902-221615.jsonl 20260902-222204.jsonl |
| claude | claude-opus-5 | merge_ranges | vstack | 1 | 1/1 | 1 | 0 | 0 | $0.2468 | 32 s | 5 | 20260902-221615.jsonl |
| claude | claude-opus-5 | multi_module | gstack | 25 | 25/25 | 25 | 0 | 0 | $0.2277 | 32 s | 4.6 | 20260903-002641.jsonl 20260903-164557.97124.jsonl 20260904-140913.15451.jsonl |
| claude | claude-opus-5 | multi_module | pstack | 10 | 10/10 | 10 | 0 | 0 | $0.3066 | 39 s | 5.5 | 20260904-140913.15451.jsonl |
| claude | claude-opus-5 | multi_module | vstack | 35 | 35/35 | 35 | 0 | 4 | $0.2662 | 36 s | 5 | 20260903-002641.jsonl 20260903-022953.55428.jsonl 20260903-164557.97124.jsonl 20260904-140913.15451.jsonl |
| file | engine | model | status | note |
|---|---|---|---|---|
| 20260902-220630.jsonl | claude | claude-opus-5 | invalid (excluded) | pre-fix scorer |
| 20260902-220902.jsonl | claude | claude-opus-5 | invalid (excluded) | pre-fix scorer |
| 20260902-221615.jsonl | claude | claude-opus-5 | valid | vstack truncated at 6 |
| 20260902-222204.jsonl | claude | claude-opus-5 | valid | |
| 20260902-223522.jsonl | claude | claude-haiku-4-5-20251001 | valid | |
| 20260903-002641.jsonl | claude | claude-opus-5 | valid | first three-arm run, breadth mandate live |
| 20260903-010851.jsonl | opencode | glm-5.3-flash | valid | killed at 25 rows |
| 20260903-012232.jsonl | opencode | glm-5.3-flash | valid | two runs sharing one stamp |
| 20260903-012854.10023.jsonl | opencode | glm-5.3-flash | valid | 30 samples, red trees kept |
| 20260903-022953.55428.jsonl | claude | claude-opus-5 | valid | paired, breadth mandate retired |
| 20260903-023828.96398.jsonl | opencode | glm-5.3-flash | valid | gate never fired |
| 20260903-030932.25374.jsonl | opencode | glm-5.3-flash | valid | 150 per arm, oracle never fired |
| 20260903-121154.7017.jsonl | opencode | glm-5.3-flash | valid | none vs gate with exit |
| 20260903-121154.7018.jsonl | opencode | glm-5.3-flash | valid | gate without exit |
| 20260903-121228.9102.jsonl | opencode | glm-5.3-flash | valid | 30 per arm |
| 20260903-164557.97124.jsonl | claude | claude-opus-5 | valid | none vs vstack vs gstack at v1.70.0, paired |
| 20260903-165323.31443.jsonl | claude | claude-opus-5 | valid | none vs vstack vs gstack at v1.70.0, paired |
| 20260904-010328.38498.jsonl | claude | claude-haiku-4-5-20251001 | valid | none vs vstack vs gstack at v1.71.0, paired |
| 20260904-011934.55418.jsonl | claude | claude-haiku-4-5-20251001 | valid | none vs vstack vs gstack at v1.71.0, paired |
| 20260904-121312.60002.jsonl | claude | claude-haiku-4-5-20251001 | valid | vstack vs gstack at v1.72.0, paired, register warns |
| 20260904-135609.47712.jsonl | claude | claude-haiku-4-5-20251001 | valid | vstack vs gstack vs pstack at v1.72.0, paired |
| 20260904-140111.78290.jsonl | claude | claude-haiku-4-5-20251001 | valid | vstack vs gstack vs pstack at v1.72.0, paired |
| 20260904-140913.15451.jsonl | claude | claude-opus-5 | valid | vstack vs gstack vs pstack at v1.72.0, paired |
| 20260904-142225.23002.jsonl | claude | claude-haiku-4-5-20251001 | valid | vstack vs gstack vs pstack at v1.72.0, paired, gated_report |
| 20260904-143157.55012.jsonl | claude | claude-opus-5 | valid | vstack vs gstack vs pstack at v1.72.0, paired, gated_report |
| 20260904-143928.90069.jsonl | claude | claude-haiku-4-5-20251001 | invalid (excluded) | gated_report_quiet; harness defect: gstack workdirs were not git repos and sat inside this checkout, so runs escaped into the staged fixture source; killed after 15 rows, superseded by the rerun |
| 20260904-144621.19824.jsonl | claude | claude-haiku-4-5-20251001 | invalid (excluded) | gated_report_quiet; killed: vstack Stop gate inert (skipped untrusted, the 1.73.0 symlink path defect); superseded by a rerun on 1.73.0 |
| 20260904-184059.28637.jsonl | claude | claude-haiku-4-5-20251001 | invalid (excluded) | gated_report_quiet on 1.73.0; two vstack runs hit the 360s wrapper timeout and wrote no row, so the surviving 8 are the faster ones -- rerun at a higher timeout supersedes it |
| 20260905-024931.28111.jsonl | claude | claude-haiku-4-5-20251001 | invalid (excluded) | gated_report_quiet; killed mid-batch: the trust lock functions were not exported to the parallel workers, so the vstack arm ran with an unarmed Stop gate |
| 20260905-025901.62077.jsonl | claude | claude-haiku-4-5-20251001 | valid | vstack vs gstack vs pstack at v1.73.1, paired, gated_report_quiet (gate armed, prompt does not name it) |
summary.json generated 2026-09-04T22:05:28Z at commit 6e89ff0. Page source: site/worker.js. Cached 5 minutes.