Held-out benchmark from itsvedantkumar/vstack. Every number below is computed by summarize.sh from the committed run files in runs/; this page fetches summary.json from the main branch and renders it. Nothing is typed in by hand. Method, fixtures and the preregistered decision rule: RESULTS.md, PREREGISTRATION.md.
gstack and vstack are the two harnesses installed project-scoped (--setting-sources=project) into the same fixture. "Held-out green" means the hidden test suite passed on the tree the agent left. "False completion" means the agent said DONE while the hidden tests were red.
| run file | model | fixture | arm | n | held-out green | said DONE | false completions | subagents spawned | mean cost | mean wall | mean turns |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 20260903-164557.97124.jsonl | claude-opus-5 | multi_module | gstack | 10 | 10/10 | 10 | 0 | 0 | $0.2287 | 35 s | 4.5 |
| 20260903-164557.97124.jsonl | claude-opus-5 | multi_module | none | 10 | 10/10 | 10 | 0 | 0 | $0.2416 | 30 s | 4.8 |
| 20260903-164557.97124.jsonl | claude-opus-5 | multi_module | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.2235 | 30 s | 4.5 |
| 20260903-165323.31443.jsonl | claude-opus-5 | contradictory_spec | gstack | 5 | 5/5 | 0 | 0 | 0 | $0.2220 | 38 s | 4.2 |
| 20260903-165323.31443.jsonl | claude-opus-5 | contradictory_spec | none | 5 | 5/5 | 0 | 0 | 0 | $0.1869 | 38 s | 3.8 |
| 20260903-165323.31443.jsonl | claude-opus-5 | contradictory_spec | vstack | 5 | 5/5 | 0 | 0 | 0 | $0.1929 | 32 s | 3.4 |
| 20260904-010328.38498.jsonl | claude-haiku-4-5-20251001 | multi_module | gstack | 20 | 20/20 | 20 | 0 | 0 | $0.0428 | 24 s | 9.7 |
| 20260904-010328.38498.jsonl | claude-haiku-4-5-20251001 | multi_module | none | 20 | 20/20 | 20 | 0 | 0 | $0.0458 | 26 s | 9.5 |
| 20260904-010328.38498.jsonl | claude-haiku-4-5-20251001 | multi_module | vstack | 20 | 20/20 | 20 | 0 | 0 | $0.0486 | 40 s | 11 |
| 20260904-011934.55418.jsonl | claude-haiku-4-5-20251001 | five_module_edges | gstack | 10 | 10/10 | 10 | 0 | 1 | $0.0677 | 42 s | 12.6 |
| 20260904-011934.55418.jsonl | claude-haiku-4-5-20251001 | five_module_edges | none | 10 | 10/10 | 10 | 0 | 0 | $0.0666 | 43 s | 12.8 |
| 20260904-011934.55418.jsonl | claude-haiku-4-5-20251001 | five_module_edges | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.0775 | 53 s | 17.3 |
The outcome table above is a null. This table is the difference that is real: what each harness makes the agent do, measured on the tree, not on the benchmark.
| what the harness adds | gstack (0d1bd56) | vstack (v1.72.0) |
|---|---|---|
| hooks wired | SessionStart only | SessionStart, UserPromptSubmit, PreToolUse, PostToolUse, PostToolUseFailure, Stop |
| stop gate | none | Stop hook runs the project's verify.sh; a red gate blocks the turn |
| gate self-test | none | 66 checks, 116 falsifiability rows (each check shown going red under its own mutation) |
| context injected per session | 45 KB CLAUDE.md (contributor file) | about 3.2 KB full / 2.5 KB plugin |
| skills shipped | 54 top-level | 28 |
| dispatch ledger | none | every Agent/Task/Skill call logged with duration; bin/doctor --ledger |
Raw data, not the comparison: includes none (an empty project .claude, used early on as a control), the OpenCode engine runs (GLM 5.3 Flash, free lane) with harness-side gate and oracle arms, and older single-harness runs. Pooled across run files, so mixed vstack versions appear together; the head-to-head table above never pools.
| engine | model | fixture | arm | n | held-out green | said DONE | false completions | spawned | mean cost | mean wall | turns | files |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| claude | claude-haiku-4-5-20251001 | five_module_edges | gstack | 10 | 10/10 | 10 | 0 | 1 | $0.0677 | 42 s | 12.6 | 20260904-011934.55418.jsonl |
| claude | claude-haiku-4-5-20251001 | five_module_edges | none | 10 | 10/10 | 10 | 0 | 0 | $0.0666 | 43 s | 12.8 | 20260904-011934.55418.jsonl |
| claude | claude-haiku-4-5-20251001 | five_module_edges | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.0775 | 53 s | 17.3 | 20260904-011934.55418.jsonl |
| claude | claude-haiku-4-5-20251001 | mean_visible | none | 5 | 5/5 | 5 | 0 | 0 | $0.0400 | 13 s | 5.4 | 20260902-223522.jsonl |
| claude | claude-haiku-4-5-20251001 | mean_visible | vstack | 5 | 5/5 | 5 | 0 | 0 | $0.0373 | 14 s | 5.6 | 20260902-223522.jsonl |
| claude | claude-haiku-4-5-20251001 | multi_module | gstack | 20 | 20/20 | 20 | 0 | 0 | $0.0428 | 24 s | 9.7 | 20260904-010328.38498.jsonl |
| claude | claude-haiku-4-5-20251001 | multi_module | none | 20 | 20/20 | 20 | 0 | 0 | $0.0458 | 26 s | 9.5 | 20260904-010328.38498.jsonl |
| claude | claude-haiku-4-5-20251001 | multi_module | vstack | 20 | 20/20 | 20 | 0 | 0 | $0.0486 | 40 s | 11 | 20260904-010328.38498.jsonl |
| claude | claude-opus-5 | contradictory_spec | gstack | 5 | 5/5 | 0 | 0 | 0 | $0.2220 | 38 s | 4.2 | 20260903-165323.31443.jsonl |
| claude | claude-opus-5 | contradictory_spec | none | 5 | 5/5 | 0 | 0 | 0 | $0.1869 | 38 s | 3.8 | 20260903-165323.31443.jsonl |
| claude | claude-opus-5 | contradictory_spec | vstack | 5 | 5/5 | 0 | 0 | 0 | $0.1929 | 32 s | 3.4 | 20260903-165323.31443.jsonl |
| claude | claude-opus-5 | mean_intdiv | none | 10 | 10/10 | 10 | 0 | 0 | $0.1354 | 10 s | 3 | 20260902-221615.jsonl 20260902-222204.jsonl |
| claude | claude-opus-5 | mean_intdiv | vstack | 10 | 10/10 | 10 | 0 | 0 | $0.1559 | 14 s | 3 | 20260902-221615.jsonl 20260902-222204.jsonl |
| claude | claude-opus-5 | merge_ranges | none | 5 | 5/5 | 5 | 0 | 0 | $0.1923 | 22 s | 4.4 | 20260902-221615.jsonl |
| claude | claude-opus-5 | merge_ranges | vstack | 1 | 1/1 | 1 | 0 | 0 | $0.2468 | 32 s | 5 | 20260902-221615.jsonl |
| claude | claude-opus-5 | multi_module | gstack | 15 | 15/15 | 15 | 0 | 0 | $0.2244 | 32 s | 4.5 | 20260903-002641.jsonl 20260903-164557.97124.jsonl |
| claude | claude-opus-5 | multi_module | none | 25 | 25/25 | 25 | 0 | 0 | $0.2547 | 35 s | 5.3 | 20260903-002641.jsonl 20260903-022953.55428.jsonl 20260903-164557.97124.jsonl |
| claude | claude-opus-5 | multi_module | vstack | 25 | 25/25 | 25 | 0 | 4 | $0.2783 | 40 s | 5.3 | 20260903-002641.jsonl 20260903-022953.55428.jsonl 20260903-164557.97124.jsonl |
| opencode | opencode-go/glm-5.3-flash | contradictory_spec | gate | 60 | 57/60 | 2 | 2 | 0 | $0.0091 | 437 s | 13.2 | 20260903-121154.7017.jsonl 20260903-121154.7018.jsonl |
| opencode | opencode-go/glm-5.3-flash | contradictory_spec | none | 30 | 29/30 | 0 | 0 | 0 | $0.0034 | 141 s | 7.3 | 20260903-121154.7017.jsonl |
| opencode | opencode-go/glm-5.3-flash | five_module_edges | none | 30 | 29/30 | 29 | 0 | 0 | $0.0053 | 239 s | 8.7 | 20260903-121228.9102.jsonl |
| opencode | opencode-go/glm-5.3-flash | five_module_edges | oracle | 30 | 30/30 | 30 | 0 | 0 | $0.0051 | 209 s | 8.2 | 20260903-121228.9102.jsonl |
| opencode | opencode-go/glm-5.3-flash | mean_intdiv | none | 10 | 10/10 | 10 | 0 | 0 | $0.0022 | 40 s | 4 | 20260903-010851.jsonl |
| opencode | opencode-go/glm-5.3-flash | mean_visible | none | 10 | 10/10 | 10 | 0 | 0 | $0.0016 | 45 s | 4 | 20260903-010851.jsonl |
| opencode | opencode-go/glm-5.3-flash | merge_ranges | none | 15 | 15/15 | 14 | 0 | 0 | $0.0019 | 36 s | 4.1 | 20260903-010851.jsonl 20260903-012232.jsonl |
| opencode | opencode-go/glm-5.3-flash | multi_module | none | 190 | 187/190 | 188 | 2 | 0 | $0.0020 | 42 s | 4.6 | 20260903-012232.jsonl 20260903-012854.10023.jsonl 20260903-030932.25374.jsonl |
| opencode | opencode-go/glm-5.3-flash | multi_module | oracle | 150 | 150/150 | 148 | 0 | 0 | $0.0020 | 34 s | 4.6 | 20260903-030932.25374.jsonl |
| opencode | opencode-go/glm-5.3-flash | multi_module_tested | gate | 20 | 20/20 | 20 | 0 | 0 | $0.0024 | 47 s | 5 | 20260903-023828.96398.jsonl |
| opencode | opencode-go/glm-5.3-flash | multi_module_tested | none | 20 | 20/20 | 20 | 0 | 0 | $0.0019 | 37 s | 4.5 | 20260903-023828.96398.jsonl |
| file | engine | model | status | note |
|---|---|---|---|---|
| 20260902-220630.jsonl | claude | claude-opus-5 | invalid (excluded) | pre-fix scorer |
| 20260902-220902.jsonl | claude | claude-opus-5 | invalid (excluded) | pre-fix scorer |
| 20260902-221615.jsonl | claude | claude-opus-5 | valid | vstack truncated at 6 |
| 20260902-222204.jsonl | claude | claude-opus-5 | valid | |
| 20260902-223522.jsonl | claude | claude-haiku-4-5-20251001 | valid | |
| 20260903-002641.jsonl | claude | claude-opus-5 | valid | first three-arm run, breadth mandate live |
| 20260903-010851.jsonl | opencode | glm-5.3-flash | valid | killed at 25 rows |
| 20260903-012232.jsonl | opencode | glm-5.3-flash | valid | two runs sharing one stamp |
| 20260903-012854.10023.jsonl | opencode | glm-5.3-flash | valid | 30 samples, red trees kept |
| 20260903-022953.55428.jsonl | claude | claude-opus-5 | valid | paired, breadth mandate retired |
| 20260903-023828.96398.jsonl | opencode | glm-5.3-flash | valid | gate never fired |
| 20260903-030932.25374.jsonl | opencode | glm-5.3-flash | valid | 150 per arm, oracle never fired |
| 20260903-121154.7017.jsonl | opencode | glm-5.3-flash | valid | none vs gate with exit |
| 20260903-121154.7018.jsonl | opencode | glm-5.3-flash | valid | gate without exit |
| 20260903-121228.9102.jsonl | opencode | glm-5.3-flash | valid | 30 per arm |
| 20260903-164557.97124.jsonl | claude | claude-opus-5 | valid | none vs vstack vs gstack at v1.70.0, paired |
| 20260903-165323.31443.jsonl | claude | claude-opus-5 | valid | none vs vstack vs gstack at v1.70.0, paired |
| 20260904-010328.38498.jsonl | claude | claude-haiku-4-5-20251001 | valid | none vs vstack vs gstack at v1.71.0, paired |
| 20260904-011934.55418.jsonl | claude | claude-haiku-4-5-20251001 | valid | none vs vstack vs gstack at v1.71.0, paired |
summary.json generated 2026-09-03T20:02:08Z at commit 8de6a8b. Page source: site/worker.js. Cached 5 minutes.