vbench · Method
How vbench works
Every AI setup gets the same brief and one attempt. I score what it builds, by hand.
One shot
One run per setup. Its first result is the result. No reruns, no repairs.
Nothing hidden
Whole rounds are released. Failures stay up with their recorded cause.
Scored by hand
Four criteria, 0 to 5 each, with a note for every score.
The setups
Each model runs through its maker's command-line tool, at maximum effort, from an empty folder. The tool is part of what's tested, so a result belongs to a setup, not to a model alone.
claude --print --model <model> --effort max --safe-mode --disable-slash-commands --no-session-persistence --tools "" --output-format text codex exec --model <model> -c model_reasoning_effort="max" --sandbox read-only --skip-git-repo-check --ignore-user-config --ignore-rules --color never -c skills.include_instructions=false -c web_search="disabled" --disable apps --disable browser_use --disable browser_use_external --disable code_mode_host --disable computer_use --disable goals --disable hooks --disable image_generation --disable multi_agent --disable plugins --disable shell_tool --disable skill_search --disable sleep_tool --disable tool_suggest --disable unified_exec --disable view_image In Season 2 neither model gets tools, skills or anything about me. Codex runs with my instruction file masked. Claude runs through a local filter that removes the account email Claude Code adds and blocks any request that still names me. Each run saves an audit of exactly what the model saw.
Codex's ultra effort is left out because it hands work to sub-agents.
Seasons
Season 2 · Into the Storm
GPT-6 Sol vs Opus 5.5, one three.js scene. One library is allowed, three.js 0.186.0 from a pinned address; everything else must be built in code. Scored on Runs, Brief, Physics and Craft. Physics is five checks anyone can verify on the recording, kept private until the results were in.
The first version of this brief, a three-masted ship with orbit controls, is withheld. GPT-6 Sol finished it in 11 minutes; Opus 5.5 at max effort spent its whole 128K-token output limit on it twice without finishing. Both models then got this lighter brief, one try each.
Claude Code restarts a reply that hits the output limit and asks the model to pick up in smaller pieces. Opus 5.5 hit the limit three times on the lighter brief before it finished, all inside its one run, which took 78 minutes. GPT-6 Sol took 10.
Season 1 · archived
GPT-5.6 Sol, GPT-5.6 Terra, GPT-5.6 Luna, Fable 5, Opus 4.8, Sonnet. No libraries at all. Scored on Loads clean, Complete, Behaves and Feel. Round 1 is an unscored pilot. Rounds 2 and 4 were withheld before publication and have no public record.
Season 1 ran Codex with --ephemeral and none of the Season 2 isolation, so its GPT runs kept read-only tools. In rounds 3 and 5, five of the six GPT runs used them to read a design guide installed on my machine before building. The Claude runs could not. Round 1 kept no logs, so it cannot be checked. The scores stand as recorded, with that caveat.
Safety
Every output is model-written code nobody reviewed line by line, so it runs in a sandboxed frame with no access to this site. A content policy blocks every network request; Season 2 outputs may load only the pinned three.js files.
Limits
One run per setup and one judge. It tests a specific kind of build, not general intelligence, and there is no overall ranking.