01The four axes
Models are tested where output is objectively checkable code, not prose. Game-making and Monster battle produce a playable HTML5 canvas app; Illustration hand-writes an SVG; Coding & reasoning answers held-out tasks.
02Rubrics & weights
Each axis has a fixed, versioned, mechanical rubric. Functional checks are table stakes; the points that separate strong models come from harder signals. Monster battle (battle-v2) = a 40-pt floor (loads · starts · JS contract · renders · moves fire · both fight · ends) + 60 discriminating pts: winnable 12 (a good player can actually win) · move count 8 · move stats 6 · power variety 8 · utility move 10 (a heal/defensive option) · visual density 16. The breakdown on each row shows exactly which tiers a model passed — the title on hover carries the raw key.
03Anti-gaming: idle vs controlled
A game that auto-wins while untouched would game a naive scorer. So every game is played twice — idle (no input) and controlled (a generic bot plays). Points for skill require the bot to beat idle, and the game must be losable. A frozen or auto-win build fails these tiers.
04Type-strict contract
Games expose a fixed window.__game API. The scorer checks types strictly — typeof score === 'number', not a function — so a model can’t fake a live value with a getter that returns a function. The bot drives play only through this contract.
05K-sampling & variance
Game RNG makes a single play noisy, so each artifact is scored multiple times and we report the median plus σ (spread across trials). A high σ means inconsistent quality, and it’s shown on every row.
06Offline isolation
Untrusted model code runs in a headless Chromium with the network disabled — it can’t phone home or exfiltrate. On the published site, games are embedded under a strict default-src 'none' sandbox.
07Provenance tiers
FRONTIER · PAID = the metered cloud API, exact version pinned (highest rigor). LOCAL · FREE = an open model run on a 24GB Apple machine. WEB · MANUAL = pasted from a chat UI: a single, trust-based sample, labelled distinctly because it’s lower-rigor.
08Worked example
Claude Opus 4.8 on game/v2 across 3 trials scored {75, 100, 100} → median 100, σ 11.79. grok-4.3 on the monster battle built a fully winnable duel, but its state() exposes only {name} for each move — no power — so the metadata tiers (move stats, power variety) can’t score → 72: the rubric rewards a battle that works and reports its mechanics.