Vibe Coding Nights #45 — Bench, a Wednesday-evening build sprint in the Off the Leash season. You do not know if your coding agent is good, you have a vibe, and the session replaces the vibe with a number you can defend. An eval is three parts and one arrow: a task set, a deterministic oracle per task, and a scoring harness of about 60 lines. The oracle is a test and not an opinion (exit code is the contract, never ask another model whether it looks right), and it must be probed twice for the same verdict before any number computed after it is trusted. Then the two rates that matter: pass@k, at least one of k attempts green, versus pass^k, all k consecutive green, which is the reliability number you act on. Five lab stages: assemble a task suite from your own repo (a bug with a known fix, a refactor with unchanged behaviour, a feature with a test that does not pass yet), wire and probe an oracle each, write the harness, run each task k times from a fresh repo copy per attempt, then read the leaderboard where a striped row is your flakiness made countable. 26 slides, agent-readable /llms.txt and /setup.txt, and executable /wire.sh + /wire.ps1 doctor-then-wire helpers. Attendees type the harness themselves: there is no repository to clone. Charts are provenance-marked, and anything labelled ILLUSTRATIVE is a schematic drawn to carry an argument rather than measured data.
Open the deck ↗Rayyan Zahid · Deck · Vibe Coding Nights #45 — Bench, a Wednesday-evening build sprint in the Off the Leash season. You do not know if your coding agent is good, you have a vibe, and the session replaces the vibe with a number you can defend. An eval is three parts and one arrow: a task set, a deterministic oracle per task, and a scoring harness of about 60 lines. The oracle is a test and not an opinion (exit code is the contract, never ask another model whether it looks right), and it must be probed twice for the same verdict before any number computed after it is trusted. Then the two rates that matter: pass@k, at least one of k attempts green, versus pass^k, all k consecutive green, which is the reliability number you act on. Five lab stages: assemble a task suite from your own repo (a bug with a known fix, a refactor with unchanged behaviour, a feature with a test that does not pass yet), wire and probe an oracle each, write the harness, run each task k times from a fresh repo copy per attempt, then read the leaderboard where a striped row is your flakiness made countable. 26 slides, agent-readable /llms.txt and /setup.txt, and executable /wire.sh + /wire.ps1 doctor-then-wire helpers. Attendees type the harness themselves: there is no repository to clone. Charts are provenance-marked, and anything labelled ILLUSTRATIVE is a schematic drawn to carry an argument rather than measured data.