Fast Local — Quantize and Serve Local Models Fast

Vibe Coding Nights #47 — Fast Local, a Wednesday-evening build sprint in the Off the Leash season. #41 Bare Metal got a coding agent running on local open weights with no API bill, and then it crawled; an agent that takes forty seconds to think is an agent you stop using. This session makes the local rig quick enough to live on. Slow is three separate problems, not one, so the deck separates prefill from decode and measures each. Five knobs in a fixed order, and the order is the argument: get every layer resident on the card first, then shrink, then serve properly, then tune one knob at a time and re-time after each. The method is measure, change exactly one thing, measure again. Rehearsed on real hardware before delivery rather than asserted: on an RTX 2080 Ti with llama.cpp b10289 and Qwen2.5-Coder-3B, quantizing Q8_0 to Q4_K_M is worth 1.40x (126.55 to 176.85 decode tok/s) while simply getting the model onto the card is worth 11.9x (14.83 to 176.85), which inverts the usual claim that quantization is the big lever. Quantization is not how you go fast, it is how you afford to go fast, by making the model fit. The convert-and-quantize chain on slide 9 was executed end to end and cross-checked on a second machine to the same number. 17 slides, a hosted-model speed bar timed live in the room to score against, agent-readable /setup.txt, and a five-stage hands-on hour. Provenance is marked throughout: the llama.cpp figures are measured on the named card, and the vLLM commands are documentation-derived and carry a check-your-own-help warning.