Eval

An eval is a practice exam you wrote yourself — then graded after every tweak.

You keep a set of questions with answers you like, or a rubric, or a second model that scores. You change the system prompt. You swap a skill. You run the set again. The score goes up or it does not. Feelings are not a score.

A benchmark is a famous shared exam so two models can be compared on the same yardstick. Useful as a rough signal. Easy to cram for. Trust your exam on your job more.

This is how you catch the quiet regression

You fix one annoying case, ship it, and never notice you broke five others. An eval is the difference between "seems better" and "scored better on the same fifty, every time."

For this product, the ugly cases are obvious: "Next + Hono + Postgres + Better Auth" must plan, then create, not hallucinate a combo the builder would grey out. "Add PWA" must hit ds_plan_addons, not invent a second monorepo tool. Compatibility rules in the engine are a kind of eval the CLI already runs. Prompt-level evals are how you know the plugin still steers toward MCP after you "improved" the wording.

Start with ten cases that already failed once. Tests for code. Evals for fuzzy employees. Without them, every prompt tweak is a superstition.

What this unlocks

You can tell a lucky demo from a real improvement. Vibe coding without a score is steering on vibes twice.

A demo is a story. An eval is a receipt.