Write the Spec Before You Pick the Model

Picking a model starts long before you open a leaderboard. This episode works through how to write a workload spec: the input distribution, the output contract, the success criterion, and an asymmetric failure-cost matrix that says which model's mistakes are cheaper for you. Along the way: AWS's four failure modes of judging on vibes, Anthropic's specificity test, rule-based checks versus judge-scored rubrics, and why thirty eval items can't tell a 92 from a 96. If you've ever chosen a model because it sounded confident, this one's for you.

Key sources:

• Anthropic (Claude) Platform Docs — https://platform.claude.com/docs/en/test-and-evaluate/develop-tests

• Salesforce AI Research — https://github.com/SalesforceAIResearch/AnchorBench

• AWS Machine Learning Blog — https://aws.amazon.com/blogs/machine-learning/beyond-vibes-how-to-properly-select-the-right-llm-for-the-right-task/

Episode 6043 — myweirdprompts.com/n/6043 (permanent short link: myweirdprompts.com/509927)