evals
Prompts tagged with "evals"
Work Out Why an Agent Run Went Wrong
Reads an agent trace to find the first step where the run actually diverged — usually well before the visible failure — classifies the failure type...
Choose Few-Shot Examples That Actually Help
Selects, orders, and stress-tests few-shot examples so they teach the boundaries of a task rather than one happy path — including when few-shot is ...
Design Retrieval That Finds the Right Thing
Designs a retrieval layer around the questions it actually has to answer — chunking, what gets indexed, filtering, reranking — and insists on evalu...
Choose the Right Model for a Task
Works out which model tier a task actually needs by defining the hardest case it must handle, then designs the head-to-head test that settles it on...
Write an LLM-as-Judge Scoring Rubric
Create a clear rubric for judging model outputs on a task — dimensions, a scoring scale, and anchored examples.
Write an Eval Set for a Prompt or Agent
Design a small, concrete eval set to test a prompt or agent before you change it — cases, inputs, and checkable pass criteria.
Guides on evals
-
How to Trace and Debug an AI Agent
Print statements don't survive a system whose control flow is decided at runtime by a model. What every span must carry, the OpenTelemetr...
Read the guide → -
Why AI Models Hallucinate
Hallucination isn't a malfunction: benchmarks reward confident guessing and penalise "I don't know", so training selects for it. Purpose-...
Read the guide → -
How to Evaluate and Test Your Prompts
A practical guide to building prompt test sets, choosing grading methods, gating CI, and catching regressions when you swap models.
Read the guide → -
How to Write Evals for Your AI Agent
The step everyone skips is the one that separates a demo from a product. How to evaluate an agent's whole trajectory — task completion, t...
Read the guide →