Write an Eval Set for a Prompt or Agent
Design a small, concrete eval set to test a prompt or agent before you change it — cases, inputs, and checkable pass criteria.
0 likes
0 dislikes
Sign in to rate this prompt
Prompt
Help me build an eval set to test a prompt or agent, so I can catch regressions before shipping changes.
What the prompt or agent is supposed to do:
{{purpose}}
The prompt, or a description of its inputs and outputs (optional):
{{prompt}}
Produce:
1. **10–15 test cases** covering the main use cases, important edge cases, and known failure modes. For each: a short name, the input, and what a good output must (and must not) do — as a checkable pass criterion, not a vibe.
2. **A few adversarial cases** — ambiguous, out-of-scope, or tricky inputs that probe where it's likely to fail.
3. **How to score** — for each case, whether it can be checked automatically (exact / substring / format) or needs a judge (human or LLM), and the rule to apply.
Keep criteria concrete and objective enough that two people would grade a response the same way. If the intended behavior is underspecified, list the assumptions you made so I can correct them.
Why this prompt works
-
How to Trace and Debug an AI Agent
Print statements don't survive a system whose control flow is decided at runtime by a model. What every span must carry, the OpenTelemetr...
Read the guide → -
Why AI Models Hallucinate
Hallucination isn't a malfunction: benchmarks reward confident guessing and penalise "I don't know", so training selects for it. Purpose-...
Read the guide → -
Claude vs Gemini vs ChatGPT
Google's Flash rate doubles on 1 January 2027 and its flagship is still preview; OpenAI spans 50x from Luna to Astra with no retirement d...
Read the guide →