Write an Eval Set for a Prompt or Agent
Design a small, concrete eval set to test a prompt or agent before you change it — cases, inputs, and checkable pass criteria.
0 likes
0 dislikes
Sign in to rate this prompt
Prompt
Help me build an eval set to test a prompt or agent, so I can catch regressions before shipping changes.
What the prompt or agent is supposed to do:
{{purpose}}
The prompt, or a description of its inputs and outputs (optional):
{{prompt}}
Produce:
1. **10–15 test cases** covering the main use cases, important edge cases, and known failure modes. For each: a short name, the input, and what a good output must (and must not) do — as a checkable pass criterion, not a vibe.
2. **A few adversarial cases** — ambiguous, out-of-scope, or tricky inputs that probe where it's likely to fail.
3. **How to score** — for each case, whether it can be checked automatically (exact / substring / format) or needs a judge (human or LLM), and the rule to apply.
Keep criteria concrete and objective enough that two people would grade a response the same way. If the intended behavior is underspecified, list the assumptions you made so I can correct them.