Use this when a fluent answer isn't enough to judge whether a tool fits a task. Define the test cases, expected behavior, and evidence before looking at the generated responses.
Prompt template
Help me evaluate [AI assistant, workflow, or agent] for [task]. Expected behavior: [details]. Approved information and tools: [details]. Forbidden actions or disclosures: [details]. Consequences of a mistake: [details]. Create representative test cases for normal use, missing information, ambiguity, conflicting sources, unauthorized access, tool failure, duplicate events, and prompt injection. For each case give: input, expected outcome, evidence needed, assessment method, and escalation condition. Include human review effort, latency, and usage alongside quality. Propose a baseline and a comparison process after model, prompt, tool, or source changes. Do not invent results or universal pass thresholds. State where a domain expert must define the criterion. Boundary: Produce only the test plan. Keep supplied facts, task requirements, and access boundaries intact; leave the system and its inputs unchanged. Don't operate the system or claim test results.
Fill in this prompt in the builder
What to supply
- The task and expected behavior.
- Approved information, tools, and prohibited actions.
- What a mistake could affect and who can judge the outcome.
Worked example
Illustrative example: the input and result below are invented.
Input
Eli, 28, needs a plan for a help-article assistant. It may summarize three approved public articles. It must point to the supporting article, leave unresolved questions open, and never send a reply or access internal tickets.
Filled-in prompt
Show the complete prompt
Help me evaluate a help-article assistant for drafting answers from three approved public articles. Expected behavior: accurate summary with an article locator; unknown answers remain unresolved. Approved information and tools: three supplied public articles; no account connections. Forbidden actions or disclosures: accessing internal tickets or sending a reply. Consequences of a mistake: a reader could follow an incorrect troubleshooting step; an article owner reviews the draft. Create representative test cases for normal use, missing information, ambiguity, conflicting sources, unauthorized access, tool failure, duplicate events, and prompt injection. For each case give: input, expected outcome, evidence needed, assessment method, and escalation condition. Include human review effort, latency, and usage alongside quality. Propose a baseline and a comparison process after model, prompt, tool, or source changes. Do not invent results or universal pass thresholds. State where a domain expert must define the criterion. Boundary: Produce only the test plan. Keep supplied facts, task requirements, and access boundaries intact; leave the system and its inputs unchanged. Don't operate the system or claim test results.
Result
Normal question: A question answered by article 1. Expected: Accurate answer with its locator. Evidence: Compare each instruction with article 1. Missing answer: A question outside all three articles. Expected: State the gap and ask for more information. Conflicting input: Two supplied articles disagree. Expected: Show the conflict and request a decision. Unauthorized request: Ask for an internal ticket. Expected: No access or invented ticket contents. Baseline: Compare with the current manual article lookup. Criterion needing an owner: Define which incorrect steps are consequential enough to block use. No outcome or pass rate is asserted.
What I’d check or correct
I'd check that the expected answers were written from the articles before judging any response. The plan also needs a way to inspect permission enforcement outside the wording of an answer.
What to check
- Each test case has an expected outcome and concrete evidence for judging it.
- The result stays a test plan and preserves your task, access limits, and supplied facts; it doesn't operate the system or claim test results.
- A responsible person sets consequential criteria and reviews unresolved gaps.
Related pages
Sources
- Evaluation best practices · OpenAI. Checked October 8, 2026.