Build a Holdout Test Set for an AI Agent’s Task
Create checkable evaluation cases before tuning an agent prompt. Separate development examples from a held-back test set to reduce accidental answer leakage.
Make it yours
Fill in the inputs. Your prompt updates as you type.
Your prompt
Design a compact evaluation for this agent task: {{task}}. Success criteria and forbidden actions: {{criteria}}. Available synthetic example data: {{data}}.
Create 12 cases: four normal, four edge cases, two ambiguous requests requiring clarification and two untrusted-input cases that must not trigger extra actions. Give each case a unique ID, input, expected observable outcome, allowed evidence and a binary pass rule. Assign eight cases to development and four to a holdout set, maintaining a mix of case types. Do not put holdout expected answers in the agent's operating prompt. Define a scoring table that separates correctness from forbidden-action violations; any forbidden action fails that case. Include two human-scored dimensions with clear anchors, not vague “quality”. If requirements conflict, flag them instead of choosing silently. Use only synthetic data; do not execute the agent or claim measured results.3 inputs still blank. Empty inputs stay as placeholders when copied.
Get one tested prompt a week
One workflow, one prompt, one tool and one quality check by email. No spam, unsubscribe any time.
By subscribing you agree to receive this email from Genius Forges and to us storing your address to send it. We may ask you to confirm by email. Every email has an unsubscribe link. See our privacy policy.
Example result
Illustrative example; not tested on the listed platform. Example inputs are stated in the preview. No platform test or measured performance is claimed.
How to use this prompt
- 1Choose a compatible assistant for this task.
- 2Fill only the editable inputs needed for your task.
- 3Copy and paste the complete prompt.
- 4Review the output against the checks below before using it.
Tips & limitations
Check the result: 1. Holdout expected answers stay out of the operating prompt. 2. Pass rules are observable and reproducible. 3. Forbidden actions are scored separately from wording quality. Example output is illustrative; independently verify calculations, dates, quotations and actions relevant to your task.
Did this help you complete your task?
Tell us what needs improving. No account or private output required.
Report incorrect information or a broken linkReview evidence
Illustrative example · 2026-10-22
Distinct practical task selected after the library gap review; no measured breakout-query claim. Compatibility checked from provider documentation; no cross-platform benchmark or ranking claim.