The prompts we use to guide our AI judge are critical to fair outcomes. This post details our prompt engineering process.
Why Prompts Matter
Large language models are highly sensitive to how questions are framed. A slight change in wording can significantly shift outcomes. For debate judging, this means our prompts must be: - Neutral on the debate topic - Clear about evaluation criteria - Consistent across all debates
Our Prompt Structure
Our judging prompt has several components:
1. Role Definition We establish the AI as an impartial judge focused on argument quality, not personal opinions.
2. Evaluation Criteria We explicitly list what makes a strong argument: - Logical validity - Evidence quality - Addressing opponent's points - Clarity of expression
3. Anti-Bias Instructions We include explicit instructions to ignore: - Writing style preferences - Argument length - Position on the topic itself
4. Output Format We specify exactly how to format the judgment for consistent parsing.
Iterative Refinement
We've gone through 47 major prompt revisions. Each revision is tested against our benchmark suite before deployment.
What We've Learned
- Specificity beats vagueness
- Examples in prompts help consistency
- Negative instructions ("don't favor...") are less effective than positive framing
- Regular re-evaluation is essential as models update