Selective Self-Reference for LLM-as-a-Judge: Using Self-Consistency to Reduce Error Propagation

Abstract

Self-reference, where an LLM judge first solves a task before evaluating candidate responses, has been shown to improve judgment accuracy. However, when the judge's self-generated answer is incorrect, using it as reference can propagate errors into the final judgment. We identify this error propagation problem and observe that self-consistency among multiple sampled solutions serves as a reliable proxy for answer correctness: unanimous agreement across five samples yields 78.66% correctness compared to only 35.51% for three-of-five agreement. Building on this insight, we propose Selective Self-Reference (SSR-Judge), which uses an agreement gate to conditionally enable self-reference only when the model exhibits high confidence. On MMLU-Pro pairwise preference judgment, SSR-Judge achieves 58.93% accuracy, outperforming both no-reference (52.07%) and always self-reference (58.21%) baselines by avoiding error propagation on uncertain items while preserving self-reference benefits when confidence is high.