Definition Unit Tests Improve LLM Convention Adherence

Abstract

Large language models often know multiple valid conventions for mathematical notation but default to the wrong one when a specific convention is required. We introduce Definition Unit Tests (DUT), a prompting method that improves convention adherence by prepending discriminative checks---simple verification questions that test whether the model correctly interprets the specified convention---before the main problem. On ErdosConventionsBench, a benchmark of 300 mathematical problems spanning three convention families, DUT improves accuracy by +5.0 percentage points on Qwen2.5-Math-7B-Instruct and +22.7 percentage points on Llama-3.1-8B-Instruct compared to engagement-matched baselines that control for additional computation. DUT also outperforms majority voting over five samples while using only a single generation, and reduces the rate of alternate-convention answers by approximately 80%. Our results demonstrate that discriminative definition binding effectively anchors models to specified conventions, addressing a key challenge in deploying LLMs for tasks requiring precise adherence to domain-specific terminology.