Speaker-Attested Grounding for False Memory Resistance in Agent Memory Systems

Abstract

Agent memory systems enable long-term personalization by extracting and storing user information from conversations. However, these systems are susceptible to false memory storage, where assistant-generated content that was never confirmed by the user enters persistent memory. We identify a key failure mode: when the full dialogue serves as evidence for memory verification, assistant-originated statements can ``self-verify,'' passing filtering despite lacking user attestation. We propose Speaker-Attested Grounding (SAG), a minimal intervention that restricts the evidence corpus to user turns only during memory filtering while maintaining full-dialogue extraction. On the HaluMem benchmark, SAG improves False Memory Resistance by +11.94 percentage points (58.76% $\rightarrow$ 70.70%) with a moderate recall tradeoff ($-$7.20pp). Per-source analysis reveals that 100% of SAG's gains come from assistant-only interference memories, confirming the targeted mechanism. Ablation studies show that 47.5% of the improvement stems from the speaker restriction itself, with the remainder from reduced evidence length.