Fielded Max-Sim Keying for Assistant-Side Memory Recall in Long-Term Conversational Assistants

Abstract

Long-term conversational assistants rely on retrieval systems to recall relevant information from extensive interaction histories. Current approaches index only user utterances, creating a fundamental mismatch for single-session-assistant (SSA) queries where users ask about information the assistant previously provided. We propose \emph{fielded max-sim keying}, which treats each conversation round as a two-field document with separate embeddings for user and assistant content, scoring by maximum similarity across fields. On the LongMemEval benchmark, our method achieves a \emph{zero-harm property}: identical overall Recall@10 (0.664) to user-only indexing while improving SSA Recall@10 by +0.018 and NDCG@10 by +0.038. In contrast, concatenation-based indexing suffers catastrophic overall degradation ($-$0.145 Recall@10) due to cross-field interference. The method requires no additional training and serves as a drop-in replacement for existing retrieval pipelines.