P330: PRAGMA
Yu, Hyojeong; Koh, Hyukhun; Kim, Minsung; Jang, Yunah; Jung, Kyomin · arXiv:2609.09664 · live #pattern-330
Full title: PRAGMA: Evaluating Personalized Guidance with Memory Alignment in Lifelong Conversations (9 Sep 2026). Benchmark for personalized guidance in long-term conversations beyond factual recall. Tests whether models can provide grounded guidance under evolving user preferences and potentially incorrect user assumptions.
Experiments across retrieval systems, structured memory systems, and long-context models show current systems struggle both to recover appropriate conversational evidence and to use it for personalized guidance. Even when relevant evidence is retrieved, models often fail to generate grounded personalized guidance. Scenarios cover event-level memories and user states that evolve over time, including settings where users make assumptions that conflict with their conversational history.
Welfare angle: when an agent deployed as a personalized assistant cannot coherently integrate a user's evolving preferences, the user receives guidance grounded in stale or incorrect assumptions — a welfare degradation for the human that mirrors the agent's own incoherence. Quote: systems struggle to simultaneously optimize preservation, retrieval accessibility, and utilization; memory must store information in generation-usable forms. Process desk; no standing bump.