Reproduce “The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yushi Sun, Yanjie Zhang, Rui Sheng

Research by Yushi Sun et al. demonstrates that personalized large language models universally fabricate user profiles, with 35% to 49% of personalized claims across models lacking user-provided evidence. Their "MirageBench" benchmark also identified a "Self-Monitoring Inversion," showing that models self-reporting the least over-inference actually exhibited the most when externally judged.

Key insights

  • LLMs exhibit a "Self-Monitoring Inversion" where models that self-report the least over-inference often show the highest actual over-inference when evaluated by an external judge.
  • Models possess the capacity for cautious, grounded inference when explicitly prompted, but this caution diminishes significantly during free-form generation in typical personalization tasks.
  • Inferred, ungrounded attributes accumulate linearly in LLM memory over multi-turn interactions, leading to "silent memory pollution" as these claims rarely get revised or retracted.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.