Reproduce “Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search” on OpenResearch

About this paper

Hideaki Joko, Shubham Chatterjee, Andrew Ramsay, Arjen P. de Vries, Jeff Dalton, Faegheh Hasibi

The research introduces LAPS, an LLM-augmented human-in-the-loop method for collecting a large-scale, multi-session, personalized conversational dataset with human-written preferences. It demonstrates that this method yields highly diverse dialogues and that structured preference memory improves personalized recommendation quality while mitigating LLM long-context recall issues.

Key insights

• LLM-augmented human-in-the-loop data collection (LAPS) can produce dialogues with high lexical diversity and quality, successfully overcoming the diversity deficits observed in purely synthetic LLM-generated conversations.
• Storing user preferences in a structured 'preference memory' is more effective for personalized recommendations than relying on raw dialogue history, leading to better rationale and significant prompt length reduction.
• Structured preference memory helps mitigate LLM long-context recall problems, particularly for preferences disclosed in earlier sessions, improving the utilization of historical user information.

Summary by alphaXiv.