Reproduce “Retrospective Learning from Interactions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zizhao Chen, Mustafa Omer Gul, Yiwei Chen, Gloria Geng, Anne Wu, Yoav Artzi
Researchers at Cornell University introduced RESPECT, a method enabling large language models to continuously improve from implicit user feedback within multi-turn interactions, eliminating the need for manual annotations or external, stronger models. The approach demonstrated substantial learning, boosting task completion rates from an initial 31% to 82% over six rounds in a complex multimodal task.
Key insights
- Large language models can reliably decode implicit feedback from user follow-up utterances, even when the model initially performs poorly on the main task, indicating these linguistic cues reside in an accessible, constrained subspace of language.
- Learning primarily from positive implicit feedback (Filtered Fine-Tuning) is more effective than utilizing both positive and negative signals in tasks with large action spaces, possibly due to the ambiguity of negative feedback without specific corrective guidance.
- Autonomous, annotation-free self-improvement is feasible for LLMs through a continuous loop of interaction, implicit feedback decoding, and policy refinement, paving the way for adaptive AI systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.