Reproduce “Learning on the Job: Continual Learning from Deployment Feedback for Frozen-Weights Agents” on OpenResearch

About this paper

Valentin Tablan, Scott Taylor, Kristoffer Bernhem

AI agents with frozen model weights can continually learn and improve from operational feedback, such as simple outcome verdicts or detailed corrections, by storing distilled insights in an external memory system. This approach enabled a Mistral Large agent to achieve a 2.6-fold increase in single-trial task success and allowed for the successful transfer of learned knowledge between different foundation models.

Key insights

• Operational feedback, encompassing both simple outcome verdicts and detailed corrections, provides a sufficient signal for gradient-free continual learning in frozen-weights agents without modifying their core model weights.
• External, shareable memory systems enable agents to overcome the static nature of their frozen models, dynamically acquiring and applying new knowledge from real-world experiences.
• Learned knowledge accumulated in the external memory is transferable and beneficial across different foundation models, demonstrating potential for organizational knowledge sharing and collective intelligence.

Summary by alphaXiv.