Reproduce “State Soup: In-Context Skill Learning, Retrieval and Mixing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Maciej Pióro, Maciej Wołczyk, Razvan Pascanu, Johannes von Oswald, João Sacramento
Research from IDEAS NCBR, Google Deepmind, and Google explores "state soups," a method for in-context skill learning, retrieval, and mixing in modern recurrent neural networks like Mamba. The work demonstrates that internal states encode task-specific information, allowing for reliable retrieval and improving few-shot learning performance, with mixtures of 4-shot states achieving results comparable to 32-shot sequences.
Key insights
- Internal states within modern gated-linear RNNs encode discernible task-specific information, evident through clear clustering, allowing their use as distinct "task vectors."
- Simple linear operations, such as averaging these task-representing states, can effectively enhance few-shot learning performance and potentially compose new skills.
- The inherent linearity of single State-Space Model layers enables an approximate decomposition and recombination of states from different parts of a sequence, offering a pathway for efficient long-context processing.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.