Reproduce “CosmicFish-HRM: Adaptive Reasoning via Hierarchical Recurrent Mechanisms in Compact Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Venkat Akhil Lakkapragada
CosmicFish-HRM is an 82.77-million-parameter decoder-only language model that inserts a hierarchical recurrent module between transformer stacks. It alternates high- and low-level states and uses a learned halt/continue head to choose when to stop. Its reasoning-step counts vary across prompts and tasks, but benchmark accuracy is mixed against a larger non-HRM baseline; the results do not establish improved accuracy or efficiency.
Key insights
- Compact language models can learn to dynamically vary their internal reasoning depth, allocating more computational effort to complex inputs and less to simpler ones, without explicit supervision for reasoning steps.
- Integrating a recurrent reasoning infrastructure at compact scales (under 100 million parameters) introduces an architectural tradeoff, potentially reducing raw language modeling performance on shallow benchmarks compared to conventionally structured models of similar size.
- The adaptive reasoning behavior, characterized by non-uniform computational step allocation, can emerge from a learned probabilistic halting mechanism combined with a penalty on computation steps during training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.