Reproduce “Debugging code world models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Babak Rahmani
Babak Rahmani's study diagnoses the operational characteristics of Code World Models (CWMs), revealing that while they reliably propagate state over long execution sequences when actions are correctly provided, their end-to-end performance is primarily limited by incorrect action generation and a unique brittleness in handling string-valued state, exacerbated by subword tokenization issues and token-budget exhaustion from dense trace revelations.
Key insights
- Dense full-state supervision in CWMs transforms long-horizon state tracking into a sequence of locally supervised updates, enabling accurate state propagation over many steps if actions are correct.
- Subword tokenization (BPE) creates discontinuities that make string-valued state uniquely brittle for character-level operations, leading to significant error compounding in string compositions.
- The primary cause of long-horizon degradation in CWMs is action hallucination (generating incorrect next commands), rather than fundamental errors in local state computation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.