Reproduce “SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiaoxin Lu, Ranran Haoran Zhang, Rui Zhang
SIMMER evaluates whether kitchen plans remain safe after execution, not only whether each action is executable. Its symbolic benchmark tracks typed actions, object states, and effects across 100 cooking tasks, then audits final hazards. For GPT-5.4, foresight reduced latent failures from 1.11 to 0.34 per plan and irreversible failures from 0.55 to 0.30, within this modeled kitchen setting.
Key insights
- LLM-generated plans for complex tasks exhibit pervasive failures, with fewer than 17% being error-free even for frontier models, challenging previous estimations of their planning reliability.
- Latent failures, particularly those involving state propagation (53.0%) and implicit precondition ignorance (26.4%), represent a critical and often irreversible safety risk that current LLM planning evaluations miss.
- The "Counterfactual Foresight Simulation" prompting strategy significantly mitigates both immediate (73-80% reduction) and latent (69-72% reduction) planning failures across various LLMs, proving more robust than post-hoc self-critique or interactive methods that increase latent failures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.