Reproduce “EvEntS ReaLM: Event Reasoning of Entity States via Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Evangelia Spiliopoulou, Artidoro Pagnoni, Yonatan Bisk, Eduard Hovy
EVENTS REALM investigates how Large Language Models (LLMs) can reason about physical entity state changes following events using only linguistic input. It demonstrates that effective prompting allows language-only LLMs to perform competitively with, and sometimes surpass, models relying on physical interaction for this task, achieving up to 94.1% F1 on the PiGLET dataset.
Key insights
- Language-only LLMs, when appropriately prompted, can perform comparably to physical interaction models in reasoning about event implications, suggesting that embodied interaction may not always be a prerequisite for this task.
- Task formulation and the specific prompting techniques employed are critical for eliciting and maximizing LLMs' latent reasoning abilities, significantly impacting observed performance and generalization.
- The k-attribute prompting strategy successfully mitigates training bias towards frequently occurring attributes, enhancing generalization to out-of-domain attributes while maintaining strong in-domain performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.