Reproduce “Large Language Models can Learn Rules” on OpenResearch

About this paper

Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, Hanjun Dai

The Hypotheses-to-Theories (HtT) framework enables Large Language Models (LLMs) to automatically learn and apply a verified library of explicit textual rules from standard training data. This approach significantly enhances reasoning accuracy, demonstrating absolute gains of 10-30% on tasks such as relational reasoning, counterfactual arithmetic, and concept learning by reducing the generation of incorrect intermediate steps.

Key insights

• Large Language Models (LLMs) can generate correct rule hypotheses with non-trivial probability; aggregating and verifying these hypotheses across examples yields a reliable rule set.
• Explicitly retrieving an existing, verified rule is a more reliable task for LLMs than generating a novel correct rule on the fly.
• Structuring the rule library and guiding retrieval through techniques like XML tagging significantly improves an LLM's ability to apply the learned rules effectively.

Summary by alphaXiv.