Reproduce “METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hyeonseok Lee, Gabin An, Shin Yoo
METAMON checks whether Java method documentation agrees with observed behavior in generated tests, using an LLM to judge test assertions and their inverted counterparts. On 9,482 test–documentation pairs from five Defects4J projects, it reaches an F1 score of 0.576 at its default threshold; stricter thresholds raise precision as high as 1.000 while sharply reducing recall.
Key insights
- Pairing observed test behavior with documentation gives the LLM concrete evidence to evaluate, while inverted assertions test whether its judgments change in the expected direction.
- Metamorphic prompts improve separation between consistent and inconsistent cases compared with using original prompts alone; repeated querying also helps, with diminishing returns.
- Documentation coverage and model reasoning are key constraints: many cases score near zero, and manual analysis of high-confidence errors often traced them to LLM reasoning limitations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.