Reproduce “Unmasking the Genuine Type Inference Capabilities of LLMs for Java Code Snippets” on OpenResearch

About this paper

Yiwen Dong, Zhenyang Xu, Yongqiang Tian, Chengnian Sun

The research uncovers that Large Language Models' (LLMs) high performance in Java type inference is substantially inflated by data leakage from benchmarks into their training data. On a newly created, unseen benchmark, LLMs exhibited a dramatic performance drop, indicating their actual semantic understanding for type inference is significantly weaker than previously suggested.

Key insights

• LLMs' previously reported high performance on type inference benchmarks is largely an artifact of data leakage and memorization, not genuine semantic understanding of the code.
• LLMs struggle significantly with type inference for less frequently used Fully Qualified Names (FQNs) and perform substantially worse on unseen code compared to robust constraint-based methods like SnR.
• While LLMs show some resilience to individual syntactic alterations, their vulnerability to combined semantic-preserving transformations on leaked data suggests a reliance on specific syntactic patterns rather than deep logical comprehension.

Summary by alphaXiv.