Reproduce “CAT-LM: Training Language Models on Aligned Code And Tests” on OpenResearch

About this paper

Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, Vincent J. Hellendoorn

Researchers at Carnegie Mellon University developed CAT-LM, a 2.7 billion parameter GPT-style model pretrained on aligned code and test files with an 8,192 token context window to generate complete test methods. This approach enables CAT-LM to generate more passing tests and achieve higher lexical similarity with human-written tests compared to larger general-purpose LLMs, while outperforming the specialized test completion model TeCo by 36.6% in exact match accuracy.

Key insights

• Explicitly incorporating the alignment between code and test files during LLM pretraining is crucial for generating high-quality, human-like test methods.
• Significantly increasing the context window enables LLMs to effectively utilize the full code context, leading to improved performance in test generation tasks.
• A moderately sized LLM with domain-specific training can outperform much larger general-purpose LLMs on specialized software engineering tasks like test generation.

Summary by alphaXiv.