Reproduce “Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities” on OpenResearch

About this paper

Wei Ma, Shangqing Liu, Mengjie Zhao, Xiaofei Xie, Wenhan Wang, Qiang Hu, Jie Zhang, Yang Liu

A layer-by-layer probing study compares seven pretrained code models, including encoder, encoder-decoder, and decoder-only architectures, to determine how their representations encode syntax and program semantics. The models reliably expose AST structure and token roles, but semantic information—especially control flow and long-range dependencies—is less consistent, with intermediate layers often more useful than final outputs and larger LLMs offering no uniform probing advantage.

Key insights

• Syntax is more consistently recoverable than semantics: shallow layers tend to expose AST relationships, while middle or deeper layers better support fine-grained token-role tagging.
• Data and control dependencies are generally easier to recover than immediate control flow, and data-dependency propagation is more accessible over long ranges than control-dependency propagation.
• Representation quality varies strongly by architecture and layer; larger decoder-based LLMs do not consistently outperform smaller pretrained encoders, and semantic information is concentrated in specialized attention heads rather than uniformly distributed.

Summary by alphaXiv.