Reproduce “Toward Graph-Tokenizing Large Language Models with Reconstructive Graph Instruction Tuning” on OpenResearch

About this paper

Zhongjian Zhang, Xiao Wang, Mengmei Zhang, Jiarui Tan, Chuan Shi

Researchers from Beijing University of Posts and Telecommunications and Beihang University developed RGLM (Reconstructive Graph Instruction Tuning), a method that explicitly integrates graph supervision into Graph-Tokenizing Large Language Models (GTokenLLMs) via reconstructive tasks. This approach addresses the implicit graph-text alignment issue, achieving superior performance on graph tasks, improved generalization across datasets, and enhanced zero-shot capabilities, with negligible computational overhead.

Key insights

• Existing GTokenLLMs primarily optimize textual autoregressive dependencies, leading to an implicit graph-text alignment and suboptimal utilization of graph context, as formally demonstrated by information-theoretic analysis.
• Explicitly incorporating graph supervision through reconstructive tasks optimizes a variational lower bound of the mutual information between the input graph and the LLM's hidden representations, thereby enhancing the overall graph-text alignment.
• Reconstructing raw node features and graph topology (RGLM-Decoder) or powerful latent GNN representations (RGLM-Similarizer/Denoiser) both effectively guide LLMs to better encode and reason over graph structures.

Summary by alphaXiv.