Reproduce “C$^2$DLM: Causal Concept-Guided Diffusion Large Language Models” on OpenResearch

About this paper

Kairong Han, Nuanqiao Shan, Ziyu Zhao, Zijing Hu, Xinpeng Dong, Junjian Ye, Lujia Pan, Fei Wu and 1 more

C²DLM introduces a paradigm that integrates concept-level causal knowledge into Diffusion Language Models (DLMs), enhancing their reasoning capabilities and training efficiency. By guiding attention mechanisms with extracted concept-level causal graphs, C²DLM achieved a 12% performance increase in normal Chain-of-Thought settings and a 3.2x training speedup compared to baseline DLMs.

Key insights

• Explicitly integrating concept-level causal knowledge can significantly enhance reasoning capabilities and training efficiency in Diffusion Language Models.
• The attention mechanism in current LLMs often misaligns with the causal priors inherent in natural language, hindering stable and effective reasoning.
• A 'V-aware' strategy, which incorporates the semantic importance captured by the Value matrix, is crucial for stabilizing and effectively guiding attention with causal signals.
• Automated extraction of concept-level causal meta-knowledge using a teacher LLM is a cost-effective method to acquire structured causal supervision.

Summary by alphaXiv.