Reproduce “Benchmarking and Improving Compositional Generalization of Multi-aspect Controllable Text Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Tianqi Zhong, Zhaoyi Li, Quan Wang, Linqi Song, Ying Wei, Defu Lian, Zhendong Mao

Researchers from the University of Science and Technology of China and collaborators developed CompMCTG, a benchmark to evaluate compositional generalization in multi-aspect controllable text generation (MCTG), revealing a significant performance drop in existing models. They also introduced Meta-MCTG, a meta-learning framework that improves compositional attribute accuracy by up to 3.64% for unseen attribute combinations.

Key insights

  • All evaluated MCTG baseline models and large language models exhibit a noticeable performance degradation in compositional testing, confirming a significant compositional generalization risk.
  • The Attribute Compound Divergence (ACD) protocol effectively amplifies the compositional generalization risk, providing a more realistic assessment than random sampling.
  • Meta-MCTG improves compositional generalization by encouraging the model to learn more sparsely distributed hidden representations, helping distinguish between in-distribution and compositional attribute combinations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.