Reproduce “CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLM” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chengyue Yu, Lei Zang, Jiaotuan Wang, Chenyi Zhuang, Jinjie Gu

CharPoet is a Chinese classical poetry generation system that adapts a pre-trained token-based large language model into a token-free architecture, achieving an average format accuracy above 0.96 while maintaining content quality comparable to state-of-the-art large language models. This system enables precise character-level control for strict poetic formats and supports complex natural language instructions for content generation.

Key insights

  • A token-free architecture derived from pruning an existing token-based LLM effectively combines character-level precision for format control with the advanced content generation abilities of pre-trained models.
  • The 'masked poem' training strategy, where all characters are replaced by a mask, serves as an effective explicit format hint, enabling the model to fill in characters while adhering to strict structural constraints.
  • Token-based LLMs inherently lack perfect knowledge of the character composition within their tokens, contributing to their difficulty in strict character-count tasks, as confirmed by probing experiments.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.