Reproduce “InstructCMP: Length Control in Sentence Compression through Instruction-based Large Language Models” on OpenResearch

About this paper

Juseon-Do, Jingun Kwon, Hidetaka Kamigaito, Manabu Okumura

InstructCMP introduces an instruction-based framework for sentence compression, enabling Large Language Models (LLMs) to achieve precise length control through a "length priming" technique. This approach, which avoids model modifications, significantly improves length adherence (e.g., achieving a ∆CR of -0.16 on the Google dataset after fine-tuning) and ROUGE scores, often surpassing traditional compression methods.

Key insights

• Embedding explicit numerical context, termed "length priming," directly within natural language instructions significantly improves LLMs' ability to adhere to precise length constraints in sentence compression tasks, even in zero-shot settings.
• Targeted instruction-based fine-tuning, even with a small fraction of training data, substantially enhances both the compression quality and the accuracy of length control in LLMs.
• The approach generalizes effectively across various LLM sizes (e.g., Llama2-7B to 70B) and different instruction-following architectures, including commercial models like GPT-4.

Summary by alphaXiv.