Reproduce “LIFEBench: Evaluating Length Instruction Following in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wei Zhang, Zhenhong Zhou, Kun Wang, Junfeng Fang, Yuanhe Zhang, Rui Wang, Ge Zhang, Xavier Li and 4 more
The LIFEBENCH benchmark evaluates Large Language Models' (LLMs) ability to follow explicit length instructions during text generation, revealing significant shortcomings in precise output length control, especially for longer contexts. It shows that LLMs frequently fail to meet vendor-claimed maximum output lengths, even as they often maintain the quality of the generated content according to other instructions.
Key insights
- LLMs demonstrate significant limitations in following explicit length instructions, particularly for 'Equal To' constraints and longer output requirements, with performance deteriorating sharply beyond short lengths.
- A substantial discrepancy exists between vendor-claimed maximum output lengths and models' actual ability to generate text to those limits, indicating inherent generation limitations.
- Length control is influenced by input characteristics, with summarization tasks performing worse, Chinese language contexts leading to over-generation, and increased input length surprisingly hindering precise output length control.
- Models often lack accurate internal 'length awareness' and frequently resort to 'lazy generation strategies' (e.g., direct refusal or premature termination) for more challenging long-length constraints.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.