Reproduce “UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Following” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jeet Sharma, Balpreet Kaur, Jeremiah Hong, Hamed Zamani, Haw-Shiuan Chang
The UNSPECIFIC framework, developed by researchers at the University of Massachusetts, Amherst, generates more natural and challenging instruction-following benchmarks for large language models by synthesizing generalized constraints from multiple sources and iteratively refining them. It introduces a "core satisfaction rate" metric that evaluates constraint adherence after article summarization, revealing that models often exhibit superficial instruction following, with core satisfaction rates significantly lower than original rates.
Key insights
- The "core satisfaction rate" metric, derived from evaluating constraints on summarized text, effectively distinguishes superficial from deep instruction following in LLMs.
- Synthesizing constraints that are common to multiple similar articles, combined with selective revision, creates more natural and challenging benchmarks, mitigating copy-and-paste behavior.
- LLMs exhibit varied levels of superficial instruction following across different domains (e.g., news vs. blogs) and task types (generation vs. editing), and this behavior is influenced by model scaling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.