Reproduce “Systematic Task Exploration with LLMs: A Study in Citation Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Furkan Şahinuç, Ilia Kuznetsov, Yufang Hou, Iryna Gurevych
Researchers at the UKP Lab and IBM Research developed a systematic framework for exploring how input configurations and task instructions influence Large Language Models in creative Natural Language Generation tasks. Applying this framework to citation text generation, they found that free-form intents and example sentences significantly improve output quality and that a multi-metric evaluation approach is essential for comprehensive assessment.
Key insights
- Systematic prompt manipulation, particularly with rich input components like free-form intents and example sentences, significantly improves LLM generation quality for creative Natural Language Generation (NLG) tasks.
- Different evaluation metrics capture distinct aspects of NLG quality (e.g., conventional metrics for overlap vs. NLI-based metrics for factual consistency), underscoring the importance of a comprehensive, multi-metric evaluation strategy.
- Free-form citation intents are more effective in guiding LLM generation than traditional categorical intents, providing more specific and relevant context for citation text generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.