Reproduce “Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fengxian Ji, Yuke Li, Jingpu Yang, Juanfan Wu, Fan Zhang, Zhexuan Cui, Yu Xie, Min Peng and 3 more
The study introduces SciStyleBench, a framework to assess if large language models judging scientific ideas prioritize presentation style over scientific content. It reveals that direct LLM evaluations are highly sensitive to stylistic variations, and proposes SciStyleExtractor, a mitigation module that significantly enhances substance recognition and adversarial robustness in idea assessment.
Key insights
- LLM judges, when evaluating scientific ideas directly, are often more susceptible to stylistic influences than they are to discerning actual scientific substance.
- Robust evaluation requires a balanced assessment using metrics that capture both score stability under stylistic variation and sensitivity to changes in scientific quality.
- Explicitly separating presentation style signals from scientific content can guide LLM judges towards more substance-focused and reliable evaluations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.