Reproduce “Knowledge-Dependency Scores Rank Weak Distractors Too High” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Anonymous authors

The authors test whether model-based knowledge-dependency scores respond to distractor changes by comparing paired items with the stem, fact, and answer fixed. Across two English science datasets, most evaluators score off-topic-distractor variants higher; Qwen2.5-7B’s pairwise accuracy on OpenBookQA is 0.302. The exception is the released four-solver baseline, whose result depends on recomputed weights.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.