Reproduce “Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sitao Cheng, Liangming Pan, Xunjian Yin, Xinyi Wang, William Yang Wang
ECHOQA tests whether language models use parametric knowledge (PK) when contextual knowledge (CK) is present. The authors elicit knowledge each model can answer, then provide supportive, complementary, conflicting, or irrelevant context. On ALCUNA complementary reasoning, Llama 3.1-70B’s Unknown Ratio rises from 23.89% without context to 62.72% with complementary context, under the tested conditions.
Key insights
- Large Language Models universally suppress their parametric knowledge (PK) when contextual knowledge (CK) is present, tending to over-rely on external context regardless of its relationship to internal knowledge.
- LLMs struggle to effectively leverage their PK even when complementary CK is provided, and show high fidelity to conflicting CK, demonstrating susceptibility to misinformation.
- Commonsense knowledge exhibits greater resilience to suppression and conflict compared to scientific or factual knowledge, suggesting differences in internal representation or prevalence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.