Reproduce “Language Models' Factuality Depends on the Language of Inquiry” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tushar Aggarwal, Kumar Tanmay, Ayush Agrawal, Kumar Ayush, Hamid Palangi, Paul Pu Liang
This research investigates why multilingual language models (LLMs) often fail to consistently recall factual information across different languages, suggesting knowledge is siloed rather than universally accessible. It introduces a benchmark of 10,000 country-related facts in 13 languages and three new metrics (FRS, KTS, X-FaKT) to quantify factual recall, knowledge transferability, and cross-lingual generalization. Experiments reveal that larger models exhibit better factual consistency, but all models show a trade-off between strong factual retrieval and adherence to counterfactual contexts, with performance varying significantly by language resourcefulness and script similarity.
Key insights
- Factual knowledge in multilingual LLMs is not consistently transferred across languages, with performance varying based on language resourcefulness and script similarity.
- A trade-off exists between a model's ability to retrieve factual knowledge and its adherence to contradictory in-context information (counterfactual adherence).
- Larger model sizes generally correlate with improved cross-lingual factual consistency, as indicated by higher X-FaKT scores.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.