Reproduce “Culture Cartography: Mapping the Landscape of Cultural Knowledge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Caleb Ziems, William Held, Jane Yu, Amir Goldberg, David Grusky, Diyi Yang
The research introduces "Culture Cartography," a mixed-initiative annotation methodology and web-tool, designed to elicit culturally salient knowledge that is genuinely challenging for Large Language Models. This approach produces "Google-Proof" data from under-represented cultures, leading to significant improvements of up to +19.2% in fine-tuned LLMs' performance on existing cultural benchmarks.
Key insights
- A mixed-initiative approach, combining human cultural expertise with LLM-identified knowledge gaps, is highly effective in eliciting culturally salient and genuinely challenging information.
- Much culturally salient knowledge from under-represented groups is not readily discoverable online, making the collected data "Google-Proof" and mitigating test set contamination issues.
- The data generated through CULTURE CARTOGRAPHY can significantly improve LLMs' cultural competence, demonstrating positive transfer performance to existing cultural benchmarks and helping close the gap with larger, proprietary models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.