Reproduce “RDF-Based Structured Quality Assessment Representation of Multilingual LLM Evaluations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jonas Gwozdz, Andreas Both
An RDF-based framework enables structured representation and analysis of multilingual LLM evaluations through a standardized vocabulary and context-based testing methodology, revealing that models replicate incorrect contextual information at 89-93% rates while demonstrating significant language-specific differences in baseline knowledge between English and German.
Key insights
- Models strongly favor provided context even when incorrect, with 89-93% error replication rates
- English models handle incomplete information better while German models show stronger baseline knowledge
- Knowledge leakage from training occurs in only 7-11% of cases with conflicting information
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.