Reproduce “Mitrasamgraha: A Comprehensive Classical Sanskrit Machine Translation Dataset” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sebastian Nehrdich, David Allport, Sven Sellmer, Jivnesh Sandhan, Manoj Balaji Jagadeeshan, Pawan Goyal, Sujeet Kumar, Kurt Keutzer

Researchers developed Mitrasamgraha, the largest and most diverse public Sanskrit-to-English machine translation dataset, comprising 391,548 aligned sentence pairs across three millennia and six historical domains. This resource significantly improves machine translation performance, with fine-tuned Gemma 2 9B achieving a GEMBA score of 83.34, and establishes that neural- and LLM-based metrics better correlate with human judgment for complex languages.

Key insights

  • Neural-based (BLEURT) and LLM-based (GEMBA, GEMBA*) MT metrics correlate significantly better with expert human judgments for Sanskrit than traditional string-based metrics like BLEU and chrF.
  • Fine-tuning open-source models on Mitrasamgraha drastically improves their performance, with decoder-only LLMs demonstrating stronger zero-shot capabilities than encoder-decoder models.
  • Retrieval-Augmented Generation (RAG) effectively boosts the performance of commercial Large Language Models in Sanskrit-to-English translation tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.