Reproduce “Cross-Examination Framework: A Task-Agnostic Diagnostic for Information Fidelity in Text-to-Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tathagata Raha, Clement Christophe, Nada Saadi, Hamza A Javed, Marco AF Pimentel, Ronnie Rajan, Praveenkumar Kanithi
The Cross-Examination Framework (CEF) introduces a reference-free, task-agnostic method utilizing large language models for generating and answering questions to evaluate information fidelity in text-to-text generation. CEF provides interpretable scores for content coverage, consistency, and conformity, showing strong alignment with human judgments for semantic errors across diverse tasks like machine translation, summarization, and clinical note generation.
Key insights
- Robust, reference-free evaluation of information fidelity is achievable through integrated LLM-based question generation and answering, particularly with a 'YES-only' question constraint for factual grounding.
- The stability of the judge LLM (low disagreement and deviation rates) is crucial for the reliability of LLM-as-judge evaluation frameworks.
- CEF's multi-dimensional scores effectively diagnose specific semantic errors (omissions, hallucinations, contradictions) often missed by traditional, surface-level metrics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.