Reproduce “HALLMARK: How to Diagnose LLM Citation Verifiers and When to Deploy Them” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kayoon Kim, Wieland Brendel, Patrik Reizinger
HALLMARK benchmarks citation verifiers using 2,526 BibTeX entries, 14 error types, and six diagnostic checks. It compares detection and false-positive rates, showing that Claude Sonnet 4.6 and DeepSeek-V3.2 have similar detection rates on dev_public (.916 and .911) but false-positive rates of .161 and .702. At assumed 2% prevalence, estimated PPVs are 10.4% and 2.6%; rankings also change with hallucination definitions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.