Reproduce “SFAD: Speculative Factuality-Aware Decoding” on OpenResearch

About this paper

Guanqiao Chen, Di Wang, Lijie Hu

SFAD trains a context-aligned draft model with DPO to support a larger target model during speculative decoding. It measures draft confidence and disagreement to choose ordinary verification or selectively raise target logits for draft-supported tokens. On 200 held-out LLM-AggreFact instances, SFAD reports 85.2 FaithScore and 2.48× ATGA, versus 38.5 and 2.18× for standard speculative decoding, for this evaluation.

Key insights

• Speculative verification alone cannot overcome a target model’s biased distribution: on MQUAKE, SFAD raises the target probability of faithful tokens from 18.73% to 62.45%, while standard speculative decoding reaches only 18.91%.
• Confidence-weighted disagreement is more selective than raw model divergence, allowing the system to avoid interventions when the draft is uncertain and disagreement may reflect style rather than factual conflict.
• Atomic perturbations in ConFide expose fine-grained hallucinations such as entity swaps and relation inversions, producing a more effective factuality-oriented draft model than the comparison preference data.

Summary by alphaXiv.