Reproduce “FASTTRACK: Fast and Accurate Fact Tracing for LLMs” on OpenResearch

About this paper

Si Chen, Feiyang Kang, Ning Yu, Ruoxi Jia

FASTTRACK introduces a two-stage pipeline for fact tracing in Large Language Models (LLMs) that efficiently identifies supporting training data for generated text. The method achieves a substantial F1 score of 0.72 on FTRACE-TREx, more than doubling existing methods, and runs up to 33 times faster than previous attribution techniques.

Key insights

• Harnessing LLMs for direct 'supportiveness' evaluation, rather than just similarity, is crucial for accurate fact tracing, and can be made computationally tractable through strategic clustering and batching.
• A hierarchical semantic clustering approach, coupled with LLM-assigned keywords, effectively reduces the search space for fact tracing from the entire corpus to a manageable subset of relevant clusters.
• Batch processing of candidate training examples during LLM inference significantly improves efficiency without sacrificing the LLM's fine-grained reasoning capabilities for supportiveness assessment.

Summary by alphaXiv.