Reproduce “LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lingfei Qian, Yan Wang, Xueqing Peng, Yupeng Cao, Yuechen Jiang, Zhuohan Xie, Anke Xu, Wenbo Cao and 7 more

FinDendro audits whether financial models’ historical answers reflect accessible knowledge by comparing answers with models’ claims about knowing them and testing cutoff safeguards. It evaluates 2,226 dated U.S. macroeconomic, stock-price, and Federal Reserve facts across four prompts. GPT-5’s dated instruction reduced prior-knowledge claims from 0.39 to 0.01 while accuracy stayed near 0.76, so claims cannot reliably establish prior knowledge.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.