Reproduce “Collaborative LLM Numerical Reasoning with Local Data Protection” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Min Zhang, Yuzhe Lu, Yun Zhou, Panpan Xu, Lin Lee Cheong, Chang-Tien Lu, Haozhu Wang

A collaborative framework allows low-capacity local models to perform complex numerical reasoning over documents by interacting with powerful remote LLMs, all while protecting sensitive local data. The methodology achieved up to 90.1% accuracy on the FinQA dataset with Llama-3.2-3B-Instruct, closely approaching fully remote model performance, while maintaining data leakage as low as 3.7%.

Key insights

  • The underlying mathematical structure and reasoning patterns of a numerical reasoning query can be abstracted and preserved even when its semantic topic is shifted.
  • Leveraging remote LLMs to generate executable code as 'tools' provides a robust and error-resistant mechanism for transferring complex reasoning patterns, avoiding the need for local model re-inference.
  • Decoupling semantic topic shifting from numerical value anonymization simplifies the data protection process and enhances the logical consistency of synthesized queries.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.