Reproduce “Learning to Reason from Feedback at Test-Time” on OpenResearch

About this paper

Yanyang Li, Michael Lyu, Liwei Wang

The paper introduces Feedback-based Test-Time Training (FTTT) and OPTUNE, a learnable test-time optimizer, to improve large language model performance on complex reasoning tasks. FTTT internalizes feedback into model weights, while OPTUNE efficiently learns to predict weight updates, collectively achieving superior performance and efficiency compared to existing test-time scaling methods on mathematical and code generation benchmarks.

Key insights

• Internalizing learning from feedback directly into model weights, as FTTT does, provides a more scalable approach than context-based methods, effectively circumventing issues like length generalization and attention noise.
• A specialized learnable optimizer operating in the gradient space can significantly outperform fixed optimizers and standard PEFT methods by adapting to the sparse and unique learning signals available during test-time adaptation.
• Efficient input compression and gradient decomposition allow a meta-optimizer to learn how to predict weight updates effectively, even with a minimal number of trainable parameters.

Summary by alphaXiv.