Reproduce “How to Train Your Advisor: Steering Black-Box LLMs with Advisor Models” on OpenResearch

About this paper

Parth Asawa, Alan Zhu, Abigail O'Neill, Matei Zaharia, Alexandros G. Dimakis, Joseph E. Gonzalez

The authors introduce advisor models that generate input-specific natural-language guidance for API models, which use it to answer. An advisor learns from task rewards without access to the student’s weights or gradients, then transfers from a cheaper student to a stronger model. On RuleArena Taxes, GPT-5.2 accuracy rose from 67.2% to 85.6% on easiest Level 0 subset’s 25 test examples.

Key insights

• Parametric optimization of black-box LLMs is achievable by training a separate, lightweight advisor model to generate dynamic, instance-specific natural language advice, effectively re-framing prompt engineering as an RL problem.
• Natural language advice generated by advisors is inherently transferable, allowing advisors trained with lower-cost student models to effectively improve more expensive, frontier LLMs and generalize across different model families.
• By keeping the black-box student model's weights fixed, the ADVISOR MODELS framework inherently prevents catastrophic forgetting and maintains the student model's general capabilities on tasks for which it was not specifically optimized.

Summary by alphaXiv.