Reproduce “Reasoning-Intensive Regression” on OpenResearch

About this paper

Diane Tchuindjo, Omar Khattab

MENTAT addresses reasoning-intensive regression by refining an LLM’s prompt from its errors, then training a small MLP to aggregate three predictions. On mathematical error detection with GPT-5 and 500 training-plus-validation examples, it achieves CCC .78 and NMSE .42, versus .69 and .78 for a detailed prompt. This improvement is task-specific; MENTAT does not lead every comparison across the four tasks.

Key insights

• Reasoning-Intensive Regression demands both complex analytical processing and precise continuous numerical outputs, presenting a unique challenge for current LLM-based and fine-tuning approaches.
• LLMs tend to quantize numerical outputs and can sometimes "overthink" simpler judgments, while fine-tuned Transformer encoders struggle to genuinely learn deep reasoning, often collapsing predictions.
• The Concordance Correlation Coefficient (CCC) is a more robust metric for RiR than NMSE, as it simultaneously evaluates prediction correlation and agreement, addressing issues of systematic bias and collapsed outputs.

Summary by alphaXiv.