Reproduce “Predicting Through Generation: Why Generation Is Better for Prediction” on OpenResearch

About this paper

Md Kowsher, Nusrat Jahan Prottasha, Prakash Bhat, Chun-Nam Yu, Mojtaba Soltanalian, Ivan Garibay, Ozlem Garibay, Chen Chen and 1 more

This paper from the University of Central Florida and collaborators proposes PredGen, an end-to-end framework that reformulates supervised prediction tasks for Large Language Models as token-level generation. The work demonstrates, both theoretically through the Data Processing Inequality and empirically, that this generation-based approach preserves more mutual information and consistently outperforms traditional pooling-based methods across diverse classification, regression, and arithmetic reasoning benchmarks.

Key insights

• Applying pooling operations to LLM hidden states for predictive tasks causes an irreversible loss of mutual information with the target output, as theoretically shown by the Data Processing Inequality.
• Fine-tuning LLMs by reformulating predictive tasks as token-level generation aligns more naturally with their pre-training objective, allowing for better leveraging of learned contextual information.
• Addressing challenges like exposure bias through scheduled sampling and format mismatch via a task adapter is crucial for robust generation-based prediction.

Summary by alphaXiv.