Reproduce “Safety as a Constraint: Fine-Tuning a LLM Recommender to Explain Itself” on OpenResearch

About this paper

Jiashu He, Emma Yanyang Kong, JJ Tan, David Fagnan

A production recommender LLM is fine-tuned to generate short, personalized explanations for its own recommendations while treating bias and offensiveness as explicit safety constraints. Constrained GRPO raises the combined faithfulness-and-harmlessness pass rate from 0.649 to 0.956 under in-house judges and from 0.677 to 0.931 under an independent evaluator, with no observed degradation in retrieval or language benchmarks.

Key insights

• Joint per-sample safety constraints outperform separate batch-level constraints because aggregate scores can hide explanations that fail different safety criteria.
• Domain-specific reward models capture subtle harmful framing more effectively than a general frontier-model judge, particularly when safety depends on recommendation context rather than explicit insults.
• A warm-up stage and constrained post-training can teach the recommender to explain its own outputs while preserving its original retrieval behavior, avoiding a separate explanation-generation model.

Summary by alphaXiv.