Reproduce “BingoGuard: LLM Content Moderation Tools with Risk Levels” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fan Yin, Philippe Laban, Xiangyu Peng, Yilun Zhou, Yixin Mao, Vaibhav Vats, Linnea Ross, Divyansh Agarwal and 2 more
Salesforce AI Research, in collaboration with UCLA and Microsoft Research, introduces BingoGuard, an LLM-based content moderation system capable of classifying both the presence and the severity of harmful content. The system achieves state-of-the-art performance in binary harm detection and accurately assigns severity levels, facilitated by a novel data generation framework.
Key insights
- Explicit severity classification offers a more actionable risk assessment than relying on binary unsafe probabilities, which are often weakly correlated with actual harm.
- A novel generate-then-filter framework, using specialized LLM generators fine-tuned on small human-audited seed sets, enables scalable creation of high-quality, severity-controlled datasets.
- Incorporating severity level information during training enhances the model's overall binary content detection performance, particularly for subtle, lower-severity harmful examples.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.