Reproduce “AI-generated Text Detection: A Multifaceted Approach to Binary and Multiclass Classification” on OpenResearch

About this paper

Harika Abburi, Sanmitra Bhattacharya, Edward Bowen, Nirmala Pudota

This research from Deloitte explores multifaceted approaches to detecting AI-generated text and attributing it to specific Large Language Models. Their Optimized Architecture achieved an F1 score of 0.994 for binary human/AI detection, while a simpler architecture leveraging stylometric features and semantic embeddings reached an F1 score of 0.627 for the more challenging multiclass LLM attribution task.

Key insights

• Stylometric features, when combined with deep semantic embeddings, significantly enhance the performance of AI-generated text detection, achieving near-perfect F1 scores for binary classification.
• For the more complex multiclass attribution task (identifying specific LLMs), a simpler architecture combining robust semantic embeddings (E5), stylometric features, and a gradient boosting classifier can outperform more complex neural network models.
• Different architectural strategies and feature sets are optimally suited for binary human/AI detection versus fine-grained multiclass LLM attribution, indicating that a 'one-size-fits-all' approach is not always effective.

Summary by alphaXiv.