Reproduce “Towards Reliable Detection of LLM-Generated Texts: A Comprehensive Evaluation Framework with CUDRT” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhen Tao, Yanfang Chen, Dinghao Xi, Zhiyu Li, Wei Xu
This work introduces CUDRT, a comprehensive, bilingual evaluation framework and benchmark for detecting LLM-generated texts, categorizing LLM operations into Create, Update, Delete, Rewrite, and Translate. Experiments with CUDRT demonstrate that detection performance varies significantly across different LLM operations and languages, with 'Delete', 'Update', and 'Rewrite' operations proving particularly effective for training robust detectors.
Key insights
- Training model-based detectors on text generated from 'Delete', 'Update', and 'Rewrite' operations significantly improves their generalization capabilities across various LLM activities.
- Detection performance varies considerably across different LLMs, with some models (e.g., GPT, Llama series) producing more easily detectable patterns than others (e.g., Baichuan2), highlighting the need for diverse training data.
- Significant differences in detection effectiveness between Chinese and English texts, and across operations within each language, underscore the necessity of multilingual and language-specific evaluation frameworks for robust detection.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.