Reproduce “Evaluating Pre-Trained Models for User Feedback Analysis in Software Engineering: A Study on Classification of App-Reviews” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohammad Abdul Hadi, Fatemeh H. Fard
This study evaluates pre-trained transformer models (PTMs) for classifying user feedback from app reviews in software engineering, demonstrating their superior accuracy compared to traditional methods. Custom PTMs, specifically further pre-trained on domain-specific app review data, consistently achieve improved performance in various real-world scenarios including multi-class, zero-shot, multi-task, and multi-resource classification, often with competitive efficiency.
Key insights
- PTMs consistently surpass traditional deep learning and machine learning baselines in accuracy for app review classification, with ALBERT and RoBERTa demonstrating the highest performance among the evaluated models.
- Incorporating domain-specific app review data during further pre-training (Custom PTMs) yields consistent and often substantial accuracy improvements over general-purpose PTMs.
- PTMs, particularly customized ones, show strong adaptability across complex classification settings (multi-task, multi-resource, zero-shot), which reduces the dependency on extensive manual labeling for new categories or data sources.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.