Reproduce “Bias Beneath the Tone: Empirical Characterisation of Tone Bias in LLM-Driven UX Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Heet Bodara, Md Masum Mushfiq, Isma Farah Siddiqui
Researchers at Monash University empirically characterized intrinsic tonal biases in LLM-driven UX systems using a diagnostic pipeline. The study revealed that LLMs exhibit pervasive tonal skew even when prompted for neutral responses, and ensemble models effectively detected these biases with Macro-F1 scores up to 0.92.
Key insights
- LLMs exhibit pervasive, inherent tonal skew, often defaulting to polite or positive framings even when explicitly prompted for neutral responses, indicating a fundamental stylistic bias.
- The effectiveness of detecting subtle versus pronounced biases is significantly influenced by the confidence threshold used for weak sentiment labeling, allowing for targeted analysis.
- Ensemble models and neural networks can systematically classify LLM-induced tones, with neural models showing particular aptitude for nuanced and contrastive tones like sarcasm or hedging.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.