Reproduce “ParliaBench: An Evaluation and Benchmarking Framework for LLM-Generated Parliamentary Speech” on OpenResearch

About this paper

Marios Koniaris, Argyro Tsipi, Panayiotis Tsanakas

ParliaBench introduces a benchmark for testing whether large language models can generate UK parliamentary speeches that are not only fluent and coherent, but also consistent with a specified party, ideology, topic, and Westminster context. Using 447,778 speeches from 2015–2022, the study evaluates five open-weight models with standard text metrics, LLM judgments, and embedding-based measures of political alignment; fine-tuning improved 45 of 70 reported comparisons, although gains varied substantially by model and metric.

Key insights

• Political authenticity is separable from general language quality: fine-tuning can improve ideological alignment without improving relevance, authenticity, or overall judged quality on every model.
• The benchmark’s alignment metrics respond to political conditioning, with all five fine-tuned models improving Political Spectrum Alignment and four improving Party Alignment at the reported significance level of p < 0.001.
• Performance depends on data representation and category coherence: major parties such as Conservative and Labour are handled more consistently than smaller or heterogeneous affiliations, while centrist positions are easier than sparsely represented extremes.

Summary by alphaXiv.