Reproduce “PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohammad Rifqi Farhansyah, Hanif Muhammad Zhafran, Farid Adilazuarda, Shamsuddeen Hassan Muhammad, Maryam Ibrahim Mukhtar, Nedjma Ousidhoum, Genta Indra Winata, Ayu Purwarianti and 1 more
Researchers introduce PINGPONG, a new benchmark designed to evaluate language models on natural, multi-party, code-switched dialogues across five diverse language combinations. The evaluation reveals existing model limitations in understanding and generating complex code-switched conversations, particularly in generative and semantic tasks.
Key insights
- Human-authored dialogues in PINGPONG exhibit significantly greater naturalness and structural complexity, including varied message lengths and multi-threaded reply structures, compared to machine-generated conversations.
- Explicit reasoning mechanisms, whether native to the model or introduced via prompting, consistently improve model performance across the benchmark's tasks.
- Few-shot learning did not consistently improve performance for Question Answering and Topic Classification, but it yielded notable improvements for Dialogue Summarization across evaluated models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.