Reproduce “Context-Aware Pragmatic Metacognitive Prompting for Sarcasm Detection” on OpenResearch

About this paper

Michael Iskandardinata, William Christian, Derwin Suhartono

Researchers from Bina Nusantara University developed a "Context-Aware Pragmatic Metacognitive Prompting" method to enhance sarcasm detection in Large Language Models (LLMs) by integrating external and internal contextual knowledge. The approach improved macro-F1 scores on a challenging Indonesian Twitter dataset from 0.5173 to 0.6160, demonstrating increased performance for culturally nuanced language.

Key insights

• Providing dynamic, real-world knowledge from external sources is crucial for enhancing LLM performance in interpreting evolving and culturally specific language, as demonstrated with the Indonesian Twitter dataset.
• Eliciting an LLM's internal knowledge and incorporating few-shot examples can effectively improve its reasoning and contextual awareness, even for domains where it is expected to have existing parametric knowledge.
• The effectiveness of context augmentation strategies is highly dependent on the specific LLM's pre-training, multilingual capabilities, and fine-tuning, with clear performance differences observed between models like Qwen-3-8B and Llama-3-8B.

Summary by alphaXiv.