Reproduce “PedagoSense: A Pedology Grounded LLM System for Pedagogical Strategy Detection and Contextual Response Generation in Learning Dialogues” on OpenResearch

About this paper

Shahem Sultan, Shahem Fadi, Yousef Melhim, Ibrahim Alsarraj, Besher Hassan

The PedagoSense system, or "Tutor Copilot," integrates NLP and LLMs to detect pedagogical strategies within learning dialogues and generate contextually appropriate tutor responses. This system demonstrated strong performance, achieving an F1-score of 98.5% for binary strategy detection and an F1-score of 0.4815 for strategy recommendation through a hybrid probabilistic voting method.

Key insights

• GPT-4o-based data augmentation is crucial for improving model performance across various tasks, particularly for models like fine-tuned BERT, by expanding the training data and balancing class distribution in domain-specific datasets.
• Ensemble approaches, such as the Hybrid-BERT Probabilistic Voting method, effectively combine complementary strengths of different algorithms (BERT, LPD, BES) to achieve superior performance in pedagogical strategy recommendation.
• Transformer-based models outperform traditional methods in fine-grained classification, though class imbalance and subtle distinctions between certain pedagogical strategies continue to present challenges for accurate identification.

Summary by alphaXiv.