Reproduce “Assessing Personalized AI Mentoring with Large Language Models in the Computing Field” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xiao Luo, Sean O'Connell, Shamima Mithun
Researchers from Oklahoma State and Purdue Universities assessed state-of-the-art Large Language Models for personalized career mentoring in computing, evaluating their ability to tailor advice based on students' social and educational backgrounds. Their study, utilizing both NLP analysis and human expert evaluation, found GPT-4o and LLaMA 3 demonstrated superior personalization, especially in acknowledging social identities, with GPT-4o performing best overall.
Key insights
- Current LLMs, especially GPT-4o and LLaMA 3, can personalize mentoring advice, demonstrating an ability to incorporate students' social and educational backgrounds into their responses.
- While LLMs excel at providing general advice on competency development and networking, they show limitations in assisting with specific goal-setting without interactive dialogue.
- Human oversight remains crucial for truly effective and nuanced personalized mentoring, suggesting future AI systems will likely augment rather than replace human mentors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.