Reproduce “Contrastive Learning for Knowledge-Based Question Generation in Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhenhong Zhang, Jiajing Chen, Weiyan Shi, Lingjie Yi, Chihang Wang, Qian Yu

Researchers developed a method for knowledge-based question generation that leverages Large Language Models (LLMs) with contrastive learning and Chain-of-Thought (CoT) prompting. This approach successfully reduces hallucinations and noise, leading to 98.80% accuracy for high-quality question generation and enhancing downstream QA model performance to 74.61% accuracy on the TriviaQA dataset.

Key insights

  • Combining explicit task instructions with carefully selected contrasting positive and negative examples in prompts dramatically improves LLM-based question generation quality, achieving nearly perfect results.
  • Contrastive learning serves as an effective mechanism for mitigating common LLM limitations like hallucinations and noise by guiding the models to learn from desirable and undesirable output characteristics.
  • Automated quality evaluation using Chain-of-Thought prompting offers a scalable and robust method for screening LLM outputs and continuously refining the quality of generated data through a dynamic feedback loop.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.