Reproduce “Perception of Knowledge Boundary for Large Language Models through Semi-open-ended Question Answering” on OpenResearch

About this paper

Zhihua Wen, Zhiliang Tian, Zexin Jian, Zhen Huang, Pei Ke, Yifu Gao, Minlie Huang, Dongsheng Li

Researchers at National University of Defense Technology and Tsinghua University introduce a framework to perceive Large Language Models' (LLMs) knowledge boundaries using semi-open-ended questions and a novel ambiguous answer discovery method. Their work reveals that advanced LLMs like GPT-4 frequently generate unqualified responses and demonstrate inaccurate self-evaluation when pushed to the limits of their knowledge, often failing to recognize their own limitations.

Key insights

• Semi-open-ended questions are particularly effective for probing the fine-grained knowledge boundaries of LLMs, revealing areas where models struggle with long-tail information or produce misinformation.
• Even state-of-the-art LLMs like GPT-4 exhibit significant overconfidence and a lack of meta-awareness regarding their knowledge boundaries, frequently generating incorrect or unverifiable answers without admitting uncertainty.
• Leveraging an auxiliary model with a carefully designed probability reduction technique can effectively uncover 'hidden correct answers' and 'unexpected wrong evaluations,' which highlight the target LLM's blind spots.

Summary by alphaXiv.