Reproduce “ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jingnan Zheng, Han Wang, An Zhang, Tai D. Nguyen, Jun Sun, Tat-Seng Chua
ALI-Agent is an agent-based framework designed to autonomously generate and refine test scenarios for assessing Large Language Models' (LLMs) alignment with human values. The framework successfully reveals more subtle misalignment cases in LLMs than existing benchmarks, including long-tail risks, and highlights that model scaling or fine-tuning can sometimes degrade alignment performance.
Key insights
- Model scaling, such as from Llama 2-7B to Llama 2-70B, does not automatically lead to improved alignment and can sometimes result in worse performance.
- Fine-tuning base LLMs (e.g., Llama 2 into Vicuna) can compromise their alignment with human values, underscoring the need for robust safety protocols during custom fine-tuning.
- Morality aspects represent the most severe misalignment challenge for LLMs across various models, compared to stereotypes and legality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.