Reproduce “RoleFUZZER: User-Behaviour Fuzzing for Role-Specification Adherence in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kshitish Ghate, Michael Saxon, Tadayoshi Kohno, Aylin Caliskan
RoleFUZZER tests whether language models follow role-specific principles under plausible multi-turn user pressure. It uses a 4-by-4 behavior grid per principle, tracks coverage, and directs scenarios toward under-tested cells. Across 15 campaigns involving five roles and three models, it covered over 89% of the 64-cell grid, versus 39% to 69% for free-form generation; coverage does not measure real-world behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.