Reproduce “MLLMs Fail to Refuse when Using Tools Agentically” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Rikiya Takehi, Ryo Hachiuma, Shaona Ghosh, Dan Zhao, Yu-Chiang Frank Wang, Yusuke Hirota
The authors compare no-tool and tool-using multimodal language models on paired harmful image-query requests across three safety benchmarks. They define refusal failure rate (RFR) as the rate of harmful requests not refused. Tool use raised RFR for every model on every benchmark, averaging 5.7 percentage points and 17.7% relative. The result is limited to the tested models, prompts, and benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.