Reproduce “Preemptive Detection and Correction of Misaligned Actions in LLM Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haishuo Fang, Xiaodan Zhu, Iryna Gurevych
InferAct uses an LLM to infer an agent’s goal from its actions and compare it with the user’s request at designated high-stakes actions. It alerts a person when they diverge, leaving intervention to the person. On ALFWorld with Llama-3-70B, it achieves Macro-F1 of 0.827 versus 0.767 for Multi-step, but evaluations use clear instructions and predefined consequential actions.
Key insights
- Leveraging LLMs' belief reasoning (Theory-of-Mind) is effective for interpreting agent intent and detecting misalignment in a proactive manner.
- A third-person perspective in evaluating an agent's actions helps to mitigate self-serving biases, offering a more objective assessment of intent.
- Selective intervention on critical actions, combined with human feedback, allows for a balanced approach to agent autonomy and safety, significantly reducing the human oversight burden.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.