Reproduce “F.A.C.U.L.: Language-Based Interaction with AI Companions in Gaming” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenya Wei, Sipeng Yang, Qixian Zhou, Ruochen Liu, Xuelei Zhang, Yifu Yuan, Yan Jiang, Yongle Luo and 7 more

Tencent Games' F.A.C.U.L. is a Vision-Language-Action model designed for natural language interaction with AI companions in commercial FPS games. It employs a confidence-based instruction reasoning system and multi-modal environmental perception to process player commands in real-time, achieving 87.2% overall accuracy and significantly reducing inference costs in "Arena Breakout: Infinite."

Key insights

  • A hybrid, confidence-based architecture efficiently integrates advanced LLMs into real-time gaming by routing complex commands to an LLM only when necessary, balancing performance and cost.
  • Combining text-based similarity with fine-tuned visual (CLIP) embeddings and dynamic, context-aware ranking is effective for precise object identification in dynamic 3D game environments.
  • Natural language interaction significantly enhances player experience, enjoyment, and perceived collaboration efficiency compared to traditional command interfaces in cooperative gaming.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.