Reproduce “Lightweight Neural App Control” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, Kun Shao
Huawei Noah’s Ark Lab developed Lightweight Multi-modal App Control (LiMAC), an efficient architecture for smartphone app agents that combines a compact transformer with a selectively invoked, fine-tuned Vision-Language Model. This framework achieved up to 42% higher action prediction accuracy and was 30 times faster than GPT-4o-based agents, demonstrating robust performance on mobile control tasks while operating efficiently on resource-constrained devices.
Key insights
- A modular, gated architecture can balance efficiency and advanced capabilities, allowing a lightweight model to handle most tasks while selectively engaging a VLM for computationally intensive natural language generation.
- Integrating visual and textual UI element embeddings into a compact transformer, coupled with fine-tuning, is crucial for robust performance, especially when UI tree quality is inconsistent.
- A contrastive learning objective for click prediction significantly enhances accuracy by effectively distinguishing target UI elements from all other screen elements.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.