Reproduce “FAVE: Foveated Adaptive Visual Encoding for Efficient Fine-Grained Visual Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amitangshu Mukherjee, Kaushik Roy
FAVE re-encodes externally selected image regions at variable resolution while preserving native geometry, complementing a global representation that retains scene context. External mechanisms provide proposals; FAVE does not select regions. On small-object ImageNet classification with oracle boxes, it reaches 50.45% Top-1 versus 41.05% for fixed-resolution ViT-T/16, using 12.7× lower analytical FLOPs. It also improves TextVQA and GQA attribute-question accuracy.
Key insights
- Allocating extra visual capacity to selected regions is more effective than distributing the same increase uniformly across the whole image; local magnification improves FAVE accuracy, whereas higher-resolution global processing does not improve the adaptive baseline.
- Object-centric training is important: generic variable-resolution models such as an adaptive full-image ViT and NaViT do not match FAVE on small-object recognition, even when evaluated with the same crop geometry.
- Targeted spatial capacity can outperform backbone scaling at a much lower cost, indicating that preserving localized evidence may be more valuable than simply increasing model size.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.