Reproduce “Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mateusz Michalkiewicz, Sheena Bai, Mahsa Baktashmotlagh, Varun Jampani, Guha Balakrishnan
This research explores and quantifies viewpoint instability in nine vision foundation models, showing they struggle with maintaining consistent feature representations across different viewing angles. It identifies "accidental viewpoints" as a major source of this instability, which is predictable from feature embeddings and significantly reduces performance on downstream tasks such as classification, visual question answering, and 3D reconstruction by up to 25%.
Key insights
- All nine evaluated foundation models exhibit viewpoint instability, with "accidental viewpoints" (visual ambiguities) being a consistent source of instability across models.
- Unstable viewpoints can be categorized into "accidental" and "out-of-distribution" (uncommon viewing angles), with models showing more agreement on accidental viewpoints.
- Viewpoint instability is predictable directly from a single image's feature representation, with SVM classifiers achieving 71-96% accuracy on prediction tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.