Reproduce “Test-Time Canonicalization by Foundation Models for Robust Perception” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Utkarsh Singhal, Ryan Feng, Stella X. Yu, Atul Prakash

FOCAL, a training-free framework, leverages pre-trained foundation models to perform test-time canonicalization, enhancing the robustness of perception systems. The method dynamically transforms inputs into their "most typical views" using energy functions derived from CLIP and Stable Diffusion, achieving significant improvements in classification and segmentation accuracy across diverse 2D and 3D transformations.

Key insights

  • Robust invariance to complex transformations can be achieved adaptively at inference time by canonicalizing inputs to their most typical views, drawing inspiration from human mental rotation.
  • Pre-trained large foundation models implicitly encode powerful visual priors about natural images, which can be effectively extracted and utilized as training-free, data-driven energy functions to guide canonicalization.
  • Combining semantic priors (from CLIP) with appearance-based priors (from Stable Diffusion) provides a comprehensive and generalizable criterion for visual typicality, applicable across a broad spectrum of transformations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.