Reproduce “Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Bin Ren, Yahui Liu, Yue Song, Wei Bi, Rita Cucchiara, Nicu Sebe, Wei Wang
The Masked Jigsaw Puzzle (MJP) method introduces a versatile position embedding for Vision Transformers, effectively addressing privacy leakage from gradient inversion attacks while simultaneously boosting model accuracy, consistency, and robustness. This approach achieves substantial privacy preservation, improved performance on various vision tasks, and enhanced resilience to corruptions and adversarial examples with minimal computational cost.
Key insights
- Position Embeddings (PEs) in Vision Transformers explicitly learn and encode 2D spatial relationships, which is a root cause of privacy leakage under gradient inversion attacks.
- By strategically introducing ambiguity for shuffled patches (`E_unk`) and reinforcing explicit localization for unshuffled ones (DAL), ViTs can simultaneously enhance privacy, robustness, and consistency without sacrificing accuracy.
- The MJP method effectively navigates the inherent trade-off between a ViT's reliance on explicit positional information for accuracy and the associated privacy risks and consistency issues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.