Reproduce “Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lorenz Hufe, Constantin Venhoff, Erblina Purelku, Maximilian Dreyer, Sebastian Lapuschkin, Wojciech Samek
Dyslexify identifies sparse attention heads that route image text to CLIP’s classification token, then suppresses their contributions without fine-tuning. It ranks heads using a typographic attention score. On synthetic ImageNet-100-Typo, OpenCLIP ViT-G accuracy reaches 68.76%, up 22.06 percentage points; on non-typographic ImageNet-100, it falls 0.66 points. These results concern CLS-based classification, not downstream systems using spatial tokens.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.