Reproduce “ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillation” on OpenResearch

About this paper

Linhan Cao, Siyuan Li, Jun Lan, Liangbo He, Guannan Li, Xiaolei Huang, Jun Jia, Shuheng Zhou and 3 more

Researchers at Ant Group, Shanghai Jiao Tong University, and East China Normal University present AdvSpot, a benchmark for grounded adversarial OCR perception, and ArmorOCR, a two-stage training framework. ArmorOCR significantly improves LMMs' ability to perceive adversarial text, achieving a 24.5% VQA accuracy gain on AdvSpot while preserving general OCR capabilities, by internalizing robust perception for efficient single-pass inference.

Key insights

• Grounded adversarial OCR, requiring both precise text localization and recognition, offers a more granular and informative evaluation of LMM failures compared to recognition-only benchmarks.
• Internalizing visual transformation-derived knowledge directly into the model during training (via OTSD) provides a more efficient and effective strategy for robust adversarial perception than applying transformations at inference time.
• A multi-objective, reward-driven refinement process (GRPO) is crucial for systematically enhancing a model's ability to handle the multifaceted challenges of grounded adversarial text perception.

Summary by alphaXiv.