Reproduce “Semi-Autoregressive Image Captioning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xu Yan, Zhengcong Fei, Zekang Li, Shuhui Wang, Qingming Huang, Qi Tian
Researchers from the Chinese Academy of Sciences introduced Semi-Autoregressive Image Captioning (SAIC), a two-stage framework that generates captions by first outlining key words autoregressively and then filling in the gaps non-autoregressively. This approach achieves a superior balance of decoding speed and caption quality, outperforming non-autoregressive models and offering up to a 3.42x speedup over autoregressive models while maintaining competitive performance.
Key insights
- Providing strategically chosen, uniformly distributed partial prior knowledge can drastically reduce the number of iterations required for high-quality non-autoregressive generation.
- A hierarchical decoding approach, leveraging autoregressive generation for structural outlines and non-autoregressive generation for parallel gap-filling, balances speed and caption quality.
- Integrating competence-aware curriculum learning and hybrid knowledge distillation is effective for training models that combine both sequential and parallel generation paradigms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.