Reproduce “Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Donghoon Kim, Minji Bae, Kyuhong Shim, Byonghyo Shim

Visually Guided Decoding (VGD) introduces a gradient-free hard prompt inversion method that combines Large Language Models (LLMs) with CLIP-based visual guidance to generate interpretable and coherent text prompts from images. This approach consistently achieves high visual fidelity and prompt editability, while being significantly faster than previous training-based techniques.

Key insights

  • Effectively combining a pre-trained LLM's linguistic coherence with a CLIP model's visual semantic alignment, without requiring further training, is highly effective for generating interpretable prompts.
  • The reformulated objective, which includes the LLM's text prior as a regularization term, is crucial for ensuring the linguistic naturalness and coherence of the generated prompts.
  • A gradient-free, token-by-token beam search decoding approach provides a highly efficient and flexible mechanism for hard prompt inversion, overcoming the complexities of gradient-based methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.