Reproduce “CAPTOR: text-encoder-free grounding by counting caption co-occurrence over the visual words of a frozen encoder” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Elad Yoshai, Natan T. Shaked
CAPTOR gives a frozen image encoder a text-query interface without a text encoder by counting caption–image co-occurrence to score query terms against image patches. Its image representation is query-independent, while its term-to-visual-word table is limited to caption-supported concepts. On held-out breast-cancer patients, UNI with CAPTOR reaches 0.456 mIoU versus 0.324 for the CONCH text-encoder baseline, but this is not a same-image-encoder comparison.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.