Reproduce “Textual Inversion for Efficient Adaptation of Open-Vocabulary Object Detectors Without Forgetting” on OpenResearch

About this paper

Frank Ruis, Gertjan Burghouts, Hugo Kuijf

Researchers at TNO developed a Textual Inversion (TI) method for open-vocabulary object detectors, enabling efficient learning of new concepts from as few as three examples while preventing catastrophic forgetting of the base model's capabilities. This approach achieves fine-grained object detection and robust domain transfer, performing comparably to prompt tuning on benchmarks while preserving semantic knowledge.

Key insights

• Textual Inversion effectively prevents catastrophic forgetting, allowing new knowledge acquisition without compromising the original VLM's zero-shot and natural language capabilities.
• The success of TI for object detection hinges on specific VLM architectural properties, particularly early vision-language fusion and gradient flow through a pre-trained language backbone like BERT.
• TI enables the integration of new concepts with prior semantic knowledge, allowing the model to disambiguate fine-grained categories and transfer learned domain contexts without concept leakage.

Summary by alphaXiv.