Reproduce “HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ji Ha Jang, Hayeon Kim, Chulwon Lee, Junghun James Kim, Se Young Chun
HyFL-CLIP adapts pretrained Euclidean CLIP models to long, structured captions by combining hyperbolic representation learning, cross-manifold distillation, and hierarchical entailment. Extending the context window to 248 tokens, it improves long-caption retrieval and remains more stable under word dropout, sentence deletion, reordering, and partial-caption inputs than several Euclidean long-context baselines, while largely preserving short-caption alignment.
Key insights
- Hyperbolic geometry provides a tolerance region for semantic alignment, allowing perturbed or incomplete captions to remain related to the same image instead of requiring exact embedding coincidence.
- Combining long-context learning with distillation from the original CLIP geometry helps improve robustness without substantially degrading conventional short-caption retrieval.
- The hierarchical objective is especially useful when visual information is distributed across multiple sentences or represented by object-level attributes, relations, and actions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.