Reproduce “Jasper-Token-Compression-600M Technical Report” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dun Zhang, Ziyang Zeng, Yudong Zhou, Shuyang Lu
Jasper-Token-Compression-600M, a 0.6B parameter model, delivers performance comparable to 4B-8B parameter models on English and Chinese MTEB benchmarks by employing multi-teacher knowledge distillation and a novel 1D convolution-based token compression module. This approach reduces encoding latency for long sequences by up to 7.2x while maintaining high embedding quality across different compression rates.
Key insights
- Combining multi-teacher knowledge distillation with novel token compression and targeted contrastive learning effectively creates compact, high-performing bilingual embedding models.
- Dynamic adjustment of token compression ratios during training enables the model to learn robust representations adaptable to varying efficiency demands with minimal performance degradation.
- Leveraging complementary strengths from multiple teacher models through a sophisticated embedding combination strategy improves student model generalization across diverse tasks and languages.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.