Reproduce “JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yebin Yang, Huaijin Wu, Fu Guo, Lin Yao, Xiaohan Qin, Jingzhi Wang, Debing Zhang, Junchi Yan

Researchers at Shanghai Jiao Tong University and Hi Lab, Xiaohongshu Inc. developed JTok and JTok-M, which introduce token-indexed parameters as a novel scaling axis for large language models, injecting substantial capacity with minimal computational cost. This approach consistently reduces training loss and improves downstream task performance, yielding up to a 35% compute saving compared to traditional scaling, while maintaining low operational overheads.

Key insights

  • Token-indexed parameters present an orthogonal and complementary scaling dimension, allowing for substantial model capacity expansion without proportional increases in computational cost.
  • Token-specific modulation can be efficiently integrated into both dense and Mixture-of-Experts (MoE) Transformer architectures, serving as a plug-in to universally enhance performance.
  • The capacity introduced by token-indexed parameters adheres to predictable power-law scaling, providing a stable and systematic path for future model development and optimization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.