Reproduce “Token Weighting for Long-Range Language Modeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Falko Helm, Nico Daheim, Iryna Gurevych
Research from the UKP Lab at TU Darmstadt explores non-uniform token weighting in the loss function to improve large language models' understanding of long contexts. Their two-step framework for scoring and postprocessing tokens demonstrates consistent performance gains on long-context benchmarks like RULER and Longbench compared to uniform weighting, while also investigating trade-offs with short-context abilities.
Key insights
- Non-uniform token weighting in the loss function consistently improves LLMs' long-context understanding, indicating that not all tokens contribute equally to learning long-range dependencies.
- The proposed scoring function, based on the absolute difference in negative log-likelihoods (CPMI), effectively identifies tokens crucial for long-range dependencies and learnable tokens.
- Perplexity on the validation set does not correlate well with actual long-range reasoning capabilities, suggesting it is not an effective proxy for evaluating long-context models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.