Reproduce “RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiarui Wang, Huichao Chai, Yuanhang Zhang, Zongjin Zhou, Wei Guo, Xingkun Yang, Qiang Tang, Bo Pan and 33 more
RelayGR provides a multi-stage inference system that enables Generative Recommendation models to process significantly longer user-behavior sequences in real-time production environments without violating strict latency budgets. It achieves this by pre-inferring item-independent user-behavior prefixes, leading to up to 1.5 times longer supported sequences and 3.6 times higher SLO-compliant throughput.
Key insights
- A substantial portion of GR model computation involves encoding user behaviors that are independent of candidate items, forming a stable prefix suitable for early pre-inference and caching.
- Ensuring cache survivability across multi-stage pipelines and local consumption without remote fetches is critical for meeting strict P99 latency SLOs, particularly with late-binding placement decisions.
- Selective pre-inference and careful memory management (HBM for lifecycle, DRAM for reuse) are essential to operate safely under high Query Per Second (QPS) and bounded resource contention in industrial recommender systems.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.