Reproduce “ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhongpan Tang
ProxyFormer introduces a dual-stream proxy architecture that enables efficient processing of ultra-long contexts and high-resolution generation by performing global interactions in a compressed proxy space while preserving fine-grained information. It dramatically reduces VRAM consumption and increases training throughput, allowing for 0.7M token history on a 16 GB GPU and maintaining high retrieval accuracy over contexts up to 1M tokens.
Key insights
- Information fidelity can be decoupled from the cost of global interaction by performing global computations in a compressed 'proxy space' while retaining a full-fidelity 'local stream' via residual connections.
- Iterative, layer-wise compression, global interaction, and decompression with residual information flow effectively mitigate the irreversible information loss often seen in prior one-shot compression methods.
- The modular design, which allows for pluggable global interaction functions (e.g., attention, SSMs) and uniform application across 1D, 2D, and 3D data, offers a generalizable solution for diverse modalities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.