Reproduce “Going Deeper With Memories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Afshin Khadangi
DeepMem tackles tokenizer-free language modeling by compressing semantic representations while preserving byte identity in a separate invertible carrier. At a 16-position bottleneck, it applies shared-state refinement. A 100-million-parameter model achieves 1.0134 BPB on FineWeb-Edu, versus 1.0457 for a matched byte-level Transformer. It leads on PG19 and FLORES results, but the Transformer performs better on Stack-smol, and evaluation throughput is lower.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.