Reproduce “Ordo-M: An Externally Addressed Sparse Memory Grafted onto a Frozen Language Model” on OpenResearch

About this paper

Russel Gavery

The Ordo-M system introduces a sparse, externally addressed memory module grafted onto frozen language models, enabling localized, auditable knowledge updates without modifying base model weights. This approach demonstrates zero drift of existing records during incremental learning and reduces unintended text damage by 96-97% through a parameter-free gate, making LLM augmentation feasible on consumer-grade GPUs.

Key insights

• A frozen network can effectively learn to read and utilize a grafted memory module, achieving "zero collateral damage" to other stored knowledge or base model integrity when shared parameters on the read path are frozen.
• The memory module functions as a "steering vector" by adding a fixed vector per entity broadcast over a span, rather than a sparse retriever, which impacts its effective capacity.
• The trade-off between knowledge efficacy and general text damage can be substantially mitigated by an answer-scoped loss combined with a parameter-free positional gate, reducing damage by 96-97%.

Summary by alphaXiv.