Reproduce “LMEB: Long-horizon Memory Embedding Benchmark” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xinping Zhao, Xinshuo Hu, Jiaxin Xu, Danyu Tang, Xin Zhang, Mengjia Zhou, Yan Zhong, Yao Zhou and 4 more

LMEB evaluates memory-retrieval embedding models across 22 English datasets and 193 zero-shot tasks spanning episodic, dialogue, semantic, and procedural memory. It tests 30 models against MTEB English-v2 retrieval. Overall correlations are weak and not statistically significant: Pearson r = 0.267 and Spearman ρ = 0.261, indicating these scores are not interchangeable here, without proving retrieval ability never transfers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.