Reproduce “LEDD: Large Language Model-Empowered Data Discovery in Data Lakes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qi An, Chihua Ying, Yuqing Zhu, Yihao Xu, Manwei Zhang, Jianmin Wang
Researchers from Tsinghua University and Xiamen University developed LEDD, an end-to-end system that leverages Large Language Models to automate the generation of hierarchical semantic data catalogs and provide natural language search capabilities for data lakes. The system was demonstrated to organize 1.7 million tables into a navigable hierarchy and identify relations between data assets in real-time, facilitating data discovery.
Key insights
- Large Language Models can effectively bridge the gap between human semantic understanding and the automated organization and discovery of data within complex data lake environments.
- An automatically generated, hierarchical, and semantically rich global catalog significantly improves data usability and comprehension, surpassing flat or scattered data organization methods.
- An extensible architecture, implemented through User-Defined Functions, is crucial for integrating and adapting various data discovery algorithms as LLM and data management technologies evolve.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.