Reproduce “The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lukas Gienapp, Christopher Schröder, Stefan Schweter, Christopher Akiki, Ferdinand Schlatt, Arden Zimmermann, Phillipe Genêt, Martin Potthast

The German Commons provides the largest collection of verifiably openly licensed German text to date, comprising 154.56 billion tokens from 35.78 million documents, processed to ensure high quality and legal compliance for training German language models.

Key insights

  • Building large-scale, ethically compliant datasets for LLMs is feasible through systematic sourcing from established institutional providers and strict adherence to explicit open licensing.
  • A comprehensive data processing pipeline is essential to transform heterogeneous raw inputs into a high-quality, clean corpus, effectively mitigating noise and ensuring usability for LLM training.
  • Rigorous PII removal and detailed content analysis (e.g., toxicity, sentiment) are critical for addressing legal and ethical challenges, thereby fostering a more transparent and trustworthy AI ecosystem, especially for non-English languages.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.