Reproduce “Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gautam Veldanda
The authors compare roughly 60,000-parameter architectures for byte-level TinyStories V2 under two training budgets. At 16 million bytes, full-precision transformer validation loss ranges from 2.835 to 3.477 nats per byte, making the apparent routed-model advantage depend on baseline shape. At 130 million bytes, a gated recurrence performs better in full precision and matches it in ternary; these findings do not establish deployment performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.