Reproduce “Schrödinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Miloš Nikolić, Enrique Torres Sanchez, Jiahui Wang, Ali Hadi Zadeh, Mostafa Mahmoud, Ameer Abdelhadi, Kareem Ibrahim, Andreas Moshovos
Researchers at the University of Toronto developed 'Schrödinger’s FP,' a framework that dynamically adjusts mantissa and exponent bitlengths for neural network activations and weights during training. This automated approach reduces the memory footprint by up to 5.64x on average compared to FP32, with minimal impact on accuracy, outperforming static mixed-precision methods like FP8 and BFloat16.
Key insights
- Optimal floating-point bitlengths for mantissas and exponents are highly dynamic, varying significantly per tensor and evolving throughout the training process, validating the need for adaptive rather than static selection.
- Tailored compression for mantissas (precision) and exponents (range) is crucial, as their statistical distributions and numerical roles differ, allowing for more efficient bitlength allocation.
- Automating datatype discovery directly within the training loop, via differentiable quantization or loss observation, eliminates manual effort and enables robust, fine-grained precision tuning without user intervention.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.