Reproduce “MCEL: Margin-Based Cross-Entropy Loss for Error-Tolerant Quantized Neural Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mikail Yayla, Akash Kumar
MCEL introduces a margin-based cross-entropy loss function to enhance neural network robustness against hardware-induced bit errors by explicitly optimizing output-layer classification margins, achieving improved bit error tolerance without requiring training-time error injection. Models trained with MCEL consistently show higher accuracy under bit errors, for instance, providing up to 15.32% accuracy improvement at 1% bit error rate for 4-bit quantized FashionMNIST networks.
Key insights
- Neural network bit error tolerance is directly correlated with the magnitude of classification margins at the output layer.
- Standard loss functions like hinge loss and cross-entropy are not optimally suited for explicitly maximizing these margins in quantized neural networks (QNNs).
- Combining a tunable margin parameter with smooth logit clamping enables effective and principled enforcement of relative logit separation (RLS) within a differentiable loss function.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.