Reproduce “LAuReL: Learned Augmented Residual Layer” on OpenResearch

About this paper

Gaurav Menghani, Ravi Kumar, Sanjiv Kumar

LAuReL modifies the skip path in deep networks with learned weights and a low-rank linear adjustment. In a ResNet-50 ImageNet-1K test, it matched added-layer accuracy, 75.20%, with 1.68% rather than 4.37% additional parameters. In a 4B language model, BookQA rose from 50.36 to 60.46, while latency increased 1–2%. These results concern specific comparisons, not faster training or inference.

Key insights

• Augmenting the foundational residual connection with lightweight learned components (scalars or low-rank matrices) can significantly improve model quality and efficiency.
• Learning adaptive linear transformations within the residual stream allows the primary non-linear components of the network to focus better on their tasks, leading to improved overall learning capacity and performance.
• LAUREL offers a more Pareto-efficient way to scale deep learning models, achieving better quality for a given (or even reduced) computational and memory footprint compared to simply adding more layers or parameters.

Summary by alphaXiv.