Reproduce “Fractional Concepts in Neural Networks: Enhancing Activation Functions” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zahra Alijani, Vojtech Molek

Integrating fractional calculus into neural network activation functions offers a more flexible design space, as demonstrated by the University of Ostrava research. A fractional Sigmoid variant consistently improved performance over its conventional counterpart across diverse image classification datasets and architectures, while other fractional functions faced challenges with computational overhead and training stability.

Key insights

  • Fractional order derivatives provide a mechanism for activation functions to adapt their shape dynamically throughout training, offering increased flexibility beyond fixed mathematical forms.
  • The computational and memory complexity of fractional activation functions increase linearly with the `N` parameter, which controls the number of summation terms in the Grünwald-Letnikov approximation.
  • Fractional activation functions can introduce more complex loss landscapes characterized by 'narrow local minima,' potentially complicating optimization and affecting model generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.