Reproduce “Learning concise representations for regression by evolving networks of trees” on OpenResearch

About this paper

William La Cava, Tilak Raj Singh, James Taggart, Srinivas Suri, Jason H. Moore

The University of Pennsylvania developed FEAT, a method that learns concise and interpretable representations for regression by evolving networks of trees that incorporate neural network activation functions. This approach achieves predictive performance comparable to XGBoost while yielding models two to four orders of magnitude less complex than other non-linear methods.

Key insights

• Evolving networks of trees with integrated neural network components and gradient-based training can produce highly accurate regression models that are also significantly more interpretable.
• Guiding the evolutionary search using feedback from linear model coefficients (e.g., mutating features with smaller absolute coefficients) effectively steers the process toward more parsimonious and effective model structures.
• Simply adding explicit disentanglement objectives, such as minimizing feature correlation or the condition number, does not guarantee less correlated feature representations in the final selected models.

Summary by alphaXiv.