Reproduce “A Boltzmann-machine-enhanced Transformer For DNA Sequence Classification” on OpenResearch

About this paper

Zhixuan Cao, Yishu Xu, Xuang WU

A Boltzmann-machine-enhanced Transformer (BM-Transformer) is introduced for DNA sequence classification, designed to learn explicit structural interactions instead of continuous attention weights. The model maintains competitive classification accuracy while revealing interpretable latent interaction graphs and higher-order combinatorial dependencies.

Key insights

• Integrating energy-based models (like Boltzmann machines) into Transformer attention allows for explicitly modeling a prior distribution over latent interaction graphs, guiding the discovery of interpretable biological structures.
• Differentiable approximation techniques, such as mean-field inference and Gumbel-Softmax, can effectively overcome the computational challenges of training models with discrete structural components in an end-to-end manner.
• By explicitly representing pairwise interactions and higher-order dependencies through latent units, the model can capture complex biological phenomena like synergy, competition, and modular regulation within DNA sequences.

Summary by alphaXiv.