Reproduce “The Disappearance of Timestep Embedding in Modern Time-Dependent Neural Networks” on OpenResearch

About this paper

Bum Jun Kim, Yoshinobu Kawahara, Sang Woo Kim

Researchers identified a critical vulnerability where timestep embedding information vanishes in Neural Ordinary Differential Equations (NODEs) and diffusion models due to normalization layers neutralizing scalar offsets. The study proposes three solutions, including a positional timestep embedding, which collectively improve model performance on image classification tasks and enhance the quality of generated data.

Key insights

• Normalization layers like Batch Normalization and Group Normalization can effectively cancel out the influence of timestep embeddings if they act as uniform scalar offsets across spatial dimensions.
• The 'ConcatConv' operation, common in Neural Ordinary Differential Equations (NODEs) and diffusion models, contributes to the vanishing timestep embedding problem by introducing time information as a channel-wise scalar.
• Careful architectural design, particularly concerning how timestep embeddings are integrated and normalized, is crucial for preserving the temporal awareness of time-dependent neural networks.

Summary by alphaXiv.