Reproduce “Convolutional Tensor-Train LSTM for Spatio-temporal Learning” on OpenResearch

About this paper

Jiahao Su, Wonmin Byeon, Jean Kossaifi, Furong Huang, Jan Kautz, Animashree Anandkumar

NVIDIA Research and University of Maryland introduced Conv-TT-LSTM, a higher-order Convolutional LSTM, employing a novel convolutional tensor-train decomposition to efficiently model long-term spatio-temporal correlations. The model achieves state-of-the-art performance in multi-frame video prediction and early activity recognition, demonstrating superior perceptual quality and accuracy with significantly reduced parameters.

Key insights

• Efficiently modeling long-term spatio-temporal dependencies in recurrent networks is achievable by integrating higher-order states through a specialized convolutional tensor-train decomposition.
• The CTTD approach effectively resolves the challenge of exponential complexity in higher-order convolutional models, allowing parameter and FLOPs reduction while maintaining crucial properties like spatial structure and growing receptive fields.
• The inherent low-rank factorization in the tensor-train structure acts as an implicit regularizer, promoting the learning of more compact and generalizable representations for video data.

Summary by alphaXiv.