Reproduce “AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection” on OpenResearch

About this paper

Saleh Momeni, Changnan Xiao, Bing Liu

Researchers from the University of Illinois Chicago developed AnaCP, an analytic method for continual feature adaptation in Class-Incremental Learning (CIL). This approach, which bypasses gradient-based training, achieved state-of-the-art accuracy comparable to joint training, delivered a relative error reduction between 4.8% and 31.4% in "Last Accuracy" over best baselines, and mitigated catastrophic forgetting with high computational efficiency, completing core training for CIFAR100 in 5 seconds.

Key insights

• Continual feature adaptation in CIL can be achieved analytically through a contrastive projection mechanism that explicitly optimizes both intra-class compactness and inter-class separability.
• Combining a frozen PTM with a series of fixed random projections and an analytically derived contrastive projection layer can effectively prevent catastrophic forgetting while adapting representations for new tasks.
• By employing a pseudo-replay strategy with an ELM classifier, it's possible to maintain state-of-the-art performance, approaching the joint training upper bound, with exceptional computational efficiency and minimal memory overhead for prior task data.

Summary by alphaXiv.