2021 was the year Transformers decisively entered computer vision: Swin Transformer won the ICCV Best Paper award, DINO showed that a self-supervised ViT could learn object segmentation without labels, and NeRF grew from one paper into an entire subfield. CVPR and ICCV both moved fully online because of the pandemic, yet the work published that year shaped architectural choices across computer vision for years to come.
2021 was the year diffusion models surpassed GANs, self-supervised learning made theoretical breakthroughs, and reinforcement learning confronted weaknesses in its evaluation methodology. NeurIPS received a then-record 9,122 submissions, ICLR’s Score-Based Generative Modeling paper became a theoretical foundation for the diffusion ecosystem, and ICML delivered substantial work on optimization theory and the dynamics of self-supervised learning.
2021 marked NLP’s shift from fine-tuning an entire model to adapting only a small fraction of its parameters. Prefix-Tuning at ACL, LoRA on arXiv, and Prompt Tuning at EMNLP all appeared that year; ACL Rolling Review launched; and the Findings track established itself as a second publication channel.
2021 was a dividing line for major AI conferences. Transformers spread from NLP throughout computer vision and time-series research, self-supervised learning became the most common cross-conference theme, and a diffusion model won an ICLR Outstanding Paper award before anyone realized it would displace GANs. Meanwhile, GNNs and federated learning reached historic peaks in paper volume before beginning to decline.