MIT 6.7960 L06: Modern CNN Architectures — ResNet, EfficientNet, ConvNeXt

🌏 中文版

Material Version: Based on MIT 6.7960 Fall 2024 OCW. Videos, slides, assignments fully public on MIT OCW.

TL;R

ResNet skip connections solve degradation; EfficientNet compound scales; ConvNeXt absorbs Transformer design to reclaim CV crown.

Background / Course Context

Lecture 6 (L06), corresponding to the official schedule for September 23 (estimated). L05 established basic CNN; this lecture enters modern architectures, exploring "how to stack networks deeper, more efficient, more accurate" — directly affecting which backbone you choose when training your own models.

Video: Lecture 6: Modern CNN (start ~2:30:00) Slides: Lecture 6: Modern CNN

Core Concept Comparison

Method Architecture

1. ResNet Residual Connection

x → [Conv-BN-ReLU] → [Conv-BN] → + (skip x) → ReLU
                              ↑
                          identity shortcut

• Learns residual $F(x) = H(x) - x$, optimizing "increment" instead of "absolute mapping"
• Degradation problem: plain nets deepen → training error rises; ResNet lets worst case degrade to identity (copy previous layer)

2. Depthwise Separable Convolution

Standard conv:  C_in × C_out × k × k params
Separable:      (C_in × k × k) + (C_in × C_out × 1 × 1) params
               compute reduced ~ k² × C_out / (1 + k²)

• MobileNet: 3×3 depthwise + 1×1 pointwise
• Example: 64 in-channels, 128 out, kernel 3×3
  • Standard: $64 \times 128 \times 9 = 73728$
  • Separable: $64 \times 9 + 64 \times 128 = 8768$ (8.4× savings)

3. EfficientNet Compound Scaling

$$\text{depth}: d=\alpha^\phi, \quad \text{width}: w=\beta^\phi, \quad \text{resolution}: r=\gamma^\phi$$ Constraint $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$ (fixed total compute)

• B0→B7: $\phi$ from 0 to 7, systematically scaled
• Key: scaling any single dimension alone is suboptimal; compound scaling is optimal

4. ConvNeXt Design Choices

Implementation Insights

PyTorch Implementation: ResNet Residual Block

import torch
import torch.nn as nn
import torch.nn.functional as F

class ResidualBlock(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_ch, out_ch, 3, stride, 1, bias=False)
        self.bn1 = nn.BatchNorm2d(out_ch)
        self.conv2 = nn.Conv2d(out_ch, out_ch, 3, 1, 1, bias=False)
        self.bn2 = nn.BatchNorm2d(out_ch)
        
        # shortcut: 1×1 conv to align dims (when in_ch ≠ out_ch or stride>1)
        self.shortcut = nn.Sequential()
        if in_ch != out_ch or stride != 1:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_ch, out_ch, 1, stride, bias=False),
                nn.BatchNorm2d(out_ch)
            )
    
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)   # residual connection
        return F.relu(out)

# Stack ResNet-18 style
class ResNet18(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 64, 7, 2, 3, bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        self.pool = nn.MaxPool2d(3, 2, 1)
        self.layer1 = self._make_layer(64, 64, 2)
        self.layer2 = self._make_layer(64, 128, 2, stride=2)
        self.layer3 = self._make_layer(128, 256, 2, stride=2)
        self.layer4 = self._make_layer(256, 512, 2, stride=2)
        self.gap = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Linear(512, num_classes)
    
    def _make_layer(self, in_ch, out_ch, blocks, stride=1):
        layers = [ResidualBlock(in_ch, out_ch, stride)]
        for _ in range(1, blocks):
            layers.append(ResidualBlock(out_ch, out_ch))
        return nn.Sequential(*layers)
    
    def forward(self, x):
        x = self.pool(F.relu(self.bn1(self.conv1(x))))
        x = self.layer1(x); x = self.layer2(x)
        x = self.layer3(x); x = self.layer4(x)
        x = self.gap(x).flatten(1)
        return self.fc(x)

Depthwise Separable Convolution

class DepthwiseSeparableConv(nn.Module):
    def __init__(self, in_ch, out_ch, kernel=3):
        super().__init__()
        self.depthwise = nn.Conv2d(in_ch, in_ch, kernel, padding=kernel//2, groups=in_ch, bias=False)
        self.pointwise = nn.Conv2d(in_ch, out_ch, 1, bias=False)
        self.bn = nn.BatchNorm2d(out_ch)
    
    def forward(self, x):
        return F.relu(self.bn(self.pointwise(self.depthwise(x))))

Use Pretrained Models (Recommended in Practice)

import torchvision.models as models

# ResNet-50 pretrained (ImageNet weights)
resnet = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
# EfficientNet-B0
effnet = models.efficientnet_b0(weights=models.EfficientNet_B0_Weights.IMAGENET1K_V1)
# ConvNeXt-Tiny
convnext = models.convnext_tiny(weights=models.ConvNeXt_Tiny_Weights.IMAGENET1K_V1)

Critical Reflection

Course Emphasis: Choose backbone by three factors — (1) data volume (small data → CNN/ConvNeXt, large data → try ViT); (2) deployment constraints (mobile → MobileNet/EfficientNet); (3) accuracy needs (extreme → EfficientNetV2/ConvNeXt-XXL). Don't blindly chase novelty; match constraints.

References

• Lecture 6: Modern CNN (MIT OCW 6.7960 Fall 2024) — Slides PDF
• Lecture 6 Video (YouTube) — Key segments: 0:00–15:00 ResNet; 15:00–30:00 MobileNet/EfficientNet; 30:00–45:00 ConvNeXt
• He et al., "Deep Residual Learning for Image Recognition" (CVPR 2016) — ResNet
• Howard et al., "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications" (2017)
• Tan & Le, "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks" (ICML 2019)
• Liu et al., "A ConvNet for the 2020s" (CVPR 2022) — ConvNeXt
• PyTorch Official: torchvision.models