← Course Overview ML2: Advanced Machine Learning
← Back to Week 8

Week 8: Convolutional Neural Networks

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Analysis

1. Why Are Convolutional Inductive Biases So Effective for Images?

The design of CNNs encodes two strong assumptions about image data: locality (nearby pixels are more related than distant ones) and translation equivariance (a feature is equally meaningful wherever it appears in the image). These inductive biases dramatically reduce the hypothesis space the model must search, enabling effective learning from relatively small datasets. However, these same biases can be limitations in other settings.

  • Fully connected networks can theoretically learn any function that a CNN can, yet CNNs consistently outperform them on image tasks when training data is limited. What does this tell us about the role of inductive biases in generalization?
  • Vision Transformers (ViT) use self-attention rather than convolutions and have achieved state-of-the-art results on image benchmarks. What inductive biases does ViT lack compared to a CNN, and how does it compensate for this (in terms of data or architecture choices)?
  • CNNs are effective for 2D images. If you were applying a CNN to 3D medical imaging data (e.g., CT scans), what architectural changes would you make, and what new challenges would arise?

Try: Ask students to list properties of image data that justify the locality assumption (e.g., a pixel is more correlated with its neighbors than with a pixel on the opposite side of the image), then brainstorm data types where locality does not hold — which opens the door to discussing when CNNs would fail.

Evaluation

2. The Residual Connection: A Simple Idea with Profound Consequences

Before ResNet, practitioners found that adding more layers to a CNN beyond a certain depth would actually hurt accuracy — not just on the test set (overfitting) but also on the training set, which was unexpected. The authors of the ResNet paper (He et al., 2016) called this the "degradation problem" and proposed skip connections as a solution. This seemingly minor architectural change enabled networks of 100+ layers and fundamentally changed the field.

  • The degradation problem showed that deeper networks can perform worse on training data — not just test data. Why does this rule out overfitting as the explanation, and what does it suggest about the optimization landscape of deep networks?
  • A residual block with a skip connection can represent the identity function by setting all learned weights to zero. Why is this property valuable, and how does it relate to the concept of a good initialization point for optimization?
  • Skip connections have appeared in many architectures beyond ResNet — U-Net for segmentation, DenseNet, and even the Transformer's residual connections. What general principle do they all share, and what might this imply about what makes deep networks hard to train?

Try: Present the class with the original training accuracy curves from the ResNet paper (plain 20-layer vs. plain 56-layer networks on CIFAR-10) and ask them to explain the result before revealing the skip-connection solution — this recreates the original puzzle the authors faced.

Application

3. Transfer Learning: When Does It Work and When Does It Fail?

Transfer learning from ImageNet-pre-trained CNNs has become the default starting point for most computer vision tasks. Instead of training from scratch, practitioners fine-tune a model that already knows how to detect edges, textures, object parts, and even entire objects. Yet transfer learning is not universally effective, and understanding its failure modes is as important as understanding its successes.

  • ImageNet contains natural photos of everyday objects. If you were training a model to classify microscopy images of cells, which layers of a pre-trained ResNet would you expect to transfer best, and which would you expect to transfer poorly? Why?
  • Domain shift refers to the statistical difference between the source training distribution and the target distribution. What are the risks of applying a fine-tuned model to data that shifts over time — for example, a medical imaging model deployed across different hospital scanner settings?
  • Some research suggests that for very large target datasets, training from scratch can eventually match or exceed fine-tuning from a pre-trained model. What does this imply about the value of the pre-training signal, and how should a practitioner decide which approach to use given their dataset size?

Try: Run a quick in-class poll asking which fine-tuning strategy students would choose given three scenarios (tiny dataset, moderate dataset, very large dataset). Use the responses to structure a small-group discussion, then reconvene to compare conclusions.

Analysis

4. Pooling vs. Strided Convolutions: Architectural Trade-offs

Max pooling has long been the standard way to perform spatial downsampling in CNNs. However, many modern architectures (e.g., ResNet variants, all-convolutional networks) replace pooling with strided convolutions that perform learned downsampling. Both approaches reduce spatial resolution, but they differ in what they preserve, what they discard, and how they are learned.

  • Max pooling is a fixed, non-learned operation that always selects the maximum value. Strided convolutions are learned operations. What is the potential advantage of learned downsampling, and under what conditions might max pooling be preferable?
  • Average pooling computes the mean of a local region while max pooling takes the maximum. In what types of tasks or input patterns would average pooling be more appropriate than max pooling, and vice versa?
  • Global average pooling (GAP) collapses an entire feature map to a single value per channel and is commonly used just before the classification layer in modern architectures. What advantages does GAP offer over a fully connected layer at the end of a CNN?

Try: Give each small group a different target task (e.g., object detection, texture recognition, fine-grained classification) and ask them to argue for max pooling, average pooling, or strided convolution as the downsampling strategy. Share reasoning in a whole-class debrief.

Synthesis/Creation

5. What Can and Cannot Be Captured by Local Receptive Fields?

A core limitation of convolutions is that each neuron in an early layer has a small receptive field — it can only "see" a local patch of the input. As the network deepens, the effective receptive field grows as local features are combined, but for shallow networks or for tasks requiring very long-range spatial reasoning, this locality constraint can be a bottleneck. This has motivated architectural innovations including dilated convolutions, self-attention in vision, and multi-scale feature pyramids.

  • Give a concrete example of an image understanding task where capturing long-range spatial relationships is critical for correct predictions, and explain why a shallow CNN with a small receptive field would struggle with it.
  • Dilated (atrous) convolutions expand the receptive field without increasing the number of parameters by inserting gaps (dilation) between filter elements. What trade-off does this introduce in terms of what the filter can detect versus what it might miss?
  • Feature Pyramid Networks (FPN) and U-Net architectures combine features from multiple scales by fusing representations from different depths of the network. What specific visual recognition problems does multi-scale feature fusion address, and how does it complement the hierarchical nature of CNN feature extraction?

Try: Display a high-resolution image on screen and ask students to physically cover part of it with their hand, then judge how much of the hidden region they can infer from context. This builds intuition for why large receptive fields matter for scene understanding tasks.

Discussion Facilitation Tips

  • Use visualizations of learned filters from early CNN layers (e.g., from AlexNet or VGGNet) to open the session. Showing that the first layer genuinely learns Gabor-like edge detectors makes the hierarchical representation story concrete and memorable before abstract discussion begins.
  • The transfer learning discussion benefits greatly from a concrete domain mismatch example. Prepare two side-by-side images — an ImageNet photo and a medical X-ray — and ask students to articulate what pixel statistics are shared and what is fundamentally different. This makes domain shift tactile rather than theoretical.
  • When discussing ResNet and the degradation problem, resist the urge to immediately explain skip connections. Let students attempt to explain why a 56-layer plain network trains worse than a 20-layer one. The productive confusion that arises when their initial explanations fail (overfitting does not explain training set degradation) creates better motivation for the skip-connection insight.
  • The pooling vs. strided convolution debate can be made more concrete by noting that modern architectures have largely moved away from max pooling for internal downsampling. Asking students why the field made this shift — and whether anything was lost — connects architectural history to present-day design decisions.