Week 9: From Supervised to Generative Learning

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Evaluation

1. The Annotation Bottleneck: Is Supervised Learning Fundamentally Constrained?

Supervised learning has driven remarkable progress in computer vision, NLP, and speech recognition, but it depends on labeled datasets whose construction is costly, slow, and in specialized domains often impossible to scale. The textbook frames this as a fundamental epistemological constraint — the information-theoretic demands of the task grow faster than our ability to provide labeled examples. Generative and self-supervised approaches have emerged partly as engineering solutions to this problem.

In what domains do you think the annotation bottleneck is most severe and least likely to be solved by crowdsourcing or automation? What properties of those domains create the bottleneck?
Self-supervised learning creates its own supervisory signal from unlabeled data. In what sense is this "free" supervision, and in what sense does the task design (e.g., which tokens to mask, which augmentations to use) encode human knowledge about the problem?
Some researchers argue that large-scale self-supervised pre-training followed by minimal fine-tuning is converging on a "foundation model" paradigm that will replace task-specific supervised training. What are the strongest arguments for and against this view?

Try: Ask each student to estimate how long it would take a layperson to annotate 1000 images for general object classification, then ask the same question for tumor segmentation. Use the gap in their estimates to open a discussion on what makes expert annotation irreplaceable.

Analysis

2. Diffusion Models vs. GANs: Two Paths to Generative Modeling

Generative Adversarial Networks (GANs) dominated image generation for several years and produced remarkably realistic outputs, but they are notoriously difficult to train due to mode collapse, training instability, and the adversarial minimax objective. Diffusion models have largely displaced GANs in image synthesis benchmarks, offering more stable training and better sample diversity. Understanding why diffusion models succeeded where GANs struggled reveals fundamental trade-offs in generative model design.

GANs train a generator and a discriminator in an adversarial game. Diffusion models train a single denoising network using a simple regression objective. What practical training challenges arise from adversarial objectives that are avoided by the diffusion approach?
GANs can generate samples in a single forward pass, while diffusion models require hundreds or thousands of sequential denoising steps at inference time. How significant is this computational cost in practice, and what recent techniques (e.g., DDIM, consistency models) attempt to address it?
Mode collapse in GANs means the generator learns to produce only a narrow subset of the true data distribution. How does the diffusion model's training objective — which minimizes reconstruction error at every noise level — help ensure better coverage of the data distribution?

Try: Show a side-by-side grid of GAN-generated faces (StyleGAN2) and diffusion-generated images (Stable Diffusion) without labels and ask students to identify differences in diversity, artifacts, and failure modes before revealing which is which.

Knowledge/Comprehension

3. What Does "Learning the Data Distribution" Actually Mean?

Generative models are described as learning the underlying data distribution P(X). But what does it mean for a model to have learned a distribution? A diffusion model never writes down an explicit probability for any input — it only knows how to denoise. A GAN cannot evaluate the likelihood of a given sample. An autoregressive model can compute exact likelihoods but generates samples differently from how it computes them. The gap between the intuitive description and the mathematical reality is worth examining carefully.

What does it mean for a diffusion model to have "learned" P(X) if it cannot assign an explicit probability to an arbitrary input image? In what practical sense does it capture the data distribution?
A generative model could achieve a low reconstruction loss on training data by simply memorizing examples. How would you distinguish between a model that has genuinely learned the data distribution and one that has memorized training examples? What experiments would help?
Evaluation of generative models is notoriously difficult. Metrics like FID (Frechet Inception Distance) measure distributional similarity between real and generated samples. What are the limitations of using FID as the primary evaluation metric for a generative model?

Try: Present two generated images — one clearly a memorized training sample and one novel interpolation — without identifying which is which, and ask students to design a test that could distinguish them. Then discuss what those tests reveal about what "generalization" means for generative models.

Synthesis/Creation

4. Self-Supervised Learning: Designing the Right Pretext Task

Self-supervised learning works by defining a "pretext task" — a task the model can be trained on without labels, where solving the task forces the model to learn useful representations. Masked token prediction (BERT), contrastive learning (SimCLR, MoCo), image patch prediction (MAE), and denoising (diffusion models) are all pretext tasks. The choice of pretext task fundamentally determines what the resulting representations capture and what downstream tasks they will support well.

BERT masks random tokens and predicts them from context. This forces the model to understand language semantics and syntax. What would happen if instead you trained BERT to predict only the next character (rather than masked words from context)? Would the representations be better or worse for downstream NLP tasks?
In contrastive learning, "positive pairs" are defined by data augmentation (e.g., two crops of the same image). The choice of augmentation encodes a strong assumption: the model should be invariant to whatever transformations define the positive pair. What if the augmentation strategy is wrong for your downstream task?
The textbook notes that representations learned through self-supervised approaches often exceed what can be achieved through supervised learning. Why might learning to reconstruct or denoise data produce richer internal representations than learning to classify it?

Try: Divide the class into groups and assign each a domain (audio, video, protein sequences, time-series sensor data). Ask each group to design a self-supervised pretext task for their domain and present their rationale. Discuss what assumptions each task encodes about the structure of that data type.

Evaluation

5. Cross-Modal Generation and the Limits of Generative Models

One of the most exciting capabilities of modern diffusion models is cross-modal generation: given a text description, generate a photorealistic image (text-to-image); given an image, generate a description (image captioning); or translate between modalities in ways that were previously impossible. Systems like Stable Diffusion and DALL-E demonstrate that generative models can learn deep correspondences between language and visual content. But these same capabilities raise important questions about what the models actually understand versus what they have statistically associated.

Text-to-image models generate images consistent with a prompt by conditioning the diffusion process on text embeddings. Does the model "understand" the text, or is it performing sophisticated pattern matching between text and visual features seen during training? How would you design an experiment to distinguish between these two hypotheses?
Diffusion models allow fine-grained control over the generation process through techniques like classifier-free guidance. This lets users interpolate between a guided and unguided generation. What does it mean for a model to be "guided" by a text prompt at the level of the reverse diffusion process?
Generative models trained on internet-scale image-text pairs inevitably learn and amplify biases present in that data (stereotypes, demographic imbalances, cultural assumptions). How should practitioners think about the relationship between scale, capability, and risk when deploying cross-modal generative systems?

Try: Have students generate images using a public text-to-image tool with a deliberately ambiguous or compositionally complex prompt (e.g., "a red ball to the left of a blue cube on top of a green box"), then analyze where the model fails to follow instructions and discuss what this reveals about its compositional understanding.

Discussion Facilitation Tips

The discriminative vs. generative distinction is abstract until grounded in examples. Begin by asking students what a classifier predicts versus what a generative model produces, then formalize the P(Y|X) vs. P(X,Y) distinction using their intuitions as a foundation.
The GAN vs. diffusion model comparison is a rich topic, but students often focus on output quality rather than training dynamics. Redirect discussions toward the training stability question: ask whether they have ever seen a minimax optimization in another context and what made it difficult, which connects to the adversarial instability problem in GANs.
When discussing self-supervised pretext tasks, emphasize that the choice of task is a form of inductive bias just as architectural choices are. Students who understand this can reason about why a task designed for images (contrastive crops) would transfer poorly to text, building cross-topic synthesis.
The cross-modal generation discussion can become philosophical quickly. Keep it grounded by requiring students to propose a specific, falsifiable experiment that would help answer the "understanding vs. pattern matching" question — this prevents the conversation from drifting into unproductive abstraction.