← Course Overview ML2: Advanced Machine Learning
← Back to Week 5

Week 5: Autoencoders & Embeddings

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Evaluation

1. Autoencoders for Anomaly Detection: Promise and Peril

Credit card fraud detection systems have been built using autoencoders trained on "normal" transactions. The assumption is that fraudulent transactions will have high reconstruction error because they differ from the training distribution. However, sophisticated fraudsters may learn to mimic normal transaction patterns, and rare-but-legitimate transactions (e.g., a customer's first international purchase) may also appear anomalous.

  • How would you design an autoencoder-based fraud detection system to minimize false positives (blocking legitimate transactions) while still catching fraud?
  • What does the reconstruction error threshold represent as a business decision? Who should set it and using what criteria?
  • If an autoencoder is continuously retrained on recent transactions, how might adversarial fraudsters gradually shift what the model considers "normal"?

Try: Present students with three transaction profiles — one clearly fraudulent, one clearly normal, and one ambiguous — and have small groups set a reconstruction error threshold and justify the business trade-off their threshold implies.

Analysis

2. The VAE Latent Space: Generative Power vs. Reconstruction Fidelity

The beta-VAE framework adjusts the weight on the KL divergence term. A high beta value produces a more structured, disentangled latent space suitable for generation and interpolation, but typically at the cost of lower reconstruction quality. A low beta collapses toward a standard autoencoder with sharper reconstructions but a less regular latent space.

  • For which downstream applications would you prioritize reconstruction fidelity, and for which would you prioritize latent space structure?
  • Is there a principled way to choose the beta hyperparameter, or is it always a task-specific judgment call?
  • The tradeoff between compression and reconstruction quality is a fundamental tension in information theory. How does this connect to Shannon's rate-distortion theory?

Try: Show side-by-side image reconstructions from a beta-VAE trained with a low and a high beta on the same dataset and run a live poll asking students which result they would prefer for a medical imaging task versus a creative generation task.

Evaluation

3. Denoising Autoencoders and the Definition of "Signal" vs. "Noise"

Denoising autoencoders are trained to remove corruption from inputs. In medical imaging, this has been used to enhance low-dose CT scans by training a network to recover high-dose-quality images from noisy low-dose inputs. However, critics point out that aggressive denoising can hallucinate structures that were not present in the original scan.

  • Who defines what counts as "noise" versus "signal" in a clinical context, and what happens when a model's learned definition of noise conflicts with clinical reality?
  • How should we evaluate a denoising model in a medical setting — what metrics are appropriate, and are there metrics that could be dangerously misleading?
  • What regulatory or ethical frameworks should govern the deployment of AI denoising in medical imaging pipelines?

Try: Show students a pair of images (denoised vs. original noisy scan) and ask them to identify which structures they would trust — then reveal whether any structures were hallucinated to spark discussion about evaluation standards.

Analysis

4. Autoencoders vs. PCA: When Does Nonlinearity Actually Matter?

Undercomplete autoencoders with linear activations and MSE loss are mathematically equivalent to PCA — they learn the same subspace. Only when nonlinear activations are introduced does the autoencoder diverge meaningfully. In practice, many practitioners default to deep autoencoders without asking whether the added complexity is justified by the data.

  • What properties of a dataset would signal that the data lies on a nonlinear manifold, making a nonlinear autoencoder clearly preferable to PCA?
  • If a deep autoencoder only marginally outperforms PCA on a reconstruction benchmark, should you still use the autoencoder? What other factors matter?
  • In what scientific domains might interpretability of PCA components outweigh the representational power of a neural autoencoder?

Try: Have students train both a linear autoencoder and PCA on the same toy dataset in a shared notebook and compare their reconstructed outputs — seeing identical results live makes the mathematical equivalence concrete.

Synthesis/Creation

5. Latent Space Interpolation and the Ethics of Generative Control

VAEs trained on face images allow smooth interpolation through the latent space, generating photorealistic intermediate faces. Researchers have demonstrated that individual latent dimensions can independently control attributes like perceived age, skin tone, or facial expression. These capabilities have legitimate research uses but have also been used to create manipulated imagery.

  • Should the ability to controllably manipulate demographic attributes in generated images be considered a feature or a vulnerability? Does the answer change depending on the use case?
  • If a VAE latent space entangles attributes in unexpected ways — for instance, if changing "apparent age" also changes "perceived gender" — what does that reveal about the training data?
  • What technical or policy safeguards, if any, could reasonably limit misuse of generative latent spaces without preventing legitimate research?

Try: Ask students to work in small groups to draft a one-paragraph acceptable-use policy for a generative face VAE tool — then compare the policies and discuss where they converge and where they reflect genuine value disagreements.

Discussion Facilitation Tips

  • When introducing the encoder-decoder architecture, draw the information bottleneck on the board and ask students to predict what the network will be forced to learn before revealing how autoencoders work. This activates prior knowledge about compression and feature selection.
  • The VAE reparameterization trick is conceptually subtle — use a concrete analogy: instead of sampling directly (which blocks gradients), we sample noise separately and shift it by the learned mean and scale. Drawing this computation graph on the board helps students see exactly where the gradient flows.
  • For the ethics topics (topics 1, 3, and 5), consider assigning roles before discussion — one student argues from a developer perspective, one from a regulator perspective, and one from an affected-user perspective. Role assignment reduces the tendency to converge on a single comfortable answer.
  • Close the session by returning to the question of what "good" reconstruction means and who gets to decide — this threads through all five topics and helps students see autoencoders not just as neural architectures but as systems that encode assumptions about what is important in data.