← Course Overview ML2: Advanced Machine Learning
← Back to Week 2

Week 2: Neural Networks & Backpropagation

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Evaluation

1. The Learning Rate Dilemma

The learning rate η is arguably the most consequential hyperparameter in gradient-based training. Too small and training is agonizingly slow and prone to getting stuck; too large and the optimization diverges. Practitioners often spend as much time tuning the learning rate as designing the model architecture itself. Techniques like learning rate warmup, decay schedules, and adaptive methods (Adam, RMSProp) attempt to automate this challenge.

  • Why does the optimal learning rate depend on both the model architecture and the dataset, making it impossible to specify a universal default?
  • Adaptive optimizers like Adam maintain per-parameter learning rates. What is the trade-off between using Adam and vanilla SGD in terms of generalization performance?
  • Learning rate warmup starts training with a very small rate before increasing it. Why might jumping to the target rate immediately be harmful at the start of training?

Try: Show a live training curve where the learning rate is too large (diverging loss) and one where it is too small (barely decreasing loss), then ask students to guess a better value before revealing the well-tuned result.

Analysis

2. Loss Functions as Inductive Biases

The choice of loss function encodes assumptions about what kinds of errors matter and how they should be penalized. Mean Squared Error penalizes large errors quadratically, making it sensitive to outliers. Cross-entropy is derived from information theory and is natural for probabilistic classification. Selecting the wrong loss function can lead a model to learn the right pattern in entirely the wrong way.

  • If you are predicting house prices and your dataset contains several luxury outlier properties, should you prefer MSE or MAE? What does this choice say about what you value?
  • In a medical diagnosis task with severe class imbalance (1% positive cases), standard cross-entropy may train a model that ignores the minority class. What modifications to the loss function address this?
  • Can you design a custom loss function for a specific business problem? What considerations guide that design?

Try: Present two loss curves (MSE vs. MAE) on the same dataset with outliers included, and ask groups to predict which metric their model would report as "better" and why before checking the actual values.

Analysis

3. Understanding the Loss Landscape

The loss landscape of a deep neural network is a high-dimensional surface where each point represents a set of parameter values and the height represents the loss. Gradient descent navigates this landscape seeking a minimum. Research has shown that this landscape has many saddle points (where the gradient is zero but the point is not a minimum) and that local minima in overparameterized networks tend to have similar loss values to global minima.

  • If many local minima give similarly good generalization performance, does it still matter that gradient descent does not find the global minimum?
  • The mini-batch noise in SGD is often cited as helping escape saddle points. Is this the whole story, or do other factors also contribute?
  • How does increasing model size (more parameters) qualitatively change the structure of the loss landscape, and what are the implications for optimization?

Try: Use a low-dimensional loss surface visualization tool (such as the loss-landscape library) to project a trained model onto 2D and let students identify valleys, saddle points, and ridges before discussing what gradient descent would do.

Evaluation

4. Backpropagation: Elegant Algorithm or Black Box?

Backpropagation is often described as a beautiful application of the chain rule, enabling the efficient computation of gradients in networks with millions of parameters. At the same time, it is the engine behind models whose internal reasoning is opaque. Critics argue that gradient-based optimization produces networks that work but cannot be interrogated, while proponents argue that the outcomes speak for themselves.

  • Backpropagation was known in the 1970s but was not widely adopted until the 1980s-1990s. What factors explain this delay, and what does it tell us about how scientific ideas diffuse?
  • Gradient-based training optimizes a single scalar loss — but real-world objectives are rarely single-dimensional. How do practitioners handle multi-objective trade-offs (e.g., accuracy vs. fairness vs. speed)?
  • Are there classes of problems where backpropagation is fundamentally unsuitable, and what alternative learning algorithms exist for those cases?

Try: Ask students to manually compute gradients for a tiny two-layer network with toy numbers, working through the chain rule step by step on paper, to build intuition before the algorithm becomes a black box.

Synthesis/Creation

5. Batch Size as a Design Decision

The choice of batch size in mini-batch gradient descent is rarely discussed as a principled design decision, yet it has significant consequences. Large batches provide accurate gradient estimates but use more memory, train more slowly per update (though fewer updates are needed), and have been shown empirically to generalize worse than small batches in some settings. This "generalization gap" between large and small batch training is an active area of research.

  • What is the "sharp minima vs. flat minima" hypothesis, and why would small-batch training favor flat minima that generalize better?
  • Distributed training across many GPUs effectively increases the batch size — what strategies have researchers proposed to recover the generalization benefits of small batches in distributed settings?
  • Is there a principled way to choose batch size for a new problem, or is it always empirically determined through experimentation?

Try: Assign small groups different batch sizes (8, 64, 512) for the same simple problem, run them live in a notebook, then compare final validation accuracy and training time to ground the theoretical discussion in observed data.

Discussion Facilitation Tips

  • When discussing backpropagation, resist moving too quickly to the matrix formulation; students benefit from tracing the chain rule through a concrete three-node toy network before generalizing to the full algorithm.
  • The loss landscape discussion can feel abstract; anchor it with a published visualization (e.g., Li et al. 2018 "Visualizing the Loss Landscape of Neural Nets") to give students a concrete mental image before asking conceptual questions.
  • For the batch size topic, help students distinguish between what is mathematically well-understood (gradient variance decreases with larger batches) and what is empirically observed but not fully explained (the generalization gap), to model good scientific epistemic hygiene.
  • The learning rate discussion pairs well with a live demo: use a simple 1D quadratic loss and animate gradient descent steps at different learning rates so students can see overshooting and slow convergence before abstracting to higher dimensions.