← Course Overview ML2: Advanced Machine Learning
← Back to Week 1

Week 1: Introduction to Deep Learning

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Analysis

1. The Feature Engineering Trade-off

Traditional machine learning required data scientists to manually engineer features — carefully crafting representations like edge detectors or bag-of-words encodings before a model ever sees the data. Deep learning's core promise is that it can learn these representations automatically from raw data. However, this shift comes with its own costs and assumptions.

  • In what domains might handcrafted features still outperform automatically learned ones, and why?
  • What does "end-to-end learning" mean in practice, and what are its failure modes?
  • How does the scale of available data change the trade-off between manual feature engineering and learned representations?

Try: Ask students to brainstorm one domain where they believe handcrafted features would beat learned features, then hold a quick class vote and debate the top two contenders.

Evaluation

2. The Deep Learning Hardware Revolution

The lecture notes that three factors converged to enable the deep learning revolution: large-scale datasets, powerful GPU hardware, and open-source frameworks. GPUs were originally designed for rendering graphics, yet they became the backbone of deep learning training. This was not an obvious outcome at the time.

  • Why are GPUs particularly well-suited to the matrix operations used in neural network training?
  • How might the next generation of hardware (TPUs, neuromorphic chips) change what kinds of models are practical?
  • What are the environmental and accessibility implications of a field that depends on large-scale computational infrastructure?

Try: Show a live energy-cost calculator for training a large model (e.g., using published figures from papers on GPT training), then ask small groups to discuss who benefits and who bears the cost of this infrastructure.

Application

3. Choosing an Activation Function

The choice of activation function has significant consequences for training dynamics. The sigmoid function dominated early neural network research, but ReLU (f(x) = max(0, x)) became the de facto standard for hidden layers in modern networks. This seemingly simple change had profound practical effects on training deep models.

  • Why does sigmoid cause gradients to vanish in deep networks, and how does ReLU address this?
  • ReLU "kills" neurons whose pre-activation is always negative — the "dying ReLU" problem. What variants (Leaky ReLU, ELU, GELU) were proposed to address this, and what are their trade-offs?
  • Should the output layer always use a different activation than hidden layers? What determines the right choice?

Try: Draw the sigmoid and ReLU curves side by side on the board, then ask students to trace the derivative through three layers by hand to viscerally experience gradient shrinkage with sigmoid.

Evaluation

4. PyTorch vs. TensorFlow: Research vs. Production

The dominance of PyTorch in academic research and TensorFlow in production deployment reflects deeper philosophical differences between the two frameworks — dynamic versus static computation graphs, eager versus deferred execution. With TensorFlow 2.0 adopting eager execution by default, the lines have blurred, yet the two communities remain distinct.

  • What specific aspects of dynamic computation graphs make PyTorch more natural for research, and what did this cost in early production deployments?
  • As a practitioner, at what point in a project's lifecycle would you consider switching from a research framework to a production-focused one?
  • What risks arise when research code written in one framework must be redeployed or reproduced using another?

Try: Pose a scenario — "You need to ship a model to a mobile app in 6 weeks. Which framework do you start with and why?" — and run a quick poll before opening the floor to debate.

Synthesis/Creation

5. Machine Learning Paradigms and Their Ethical Assumptions

Supervised learning requires labeled data, unsupervised learning looks for patterns without labels, and reinforcement learning learns from environmental feedback. Each paradigm embeds implicit assumptions about what counts as "correct" behavior and who decides it. These assumptions have significant ethical implications when models are deployed at scale.

  • In supervised learning, labels reflect the judgments of whoever created them — how does labeling bias propagate into a deployed model?
  • Unsupervised clustering can reinforce existing social categories (e.g., demographic groupings). How should practitioners decide when discovered clusters are meaningful vs. spurious?
  • Reinforcement learning optimizes a reward signal — what happens when the reward is misspecified? Can you think of real-world examples where reward misspecification led to unintended behavior?

Try: Split students into three groups, one per paradigm, and ask each group to design a labeling or reward scheme for a hiring tool, then compare the embedded assumptions across groups.

Discussion Facilitation Tips

  • Ground abstract concepts early: before discussing deep learning's advantages, ask students what they already know about traditional ML pipelines so you can build on prior knowledge rather than starting from scratch.
  • Use the feature engineering topic to establish a recurring theme for the course — the tension between automation and control — which will resurface in regularization, architecture design, and AutoML discussions.
  • When discussing GPUs and compute costs, acknowledge that students have varying access to hardware; emphasize free-tier cloud options (Google Colab, Kaggle) so the conversation stays grounded and inclusive.
  • For the ethics discussion, model intellectual humility by acknowledging genuine uncertainty — there are no clean answers to reward misspecification or labeling bias, and framing this as an open research area helps students engage critically rather than seeking a "right" answer.