← Course Overview ML2: Advanced Machine Learning
← Back to Week 3

Week 3: Building a Real-World Housing Price Predictor

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Evaluation

1. When Deep Learning Is and Is Not the Right Tool

This week applies deep learning to the California Housing dataset — a structured tabular regression problem. However, deep learning is not always the best approach for tabular data. Classical methods like gradient-boosted trees (XGBoost, LightGBM) frequently outperform neural networks on structured data with hundreds or thousands of rows. Understanding when to use each approach is a crucial practical skill.

  • What properties of tabular data make classical ensemble methods competitive with neural networks, even at moderate data scales?
  • At what scale of data (in terms of rows and features) do neural networks typically begin to outperform gradient-boosted trees, and why?
  • In a real project, how would you decide which approach to try first — and what would your evaluation criteria be?

Try: Run both an XGBoost baseline and a neural network on the California Housing dataset live in class, compare their R-squared scores side by side, and ask students to predict which will win before revealing the results.

Synthesis/Creation

2. The Ethics of Algorithmic Housing Valuation

Automated Valuation Models (AVMs) — algorithmic systems that estimate property values — are now widely used in mortgage lending, property tax assessment, and real estate investment. The California Housing dataset includes geographic features like latitude and longitude that correlate strongly with historical patterns of racial and economic segregation. A model trained on this data could perpetuate or amplify historical inequities.

  • How might a housing price model trained on historical data encode and perpetuate discriminatory patterns, even if race is not an explicit input feature?
  • If you were asked to deploy a housing price model for a lending institution, what fairness audits would you require before approving it for use?
  • Is there a tension between model accuracy and fairness in this domain? Can a model be optimized for both simultaneously, or must trade-offs be made?

Try: Ask small groups to design a fairness audit checklist for a hypothetical AVM deployment, then compare the checklists across groups to surface different conceptions of fairness (individual vs. group, procedural vs. outcome-based).

Analysis

3. Overfitting, Regularization, and the Bias-Variance Trade-off

L2 regularization (weight decay), dropout, and early stopping are three distinct strategies for combating overfitting. Each operates through a different mechanism: L2 penalizes weight magnitudes, dropout stochastically disables neurons during training, and early stopping terminates training before the model memorizes the training set. Understanding these mechanisms helps practitioners choose the right tool for a given situation.

  • L2 regularization and dropout both reduce overfitting — but through fundamentally different mechanisms. In what settings might one be preferable over the other?
  • Early stopping relies on the validation loss as a stopping criterion. What happens if the validation set is not representative of the true deployment distribution?
  • The bias-variance trade-off suggests that reducing variance (overfitting) often increases bias (underfitting). How do practitioners navigate this trade-off in practice, and what does "good enough" look like?

Try: Train the same network with each of the three regularization strategies separately, plot all three validation curves on one figure, and invite students to compare where each strategy kicks in and at what cost to training performance.

Evaluation

4. Evaluation Metrics and What They Obscure

MSE, MAE, and R² are the standard metrics for regression, but each tells a different story about model quality. Practitioners often report one metric and move on, but the choice of metric can significantly influence what a model is optimized for and who benefits or is harmed by its predictions. In high-stakes domains, the choice of evaluation metric is as important as the choice of model.

  • A housing price model with a low MSE might still perform poorly for homes in certain price ranges. What disaggregated evaluation strategies would help identify these failure modes?
  • R² can appear high even when a model fails on a meaningful subpopulation. How would you design an evaluation protocol that goes beyond a single aggregate metric?
  • In what real-world scenarios is optimizing for MSE the wrong objective, and what alternative objectives (e.g., minimizing worst-case error) might be more appropriate?

Try: Compute MSE, MAE, and R-squared separately for different price quintiles of the California Housing dataset, display the disaggregated results, and ask students which quintile the model is most and least reliable for.

Application

5. Iterative Model Improvement as a Scientific Process

The iterative improvement process described in this week — tuning the learning rate, adjusting architecture depth and width, applying regularization, conducting error analysis — resembles the scientific method: form a hypothesis, run an experiment, observe results, update your understanding. However, without discipline, this process can devolve into random tinkering that produces overly optimistic results through implicit data leakage.

  • What practices help maintain the integrity of the evaluation process when iterating on a model many times — especially to avoid inadvertently tuning to the test set?
  • Hyperparameter optimization methods (grid search, random search, Bayesian optimization) can automate much of the iteration process. What are the risks and benefits of delegating this to automation?
  • When should a practitioner stop iterating and declare a model "good enough"? What external factors (business constraints, deployment deadlines, risk tolerance) should influence this decision?

Try: Present a scenario where a student has run 50 experiments using the validation set to select the best model — ask the class to vote on whether that final validation score is still a trustworthy estimate, then discuss implicit leakage and the importance of a held-out test set.

Discussion Facilitation Tips

  • Anchor the "right tool" discussion in concrete numbers: bring published benchmarks comparing XGBoost and neural networks on tabular datasets so students argue from evidence rather than intuition alone.
  • For the ethics discussion, acknowledge upfront that the California Housing dataset is a teaching artifact and that real AVMs involve far more complex data pipelines — this helps students generalize the ethical reasoning rather than treating it as a single dataset problem.
  • When discussing evaluation metrics, use the phrase "a metric is a contract" to help students see that choosing MSE commits you to a specific definition of model quality, and that contract has downstream social consequences in high-stakes applications.
  • For the iterative improvement topic, draw a timeline on the board mapping when each change was made and which dataset was consulted, making implicit data leakage visible and concrete before discussing how to prevent it.