Linear Regression

Fundamentals of Supervised Learning

📈 Core Concepts

Understand linear relationships and mathematical foundations

🎯 Model Training

Learn ordinary least squares optimization

📊 Evaluation

Master R-squared, MSE, and error interpretation

🔍 Applications

Recognize when to use linear regression

What is Linear Regression?

Linear regression models the relationship between variables using a straight line that minimizes prediction errors.

Key Properties

Supervised learning algorithm
Predicts continuous values
Assumes linear relationships
Provides interpretable coefficients
Fast training and prediction

Applications: House prices, sales forecasting, medical outcomes, financial modeling

Visualization: Scatter plot with regression line

Simple Linear Regression

Start with the simplest case: one feature predicting one target variable.

$$y = \beta_0 + \beta_1 x + \varepsilon$$

Components

y: Target variable (dependent)
x: Input feature (independent)
β₀: Intercept (baseline prediction)
β₁: Slope (rate of change)
ε: Error term (noise)

Interpretation

For every unit increase in x, y changes by β₁ units
β₀ represents y when x = 0
ε captures measurement error and omitted factors
Assumes constant rate of change (linearity)

Example: House Price = $150 × Square Footage + $50,000
Each additional square foot adds $150 to the price.

Multiple Linear Regression

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_p x_p + \varepsilon$$

Key Differences

Partial effects: Each β_i shows effect of x_i holding other features constant
Conditional relationships: Effects depend on other variables
More realistic: Real-world outcomes depend on multiple factors
Higher complexity: More parameters to estimate

Error Term Properties

Captures omitted variables
Includes measurement error
Represents inherent randomness
Assumed to be normally distributed

Example: House Price = β₀ + β₁×Size + β₂×Bedrooms + β₃×Age + ε
Each coefficient shows the effect of that feature, controlling for the others.

Matrix Notation

Compact representation enables efficient computation and theoretical analysis.

$$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$$

Matrix Components

y: n×1 vector of target values
X: n×p matrix of features
β: p×1 vector of coefficients
ε: n×1 vector of error terms

Dimensions:
n = number of observations
p = number of features

Computational Advantages

Enables vectorized operations
Scales to large datasets efficiently
Provides unified treatment regardless of p
Facilitates theoretical analysis

Requirement: X must have full column rank (no multicollinearity)

Ordinary Least Squares (OLS)

Find parameters that minimize the sum of squared residuals.

$$\min_{\boldsymbol{\beta}} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Why Squared Errors?

Penalizes large errors more heavily
Differentiable everywhere
Leads to closed-form solution
Maximum likelihood under normality

Connection to Correlation

Strong correlation → small residuals
Weak correlation → large residuals
OLS finds the best possible linear fit
Quality depends on data's linear structure

Error Minimization Visualization

Visualization: Error minimization with residual lines

The visualization shows how OLS finds the optimal line by minimizing the sum of squared residuals (vertical distances from points to the line).

Training Methods

Normal Equation

$$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

✅ Advantages:

Exact global optimum
No hyperparameters needed
Single computation step
Mathematically elegant

❌ Disadvantages:

O(p³) complexity
Requires matrix inversion
Memory intensive for large data

Gradient Descent

$$\boldsymbol{\beta}_{t+1} = \boldsymbol{\beta}_t - \alpha \nabla J(\boldsymbol{\beta}_t)$$

✅ Advantages:

Scales to large datasets
O(p) memory complexity
Works with streaming data
Extensible to other loss functions

❌ Disadvantages:

Requires tuning learning rate α
May need many iterations
Sensitive to feature scaling

Cost Function: Mean Squared Error

$$\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2$$

Properties

Always non-negative: Minimum value is 0
Differentiable: Enables gradient-based optimization
Convex: Guarantees global minimum
Outlier sensitive: Large errors dominate

Alternative Loss Functions

Mean Absolute Error (MAE): More robust to outliers
Huber Loss: Combines MSE and MAE benefits
Quantile Loss: For quantile regression
Custom losses: Domain-specific objectives

MSE is the maximum likelihood estimator under the assumption of normally distributed errors.

Understanding Correlation

The strength of linear correlation determines how well linear regression will perform.

Key Insights

Strong correlation means data points cluster tightly around a line
Weak correlation means data points are scattered widely
Linear regression works best with strong linear relationships
Correlation coefficient ranges from -1 to +1

Impact on Performance

Strong correlation (|r| > 0.7): High R², low prediction error
Moderate correlation (0.3 < |r| < 0.7): Decent performance
Weak correlation (|r| < 0.3): Poor predictive power
No correlation (r ≈ 0): Linear regression ineffective

Correlation Examples

Weak Correlation (r ≈ 0.3)

Weak correlation scatter plot

Data forms a cloud pattern. Linear regression will have poor predictive power.

Strong Correlation (r ≈ 0.9)

Strong correlation scatter plot

Data follows a clear linear pattern. Linear regression will provide accurate predictions.

Feature Scaling & Practical Considerations

Feature Scaling

Why needed: Features with different scales can dominate
Standardization: (x - μ) / σ → mean=0, std=1
Normalization: (x - min) / (max - min) → range [0,1]
When to use: Always for gradient descent

Example: House size (2000 sqft) vs bedrooms (3)
Without scaling, size dominates the optimization

Handling Outliers

Detection: Residual plots, leverage, Cook's distance
Impact: Can severely skew coefficient estimates
Options: Remove, transform, robust regression
Caution: Outliers might contain valuable information

Trade-off: Robust fit vs. preserving data integrity

When Linear Regression Fails

Recognizing limitations is crucial for appropriate model selection.

Common Failure Modes

Nonlinear relationships: Polynomial, exponential, logarithmic patterns
Interaction effects: Features affect each other's impact
Heteroscedasticity: Non-constant error variance
Multicollinearity: Highly correlated features

Alternative Approaches

Polynomial regression: Add x², x³ terms
Regularization: Ridge, Lasso, Elastic Net
Nonlinear models: Decision trees, neural networks
Transformations: Log, square root, Box-Cox

Rule of thumb: Always plot residuals vs. fitted values to check assumptions

Model Evaluation Metrics

R-squared

$$R^2 = 1 - \frac{\text{SS}_{\text{res}}}{\text{SS}_{\text{tot}}}$$

Proportion of variance explained (0 to 1)

Mean Squared Error

$$\text{MSE} = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2$$

Average squared prediction error

Root MSE

$$\text{RMSE} = \sqrt{\text{MSE}}$$

Error in original units

Model Performance Comparison

Visualization: Model comparison metrics

Compare different models using R², MSE, and RMSE. Higher R² and lower MSE/RMSE indicate better performance.

Linear Regression Assumptions

1. Linearity
The relationship between features and target is linear.

2. Independence
Observations are independent of each other.

3. Homoscedasticity
Constant variance of residuals across predictions.

4. Normality
Residuals are approximately normally distributed.

Residual plot visualization

Key Takeaways

Strengths

Simple and interpretable
Fast training and prediction
No hyperparameter tuning
Probabilistic interpretation
Good baseline for comparison

Limitations

Assumes linear relationships
Sensitive to outliers
Requires feature scaling for optimal performance
Can overfit with many features
Limited to linear decision boundaries

Linear regression is fundamental to machine learning. Master it well as it forms the foundation for many advanced techniques including regularization, ensemble methods, and neural networks.