Feedforward Neural Networks

The Foundation of Deep Learning

Understanding the basic building blocks of artificial neural networks

Key Concept: Neural networks are computational models inspired by biological neural networks that can learn complex patterns from data through layers of interconnected nodes.

Notation Used Throughout:

$y$ = true target values (ground truth)
$\hat{y}$ = predicted values (network output)
$\mathcal{L}$ = loss function (what we minimize)
Goal: Minimize $\mathcal{L}(y, \hat{y})$ to make predictions closer to truth

Mathematical functions that approximate complex relationships
Universal function approximators
Foundation for modern deep learning architectures

Historical Context

From Perceptrons to Deep Networks

1943: McCulloch-Pitts neuron model
1957: Rosenblatt's Perceptron
1969: Minsky & Papert show perceptron limitations
1986: Backpropagation algorithm popularized
2006: Deep learning renaissance begins
2012: AlexNet wins ImageNet competition

Key Insight: Neural networks experienced periods of decline ("AI winters") and resurgence, ultimately leading to today's deep learning revolution.

The Artificial Neuron

Basic Computational Unit

x₁

→

Σ

→

f(·)

→

ŷ

Mathematical Formula: $$\hat{y} = f\left(\sum_{i=1}^{n} w_i x_i + b\right)$$

Where $w_i$ are weights, $x_i$ are inputs, $b$ is bias, $f$ is the activation function, and $\hat{y}$ is the predicted output

Network Architecture

Layers and Connections

Input Layer: Receives raw data features
Hidden Layer(s): Process and transform information
Output Layer: Produces final predictions
Feedforward: Information flows in one direction

Layer Dimensions:

Input layer: n neurons (n = number of features)
Hidden layers: arbitrary number of neurons
Output layer: k neurons (k = number of classes/outputs)

Activation Functions

Introducing Non-linearity

Common Activation Functions:

Sigmoid: $\sigma(x) = \frac{1}{1 + e^{-x}}$
Tanh: $\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$
ReLU: $\text{ReLU}(x) = \max(0, x)$
Softmax: $\text{softmax}(x_i) = \frac{e^{x_i}}{\sum_{j=1}^{k} e^{x_j}}$

Purpose: Enable networks to learn non-linear patterns and relationships

Forward Propagation

Computing Network Output

Step-by-step Process:

Initialize input values: $\mathbf{x} = [x_1, x_2, ..., x_n]$ (where $\mathbf{a}^{(0)} = \mathbf{x}$)
For each layer $l$: $\mathbf{z}^{(l)} = \mathbf{W}^{(l)} \mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}$
Apply activation: $\mathbf{a}^{(l)} = f(\mathbf{z}^{(l)})$
Repeat until output layer: $\hat{\mathbf{y}} = \mathbf{a}^{(L)}$ (final prediction)

Matrix Representation: Enables efficient computation using vectorized operations

Loss Functions

Measuring Network Performance

Common Loss Functions:

Mean Squared Error (Regression): $\mathcal{L}_{MSE} = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$
Cross-Entropy (Classification): $\mathcal{L}_{CE} = -\sum_{i=1}^{k} y_i \log(\hat{y}_i)$
Binary Cross-Entropy: $\mathcal{L}_{BCE} = -[y \log(\hat{y}) + (1-y) \log(1-\hat{y})]$

Where $y$ is the true target value and $\hat{y}$ is the predicted value

Purpose: Quantify the difference between predicted ($\hat{y}$) and actual outputs ($y$)

Goal: Minimize the loss function $\mathcal{L}$ to improve model predictions

Backpropagation Algorithm

Learning Through Gradient Descent

Key Steps:

Forward pass: Compute predictions $\hat{y}$ and loss $\mathcal{L}(y, \hat{y})$
Backward pass: Compute gradients $\frac{\partial \mathcal{L}}{\partial w}$ using chain rule
Update weights: $w_{new} = w_{old} - \alpha \frac{\partial \mathcal{L}}{\partial w}$
Repeat: Until convergence or stopping criteria

Chain Rule Application: $$\frac{\partial \mathcal{L}}{\partial w_{ij}^{(l)}} = \frac{\partial \mathcal{L}}{\partial a_j^{(l)}} \cdot \frac{\partial a_j^{(l)}}{\partial z_j^{(l)}} \cdot \frac{\partial z_j^{(l)}}{\partial w_{ij}^{(l)}}$$

Where $\mathcal{L}$ is the loss function we want to minimize

Training Process

Optimizing Network Parameters

Training Algorithm:


                for epoch in range(num_epochs):

                    for batch in data_loader:

                        # Forward pass: compute predictions

                        y_hat = network(batch.inputs)  # predicted values

                        loss = loss_function(y_hat, batch.y)  # minimize this

                        # Backward pass: compute gradients

                        gradients = compute_gradients(loss)

                        # Update parameters to minimize loss

                        optimizer.step(gradients)

Objective: Minimize loss $\mathcal{L}(y, \hat{y})$ by adjusting weights and biases

Key Hyperparameters: Learning rate, batch size, number of epochs

Training Challenges

Common Issues and Solutions

Vanishing Gradients: Gradients become too small in deep networks
- Solutions: Better activation functions (ReLU), normalization
Exploding Gradients: Gradients become too large
- Solutions: Gradient clipping, proper initialization
Overfitting: Model memorizes training data
- Solutions: Regularization, dropout, early stopping
Local Minima: Optimization gets stuck
- Solutions: Better optimizers (Adam, RMSprop), momentum

Regularization Techniques

Preventing Overfitting

Common Techniques:

L1 Regularization: $\mathcal{L}_{total} = \mathcal{L}_{original} + \lambda \sum |w_i|$
L2 Regularization: $\mathcal{L}_{total} = \mathcal{L}_{original} + \lambda \sum w_i^2$
Dropout: Randomly set neurons to zero during training
Early Stopping: Stop training when validation loss increases

We minimize the total loss $\mathcal{L}_{total}$ which includes penalty terms

Goal: Balance model complexity with generalization ability

Modern Optimization

Beyond Basic Gradient Descent

Advanced Optimizers:

Momentum: $v_t = \gamma v_{t-1} + \alpha \nabla \mathcal{L}$
Adam: Combines momentum with adaptive learning rates
RMSprop: Adapts learning rate per parameter
AdaGrad: Accumulates squared gradients

All optimize by minimizing the loss function $\mathcal{L}$ more effectively than basic gradient descent

Benefits: Faster convergence, better handling of sparse gradients, adaptive learning rates

Key Takeaways

Feedforward Neural Networks

Neural networks are powerful universal function approximators
Architecture matters: depth, width, and activation functions
Training requires careful consideration of:
- Loss functions and optimization algorithms
- Regularization and overfitting prevention
- Hyperparameter tuning
Backpropagation enables efficient gradient computation
Modern techniques address traditional challenges
Foundation for advanced architectures (CNNs, RNNs, Transformers)

Next Steps: Explore specialized architectures for specific domains (computer vision, NLP, etc.)