Autoencoders

Neural Networks for Unsupervised Representation Learning

Learning compressed representations through reconstruction

Core Concept: Train neural networks to copy their input to output through a bottleneck layer
Learning Goals:
  • Understand autoencoder architecture and motivation
  • Learn different types and their applications
  • Compare with other dimensionality reduction methods
  • Apply autoencoders to real-world problems

Why Autoencoders?

The Challenge

  • High-dimensional data: Images, text, sensor data
  • Hidden structure: Meaningful patterns in complex data
  • Unlabeled data: Most real-world data lacks labels
  • Feature learning: Manual feature engineering is difficult
Goal: Learn function $f: \mathbb{R}^d \rightarrow \mathbb{R}^k$
where $k < d$ (compression)
Applications: Dimensionality reduction, denoising, anomaly detection, feature learning, data generation

Autoencoder Architecture

Encoder Network

$z = f_\theta(x)$
  • Maps input $x \in \mathbb{R}^d$ to latent code $z \in \mathbb{R}^k$
  • Typically $k < d$ for compression
  • Uses standard activation functions (ReLU, tanh)

Decoder Network

$\hat{x} = g_\phi(z)$
  • Maps latent code back to reconstruction $\hat{x}$
  • Often mirrors encoder architecture
  • Output activation depends on data type
Key Insight: Bottleneck forces network to learn compressed, meaningful representations

Training Autoencoders

Reconstruction Loss

MSE: $L = \frac{1}{n}\sum_{i=1}^{n}||x_i - \hat{x}_i||^2$
Binary CE: $L = -\sum_{i=1}^{d}[x_i \log(\hat{x}_i) + (1-x_i)\log(1-\hat{x}_i)]$

Choice depends on input data type:

  • Continuous data: MSE, MAE
  • Binary data: Binary cross-entropy
  • Categorical: Categorical cross-entropy
Training Process: Standard backpropagation, no labels required, minimize reconstruction error

Undercomplete vs Overcomplete Autoencoders

Undercomplete ($k < d$)

  • Natural compression: Bottleneck smaller than input
  • Forces feature extraction: Cannot simply copy
  • Risk: May learn trivial features if too constrained
  • Good for: Dimensionality reduction, visualization

Overcomplete ($k > d$)

  • Potential copying: Could learn identity function
  • Requires regularization: Prevent trivial solutions
  • Techniques: Sparsity, noise, weight decay
  • Good for: Feature learning with constraints
Key Trade-off: Compression vs. Reconstruction Quality

Sparse Autoencoders

Sparsity Constraint

$L = L_{recon} + \lambda \sum_j |h_j|$

Or KL-divergence sparsity:

$L = L_{recon} + \beta \sum_j KL(\rho || \hat{\rho}_j)$
  • $\rho$: Target sparsity (e.g., 0.05)
  • $\hat{\rho}_j$: Average activation of neuron $j$
  • $\beta$: Sparsity weight
Benefits: Learns distributed but sparse representations, prevents overfitting, interpretable features
Biological Inspiration: Neurons in the brain are typically sparsely active

Denoising Autoencoders (DAE)

Training Process

  1. Corrupt input: $\tilde{x} = x + \epsilon$
  2. Encode corrupted: $z = f_\theta(\tilde{x})$
  3. Decode to clean: $\hat{x} = g_\phi(z)$
  4. Loss vs. original: $L = ||x - \hat{x}||^2$

Corruption Types

  • Gaussian noise: $\tilde{x} = x + \mathcal{N}(0, \sigma^2)$
  • Masking: Set random pixels to 0
  • Salt-and-pepper: Random black/white pixels

Benefits

  • Robust features: Invariant to small perturbations
  • Prevents copying: Must learn meaningful structure
  • Data augmentation: Implicit regularization
  • Real-world noise: Handles imperfect data
Key Insight: Corruption forces extraction of robust, essential features

Advanced Autoencoder Variants

Variational Autoencoders (VAE)

$L = L_{recon} + \beta D_{KL}(q_\phi(z|x)||p(z))$
  • Probabilistic encoding: $q_\phi(z|x) = \mathcal{N}(\mu, \sigma^2)$
  • Regularized latent space: KL divergence term
  • Generation capability: Sample from $p(z)$ and decode
  • Smooth interpolation: Continuous latent space

Convolutional Autoencoders

  • Spatial structure: Preserve image topology
  • Encoder: Conv + pooling layers
  • Decoder: Transposed conv + upsampling
  • Applications: Image compression, super-resolution

Sequence Autoencoders

  • RNN/LSTM based: Handle temporal data
  • Encoder: Sequence → Fixed vector
  • Decoder: Vector → Sequence
  • Applications: Text, time series, speech

Autoencoders vs Other Methods

Method Type Advantages Disadvantages Best For
PCA Linear Fast, closed-form, interpretable Linear only, global transformation Linear relationships, quick analysis
Autoencoders Non-linear Complex manifolds, invertible, flexible Requires training, local minima Complex data, feature learning
t-SNE/UMAP Non-linear Great visualization, local structure Not invertible, slow, 2D/3D only Data visualization, exploration
When to Use Autoencoders: Need non-linear reduction + invertible transformation + feature learning

Real-World Applications

Image Processing

  • Compression: Better than JPEG for specific domains
  • Denoising: Medical images, old photos
  • Super-resolution: Enhance low-res images
  • Inpainting: Fill missing image regions

Anomaly Detection

  • Fraud detection: Unusual transaction patterns
  • System monitoring: Network intrusion detection
  • Quality control: Manufacturing defects
  • Medical diagnosis: Abnormal scan patterns
Anomaly Detection Process: Train on normal data → High reconstruction error indicates anomaly
Anomaly Score: $S(x) = ||x - \hat{x}||^2$

Key Takeaways

🎯 Core Concepts

  • Autoencoders learn compressed representations through reconstruction
  • Bottleneck architecture forces meaningful feature extraction
  • Unsupervised learning - no labels required
  • Balance between compression and reconstruction quality

🔧 Design Choices

  • Architecture: Undercomplete vs overcomplete
  • Regularization: Sparsity, denoising, variational
  • Loss function: MSE, cross-entropy, KL divergence
  • Network type: Dense, convolutional, recurrent

🎨 Applications

  • Dimensionality reduction: Non-linear alternative to PCA
  • Feature learning: Pre-training for downstream tasks
  • Data cleaning: Denoising and reconstruction
  • Anomaly detection: Outlier identification
  • Generation: VAEs for new sample creation
Remember: Autoencoders provide flexible, learnable dimensionality reduction but require careful design and training
1 / 11