← Course Overview ML2: Advanced Machine Learning

Week 10: Introduction to Large Language Models

Videos Textbook Colab Colab 2 Tokenizer Explorer Homework Knowledge Check Discussion

Course Generated Slides: Transformers

1. Evolution of Language Models

  • Historical perspective
    • N-gram models
    • RNN-based models
    • Transformer revolution
  • Scaling laws
    • Model size trends
    • Compute requirements
    • Data scaling
  • Architecture developments
    • GPT series evolution
    • PaLM architecture
    • Emergent capabilities

2. Pre-training and Foundation Models

  • Training objectives
    • Next token prediction
    • Masked language modeling
    • Causal language modeling
  • Data considerations
    • Web-scale datasets
    • Data quality
    • Filtering strategies
  • Computational challenges
    • Distributed training
    • Memory optimization
    • Training stability

3. Understanding LLM Behavior

  • Model capabilities
    • In-context learning
    • Few-shot learning
    • Zero-shot generalization
  • Internal mechanics
    • Attention patterns
    • Knowledge storage
    • Token representations
  • Limitations
    • Hallucinations
    • Reasoning gaps
    • Bias issues

Required Reading

  • "Language Models are Few-Shot Learners" (GPT-3 paper)
  • "On the Opportunities and Risks of Foundation Models"

Learning Objectives

  • Understand the architecture and capabilities of Large Language Models
  • Learn about pre-training and fine-tuning approaches
  • Explore practical applications and limitations
← Week 9: From Supervised to Generative Learning Week 11: Practical LLM Integration & API Development →