← Back to Week 10

Week 10: Introduction to Large Language Models

Knowledge Check — Select an answer to see immediate feedback.

Questions 10

Question 1

Which mathematical operation is at the core of the transformer self-attention mechanism?

A) A convolution over the input sequence using learned filters
B) A weighted sum of value vectors, where weights come from softmax-scaled query-key dot products
C) A recurrent update of a hidden state using the previous hidden state and the current input
D) A feedforward projection of each token embedding independently

Question 2

Early n-gram language models estimated the probability of a word given the previous n-1 words. What was a primary weakness of this approach compared to transformer-based LLMs?

A) They required GPU hardware that was not available before 2012
B) They could not be trained on text data of any kind
C) They struggled to capture long-range dependencies and suffered from data sparsity
D) They generated outputs that were too grammatically complex for practical use

Question 3

Scaling laws in deep learning describe a relationship between model performance and scale factors. Which of the following best describes this relationship?

A) Performance improves linearly and without limit as model size increases
B) Test loss scales as a power law with respect to model size, compute budget, and dataset size
C) Performance is determined solely by dataset size and is independent of model size
D) Larger models always perform worse because they overfit to training data

Question 4

GPT-3 grew from GPT-1's 117 million parameters to 175 billion parameters. What primary capability did this dramatic increase in scale unlock?

A) The ability to process images alongside text for the first time
B) Remarkable few-shot learning across diverse tasks without task-specific fine-tuning
C) Perfect factual accuracy with zero hallucinations
D) Real-time internet search integration

Question 5

Masked language modeling (MLM), used in BERT, differs from causal language modeling (CLM), used in GPT, in what fundamental way?

A) MLM trains on images while CLM trains only on text
B) MLM predicts randomly masked tokens using bidirectional context; CLM predicts the next token using only prior context
C) MLM requires labeled data while CLM is entirely unsupervised
D) MLM generates text autoregressively while CLM uses a fill-in-the-blank objective

Question 6

In the context of LLM pre-training, what technique involves discarding intermediate activations during the forward pass and recomputing them during backpropagation to reduce memory usage?

A) Data parallelism
B) Mixed-precision training
C) Gradient checkpointing
D) Layer normalization

Question 7

"Hallucination" in an LLM refers to which of the following behaviors?

A) The model refusing to answer questions outside its training domain
B) The model generating text that is fluent and plausible but factually incorrect or unsupported
C) The model producing outputs that contain explicit content from its training data verbatim
D) The model failing to generate any output when given an ambiguous prompt

Question 8

In-context learning, few-shot learning, and zero-shot generalization differ in how much task information the prompt provides. A colleague asks an LLM to classify support tickets using only a written instruction — no example tickets, no labels, no demonstrations. Which setting is this?

A) Few-shot learning
B) Zero-shot generalization
C) Fine-tuning
D) In-context learning with demonstrations

Question 9

Google's PaLM architecture uses an approach called "Pathways." What distinguishes this approach from standard single-task model training?

A) It trains a separate expert model for each task and combines their outputs at inference time
B) It aims to train a single model to perform multiple tasks across different modalities using shared knowledge
C) It uses reinforcement learning from human feedback as its primary training objective
D) It reduces model size by pruning weights that are not used across tasks

Question 10

When a language model is about to predict the next word, it first builds a rich internal representation of everything it has read so far. What does this final internal state capture?

A) Just the most recent word that appeared in the input
B) A summary of all the context up to that point — what the model has understood about the full conversation so far
C) The raw probabilities from training data for what word typically follows
D) The specific attention weights used in the final transformer layer