Week 15: Future Trends in LLMs — Knowledge Check

Question 1

A) It makes the final model smaller and faster at inference time
B) It allows fine-tuning on modest hardware by drastically reducing the number of parameters that need to be trained
C) It prevents the model from forgetting what it learned during pre-training
D) It automatically selects the most important layers to update based on the task

Question 2

A) All expert sub-networks process every input token and their outputs are averaged, spreading the compute
B) A gating network routes each input to only a small subset of expert sub-networks, so most experts remain inactive for any given token
C) Expert weights are quantized to 1-bit precision to reduce memory bandwidth during inference
D) The model uses separate tokenizers for each expert to reduce vocabulary processing overhead

Question 3

A) It replaces the language model with a symbolic rule engine for all safety-critical outputs
B) It uses a set of principles to guide the model in self-critiquing and revising its own outputs, reducing dependence on large volumes of direct human preference labels
C) It trains a separate constitutional model that checks outputs after deployment, with no changes to the base model training
D) It applies differential privacy during training to prevent the model from memorizing sensitive constitutional documents

Question 4

A) Larger models always outperform smaller models regardless of dataset size, so compute should be spent entirely on increasing parameters
B) Previous large models were undertrained; optimal compute allocation requires scaling dataset size proportionally with model parameters, not focusing solely on parameter count
C) Training loss follows an exponential rather than a power-law relationship with compute, invalidating earlier scaling predictions
D) The optimal dataset size is fixed at approximately 300 billion tokens regardless of model scale

Question 5

A) It reduces the number of parameters in the feed-forward layers by using sparse activations
B) It eliminates the quadratic memory complexity of standard attention by reorganizing computation to avoid materializing the full attention matrix, achieving 2-4x speedups and enabling longer contexts
C) It replaces softmax attention with a linear approximation, changing the expressive power of each attention head
D) It quantizes attention weights to 4-bit precision during the forward pass to reduce memory bandwidth

Question 6

A) The gradual, steady improvement in generalization performance throughout training that scales predictably with compute
B) A sudden, sharp jump to near-perfect generalization that occurs after prolonged training, even after the model has already achieved low training loss through apparent memorization
C) The phenomenon where models copy exact training examples verbatim into their outputs, indicating overfitting
D) The process by which a model's internal representations become interpretable to human researchers after sufficient training

Question 7

A) Running the fine-tuning for fewer training steps so the model does not drift too far
B) Adding a penalty that keeps the model's outputs close to the original, so it improves without losing its general capabilities
C) Using a smaller dataset of human preferences to limit how much the model can change
D) Freezing the first half of the model's layers and only updating the later ones

Question 8

A) It is pre-trained with a masked image modeling objective, predicting missing image patches from surrounding context
B) It maximizes the cosine similarity between matching image-text pairs while minimizing similarity between non-matching pairs through contrastive learning on large image-caption datasets
C) It is fine-tuned on labeled ImageNet categories using standard cross-entropy loss, then adapted to text descriptions
D) It uses a generative decoder to produce image captions and is evaluated on BLEU score against human-written captions

Question 9

A) Capabilities that are explicitly trained at every scale but become more accurate as model size increases
B) Capabilities that are absent or near-random in smaller models but appear sharply and dramatically at certain scales, making them difficult to predict in advance
C) Capabilities that emerge only after fine-tuning on task-specific data, regardless of the base model scale
D) Capabilities that degrade with scale due to interference from increasingly diverse training data

Question 10

A) Full-precision fine-tuning combined with gradient accumulation over many small batches
B) Quantizing pre-trained model weights to 4-bit precision while keeping LoRA adapter parameters in higher precision, combined with paged optimizers and gradient checkpointing
C) Replacing all attention layers with linear recurrent layers to reduce the quadratic memory cost of attention
D) Distributing fine-tuning across hundreds of GPUs using data parallelism with synchronized gradient updates