Week 7: Sequence Models & Attention — Knowledge Check

Question 1

A) They require significantly more memory to store.
B) They assign a single fixed vector to each word regardless of context, failing to capture multiple meanings.
C) They cannot be used in downstream NLP tasks like sentiment analysis.
D) They are too slow to compute for large vocabularies.

Question 2

A) The scores become identical for all word pairs, making attention useless
B) The scores can get very large, making the attention output so peaked that learning effectively stops
C) The scores grow too slowly, making the model unable to distinguish between relevant and irrelevant words
D) The scores become negative, causing the attention weights to sum to zero

Question 3

A) Transformers use larger embedding dimensions and need positional information to reduce parameter count.
B) Transformers process all input positions in parallel without inherent sequential ordering, so position must be explicitly injected.
C) Positional encodings replace the need for multi-head attention in Transformers.
D) RNNs also require positional encodings but encode them implicitly through hidden state initialization.

Question 4

A) Sinusoidal encodings add no additional trainable parameters and can generalize to sequence lengths not seen during training.
B) Sinusoidal encodings are faster to compute because they avoid matrix multiplication.
C) Sinusoidal encodings are always more accurate than learned embeddings on downstream tasks.
D) Learned positional embeddings cannot handle variable-length input sequences.

Question 5

A) It reduces the total number of parameters compared to single-head attention.
B) It allows the model to attend to different representation subspaces simultaneously, capturing diverse relationship types.
C) It ensures that all attention heads converge to the same representation for stability.
D) It replaces the need for feed-forward layers within each Transformer block.

Question 6

A) It computes the attention weights used by the next layer's multi-head attention module.
B) It introduces non-linearity and increases the model's capacity to learn complex relationships between attended features.
C) It reduces the dimensionality of the sequence to prevent overfitting.
D) It aggregates information across all sequence positions to form a single context vector.

Question 7

A) To mask padding tokens so they do not contribute to the loss calculation.
B) To prevent the model from attending to future positions, ensuring autoregressive generation remains causal.
C) To randomly drop attention weights during training as a form of regularization.
D) To focus attention exclusively on the most informative tokens identified by a separate classifier.

Question 8

A) It normalizes activations across the batch dimension using running statistics accumulated during training.
B) It normalizes activations across the feature dimension for each individual example, then applies learnable scale and shift parameters.
C) It clips activation values to a fixed range to prevent gradient explosion.
D) It applies dropout to each layer's output before passing it to the next sub-layer.

Question 9

A) Faster inference because fewer unique vectors need to be stored in the embedding table.
B) Word sense disambiguation — correctly interpreting polysemous words based on the surrounding text.
C) The ability to process arbitrarily long documents without truncation.
D) Training with smaller datasets because context reduces data requirements.

Question 10

A) Self-attention processes only a fixed window of tokens at a time, reducing memory requirements.
B) Self-attention computations for all positions can be performed simultaneously via matrix operations, while RNNs must process tokens sequentially.
C) Self-attention does not require backpropagation, eliminating the sequential dependency during training.
D) RNNs require GPU memory proportional to sequence length, while self-attention has constant memory usage.