← Back to Week 13

Week 13: Evaluating LLMs - Metrics and Methods

Knowledge Check — Select an answer to see immediate feedback.

Questions 10

Question 1

A translation model tries to cheat by outputting just one word that happens to appear in the correct translation — and gets 100% precision. What does this reveal about using precision alone to evaluate translation quality?

A) Precision is too slow to compute for long texts
B) A one-word output that is technically correct is useless — precision alone does not check whether the translation is complete
C) Precision favors longer translations and needs to be balanced against output speed
D) Precision can only be applied to multiple-choice tasks, not open-ended generation

Question 2

ROUGE measures how much of a reference summary is covered by a model's output. If the reference says "The cat sat on the mat" and the model outputs "The mat had a cat," ROUGE would still award partial credit because:

A) ROUGE requires word-for-word matching, so this example would actually score zero
B) ROUGE counts individual word overlap regardless of order, and several words appear in both outputs
C) ROUGE only checks the first and last word of each sentence
D) ROUGE uses semantic meaning, so synonyms like "sat" and "rested" count as equivalent

Question 3

Perplexity measures how surprised a language model is when it sees text it hasn't encountered before. What does a lower perplexity value indicate?

A) The model requires more memory to make predictions
B) The model is better at predicting text — it is less surprised by typical word sequences
C) The model was trained on a smaller dataset
D) The model is slower but more accurate

Question 4

The F1 score is defined as the harmonic mean of precision and recall. Why is the harmonic mean preferred over the arithmetic mean in this context?

A) It is easier to compute when class distributions are balanced
B) It penalizes extreme imbalances between precision and recall, requiring both to be reasonably high for a good F1 score
C) It always produces higher values than the arithmetic mean, making models look better
D) It is identical to the geometric mean and therefore more statistically robust

Question 5

The GLUE and SuperGLUE benchmark suites are specifically designed to evaluate which aspect of large language models?

A) The model's training throughput and inference latency
B) The model's ability to generate photorealistic images from text prompts
C) Natural language understanding capabilities across tasks like sentiment analysis, textual entailment, and question answering
D) Code generation ability measured by unit test pass rates

Question 6

One fundamental limitation of traditional metrics like BLEU and perplexity is that they focus on surface-level word overlap rather than meaning. Which evaluation approach has been proposed to better capture semantic understanding?

A) Measuring the vocabulary size of generated outputs
B) Using contextualized embeddings from models like BERT to compute semantic similarity between generated and reference text
C) Counting the number of unique sentences the model can produce per second
D) Restricting evaluation to single-sentence outputs to eliminate context ambiguity

Question 7

In human evaluation protocols, inter-rater reliability measures agreement among multiple annotators. Which statistic is commonly used for this purpose?

A) Pearson's r correlation coefficient
B) Cohen's kappa or Fleiss' kappa
C) The BLEU brevity penalty factor
D) A paired t-test comparing annotator scores

Question 8

Behavioral evaluation of LLMs assesses chain-of-thought accuracy. What does this specifically evaluate?

A) Whether the model can generate text faster than a baseline system
B) The model's ability to provide step-by-step reasoning and logically sound explanations for its outputs
C) The percentage of tokens the model correctly predicts in a masked language modeling task
D) Whether the model's outputs contain fewer grammatical errors than a reference baseline

Question 9

Blind evaluation is used as a systematic assessment technique in human evaluation protocols. What is its primary purpose?

A) To reduce annotation costs by having evaluators skip difficult examples
B) To prevent evaluators' prior knowledge of whether text is human-written or AI-generated from biasing their ratings
C) To ensure the model never sees its own outputs during self-evaluation
D) To eliminate the need for reference translations when computing BLEU scores

Question 10

The HELM (Holistic Evaluation of Language Models) framework takes a more comprehensive approach than task accuracy alone. Which additional dimensions does HELM consider?

A) Model architecture depth and the number of attention heads
B) Only the BLEU and ROUGE scores aggregated across all tasks
C) Robustness, fairness, and efficiency alongside task performance
D) The geographic location of the servers used for inference