Week 13: Evaluating LLMs - Metrics and Methods

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Evaluation

1. The Gap Between Automated Metrics and Human Judgment

BLEU and ROUGE scores are convenient and reproducible, but research has repeatedly shown they correlate poorly with human judgments of translation and summarization quality. A system can achieve a high BLEU score by closely matching surface-level word patterns while producing output that is semantically incoherent or factually wrong. At the same time, a creative but accurate paraphrase will score poorly because it does not match reference n-grams.

In what real-world applications would a mismatch between BLEU and human quality be most dangerous?
Should automated metrics be abandoned, improved, or used only in combination with human evaluation? What does each choice cost?
How would you design a study to measure how well a new automated metric correlates with expert human judgment in a domain like medical report generation?

Try: Show students two model outputs side by side — one with a high BLEU score and one with a low BLEU score — and have the class vote on which is higher quality before revealing the scores. Discuss the surprise factor.

Analysis

2. Benchmark Saturation and the Arms Race in LLM Evaluation

Benchmark suites like GLUE, SuperGLUE, and BIG-bench were introduced to provide standardized, challenging tests for LLMs. However, frontier models now approach or exceed human-level performance on many of these benchmarks, sometimes through contamination of training data rather than genuine understanding. This phenomenon — often called "benchmark saturation" — raises questions about what benchmarks actually measure and how to keep evaluation meaningful as models improve.

What does it mean for a model to "pass" a benchmark, and does passing indicate true comprehension or sophisticated pattern matching?
How can the research community design benchmarks that remain challenging and contamination-resistant as training datasets grow?
Should benchmark performance be weighted differently based on the risk of data contamination from pre-training corpora?

Try: Split the class into two groups — one argues that benchmark saturation proves models truly understand language, the other argues it proves the benchmarks are flawed. Hold a structured four-minute debate, then debrief together.

Synthesis/Creation

3. Designing Reliable Human Evaluation Protocols

Human evaluation is considered the gold standard for assessing LLM output quality, but it is expensive, slow, and susceptible to annotator bias and inconsistency. Protocols based on Likert scales, A/B comparisons, and expert review each have different trade-offs in reliability and cost. Inter-rater reliability measured by Cohen's kappa can reveal significant disagreement even among trained annotators, especially for subjective qualities like creativity or coherence.

How would you structure an annotation rubric for evaluating LLM-generated patient-facing medical explanations? What criteria matter most?
What sample size and annotator diversity would you need to obtain statistically significant results in an A/B comparison of two chatbot systems?
How should annotation guidelines handle cases where two responses are equally good but in different ways, such as one being more accurate and the other more readable?

Try: Give small groups of three students the same three LLM outputs to rate independently on a 1-5 scale for quality, then calculate the inter-rater agreement in class. Use the results to illustrate why annotation guidelines and training matter.

Analysis

4. Context Sensitivity and the Limits of Token-Level Metrics

Traditional NLP metrics evaluate language in isolation, scoring individual sentences or n-grams without regard for the broader discourse. Yet language is inherently context-dependent: the same sentence can be coherent or absurd depending on what came before it. LLMs used in dialogue, story generation, or long-form writing need to maintain consistency and relevance across hundreds or thousands of tokens, which token-level metrics like perplexity completely miss.

What would a "contextual coherence" metric need to measure, and how would you collect ground-truth data to train or validate it?
Are there tasks — such as short factual question answering — where token-level metrics remain fully adequate? What distinguishes these tasks from those that demand context-aware evaluation?
How do recent approaches like G-Eval (using an LLM as an evaluator) attempt to capture context sensitivity, and what are the risks of using one LLM to judge another?

Try: Present the class with a passage from a long generated story where a character's name or trait is inconsistent with an earlier paragraph. Ask students to identify what kind of metric would catch this error and what data you would need to build it.

Evaluation

5. Evaluating Reasoning, Knowledge, and Behavioral Consistency

LLM-specific evaluation goes beyond text quality to probe whether models genuinely reason, retain factual knowledge, and behave consistently when prompts are rephrased. Chain-of-thought accuracy, knowledge probing, and instruction-following assessments reveal that models can give correct final answers through flawed reasoning, "hallucinate" facts with high confidence, and respond differently to semantically identical questions depending on phrasing.

Should a model that arrives at a correct answer through incorrect reasoning receive the same credit as one that reasons correctly? How does the answer change depending on the application?
What techniques can be used to distinguish genuine knowledge retention from superficial pattern matching on knowledge-probing tasks?
If a model gives contradictory answers to the same question asked in different ways, what does this reveal about its underlying representations, and how should this inconsistency be factored into evaluation scores?

Try: Live demo — ask a public LLM the same factual question five different ways and display all responses. Have students classify each response as consistent or inconsistent, then discuss what the variation implies for deployment in a high-stakes setting.

Discussion Facilitation Tips

Ground abstract metric debates in concrete examples: bring in published side-by-side comparisons of BLEU scores and human ratings from MT or summarization papers to show the disconnect is real and quantifiable.
When discussing benchmark saturation, encourage students to distinguish between two distinct failure modes — data contamination versus genuine generalization — since they have different policy implications.
For the human evaluation protocol discussion, emphasize that inter-rater reliability is not just a statistical formality: low kappa indicates the task definition itself is ambiguous and needs revision before data collection scales.
Connect the reasoning and consistency topic to real stakes: a student loan approval system that gives different decisions based on how an application is worded is not a hypothetical — help students see the regulatory and ethical weight of behavioral inconsistency.