← Course Overview ML2: Advanced Machine Learning
← Back to Week 12

Week 12: Retrieval Augmented Generation (RAG)

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Analysis

1. Chunking Strategy: The Hidden Foundation of RAG Performance

Before any retrieval can happen, documents must be divided into chunks that are embedded and stored in the vector database. The textbook describes several strategies — fixed-size, sentence-based, paragraph-based, and semantic chunking — each with different trade-offs between precision, context preservation, and computational cost. The choice of chunking strategy is often the single most impactful decision in building a RAG system, yet it receives far less attention than model selection.

  • Smaller chunks tend to improve retrieval precision by reducing noise, but may lose important surrounding context. Larger chunks preserve context but may dilute relevance. How would you decide on an appropriate chunk size for a technical documentation corpus versus a corpus of news articles?
  • Semantic chunking uses embeddings to identify topic boundaries — it is potentially the most effective but also the most computationally expensive. Under what circumstances would the added cost be justified?
  • The textbook mentions metadata enrichment — augmenting chunks with document title, section headers, and creation dates. How might this metadata change the retrieval behavior, and what retrieval scenarios would benefit most from it?

Try: Give student groups a printed multi-page technical document and ask each group to manually chunk it using a different strategy (fixed-size, sentence-level, paragraph-level, semantic topic boundaries). Then have groups compare their chunk boundaries and discuss which strategy would produce the best retrievable units for different types of queries against that document.

Application

2. Semantic Search vs. Keyword Search: When Does Meaning Matter?

Traditional keyword-based search (including BM25) matches documents to queries based on term frequency and exact vocabulary overlap. Semantic search using dense vector embeddings captures conceptual similarity even when no exact terms are shared. RAG systems increasingly use hybrid approaches that combine both. Understanding when each mode of search excels — and when it fails — is essential for designing effective retrieval systems.

  • Give a concrete example of a query where pure semantic search would outperform BM25, and one where BM25 would outperform semantic search. What do these examples reveal about the failure modes of each approach?
  • The choice of similarity metric (cosine similarity, Euclidean distance, dot product) should align with how the embedding model was trained. Why does the training objective of the embedding model determine which metric is most appropriate?
  • Approximate nearest neighbor (ANN) algorithms like HNSW trade accuracy for speed. In a production RAG system, how much retrieval accuracy loss would you consider acceptable, and how would you measure it?

Try: Run a live demo using a simple keyword search and a semantic search on the same small document corpus, querying it with paraphrases and rare technical terms. Ask students to predict which system will retrieve better results before seeing the output, then analyze the cases where predictions were wrong.

Synthesis/Creation

3. Advanced Retrieval: HyDE, Multi-Query, and Active RAG

The basic RAG retrieval step — embed query, find nearest neighbors, retrieve top-k — is a starting point, not an endpoint. The textbook describes several advanced retrieval strategies: HyDE (embedding a hypothetical answer), multi-query retrieval (generating variations of the query), and Active RAG (iterative multi-step retrieval). Each addresses a different failure mode of naive retrieval, but each also adds latency and complexity.

  • HyDE generates a hypothetical answer using the LLM before retrieval. This means the LLM is run twice — once before and once after retrieval. When do you think the added latency and cost of HyDE would be justified by improved retrieval quality?
  • Multi-query retrieval generates multiple reformulations of the original query. What types of questions would benefit most from this? What are the risks of combining results from multiple queries?
  • Active RAG (iterative retrieval) is described as particularly powerful for complex multi-step questions. Design a concrete example of a complex question and sketch out how Active RAG would proceed to answer it over multiple retrieval rounds.

Try: Present students with a complex multi-hop question (e.g., one that requires finding information from two separate documents and synthesizing it) and ask small groups to manually trace through how Active RAG would retrieve, evaluate, and re-query to build toward an answer — then compare the paths different groups took.

Evaluation

4. Evaluating RAG: What Does a "Good" Answer Mean?

Evaluating RAG systems is more complex than evaluating traditional information retrieval because quality depends on both the retrieval step and the generation step, and these can fail independently. The textbook describes retrieval metrics (Recall@k, Precision@k, MRR, NDCG), generation metrics (faithfulness, answer relevance, hallucination rate), and end-to-end metrics (answer correctness, human preference). Building a comprehensive evaluation framework requires balancing all of these.

  • A RAG system might retrieve the right documents but generate an unfaithful response, or retrieve the wrong documents but still produce a correct answer from parametric knowledge. How would you design an evaluation pipeline that detects and distinguishes these different failure modes?
  • Ground truth labels for retrieval evaluation (knowing which documents are relevant for each query) are expensive to obtain. What alternative approaches could you use to evaluate retrieval quality without human-annotated relevance labels?
  • Human preference is listed as a key end-to-end metric, but it can conflict with faithfulness — users sometimes prefer confident, well-written responses even when they are not fully grounded in the retrieved context. How would you handle this tension in a real application?

Try: Have students rate a set of five RAG-generated answers on two separate dimensions (how much they prefer the response overall, and how well they think it is grounded in a provided source document), then compare the rankings across both dimensions and discuss where they diverge and why.

Evaluation

5. Privacy, Security, and Access Control in RAG Systems

RAG systems often access proprietary or sensitive knowledge bases — internal company documents, patient records, legal case files. This creates security challenges that are distinct from those in standard software systems: the retrieval step must respect document-level access permissions, retrieved content must be sanitized before being included in prompts, and every retrieval action should be auditable. These requirements add significant architectural complexity to what might otherwise seem like a straightforward pipeline.

  • In a multi-tenant RAG system where different users have access to different subsets of documents, how would you implement access control at the retrieval layer? What are the risks if this control is implemented incorrectly?
  • The textbook mentions "sanitizing retrieved content before including in prompts." What types of sensitive information would need to be sanitized, and how would you implement this without destroying the informational value of the retrieved content?
  • Audit logging is described as a requirement for RAG systems handling sensitive data. What events should be logged, what metadata should be captured for each, and how would you use these logs to investigate a potential data access incident?

Try: Present a small-group scenario: a healthcare company is building a RAG system over patient records, and a user submits a query that retrieves a record they are not supposed to access. Ask each group to trace where the access control failure could have occurred and what logging would be needed to detect and investigate it.

Discussion Facilitation Tips

  • The chunking discussion is most productive when students have handled real text. If time allows, paste a multi-page document into a shared editor before class and ask students to draw chunk boundaries in different colors using different strategies. The disagreements that emerge — especially around paragraph breaks, tables, and figures — reveal the genuine difficulty of the problem far better than any abstract description.
  • When discussing retrieval metrics like Recall@k and Precision@k, it helps to frame them as two different types of librarian failures: a librarian who retrieves irrelevant books (low precision) versus one who misses important books that exist in the collection (low recall). This analogy lets students reason about the metrics intuitively before working through the formal definitions.
  • The HyDE and advanced retrieval discussion can easily become too hypothetical. Ground it by asking students to identify a specific domain (customer support, academic research, legal discovery) and explain concretely which advanced technique would add the most value for queries in that domain — and which would be wasted overhead. Specificity forces genuine reasoning rather than generic answers.
  • Privacy and access control in RAG is an area where students often underestimate the complexity. A useful provocation: ask students whether a retrieval system that returns only document summaries (rather than full text) is sufficient to protect sensitive data. This usually surfaces the insight that semantic information can be leaked even without returning raw content, leading to a richer discussion about what "sanitization" really means.