← Course Overview ML2: Advanced Machine Learning
← Back to Week 11

Week 11: Practical LLM Integration & API Development

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Application

1. The Art and Science of Prompt Engineering

System prompts, few-shot examples, and output formatting instructions are the primary tools a developer has to steer LLM behavior without modifying model weights. Crafting effective prompts can be the difference between a reliable production system and one that fails unpredictably. Yet prompt engineering remains more craft than science — changes that seem minor can have large and unexpected effects on output quality.

  • Why do you think specifying output format (e.g., "respond only with a JSON object") tends to improve reliability in automated pipelines? What is happening from the model's perspective?
  • Few-shot examples guide the model through demonstration rather than instruction. In what situations might few-shot prompting outperform a detailed system prompt, and vice versa?
  • As LLM capabilities improve, do you think prompt engineering will become less important or more nuanced? Is it a skill that will remain valuable in five years?

Try: Give student pairs the same task (e.g., extracting structured data from a paragraph) and have each pair write a different system prompt. Then test all versions live on the same inputs and compare which prompts produce the most consistent outputs, discussing what differences in phrasing drove different results.

Analysis

2. Chain-of-Thought Reasoning: Transparency or Theater?

Chain-of-thought (CoT) prompting encourages the model to show its reasoning step by step before arriving at a final answer. This approach demonstrably improves accuracy on complex tasks and produces outputs that are easier to debug. However, it raises a deeper question: does the visible reasoning reflect the model's actual internal computation, or is it a fluent rationalization generated after the fact?

  • If a model produces a correct answer via an incorrect chain of thought (or vice versa), what does this tell us about the relationship between the visible reasoning and the underlying computation?
  • In a high-stakes domain like medical diagnosis or financial risk assessment, how much weight should a human reviewer place on the model's stated reasoning chain when deciding whether to trust its conclusion?
  • The textbook mentions validation checkpoints within a CoT to catch errors early. Design a CoT prompt structure with embedded checkpoints for a specific domain problem of your choosing.

Try: Present the class with a multi-step math or logic problem, show two model outputs side by side — one with a confident but subtly incorrect chain of thought and one with a correct chain — and ask students to evaluate which they would trust and why before revealing which is actually correct.

Evaluation

3. Cost, Rate Limits, and the Economics of Production LLM Systems

LLM API costs follow a per-token pricing model, with charges for both input and output tokens. At scale, token consumption costs can become substantial. The textbook describes a range of optimization strategies — caching, batching, prompt compression, model selection based on task requirements — that together form the cost management discipline for production LLM systems.

  • Caching responses to repeated prompts can dramatically reduce costs, but introduces the risk of serving stale responses. How would you design a cache invalidation policy for a knowledge-intensive application like a customer support bot?
  • The textbook recommends implementing per-user limits, project-level budgets, and organization-wide caps. What organizational dynamics or failure modes do these layers protect against?
  • Consider an application that could use either a small, fast, inexpensive model or a large, capable, expensive one. What criteria would you use to decide which to route each request to?

Try: Give small groups a hypothetical monthly API budget of $500 and a set of sample request volumes and query types, then ask each group to design a cost management strategy (caching policy, model routing rules, batch scheduling) and present it to the class for critique.

Synthesis/Creation

4. Human-in-the-Loop Systems: When Is AI Oversight Enough?

For critical applications, production LLM systems include human review as a safety net. The textbook describes systems that automatically flag outputs for human review based on confidence scores, unusual patterns, or domain-specific risk factors. But implementing effective human oversight requires more than just routing uncertain outputs to a queue — it requires designing interfaces that enable reviewers to make informed, efficient decisions.

  • What information would you include in a human review interface to help a reviewer quickly and accurately assess whether an LLM output is acceptable? What makes reviewer interfaces succeed or fail?
  • If a system routes 5% of cases for human review, what happens to those 95% that are automatically approved? How do you build confidence that the automated approval threshold is set correctly?
  • Human review introduces latency and cost. How would you decide which applications warrant human-in-the-loop oversight versus fully automated processing? Where do you draw the line?

Try: Run a small-group design sprint where each group sketches a human review interface for a specific domain (medical triage, content moderation, legal document processing), then presents it to the class and receives critique on what information was included or omitted.

Analysis

5. Monitoring, Observability, and the Challenge of Measuring LLM Quality in Production

Unlike traditional software systems where correctness is binary and deterministic, LLM application quality is probabilistic and context-dependent. The textbook describes a range of metrics — token efficiency, prompt success rates, response quality metrics, cache hit rates, human review triggers — that together constitute an observability framework for LLM systems. Defining and measuring "quality" in this context is a non-trivial challenge.

  • Traditional software monitoring focuses on error rates and latency. What additional dimensions of quality are specific to LLM applications, and how would you instrument a system to track them?
  • The textbook mentions anomaly detection for "unexpected changes in response patterns." How would you define a baseline for normal LLM response patterns, and what would constitute a meaningful anomaly in that context?
  • User satisfaction is listed as a key metric, but it can be difficult to collect at scale. What proxy metrics could you use to infer user satisfaction without requiring explicit ratings from users?

Try: Ask students to design a monitoring dashboard for a fictional LLM-powered customer service application, specifying which five metrics they would display on the primary screen and what alert thresholds they would set, then compare and debate the choices across groups.

Discussion Facilitation Tips

  • Prompt engineering discussions benefit enormously from live experimentation. If possible, have a shared LLM interface open during class so students can test prompt variations in real time. Seeing the model respond differently to small phrasing changes is far more instructive than describing it hypothetically.
  • The chain-of-thought transparency debate often splits the class. Some students find visible reasoning highly reassuring; others are skeptical. Encourage both camps to articulate their criteria for what would actually count as evidence that the reasoning is genuine versus post-hoc rationalization — this pushes the discussion toward more rigorous epistemics.
  • When discussing cost management, make the numbers concrete. Show a sample API pricing page and walk through what a real-world request volume (e.g., 1 million customer support queries per month) would cost at different token counts. This grounds the optimization strategies in genuine economic stakes rather than abstract principles.
  • For the human-in-the-loop topic, bring in the concept of automation bias — the tendency of human reviewers to over-trust automated recommendations. Ask students how a well-designed interface might counteract this bias, and what the failure mode looks like when reviewers rubber-stamp AI outputs without genuine evaluation.