← Course Overview ML2: Advanced Machine Learning
← Back to Week 14

Week 14: LLMs as Decision Makers and Agents

Discussion Topics — Use these prompts for class discussion, online forums, or personal reflection.

Evaluation

1. When Should an LLM Make Autonomous Decisions vs. Defer to Humans?

LLM-based decision frameworks include confidence thresholds and fallback mechanisms precisely because no model is infallible. Setting the threshold too high forces most decisions to humans, negating efficiency gains; setting it too low allows the model to act on weak signals. The stakes of the domain — medical triage, financial lending, legal analysis — profoundly affect where that threshold should sit and who bears accountability for errors.

  • In a high-stakes domain of your choice, what criteria should determine when a model's confidence is high enough to act autonomously versus escalate to a human?
  • How do liability, regulatory requirements, and public trust affect your threshold design, and who is accountable when an automated decision causes harm?
  • Could overreliance on high-confidence scores create complacency in human reviewers who see escalated cases as automatically problematic?

Try: Poll the class with a slider question — for a given domain (e.g., flagging fraudulent transactions), what confidence threshold percentage would they set before allowing autonomous action? Compare responses and discuss the spread of answers as a class.

Analysis

2. Reward Specification and Goal Misalignment in Agentic Systems

Agentic workflows frame the agent's objective as maximizing expected cumulative reward. However, specifying a reward function that exactly captures human intent is notoriously difficult. An agent optimizing a proxy reward may find unintended shortcuts — for example, a customer service agent optimizing for short resolution times might close tickets prematurely rather than actually solving problems. This is sometimes called "reward hacking" or "specification gaming."

  • Can you identify a real-world agentic use case where a plausible reward function would incentivize behavior that satisfies the letter but not the spirit of the goal?
  • What design choices — such as including multiple reward components, periodic human audits, or decision boundaries — could reduce reward hacking without making the system too conservative?
  • How does the discount factor gamma affect the risk of specification gaming, and should long-horizon tasks use different discount strategies than short-horizon ones?

Try: Small group activity — each group receives a different deployment scenario (e.g., content moderation, hospital scheduling, code review) and must write a one-sentence reward function, then swap with another group to find the loophole. Share the most creative exploits with the class.

Application

3. Scalability vs. Reliability in Multi-Agent Coordination

Multi-agent systems gain power from parallelism — multiple agents can tackle different parts of a problem simultaneously. However, coordination mechanisms (message passing, shared memory, centralized control) introduce latency, potential single points of failure, and conflict resolution overhead. As the number of agents scales up, the complexity of maintaining consistent state and resolving conflicts grows significantly.

  • For a large-scale data pipeline with many specialized LLM agents, which coordination mechanism — centralized control, shared memory, or peer message passing — offers the best balance of reliability and scalability, and why?
  • How should conflict resolution strategies differ between time-sensitive applications (e.g., real-time fraud detection) and non-time-sensitive ones (e.g., overnight report generation)?
  • What monitoring and observability infrastructure would you build to detect when coordination failures are silently degrading system output quality?

Try: Draw a live whiteboard diagram of a two-agent system and progressively add agents while asking students to call out where new failure points appear. Use this to make the scalability-reliability tension concrete and visible.

Evaluation

4. Audit Trails, Transparency, and the Right to Explanation

Decision logging and audit trails are described as essential components of safe LLM-based systems. They create a record of what the model decided, on what basis, and with what confidence. This is increasingly relevant as regulations such as the EU AI Act and GDPR's right to explanation require that automated decisions affecting individuals be explainable and contestable. However, logging detailed reasoning from opaque LLMs is technically challenging.

  • What information should a decision log minimally contain to satisfy a legal right-to-explanation requirement — and is a model's internal chain-of-thought reasoning sufficient as an "explanation"?
  • How do you balance the privacy risks of storing detailed interaction logs against the accountability benefits they provide?
  • If a model's decision cannot be explained in terms a non-technical person understands, should that model be permitted to make certain types of consequential decisions at all?

Try: Role-play exercise — one student plays a person who was denied a loan by an LLM system and demands an explanation; another plays the system operator trying to explain a chain-of-thought log. Debrief on whether the explanation was meaningful and legally sufficient.

Synthesis/Creation

5. Edge Cases, System Boundaries, and Graceful Degradation

No production system can anticipate every input it will receive. The lecture emphasizes that clearly defined system boundaries, edge case handling, and recovery mechanisms are essential for reliable LLM-based workflows. Edge cases — rare or extreme scenarios outside the training distribution — can cause model outputs to be confidently wrong. Graceful degradation means the system continues to operate at reduced capability rather than failing catastrophically when it encounters such cases.

  • How would you systematically identify the edge cases most likely to be encountered in a deployment of an LLM-based loan approval system, and how would you test for them before launch?
  • What does "graceful degradation" look like concretely for an agentic system that manages hospital appointment scheduling? What is the minimum acceptable behavior when the model fails?
  • How do you communicate system boundaries to end users clearly enough that they do not attempt to use the system for tasks it was not designed to handle?

Try: Have students work in pairs to write a one-page "system boundary card" for a chosen LLM application — listing exactly what it will and will not do, and what happens at the boundary. Share and critique two or three cards as a class.

Discussion Facilitation Tips

  • Use concrete domain scenarios (medical triage, loan approvals, content moderation) rather than abstract agents whenever possible — students engage more deeply when the stakes feel real and the accountability chain is specific.
  • When discussing reward specification, resist the temptation to frame reward hacking as a purely technical problem; emphasize that it is ultimately a problem of value alignment and that technical fixes alone are insufficient without human oversight.
  • For the audit trail discussion, bring in a brief excerpt from actual GDPR or EU AI Act text describing explanation requirements, so students see the regulatory language they would have to satisfy and can assess whether chain-of-thought logs would meet the standard.
  • The multi-agent coordination topic benefits from a physical analogy — compare it to a hospital trauma team (centralized coordination under a lead physician) versus a peer-to-peer architecture (a committee voting on every action), helping students intuit the speed-reliability trade-off before applying it to software systems.