← Back to Week 14

Week 14: LLMs as Decision Makers and Agents

Knowledge Check — Select an answer to see immediate feedback.

Questions 10

Question 1

When using an LLM as a decision maker, structured output formats such as JSON are preferred over free-form text. What is the primary reason for this?

A) JSON outputs require fewer tokens and therefore reduce API costs
B) Structured formats make the decision interpretable and easier to integrate with downstream systems by capturing the selected option, rationale, and metadata in a consistent schema
C) Free-form text outputs always contain hallucinations, while JSON outputs do not
D) Structured formats eliminate the need for confidence scoring

Question 2

Confidence scoring is an important component of LLM-based decision frameworks. What is its primary purpose?

A) To rank competing LLM providers by their average response accuracy
B) To allow the model to communicate the reliability of its output, enabling downstream systems or humans to assess trustworthiness before acting on a decision
C) To automatically retrain the model when its confidence falls below a threshold
D) To measure the computational cost of each decision in terms of FLOPs

Question 3

In agentic workflows, what does the agent's "action space" define?

A) The memory capacity available to the agent for storing past observations
B) The set of actions the agent can perform, subject to the constraints and limitations imposed by the environment and the agent's capabilities
C) The maximum number of tokens the agent can generate per request
D) The physical hardware on which the agent is deployed

Question 4

An AI agent can take a shortcut that gives a small reward right now, or take a longer path that gives a bigger reward later. What concept in agent design controls how much the agent values future rewards compared to immediate ones?

A) The learning rate — how quickly the agent updates its beliefs based on new information
B) The temperature — how randomly the agent explores new actions
C) The discount factor — how much the agent values rewards that arrive in the future vs. right now
D) The confidence threshold — how certain the agent must be before committing to an action

Question 5

Fallback mechanisms in LLM decision frameworks are designed to handle situations where the model cannot make a confident decision. Which of the following is a valid example of a fallback strategy?

A) Automatically accepting the model's output regardless of confidence score to maintain throughput
B) Deferring to a human expert or a more specialized model when the LLM's confidence score falls below a threshold
C) Ignoring edge cases because they represent a small fraction of production traffic
D) Switching to a higher-temperature sampling setting to generate more diverse outputs

Question 6

In router architectures for LLM-based data science workflows, what is the primary role of feedback loops?

A) To batch requests together to reduce latency
B) To enable the system to learn from past routing decisions and adapt over time, so the workflow remains optimized as conditions change
C) To duplicate data across multiple nodes for redundancy
D) To translate structured outputs back into natural language for human review

Question 7

Conflict resolution in multi-agent systems is needed when agents' goals or actions are incompatible. Which approach uses predefined rules and constraints to guide resolution rather than having agents negotiate directly?

A) Negotiation-based resolution
B) Consensus-based resolution
C) Rule-based resolution
D) Auction-based resolution

Question 8

Decision logging and audit trails are cited as key components of safety and reliability in LLM-based data science systems. What primary function do they serve?

A) They compress model outputs to reduce storage costs
B) They enable tracking and analysis of system behavior over time, facilitating identification of issues and supporting corrective measures and accountability
C) They automatically roll back the model to a previous version when errors are detected
D) They replace the need for human oversight by providing an automated review pipeline

Question 9

Parallel processing is described as a control flow mechanism in agentic workflows. In which scenario is it most beneficial compared to sequential decision-making?

A) When each step of a task strictly depends on the outcome of the previous step
B) When the agent needs to process large volumes of data or coordinate with multiple entities simultaneously in real time
C) When the action space contains only a single available action at each time step
D) When the environment is fully deterministic and the optimal policy is already known

Question 10

System boundaries are described as critical for defining the scope of LLM-based decision systems. What is the main risk of deploying a system without clearly defined boundaries?

A) The system will always refuse to generate outputs outside the training distribution
B) The model may exceed its intended purpose and make decisions in domains where it lacks sufficient knowledge or expertise, potentially causing unintended consequences
C) The system will require more GPU memory at inference time
D) Users will be unable to provide natural language inputs to the system