Bayes' Theorem

The Foundation of Probabilistic Reasoning

$$P(A|B) = \frac{P(B|A) \times P(A)}{P(B)}$$

Named after: Reverend Thomas Bayes (1701-1761)

Core Principle: How to update our beliefs when we observe new evidence

Foundation of machine learning and artificial intelligence
Bridges prior knowledge with observed data
Quantifies uncertainty in a principled way
Essential for probabilistic models and decision making

Conditional Probability

The Building Block

Conditional Probability: The probability of event A occurring given that event B has occurred $$P(A|B) = \frac{P(A \cap B)}{P(B)}$$

Example: What's the probability it's raining given that you see people with umbrellas?

$P(\text{Rain}|\text{Umbrellas})$ = ?
We need to consider how often people carry umbrellas when it rains
And how often it rains in general

            Key Insight: $P(A|B) \neq P(B|A)$ in general

            The probability of rain given umbrellas ≠ probability of umbrellas given rain

Deriving Bayes' Theorem

From Basic Probability Rules

Starting with conditional probability:

$$P(A|B) = \frac{P(B|A) \times P(A)}{P(B)}$$

            This is Bayes' Theorem! A simple rearrangement of conditional probability definitions
        

Components of Bayes' Theorem

Understanding Each Term

$$P(\text{Hypothesis}|\text{Evidence}) = \frac{P(\text{Evidence}|\text{Hypothesis}) \times P(\text{Hypothesis})}{P(\text{Evidence})}$$

Four Key Components:

Prior P(A): What we believe before seeing evidence
- Our initial belief about the hypothesis
Likelihood P(B|A): How well the evidence fits our hypothesis
- How probable is this evidence if our hypothesis is true?
Evidence P(B): The probability of observing the evidence
- Normalizing constant (often computed via law of total probability)
Posterior P(A|B): Updated belief after seeing evidence
- What we believe after incorporating the new information

Intuition: Posterior ∝ Likelihood × Prior

Medical Diagnosis Example

Disease Testing Scenario

Problem Setup:

Disease affects 1% of population (base rate)
Test is 95% accurate (both sensitivity and specificity)
You test positive. What's the probability you have the disease?

Given information:
• P(Disease) = 0.01 (prior)
• P(Positive|Disease) = 0.95 (likelihood)
• P(Positive|No Disease) = 0.05

Need to find: P(Disease|Positive) = ?

$$P(\text{Disease}|\text{Positive}) = \frac{P(\text{Positive}|\text{Disease}) \times P(\text{Disease})}{P(\text{Positive})}$$

Intuitive guess: 95%? Actual answer: Much lower!

Medical Example: Full Calculation

Step-by-Step Solution

Step 1: Calculate P(Positive) using law of total probability
P(Positive) = P(Positive|Disease) × P(Disease) + P(Positive|No Disease) × P(No Disease)
P(Positive) = 0.95 × 0.01 + 0.05 × 0.99
P(Positive) = 0.0095 + 0.0495 = 0.059

Step 2: Apply Bayes' theorem
P(Disease|Positive) = P(Positive|Disease) × P(Disease) / P(Positive)
P(Disease|Positive) = (0.95 × 0.01) / 0.059 ≈ 0.161

$$P(\text{Disease}|\text{Positive}) = 16.1\%$$

            Key Insight: Even with a 95% accurate test, a positive result only gives 16% chance of having the disease!
        

Base Rate Fallacy: Ignoring the low prevalence (1%) leads to massive overestimation of risk

Bayesian vs. Frequentist Perspectives

Two Philosophies of Probability

Aspect	Frequentist	Bayesian
Probability	Long-run frequency of events	Degree of belief or uncertainty
Parameters	Fixed but unknown constants	Random variables with distributions
Prior Knowledge	Not formally incorporated	Explicitly included via priors
Inference	Confidence intervals, p-values	Credible intervals, posterior distributions
Uncertainty	About the procedure	About the parameter values

When to use each:

Frequentist: Large datasets, objective analysis, regulatory approval
Bayesian: Small datasets, prior knowledge available, decision making under uncertainty

Applications in Machine Learning

Bayes' Theorem in Action

Key Applications:

Naive Bayes Classifier:
- Text classification, spam detection
- Assumes feature independence (hence "naive")
Bayesian Optimization:
- Hyperparameter tuning
- Expensive function optimization
Bayesian Neural Networks:
- Uncertainty quantification in deep learning
- Distributions over weights instead of point estimates
A/B Testing:
- Online experimentation
- Sequential decision making
Probabilistic Programming:
- PyMC, Stan, Edward
- Complex hierarchical models

$$P(\text{Class}|\text{Features}) = \frac{P(\text{Features}|\text{Class}) \times P(\text{Class})}{P(\text{Features})}$$

Common Misconceptions

Avoiding Pitfalls

Misconception 1: Confusing P(A|B) with P(B|A)

P(Positive Test|Disease) ≠ P(Disease|Positive Test)
These can be dramatically different!

Misconception 2: Ignoring Base Rates (Priors)

Rare events stay rare even with strong evidence
Prior probability matters enormously

Misconception 3: Assuming Independence

Naive Bayes assumes features are independent
Often violated in practice, but algorithm still works well

Pro Tips:

Always consider the base rate
Be careful about the direction of conditioning
Check independence assumptions
Use natural frequencies instead of percentages for intuition

Key Takeaways

Essential Points to Remember

$$P(\text{Hypothesis}|\text{Evidence}) = \frac{P(\text{Evidence}|\text{Hypothesis}) \times P(\text{Hypothesis})}{P(\text{Evidence})}$$

Foundation of Probabilistic Reasoning:
- Formalizes how to update beliefs with evidence
Prior Knowledge Matters:
- Base rates and prior probabilities are crucial
Uncertainty Quantification:
- Provides principled way to handle uncertainty
Machine Learning Foundation:
- Underlies many algorithms and approaches
Decision Making Tool:
- Optimal decisions under uncertainty

            Remember: Bayes' theorem is not just a formula—it's a way of thinking about evidence, belief updating, and rational decision making in the face of uncertainty.
        

Next Steps: Explore Bayesian statistics, probabilistic programming, and Bayesian machine learning methods