Simpson's Paradox and its Implications in Data Science - KDnuggets

Abstraction: Aggregate trends reversing upon disaggregation mislead data-driven decisions

Key points:

• Simpson's Paradox (named 1972 by Colin R. Blyth): a trend present in aggregated data reverses or disappears when the data is split into subgroups
• UC Berkeley 1973: overall admission rates showed 44% male vs 35% female; disaggregating by department revealed statistical bias in favor of women in 4 of 6 depts — women applied disproportionately to more competitive departments (the hidden confounding variable)
• Same dataset can yield opposite conclusions depending on grouping choice; forces practitioners to find the "best" conclusion that fairly represents the data
• Real-time data analytics is especially vulnerable: short time windows can hide true long-run trends
• Excessive aggregation introduces bias; insufficient aggregation limits discoverable patterns — the paradox defines a calibration problem

Connections: Statistics · Data Science · Simpsons Paradox · Confounding Variables

Source: https://www.kdnuggets.com/2023/03/simpson-paradox-implications-data-science.html