[99] Hyping Fisher: The Most Cited 2019 QJE Paper Relied on an Outdated Stata Default to Conclude Regression p-values Are Inadequate

Abstraction: Data Colada showing QJE randomization-test conclusion collapses when using HC3 robust standard errors

Key points:

• Young (2019 QJE), most-cited QJE paper of 2019 (372 Google cites), claimed randomization tests are "superior to all other methods" over regression p-values
• Conclusion relied on Stata's default HC1 robust standard errors, which Long & Ervin (2000) showed to perform poorly for N<250; HC3 should be used instead
• R's default sandwich package uses HC3; when HC3 is applied, the supposed superiority of randomization tests disappears
• Five variants of robust SEs: HC0–HC4; Stata defaults to HC1, which underperforms on "small" samples (N=20 or N=200 used in QJE simulations)
• HC1 false-positive rates reached 21% vs nominal 5% in simulations; HC3 brings this in line, matching or outperforming randomization tests
• Author notes randomization tests still useful for non-standard test statistics and complex data structures, just not as a general replacement for regression

Connections: Data Colada · Robust Standard Errors · P Values · Statistics · Hypothesis Testing

Source: https://datacolada.org/99