A Short Chronology Of Deep Learning For Tabular Data

tabular-datadeep-learningxgboosttransformerssurveygradient-boosting

Abstraction: Chronological survey comparing deep learning versus gradient boosting on tabular data

Key points:

Sebastian Raschka's curated timeline (2019–2022) of ~20+ deep tabular learning papers; key architectures include TabNet (2019), NODE (2019), FT-Transformer (2021), SAINT (2021), TabPFN (2022), TabDDPM (2022), and GReaT (2022).
Consistent finding: gradient-boosted tree ensembles (XGBoost, LightGBM, CatBoost) still mostly outperform deep learning on medium-sized tabular datasets; the gap narrows as dataset size grows to 50k+ examples.
Uninformative features harm deep learning more than tree-based methods; well-tuned MLPs with modern regularization (13 techniques evaluated) can outperform specialized architectures and gradient boosting on many datasets.
TabPFN introduces prior-data fitted networks combining Bayesian inference and transformer tokenization; requires no training on new data but scales quadratically with dataset size.
Practical recommendation: start with random forests, then HistGradientBoosting, then XGBoost; only experiment with deep learning if time permits and dataset is large enough.

Connections: Sebastian Raschka · Xgboost · Deep Learning · Tabular Data · Gradient Boosting · Transformers