6 Tricks I Learned From The OTTO Kaggle Challenge

kagglemachine-learningensemblegradient-boostinghyperparameter-tuning

Abstraction: Practical Kaggle competition tricks for ensembling and hyperparameter search

Key points:

Stacking pipeline: add N classifier predictions as new features to 93 original features, retrain P classifiers, then take weighted average; harmonic mean and logit-transformed predictions both improved score
scikit-learn 0.16 calibration rescales predicted probabilities against actual truth ratios within probability segments, meaningfully improving log-loss scores
GBM key hyperparameters beyond learning rate and n_estimators: subsample (fraction of samples per tree) and max_features (features per split) were crucial but often overlooked
RandomizedSearchCV outperforms exhaustive GridSearchCV when parameter space is large; typically finds good params in a few hours vs. days
XGBoost significantly faster than scikit-learn's GBM and yields better predictions; became a standard Kaggle tool
BaggingClassifier meta-estimator wraps arbitrary classifiers to average multiple random-subset fits without manual looping

Connections: Kaggle · Xgboost · Scikit Learn · Ensemble Learning · Gradient Boosting · Hyperparameter Optimization