BOOM: Benchmarking Out-Of-distribution Molecular Property Predictions of Machine Learning Models

Bhavya Kailkhura (Lawrence Livermore National Laboratory) · James Diffenderfer (Lawrence Livermore National Laboratory) · Tal Ben-Nun (Lawrence Livermore National Laboratory) · Evan Antoniuk (Lawrence Livermore National Laboratory) · Shehtab Zaman (Binghamton University) · Peggy Li (Lawrence Livermore National Labs) · Busra Sahin (State University of New York at Binghamton) · Obadiah Smolenski (SUNY Binghamton) · Everett Grethel (University of California, Los Angeles) · Tim Hsu (Lawrence Livermore National Labs) · Anna Hiszpanski (Lawrence Livermore National Labs) · Kenneth Chiu (Binghamton University) · Brian Van Essen (Lawrence Livermore National Labs)

ablation experiments benchmarkschemical foundation modelsdata generationdata-driven molecular discoverydeep learning models generative modeling hyperparameter optimization inductive biasmolecular property predictionsmolecular representationood performanceout-of-distribution predictions

Data-driven molecular discovery leverages artificial intelligence/machine learning (AI/ML) and generative modeling to filter and design novel molecules. Discovering novel molecules requires accurate out-of-distribution (OOD) predictions, but ML models struggle to generalize OOD. Currently, no systematic benchmarks exist for molecular OOD prediction tasks. We present BOOM, $\textbf{b}$enchmarks for $\textbf{o}$ut-$\textbf{o}f$-$\textbf{d}$istribution $\textbf{m}$olecular property predictions: a chemically-informed benchmark for OOD performance on common molecular property prediction tasks. We evaluate over 150 model-task combinations to benchmark deep learning models on OOD performance. Overall, we find that no existing model achieves strong generalization across all tasks: even the top-performing model exhibited an average OOD error 3$\times$ higher than in-distribution. Current chemical foundation models do not show strong OOD extrapolation, while models with high inductive bias can perform well on OOD tasks with simple, specific properties. We perform extensive ablation experiments, highlighting how data generation, pre-training, hyperparameter optimization, model architecture, and molecular representation impact OOD performance. Developing models with strong OOD generalization is a new frontier challenge in chemical ML. This open-source benchmark is available at https://github.com/FLASK-LLNL/BOOM