SMMILE: An expert-driven benchmark for multimodal medical in-context learning

Maya Varma (Stanford University) · Jean-Benoit Delbrouck (HOPPR - Stanford) · Melanie Rieff (ETHZ - ETH Zurich) · Ossian Rabow (Stanford University) · Subathra Adithan (Jawaharlal Institute of Postgraduate Medical Education and Research (JIPMER)) · Julie Kim (Stanford University) · Ken Chang (Stanford University) · Hannah Lee (University of California, San Francisco) · Nidhi Rohatgi (Stanford University) · Christian Bluethgen (Stanford University) · Mohamed Muneer (Stanford University) · Michael Moor (ETH Zurich)

augmented variantcontextual learningevaluation metricsexpert-driven benchmarkimaging modalitiesirrelevant examplesmedical specialtiesmedical visual question answeringmultimodal in-context learningmultimodal querymultimodal tasks performance degradationrecency biastask demonstrationszero-shot evaluation

Multimodal in-context learning (ICL) remains underexplored despite significant potential for domains such as medicine. Clinicians routinely encounter diverse, specialized tasks requiring adaptation from limited examples, such as drawing insights from a few relevant prior cases or considering a constrained set of differential diagnoses. While multimodal large language models (MLLMs) have shown advances in medical visual question answering (VQA), their ability to learn multimodal tasks from context is largely unknown. We introduce SMMILE, the first expert-driven multimodal ICL benchmark for medical tasks. Eleven medical experts curated problems, each including a multimodal query and multimodal in-context examples as task demonstrations. SMMILE encompasses 111 problems (517 question-image-answer triplets) covering 6 medical specialties and 13 imaging modalities. We further introduce SMMILE++, an augmented variant with 1038 permuted problems. A comprehensive evaluation of 15 MLLMs demonstrates that most models exhibit moderate to poor multimodal ICL ability in medical tasks. In open-ended evaluations, ICL contributes only an 8% average improvement over zero-shot on SMMILE and 9.4% on SMMILE++. We observe a susceptibility for irrelevant in-context examples: even a single noisy or irrelevant example can degrade performance by up to 9.5%. Moreover, we observe that MLLMs are affected by a recency bias, where placing the most relevant example last can lead to substantial performance improvements of up to 71%. Our findings highlight critical limitations and biases in current MLLMs when learning multimodal medical tasks from context. SMMILE is available at https://smmile-benchmark.github.io.