Learning Task-Agnostic Representations through Multi-Teacher Distillation

Jackie CK Cheung (Mila / McGill University) · Ismail Ayed (ETS Montreal) · Eric Granger (ETS Montreal) · Philippe Formont (École de technologie supérieure, Université Paris-Saclay, MILA) · Maxime Darrin (Mcgill University / Paris Saclay University) · Banafsheh Karimian (École de technologie supérieure) · Mohammadhadi Shateri (École de technologie supérieure, Université du Québec) · Pablo Piantanida (ILLS & MILA | CNRS Paris-Saclay University)

architectures classification clusteringdistillation lossdownstream tasks embedding models input modalities loss functionsmajority vote objectivemolecular modelingmulti-teacher distillationmutual information regression representationstask-agnostic frameworkteacher diversity

Casting complex inputs into tractable representations is a critical step across various fields. Diverse embedding models emerge from differences in architectures, loss functions, input modalities and datasets, each capturing unique aspects of the input. Multi-teacher distillation leverages this diversity to enrich representations but often remains tailored to specific tasks. We introduce a task-agnostic framework based on a ``majority vote" objective function. We demonstrate that this function is bounded by the mutual information between the student and the teachers' embeddings, leading to a task-agnostic distillation loss that eliminates dependence on task-specific labels or prior knowledge. Comprehensive evaluations across text, vision models, and molecular modeling show that our method effectively leverages teacher diversity, resulting in representations enabling better performance for a wide range of downstream tasks such as classification, clustering, or regression. Additionally, we train and release state-of-the-art embedding models, enhancing downstream performance in various modalities.