Self-supervised learning is the future of AI

self-supervised-learningdeep-learningtransformersyann-lecun

Abstraction: LeCun's blueprint for self-supervised learning over supervised AI

Key points:

LeCun clarifies deep learning's bottleneck is supervised learning, not deep learning itself — deep learning can be applied to self-supervised paradigms
Self-supervised = "learning to fill in the blanks": mask part of input (text word, video frame) and train model to predict the missing piece
Transformers (BERT, RoBERTa, GPT-2) are the closest existing approximation to self-supervised systems, excelling in discrete text but failing at continuous image/video due to uncertainty representation
Core unsolved problem: representing a distribution over all possible video frames — averaging possible outcomes produces blurry output
LeCun's proposed solution: latent variable energy-based models, where variable Z selects best-compatible future Y for input X
"If AI is a cake, self-supervised learning is the bulk of the cake" — next revolution will be neither purely supervised nor purely reinforcement learning

Connections: Yann LeCun · Yoshua Bengio · Self Supervised Learning · Transformers · Energy Based Models