Self-supervised learning: The plan to make deep learning data-efficient - TechTalks

self-supervised-learningdeep-learningyann-lecunaaai-2020energy-based-modelsdata-efficiency

Abstraction: Yann LeCun's AAAI 2020 roadmap for data-efficient self-supervised learning

Key points:

LeCun argues that data hunger is a limitation of supervised learning specifically, not deep learning in general; the fix is self-supervised learning ("learning to fill in the blanks")
Three challenges: (1) learning from fewer samples, (2) making deep learning systems capable of reasoning, (3) enabling planning of complex action sequences decomposed into subtasks
Transformers are the closest existing self-supervised systems; they underpin BERT, RoBERTa, GPT-2, and Meena, but their success has not transferred to visual/video data due to high-dimensional continuous output uncertainty
For video, LeCun proposes "latent variable energy-based models" — a latent variable Z scores compatibility between current state X and predicted future Y, selecting the highest-compatibility outcome
"If artificial intelligence is a cake, self-supervised learning is the bulk of the cake. The next revolution in AI will not be supervised, nor purely reinforced."

Connections: Yann LeCun · Facebook AI · Self Supervised Learning · Deep Learning · Transformers