Bert

entities · 5 notes linked

Related: Transformers · Large Language Models · Attention Mechanism · Roberta · Knowledge Distillation · Google Brain · Self Attention · Google

Notes

• DeepMind Paper Provides a Mathematically Precise Overview of Transformer Architectures and Algorithms | Synced — DeepMind formal pseudocode reference for all major transformer architectures
• Google Replaces BERT Self-Attention with Fourier Transform: 92% Accuracy, 7 Times Faster on GPUs — FNet replaces transformer self-attention with Fourier Transform for faster training
• Transforming the Retail Industry with Transformers — BERT transformers automating retail product attribute classification in production
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension — Hybrid convolutional-transformer model via knowledge distillation for fast NLU inference
• Why multi-head self attention works: math, intuitions and 10+1 hidden insights | AI Summer — Mathematical and intuitive deep-dive into multi-head self-attention mechanisms