WaLDORf: Wasteless Language-model Distillation On Reading-comprehension

Abstraction: Hybrid convolutional-transformer model via knowledge distillation for fast NLU inference

Key points:

• Targets deployment problem of Very Large Language Models (BERT, XLNet, RoBERTa) which are resource-intensive at inference time
• WaLDORf is a task-specific hybrid of convolutional and transformer layers
• Achieves state-of-the-art inference speed among distilled models while exceeding their accuracy
• Focuses on reading-comprehension as the target NLU task
• Presents novel distillation techniques rather than simple layer-dropping approaches

Connections: Bert · Roberta · Knowledge Distillation · Transformers

Source: https://arxiv.org/abs/1912.06638