VanillaNet — A Novel Neural Network Architecture for Computer Vision

Abstraction: Simple convolutional network without shortcuts or attention matches complex model performance

Key points:

• VanillaNet proposed by Huawei Noah's Ark Lab and University of Sydney; avoids shortcuts, self-attention, and excessive depth unlike ResNets and ViTs
• Uses a "deep training" technique: starts with non-linear activation layers, then progressively removes them during training to enable fast inference
• Introduces a series-based activation function with learnable affine transformations to maintain non-linearity
• Targets low-resource deployment contexts where complex architectures create off-chip memory and implementation overhead
• Outperforms complex architectures in efficiency while retaining competitive accuracy; PyTorch implementation on GitHub
• Challenges the dominant paradigm that depth and complexity are required for high CV performance

Connections: Huawei Noahs Ark Lab · Neural Network Architecture · Computer Vision · Deep Learning

Source: https://www.marktechpost.com/2023/07/12/a-new-ai-research-proposes-vanillanet-a-novel-neural-network-architecture-emphasizing-the-elegance-and-simplicity-of-design-while-retaining-remarkable-performance-in-computer-vision-tasks/