Revisiting Residual Connections: Orthogonal Updates for Stable and Efficient Deep Networks

Suhwan Choi (Seoul National University) · Giyeong Oh (Yonsei University) · Youngjae Yu (Yonsei University) · Woohyun Cho (Yonsei University) · Siyeol Kim (Yonsei University)

architectural diversitycifarsdeep neural networks feature learninggeneralization accuracyimagenet-1kmodule capacityorthogonal residual updaterepresentation learning residual connectionsresnetv2tinyimagenettraining stabilityupdate strategyvanishing gradients vision transformers

Residual connections are pivotal for deep neural networks, enabling greater depth by mitigating vanishing gradients. However, in standard residual updates, the module’s output is directly added to the input stream. This can lead to updates that predominantly reinforce or modulate the existing stream direction, potentially underutilizing the module’s capacity for learning entirely novel features. In this work, we introduce _Orthogonal Residual Update_: we decompose the module’s output relative to the input stream and add only the component orthogonal to this stream. This design aims to guide modules to contribute primarily new representa-tional directions, fostering richer feature learning while promoting more efficient training. We demonstrate that our orthogonal update strategy improves generalization accuracy and training stability across diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs, TinyImageNet, ImageNet-1k), achieving, for instance, a +3.78 pp Acc@1 gain for ViT-B on ImageNet-1k. Code and models are available at https://github.com/BootsofLagrangian/ortho-residual.