QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks

Yin Zhang (The Chinese University of Hong Kong, Shenzhen) · Qian Chen (Alibaba Group) · Linxin Yang (The Chinese University of Hong Kong, Shenzhen) · Akang Wang (Shenzhen Research Institute of Big Data) · Xiaodong Luo (Chinese University of HongKong (Shenzhen))

architectural enhancementcomputational burden deep neural networks fine-tuning image classificationlightweight quadratic enhancermatrix decompositionmodel performancenonlinear activation functionsparameter minimizationperformance gainsquadratic transformationssparsification techniquestext classificationweight sharing

The combination of linear transformations and nonlinear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex functions. This paper explores the introduction of quadratic transformations to further increase the nonlinearity of the model, with the aim of enhancing the performance of existing architectures. To minimize the additional parameters and computational burden, we propose a lightweight quadratic enhancer that leverages matrix decomposition, weight sharing, and sparsification techniques. This approach introduces only a minimal and negligible increase in parameters and forward computation, while still yielding substantial improvements in model performance. We evaluate the effectiveness of the proposed method across three tasks: text classification, image classification, and fine-tuning large language models (LLMs). In all tasks, our approach demonstrates significant performance gains.