Jacobian-Based Interpretation of Nonlinear Neural Encoding Model

Xiaohui Gao (School of Automation, Northwestern Polytechnical University) · Haoran Yang (Northwest Polytechnical University Xi'an) · cheng yue · Mengfei Zuo (Northwestern Polytechnical University, Northwest Polytechnical University Xi'an) · Yiheng Liu (Shaanxi Normal University) · Peiyang Li (Chongqing University of Post and Telecommunications)
activation functionsartificial neural networksbold responsesbrain information processingcortical organizationfunctional magnetic resonance imaginghierarchical progressioninterpretability metricjacobian-based nonlinearity evaluationlocal linear mappingsnetwork architecturesneural encoding modelsnonlinear response patternsnonlinearity quantificationsample-specificitystimulus-selective patterns

In recent years, the alignment between artificial neural network (ANN) embeddings and blood oxygenation level dependent (BOLD) responses in functional magnetic resonance imaging (fMRI) via neural encoding models has significantly advanced research on neural representation mechanisms and interpretability in the brain. However, these approaches remain limited in characterizing the brain’s inherently nonlinear response properties. To address this, we propose the Jacobian-based Nonlinearity Evaluation (JNE), an interpretability metric for nonlinear neural encoding models. JNE quantifies nonlinearity by statistically measuring the dispersion of local linear mappings (Jacobians) from model representations to predicted BOLD responses, thereby approximating the nonlinearity of BOLD signals. Centered on proposing JNE as a novel interpretability metric, we validated its effectiveness through controlled simulation experiments on various activation functions and network architectures, and further verified it on real fMRI data, demonstrating a hierarchical progression of nonlinear characteristics from primary to higher-order visual cortices, consistent with established cortical organization. We further extended JNE with Sample-Specificity (JNE-SS), revealing stimulus-selective nonlinear response patterns in functionally specialized brain regions. As the first interpretability metric for quantifying nonlinear responses, JNE provides new insights into brain information processing. Code available at https://github.com/Gaitxh/JNE.