Revisiting Glorot Initialization for Long-Range Linear Recurrences
dimension-aware rescalingexploding signalsglorot initializationhidden stateinitializationinstabilitylong-range reasoninglong-sequence regimerecurrent initializationrecurrent neural networkssignal propagationspectral radiusstability theorytheoretical analysisvanishing signals
Proper initialization is critical for Recurrent Neural Networks (RNNs), particularly in long-range reasoning tasks, where repeated application of the same weight matrix can cause vanishing or exploding signals. A common baseline for linear recurrences is Glorot initialization, designed to ensure stable signal propagation