How to implement an RNN (2/2) - Tensor data and non-linearities

Abstraction: From-scratch NumPy RNN with tensor input, tanh activations, and RMSProp optimizer

Key points:

• Training data stored as 3rd-order tensor (samples × timesteps × features); np.tensordot enables parallel computation across all samples and timesteps at once
• Glorot initialization: weights sampled uniformly from ±√(6/(n_in + n_out)) to control variance
• tanh preferred over logistic for state updates because its maximum gradient (1.0) is higher than logistic's (0.25), reducing vanishing gradient risk
• RProp replaced by RMSProp because RProp fails with minibatches — stochastic error surface causes sign changes that break RProp's sign-based update rule
• RMSProp normalizes gradients by dividing by √(moving-average of squared gradient); λ=0 reduces it to sign-based update
• Nesterov momentum first moves parameters by current velocity, then computes gradient at new position — more informative correction than classical momentum

Connections: Recurrent Neural Networks · Backpropagation Through Time · Gradient Descent

Source: https://peterroelants.github.io/posts/rnn-implementation-part02/