residual stream
In the context of neural networks, the residual stream refers to the mechanism of bypassing or 'skipping' certain layers in a deep architecture to alleviate the vanishing gradient problem and facilitate the learning of deep representations, often seen in ResNet architectures.
- Attention Sinks: A 'Catch, Tag, Release' Mechanism for Embeddings
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models
- Counterfactual reasoning: an analysis of in-context emergence
- Dense SAE Latents Are Features, Not Bugs
- Do Language Models Use Their Depth Efficiently?
- Interpreting vision transformers via residual replacement model
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models