positional encodings
In the context of models like Transformers, positional encodings are added to input embeddings to provide the model with information about the position of items in a sequence, compensating for the lack of inherent sequence information, as these models do not operate on sequences directly.
- Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs
- Bridging Time and Linguistics: LLMs as Time Series Analyzer through Symbolization and Segmentation
- Characterizing the Expressivity of Fixed-Precision Transformer Language Models
- Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models
- Knee-Deep in C-RASP: A Transformer Depth Hierarchy
- LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding
- On Geometry-Enhanced Parameter-Efficient Fine-Tuning for 3D Scene Segmentation
- Return of ChebNet: Understanding and Improving an Overlooked GNN on Long Range Tasks
- Set-LLM: A Permutation-Invariant LLM