key-value cache
A data storage mechanism used in machine learning models to store and retrieve data efficiently, particularly in transformer models. The key-value cache enhances computation speed and memory management during training and inference.
- Activated LoRA: Fine-tuned LLMs for Intrinsics
- Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
- AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
- EasySpec: Layer-Parallel Speculative Decoding for Efficient Multi-GPU Utilization
- Efficient Low Rank Attention for Long-Context Inference in Large Language Models
- Efficient Prompt Compression with Evaluator Heads for Long-Context Transformer Inference
- FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering
- Hogwild! Inference: Parallel LLM Generation via Concurrent Attention
- Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads
- Inference-Time Hyper-Scaling with KV Cache Compression
- InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understanding
- KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse
- Multi-head Temporal Latent Attention
- NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cache
- PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation
- R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
- SALS: Sparse Attention in Latent Space for KV Cache Compression
- Spotlight Attention: Towards Efficient LLM Generation via Non-linear Hashing-based KV Cache Retrieval
- TransMLA: Migrating GQA Models to MLA with Full DeepSeek Compatibility and Speedup