gradient projection
A technique used in optimization where gradients are projected onto feasible regions of the parameter space to ensure that updates remain within valid constraints during training.
- Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretraining
- Continual Multimodal Contrastive Learning
- Continuous Subspace Optimization for Continual Learning
- Turning the Tables: Enabling Backward Transfer via Causal-Aware LoRA in Continual Learning
- What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions