FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing

Min Zhang (Harbin Institute of Technology, Shenzhen) · Shoutao Guo (Institute of computing technology, Chinese Academy of Sciences) · Shaolei Zhang (Renmin University of China) · Qingkai Fang (Institute of Computing Technology, Chinese Academy of Sciences) · Zhengrui Ma (Institute of Computing Technology, Chinese Academy of Sciences) · Yang Feng (Institute of Computing Technology, Chinese Academy of Sciences)
compression ratioscomputational costsdynamic compression traininginference efficiencyiterative fusion strategylarge speech-language modelslong-form speechlong-speech taskslong-speech training datasetslong-speech understanding benchmarklongspeech-evalmodel adaptationshort-speech sequencesspeech generationspeech understanding

The rapid advancement of Large Language Models (LLMs) has spurred significant progress in Large Speech-Language Models (LSLMs), enhancing their capabilities in both speech understanding and generation. While existing LSLMs often concentrate on augmenting speech generation or tackling a diverse array of short-speech tasks, the efficient processing of long-form speech remains a critical yet underexplored challenge. This gap is primarily attributed to the scarcity of long-speech training datasets and the high computational costs associated with long sequences. To address these limitations, we introduce FastLongSpeech, a novel framework designed to extend LSLM capabilities for efficient long-speech processing without necessitating dedicated long-speech training data. FastLongSpeech incorporates an iterative fusion strategy that can compress excessively long-speech sequences into manageable lengths. To adapt LSLMs for long-speech inputs, it introduces a dynamic compression training approach, which exposes the model to short-speech sequences at varying compression ratios, thereby transferring the capabilities of LSLMs to long-speech tasks. To assess the long-speech capabilities of LSLMs, we develop a long-speech understanding benchmark called LongSpeech-Eval. Experiments show that our method exhibits strong performance in both long-speech and short-speech tasks, while greatly improving inference efficiency.