SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs

zhicheng li (ICT) · Shuoming Zhang (Institute of Computing Technology, Chinese Academy of Sciences) · Jiacheng Zhao (Institute of Computing Technology, Chinese Academy of Sciences) · Siqi Li (Beijing University of Technology) · Xiyu Shi (KTH Royal Institute of Technology) · Yangyu Zhang (University of the Chinese Academy of Sciences) · Shuaijiang Li (, Chinese Academy of Sciences) · Donglin Yu (University of Illinois at Urbana-Champaign) · Zheming Yang (Institute of Computing Technology, Chinese Academy of Science) · YUAN WEN (University of Aberdeen) · Huimin Cui (Institute of Computing Technology, Chinese Academy of Sciences)
completion latencycompute-intensivecost-model-guided schedulerdecoder arrivalsdynamic resource allocationfine-grained sm partitioningmemory-lightmodality-specific encodersmultimodal large language modelsnvidia a100 gpusperformance evaluationshared text decoderspace-inference schedulerspace-multiplexingtime-windowed shortest-remaining-first policy

Recent multimodal large language models (MLLMs) marry modality-specific vision or audio encoders with a shared text decoder. While the encoder is compute- intensive but memory-light, the decoder is the opposite, yet state-of-the-art serving stacks still time-multiplex these complementary kernels, idling SMs or HBM in turn. We introduce SpaceServe, a serving system that space-multiplexes MLLMs: it decouples all modality encoders from the decoder, and co-locates them on the same GPU using fine-grained SM partitioning available in modern runtimes. A cost-model-guided Space-Inference Scheduler (SIS) dynamically assigns SM slices, while a Time-Windowed Shortest-Remaining-First (TWSRFT) policy batches en- coder requests to minimise completion latency and smooth decoder arrivals. Evaluation shows that SpaceServe reduces time-per-output-token by 4.81× on average and up to 28.9× on Nvidia A100 GPUs. SpaceServe is available at https://github.com/gofreelee/SpaceServe