ClusterFusion: Expanding Operator Fusion Scope for LLM Inference via Cluster-Level Collective Primitive

Chen Zhang (Tsinghua University, Tsinghua University) · Minyi Guo (Shanghai Jiao Tong University) · Yu Feng (Shanghai Jiaotong University) · Shihan Fang (Shanghai Jiaotong University) · Zihan Liu (MBZUAI IFM) · Yangjie Zhou (National University of Singapore) · Jingwen Leng (Shanghai Jiao Tong University) · Xinhao Luo (Shanghai Jiao Tong University) · Ziyu Huang (Shanghai Jiaotong University) · Shixuan Sun (National University of Singapore) · Zhenzhe Zheng (Shanghai Jiaotong University)

clusterfusionclustergatherclusterreducecollective communicationdata exchangedecoding latencydistributed shared memoryfragmented executionhigh-speed data exchangeinference frameworkskernel launch overheadlarge language modellow-latency interconnectsoff-chip memoryoperator fusionqkv projection

Large language model (LLM) decoding suffers from high latency due to fragmented execution across operators and heavy reliance on off-chip memory for data exchange and reduction. This execution model limits opportunities for fusion and incurs significant memory traffic and kernel launch overhead. While modern architectures such as NVIDIA Hopper provide distributed shared memory and low-latency intra-cluster interconnects, they expose only low-level data movement instructions, lacking structured abstractions for collective on-chip communication. To bridge this software-hardware gap, we introduce two cluster-level communication primitives, ClusterReduce and ClusterGather, which abstract common communication patterns and enable structured, high-speed data exchange and reduction between thread blocks within a cluster, allowing intermediate results to be on-chip without involving off-chip memory. Building on these abstractions, we design ClusterFusion, an execution framework that schedules communication and computation jointly to expand operator fusion scope by composing decoding stages such as QKV Projection, Attention, and Output Projection into a single fused kernels. Evaluations on H100 GPUs show that ClusterFusion outperforms state-of-the-art inference frameworks by $1.61\times$ on average in end-to-end latency across different models and configurations.