NegoCollab: A Common Representation Negotiation Approach for Heterogeneous Collaborative Perception

Bo Chen (Beijing University of Posts and Telecommunications) · Rui Pan (Beijing University of Posts and Telecommunications) · CONGZHANG SHAO (Beijing University of Posts and Telecommunications) · Quan Yuan (Beijing University of Posts and Telecommunications) · Guiyang Luo (Beijing University of Posts and Telecommunications) · Yue Hu (University of Michigan - Ann Arbor) · Danni Wang (Beijing University of Posts and Telecommunications) · Liu Yilin (Beijing University of Posts and Telecommunications) · Jinglin Li (Beijing University of Posts and Telecommunications)

bidirectional transformationcollaborative perceptioncommon representationdistribution alignment lossdomain gaps feature alignmentheterogeneous collaborationimmutable heterogeneityknowledge distillationlocal representation spacemultimodal representationnegotiated common representationpaired sender-receiverpragmatic alignment lossstructural alignment losstraining cost

Collaborative perception expands the perception range by sharing information among agents, effectively improving task performance. Immutable heterogeneity poses a significant challenge in collaborative perception, as participating agents may employ different and fixed perception models. This leads to domain gaps in the intermediate features shared among agents, consequently degrading collaborative performance. Aligning the features of all agents to a common representation can eliminate domain gaps with low training cost. However, in existing methods, the common representation is designated as the representation of a specific agent, making it difficult for agents with significant domain discrepancies from this specific agent to achieve proper alignment. This paper proposes NegoCollab, a heterogeneous collaboration method based on negotiated common representation. It achieves bidirectional transformation of each modality's features between local representation space and common representation space through paired sender-receiver, thereby eliminating domain gaps. The common representation in NegoCollab is negotiated from local representations of each modality's agent via a negotiator introduced during training, effectively reducing inherent domain discrepancies with each local representation. Furthermore, to better align local representations with the multimodal common representation, we introduce both structural alignment loss and pragmatic alignment loss alongside the conventional distribution alignment loss during supervised training, enabling comprehensive knowledge distillation from the common representation to the senders. The experimental results demonstrate that NegoCollab significantly outperforms existing methods in common representation-based collaboration approaches. The negotiation-based mechanism for acquiring common representations provides more diverse and reliable alternatives for establishing common representations required in heterogeneous collaboration perception.