Automatic Synthetic Data and Fine-grained Adaptive Feature Alignment for Composed Person Retrieval

Fei Su (Beijing University of Posts and Telecommunications) · Delong Liu (Beijing University of Posts and Telecommunications) · Haiwen Li (Beijing University of Posts and Telecommunications) · Zhaohui Hou (Sensetime) · Zhicheng Zhao (Beijing University of Posts and Telecommunications) · Yuan Dong (Beijing University of Posts and Telecommunications)

composed person retrievaldata synthesis pipelinedynamic alignmentfine-grained adaptive feature alignmentgenerative modelshigh-quality synthetic tripletsidentity-consistent image synthesisimage-text composed person retrievalmasked feature reasoningmultimodal data generationmultimodal filteringperson retrievalrepresentation learning state-of-the-art methodssyncpr datasettextual quadruples

Person retrieval has attracted rising attention. Existing methods are mainly divided into two retrieval modes, namely image-only and text-only. However, they are unable to make full use of the available information and are difficult to meet diverse application requirements. To address the above limitations, we propose a new Composed Person Retrieval (CPR) task, which combines visual and textual queries to identify individuals of interest from large-scale person image databases. Nevertheless, the foremost difficulty of the CPR task is the lack of available annotated datasets. Therefore, we first introduce a scalable automatic data synthesis pipeline, which decomposes complex multimodal data generation into the creation of textual quadruples followed by identity-consistent image synthesis using fine-tuned generative models. Meanwhile, a multimodal filtering method is designed to ensure the resulting SynCPR dataset retains 1.15 million high-quality and fully synthetic triplets. Additionally, to improve the representation of composed person queries, we propose a novel Fine-grained Adaptive Feature Alignment (FAFA) framework through fine-grained dynamic alignment and masked feature reasoning. Moreover, for objective evaluation, we manually annotate the Image-Text Composed Person Retrieval (ITCPR) test set. The extensive experiments demonstrate the effectiveness of the SynCPR dataset and the superiority of the proposed FAFA framework when compared with the state-of-the-art methods. All code and data will be provided at https://github.com/Delong-liu-bupt/Composed_Person_Retrieval.