MSTAR: Box-free Multi-query Scene Text Retrieval with Attention Recycling

Xiang Bai (Huazhong University of Science and Technology) · Zhang Li (Huazhong University of Science and Technology) · Yuliang Liu (Huazhong University of Science and Technology) · Liang Yin (Huazhong University of Science and Technology) · Xudong Xie (Huazhong University of Science and Technology)

attention recyclingbenchmark datasetbounding box annotationsfree-style text queriesmap performancemulti-grained representationmulti-instance matchingmulti-querymulti-query text retrievalprogressive vision embeddingretrieval strategyscene text retrievalstyle-aware instructionstext localization vision-language alignment

Scene text retrieval has made significant progress with the assistance of accurate text localization. However, existing approaches typically require costly bounding box annotations for training. Besides, they mostly adopt a customized retrieval strategy but struggle to unify various types of queries to meet diverse retrieval needs. To address these issues, we introduce Multi-query Scene Text retrieval with Attention Recycling (MSTAR), a box-free approach for scene text retrieval. It incorporates progressive vision embedding to dynamically capture the multi-grained representation of texts and harmonizes free-style text queries with style-aware instructions. Additionally, a multi-instance matching module is integrated to enhance vision-language alignment. Furthermore, we build the Multi-Query Text Retrieval (MQTR) dataset, the first benchmark designed to evaluate the multi-query scene text retrieval capability of models, comprising four query types and $16k$ images. Extensive experiments demonstrate the superiority of our method across seven public datasets and the MQTR dataset. Notably, MSTAR marginally surpasses the previous state-of-the-art model by 6.4\% in MAP on Total-Text while eliminating box annotation costs. Moreover, on the MQTR benchmark, MSTAR significantly outperforms the previous models by an average of 8.5\%. The code and datasets are available at \href{https://github.com/yingift/MSTAR}{https://github.com/yingift/MSTAR}.