Enhancing Sketch-Based Image Retrieval by CNN Semantic Re-ranking

Luo Wang, Xueming Qian, Yuting Zhang, Jialie Shen, Xiaochun Cao

Research output: Contribution to journalArticlepeer-review

26 Citations (Scopus)


This paper introduces a convolutional neural network (CNN) semantic re-ranking system to enhance the performance of sketch-based image retrieval (SBIR). Distinguished from the existing approaches, the proposed system can leverage category information brought by CNNs to support effective similarity measurement between the images. To achieve effective classification of query sketches and high-quality initial retrieval results, one CNN model is trained for classification of sketches, another for that of natural images. Through training dual CNN models, the semantic information of both the sketches and natural images is captured by deep learning. In order to measure the category similarity between images, a category similarity measurement method is proposed. Category information is then used for re-ranking. Re-ranking operation first infers the retrieval category of the query sketch and then uses the category similarity measurement to measure the category similarity between the query sketch and each initial retrieval result. Finally, the initial retrieval results are re-ranked. The experiments on different types of SBIR datasets demonstrate the effectiveness of the proposed re-ranking method. Comparisons with other re-ranking algorithms are also given to show the proposed method's superiority. Further, compared to the baseline systems, the proposed re-ranking approach achieves significantly higher precision in the top ten different SBIR methods and datasets.
Original languageEnglish
Pages (from-to)3330 - 3342
Number of pages13
JournalIEEE Transactions on Cybernetics
Issue number7
Early online date15 Mar 2020
Publication statusPublished - Jul 2020


Dive into the research topics of 'Enhancing Sketch-Based Image Retrieval by CNN Semantic Re-ranking'. Together they form a unique fingerprint.

Cite this