A Perception-Inspired Deep Learning Framework for Predicting Perceptual Texture Similarity

Ying Gao; Yanhai Gan; Lin Qi; Huiyu Zhou; Xinghui Dong; Junyu Dong

doi:10.1109/TCSVT.2019.2944569

A Perception-Inspired Deep Learning Framework for Predicting Perceptual Texture Similarity

Ying Gao, Yanhai Gan, Lin Qi, Huiyu Zhou, Xinghui Dong, Junyu Dong^*

^*Corresponding author for this work

Research output: Contribution to journal › Article › peer-review

9 Citations (Scopus)

Abstract

Similarity learning plays a fundamental role in the fields of multimedia retrieval and pattern recognition. Prediction of perceptual similarity is a challenging task as in most cases we lack human labeled ground-truth data and robust models to mimic human visual perception. Although in the literature, some studies have been dedicated to similarity learning, they mainly focus on the evaluation of whether or not two images are similar, rather than prediction of perceptual similarity which is consistent with human perception. Inspired by the human visual perception mechanism, we here propose a novel framework in order to predict perceptual similarity between two texture images. Our proposed framework is built on the top of Convolutional Neural Networks (CNNs). The proposed framework considers both powerful features and perceptual characteristics of contours extracted from the images. The similarity value is computed by aggregating resemblances between the corresponding convolutional layer activations of the two texture maps. Experimental results show that the predicted similarity values are consistent with the human-perceived similarity data.

Original language	English
Pages (from-to)	3714-3726
Number of pages	13
Journal	IEEE Transactions on Circuits and Systems for Video Technology
Volume	30
Issue number	10
DOIs	https://doi.org/10.1109/TCSVT.2019.2944569
Publication status	Published - 30 Sept 2020
Externally published	Yes

Bibliographical note

Funding Information:
Manuscript received June 12, 2019; revised September 10, 2019; accepted September 25, 2019. Date of publication September 30, 2019; date of current version October 2, 2020. The work of J. Dong was supported in part by the National Key Research and development Program of China under Grant 2018AAA0100602, and in part by the National Natural Science Foundation of China (NSFC) under Grant 41576011, Grant U1706218, and Grant 41927805. The work of H. Zhou was supported in part by the U.K. Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/N011074/1, in part by the Royal Society-Newton Advanced Fellowship under Grant NA160342, and in part by the European Unions Horizon 2020 Research and Innovation Program through the Marie-Sklodowska-Curie under Grant 720325. The work of L. Qi was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 61501417. This article was recommended by Associate Editor H. Lu. (Corresponding author: Junyu Dong.) Y. Gao, Y. Gan, L. Qi, and J. Dong are with the Department of Information Science and Technology, Ocean University of China, Qingdao 266100, China (e-mail: [email protected]).

Publisher Copyright:
© 1991-2012 IEEE.

Copyright:
Copyright 2020 Elsevier B.V., All rights reserved.

Keywords

convolutional neural networks
perceptual similarity
Similarity learning
texture similarity

ASJC Scopus subject areas

Media Technology
Electrical and Electronic Engineering

Access to Document

10.1109/TCSVT.2019.2944569Licence: Unspecified

Cite this

@article{41dab4743c274d8bb02ab603f0f5f299,

title = "A Perception-Inspired Deep Learning Framework for Predicting Perceptual Texture Similarity",

abstract = "Similarity learning plays a fundamental role in the fields of multimedia retrieval and pattern recognition. Prediction of perceptual similarity is a challenging task as in most cases we lack human labeled ground-truth data and robust models to mimic human visual perception. Although in the literature, some studies have been dedicated to similarity learning, they mainly focus on the evaluation of whether or not two images are similar, rather than prediction of perceptual similarity which is consistent with human perception. Inspired by the human visual perception mechanism, we here propose a novel framework in order to predict perceptual similarity between two texture images. Our proposed framework is built on the top of Convolutional Neural Networks (CNNs). The proposed framework considers both powerful features and perceptual characteristics of contours extracted from the images. The similarity value is computed by aggregating resemblances between the corresponding convolutional layer activations of the two texture maps. Experimental results show that the predicted similarity values are consistent with the human-perceived similarity data.",

keywords = "convolutional neural networks, perceptual similarity, Similarity learning, texture similarity",

author = "Ying Gao and Yanhai Gan and Lin Qi and Huiyu Zhou and Xinghui Dong and Junyu Dong",

note = "Funding Information: Manuscript received June 12, 2019; revised September 10, 2019; accepted September 25, 2019. Date of publication September 30, 2019; date of current version October 2, 2020. The work of J. Dong was supported in part by the National Key Research and development Program of China under Grant 2018AAA0100602, and in part by the National Natural Science Foundation of China (NSFC) under Grant 41576011, Grant U1706218, and Grant 41927805. The work of H. Zhou was supported in part by the U.K. Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/N011074/1, in part by the Royal Society-Newton Advanced Fellowship under Grant NA160342, and in part by the European Unions Horizon 2020 Research and Innovation Program through the Marie-Sklodowska-Curie under Grant 720325. The work of L. Qi was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 61501417. This article was recommended by Associate Editor H. Lu. (Corresponding author: Junyu Dong.) Y. Gao, Y. Gan, L. Qi, and J. Dong are with the Department of Information Science and Technology, Ocean University of China, Qingdao 266100, China (e-mail: [email protected]). Publisher Copyright: {\textcopyright} 1991-2012 IEEE. Copyright: Copyright 2020 Elsevier B.V., All rights reserved.",

year = "2020",

month = sep,

day = "30",

doi = "10.1109/TCSVT.2019.2944569",

language = "English",

volume = "30",

pages = "3714--3726",

journal = "IEEE Transactions on Circuits and Systems for Video Technology",

issn = "1051-8215",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "10",

}

TY - JOUR

T1 - A Perception-Inspired Deep Learning Framework for Predicting Perceptual Texture Similarity

AU - Gao, Ying

AU - Gan, Yanhai

AU - Qi, Lin

AU - Zhou, Huiyu

AU - Dong, Xinghui

AU - Dong, Junyu

N1 - Funding Information: Manuscript received June 12, 2019; revised September 10, 2019; accepted September 25, 2019. Date of publication September 30, 2019; date of current version October 2, 2020. The work of J. Dong was supported in part by the National Key Research and development Program of China under Grant 2018AAA0100602, and in part by the National Natural Science Foundation of China (NSFC) under Grant 41576011, Grant U1706218, and Grant 41927805. The work of H. Zhou was supported in part by the U.K. Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/N011074/1, in part by the Royal Society-Newton Advanced Fellowship under Grant NA160342, and in part by the European Unions Horizon 2020 Research and Innovation Program through the Marie-Sklodowska-Curie under Grant 720325. The work of L. Qi was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 61501417. This article was recommended by Associate Editor H. Lu. (Corresponding author: Junyu Dong.) Y. Gao, Y. Gan, L. Qi, and J. Dong are with the Department of Information Science and Technology, Ocean University of China, Qingdao 266100, China (e-mail: [email protected]). Publisher Copyright: © 1991-2012 IEEE. Copyright: Copyright 2020 Elsevier B.V., All rights reserved.

PY - 2020/9/30

Y1 - 2020/9/30

N2 - Similarity learning plays a fundamental role in the fields of multimedia retrieval and pattern recognition. Prediction of perceptual similarity is a challenging task as in most cases we lack human labeled ground-truth data and robust models to mimic human visual perception. Although in the literature, some studies have been dedicated to similarity learning, they mainly focus on the evaluation of whether or not two images are similar, rather than prediction of perceptual similarity which is consistent with human perception. Inspired by the human visual perception mechanism, we here propose a novel framework in order to predict perceptual similarity between two texture images. Our proposed framework is built on the top of Convolutional Neural Networks (CNNs). The proposed framework considers both powerful features and perceptual characteristics of contours extracted from the images. The similarity value is computed by aggregating resemblances between the corresponding convolutional layer activations of the two texture maps. Experimental results show that the predicted similarity values are consistent with the human-perceived similarity data.

AB - Similarity learning plays a fundamental role in the fields of multimedia retrieval and pattern recognition. Prediction of perceptual similarity is a challenging task as in most cases we lack human labeled ground-truth data and robust models to mimic human visual perception. Although in the literature, some studies have been dedicated to similarity learning, they mainly focus on the evaluation of whether or not two images are similar, rather than prediction of perceptual similarity which is consistent with human perception. Inspired by the human visual perception mechanism, we here propose a novel framework in order to predict perceptual similarity between two texture images. Our proposed framework is built on the top of Convolutional Neural Networks (CNNs). The proposed framework considers both powerful features and perceptual characteristics of contours extracted from the images. The similarity value is computed by aggregating resemblances between the corresponding convolutional layer activations of the two texture maps. Experimental results show that the predicted similarity values are consistent with the human-perceived similarity data.

KW - convolutional neural networks

KW - perceptual similarity

KW - Similarity learning

KW - texture similarity

U2 - 10.1109/TCSVT.2019.2944569

DO - 10.1109/TCSVT.2019.2944569

M3 - Article

AN - SCOPUS:85092536290

SN - 1051-8215

VL - 30

SP - 3714

EP - 3726

JO - IEEE Transactions on Circuits and Systems for Video Technology

JF - IEEE Transactions on Circuits and Systems for Video Technology

IS - 10

ER -

A Perception-Inspired Deep Learning Framework for Predicting Perceptual Texture Similarity

Abstract

Bibliographical note

Keywords

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this