CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs

Lanyu Shang; Zhang Daniel Yue; Khan Siamul Karim; Jialie Shen; Dong Wang

doi:10.1109/ASONAM49781.2020.9381371

CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs

Lanyu Shang, Zhang Daniel Yue, Khan Siamul Karim, Jialie Shen, Dong Wang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

7 Citations (Scopus)

Abstract

With the ubiquitous network connectivity and the proliferation of mobile devices, people are increasingly consuming digital contents from social media driven music sharing platforms (e.g., YouTube, Soundcloud). In this paper, we study a novel problem of connotation-Aware music retrieval that focuses on the connotation which expresses the implicit feeling or emotion beyond the explicit content in artworks. Our goal is to automatically retrieve relevant music on social media based on the connotation of visual inputs (e.g., images, photos) provided by the users. The problem is challenging as it requires the accurate identification of the implicit connotation from both images and music pieces, and the precise matching of the identified connotation across different data modalities. We develop a connotation-Aware music retrieval (CaMR) framework to address the above challenges. Evaluation results from a real-world social media dataset demonstrate that the CaMR framework can retrieve music that is highly relevant to the connotation of the input image.

Original language	English
Title of host publication	Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2020)
Editors	Martin Atzmuller, Michele Coscia, Rokia Missaoui
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	425-429
Number of pages	5
ISBN (Electronic)	9781728110561
DOIs	https://doi.org/10.1109/ASONAM49781.2020.9381371
Publication status	Published - 24 Mar 2021
Event	12th IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020 - Virtual, Online, Netherlands Duration: 07 Dec 2020 → 10 Dec 2020

Publication series

Name	Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020
Publisher	IEEE
ISSN (Print)	2473-9928
ISSN (Electronic)	2473-991X

Conference

Conference	12th IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020
Country/Territory	Netherlands
City	Virtual, Online
Period	07/12/2020 → 10/12/2020

Bibliographical note

Funding Information:
This research is supported in part by the National Science Foundation under Grant No. CNS-1845639, CNS-1831669, Army Research Office under Grant W911NF-17-1-0409. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.

Publisher Copyright:
© 2020 IEEE.

Copyright:
Copyright 2021 Elsevier B.V., All rights reserved.

ASJC Scopus subject areas

Artificial Intelligence
Computer Networks and Communications
Information Systems and Management
Social Psychology
Communication

Access to Document

10.1109/ASONAM49781.2020.9381371Licence: Unspecified

Cite this

Shang, L., Daniel Yue, Z., Siamul Karim, K., Shen, J., & Wang, D. (2021). CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs. In M. Atzmuller, M. Coscia, & R. Missaoui (Eds.), Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2020) (pp. 425-429). Article 9381371 (Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ASONAM49781.2020.9381371

Shang, Lanyu ; Daniel Yue, Zhang ; Siamul Karim, Khan et al. / CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs. Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2020). editor / Martin Atzmuller ; Michele Coscia ; Rokia Missaoui. Institute of Electrical and Electronics Engineers Inc., 2021. pp. 425-429 (Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020).

@inproceedings{735434af0c5d46b5be84d7a1dc4ed355,

title = "CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs",

abstract = "With the ubiquitous network connectivity and the proliferation of mobile devices, people are increasingly consuming digital contents from social media driven music sharing platforms (e.g., YouTube, Soundcloud). In this paper, we study a novel problem of connotation-Aware music retrieval that focuses on the connotation which expresses the implicit feeling or emotion beyond the explicit content in artworks. Our goal is to automatically retrieve relevant music on social media based on the connotation of visual inputs (e.g., images, photos) provided by the users. The problem is challenging as it requires the accurate identification of the implicit connotation from both images and music pieces, and the precise matching of the identified connotation across different data modalities. We develop a connotation-Aware music retrieval (CaMR) framework to address the above challenges. Evaluation results from a real-world social media dataset demonstrate that the CaMR framework can retrieve music that is highly relevant to the connotation of the input image.",

author = "Lanyu Shang and {Daniel Yue}, Zhang and {Siamul Karim}, Khan and Jialie Shen and Dong Wang",

note = "Funding Information: This research is supported in part by the National Science Foundation under Grant No. CNS-1845639, CNS-1831669, Army Research Office under Grant W911NF-17-1-0409. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. Publisher Copyright: {\textcopyright} 2020 IEEE. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.; 12th IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020 ; Conference date: 07-12-2020 Through 10-12-2020",

year = "2021",

month = mar,

day = "24",

doi = "10.1109/ASONAM49781.2020.9381371",

language = "English",

series = "Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "425--429",

editor = "Martin Atzmuller and Michele Coscia and Rokia Missaoui",

booktitle = "Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2020)",

address = "United States",

}

Shang, L, Daniel Yue, Z, Siamul Karim, K, Shen, J & Wang, D 2021, CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs. in M Atzmuller, M Coscia & R Missaoui (eds), Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2020)., 9381371, Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020, Institute of Electrical and Electronics Engineers Inc., pp. 425-429, 12th IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020, Virtual, Online, Netherlands, 07/12/2020. https://doi.org/10.1109/ASONAM49781.2020.9381371

CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs. / Shang, Lanyu; Daniel Yue, Zhang; Siamul Karim, Khan et al.
Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2020). ed. / Martin Atzmuller; Michele Coscia; Rokia Missaoui. Institute of Electrical and Electronics Engineers Inc., 2021. p. 425-429 9381371 (Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs

AU - Shang, Lanyu

AU - Daniel Yue, Zhang

AU - Siamul Karim, Khan

AU - Shen, Jialie

AU - Wang, Dong

N1 - Funding Information: This research is supported in part by the National Science Foundation under Grant No. CNS-1845639, CNS-1831669, Army Research Office under Grant W911NF-17-1-0409. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on. Publisher Copyright: © 2020 IEEE. Copyright: Copyright 2021 Elsevier B.V., All rights reserved.

PY - 2021/3/24

Y1 - 2021/3/24

N2 - With the ubiquitous network connectivity and the proliferation of mobile devices, people are increasingly consuming digital contents from social media driven music sharing platforms (e.g., YouTube, Soundcloud). In this paper, we study a novel problem of connotation-Aware music retrieval that focuses on the connotation which expresses the implicit feeling or emotion beyond the explicit content in artworks. Our goal is to automatically retrieve relevant music on social media based on the connotation of visual inputs (e.g., images, photos) provided by the users. The problem is challenging as it requires the accurate identification of the implicit connotation from both images and music pieces, and the precise matching of the identified connotation across different data modalities. We develop a connotation-Aware music retrieval (CaMR) framework to address the above challenges. Evaluation results from a real-world social media dataset demonstrate that the CaMR framework can retrieve music that is highly relevant to the connotation of the input image.

AB - With the ubiquitous network connectivity and the proliferation of mobile devices, people are increasingly consuming digital contents from social media driven music sharing platforms (e.g., YouTube, Soundcloud). In this paper, we study a novel problem of connotation-Aware music retrieval that focuses on the connotation which expresses the implicit feeling or emotion beyond the explicit content in artworks. Our goal is to automatically retrieve relevant music on social media based on the connotation of visual inputs (e.g., images, photos) provided by the users. The problem is challenging as it requires the accurate identification of the implicit connotation from both images and music pieces, and the precise matching of the identified connotation across different data modalities. We develop a connotation-Aware music retrieval (CaMR) framework to address the above challenges. Evaluation results from a real-world social media dataset demonstrate that the CaMR framework can retrieve music that is highly relevant to the connotation of the input image.

U2 - 10.1109/ASONAM49781.2020.9381371

DO - 10.1109/ASONAM49781.2020.9381371

M3 - Conference contribution

AN - SCOPUS:85103692239

T3 - Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020

SP - 425

EP - 429

BT - Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2020)

A2 - Atzmuller, Martin

A2 - Coscia, Michele

A2 - Missaoui, Rokia

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 12th IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020

Y2 - 7 December 2020 through 10 December 2020

ER -

Shang L, Daniel Yue Z, Siamul Karim K, Shen J, Wang D. CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs. In Atzmuller M, Coscia M, Missaoui R, editors, Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2020). Institute of Electrical and Electronics Engineers Inc. 2021. p. 425-429. 9381371. (Proceedings of the 2020 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2020). doi: 10.1109/ASONAM49781.2020.9381371

CaMR: Towards Connotation-Aware Music Retrieval on Social Media with Visual Inputs

Abstract

Publication series

Conference

Bibliographical note

ASJC Scopus subject areas

Access to Document

Fingerprint

Cite this