It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization

Research output: Chapter in Book/Report/Conference proceedingConference contribution

76 Downloads (Pure)

Abstract

Record linkage (RL) is a process of identifying records that refer to the same real-world entity. Many existing approaches to RL apply supervised machinelearning (ML) techniques to generate a classification model that classifiesa pair of records as either linked or non-linked. In such techniques, the labeleddata helps guide the choice and relative importance to similarity measures to beemployed in RL. Unsupervised RL is therefore a more challenging problem sincethe quality of similarity measures needs to be estimated in the absence of linkage labels. In this paper we propose a novel optimization approach to unsupervised RL. We define a scoring technique which aggregates similarities between two records along all attributes and all available similarity measures using a weighted sum formulation. The core idea behind our method is embodied in an objective function representing the overall ambiguity of the scoring across a dataset. Our goal is to iteratively optimize the objective function to progressively refine estimates of the scoring weights in the direction of lesser overall ambiguity. We have evaluated our approach on multiple real world datasets which are commonly used in the RL community. Our experimental results show that our proposed approach outperforms state-of-the-art techniques, while being orders of magnitude faster.
Original languageEnglish
Title of host publicationAdvances in Knowledge Discovery and Data Mining
PublisherSpringer
Publication statusEarly online date - 17 Jun 2018
EventThe 22nd Pacific-Asia Conference on Knowledge Discovery and Data Mining - Melbourne, Melbourne, Australia
Duration: 03 Jun 201806 Jun 2018
http://prada-research.net/pakdd18/

Publication series

Name Lecture Notes in Computer Science
PublisherSpringer
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceThe 22nd Pacific-Asia Conference on Knowledge Discovery and Data Mining
Abbreviated titlePAKDD
CountryAustralia
CityMelbourne
Period03/06/201806/06/2018
Internet address

Fingerprint Dive into the research topics of 'It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization'. Together they form a unique fingerprint.

  • Cite this