It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Record linkage (RL) is a process of identifying records that refer to the same real-world entity. Many existing approaches to RL apply supervised machinelearning (ML) techniques to generate a classification model that classifiesa pair of records as either linked or non-linked. In such techniques, the labeleddata helps guide the choice and relative importance to similarity measures to beemployed in RL. Unsupervised RL is therefore a more challenging problem sincethe quality of similarity measures needs to be estimated in the absence of linkage labels. In this paper we propose a novel optimization approach to unsupervised RL. We define a scoring technique which aggregates similarities between two records along all attributes and all available similarity measures using a weighted sum formulation. The core idea behind our method is embodied in an objective function representing the overall ambiguity of the scoring across a dataset. Our goal is to iteratively optimize the objective function to progressively refine estimates of the scoring weights in the direction of lesser overall ambiguity. We have evaluated our approach on multiple real world datasets which are commonly used in the RL community. Our experimental results show that our proposed approach outperforms state-of-the-art techniques, while being orders of magnitude faster.
LanguageEnglish
Title of host publicationAdvances in Knowledge Discovery and Data Mining
PublisherSpringer
Publication statusEarly online date - 17 Jun 2018
EventThe 22nd Pacific-Asia Conference on Knowledge Discovery and Data Mining - Melbourne, Melbourne, Australia
Duration: 03 Jun 201806 Jun 2018
http://prada-research.net/pakdd18/

Publication series

Name Lecture Notes in Computer Science
PublisherSpringer
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

ConferenceThe 22nd Pacific-Asia Conference on Knowledge Discovery and Data Mining
Abbreviated titlePAKDD
CountryAustralia
CityMelbourne
Period03/06/201806/06/2018
Internet address

Fingerprint

Labels

Cite this

Jurek, A., & Padmanabhan, D. (2018). It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization. In Advances in Knowledge Discovery and Data Mining ( Lecture Notes in Computer Science ). Springer.
Jurek, Anna ; Padmanabhan, Deepak. / It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization. Advances in Knowledge Discovery and Data Mining. Springer, 2018. ( Lecture Notes in Computer Science ).
@inproceedings{d353c1ae94ca4d9bb658ee6a14fe884d,
title = "It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization",
abstract = "Record linkage (RL) is a process of identifying records that refer to the same real-world entity. Many existing approaches to RL apply supervised machinelearning (ML) techniques to generate a classification model that classifiesa pair of records as either linked or non-linked. In such techniques, the labeleddata helps guide the choice and relative importance to similarity measures to beemployed in RL. Unsupervised RL is therefore a more challenging problem sincethe quality of similarity measures needs to be estimated in the absence of linkage labels. In this paper we propose a novel optimization approach to unsupervised RL. We define a scoring technique which aggregates similarities between two records along all attributes and all available similarity measures using a weighted sum formulation. The core idea behind our method is embodied in an objective function representing the overall ambiguity of the scoring across a dataset. Our goal is to iteratively optimize the objective function to progressively refine estimates of the scoring weights in the direction of lesser overall ambiguity. We have evaluated our approach on multiple real world datasets which are commonly used in the RL community. Our experimental results show that our proposed approach outperforms state-of-the-art techniques, while being orders of magnitude faster.",
author = "Anna Jurek and Deepak Padmanabhan",
year = "2018",
month = "6",
day = "17",
language = "English",
series = "Lecture Notes in Computer Science",
publisher = "Springer",
booktitle = "Advances in Knowledge Discovery and Data Mining",

}

Jurek, A & Padmanabhan, D 2018, It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization. in Advances in Knowledge Discovery and Data Mining. Lecture Notes in Computer Science , Springer, The 22nd Pacific-Asia Conference on Knowledge Discovery and Data Mining, Melbourne, Australia, 03/06/2018.

It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization. / Jurek, Anna; Padmanabhan, Deepak.

Advances in Knowledge Discovery and Data Mining. Springer, 2018. ( Lecture Notes in Computer Science ).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization

AU - Jurek, Anna

AU - Padmanabhan, Deepak

PY - 2018/6/17

Y1 - 2018/6/17

N2 - Record linkage (RL) is a process of identifying records that refer to the same real-world entity. Many existing approaches to RL apply supervised machinelearning (ML) techniques to generate a classification model that classifiesa pair of records as either linked or non-linked. In such techniques, the labeleddata helps guide the choice and relative importance to similarity measures to beemployed in RL. Unsupervised RL is therefore a more challenging problem sincethe quality of similarity measures needs to be estimated in the absence of linkage labels. In this paper we propose a novel optimization approach to unsupervised RL. We define a scoring technique which aggregates similarities between two records along all attributes and all available similarity measures using a weighted sum formulation. The core idea behind our method is embodied in an objective function representing the overall ambiguity of the scoring across a dataset. Our goal is to iteratively optimize the objective function to progressively refine estimates of the scoring weights in the direction of lesser overall ambiguity. We have evaluated our approach on multiple real world datasets which are commonly used in the RL community. Our experimental results show that our proposed approach outperforms state-of-the-art techniques, while being orders of magnitude faster.

AB - Record linkage (RL) is a process of identifying records that refer to the same real-world entity. Many existing approaches to RL apply supervised machinelearning (ML) techniques to generate a classification model that classifiesa pair of records as either linked or non-linked. In such techniques, the labeleddata helps guide the choice and relative importance to similarity measures to beemployed in RL. Unsupervised RL is therefore a more challenging problem sincethe quality of similarity measures needs to be estimated in the absence of linkage labels. In this paper we propose a novel optimization approach to unsupervised RL. We define a scoring technique which aggregates similarities between two records along all attributes and all available similarity measures using a weighted sum formulation. The core idea behind our method is embodied in an objective function representing the overall ambiguity of the scoring across a dataset. Our goal is to iteratively optimize the objective function to progressively refine estimates of the scoring weights in the direction of lesser overall ambiguity. We have evaluated our approach on multiple real world datasets which are commonly used in the RL community. Our experimental results show that our proposed approach outperforms state-of-the-art techniques, while being orders of magnitude faster.

M3 - Conference contribution

T3 - Lecture Notes in Computer Science

BT - Advances in Knowledge Discovery and Data Mining

PB - Springer

ER -

Jurek A, Padmanabhan D. It pays to be Certain: Unsupervised Record Linkage via Ambiguity Minimization. In Advances in Knowledge Discovery and Data Mining. Springer. 2018. ( Lecture Notes in Computer Science ).