Data Analysis and Integration

Data Analysis and Integration
Duplicate detection and data profiling

Introduction
• String matching
– edit distance, Damerau-Levenshtein, Needleman-Wunsch
– Jaro, Jaro-Winkler
– Jaccard
– Soundex, Refined Soundex
• Function that gives the similarity between two strings:

𝑠 𝑥, 𝑦 = …
IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 2

Data matching
• How to find these matches?

Data matching
• How to find these matches?
– compare every 𝑥 with every 𝑦

– calculate similarity 𝑠(𝑥, 𝑦)
– use 𝑡 as a threshold

Data matching

Data matching

Data matching

Data matching

Data matching

Data matching
• Potential problems
– number of comparisons may be too large
• e.g. 1000 x 1000 rows = 106 comparisons
– difficult to find single perfect measure
• e.g. there might be false negatives
– there may be no clear-cut threshold
• e.g. no way to avoid false positives or false negatives

Duplicate detection
• Similar approach for duplicate detection

– multiple records refer to the same real-world entity
customer_id first_name last_name address city country

169 Isabel de Castro Estrada da saúde n. 58 Lisboa Portugal
504 Isabel Castro Estrada da saúde 58 Lisbon Portugal
568 Isabella de Castro Estrada da saúde 58 Lisboa Portugal
369 Manuel Rodriguez Jardim das Rosas 32 Lisboa Portugal
535 Manuel Rodrigues Jardim das Rosas n. 32 Lisboa Portugal
576 Emanuel Rodrigues Jardim das Rosas 32 Lisbon Portugal
– compare these records to find duplicates

• focus on first_name, last_name, address

Duplicate detection
• If we wanted to find exact duplicates

select *
from customers as a, customers as b
where a.first_name = b.first_name
and a.last_name = b.last_name
and a.address = b.address;
– but we want to find approximate duplicates

• where first_name, last_name, address are similar, not equal
– similar according to some measure

Duplicate detection
• We could do:
select *
where similarity(a.first_name, b.first_name) >= …
and similarity(a.last_name, b.last_name) >= …
and similarity(a.address, b.address) >= …;
– assuming that the similarity(…, …) function is available

Duplicate detection
• Or:
select *
where similarity(a.first_name, b.first_name) +
similarity(a.last_name, b.last_name) +
similarity(a.address, b.address) >= …;
– using the similarities together in the same formula

Duplicate detection
• Or even:
select *
where 0.3*similarity(a.first_name, b.first_name) +
0.3*similarity(a.last_name, b.last_name) +
0.4*similarity(a.address, b.address) >= …;
– using different weights for the similarities

General approach
• Similarity between two records 𝑥 and 𝑦

𝑛
𝑠 𝑥, 𝑦 = ෍ 𝛼𝑖 ∙ 𝑠𝑖 (𝑥, 𝑦)
𝑖=1
– where:
• 𝑠𝑖 (𝑥, 𝑦) is the similarity between the 𝑖 𝑡ℎ attributes of 𝑥 and 𝑦
• 𝛼𝑖 is the weight given to the similarity 𝑠𝑖 (𝑥, 𝑦)

General approach
• The goal is to find every match 𝑥, 𝑦 for which

𝑛
𝑠 𝑥, 𝑦 = ෍ 𝛼𝑖 ∙ 𝑠𝑖 (𝑥, 𝑦) ≥ 𝑡
𝑖=1
– where 𝑡 is a threshold

Duplicate detection
• Example
customer_id,first_name,last_name,address,city,country
169,Isabel,de Castro,Estrada da saúde n. 58,Lisboa,Portugal
504,Isabel,Castro,Estrada da saúde 58,Lisbon,Portugal
568,Isabella,de Castro,Estrada da saúde 58,Lisboa,Portugal
369,Manuel,Rodriguez,Jardim das Rosas 32,Lisboa,Portugal
535,Manuel,Rodrigues,Jardim das Rosas n. 32,Lisboa,Portugal
576,Emanuel,Rodrigues,Jardim das Rosas 32,Lisbon,Portugal

Duplicate detection
• Rename

Duplicate detection
• Compare

Duplicate detection
• No need to compare 𝑥 ↔ 𝑥

Duplicate detection
• No need to compare 𝑥 ↔ 𝑥

Duplicate detection
• No need to compare both 𝑥 ↔ 𝑦 and 𝑦 ↔ 𝑥

Duplicate detection
• No need to compare both 𝑥 ↔ 𝑦 and 𝑦 ↔ 𝑥

Duplicate detection
• Calculating the measures

Duplicate detection
• Combining the measures

Duplicate detection
• Combining the measures

Duplicate detection
• Applying the threshold

Duplicate detection
• Getting the results

Duplicate detection
• Using the results

Duplicate elimination



Data profiling
• From duplicate detection to data profiling

– duplicate detection
– data cleaning
• much more than duplicate elimination
– data quality
• much more than finding duplicates
– data profiling

Data profiling
• What is data profiling?

– analyze the contents of a data source
– gather statistics about the data contained therein
• minimum, maximum, average, range, value distribution, etc.
– identify data quality problems
• missing or incomplete data, errors, inconsistencies, etc.
– understand the logic and relationships between data
• unique values, keys, foreign keys and other constraints

Data profiling
• These tasks (and many more) are data profiling

– number of rows, number of null values, number of distinct values
– minimum value, maximum value, minimum length, maximum length
– single- and multi-column frequency histogram
– precision of numeric values, length of string values
– data type discovery
– uniqueness and constancy
– single- and multi-column primary key discovery
– single- and multi-column foreign key discovery
– value overlap (cross-domain analysis)
– text field profiling
– pattern discovery (e.g. phone number patterns)
– soundex frequency histogram
– etc.
Data profiling
Z. Abedjan, L. Golab, F. Naumann

Profiling relational data: a survey
VLDB Journal, vol. 24, no. 4, 2015

Data profiling
• Data profiling tools

data quality problems
Data metadata
profiling • data types
… • keys
• dependencies
• …
data statistics

Data profiling
• Data profiling tools

Data profiling
• A datastore can be a file, database, etc.

Data profiling
• Creating a new datastore

Data profiling
• Creating a new datastore

Data profiling
• Several options for data profiling tasks

Data profiling
• Selecting the data source

Data profiling
• Completeness analysis

Data profiling

Data profiling

Data profiling
• Value distribution

Data profiling

Data profiling

Data profiling

Data profiling
• Value distribution – detailed results

Data profiling
• String analysis

Data profiling
• String analysis

Data profiling
• String analysis

Data profiling
• Integration between tools

– Pentaho Data Integration (PDI) with DataCleaner plugin
– the output of any transformation step can be a data source for
data profiling

Duplicate detection

Data profiling
• Example
– perform data profiling after duplicate elimination

Data profiling

Data profiling

Data profiling

Data profiling

Data profiling

Data profiling
• String analysis

Data profiling
• String analysis

Data Analysis and Integration

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Data Analysis and Integration

Uploaded by

Copyright:

Available Formats

Data Analysis and Integration

Duplicate detection and data profiling

• Function that gives the similarity between two strings:

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 2

• How to find these matches?

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 3

• How to find these matches?

– compare every 𝑥 with every 𝑦

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 4

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 5

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 6

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 7

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 8

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 9

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 10

• Similar approach for duplicate detection

customer_id first_name last_name address city country

– compare these records to find duplicates

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 11

• If we wanted to find exact duplicates

– but we want to find approximate duplicates

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 12

– assuming that the similarity(…, …) function is available

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 13

– using the similarities together in the same formula

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 14

– using different weights for the similarities

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 15

• Similarity between two records 𝑥 and 𝑦

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 16

• The goal is to find every match 𝑥, 𝑦 for which

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 17

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 18

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 19

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 20

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 21

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 22

• No need to compare both 𝑥 ↔ 𝑦 and 𝑦 ↔ 𝑥

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 23

• No need to compare both 𝑥 ↔ 𝑦 and 𝑦 ↔ 𝑥

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 24

• Calculating the measures

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 25

• Combining the measures

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 26

• Combining the measures

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 27

• Applying the threshold

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 28

• Getting the results

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 29

• Using the results

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 30

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 31

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 32

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 33

• From duplicate detection to data profiling

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 34

• What is data profiling?

IST/MEIC Análise e Integração de Dados - 2022/2023 - 1º Sem 35

• These tasks (and many more) are data profiling

Z. Abedjan, L. Golab, F. Naumann