Near-duplicate records

• The same person signed up twice, or was entered by two staff members.
• Two lists were combined without checking for overlap.
• A record was re-created instead of updated when a detail changed.

• Counts are inflated: more customers, donors, or patients than really exist.
• History is split, so no single record shows the full picture.
• People are contacted twice, and the second contact often contradicts the first.

• Standardize the fields first. Many near-duplicates become exact duplicates once phone and email formats agree.
• Group by one strong field at a time, such as email, and inspect groups larger than one.
• Sort by name and address together and read adjacent rows.

• Decide which fields count as evidence of sameness, and how many must agree.
• Merge deliberately: choose which record survives and which value wins when they differ.
• Keep a record of what was merged. Merging is hard to reverse once the originals are gone.

• Name variantsThe same person or organization appears under several spellings, orders, and abbreviations.
• Stray whitespace and invisible charactersTwo values look identical on screen but differ by a trailing space, a non-breaking space, or a hidden character.
• Case example: Non-profitsA donor list that counted the same people twice
• Case example: Finance teamsA vendor list that paid the same invoice twice

TermsDeduplication, Entity resolution, Fuzzy matching, Primary key