Deduplication
Finding records that describe the same real thing and reducing them to one.
Exact deduplication removes rows that are identical. The harder and more common task is deciding that two different-looking rows are the same, which is entity resolution.
Where it comes up
- Near-duplicate recordsTwo rows describe the same real thing but differ enough that an exact comparison misses them.
See also
- Entity resolutionDeciding which records refer to the same real-world person, company, or thing, when no shared identifier says so.
- Fuzzy matchingComparing two pieces of text for similarity instead of exact equality, so that Jon Smith and John Smith count as close.
- Primary keyA column, or set of columns, whose value is unique for every row and identifies it: a customer number, an order ID.