Tag Info

About

Duplication refers to the inadvertent creation of duplicate entries in tables and the process for resolving the repeated entries.

In the simple case, this involves identifying a survivor row and eliminating the non-survivors.

A slightly more complex situation, survivorship logic must be employed. In the following example, the rule was to simply collapse non-empty values.

Before

CustomerName | Address | Phone
Bob          | 100 Elm | 
Bob          |         | 555.1234

After

CustomerName | Address | Phone
Bob          | 100 Elm | 555.1234

The truly complex scenarios stem from conflicting values. Consider the following example, there is a conflicting value for the phone value in our deduplication candidate. Which value is the "right" is likely to be dependent on the supplied business rules and/or data collection logic.

Before

CustomerName | Address | Phone
Bob          | 100 Elm | 555.4234
Bob          |         | 555.1234

After 1

CustomerName | Address | Phone
Bob          | 100 Elm | 555.1234

After 2

CustomerName | Address | Phone
Bob          | 100 Elm | 555.4234

After 3

CustomerName | Address | Phone
Bob          |         | 555.1234

Which After is correct is an "it depends" answer.

The first After merged the changes together. Perhaps the data collection system only records changed data (and we further assume said system required address for an entry to be valid)

The second After declared the first row completely correct. Perhaps there was an additional field showing it was the most recently updated. Or because it was most completely filled out, the assumption is that it was the survivor.

The third After declared the second row completely correct. It could be that there was another field showing it to be the most recently updated (Bob is now homeless, pity him).

history | show excerpt | excerpt history