Categories that changed meaning
A label stays the same across the file, but what it covers was redefined partway through.
What it looks like
Before
| sale_date | territory | territory_definition |
|---|---|---|
| 2021-06-14 | West | · |
| 2024-06-14 | West | · |
| 2024-06-14 | Mountain | · |
After
| sale_date | territory | territory_definition |
|---|---|---|
| 2021-06-14 | West | map before 2023 |
| 2024-06-14 | West | map from 2023 |
| 2024-06-14 | Mountain | map from 2023 |
How it happens
- A territory, tier, or product line was split, merged, or redrawn, and the old name was reused.
- A threshold moved, so the same label now applies to a different group.
- A code was retired and later assigned to something new.
What it breaks
- Trends show a jump or a drop on the day the definition changed, and it is read as a real change in the business.
- Comparisons between periods set two different groups against each other under one name.
- No check catches it, because every value is a valid label.
How to spot it
- Chart the row count for each category over time. A sudden step with no known cause is a sign.
- Look for categories that first appear partway through the file, and see which older category shrank at the same moment.
- Ask the people who maintain the list whether it has been revised, and when.
How to fix it
- Establish the date of each change from outside the file. The data alone rarely proves it.
- Record which definition each row was written under, in its own column.
- Restate old rows under the new definition only where a finer field, such as state or product, makes that possible.
- Where old rows cannot be restated, say so, and compare periods only within one definition.
- Codes without a lookupA column holds short codes such as 3, A, or XR, and the table that says what each code means is not in the file.
- Inconsistent category labelsA column meant to hold a few fixed values holds many spellings of them.