Schema drift
Unannounced change in the structure of a recurring data feed: a column renamed, added, dropped, reordered, or changed in type between one delivery and the next.
Drift does the most harm when nothing fails. A process that reads columns by position keeps running after a reorder and loads the wrong values into the right names.
Where it comes up
- Inconsistent column names across filesFiles that should share one layout name the same column differently, or use the same name for different things.
- Categories that changed meaningA label stays the same across the file, but what it covers was redefined partway through.
See also
- SchemaThe declared structure of a dataset: its columns, their names and types, and the rules they must satisfy.
- HeaderThe row at the top of a table that holds the column names. In data files it is normally the first line.
- ValidationChecking values against rules, such as a date being in the past or a state code being on the allowed list, and reporting what fails.