Inconsistent column names across files
Files that should share one layout name the same column differently, or use the same name for different things.
What it looks like
Before
| file | header as written | standard name |
|---|---|---|
| march.csv | · | |
| april.csv | E-mail Address | · |
| may.csv | email_addr | · |
| may.csv | Contact | · |
After
| file | header as written | standard name |
|---|---|---|
| march.csv | ||
| april.csv | E-mail Address | |
| may.csv | email_addr | |
| may.csv | Contact | needs a decision |
How it happens
- The export template was changed between one period and the next.
- Each department or supplier built its own version of the sheet.
- Headers were retyped by hand, with different spacing, case, and abbreviations.
What it breaks
- Stacking the files produces several partly filled columns where one full column was expected.
- A script that looks for a column by name fails, or quietly skips files that lack it.
- Columns that share a name but not a meaning are merged, and the mix is hard to detect later.
How to spot it
- List the headers of every file side by side before combining anything.
- After stacking, look for columns that are empty for whole files.
- Compare the kind of values under each header, not just the header text.
How to fix it
- Define one standard set of column names, with a short description of what each holds.
- Map every header in every file to a standard name, and keep the mapping so the next batch can reuse it.
- Decide from the values, not the name, when two headers look alike. Where the values do not settle it, ask whoever produced the file.
- Rename before combining, so the files stack on matching columns.
- One table split across sheetsRows that belong in one table are spread over several sheets or files, one per month, region, or person.
- Merged and multi-row headersThe column names are spread over two or more rows, often with merged cells, so no single row names every column.
- The same entity in two systemsOne person or organization exists in two systems under different identifiers, with nothing linking the records.
- Case example: Marketing analyticsFive clients, five definitions of a campaign
TermsSchema, Schema drift, Mapping, Data dictionary