Broken accented characters
Accented letters appear as pairs of odd symbols or as question marks, because the file was read with the wrong encoding.
What it looks like
Before
| name | city |
|---|---|
| José Marin | München |
| Ana Peña | Logroño |
| Ren?e Dubois | Orl?ans |
After
| name | city |
|---|---|
| José Marin | München |
| Ana Peña | Logroño |
| needs the source | needs the source |
How it happens
- A file saved as UTF-8 was opened as an older single-byte encoding, so each accented letter became two characters.
- A file passed through a system that could not store the letter and substituted a question mark.
- The file was opened and saved again while it was being misread, which made the damage permanent.
What it breaks
- Names and places are wrong on everything sent to the people they belong to.
- The damaged spelling does not match the correct one, so joins and duplicate checks miss it.
- Sorting and searching by name put the affected rows in the wrong place.
How to spot it
- Search for the telltale pairs, most often a capital A with a tilde or a circumflex followed by another symbol.
- Search for question marks and replacement symbols in the middle of words.
- Check columns likely to hold accents: names, cities, and anything typed in a language other than English.
How to fix it
- Read the original file again with the correct encoding. This is the cleanest fix when the original still exists.
- Where only the damaged copy remains and the pairs are intact, reverse the misreading to recover the letters.
- A question mark cannot be reversed. Recover the value from the source, or leave it flagged; guessing the letter is still guessing.
- Save the result as UTF-8 and state the encoding wherever the file is handed on.
- Stray whitespace and invisible charactersTwo values look identical on screen but differ by a trailing space, a non-breaking space, or a hidden character.
- Smart quotes and look-alike charactersA character in the cell looks like an ordinary quote, hyphen, or letter but is a different character underneath.
- Name variantsThe same person or organization appears under several spellings, orders, and abbreviations.
- Case example: E-commerceA catalog where the same product was listed four times