Smart quotes and look-alike characters

• Word processors and phones replace straight quotes and hyphens with typographic ones as you type.
• Text was pasted from documents, web pages, or email.
• A person or a scanner confused characters that look alike: the letter O and zero, lowercase L and one, or letters from another alphabet.

• Matching fails between values that look the same on screen.
• Curly quotes inside a delimited file are not treated as quoting, so fields split in the wrong places.
• Codes with a swapped character point to nothing, or to the wrong item.

• Search for any character outside the plain keyboard set in columns that should not need one.
• Compare two values that look equal by checking their characters one at a time.
• Test codes against their expected pattern. A letter in a position that is always a digit stands out.

• In fields used for matching, replace typographic quotes and dashes with their plain equivalents.
• Leave typographic characters alone in text meant for reading, where they are correct.
• Check suspect codes against the list of valid ones before changing a character. Without that list, flag the value.

• Stray whitespace and invisible charactersTwo values look identical on screen but differ by a trailing space, a non-breaking space, or a hidden character.
• Broken accented charactersAccented letters appear as pairs of odd symbols or as question marks, because the file was read with the wrong encoding.
• Negative numbers in several notationsNegative amounts are written with a leading minus, a trailing minus, parentheses, or a credit marker, all in one column.

TermsUnicode, Normalization, Encoding, Regular expression