Unicode
The standard that assigns a number to the characters of nearly all the world's writing systems, along with symbols and emoji, so that text in any of them can be represented in one scheme.
Unicode says which characters exist, and an encoding such as UTF-8 says how they are stored as bytes. Some characters can be built in more than one way, so two strings can look identical and still compare as different.
Where it comes up
- Smart quotes and look-alike charactersA character in the cell looks like an ordinary quote, hyphen, or letter but is a different character underneath.
- Broken accented charactersAccented letters appear as pairs of odd symbols or as question marks, because the file was read with the wrong encoding.
- Stray whitespace and invisible charactersTwo values look identical on screen but differ by a trailing space, a non-breaking space, or a hidden character.
See also
- UTF-8The most widely used encoding for Unicode text. It stores each character in one to four bytes, with plain English letters and digits taking one byte each.
- EncodingThe rule that maps the bytes in a file to characters. Reading a file with the wrong encoding turns accented letters and symbols into nonsense.
- MojibakeThe garbled text produced when a file is read with a different encoding from the one it was written in, so that José appears as José.