Character Encoding: ASCII, Unicode and UTF-8

The problem encoding solves

Computers store numbers. Text is not numbers. Every writing system on earth has to be reduced to integers before a machine can hold it, and the mapping from characters to integers is a character set. The rule for turning those integers into bytes is an encoding.

Those two things are different, and conflating them is the source of most confusion. Unicode is a character set: it says the euro sign is code point U+20AC. UTF-8 is an encoding: it says U+20AC is stored as the three bytes E2 82 AC. The same character set can be encoded several ways.

When text arrives as bytes with no reliable statement of which encoding produced them, the receiver has to guess. Guessing wrong produces mojibake — from the Japanese for “character transformation” — the garbled text everyone has seen.

1963: ASCII and the seven-bit world

Before ASCII, every manufacturer had its own code. Moving a tape from an IBM machine to a Univac meant running it through a translation table, and often losing characters that had no equivalent.

ASCII, published in 1963, fixed 128 characters to the numbers 0–127: the English alphabet in both cases, digits, punctuation and 33 control characters inherited from teleprinters. Seven bits was a deliberate compromise between coverage and the cost of memory and transmission.

It worked brilliantly for English and not at all for anything else. No accented letters, no Greek, no Cyrillic, no Arabic, no CJK. Since bytes were eight bits and ASCII used seven, there were 128 spare codes — and everyone filled them differently.

The result was a proliferation of incompatible “code pages”: ISO-8859-1 for Western Europe, ISO-8859-5 for Cyrillic, KOI8-R also for Cyrillic but ordered differently, Windows-1252 which is Latin-1 plus curly quotes, code page 437 for the IBM PC, Shift-JIS for Japanese, Big5 for Traditional Chinese. Byte 0xE9 was é in one and something entirely different in another. Multilingual documents were impossible.

1991: Unicode

The Unicode Consortium was founded in 1991 with one goal: a single character set containing every character in every writing system, historical and modern.

The current version defines about 149,800 characters across 161 scripts, with a code space running from U+0000 to U+10FFFF — 1.1 million slots, of which around 13% are used. There is room for everything anyone is likely to encode.

The first design mistake was assuming 65,536 characters would be enough. Early Unicode was a fixed 16-bit encoding, and Windows NT, Java and JavaScript all built their string types around that assumption. When Unicode outgrew 16 bits in 1996, surrogate pairs were bolted on to encode the overflow — which is why '😀'.length is 2 in JavaScript to this day.

Browse the ranges in the Unicode blocks reference.

1992: UTF-8, the encoding that won

UTF-8 was sketched by Ken Thompson and Rob Pike on a placemat in a New Jersey diner in September 1992. It is arguably the single most successful piece of design in computing infrastructure, and it is used by over 98% of web pages.

The scheme is variable-width: a character takes 1 to 4 bytes depending on its code point.

Four properties made it win:

1. ASCII compatibility. Every ASCII file is already valid UTF-8, byte for byte. Decades of existing files and code kept working.
2. No null bytes. C string functions, which stop at a zero byte, continue to work on UTF-8 text.
3. Self-synchronising. Leading bytes and continuation bytes are distinguishable, so you can find a character boundary from any position. Corruption affects one character, not the rest of the file.
4. Byte-order independent. There is no endianness question, so no byte order mark is needed.

UTF-16, by contrast, needs a BOM or an out-of-band agreement, wastes a byte on every ASCII character, and contains null bytes that break C tooling. It survives mainly inside Windows, Java and JavaScript for historical reasons.

Diagnosing mojibake

Garbled text is not random. The pattern tells you exactly what went wrong.

Decode the actual bytes in the hex to text converter to confirm which encoding produced them before changing anything.

Seven rules that prevent every encoding bug

1. Use UTF-8 everywhere. Files, databases, APIs, source code, file names. Never mix encodings within a system.
2. Declare it explicitly. <meta charset="utf-8"> as the first element in <head>, and Content-Type: text/html; charset=utf-8 in the HTTP response. The header wins if they disagree.
3. Use utf8mb4 in MySQL. MySQL’s utf8 is a three-byte subset that cannot store emoji or many CJK characters. Only utf8mb4 is real UTF-8.
4. Do not add a BOM. It is unnecessary for UTF-8 and breaks shell scripts, JSON parsers and PHP files.
5. Normalise before comparing. Apply NFC to user input so that precomposed and decomposed forms of the same character compare equal.
6. Count characters, not bytes. Truncating a UTF-8 string at a byte offset can cut a character in half.
7. Test with real data. A string containing é 中 😀 العربية exercises 2, 3 and 4-byte sequences plus right-to-left text.

Handling encoding correctly in code

// JavaScript: bytes to text and back
const bytes = new TextEncoder().encode('caf\u00e9');   // Uint8Array [99,97,102,195,169]
const text  = new TextDecoder('utf-8').decode(bytes);

// Character count, not UTF-16 unit count
'\u{1F600}'.length          // 2  — wrong
[...'\u{1F600}'].length     // 1  — correct

// Grapheme clusters — what a user calls 'a character'
const seg = new Intl.Segmenter('en', { granularity: 'grapheme' });
[...seg.segment(str)].length

# Python 3: str is Unicode, bytes are bytes
s = 'caf\u00e9'
b = s.encode('utf-8')        # b'caf\xc3\xa9'
s2 = b.decode('utf-8')

# Always be explicit when opening files
open('data.txt', encoding='utf-8')

The rule in every language is the same: decode bytes to text at the boundary of your system, work in text internally, and encode back to bytes only on the way out. Bugs cluster wherever that boundary is implicit.

Frequently Asked Questions