From ASCII to Unicode — numbering the characters
Trace the history of stage 1, assigning numbers to characters: ASCII's 128 characters, the tangle of competing Japanese character sets, and finally Unicode, giving every character in the world a unique number U+XXXX.
The ancestor of every character code — ASCII's 128 characters
The idea of "assigning numbers to characters" originates with ASCII (American Standard Code for Information Interchange), born in 1960s America. ASCII assigned uppercase and lowercase letters, digits, symbols, and control characters such as the line feed to 7 bits — the 128 numbers 0 through 127.
For the English-speaking world this was enough. But the world holds far more characters than 128 slots can accommodate. Here began the long confusion over how to assign the numbers.
Quiz 2-1 — Read ASCII as numbers
ASCII underlies every character code. Try actually reading it as numbers.
Q1. Which statement about ASCII is correct?
ASCII assigns letters, digits, symbols, and control characters (such as line feeds) to 7 bits — the 128 numbers 0 through 127. It is not 256 characters: the 8th bit became territory that each country fought over for its own extensions, one cause of the later proliferation of encodings. It contains no Japanese, and dates from the 1960s, long before Unicode.
Q2. In ASCII the letter 'A' has number 0x41 in hexadecimal. What is that in decimal? (The higher hex digit carries a weight of 16: 0x41 = 4 × 16 + 1)
0x41 = 4 × 16 + 1 = 65. 'A = 65' is a classic value that shows up in many programming languages, followed by B = 66, C = 67, and so on. Lowercase 'a' is 0x61 = 97, exactly 0x20 (32) away from uppercase — that regularity, allowing case conversion by flipping a single bit, is part of ASCII's elegant design.
Q3. ASCII contains not a single Japanese character. What actually happened afterwards?
Japan standardized character sets including kanji as JIS standards, and several schemes for rendering them into bytes coexisted: Shift_JIS, EUC-JP, and the JIS code (ISO-2022-JP). The same Japanese document had different bytes depending on the environment that produced it — this proliferation was the breeding ground of Japanese mojibake, and the era of cross-platform email garbling dragged on for years.
Japan's era of proliferation — the same document, different bytes
In Japan, character sets including kanji — which kanji gets which number — were standardized as JIS standards. But multiple schemes arose for turning those numbers into bytes (encodings). The PC world settled on Shift_JIS, the UNIX world on EUC-JP, and email on the JIS code (ISO-2022-JP) — a different scheme dominated each environment.
The same Japanese word "こんにちは" is an entirely different byte sequence in a Shift_JIS file and an EUC-JP file. Recall Chapter 1's motto: the receiver of untagged bytes can only guess the reading, and a wrong guess garbles. In the era of encoding proliferation, Japanese text was a mojibake regular. The star of that era, Shift_JIS, is still in service today, so Chapter 4 digs into its structure.
Unicode — a unique number for every character in the world
If number tables keep multiplying per language and per platform, why not gather all the world's characters into one table and assign unique numbers once and for all — that is the idea of Unicode. Hiragana, kanji, Arabic script, emoji: all sit in the same single number space. The number is called a code point and written U+XXXX (XXXX in hexadecimal).
The hex after U+ reads like an ordinary number once you are used to it. One hex digit covers 0–15; the digit weights are 1, 16, 256, 4096. 0x3042 = 3 × 4096 + 4 × 16 + 2 = 12354 — work through this conversion once by hand in the quiz below.
Unicode only decides stage 1 (the number). The same U+3042 becomes different byte sequences under different stage-2 encodings. "Unicode ≠ UTF-8" is the point of this diagram.
Unicode stops at the number — turning it into bytes is still open
Here Chapter 1's two-stage view pays off. What Unicode solved is only stage 1: unifying how numbers are assigned. For stage 2 — turning numbers into bytes — multiple schemes remain: UTF-8, UTF-16, and UTF-32. All carry the same Unicode numbers; they are different encodings.
The phrase "it supports Unicode" alone does not pin down the bytes in a file. What practice always asks is the stage-2 choice: is it UTF-8 or UTF-16, and is there a BOM? The next chapter takes the leading player of stage 2, UTF-8, and assembles it by hand, bit by bit.
Quiz 2-2 — Unicode and code points
Check the Unicode idea of giving every character in the world a unique number, and how to read U+XXXX notation.
Q4. Which statement captures the basic idea of Unicode?
Reflecting on the chaos of per-language number tables, Unicode chose to gather all the world's characters into a single table with unique numbers. Hiragana, kanji, and emoji all live in the same single number space. 'Fixed 2 bytes' was only an early expectation — code points now run up to U+10FFFF and no longer fit in 2 bytes, a story that leads to surrogate pairs in Chapter 5.
Q5. The code point of the Japanese character 'あ' is U+3042. The part after U+ is hexadecimal. What is 0x3042 in decimal? (3 × 4096 + 0 × 256 + 4 × 16 + 2)
The four hex digits carry weights 4096, 256, 16, and 1, so 3 × 4096 + 0 + 4 × 16 + 2 = 12288 + 64 + 2 = 12354. U+XXXX notation is nothing more than a number written in hex; once you can read it, Unicode reference material becomes far more approachable. Incidentally, two places after 'あ' sits U+3044, 'い' (with the small 'ぃ' at U+3043 in between).
Q6. What does the notation 'U+3042' refer to?
U+3042 is the stage-1 'number'. To store it you must convert the number into bytes with a stage-2 encoding: E3 81 82 in UTF-8, 30 42 in UTF-16. The crux is that the number stays the same no matter which encoding you pick. Fonts are yet another layer, giving the number a visual shape — separate from encoding altogether.
Key takeaways from this chapter
- ASCII is 7 bits, 128 characters. 'A' = 0x41 = 65. The foundation of most character codes
- Japanese text lived through a long era of competing encodings — Shift_JIS, EUC-JP, and others — a breeding ground for mojibake
- Unicode assigns every character in the world a unique number (code point, U+XXXX). 'あ' = U+3042 = 12354
- Unicode decides only the number. Stage 2 — bytes (UTF-8 and friends) — remains a separate choice
Next up is the leading player of stage 2: UTF-8. You will compute with your own hands how U+3042 becomes E3 81 82.