CE101

Understand Character Encodings with Diagrams and Exercises

You open a CSV in Excel and every Japanese line reads "縺薙ｓ縺ｫ縺｡縺ｯ". An email subject turns into a string of symbols. A name you stored in the database comes back as "????". Garbled text — mojibake, as it is known by its Japanese name — is an old acquaintance for anyone who works with text, yet explaining its cause is surprisingly hard. Instead of memorizing fixes, this course aims to let you look at a garbled screen and deduce for yourself which encoding wrote the bytes and which encoding misread them. You grasp the mechanism with diagrams, assemble UTF-8 byte sequences by hand, and finish by solving realistic mojibake cases.

Diagrams → numbers → bytes → deduction 43 questions Graded in-browser Saved to localStorage

Time

3–4 hours

Questions

43 total

Format

7 chapters + quizzes + simulator

Cost

Free

Diagram showing that the same byte sequence E3 81 AB reads as the Japanese character に in UTF-8, as 縺ｫ in Shift_JIS, and as ã« in Latin-1

All a file really contains is bytes — there is no tag saying which encoding they are in. The same byte sequence looks like different characters when read differently. This single diagram is the starting point of the whole course.

What makes this course different

Quizzes right below the text

You answer 1–3 questions immediately after each section, so it is hard to move on with only the illusion of understanding.

Build UTF-8 by hand

You compute for yourself, bit by bit, how the Japanese character "あ" (U+3042) becomes the 3 bytes E3 81 82.

A byte-sequence simulator

Type any string into the Chapter 3 simulator and see its code points, UTF-8 bytes, UTF-16 code units, and even "what it looks like when misread as Shift_JIS" with your own eyes.

Graded in-browser, no sign-up

All 43 questions are graded inside your browser, and results are stored only in your browser's localStorage. Nothing is sent to any server.

The one viewpoint that runs through this course

Inside a computer, characters are always handled in two stages. Learning to keep these two stages apart is everything this course teaches.

Stage 1: numbers (code points)

A character set decides which character gets which number. In Unicode, the Japanese character "あ" = U+3042. (Chapter 2)

Stage 2: bytes (encoding)

An encoding decides how that number becomes bytes. In UTF-8, U+3042 → E3 81 82. (Chapters 3–5)

The pattern behind garbling

Files carry no tag saying which encoding they use, so when the "writing" encoding and the "reading" encoding disagree, the same bytes appear as different characters. (Chapters 1, 6, 7)

The course motto: when you see mojibake, do not stop thinking — ask: which encoding wrote these bytes, and which encoding was used to read them?

Chapter overview

1 Introduction — Why mojibake happens 5 questions. A file is nothing but bytes; the same bytes become different characters under different readings; and the distinction between character sets and encodings. 2 From ASCII to Unicode — numbering the characters 6 questions. From 7-bit ASCII, through the proliferation of competing Japanese character sets, to Unicode, which gives every character in the world a unique number. You also practice reading U+XXXX notation. 3 Build UTF-8 by hand 6 questions. Learn the variable-length rules and hand-compute U+3042 into E3 81 82. Why ASCII compatibility works, plus the BOM — simulator included. 4 Shift_JIS, Japan's legacy encoding — dame-moji and CP932 6 questions. From the way second bytes intrude into the ASCII range, derive the "dame-moji" (troublemaker character) problem, then cover half-width kana, CP932, and why Shift_JIS is still in service. 5 UTF-16 and surrogate pairs — the character-count trap 6 questions. Follow how characters like "𠮷" and emoji take 2 code units, and verify by calculation why length does not match the number of characters you see. 6 Practical pitfalls — make the declaration match the bytes 6 questions. Mismatched charset declarations, Excel and UTF-8 with a BOM, newline codes, database connection settings — a sweep of the classic real-world failures. 7 Final exercises — deducing the mojibake 8 questions. Read classic patterns such as "縺薙ｓ縺ｫ縺｡縺ｯ" and "ã“ã‚“…" as fingerprints, and cross all chapters with realistic field cases.

Tips for studying

Go in the order text → the quiz right below it. Answering just after reading fixes things in memory far better than saving all the questions for later.
For the bit-packing in Chapter 3, write the binary out on paper and work it with your own hands. Once you have derived E3 81 82 yourself, UTF-8 turns from something to memorize into something you can compute.
Feed the Chapter 3 simulator your own name or strings you handle at work. The gap between byte count and character count becomes something you have felt, not just read about.
If a case question in Chapter 7 stumps you, go back to the corresponding chapter for just a pass or two of review. There is no need to redo everything.

Prerequisites before you start

Having handled text files or CSV files is enough. Programming experience is not required.
For hexadecimal, we assume only that you know it as "writing numbers with 0–9 and A–F"; you practice reading it inside the chapters.
Converting between binary and hexadecimal (4 bits = 1 hex digit) is the one skill used in Chapter 3, so the text includes a refresher.