UTF-16 and surrogate pairs — the character-count trap
UTF-16 encodes in 16-bit units. Characters outside the BMP become surrogate pairs of 2 units, so length in C# and JavaScript stops matching the number of characters you see.
"A 3-character name will not fit" — an incident about counting
A form that allows "names up to 4 characters" rejects the 3 characters of "𠮷野家". A post with 2 emoji counts as "4 characters". Slicing by Length in C# left fragments of a character behind. — After byte counts, this chapter covers the other numeric mismatch: how characters are counted.
The protagonist is UTF-16: the encoding used internally by Windows and by the string types of Java, C#, and JavaScript. Its basic building block is a 16-bit (2-byte) code unit. The Japanese character "あ" (U+3042) fits into 16 bits as-is — 1 unit. So far, so straightforward.
But as Chapter 2 showed, Unicode numbers run up to U+10FFFF. The range representable in 16 bits — up to U+FFFF — is called the BMP (Basic Multilingual Plane); emoji (the U+1F600 block) and characters like "𠮷" (U+20BB7) live outside it. They do not fit in one 16-bit unit.
Surrogate pairs — 2 units representing 1 character
UTF-16's answer was to pair two code units to represent one character. The procedure closely resembles UTF-8's templates.
D842 DFB7. D800–DFFF is a reserved region whose values are never characters on their own, so a reader recognizes a pair instantlyThis two-unit combination is called a surrogate pair. Emoji work the same way: "😀" U+1F600 is the 2 units D83D DE00.
"あ", inside the BMP, is 1 unit; "𠮷", outside it, is 2. What length counts is this number of boxes — not the number of characters you see.
Quiz 5-1 — UTF-16 and surrogate pairs
Check how characters with large numbers are represented in a 16-bit-unit world.
Q1. Which statement about UTF-16 is correct?
UTF-16's basic unit is 16 bits. Unicode's early expectation that 2 bytes would suffice for all characters collapsed, and surrogate pairs — representing characters beyond U+FFFF in 2 units — were bolted on later. 'Fixed-length' describes only a bygone BMP-only past; today it is variable-length (1 or 2 units). The byte-oriented, ASCII-compatible one is UTF-8.
Q2. The kanji '𠮷' (the variant of 吉) has code point U+20BB7, beyond U+FFFF. How is it represented in UTF-16?
Subtract 0x10000 from U+20BB7 to get 0x10BB7, express it in 20 bits, put the top 10 bits on the high surrogate (0xD800-) and the bottom 10 bits on the low surrogate (0xDC00-), and you get D842 DFB7. D800-DFFF is a region reserved for exactly this purpose — units that are not characters on their own — and only as a pair do they form one character. Emoji (U+1F600 and up) use precisely the same mechanism.
Q3. C#'s string.Length and JavaScript's .length return the number of UTF-16 code units. What is the length of '𠮷野家' (𠮷 = U+20BB7; 野 and 家 are inside the BMP)?
'𠮷' is a surrogate pair, 2 units; '野' and '家' are 1 unit each, so 2 + 1 + 1 = 4. Three visible characters, yet 4 comes back — the moment the assumption 'length = character count' collapses. Enforcing character limits or slicing the first N characters by code units can cut a pair down the middle and produce broken characters.
What length returns is the number of boxes
Now the opening incidents solve themselves. Both C#'s string.Length and JavaScript's .length return the number of UTF-16 code units — a value that counts a surrogate pair's boxes as 2.
| String | Looks like | length (code units) | Why |
|---|---|---|---|
| あいう | 3 characters | 3 | All inside the BMP, 1 unit each |
| 𠮷野家 | 3 characters | 4 | 𠮷 is 2 units |
| 😀😀😀 | 3 characters | 6 | Each emoji is 2 units |
This is not a runtime bug — it is exactly as specified. The danger comes when length is read as "character count" and used for limit checks or slicing the first N characters. If a cut lands in the middle of a pair, a "broken character" — a stranded high surrogate — is born, seeding errors in downstream processing and databases.
One step further — combining characters and the "visible character"
In fact, "count code points and you get the character count" is still not the full answer. The family emoji "👨👩👧" is the three emoji "man", "woman", "girl" joined by invisible code points called zero-width joiners (ZWJ): 5 code points, 8 UTF-16 code units — yet what appears on screen is one character. Combining characters, such as writing the Japanese "が" as the 2 code points "か + voicing mark", belong to the same family.
The unit that corresponds to one character as humans see it is called the grapheme cluster, and counting it takes a dedicated API (JavaScript's Intl.Segmenter, for example). In sum, "character count" has at least four definitions — bytes, code units, code points, graphemes — and being able to say which one you are counting is the goal of this chapter. Feed "𠮷野家" or the family emoji to the Chapter 3 simulator and watch all four numbers disagree.
Quiz 5-2 — The length trap and combining characters
Finish off the surprising difficulty of 'counting characters' with emoji.
Q4. The emoji '😀' is U+1F600, 2 code units in UTF-16. In JavaScript, what is the .length of '😀😀😀' (3 emoji)?
2 units × 3 = 6. An input check written as length <= 3, intended to allow 'up to 3 emoji characters', rejects the input at just 2 emoji. This is a thoroughly modern pitfall that recurs in social-media character counters and database length limits.
Q5. When length (Length) disagrees with the visible character count, what is the correct understanding?
length behaves exactly as specified: it counts UTF-16 code units. The problem is on the caller's side, reading it as 'character count'. And avoiding emoji does not help — kanji like '𠮷' and the combining characters in the next question produce the same gap. Choose consciously which unit you are counting — bytes, code units, code points, or graphemes. That is this chapter's conclusion.
Q6. The family emoji '👨👩👧' is several emoji joined by zero-width joiners (ZWJ), and its JavaScript .length is 8. When you want to count 'one visible character' correctly, which unit is appropriate?
👨👩👧 is 'man + ZWJ + woman + ZWJ + girl' — 5 code points (3 of which are emoji at 2 units each), 8 code units, and still 5 even counted by code points. The unit corresponding to a human-perceived 'one character' is the grapheme cluster; count those and you finally get 1. Many languages have dedicated APIs (JavaScript's Intl.Segmenter, for example), and the 'visible characters' field of the Chapter 3 simulator counts in this unit too.
Key takeaways from this chapter
- UTF-16 is built on 16-bit code units. The BMP (up to U+FFFF) takes 1 unit; beyond it, a surrogate pair takes 2
- '𠮷' U+20BB7 = D842 DFB7; '😀' U+1F600 = D83D DE00
- In C# and JavaScript, length counts code units: '𠮷野家' is 4, '😀😀😀' is 6
- One visible character = one grapheme cluster. Distinguish the four counts: bytes, code units, code points, graphemes
The encoding machinery is now fully on the table. From the next chapter we turn to practice — clearing up the field troubles born of "declaration versus reality" mismatches.