Japanese Font and Character Pitfalls — Handling JIS2004, IVS, and Gaiji in Business Apps

“The 葛 character in the customer list looks different on screen and on the printed form. The customer complained that the data must be corrupted.” — In business-system maintenance, this kind of consultation is not rare. Another common one is “a character in a person’s name will not display on a document we submit to the government office. It used to display on the old PC; after we replaced it, it became □.”

Both tend to be called “mojibake” in the field, but they are a different problem from the mojibake that comes from an encoding mismatch. In the former, not one bit of the data has changed and only the appearance has changed; in the latter, a “gaiji” that existed only on that PC has been lost.

flowchart TB
    accTitle: What the two common consultations really are
    accDescr: The consultation that 葛 looks different on screen and on the form is a case where only the appearance changed while the data stayed the same; the consultation that a character became □ after a PC replacement is a case where a gaiji that existed only on that PC was lost; both are a different problem from encoding-mismatch mojibake
    c1["Consultation 1: the shape differs on screen and on the form"] --> r1["The data is unchanged; only the appearance changed"]
    c2["Consultation 2: it became □ after a replacement"] --> r2["A gaiji that existed only on that PC was lost"]
    r1 --> diff["A different problem from encoding mojibake"]
    r2 --> diff

Figure 1: The two consultations that tend to be called “mojibake” are both a different problem from an encoding mismatch.

This article’s promise is simple. If you separate the character-code (data) layer from the font (appearance) layer, most Japanese character trouble becomes tractable. From the JIS2004 glyph change, ideographic variation selectors (IVS), and gaiji (EUDC), through the government’s character platform, to choosing and embedding fonts, it is organised in a form developers of business systems and IT staff can use for decisions.

The “mojibake” itself that happens in Shift_JIS ↔ UTF-8 conversion is covered in existing articles, so this article concentrates on the problem of “the codes round-trip correctly, but appearance or displayability is off”.

1. The Bottom Line First

• “Mojibake” and “the glyph is different” are different problems. Mojibake is an accident at the data layer of misinterpreting a byte sequence; a glyph difference is an accident at the appearance layer of a difference in the glyphs a font holds; the remedies are completely different.
• Even with the same Unicode code point, the glyph that is displayed depends on the font. JIS X 0213:2004 revised the exemplar glyphs of 168 characters such as 葛, 辻, and 飴 to the printing-standard forms, and Windows too made the JIS2004 glyphs the default in MS Gothic / MS Mincho from Vista onward.¹²
• The standard means of fixing a glyph as data is an ideographic variation selector (IVS). You specify a glyph with the sequence of a base character plus a selector from U+E0100 onward; collections such as Adobe-Japan1, Hanyo-Denshi, and Moji_Joho (the Character Information Platform) are registered in Unicode’s IVD.³⁴
• In a non-supporting environment the specified behaviour of IVS is that the selector is ignored and the base character’s default glyph is displayed. One IVS-bearing character, however, can be up to four code units in UTF-16, so implementations of character-count and slicing need care.⁵
• Gaiji (EUDC) has the fate of “it can only display on that PC”. There is no agreed meaning for a Private Use Area code point, and a glyph registered in eudc.tte does not travel to another PC, to mail, or to a PDF.⁶⁷
• A system that handles personal names should decide the accepted character set and state it. On the government side, building on Koseki Unified Characters and the Character Information Platform, systems that conform to the standard are moving toward using “Standard Characters for Administrative Affairs”.⁸⁹¹⁰
• For forms and PDFs, “align the font with the screen, and embed it” is the baseline. Whether embedding is allowed is determined by the font’s licence (fsType), and long-term-retention PDF/A requires font embedding.¹¹¹²
• Do not casually apply normalization (NFKC) to personal-name data. Unifying fullwidth and halfwidth, and replacing compatibility characters, loses distinctions you should keep.¹³

In one sentence: “which byte sequence you store” is a data-design problem; “how it looks” is a font-design problem. If you discuss the two mixed together, even problems you could fix become unfixable.

2. Thinking of Data and Appearance Separately — Code Points and Glyphs

In Unicode, a character is represented by a number called a code point. 葛 is U+845B, and this number is the same on every PC. How that number is drawn on a screen or on paper, on the other hand, is decided by the glyph the font holds. It is normal behaviour for the same U+845B to differ in the details of its shape between font A and font B.

With these two layers as the premise, field symptoms can be split as follows.

As a clue for the split, it is useful to remember the difference between “�” and “□”. “�” of U+FFFD (REPLACEMENT CHARACTER) is a trace of a conversion failure at the data layer, and the original character is already lost. “□”, on the other hand, is in many cases only that the data is still there but the font has no glyph, and changing the font may make it displayable.

flowchart TB
    accTitle: Splitting the symptom by � versus □
    accDescr: When a character does not display correctly, � is a trace of a conversion failure at the data layer in which the original character has been lost; □ is only that the data is still there but the font has no glyph, and changing the font may make it displayable
    symptom["A character does not display correctly"] --> which{"What do you see?"}
    which -->|You see �| datalayer["A data-layer accident"]
    datalayer -.-> lost["A trace of conversion failure (the original character is lost)"]
    which -->|You see □| viewlayer["An appearance-layer accident"]
    viewlayer -.-> noglyph["Only that the font has no glyph"]
    noglyph --> fixable["Changing the font may make it displayable"]

Figure 2: � is a sign of a data-layer accident, □ of an appearance-layer accident, and the entry point of the investigation changes.

The basics of encodings themselves (CP932 and UTF-8, BOM, newline codes) are covered in “An Introduction to Windows Text Encodings - The Mojibake That Happens When Integrating with Linux” and “Windows Text Encodings and Line Endings - The Basics of Mojibake and CRLF/LF”. From here on it is the appearance layer, and the problems that occur at its boundary.

3. From JIS90 to JIS2004 — The Glyph Changed While the Code Stayed the Same

The identity of the opening “葛 looks different on screen and on the form” is, in many cases, here.

Following the National Language Council’s 2000 report “Hyogai Kanji Jitaihyo” (the table of character forms for kanji outside the joyo list), the 2004 revision JIS X 0213:2004 (commonly JIS2004) revised the exemplar glyphs of 168 kanji to the printing-standard forms, close to the so-called Kangxi Dictionary forms. 葛, 辻, 飴, 芦, 溢, 餅 and the like are representative examples.¹

Windows aligned with this and made the JIS2004 glyphs the default in MS Gothic / MS Mincho (and the newly introduced Meiryo) from Windows Vista onward. Current MS Gothic, too, has a JIS2004-based default glyph, with the structure that JIS90-era glyphs are accessible via the OpenType jp90 feature.²¹

flowchart TB
    accTitle: The glyph structure of current MS Gothic
    accDescr: From Vista onward, MS Gothic has JIS2004 glyphs as the default, and accessing JIS90-era glyphs via the OpenType jp90 feature is the structure
    msg["MS Gothic (Vista onward)"] --> def["Default glyph: JIS2004-based"]
    msg --> feat["Via the jp90 feature"]
    feat --> old["JIS90-era glyphs"]

Figure 3: Current MS Gothic has JIS2004 glyphs as the default, and can switch to JIS90 glyphs with the jp90 feature.

What matters here is that only the font changed; the data changed not at all.

• The code point of 葛 is U+845B on both XP and Windows 11
• On XP (JIS90 glyphs) it is displayed in the form that simplifies the inside of the wrapping radical to ヒ; from Vista onward (JIS2004 glyphs) it is displayed in the form that writes 人 inside as well
• Therefore the scanned image of a form printed on the old system and the screen display on a new PC disagree in the shape of the character. A data comparison matches completely

Whether 辻’s shinnyo radical has one dot or two, the form of 飴’s “eat” radical, and the like are the same. If you do not know this history, the investigation tends to go in the wrong direction of “the data was corrupted in the migration”. When you are told that a character’s appearance differs before and after a migration, first compare the code points, and if they match, suspect a font glyph difference — that is the correct order.

flowchart TB
    accTitle: The same code point, a different glyph depending on the font
    accDescr: The code point U+845B of 葛 stays the same on both XP and Windows 11; only the displayed shape changes between a JIS90-glyph font and a JIS2004-glyph font, and a data comparison matches completely
    cp["Code point U+845B (葛)"] --> f90["A JIS90-glyph font (XP)"]
    cp --> f04["A JIS2004-glyph font (Vista onward)"]
    f90 --> g90["The form that simplifies the inside to ヒ"]
    f04 --> g04["The printing-standard form that writes 人 inside"]
    g90 -.-> same["A data comparison matches completely"]
    g04 -.-> same

Figure 4: Only the font changed; the code point U+845B stays the same in every environment.

Note that because the character itself did not change, both glyphs are “the same character”. In personal names, however, the person or a government office sometimes insists on a particular form, and answering the demand to distinguish that “as data” is the next topic, IVS.

4. Ideographic Variation Selectors (IVS) — Specifying a Glyph as Data

IVS (Ideographic Variation Sequence) is a mechanism that places an invisible code point called an “ideographic variation selector” immediately after a kanji, to specify a glyph variant as data. The selectors used are U+E0100–U+E01EF (VS17–VS256).³

Which “base character + selector” sequence refers to which glyph is decided by a registry called the IVD (Ideographic Variation Database), managed by the Unicode Consortium. The main collections are as follows.⁴

Microsoft’s documentation, for example, gives the example of U+845B alone (葛) being used in the writing of Nishi-Kasai Station, and U+845B+U+E0100 (VS17) being used in the writing of Katsuragi City, Nara. The same 葛, but which glyph it is can be distinguished as data.³

flowchart TB
    accTitle: An example of distinguishing the same 葛 as data with IVS
    accDescr: 葛 as U+845B alone is used in the writing of Nishi-Kasai Station; the sequence of U+845B followed by VS17 is used in the writing of Katsuragi City; which sequence refers to which glyph is decided by the IVD registry
    seq1["U+845B alone"] --> gl1["The glyph used in the writing of Nishi-Kasai Station"]
    seq2["U+845B + VS17"] --> gl2["The glyph used in the writing of Katsuragi City"]
    ivd["IVD (the registry)"] -.-> gl1
    ivd -.-> gl2

Figure 5: Even with the same 葛, the presence or absence of a selector lets you distinguish which glyph it is as data.

4.1. Behaviour in an Environment That Does Not Support It

On the font side, the correspondence between an IVS and a glyph is implemented in the OpenType cmap table (format 14).⁵ When a supporting font (IPAmj Mincho and the like) and a supporting app are both present, the specified glyph appears; when they are not, it goes as follows.

• The specified correct behaviour: the selector is ignored and the base character’s default glyph is displayed (the selector itself is invisible)
• Older apps and some drawing stacks: the selector is treated as an independent unknown character, and an extra □ is displayed

In other words IVS is designed so that “even if it degrades, the base character is readable”, but a guarantee that “it will always display in the specified glyph” depends on the receiver’s environment. Government resident-record and family-register systems use the combination of a Character Information Platform font plus IVS, but if a general business system accepts it casually, the glyph will drop somewhere in display, print, or a downstream system.

flowchart TB
    accTitle: How IVS-bearing data is displayed
    accDescr: When a supporting font and a supporting app are both present it displays in the specified glyph; when they are not, the selector is ignored and the base character's default glyph is shown; in older apps and some drawing stacks the selector is treated as an unknown character and an extra □ is displayed
    ivs["Base + IVS selector"] --> env{"Supporting font + app?"}
    env -->|Yes| ok["Specified glyph"]
    env -->|No| other{"How is it drawn?"}
    other -->|Ignored| ignore["Default glyph"]
    other -->|Older / some stacks| tofu["An extra □"]
    ignore -.-> spec["Specified correct"]

Figure 6: IVS remains readable as the base character even if it degrades, but whether the specified glyph appears depends on the receiver’s environment.

4.2. An Implementation Caveat — “One Character” Can Be up to Four Code Units

IVS selectors from U+E0100 onward are code points on a supplementary plane, so in UTF-16 they are always a surrogate pair (two code units). If the base character is a supplementary-plane kanji (for example 𠮟 (U+20B9F), added in JIS2004), the base alone is already two code units, and the sequence a user recognises as “one character” is up to four code units in UTF-16, and up to eight bytes in UTF-8.

• C#’s "葛󠄀" (葛+VS17) has string.Length == 3. Substring and fixed-length slicing risk splitting the base character from the selector
• Validation of character count and slicing should be done in grapheme units (APIs such as StringInfo), not code units
• For DB column length (SQL Server’s nvarchar(n) is in UTF-16 code units), if you accept IVS, provision two to four times the apparent character count
• In search and comparison, the presence or absence of a selector makes a different string. Whether a search for “葛” hits “葛+VS17” is something you need to decide as a requirement and implement

flowchart TB
    accTitle: One IVS-bearing character and UTF-16 code units
    accDescr: The sequence of a base character and an ideographic variation selector that a user recognises as one character is always a surrogate pair for the selector, and if the base character is a supplementary-plane kanji another two code units, for a maximum of four code units in UTF-16
    one["One visible character"] --> base["Base character"]
    one --> vs["Variation selector"]
    base -.-> bnote["+2 if supplementary"]
    vs -.-> vnote["Always 2 code units"]
    base --> total["Up to 4 UTF-16 units"]
    vs --> total
    total -.-> risk["Split in fixed slicing"]

Figure 7: One IVS-bearing character can be up to four code units in UTF-16; slicing by code unit is dangerous.

5. Gaiji (EUDC) — Characters That Display Only on That PC

Gaiji is a mechanism in which a user assigns a glyph of their own to a code point in the Unicode Private Use Area (PUA: U+E000–U+F8FF and the like). A Private Use Area code point has no worldwide-agreed meaning; the same U+E000 can be assigned a different character per PC and per organisation.⁶

On Windows you create the glyph with the Private Character Editor (eudcedit.exe), and it is saved in a font file called eudc.tte. This file is installed as a hidden font and associated with each font in the HKEY_CURRENT_USER\EUDC registry.⁷ In the Shift_JIS (CP932) era the gaiji range was 0xF040–0xF9FC, and on conversion to Unicode it is mapped to the Private Use Area.

The consequence of this mechanism is clear.

• eudc.tte belongs to that PC (that user) and does not travel to the other party with the data
• The moment it is passed to mail, a PDF, the Web, or another system, it becomes □ or looks like the other side’s different gaiji
• If you forget to migrate eudc.tte in an OS migration or a PC replacement, “a character that displayed on the old PC will not display” occurs

This is the identity of the second consultation in the opening.

flowchart TB
    accTitle: Why gaiji display only on that PC
    accDescr: A glyph created in the Private Character Editor is saved in eudc.tte and associated with fonts in that PC's registry, so if only the Private Use Area code is passed to mail, a PDF, or another system it becomes □ or looks like a different character
    edit["Create a PUA glyph"] --> tte["Save in eudc.tte"]
    edit -.-> editN["Private Char. Editor"]
    tte --> reg["Registry font mapping"]
    reg --> local["Displays on that PC"]
    tte -.-> stay["eudc.tte stays behind"]
    send["Only the PUA code goes"] --> dest["Mail / PDF / other sys"]
    dest --> broken["□ or a wrong character"]
    local ~~~ send

Figure 8: The glyph lives in eudc.tte; only a Private Use Area number remains in the data, so gaiji look broken once they leave the PC.

5.1. A Realistic Answer for a System That Has Already Received Gaiji

The problem is when data inherited from a legacy system already has gaiji mixed in. The procedure we recommend on migration engagements is as follows.

1. Investigate: scan databases and files with a regular expression for the Private Use Area (U+E000–U+F8FF), and inventory the gaiji codes in use and their counts. Collect eudc.tte from PCs at each site and confirm the glyphs
2. Identify: for each gaiji, investigate “can it be represented as a regular Unicode character”, “can it be represented with IVS”, “is there a corresponding character in the Character Information Platform (MJ)”, and build a substitute-character correspondence table. In practice the majority of cases are simply that an old form had been made as a JIS gaiji
3. Replace: replace the data from the correspondence table. Only when there is truly no corresponding character, keep it as an image or attach a note to that record
4. Cut off: in the new system, reject Private Use Area input in validation, and do not create new gaiji

flowchart TB
    accTitle: The procedure for migrating data that contains gaiji
    accDescr: Inventory gaiji in use by scanning the Private Use Area and collecting eudc.tte, build a substitute-character correspondence table and replace, and in the new system reject Private Use Area input in validation and do not create new gaiji
    st1["Investigate: scan PUA"] --> st2["Identify: subst. table"]
    st2 --> st3["Replace from the table"]
    st3 --> st4["Cut off: no new gaiji"]
    st1 -.-> tte["Collect eudc.tte"]
    st2 -.-> nomap["No map: image or note"]

Figure 9: Migrate gaiji in the four stages investigate, identify, replace, and cut off, and do not create new gaiji.

The direction is the same on the government side: a policy has been stated of uniquely identifying the gaiji municipalities have created on their own (said to be about two million characters nationwide) against the Standard Characters for Administrative Affairs described later, and stopping their use.¹⁰ “Do not increase gaiji; identify them against a standardised character set” is becoming the established migration pattern in both the public and private sectors.

6. The Government Character Platform — From Koseki Unified Characters to Standard Characters for Administrative Affairs

In the design of a system that handles personal names, knowing the government-side character platform becomes material for deciding “how far to accept”.

In municipal core-business systems (systems that conform to the standard), a two-tier structure is in the standard specification: use Standard Characters for Administrative Affairs for information interoperation of personal names and the like, and interoperate with external systems that have no unified interoperation rules — smartphones and the like — in the scope of JIS X 0213:2012.¹⁰ The structure itself of “hold a wide character set internally, and exchange with the outside in a range that a general environment can display” is also a reference for private-sector systems.

flowchart TB
    accTitle: The two-tier interoperation of a standard-conforming system
    accDescr: A municipal standard-conforming system uses Standard Characters for Administrative Affairs for information interoperation of personal names and the like, and interoperates with external systems such as smartphones that have no unified interoperation rules in the scope of JIS X 0213:2012
    sys["Municipal standard sys"] --> renkei["Name interoperation"]
    sys --> gaibu["External systems"]
    renkei --> mjp["Admin standard chars"]
    mjp -.-> mjpN["Personal names etc."]
    gaibu --> jis["JIS X 0213:2012 scope"]
    gaibu -.-> sumaho["No rules（smartphone）"]
    mjp -.-> naibu["Wide set held inside"]

Figure 10: A two-tier structure: government interoperation uses Standard Characters for Administrative Affairs; external interoperation with no rules uses JIS X 0213:2012.

As practical guidance for a general business system, we recommend the following.

• Decide the accepted character set and state it in both the specification and input validation. For example “the scope of JIS X 0213:2012”, “the Private Use Area and combining characters are not allowed”, “IVS is not accepted (or is accepted, but display is guaranteed only in an IPAmj Mincho environment)”
• Do not accept without limit. A design of “it is Unicode, so anything goes” will break somewhere in display, print, or interoperation
• Decide the operation for out-of-range characters in advance. The rule for substituting an alternative representation (a new form, katakana) and the wording you explain to the person are themselves system specification
• When a downstream system such as government or finance has a character-set rule, take that as authoritative and align with it

flowchart TB
    accTitle: Designing and operating an accepted character set
    accDescr: Decide the accepted character set and state it in both the specification and input validation; accept in-range characters; for out-of-range characters, decide the operation including the rule for substituting an alternative representation and the wording you explain to the person
    decide["Decide the accepted character set"] --> spec["State it in the specification"]
    decide --> valid["State it in input validation"]
    valid --> range{"In range?"}
    range -->|Yes| ok["Accept"]
    range -->|No| alt["Substitute an alternative representation"]
    alt -.-> word["The wording you explain to the person is also specification"]

Figure 11: State the accepted character set in both the specification and input validation, and decide the out-of-range operation as well.

7. Choosing and Embedding Fonts — Aligning Screen and Form

7.1. The Character of the Usual Fonts

What matters in the choice is less a preference of typeface than whether that font exists in every environment involved in display, print, and PDF generation. Japanese supplemental fonts on Windows 10/11 (BIZ UD and the like) are sometimes not present depending on the configuration, and in a configuration that generates PDFs on the server side, the presence or absence of the font on the server has a direct effect.

flowchart TB
    accTitle: The environments to confirm when choosing a font
    accDescr: In choosing a font, what matters is less a preference of typeface than whether that font exists in every environment involved in display, print, and PDF generation; the configuration of supplemental fonts and the presence or absence of the font on the server have a direct effect
    cand["A candidate font"] --> exist["On every environment?"]
    exist --> scr["Display environment"]
    exist --> more{"Print or PDF server?"}
    more --> prn["Print environment"]
    more --> srv["PDF-generation server"]
    scr -.-> hojo["Supplemental font?"]
    hojo -.-> hojoN["May be absent"]
    srv -.-> eikyo["Font on server matters"]

Figure 12: Choose a font less by typeface preference than by whether it is present in every environment of display, print, and PDF generation.

7.2. The Basics of Form Design — Align, and Embed

• Specify the same font on screen and on the form. If the fonts differ, the same data can look like a different glyph, and you get the opening complaint. A configuration such as “Meiryo on screen, MS Mincho on the form” should at least be checked for whether there is a glyph difference on the 168 JIS2004 characters
• Embed the font in the PDF. If you do not embed, the viewing side substitute-draws with a font it has on hand, and not only the glyph but the layout can change
• Whether embedding is allowed is determined by the licence. An OpenType font declares embedding permissions in the fsType field (Installable / Restricted / Preview & Print / Editable, no-subsetting, and the like), and you must not embed a font whose embedding is not licensed.¹¹ For a commercial font, confirming the contract is required
• Make subset embedding the baseline. If you embed only the glyphs of the characters used, you do not have to take on a whole Japanese font (several MB to tens of MB)
• If there is a long-term-retention requirement, PDF/A. PDF/A (ISO 19005) is a standard that completes the resources needed for display inside the file, and font embedding is required.¹² It is also the most reliable way to prevent “ten years later I opened it and the glyphs had changed”

flowchart TB
    accTitle: The decision flow for font embedding
    accDescr: Before embedding a font in a PDF, confirm the fsType embedding licence; if it is licensed, make subset embedding the baseline; if there is a long-term-retention requirement, consider PDF/A, which requires embedding
    emb["Embed the font in the PDF"] --> lic{"Is embedding licensed by fsType?"}
    lic -->|Licensed| sub["Subset embedding is the baseline"]
    lic -->|Not licensed| ng["You must not embed"]
    sub -.-> gly["Only the glyphs of the characters used"]
    sub -->|A long-term-retention requirement| pdfa["Consider PDF/A"]
    pdfa -.-> must["Font embedding is required"]

Figure 13: Embedding assumes confirming the fsType licence; subset embedding and PDF/A are the baseline.

How to choose an implementation means for print and PDF output is covered in depth in “Printing and PDF Output in Windows Business Apps”.

8. Font Linking and Fallback — The Phenomenon of “A Different Font Gets Mixed In”

A character for which the specified font has no glyph is not displayed as nothing; substitute-drawing in another font is the default behaviour of modern drawing stacks. In GDI, “font linking” defined in the registry (FontLink\SystemLink) does this; in DirectWrite, WPF, and browsers, “font fallback” does.¹⁵

flowchart TB
    accTitle: The flow of font linking and fallback
    accDescr: If the specified font has a glyph it is displayed as-is; if not, it is substitute-drawn in a font-link or fallback font; if no glyph exists anywhere it becomes □, but the data is often still alive
    disp["Display a character"] --> has{"Does the specified font have a glyph?"}
    has -->|Yes| draw["Display in the specified font"]
    has -->|No| fb{"Is it in a link or fallback target?"}
    fb -->|Yes| alt["Substitute-draw in another font"]
    alt -.-> mixed["The cause of a mixed typeface feel"]
    fb -->|No| tofu["□ (tofu) is displayed"]
    tofu -.-> alive["The data is often still alive"]

Figure 14: □ is the trace of a fallback failure; whether substitute-drawing succeeds is the fork between “mixed” and “tofu”.

Knowing this mechanism lets you explain the following common cases.

• The typeface feel differs between Latin and Japanese: because a Latin font was specified first, only the Japanese portion is being drawn in a linked or fallback Japanese font
• Only the kanji in a Japanese sentence become a Chinese-style glyph: the fallback target resolved to a Chinese font. Easy to happen in a Web page or app that is not passing language information (a lang attribute or a locale) correctly
• Tofu (□) appears: neither the specified font nor the fallback target has a glyph. In other words □ is the “trace of a fallback failure”, and the data is often still alive

Fallback is a relief mechanism; it is not a substitute for choosing the correct font from the start.¹⁵ In a business app the healthy position is “on the main display and print paths, complete with the designed fonts alone; fallback is insurance for unexpected characters”. For the thinking of font selection in a multilingual UI, see also “Localizing WinForms/WPF Apps”.

9. An Implementation Checklist for Business Apps

Finally, the points to confirm at each layer from input through interoperation are summarised in a table.

Normalization in particular is a trap that is this article’s theme itself: a process applied “with good intentions” that crushes the distinction of variant characters and of fullwidth versus halfwidth. The original as-is; processing on a copy is the principle. Character-encoding accidents in CSV interoperation are covered in depth in “CSV Is Not "Just Text"”.

flowchart TB
    accTitle: The original as-is; processing on a copy
    accDescr: Store the entered string as-is as the original; apply normalization to a copy limited to a use such as generating a search key; applying NFKC to the original loses the distinction of variant characters and of fullwidth versus halfwidth
    input["The entered string"] --> orig["Original: store as entered"]
    input --> copy["Copy: normalize, limited to a use"]
    copy -.-> use["Generating a search key, and the like"]
    orig -.-> ng["NFKC on the original crushes distinctions"]

Figure 15: Limit normalization to a use and apply it to a copy; store the original as entered.

10. Summary

• Split character trouble first into the “data layer (character encoding)” and the “appearance layer (fonts)”. � is a sign of a data-layer accident, □ of an appearance-layer accident.
• JIS X 0213:2004 changed the exemplar glyphs of 168 characters, and Windows has JIS2004 glyphs as the default from Vista onward. 葛, 辻, and 飴 looking different by environment is the history of fonts, not data corruption.
• The standard means of fixing a glyph as data is IVS, but without a supporting font and a supporting app it falls back to the default glyph. Do not forget the implementation impact that one character can be up to four UTF-16 code units.
• Gaiji (EUDC) is an asset specific to that PC and cannot travel with the data. The realistic answer is to inventory at migration time, replace from a correspondence table to regular characters or IVS, and stop creating new ones.
• A system that handles personal names decides the accepted character set and states it. The government is standardising toward Standard Characters for Administrative Affairs on the foundation of Koseki Unified Characters and the Character Information Platform, and a system that interoperates needs to follow that movement.
• For forms and PDFs, “align the font with the screen, confirm the licence, and embed” is the baseline. For long-term retention, consider PDF/A.
• NFKC normalization, slicing by code unit, and CP932 conversion are the three points that quietly break variant characters and gaiji. Make storing the original and processing in grapheme units the principle.

The next time you are told “the character is different”, first recast the question this way. Are the code points the same, or different? If they are the same it is a font problem; if they are different it is a data problem. That one move keeps you from taking the wrong entry point for the investigation.

flowchart TB
    accTitle: The first question that decides the entry point of the investigation
    accDescr: When you are told the character is different, first compare whether the code points are the same or different; if they are the same start the investigation as a font problem, if they are different as a data problem
    said["You were told the character is different"] --> cmp{"Are the code points the same?"}
    cmp -->|The same| fontp["A font problem"]
    cmp -->|Different| datap["A data problem"]

Figure 16: If the code points are the same, start the investigation as a font problem; if they are different, as a data problem.

Related Articles

• An Introduction to Windows Text Encodings - The Mojibake That Happens When Integrating with Linux
• Windows Text Encodings and Line Endings - The Basics of Mojibake and CRLF/LF
• Printing and PDF Output in Windows Business Apps — Choosing Between System.Drawing.Printing, WPF, and Report Libraries
• Localizing WinForms/WPF Apps — resx, Satellite Assemblies, and Culture Switching in Practice
• CSV Is Not “Just Text”: A Practical Guide to CSV Handling in C# Business Apps (Encoding, Excel Compatibility, Injection Defense)
• An Introduction to Windows App Accessibility — Preparing for UI Automation and Reasonable-Accommodation Requirements

Related Consulting Areas

KomuraSoft LLC handles design and investigation around characters in business systems. From isolating the cause of symptoms such as “the character differs on screen and on the form” or “after the migration a personal name became □”, through inventorying gaiji and building a substitute-character table at migration from a legacy system, designing the accepted character set of a system that handles personal names, and reviewing the font-embedding configuration of forms and PDFs, we cover both the code layer and the font layer.

• Windows Application Development
• Legacy Asset Migration
• Technical Consulting & Design Review
• Contact Us

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

Why does the same 葛 character look different depending on the PC or the printed form?

It is more likely a difference in font glyphs than mojibake. JIS X 0213:2004 (JIS2004) revised the exemplar glyphs of 168 kanji to the printing-standard forms, and Windows too made the JIS2004 glyphs the default in MS Gothic / MS Mincho and others from Vista onward. 葛, 辻, 飴 and the like are representative examples: the Unicode code point (the data) stays the same, and only the glyph the font holds (the appearance) has changed. Compare the data and they match; a mismatch of shape between an XP-era form image and the screen of a new PC is specified behaviour. If you want the glyphs aligned as well, use the same font on screen and on the form, or specify the glyph with an ideographic variation selector.

If we use ideographic variation selectors (IVS), does that solve every personal-name glyph problem?

It does not. IVS is a mechanism that places a selector from U+E0100 onward immediately after a base character to specify a glyph as data, and the specified glyph is displayed only when a supporting font such as IPAmj Mincho and a supporting app are both present. In a non-supporting environment the specified behaviour is that the selector is ignored and the base character's default glyph is shown; in some environments the selector can also appear as □. Further, one IVS-bearing character can be up to four code units in UTF-16, which affects character-count, slicing, and the design of DB column lengths. If you introduce it, confirm the scope of support through display, print, and downstream systems before you use it.

Can a character registered as gaiji (EUDC) display on another PC or in a PDF?

In principle it cannot. Gaiji is a mechanism in which a user registers a glyph into that PC's eudc.tte file at a code point in the Unicode Private Use Area (U+E000 onward); the same code point is undefined or a different glyph on another PC. The fate of passing it to mail, a PDF, or another system is therefore that it becomes □ or looks like a different character. If you have already inherited data that contains gaiji, the realistic path at migration time is to inventory uses of the Private Use Area, build a correspondence table to regular Unicode characters or ideographic variation selectors, and replace. You should avoid creating new gaiji in a new system.

How far should a business system accept characters in personal names?

The first thing is to "decide the accepted character set and state it as a specification". Family registers have about 56,000 Koseki Unified Characters, and systems that conform to the government standard are moving toward using Standard Characters for Administrative Affairs, an extension of the Character Information Platform — but a general business system is under no obligation to accept the same level without limit. A realistic design is to decide a range such as "up to the scope of JIS X 0213" or "do not accept ideographic variation selectors or the Private Use Area", validate at input time, and operate out-of-range cases with an alert or an alternative representation. Only systems that interoperate with government systems or municipalities need to follow the movement of Standard Characters for Administrative Affairs and the JIS X 0221-based interoperation requirements.

How do we make a form or PDF show the same characters as the screen?

The baseline is to specify the same font on screen and on the form, and to embed the font in the PDF. If the fonts differ, the same data can change glyph; if the viewing PC does not have the font, a substitute font is used for drawing and the appearance breaks. Whether embedding is allowed is determined by the font's licence (OpenType fsType), so confirm it rather than leaving it to the report library. Subset embedding, which embeds only the characters used, also keeps the file size down. If long-term retention is a requirement, consider PDF/A, which requires font embedding.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.