An Introduction to Windows Text Encodings - The Mojibake That Happens When Integrating with Linux

Mojibake on Windows does not happen because Japanese is difficult. Almost all of it is caused by reading the same byte sequence as a different encoding, or by saving the result of a misread in yet another encoding.

Crossing between Windows and Linux makes this especially visible. The Windows side still carries several contexts at once - CP932, UTF-8, UTF-16, the console code page, PowerShell version differences - while the Linux side mostly flows on a UTF-8 assumption, so mismatched assumptions that had been invisible until then surface all at once.

flowchart TB
    accTitle: How assumptions diverge between Windows and Linux
    accDescr: A diagram showing that the Windows side keeps several contexts alive such as CP932, UTF-16, the console code page and PowerShell version differences, and that combining them with a Linux side that mostly runs on a UTF-8 assumption brings the mismatched assumptions to the surface.
    w1["Windows side"] --> w2["CP932 / UTF-16 / code page / version differences"]
    l1["Linux side"] --> l2["Strong UTF-8 assumption"]
    w2 --> z1["Mismatched assumptions surface"]
    l2 --> z1

Figure 1: Where the several contexts left on the Windows side meet the UTF-8 assumption of the Linux side, the mismatch surfaces all at once.

This is less about the difficulty of Japanese text processing and more about whether the assumptions under which bytes are handled have been lined up. This article organizes Windows text encodings from the angle of why mojibake happens, with a practical focus on the points where things break more often once Linux is part of the picture.

The intended reader is someone who hands CSVs, logs, or configuration files created on Windows over to Linux (or the other way around) and wants to isolate and repair mojibake without help. No specific language or framework knowledge is assumed. The command examples use iconv and PowerShell.

1. What to Grasp First

Stating the essentials up front, six points matter.

• Mojibake is not a problem of characters. It is a problem of how a byte sequence was interpreted.
• On Windows the Unicode world and the legacy code page world coexist, and even within a single machine the assumption differs by context.
• The Linux side carries a strong UTF-8 assumption, so mixing in CP932 or UTF-16 from the Windows side breaks things easily.
• The stage where only the display is garbled and the stage where the corrupted content has been saved should be thought about separately.
• The safe default is to make UTF-8 the first choice for new text and to leave existing legacy files exactly as they are until an explicit migration task.
• A file’s encoding, an editor’s encoding, the console code page, and an app’s internal string format are different things. Confuse them and the investigation goes in circles.

The phrase “it got garbled on Windows” is not enough to pin down a cause. At minimum you need to separate which of the following is out of line.

• The encoding of the file itself
• The encoding used at save time
• The editor’s interpretation
• The console’s input and output code pages
• The app’s internal string format
• The locale and assumed encoding on the Linux side

flowchart TB
    accTitle: The layers to separate when isolating mojibake
    accDescr: A diagram showing that saying it got garbled on Windows is not enough to pin down a cause, and that you need to separate which of the file itself, the save-time encoding, the editor, the console code page, the app internals or the Linux locale is out of line.
    s0["It got garbled on Windows"] --> q1{"Which one is out of line"}
    q1 --> a1["The encoding of the file itself"]
    q1 --> a2["The encoding used at save time"]
    q1 --> a3["The editor interpretation"]
    a1 --> a4["The console code page"]
    a2 --> a5["The app internal string format"]
    a3 --> a6["The Linux locale"]

Figure 2: “It got garbled” does not settle the cause on its own. You have to look at which of these six layers slipped.

1.1 Terms used up front

Here are short definitions for the abbreviations that appear later without explanation.

Knowledge map for this article

Mojibake on Windows comes from a mismatch between the assumptions used to encode and to decode, such as reading a byte sequence saved as CP932 as if it were UTF-8 or the other way around, and the corruption looks different depending on the direction. Windows PowerShell is split between 5.1 and 7 or later in its default encoding, and differences per path, such as Out-File writing UTF-16LE while Set-Content writes CP932, are a breeding ground for accidents. The Linux side reads on the assumption of UTF-8 according to the locale, so handing it CP932 or UTF-8 with a BOM leads to mojibake and data corruption, and this happens especially easily in environments where the two live side by side, such as WSL. As long as the original bytes are still there, the data can be recovered by reading it back with the correct encoding and writing it out again with iconv or PowerShell, but once the misread content has been saved it cannot be restored. The article recommends making UTF-8 the first choice for new files and writing the encoding itself down explicitly as part of the I/O contract.

flowchart LR
    accTitle: Windows mojibake and Linux integration
    accDescr: Diagram showing how the symptoms of mojibake change depending on whether the same byte sequence is read as CP932 or as UTF-8, how the BOM, UTF-16LE, and the console code page are independent layers, how the default encoding differs by PowerShell version, and how recovery with iconv or PowerShell relates to an operational policy that makes UTF-8 the first choice
    mojibake["Mojibake (Garbled Text)"]
    cp932["CP932 (Windows-31J)"]
    utf_8["UTF-8"]
    utf_16le["UTF-16LE"]
    bom["BOM (Byte Order Mark)"]
    encoding_overwrite_corruption["Corruption from Saving Mis-Decoded Text"]
    console_code_page["Console Code Page"]
    windows_powershell_5_1["Windows PowerShell 5.1"]
    powershell_7["PowerShell 7"]
    iconv["iconv"]
    powershell["PowerShell"]
    linux_locale["Linux Locale (LANG/LC_ALL)"]
    wsl["WSL (Windows Subsystem for Linux)"]
    encoding_as_interface["Encoding as an I/O Contract"]
    utf8_first_policy["UTF-8 First Policy for New Files"]
    cp932 -.->|"may cause"| mojibake
    utf_8 -.->|"may cause"| mojibake
    utf_16le -.->|"may cause"| mojibake
    bom -.->|"may cause"| mojibake
    mojibake -.->|"may cause"| encoding_overwrite_corruption
    console_code_page -.->|"may cause"| mojibake
    windows_powershell_5_1 -->|"uses"| utf_16le
    windows_powershell_5_1 -->|"uses"| cp932
    powershell_7 -->|"uses"| utf_8
    iconv -.->|"mitigates"| mojibake
    powershell -.->|"mitigates"| mojibake
    mojibake -->|"verified by"| iconv
    linux_locale -.->|"uses"| utf_8
    wsl -->|"uses"| linux_locale
    wsl -.->|"may cause"| mojibake
    encoding_as_interface -->|"recommended for"| mojibake
    utf8_first_policy -->|"recommended for"| mojibake

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (17 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

2. What Mojibake Actually Is

What mojibake actually is turns out to be quite simple.

1. A string is encoded with some encoding into a byte sequence
2. That byte sequence is decoded with some encoding back into a string
3. If the encode and decode assumptions do not match, it reads as a different string

flowchart TB
    accTitle: Whether the encode and decode assumptions match
    accDescr: A diagram showing that when a byte sequence produced by encoding a string is turned back into a string by decoding, a matching assumption returns the original string while a mismatched one produces a different string.
    e1["Encode a string into bytes"] --> e2["Decode the bytes back into a string"]
    e2 --> j1{"Do the assumptions match"}
    j1 -->|"Match"| r1["You get the original string back"]
    j1 -->|"Mismatch"| r2["It reads as a different string"]

Figure 3: All mojibake really is is a mismatch between the encode assumption and the decode assumption.

For example, saving あ as UTF-8 produces these bytes.

E3 81 82

Read those bytes as UTF-8 and you get あ; read them in a CP932 context and they look like a different string, something like 縺�. That is mojibake.

The point to hold onto is that what happened here is not that “the Japanese broke.” It is only that the interpretation of the same bytes diverged.

It is worth looking at the reverse direction too. Saving あ as CP932 gives these bytes.

82 A0

Try to read those bytes as UTF-8 and neither 0x82 nor 0xA0 is valid as a UTF-8 lead byte, so both become replacement characters and you see something like ��. From the UTF-8 side, this is not even a valid character to begin with.

A slightly longer example brings out the pattern better. Saving 日本語 as CP932 gives this.

93 FA 96 7B 8C EA

Read as UTF-8, that comes out as something like ���{��. The thing to notice is that only the fourth byte, 0x7B, passes straight through as the ASCII {. The second byte of a CP932 pair can fall in the ASCII range, which is why symbols such as { or \ end up scattered through the garbled result.

So the symptoms differ by direction.

A run of unreadable kanji means UTF-8 is being read as CP932; a wall of replacement characters means CP932 is being read as UTF-8. That asymmetry gives you a quick first guess, so it is worth remembering when you are isolating a problem.

2.1 If only the display is garbled, recovery may still be possible

Mojibake has a stage where things can still be recovered. If the original bytes are unchanged, for instance, reopening the file with the correct encoding can bring it back.

What is dangerous is a flow like this.

1. A UTF-8 file is misread as CP932
2. On screen it looks like 縺�
3. The string as displayed is saved as-is
4. The original UTF-8 bytes are gone

Once you are past that point, it is no longer a display issue. It is data corruption.

flowchart TB
    accTitle: How a garbled display turns into data corruption
    accDescr: A diagram showing that going from the stage where a UTF-8 file is misread as CP932 and looks garbled on screen to the stage where the displayed string is saved and the original UTF-8 bytes are lost turns a display problem into data corruption.
    d1["A UTF-8 file is misread as CP932"] --> d2["It looks garbled on screen"]
    d2 --> d3["The displayed string is saved as-is"]
    d3 --> d4["The original UTF-8 bytes are gone"]
    d2 -.-> n1["Up to here reopening it still recovers it"]
    d4 -.-> n2["From here on it is data corruption"]

Figure 4: A misread alone is recoverable, but the moment the misread content is saved it turns into data corruption.

2.2 More dangerous still: dropping unrepresentable characters into a narrow code page

The other classic failure is converting a Unicode string down into a legacy code page such as CP932.

When the string contains characters that do not exist in the destination code page, you get things like:

• replacement with ?
• the replacement character � appearing
• conversion to a similar but different character
• outright conversion failure

This one should be judged not by readable versus unreadable but by whether a round-trip conversion returns the original. Once a character is lost, knowing the correct encoding afterwards cannot bring it back.

flowchart TB
    accTitle: What goes wrong when dropping into a narrow code page
    accDescr: A diagram showing that converting a Unicode string into a narrow code page such as CP932 turns characters missing from the destination into substitutions or conversion failures, that a round trip no longer returns the original, and that a character once lost cannot be restored even when the correct encoding is known.
    u1["Unicode string"] --> u2["Convert into a narrow code page such as CP932"]
    u2 --> j1{"Characters missing from the destination"}
    j1 --> r1["Turn into ? or a replacement character"]
    j1 --> r2["Become a different character or fail to convert"]
    r1 --> k1["A round trip no longer returns the original"]
    r2 --> k1
    k1 -.-> n1["Lost characters cannot be restored"]

Figure 5: Judge a drop into a narrow code page not by readable versus unreadable but by whether a round trip returns the original.

3. Why Things Get So Tangled on Windows

Windows is tangled not simply because it is old, but because the Unicode world and the legacy code page world still live side by side.

3.1 The Windows API carries both a Unicode lineage and a code page lineage

The Windows API has two broad lineages.

• The W family: wide character. Handles Unicode as UTF-16
• The A family: the code page lineage, known as ANSI

In other words, Windows has had both a path that handles text as Unicode and a path that handles it through the active code page from the very beginning. So even on one Windows machine, the assumption changes depending on which API and which tool the text went through.

flowchart TB
    accTitle: The two lineages of the Windows API
    accDescr: A diagram showing that the Windows API has always had both a W family that handles Unicode as UTF-16 and an A family that handles text through the active code page, so the assumption changes depending on which path the text took.
    api["Windows API"] --> w1["W family (wide character)"]
    api --> a1["A family (known as ANSI)"]
    w1 --> w2["Handles Unicode as UTF-16"]
    a1 --> a2["Handles text via the active code page"]
    w2 --> z1["The assumption changes with the path taken"]
    a2 --> z1

Figure 6: Windows has had both the Unicode path and the code page path from the start.

3.2 “Japanese on Windows” is not one thing

In day-to-day work, four things get mixed up most often around Japanese text on Windows.

• CP932: turns up constantly in legacy Japanese Windows text
• UTF-8: increasingly common in newer text assets, on the web, and in cross-platform work
• UTF-16LE: still perfectly normal in the context of Windows tools and APIs
• The console code page: a separate layer that governs input and output for cmd.exe and some console tools

The important point here is that running chcp 65001 does not make your files UTF-8. Changing the console code page and what bytes an existing file holds are separate questions.

flowchart TB
    accTitle: chcp 65001 and existing files are separate questions
    accDescr: A diagram showing that chcp 65001 only changes the console code page, that the bytes in an existing file are untouched, and that the console setting and the file contents are separate questions.
    c1["Run chcp 65001"] --> c2["The console code page changes"]
    f1["The bytes in an existing file"] --> f2["Nothing changes"]
    c2 --> n1["The console and the file are separate questions"]
    f2 --> n1

Figure 7: All chcp 65001 changes is how the console interprets bytes. The bytes in the file stay exactly as they were.

Legacy Japanese Windows text is often loosely called Shift_JIS, but in practice keeping the name CP932 in your head keeps conversations from drifting. At minimum it makes explicit that you are talking about the Windows-derived Japanese legacy encoding.

3.3 File names and file contents are separate problems

When Japanese file names show up correctly on Windows, the reflex is to assume the contents must be fine too. That is where the danger is.

• The layer that handles paths and file names
• The layer that reads file contents
• The layer that displays to the console

These three are distinct.

Japanese paths may work without a hitch while the file contents, saved as CP932, break when the Linux side reads them as UTF-8. Conversely, the contents may be UTF-8 and only the display breaks because the console code page does not match.

Drawn as a diagram, the layers look like this.

flowchart LR
    W["Writer<br/>app / editor / script"] --> FB["The bytes in the file<br/>the only fact here"]
    FB --> R1["Reader A: editor<br/>auto-detected or specified encoding"]
    FB --> R2["Reader B: console<br/>input and output code page"]
    FB --> R3["Reader C: app internals<br/>the library default encoding"]
    FB --> R4["Reader D: Linux side<br/>UTF-8 assumed per the locale"]
    R1 --> S1["Only the display breaks<br/>re-saving turns it into corruption"]
    R2 --> S2["Only the display breaks<br/>the file is intact"]
    R3 --> S3["The processing result breaks<br/>and propagates downstream"]
    R4 --> S4["Decode error or replacement characters"]

Figure 8: The only fact is the bytes in the file. The editor, the console, the app, and the Linux side are independent readers of it.

Two things are worth watching. The first is separating whether the breakage is in the byte sequence in the middle or in one of the readers on the right. The second is that the four readers on the right are independent of each other, so verifying one guarantees nothing about the other three. That shape is exactly why “it was readable in the console, so the editor will be fine” does not hold.

3.4 The defaults in PowerShell and nearby tools are not aligned either

A quiet multiplier of breakage on Windows is that the same act of “writing some text” produces different output bytes depending on the path taken.

The points to watch in particular:

• Windows PowerShell 5.1 has no consistent default encoding
• Some cmdlets and redirection produce UTF-16LE
• Other paths use the active ANSI code page
• PowerShell 7 and later defaults to UTF-8 no BOM

So “text produced by PowerShell” does not determine the encoding on its own. You have to know which version, which cmdlet, and which write path were used.

Which path produces which bytes is laid out in about_Character_Encoding on Microsoft Learn. Pulling out just the common ones:

The read side differs too. When reading a file with no BOM, Get-Content in 5.1 assumes ANSI, while Import-Csv and Select-String assume UTF-8. The assumption is split inside a single session.

What bites hardest in practice is that the same act of writing text produces UTF-16LE through Out-File and CP932 through Set-Content. The “binary-looking text riddled with NUL bytes” mentioned in 4.3 usually comes from the default behavior of > or Out-File.

flowchart TB
    accTitle: The asymmetry on the read side of Windows PowerShell 5.1
    accDescr: A diagram showing that when reading a file with no BOM, Get-Content in Windows PowerShell 5.1 assumes ANSI while Import-Csv and Select-String assume UTF-8, leaving the assumption split inside a single session.
    b1["A file with no BOM"] --> g1["Read with Get-Content"]
    b1 --> i1["Read with Import-Csv or Select-String"]
    g1 --> g2["Assumes ANSI"]
    i1 --> i2["Assumes UTF-8"]
    g2 --> z1["The assumption is split inside one session"]
    i2 --> z1

Figure 9: In 5.1 the same BOM-less file gets a different assumed encoding depending on which cmdlet reads it.

There is one more: in 5.1, specifying -Encoding UTF8 still produces a BOM. To write UTF-8 without a BOM from 5.1, you go through .NET.

# Write UTF-8 no BOM from Windows PowerShell 5.1
$text = "Body text containing Japanese"
[System.IO.File]::WriteAllText(
    "C:\work\output.txt", $text,
    (New-Object -TypeName System.Text.UTF8Encoding -ArgumentList $false))

Passing $false to UTF8Encoding is what says do not emit a BOM. On PowerShell 7, -Encoding utf8NoBOM gives the same result.

4. Typical Failures When Linux Is in the Mix

It is not unusual for something that more or less worked on Windows alone to break the moment Linux is involved. The reason is simple: the Linux side carries a strong UTF-8 assumption.

4.1 Text saved as CP932 on Windows, read as UTF-8 on Linux

The most common one.

• A legacy Windows app or an old operational process writes CSVs, TXT, or logs in CP932
• Linux-side scripts and tools read them with a UTF-8 assumption per the locale
• The result is decode errors, �, or strings that mean nothing

The Linux tool is not at fault here. The root cause is that the bytes arrived with no agreement about their encoding attached.

flowchart TB
    accTitle: CP932 text read as UTF-8 by Linux
    accDescr: A diagram showing that when a legacy Windows app or an old operational process writes CSVs or logs in CP932 and hands them over with no encoding agreement, Linux-side scripts and tools read them as UTF-8 per the locale and produce decode errors or replacement characters.
    p1["A legacy app writes CSVs and logs in CP932"] --> p2["Handed over with no encoding agreement"]
    p2 --> p3["The Linux side reads as UTF-8 per the locale"]
    p3 --> p4["Decode error / replacement characters"]
    p2 -.-> n1["The root cause is the missing agreement"]

Figure 10: The Linux tool is not the problem. The problem is bytes handed over with no encoding agreement attached.

4.2 UTF-8 no BOM created on Linux or VS Code, treated as ANSI on Windows

The reverse direction breaks too.

• A script, config, or text file is created as UTF-8 no BOM on Linux or in VS Code
• Windows PowerShell 5.1 or a legacy tool treats the BOM-less file as the ANSI-side code page
• Only the lines containing Japanese or other non-ASCII break

UTF-8 tends to take the blame here, but the actual cause is a reader in the chain that will not correctly infer BOM-less UTF-8.

flowchart TB
    accTitle: UTF-8 no BOM being treated as ANSI
    accDescr: A diagram showing that when Windows PowerShell 5.1 or a legacy tool reads a UTF-8 no BOM file created on Linux or in VS Code and treats it as the ANSI-side code page, only the lines containing Japanese or other non-ASCII break.
    v1["Create UTF-8 no BOM on Linux or VS Code"] --> v2["5.1 or a legacy tool reads it"]
    v2 --> v3["Treats the BOM-less file as ANSI"]
    v3 --> v4["Only lines with non-ASCII break"]
    v3 -.-> n1["The cause is a reader that will not infer"]

Figure 11: In the reverse direction the cause is a reader that cannot infer BOM-less UTF-8.

4.3 Windows writes UTF-16LE and Linux says it does not look like text

This one comes up a lot as well.

• Some Windows PowerShell 5.1 output or a legacy tool writes UTF-16LE
• Linux-side text tools expect a single-byte UTF-8 stream
• The result is “binary-looking text” riddled with NUL bytes

UTF-16LE itself is fine. It just often fails to mesh with the assumption of piping text straight into Linux text processing tools.

flowchart TB
    accTitle: UTF-16LE looking like binary on the Linux side
    accDescr: A diagram showing that when UTF-16LE written by some Windows PowerShell 5.1 output or a legacy tool is fed to Linux text tools that expect a single-byte UTF-8 stream, it looks like binary-ish text riddled with NUL bytes.
    u1["Some 5.1 output or a legacy tool"] --> u2["Writes UTF-16LE"]
    u2 --> u3["Linux text tools expect a single-byte stream"]
    u3 --> u4["NUL bytes mixed in and it looks like binary"]

Figure 12: UTF-16LE itself is fine, but it does not mesh with what Linux text processing assumes.

4.4 BOM presence causes friction too

A BOM is not the encoding itself, but in practice it matters a great deal.

• Some Windows-side tools are helped by a BOM
• Some Linux-side tools treat the BOM as extra leading bytes
• The result is a broken first column or first line, invisible junk, or comparisons that no longer line up

With UTF-8 in particular, the same UTF-8 with and without a BOM is a different byte sequence. Saying “we moved to UTF-8” only settles half the operational rule.

flowchart TB
    accTitle: The friction caused by BOM presence
    accDescr: A diagram showing that the same UTF-8 is a different byte sequence with and without a BOM, that some Windows-side tools are helped by a BOM while some Linux-side tools treat it as extra leading bytes, and that this produces friction such as a broken first line.
    b0["The same UTF-8 with and without a BOM is different bytes"] --> w1["Some Windows-side tools"]
    b0 --> l1["Some Linux-side tools"]
    w1 --> w2["Are helped by a BOM"]
    l1 --> l2["Treat it as extra leading bytes"]
    l2 --> l3["The start breaks or invisible junk appears"]

Figure 13: “We moved to UTF-8” is not enough. It becomes a rule only once BOM presence is settled too.

4.5 Trusting what the console shows sends you off course

The other danger when crossing between Windows and Linux is the console.

• The Windows console has input and output code pages
• Linux terminals mostly run on a UTF-8 locale assumption
• Going through WSL, SSH, containers, or CI adds more display paths

In that state, concluding “it was readable in the console, so the file is fine” or “it was garbled in the console, so the file is corrupted” is an easy way to get it wrong. It is safer to verify separately whether what you see is broken and whether the saved bytes are broken.

4.6 The typical failures in a table

5. Drive a Mojibake Investigation with These Four Questions

When a mojibake investigation stalls, going back to these four questions is the fastest way forward.

5.1 What are the original bytes

The first thing to look at is what bytes this file holds right now. You need the habit of looking at bytes rather than at appearance.

• Is it UTF-8
• Is it UTF-8 with BOM
• Is it CP932
• Is it UTF-16LE
• Was it re-saved somewhere along the way and turned into something else

5.2 Who wrote it first, and under what assumption

Next, identify the original writer.

• A legacy Windows app
• PowerShell 5.1 or 7
• A Linux script
• VS Code
• An export from Excel
• Some piece of middleware, a batch job, or CI

Leave this vague and inferring the encoding becomes a matter of luck.

5.3 Who is reading it now, and under what assumption

You need not just the writer but the reader’s assumption as well.

• Is the editor auto-detecting
• Is PowerShell looking at the BOM
• Is the Linux side treating it as UTF-8 per the locale
• Is a library falling back to its default encoding
• Is Encoding.UTF8 or cp932 being specified explicitly

This is where mojibake almost always originates.

5.4 Has the misread content already been saved

Finally, confirm whether the damage has stopped at the display stage.

• Are the bytes still the original ones
• Has someone saved the broken-looking content
• Have ? or � shown up in the diff
• Has the whole file been rewritten in a different encoding

Fill in these four questions and the cause is usually visible.

flowchart TB
    accTitle: The four questions of a mojibake investigation
    accDescr: A diagram showing that working through the four questions of what the original bytes are, who wrote it first and under what assumption, who is reading it now and under what assumption, and whether the misread content has already been saved usually makes the cause of mojibake visible.
    q1["Q1 What are the original bytes"] --> q2["Q2 Who wrote it first and under what assumption"]
    q2 --> q3["Q3 Who is reading it now and under what assumption"]
    q3 --> q4["Q4 Has the misread content already been saved"]
    q4 --> r1["The cause becomes visible"]

Figure 14: When an investigation stalls, going back to these four questions and filling them in order is the fastest route.

6. How to Repair a Broken File

Once the cause is visible, recovery is next. The first thing to settle here is whether the original bytes are still there.

• The original bytes are still there: read the file back with the correct encoding and write it out in the target encoding, and it comes back. That is what this chapter covers
• The misread result has already been saved: your only option is a backup or the Git history. Characters that became the replacement character � or ? cannot be restored even once you know the correct encoding

So the very first thing to do is take a copy of what you are working on. Convert the copy and leave the original file untouched.

flowchart TB
    accTitle: The first branch in a recovery
    accDescr: A diagram showing that recovery starts by checking whether the original bytes are still there, that if they are you read the file back with the correct encoding and write it out, that if the misread result has already been saved your only option is a backup or the Git history, and that the work is done on a copy.
    s1["Take a copy first"] --> j1{"Are the original bytes still there"}
    j1 -->|"Still there"| r1["Read back with the correct encoding and write it out"]
    j1 -->|"Saved and lost"| r2["Restore from a backup or the Git history"]
    r2 -.-> n1["Characters turned into replacement characters cannot be restored"]

Figure 15: Recovery starts with taking a copy, then forks on whether the original bytes are still there.

6.1 If you are on the Linux side, use iconv

To turn a CP932 file into UTF-8, iconv is the most straightforward option.

# CP932 -> UTF-8
iconv -f CP932 -t UTF-8 input.csv > output.csv

If you get the source encoding wrong, it stops partway with something like this.

iconv: illegal input sequence at position 0

Stopping is itself the information that the file is not in that encoding, so the fast move is to try another candidate. If nothing you pass makes it stop, the file may consist of ASCII only.

To go from UTF-16LE down to UTF-8, use -f UTF-16LE. Note, though, that converting a file that has a BOM while explicitly specifying -f UTF-16LE can leave the BOM in the output as a U+FEFF character. If you want iconv to handle the BOM for you, use -f UTF-16, and always check the first character after converting.

6.2 If you are on the Windows side, use PowerShell

From PowerShell 6.2 onward, you can pass a code page number directly to -Encoding. CP932 is 932.

# PowerShell 6.2 and later. CP932 -> UTF-8 no BOM
Get-Content -Path .\input.csv -Encoding 932 |
    Set-Content -Path .\output.csv -Encoding utf8NoBOM

Going the other way, to hand a UTF-8 file created on Linux to a consumer that can only read CP932:

Get-Content -Path .\input.csv -Encoding utf8 |
    Set-Content -Path .\output.csv -Encoding 932

This form has two side effects, though.

1. Get-Content splits the input into lines and Set-Content re-appends a newline after each one. In other words, the newline convention gets normalized for you
2. Even if the original file had no trailing newline, the output will have one

If you want to preserve the bytes right down to the newlines, handle the file whole instead of line by line.

# CP932 -> UTF-8 no BOM with the newlines preserved as-is
$text = [System.IO.File]::ReadAllText(
    "C:\work\input.csv", [System.Text.Encoding]::GetEncoding(932))
[System.IO.File]::WriteAllText(
    "C:\work\output.csv", $text,
    (New-Object -TypeName System.Text.UTF8Encoding -ArgumentList $false))

GetEncoding(932) works as-is on Windows PowerShell 5.1. In environments where it throws on PowerShell 7, run the following once first.

[System.Text.Encoding]::RegisterProvider(
    [System.Text.CodePagesEncodingProvider]::Instance)

Whether to emit a BOM is decided by what the consumer needs, as covered in 7.1. Pass $true to UTF8Encoding in the example above and you get a BOM.

6.3 What to check after converting

A conversion is not finished just because no error appeared. At minimum, check the following.

• Do two or three representative Japanese lines read correctly
• Have ? or � increased. If they have, those characters are already gone
• Has the line count changed
• Are the BOM and the newline convention what the consumer expects
• Has the file size changed drastically. Going from CP932 to UTF-8 grows the Japanese portions from 2 bytes to 3 bytes per character, so a modest increase is normal

The second item matters most. Dropping UTF-8 that contains characters missing from CP932 down into CP932 always loses information. As written in 2.2, check all the way through to whether a round trip returns the original.

flowchart TB
    accTitle: What to check after converting
    accDescr: A diagram showing that a conversion is not finished just because no error appeared, that you check whether representative Japanese lines read correctly, whether question marks or replacement characters have increased, and whether the line count, BOM, newlines and file size are as expected, and that an increase in replacement characters means those characters are already gone.
    c1["The conversion finishes with no error"] --> c2["Read two or three representative Japanese lines"]
    c2 --> c3["Have ? or replacement characters increased"]
    c3 --> c4["Check line count, BOM, newlines and size"]
    c3 -.-> n1["If they increased those characters are gone"]

Figure 16: A conversion is not done at “no error appeared.” Checking the contents is part of the same job.

6.4 Carve bulk conversion out as a migration task

A word on operations to close. Repairing one broken file and moving a whole repository from CP932 to UTF-8 are different jobs. As 7.2 in the next chapter says, do not do the latter on the side of an everyday functional fix. Plan it as an independent migration task.

7. Operational Rules That Reduce Breakage

From here on it is the practical side. In projects that span Windows and Linux, settling the following rules up front cuts the damage substantially.

7.1 Make UTF-8 the first choice for new files

For new text files, making UTF-8 the first candidate is the safe default. But do not stop there. You need to decide what happens with the BOM as well.

Here is a good way to decide.

• Text that is mostly read on the Linux side: default to UTF-8 no BOM
• Scripts read by legacy Windows tools or Windows PowerShell 5.1: state BOM presence explicitly based on what that consumer needs
• If there is a specific consumer that requires UTF-16LE, write that requirement into the spec

Write down only “standardize on UTF-8” and you will be arguing about the BOM later.

flowchart TB
    accTitle: How to decide the encoding for a new file
    accDescr: A diagram showing that a new text file starts with UTF-8 as the first choice but does not stop there, defaulting to UTF-8 no BOM when the Linux side reads it most, stating BOM presence explicitly when a legacy tool or Windows PowerShell 5.1 reads it, and writing the requirement into the spec when a consumer needs UTF-16LE.
    s1["A new text file"] --> s2["Make UTF-8 the first choice"]
    s2 --> j1{"Who reads it"}
    j1 -->|"Mostly the Linux side"| r1["Default to UTF-8 no BOM"]
    j1 -->|"5.1 or a legacy tool"| r2["State BOM presence per the consumer"]
    j1 -->|"A consumer that needs UTF-16LE"| r3["Write the requirement into the spec"]

Figure 17: Start a new file at UTF-8, then finish the decision by settling BOM handling around what the consumer needs.

7.2 Keep existing legacy files as they are until an explicit migration task

If an existing file is CP932, it is safer not to quietly move it to UTF-8 alongside an everyday functional fix.

The safe operational shape is this.

• Existing files keep their original encoding, BOM, and newlines
• Encoding changes are carved out as a migration task
• Convert in bulk only after confirming the targets, the blast radius, and the downstream consumers

Most mojibake damage begins with a well-intentioned “converting it to UTF-8 while I’m in here.”

7.3 Treat the encoding as part of the interface

For CSVs, TXT files, logs, configuration files, and simple protocols, the encoding itself is the interface, not just the content.

At minimum, a spec should state this much.

• Is this file UTF-8, CP932, or UTF-16LE
• If UTF-8, does it carry a BOM
• Are the newlines LF or CRLF
• Which side, Linux or Windows, is the producer and which the consumer
• Does an intermediate batch job or ETL step re-save it

“We hand it over as text” is not a specification.

7.4 Do not trust defaults. Be explicit when writing

In code and in scripts alike, specifying the encoding explicitly is safer.

These are the dangerous lines of thinking.

• Save with whatever the default is
• It will probably come out fine, matching the OS
• It was readable in the console, so the file is probably fine
• Auto-detect exists, so it will be fine

Defaults change routinely between Windows and Linux, between PowerShell 5.1 and 7, and across editors and runtimes. Unless you are explicit, it tends to be working by coincidence.

7.5 Verify the console and the file separately

A quietly effective rule.

• Checking the display in the console
• Checking by reopening the file

Keep these two apart.

Even if chcp and the terminal display line up, it means nothing when the saved file is in a different encoding. Conversely, the file may be perfectly fine and only the appearance breaks because the console display code page does not match.

7.6 Git will not fix your encodings

Unglamorous, but important.

Git fundamentally tracks bytes. Which means it dutifully records broken bytes into history exactly as they are.

So when

• a huge diff appears even though you changed nothing
• only the Japanese lines produce a diff you cannot explain
• only the first line changed
• newlines and encoding changed together

it is better to suspect a re-encoding failure before suspecting a content change.

flowchart TB
    accTitle: Git will not fix your encodings
    accDescr: A diagram showing that Git fundamentally only tracks bytes so broken bytes enter the history as they are, and that when a huge diff appears with no change or only the Japanese lines produce an unexplained diff you should suspect a re-encoding failure before a content change.
    g1["Git only tracks bytes"] --> g2["Broken bytes enter the history as they are"]
    g2 --> g3["Huge diffs or unexplained Japanese-only diffs appear"]
    g3 --> g4["Suspect a re-encoding failure before a content change"]

Figure 18: Git faithfully records broken bytes too, so an unexplained diff should point you at re-encoding first.

8. The Minimum Checklist

Here is the checklist worth pinning down first on a project where Windows and Linux are mixed.

flowchart TB
    accTitle: The flow of the checklist
    accDescr: A diagram showing the flow of checking the current encoding, BOM and newlines before editing, avoiding reliance on defaults and auto-detect while editing, and reopening the file and checking the diff after editing, with bulk conversion and similar work handled separately as a migration task.
    c1["Before editing check the current encoding, BOM and newlines"] --> c2["While editing avoid leaving it to defaults or auto-detect"]
    c2 --> c3["After editing reopen the file and check the diff"]
    c3 -.-> t1["Handle bulk conversion separately as a migration task"]

Figure 19: Split the checks into before, during, and after editing, and carve bulk conversion out into its own task.

8.1 Before editing

• What is this file’s current encoding
• Is there a BOM
• Are the newlines LF or CRLF
• Have you noted two or three representative Japanese lines
• Do you know which side, Linux or Windows, is the final consumer

8.2 While editing

• Are you writing in a way that depends on the default encoding
• Are you saving with auto-detect left in charge
• Are you being careless about PowerShell or shell redirection paths
• Are you taking comfort in nothing more than “the display is readable”

8.3 After editing

• Did you reopen the file and check after saving
• Are the representative lines intact on both the Linux and the Windows side
• Have ? or � increased in the diff
• Is only the first line or the first column broken
• Is the diff a huge BOM-only or newline-only change

8.4 What belongs in a migration task

• Bulk conversion from CP932 to UTF-8
• Unifying the UTF-8 BOM policy
• Taking stock of scripts that assume PowerShell 5.1
• Documenting the text paths that go through CI, containers, WSL, or SSH
• Unifying the save settings of editors, formatters, and batch jobs

9. Summary

If the Windows text encoding problem had to be put in one sentence, the essence is that the Unicode world and the legacy code page world still live side by side.

And the reason more breaks once Linux is added is that the Linux side mostly flows on a UTF-8 assumption, which brings the Windows side’s CP932, UTF-16, console code page, and PowerShell version differences all into the open at once.

Five points are worth remembering.

• Mojibake is a divergence in how bytes are interpreted
• A garbled display and data corruption are different things
• On Windows, think in separate layers: file, editor, console, API
• For text exchanged with Linux, make UTF-8 the first choice
• Keep the conversion of existing legacy files apart from ordinary maintenance

Taken at face value, “it got garbled on Windows” covers far too much ground. But cut it along

• what the original bytes are
• who wrote it and how
• who read it and how
• whether it has already been saved

and it becomes far more tractable.

Text encodings are unglamorous, but between Windows and Linux they are the I/O contract itself. Refusing to leave that ambiguous is the single most effective countermeasure.

10. References

Windows / Microsoft

• Code Pages - Win32 apps | Microsoft Learn
• Code Page Identifiers - Win32 apps | Microsoft Learn
• Unicode in the Windows API - Win32 apps | Microsoft Learn
• Console Code Pages - Windows Console | Microsoft Learn
• chcp | Microsoft Learn
• Use UTF-8 code pages in Windows apps | Microsoft Learn

PowerShell / VS Code

• about_Character_Encoding | Microsoft Learn
• Understanding file encoding in VS Code and PowerShell | Microsoft Learn

GNU / Linux locale

• GNU gettext manual: Header Entry
• Debian Reference, Chapter 8. I18N and L10N

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What causes mojibake on Windows?

Almost all of it comes from reading the same byte sequence as a different encoding, or from saving the result of a misread in yet another encoding. For example, the bytes produced by saving the character あ as UTF-8 (E3 81 82) look like an unrelated string such as 縺 when read in a CP932 context. It does not happen because Japanese is hard; the real cause is that the encode and the decode assumptions do not match.

Can a file that is already garbled be restored?

If the original bytes have not changed, reopening the file with the correct encoding can bring it back. The dangerous path is saving the misread, broken-looking content as-is: once you are past that point it is no longer a display problem but data corruption. And when a Unicode string is dropped into a narrow code page such as CP932 and characters turn into ? or a replacement character, those lost characters cannot be recovered later, even once you know the correct encoding.

Does chcp 65001 make my files UTF-8 as well?

No. Changing the console code page and what bytes an existing file holds are separate questions. On Windows, a file's own encoding, the editor's interpretation, the console's input and output code pages, and the app's internal string format are different things, and you have to keep those layers apart. Concluding that a file must be fine because it displayed correctly in the console is an easy way to get it wrong, so checking the console display and checking by reopening the file should be treated as two separate steps.

What are the safe rules for moving text between Windows and Linux?

Make UTF-8 the first choice for new files, and settle the BOM question as part of that decision. Text that is mostly read on the Linux side should default to UTF-8 no BOM; when legacy tools such as Windows PowerShell 5.1 read it, state BOM presence explicitly based on what that consumer needs. Do not convert existing CP932 files on the side while doing everyday maintenance - separate that out as an explicit migration task. And because for CSVs and logs the encoding itself is the interface, writing the encoding, the BOM, and the newline convention into the spec keeps things from breaking.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.