Windows Text Encodings and Line Endings - The Basics of Mojibake and CRLF/LF

In consultations about text handling on Windows, the following questions get tangled together with remarkable frequency.

• What is the difference between Shift_JIS and UTF-8
• Why does mojibake (garbled text) happen
• What is the difference between CRLF and LF
• We switched to UTF-8, so why can it still be unreadable
• Why does the same file look different in the editor, the console, Excel, and Git

This does not happen because Japanese is hard. Almost all of it is caused by reading the same bytes under a different assumption, or saving the misread content as is.

On top of that, Windows still hosts both the Unicode world and the code-page world side by side. Add BOMs, line endings, editor auto-detection, the console code page, and Git’s newline conversion, and the whole topic looks complicated.

This article organizes, for practical use, the Shift_JIS / UTF-8 / UTF-16 mix-ups common on Windows, why mojibake happens, the difference between CRLF and LF, and why this topic is so confusing in the first place.

The content is based on Microsoft Learn, PowerShell, Git, and W3C / Unicode public documentation as of April 2026. See the references at the end for details.

Who This Is For and What It Assumes

If you are in a hurry, read it like this. For the overall picture, chapters 1 and 2. To isolate a cause, chapters 3 and 8. To settle on operating rules, start at chapter 7.

Table of Contents

1. What to Grasp First
2. Breaking Down the Terminology
   • 2.1 How Unicode / UTF-8 / UTF-16 / CP932 Differ
   • 2.2 How to Think About Shift_JIS vs. CP932
   • 2.3 The Traps in the Words ANSI, Unicode, and UTF-8N
   • 2.4 What a BOM Is, and Why UTF-8 Gets One Too
3. Why Mojibake Happens
   • 3.1 What Mojibake Really Is
   • 3.2 Garbled Display and Data Corruption Are Different
   • 3.3 Converting Down to an Encoding That Cannot Represent the Characters Is Irreversible
4. What the Line-Ending Differences Are
   • 4.1 CRLF / LF / CR
   • 4.2 Line Endings Are a Separate Problem from Encodings
   • 4.3 \n and the Newline Bytes in the File Are Not Necessarily the Same
5. Why Windows in Particular Is So Confusing
   • 5.1 Unicode and Legacy Code Pages Coexist
   • 5.2 The Labels Don’t Line Up
   • 5.3 ASCII-Only Content Hides the Problem
   • 5.4 File Contents, File Names, Console, and Source Files Are Separate Layers
   • 5.5 BOM and Line Endings Act on Yet Other Axes
   • 5.6 Tools Change Things on Their Own
6. Common Failure Patterns
7. Rules That Reduce Breakage in Practice
8. Drive Mojibake / Line-Ending Investigations with These 5 Questions
9. Summary
10. Related Articles
11. References

Knowledge map for this article

Text on Windows is made up of a combination of separate axes: the byte sequence, the character encoding, the line ending, and the BOM. UTF-8 and UTF-16LE are character encodings that turn Unicode code points into bytes, whereas CP932 is a legacy code page specific to Japanese Windows, and mistaking one set of assumptions for the other produces mojibake. The BOM marks the byte order of UTF-16 and at the same time serves in UTF-8 as a signature saying that the file is UTF-8, and Windows PowerShell 5.1 needs that BOM in order to read a script containing non-ASCII characters correctly. The line ending is either CRLF or LF and is a separate matter from the character encoding; how a repository treats it can be pinned down with .gitattributes, but while Git does convert line endings, it does not fix the character encoding for you. Mojibake can often be undone by reopening the file with the correct encoding, but saving the misread content as it stands loses the original byte sequence and turns a display problem into irreversible data corruption.

flowchart LR
    accTitle: Windows character encodings and line endings
    accDescr: Diagram showing the relationship between Unicode and UTF-8, UTF-16LE, and CP932, the role of the BOM, the difference between mojibake and data corruption, and how the PowerShell defaults and Git line-ending conversion act on line endings and on character encoding respectively
    character_encoding["Character Encoding"]
    line_ending["Line Ending (CRLF/LF)"]
    unicode["Unicode"]
    utf_8["UTF-8"]
    utf_16le["UTF-16LE"]
    cp932["CP932 (Windows-31J)"]
    active_code_page["Active Code Page (ANSI/ACP)"]
    bom["BOM (Byte Order Mark)"]
    mojibake["Mojibake (Garbled Text)"]
    powershell_text_encoding["PowerShell Default Text Encoding"]
    gitattributes[".gitattributes"]
    reencoding_data_loss["Data Loss from Re-Saving Misdecoded Text"]
    character_encoding -.->|"uses"| unicode
    character_encoding -->|"uses"| utf_8
    character_encoding -->|"uses"| utf_16le
    character_encoding -->|"uses"| cp932
    character_encoding -->|"uses"| active_code_page
    character_encoding -.->|"uses"| bom
    character_encoding -->|"may cause"| mojibake
    utf_8 -->|"implements"| unicode
    utf_16le -->|"implements"| unicode
    cp932 -.->|"incompatible with"| utf_8
    cp932 -.->|"may cause"| mojibake
    active_code_page -.->|"incompatible with"| utf_8
    utf_16le -.->|"incompatible with"| active_code_page
    bom -->|"uses"| unicode
    powershell_text_encoding -.->|"uses"| utf_8
    powershell_text_encoding -.->|"uses"| active_code_page
    powershell_text_encoding -.->|"requires"| bom
    line_ending -.->|"configured by"| gitattributes
    mojibake -.->|"may cause"| reencoding_data_loss

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (19 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

1. What to Grasp First

Putting the conclusions first, these seven points matter most.

• A text file is not the string itself but bytes + an encoding + a line-ending convention. Sometimes a BOM (Byte Order Mark) is attached too.
• Mojibake happens when the same bytes are decoded as a different encoding.
• Line-ending trouble happens when the encoding is correct but the assumption about line separators is misaligned.
• Unicode and UTF-8 do not mean the same thing. Unicode is about the character set; UTF-8 and UTF-16 are about the encodings that turn it into bytes.
• What Windows people call “Shift_JIS” is, in practice, better thought of as CP932 / the Windows family of Japanese code pages — conversations drift less that way.
• “We switched to UTF-8” is still an incomplete specification. It only becomes an operating rule once you also decide whether there is a BOM and the line-ending style.
• The source of the confusion is not Japanese itself, but multiple assumptions with different histories surviving on the same Windows machine.

When handling text on Windows, the starting point is to separate these four:

1. What are the bytes of this file
2. With which encoding was it written
3. With which encoding was it read
4. Are the line endings CRLF or LF

Just separating these makes you far less likely to get lost.

2. Breaking Down the Terminology

2.1 How Unicode / UTF-8 / UTF-16 / CP932 Differ

First, it pays to take the words apart once.

Even the single character あ has different bytes per encoding.

Character: あ

UTF-8    : E3 81 82
CP932    : 82 A0
UTF-16LE : 42 30

What matters is that characters and bytes are different things. An application appears to handle “characters” on screen, but for saving and transmission it ultimately exchanges bytes. Things break, almost always, at the boundary of that conversion.

2.2 How to Think About Shift_JIS vs. CP932

In the field, Japanese Windows text files often get lumped together as “Shift_JIS”. That works for conversation, but for practical work it is a bit sloppy.

If you want to be more precise about Japanese legacy text on Windows, it is safer to think CP932, or the Windows family of Japanese code pages.

Be sloppy here, and conversations like these drift:

• You said “save it as Shift_JIS”, but the other party assumed Windows-side CP932
• The Linux / macOS side treated it as shift_jis, but some Windows-origin files do not reproduce cleanly
• You were told to save as ANSI, but which code page that means is environment-dependent

So in specifications and investigation notes, it is safer to write it as concretely as possible:

• Not Shift_JIS but CP932
• Not ANSI but ACP (active code page) / normally CP932 in Japanese environments
• Not text but something concrete like UTF-8 no BOM, LF

2.3 The Traps in the Words ANSI, Unicode, and UTF-8N

Around Windows, the word labels themselves are also a source of confusion.

These three are especially misleading.

• ANSI Appears in the Windows UI and older documentation, but it is not ASCII. In most cases it refers to the machine’s active code page (ACP).
• Unicode In some editors and tools, the Unicode entry in a menu means UTF-16LE. “I saved it as Unicode” does not necessarily mean UTF-8.
• UTF-8N Seen in some Japanese-market editors; it is normally a UI label to distinguish UTF-8 without a BOM. It is not an official encoding name.

In short, on Windows, the same word means different things in different tools. This is the first big point of confusion.

2.4 What a BOM Is, and Why UTF-8 Gets One Too

BOM stands for Byte Order Mark, and it is the code point U+FEFF placed at the start of a file or stream. As the name says, its original job is to indicate the order of the bytes.

UTF-16 and UTF-32 represent a character as a 2-byte or 4-byte unit, so they have to tell the reader whether that unit is laid out low byte first (little endian) or high byte first (big endian). Putting a BOM at the start solves this: the byte order of the BOM itself declares the byte order used for everything that follows.

The obvious question is this: UTF-8 works one byte at a time, so it has no byte-order problem — why does it get a BOM at all?

The reason is that a UTF-8 BOM is not about byte order but is used as a signature saying “this file is UTF-8”. UTF-8 is ASCII-compatible, so a file whose contents are only alphanumerics is indistinguishable from a legacy code page. Put EF BB BF at the front and the reader can decide to read it as UTF-8 rather than as the active code page.

Turned around, that gives you the trade-off:

• The upside of a BOM: the reader is less likely to guess the encoding wrong. Windows PowerShell 5.1 in particular reads a script file without a BOM as the active code page, so a script containing non-ASCII characters breaks unless the BOM is there.
• The downside of a BOM: three extra bytes appear at the start. A runtime that does not know about BOMs treats them as an invisible character mixed into the beginning of line 1. This is where breakage like “only the first column of the CSV header fails to match” or “the #! line of the shell script is not recognized” comes from.

So a UTF-8 BOM is not a matter of “correct” or “incorrect”; it is a setting decided by who reads the file. That is exactly why, as in 7.1, you have to decide not only “make it UTF-8” but also “with BOM or no BOM”.

3. Why Mojibake Happens

3.1 What Mojibake Really Is

What mojibake actually is, is quite simple.

1. A string is turned into bytes with some encoding
2. Those bytes are turned back into a string with a different encoding
3. If the assumptions do not match, a different string comes out

For example, saving あ as UTF-8 produces these bytes:

E3 81 82

Read as UTF-8 this is あ, but read under CP932 assumptions it appears as a different string like 縺�. What is broken at that point is not “the Japanese” but the decoding assumption.

Mojibake in one line:

The same bytes were read as a different encoding.

flowchart TB
    A["Character あ"] -->|"Encode as UTF-8 and save"| B["Bytes E3 81 82<br/>this is all the file contains"]
    B -->|"Decode as UTF-8"| OK["Character あ ── the assumptions match"]
    B -->|"Decode as CP932"| NG["A different string such as 縺 ── the assumptions are misaligned"]

Figure 1: Down both branches the file contents (the bytes) are identical. The only difference is the assumption the reader brings

3.2 Garbled Display and Data Corruption Are Different

What matters here is separating the stage where things can still be recovered from the stage where they barely can.

For example, this flow can still be recovered from:

1. A UTF-8 file is opened as CP932
2. On screen it looks like 縺�
3. Nothing has been saved yet

At this stage the original bytes are still UTF-8. Reopen with the correct encoding and it may come back intact.

The dangerous flow is this one:

1. A UTF-8 file is misread as CP932
2. The visibly broken content is saved as is
3. The original UTF-8 bytes are lost

Once you get this far, it is no longer garbled display but data corruption.

flowchart TD
    F["File saved as UTF-8<br/>the bytes are E3 81 82"]
    F -->|"Open under CP932 assumptions"| V["The screen shows something like 縺<br/>= garbled display. The bytes are still UTF-8"]
    V -->|"Reopen the file"| OK["Specify the correct encoding and it comes back"]
    V -->|"Save it as is"| BAD["The misread string is written back as CP932<br/>= data corruption. The original bytes are gone"]
    BAD --> NG["Learning the correct encoding later does not bring it back"]

Figure 2: The fork is a single move - did you reopen it, or did you save it. Always settle this question first in an investigation

In practice, instead of summarizing everything as “it got garbled”, it is important to at least separate these two questions:

• Are the bytes themselves still correct
• Has the misread content already been re-saved

3.3 Converting Down to an Encoding That Cannot Represent the Characters Is Irreversible

The other danger is converting a Unicode string down to a narrow code page like CP932.

If the string contains characters that do not exist on the target side, one of the following happens:

• They become ?
• A replacement character is inserted
• The conversion errors out
• They are mapped to some similar nearby character

For example, some emoji and extended kanji cannot be dropped into CP932 as is. Judge this kind of breakage not by “is it readable” but by does a round-trip conversion return the original.

Information once lost does not come back, even if you learn the correct encoding afterwards.

4. What the Line-Ending Differences Are

4.1 CRLF / LF / CR

Line endings are bytes too.

• CR = carriage return = 0D
• LF = line feed = 0A
• Windows text files traditionally use CRLF (0D 0A)
• Linux / Unix systems generally use LF (0A)
• Bare CR can appear in older contexts such as classic Mac

In table form:

4.2 Line Endings Are a Separate Problem from Encodings

This is quite important.

The line-ending convention is a separate problem from the encoding.

The same UTF-8 file can have either CRLF or LF line endings. For content of A, a newline, then B, the bytes change like this:

UTF-8 + LF   : 41 0A 42
UTF-8 + CRLF : 41 0D 0A 42

Which means all of these are perfectly possible:

• UTF-8, but only the line endings differ
• CP932, but the line endings are LF
• UTF-16LE, but the line endings are CRLF

So when “we switched to UTF-8 and it’s still wrong”, the reality is sometimes that only the line endings are misaligned, not the encoding.

4.3 \n and the Newline Bytes in the File Are Not Necessarily the Same

From a programmer’s perspective, this is the quietly confusing part.

Writing \n in source code does not guarantee that only 0A lands in the file. In the text mode of a language, runtime, or I/O API, \n may be converted to CRLF on Windows.

Which means the following can all be out of sync:

• the newline notation in the source code
• the string at runtime
• the bytes saved in the file
• the line endings visible in the editor

This is how the “I’m sure I wrote LF, but the file is CRLF” problem happens.

Modern editors handle bare LF just fine, but surrounding tools, legacy applications, and business operations still carry CRLF assumptions. So line-ending problems are not “ancient history” - they routinely come up in practice today.

5. Why Windows in Particular Is So Confusing

5.1 Unicode and Legacy Code Pages Coexist

This is the biggest reason Windows is complicated.

Windows retains both:

• the Unicode path
• the code-page-based path

Newer applications, the web, and cross-platform assets lean toward UTF-8, while older CSVs, TXTs, logs, the Excel ecosystem, and business-system integrations still carry CP932. On top of that, UTF-16LE routinely appears in some outputs and around certain APIs.

In other words, multiple text cultures cohabit inside a single Windows machine.

5.2 The Labels Don’t Line Up

What multiplies the confusion is less the technology than the misaligned labels.

• It says Shift_JIS, but the reality is CP932
• It says ANSI, but the reality is the active code page
• It says Unicode, but the reality is UTF-16LE
• It says UTF-8, but BOM-or-not has never been pinned down
• Editor-specific labels like UTF-8N show up

Leave this vague, and the conversation appears to work while the underlying realities never match.

5.3 ASCII-Only Content Hides the Problem

This one is big as well.

Because UTF-8 is compatible with the ASCII range, a file containing only alphanumerics and symbols can “sort of read fine” even under the wrong assumption. On the CP932 side too, the ASCII-equivalent range rarely looks broken, so the problem never surfaces.

The result is a state like this:

• The English-only config file looks fine
• The moment one line of Japanese goes in, it breaks
• A problem that lurked all along finally goes off in production

This is why encoding breakage tends to look like “it worked until yesterday and suddenly broke today”. In reality, it is usually that the landmine was there all along, and it only became visible the moment non-ASCII characters arrived.

5.4 File Contents, File Names, Console, and Source Files Are Separate Layers

On Windows, lumping all of the following together as “the encoding” gets you lost.

• file names / paths
• file contents
• display in the console
• the encoding of the source code file itself
• the string representation at runtime
• display on the clipboard and in GUI widgets

For example, even if a Japanese file name displays fine, the file’s contents may be saved in CP932. Conversely, even if the file itself is UTF-8, the display alone breaks if the console code page does not match.

Operations like chcp 65001 fundamentally act on the console-side assumption - they do not change the bytes of existing files.

Furthermore, even if the source code file is UTF-8, the log file it writes at runtime is not necessarily UTF-8. You have to separate, every single time, which layer’s encoding you are talking about.

Incidentally, on Japanese Windows the \ character can display as a yen sign, which also tends to get mixed into encoding discussions. In most cases, however, this is a display font / glyph issue - the meaning as a path separator or escape character has not changed.

5.5 BOM and Line Endings Act on Yet Other Axes

Saying UTF-8 only settles half the question.

In practice, these matter too:

• Is there a BOM or not
• Are the line endings CRLF or LF

For example, with the very same UTF-8:

• some Windows tools read it only if the BOM is present
• some Unix-side processing ends up with junk characters in the first column when a BOM is present
• some legacy-side tools struggle with bare LF
• CRLF can make shell scripts and diffs messy

So even with the encoding matched, things can still break.

5.6 Tools Change Things on Their Own

What makes it even trickier is that your local tools change things implicitly.

• The editor auto-detects
• A BOM is added / removed at save time
• Git converts between CRLF and LF
• Shells and commands save with their default encoding
• A CSV export uses an unexpected code page
• Defaults differ between versions of PowerShell or other tools

In other words, even when the operator specified nothing, some layer adds its own assumption - that is everyday Windows practice.

This is the true identity of “it broke even though I changed nothing”. More often than not, it was not a person but a tool’s default that made the change.

6. Common Failure Patterns

The typical failures, in table form:

The most dangerous pattern of all is seeing the garbled display and then saving, locking the damage in.

7. Rules That Reduce Breakage in Practice

From here: what to decide as operating rules to reduce breakage.

7.1 Decide the Baseline for New Text

For new files, making UTF-8 the first candidate is reasonable. But that alone is not enough.

At minimum, it is safer to decide as far as:

• UTF-8 with BOM or UTF-8 no BOM
• Line endings CRLF or LF
• Who reads this file
• Is compatibility with legacy Windows tools required
• Will Linux / macOS / CI / containers read it too

For example, for cross-platform source code and config files, UTF-8 no BOM + LF is the likely first choice. On the other hand, if you must align with old Windows tools or existing operations, UTF-8 with BOM or even CP932 + CRLF may still be necessary.

What matters is deciding based on who you exchange data with, not on generic notions of “what is correct”.

7.2 Don’t Convert Existing Legacy Files on a Whim

If an existing file is CP932, it is safer not to UTF-8-ify it as a side effect of a routine small fix.

The safe-side operation is:

• Existing files keep their original encoding / BOM / line endings
• Encoding conversion is split out as a separate migration task
• Bulk-convert only after confirming the conversion targets and the downstream consumers

Mojibake tends to start from well-intentioned “modernizing while we’re at it”.

7.3 Treat Encoding and Line Endings as Part of the Interface

For CSVs, TXTs, logs, config files, and simple protocols, the text format itself is the interface, not just the content.

A spec should state at least:

• the encoding
• BOM or not
• the newline style
• header or not
• the quoting / delimiter rules
• which tool it was validated with

The three letters CSV are not enough. Only once you write UTF-8 with BOM, CRLF, comma delimiter, with header does the conversation stop drifting.

7.4 Be Explicit at Read/Write Boundaries

On the code side too, it is safer not to lean on implicit defaults.

• Specify the encoding explicitly on file read / write
• Stay conscious of encoding when passing text between processes
• Pin the line endings as part of the spec in export / import logic
• Don’t let a casual shell redirect become a production path

On Windows especially, “it saved” and “it saved with the correct bytes” are not the same thing.

7.5 Share the Git and Editor Rules Too

Git is not a tool that automatically fixes encodings. Line endings, on the other hand, can get converted.

So it is safer to decide, per repository:

• Is LF the baseline for source code
• Is CRLF tolerated for Windows-only text
• How is this pinned with .gitattributes
• How are editor settings shared

It is important to think about encoding and line endings separately. Even if Git normalizes the line endings for you, the encoding problems remain untouched.

7.6 Don’t Stop at “It Got Garbled” - Say What Is Misaligned

In the field, this rephrasing works wonders.

• Bad phrasing: “It got garbled”
• Good phrasing: “It looks like a UTF-8 no BOM file is being opened under CP932 assumptions”
• Bad phrasing: “The line endings are weird”
• Good phrasing: “An LF file is being converted to CRLF, inflating the diff”

Just being able to say what is misaligned changes the speed of the investigation considerably.

7.7 How to Check - What to Look at in Each Tool

The rules above only work once you pair them with a way to check. For the three tools you reach for most, here is where to see the current encoding and where to change it.

VS Code

The default encoding in VS Code is UTF-8 (no BOM). To change it for a single file, click the indicator in the status bar and choose whether to reopen the current file with a different encoding or to save it again with a different encoding. Reopen first if all you did was misread the file - that maps directly onto the judgment in 3.2.

To change the default, set files.encoding in the settings. Values include utf8 (no BOM), utf8bom (with BOM), utf16le, and windows1252. You can also split the setting per language.

{
  "files.encoding": "utf8",
  "[powershell]": {
    "files.encoding": "utf8bom"
  }
}

The 2.4 distinction of using a BOM only for scripts that run under Windows PowerShell 5.1 is exactly what this per-language setting expresses.

Notepad

Since Windows 10 Build 18963, Notepad has a status bar column that shows the document’s encoding, and the default for new files is UTF-8 (no BOM). To save, open File > Save As and pick from the Encoding field in the dialog. That field lets you choose BOM or no BOM, so this is where you apply the policy you settled on in 7.1.

When someone reports that “it came out garbled when I opened it in Notepad”, checking this field and the status bar indicator first is the quickest route.

PowerShell

PowerShell defaults differ enormously between versions. Not knowing this is a common reason people end up asking why a tool cannot read what PowerShell wrote.

• PowerShell 6 and later (the PowerShell 7 line): every text output defaults to utf8NoBOM, that is, UTF-8 without a BOM.
• Windows PowerShell 5.1: the defaults are not consistent across cmdlets.

The main 5.1 cases, in table form:

The reading side has defaults as well. When there is no BOM, Get-Content and the PowerShell engine reading a script file assume Default (the active code page), while Import-Csv and Select-String assume UTF-8. In other words, for the same file, the assumption changes depending on which cmdlet reads it.

To find out whether the file in front of you has a BOM, look at its leading bytes.

# PowerShell 7: show the first 3 bytes in hex
Get-Content -Path .\sample.txt -AsByteStream -TotalCount 3 | Format-Hex

# Windows PowerShell 5.1 has no -AsByteStream, so use -Encoding Byte
Get-Content -Path .\sample.txt -Encoding Byte -TotalCount 3 | Format-Hex

If it starts with EF BB BF it is UTF-8 with BOM, FF FE means UTF-16LE, and anything else means no BOM. This is the same reading as the table in 2.4.

To pin the defaults explicitly, use $PSDefaultParameterValues.

# Default every cmdlet that has an Encoding parameter to UTF-8
$PSDefaultParameterValues['*:Encoding'] = 'utf8'

There is a cross-version trap here. The very same string utf8 means UTF-8 with BOM in Windows PowerShell 5.1 and UTF-8 no BOM in PowerShell 6 and later. If you want to pin BOM-or-not as well, it is safer to spell out utf8BOM / utf8NoBOM, which are available in PowerShell 6 and later.

One more thing: $OutputEncoding is the encoding used when exchanging data with external programs; it has no effect on the encoding used when a redirect or a cmdlet saves to a file. This is an easy pair to conflate.

8. Drive Mojibake / Line-Ending Investigations with These 5 Questions

When an investigation stalls, returning to these five questions is the fastest route.

1. What are the bytes of this file right now
   • UTF-8?
   • UTF-8 with BOM?
   • CP932?
   • UTF-16LE?
2. Who wrote it first, under which assumption
   • an editor
   • a legacy app
   • an Excel export
   • a shell / script
   • a batch job / middleware
3. Who is reading it now, under which assumption
   • the editor’s auto-detect
   • the console code page
   • the library’s default encoding
   • the importer’s specification
4. What are the BOM and line endings
   • BOM present / absent
   • CRLF / LF
5. Has the misread content already been saved
   • still display-only?
   • already re-saved, with the original bytes lost?

Fill in these five, and the cause usually comes into view.

9. Summary

Windows text encodings and line endings look complicated not because Japanese itself is hard. It is because bytes, encoding, BOM, newline, and tool defaults exist independently - and on top of that, old and new text cultures cohabit on Windows.

The six points especially worth remembering:

• Mojibake is the result of reading the same bytes as a different encoding
• Line-ending problems are on a separate axis from encodings
• Don’t over-trust the words Shift_JIS, CP932, ANSI, and Unicode at face value
• “We switched to UTF-8” is insufficient - BOM and line endings are needed too
• Treat garbled display and already-re-saved data corruption as separate things
• In specs, write not text but something like UTF-8 no BOM, LF

In short, when handling text on Windows, the practical mindset is that it is not “a discussion about strings” but a discussion about how to align the contract over bytes.

10. Related Articles

• An Introduction to Windows Text Encodings - The Mojibake That Happens When Integrating with Linux
• Prompting Rules That Reduce Codex Mojibake on Windows

11. References

1. Microsoft Learn, Code Page Identifiers - Win32 apps https://learn.microsoft.com/en-us/windows/win32/intl/code-page-identifiers
2. Microsoft Learn, about_Character_Encoding - PowerShell https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_character_encoding?view=powershell-7.6
3. Microsoft Learn, Understanding file encoding in VS Code and PowerShell https://learn.microsoft.com/en-us/powershell/scripting/dev-cross-plat/vscode/understanding-file-encoding?view=powershell-7.6
4. W3C Internationalization, Character encodings: Essential concepts https://www.w3.org/International/articles/definitions-characters/
5. Git documentation, gitattributes https://git-scm.com/docs/gitattributes
6. Git documentation, git-config https://git-scm.com/docs/git-config
7. Microsoft Learn, The Unicode standard - Globalization (the BOM byte sequences, and the use of a UTF-8 BOM as a signature) https://learn.microsoft.com/en-us/globalization/encoding/unicode-standard
8. Microsoft Learn, What was new in 20H1 Windows 10 Insider Preview Builds (Notepad defaulting to UTF-8 in Build 18963, and the addition of the encoding indicator to the status bar) https://learn.microsoft.com/en-us/previous-versions/windows-insider/archive/new-in-20h1

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

Why does mojibake happen?

Because the same bytes were read with a different encoding than the one they were written with. Saving あ as UTF-8 produces the bytes E3 81 82, but reading those same bytes under CP932 assumptions shows a different string such as 縺. What is broken at that point is not the Japanese text but the decoding assumption. Note, however, that re-saving the misread content as is destroys the original bytes, turning a display problem into data corruption, so the important thing is to reopen the file with the correct encoding before you save.

What is the difference between CRLF and LF?

They are different bytes for separating lines. CRLF is the two bytes 0D 0A and is traditional in Windows text files; LF is the single byte 0A and is common on Linux and macOS and in most development tools. The key point is that line endings are a separate problem from encodings. The same UTF-8 file can end its lines with either CRLF or LF, so when someone says we switched to UTF-8 and it is still wrong, sometimes only the line endings are misaligned, not the encoding.

Are Shift_JIS and CP932 the same thing?

The two are interchangeable in casual conversation, but in practice it is safer to keep them apart. If you want to be precise about Japanese legacy text on Windows, think CP932, or the Windows family of Japanese code pages. In the same way, ANSI on Windows usually means the machine's active code page, and the Unicode entry in an editor menu can mean UTF-16LE. Writing something concrete such as UTF-8 no BOM, LF in specifications and investigation notes keeps conversations from drifting.

What should a text file specification pin down?

We switched to UTF-8 is not enough on its own; it only becomes an operating rule once you also decide whether there is a BOM and which line ending is used. For handoff files such as CSVs and logs, it is safer to write down the encoding, BOM or no BOM, the line ending, whether there is a header, and the delimiter rules. For cross-platform source code and configuration, UTF-8 no BOM plus LF tends to be the first candidate, while UTF-8 with BOM or CP932 plus CRLF may still be necessary when you have to match older Windows tools.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.