The Depths of Windows I/O (Part 5) — NTFS Internals: Understanding the File System Through the MFT

The invisible Zone.Identifier attached to a downloaded file. Why copying ten thousand small files is slow even when the total size is the same. How far the explanation “NTFS is journaling, so you are safe” actually reaches. This article works through all of them starting from the way data is laid out on disk.

At the center is the MFT (Master File Table), the ledger that tracks every file. We first take a file to be a collection of attributes, then work through data, names, links, journals, and disk space in that order.

Parts 1 through 3 of this series covered how an I/O request flows, and Part 4 covered what the cache does. This time we take up NTFS, the leading example of the file system where a request finally arrives. This is the part where the viewpoint shifts from the dynamic story of how a request flows to the static structure of how data is laid out.

This is Part 5 of the series “The Depths of Windows I/O”.

Start From the Problem You Have

If you want to learn the mechanism in order, start at Section 2; if you are in the middle of an investigation, use the guide below. The commands for checking things and how to read their output are collected in Section 8.

Prerequisite Terms Used in This Part

Prerequisite for this part: it helps to have a grip on the basics of IRPs and the device stack from Part 1. That said, so that this article causes no trouble read on its own, here are definitions of the terms from earlier parts that appear in the body below.

Table 1: Terms from earlier parts assumed in this part

One more thing: the two stages covered in Part 1, cleanup (when the last handle closes) and close (when every reference inside the kernel has gone), are also used in the explanation of file deletion in Section 4.

1. The Bottom Line First

What a File Really Is, and Where Its Data Lives

• NTFS is built around the MFT (Master File Table). Every file is tracked as a record inside the MFT, and every piece of information about a file lives either inside the MFT entry or in the area outside the MFT that the entry points to (Section 2).¹
• A file’s substance is a collection of attributes. A small file fits inside its MFT record, data and all (resident); a large file holds only a reference to a run of clusters (non-resident). The slowness of processing huge numbers of small files can be explained from here (Section 2).¹
• A file can hold more than one lot of data (multiple data streams). The data you work with day to day is the unnamed stream, and file.txt:name gives a file additional streams. This is what Zone.Identifier (Mark of the Web) really is (Section 3).²

Names, and What Happens When You Open a File

• Names are attributes too. Attaching several names to the same record is a hard link. The 8.3 short name lives alongside it as one more name (Section 4).³⁴
• Reparse points are the official mechanism for “open it and you end up somewhere else.” Symbolic links, junctions, and OneDrive Files On-Demand are all applications of this tagged data (Section 5).⁵⁶

Failure Recovery and Disk Space Allocation

• There are two journals. $LogFile is for recovering metadata consistency (a write-ahead log so nothing breaks), while the USN journal is for recording change history (a ledger of what changed). Their roles are completely different (Section 6).⁷⁸
• “Size” and “size on disk” are different things. Sparse files and compression are what drive them apart. The background to “a compressed file never becomes asynchronous,” which we saw in Part 2, is here too (Section 7).⁹¹⁰

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (35 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

2. Everything Is a Record in the MFT

2.1. The Volume’s Ledger

Formatting an NTFS volume creates the MFT (master file table) along with a set of metadata files whose names begin with $. The MFT holds at least one entry for every file on the volume, and that includes an entry for the MFT itself.¹

flowchart TB
    subgraph VOL["NTFS volume"]
        MFT["$MFT — the master file table<br/>The ledger of every file's record (itself included)"]
        LOG["$LogFile — the transaction log for<br/>metadata operations (Section 6)"]
        BITMAP["$Bitmap — cluster usage status"]
        OTH["$Boot / $Secure / $UpCase and<br/>other metadata files"]
        DATA["User data area<br/>(where non-resident data lives)"]
    end
    MFT -->|"records point to the location"| DATA

Figure 1: The structure of an NTFS volume. Keeping the file system’s own management information as files as well is part of the NTFS design

Information Lives Either Inside the Record or in the External Area the Record Points To

Size, timestamps, access permissions, and even the content of the data — every piece of information about a file is stored either inside the MFT entry or in an area outside the MFT whose location the entry describes.¹

A Record Freed by Deletion Is Reused, but the MFT Itself Never Shrinks

When you delete a file, its entry is marked as free and gets reused. However, the size of the MFT itself does not shrink.

So that the MFT can use as much contiguous space as possible when it grows, an MFT zone is reserved. The official documentation also describes how this changes in the course of operation: as the volume fills up, the MFT starts to fragment.¹

2.2. A File Is a Collection of Attributes, Resident and Non-Resident

The contents of a file record are a list of attributes. Standard information (timestamps and so on), the file name, security, and the data are all managed together in this one bundle.

Where the data goes splits into the following two cases.

Figure 2 shows the branch, and Figure 3 shows how the contents of the record differ.

flowchart TB
    subgraph REC["MFT file record (the ledger entry for one file)"]
        STD["$STANDARD_INFORMATION attribute<br/>timestamps and attribute flags"]
        FN["$FILE_NAME attribute<br/>(a file can hold several — Section 4)"]
        DATA["$DATA attribute"]
    end
    Q{"Is the data small"}
    RES["Resident<br/>the data itself fits inside the record<br/>a read completes with an MFT access alone"]
    NONRES["Non-resident<br/>the record holds only a reference to a run of clusters<br/>the real data lives in the user data area"]
    DATA --> Q
    Q -->|"up to a few hundred bytes"| RES
    Q -->|"larger than that"| NONRES

Figure 2: A file record is a collection of attributes. If the data is small it stays resident inside the record

Putting the same record side by side in its resident and non-resident forms makes the difference easier to see.

flowchart LR
    subgraph RES2["Resident — a small file"]
        RA["MFT file record (fixed length)<br/>$STANDARD_INFORMATION / $FILE_NAME / security<br/>─────────────<br/>$DATA attribute = the content itself<br/>'setting=1' goes directly in here"]
        RB["There is no separate storage location on disk<br/>a read completes with an MFT access alone"]
        RA --> RB
    end
    subgraph NON2["Non-resident — a large file"]
        NA["MFT file record (fixed length)<br/>$STANDARD_INFORMATION / $FILE_NAME / security<br/>─────────────<br/>$DATA attribute = a table of data runs<br/>a list of 'from where, how many clusters'"]
        NB["User data area<br/>run 1, contiguous clusters"]
        NC["User data area<br/>run 2, contiguous clusters elsewhere"]
        NA -->|"points to the location"| NB
        NA -->|"points to the location"| NC
    end

Figure 3: Resident against non-resident. In the non-resident case all the record holds is the table of where the real data is and how much of it there is, the data runs

The more data runs there are, the more scattered areas have to be walked across to read a single file. That is exactly what the fragmentation described next amounts to.

This structure explains a whole set of the phenomena you meet in the field.

Copying Small Files Piles Up Per-File Ledger Work

Every single file triggers metadata operations: creating an MFT record, registering the name, setting security. The ledger work comes to dominate over the data transfer itself (and every one of those operations is also something the filters covered in Part 6 get to inspect).

Fragmentation Is Data Runs Being Split Across Several Areas

Non-resident data is recorded as a list of contiguous cluster intervals, the runs. When contiguous space cannot be found, the number of runs grows and the seeks needed for a read grow with it — that is fragmentation. You can peek at the actual arrangement of the runs with fsutil file layout.

A Directory Is Also a File, One That Holds an Index for Looking Up Names

A directory is a file that holds an index from file names to MFT record numbers. On the ledger, everything rides on the same mechanism.

3. Data Is Only One of the Streams

3.1. One File, Several Byte Sequences

In NTFS, a single file can hold multiple data streams. What you normally read and write with ReadFile/WriteFile is the unnamed default stream, and the syntax filename:streamname lets you create an alternate data stream (ADS).²

flowchart LR
    subgraph F["A file named report.docx (one MFT record)"]
        D0["Default stream (unnamed)<br/>= the content you normally see"]
        D1[":Zone.Identifier<br/>origin information (Mark of the Web)"]
        D2[":any name you like<br/>extra information private to an application"]
    end

Figure 4: Multiple data streams. Only the default stream shows up in Explorer’s size display

Zone.Identifier Is the Stream That Carries Origin Information

A familiar example is Zone.Identifier. Where a file downloaded through a browser came from (that it originated on the internet, for instance) is recorded there, and it becomes the evidence behind SmartScreen’s “Windows protected your PC” message and Office’s Protected View.

How the warning works was covered in Why Windows Shows “Windows protected your PC”. The mechanism on the other side, the one that stores that origin information, is NTFS’s additional stream.

3.2. Pitfalls Developers Step On

An Ordinary Listing or Size Display Will Not Show Them

They appear neither in Explorer’s size column nor in a dir listing. You can check for them with dir /r or with Sysinternals’ streams.¹¹

They Are Lost Depending on the Destination or the Route

An ADS is an NTFS feature, so it tends to disappear on a copy to a FAT-formatted USB stick or through cloud storage. “The download warning vanished after I copied the file” is this.

You Can Use Them From an Application, but Do Not Store Real Business Data There

Simply including a colon in the path, as in CreateFile("data.txt:meta", ...), is enough to read and write one.² That is convenient, but you also take on the “cannot be transported” property from the previous point, so this is not the place to put the substance of your business data.

4. Names Are Attributes Too — Hard Links and 8.3 Names

4.1. Hard Links — Several Names for the Same Record

Figure 2 noted that a file can hold more than one file name attribute. Within a single volume, several paths reference a single file — that is a hard link (CreateHardLink / mklink /H).³

flowchart TB
    subgraph DIR1["The index of C:\app\"]
        E1["config.json → record #1234"]
    end
    subgraph DIR2["The index of C:\backup\"]
        E2["config-link.json → record #1234"]
    end
    REC["MFT record #1234<br/>the data itself (or a reference to its runs)<br/>link count 2"]
    E1 --> REC
    E2 --> REC

Figure 5: Hard links. The directory indexes simply point at the same MFT record, and both names are equally real

Every Name Points at the Same File

Change the file through any of its names and it is the same file, so the content matches immediately.³ Rather than treating one as the original and the other as a copy, take it as a single entity carrying several names of equal standing.

Separate Detaching a Name From the Entity Disappearing

When hard links exist, DeleteFile comes to mean “detach one name.” The entity disappears only when the last name has been detached, every open handle has been closed, and every reference inside the kernel, such as a memory-mapped section, has gone as well.

The two stages from Part 1, cleanup (when the last handle closes) and close (when the last reference disappears), also bear on this lifetime of a deletion.

Sharing the Content and Refreshing the Attribute Display Are Not the Same Thing

Change an attribute through one link and the attribute display seen through another link can stay stale. This is a display quirk noted in the official documentation as well.³

4.2. 8.3 Names — One More Hidden Name

The Short Name Is an Alias Kept for Compatibility

For historical compatibility, NTFS can automatically generate an 8.3-format short name such as REPORT~1.DOC for a long file name. This too is one more name living inside the same record.

In a folder holding large numbers of files, generating short names and avoiding collisions between them costs something as well. fsutil 8dot3name lets you disable generation or strip existing short names.⁴

That said, if an older application records a short name in a registry path, stripping breaks it. That is exactly why there is a feature for scanning the impact before you strip.⁴

Check the Generation Setting Before Relying on Short Names

Whether a short name exists at all depends on the environment. The registry value that decides the default behavior, NtfsDisable8dot3NameCreation, has the following four settings.⁴

With 2, the setting can be switched per volume. “On Windows there is always a short name such as PROGRA~1” does not necessarily hold.

Before writing code or a procedure that depends on short names, check the state with fsutil 8dot3name query C:. Omit the volume and you get the default setting shared by all volumes.

The pitfalls around paths and names (MAX_PATH, reserved names, trailing dots) are covered in detail in MAX_PATH and Windows Path/Filename Pitfalls. Put name resolution from Part 1 (the object manager) together with this section (names inside the file system) and you have the whole picture of names in Windows.

5. Reparse Points — The Mechanism for “Open It and You End Up Somewhere Else”

The Tag Decides What Happens on Open

A file or a directory can carry a reparse point. What it really is, is an attribute holding a tag and user-defined data.

When a file with a reparse point is opened, the processing switches according to the tag. In some cases a filter driver that understands the tag takes it over; in others a name-redirection tag makes resolution start over on the target path.⁵

sequenceDiagram
    participant App as Application
    participant IOM as I/O manager
    participant FS as NTFS

    App->>IOM: CreateFile("C:\data\link.txt")
    IOM->>FS: IRP_MJ_CREATE (the world of Part 1)
    Note over FS: Reparse point found on the target<br/>the tag and data are returned
    alt Symbolic link or junction (name redirection)
        FS-->>IOM: "the real location is over here"
        IOM->>FS: Resolution starts over on the target path
    else Filter-managed tag (cloud files and so on)
        Note over FS: A filter that understands the tag<br/>takes over the processing (Part 6)
    end

Figure 6: Reparse point resolution. It is the official hook that intercepts the “open” operation

Symbolic Links, Junctions, and Files On-Demand

Familiar features line up on top of this single mechanism.

• Symbolic links (mklink) — a signpost that holds the target path. They can point to a different volume or to a UNC path.⁶
• Junctions and mount points — the long-standing mechanism for connecting a directory to a location on a different local volume.³
• OneDrive Files On-Demand — a file whose real data is not present locally is represented as a reparse point, and the moment it is opened a filter downloads it and hands the content over. This is what “it is visible in Explorer, but opening it kicks off network traffic” really is (the filter mechanism itself comes in Part 6).

Code That Walks a Tree Checks for Reparse Points

In practice, what matters is that the end of a path is not necessarily the local location it appears to be. Code that does not take their existence into account runs into problems like these.

• A recursive walk loops on a junction.
• Size totals get counted twice.
• A backup triggers mass hydration of cloud files.

The entry point for dealing with this is to check FILE_ATTRIBUTE_REPARSE_POINT with the FindFirstFile family.⁵

6. The Two Journals — $LogFile and the USN Journal

“NTFS is a journaling file system” is often said, but NTFS has two journals with different roles. Confuse them and you misread what is guaranteed.

flowchart TB
    subgraph J1["$LogFile — the write-ahead log (so nothing breaks)"]
        A1["Metadata operations (record updates, renames and so on)<br/>are logged before they are carried out"]
        A2["At the next startup after a system failure<br/>the log is replayed to restore structural consistency"]
        A1 --> A2
    end
    subgraph J2["USN journal — the change history (so you can tell what changed)"]
        B1["Every change to a file or directory records<br/>the content of the change and the name"]
        B2["Backup, search index, and sync tools learn<br/>'what changed since last time' without a full scan"]
        B1 --> B2
    end

Figure 7: The two journals. $LogFile exists so nothing breaks, the USN journal so you can learn what changed

Put the differences in a table and they come out as follows.

Table 2: The two journals compared

$LogFile Restores Structural Consistency

$LogFile is a write-ahead log of metadata operations. After a system failure, NTFS uses the log and the checkpoint information at the next startup to restore the consistency of the file system automatically.⁷

What is protected here is the consistency of the structure, not the data content written into a file. As we saw in Part 4, dirty data sitting in the cache can be lost in a power failure. Keep “the structure can be recovered” and “the content of the last write survives” as separate ideas.

The USN Journal Is the Ledger for Finding Out What Changed Since Last Time

Every time a file or directory on the volume changes, the USN journal records the content of the change and the name of the target.⁸

It is the mechanism that lets backups and indexers pick up only what changed since last time without a full scan, and it is also used to avoid rebuilding an index after a failure.⁸

In an investigation, fsutil usn readjournal can serve as a cross-checking ledger that makes up for events FileSystemWatcher drops. For dropped events themselves, see A Practical Guide to FileSystemWatcher.

7. Sparse Files and Compression — When There Are Two “Sizes”

In NTFS, a file’s logical length and the space actually allocated to it are managed separately. These are “Size” and “Size on disk” in the properties dialog. Two things in particular drive them apart.

Sparseness Keeps Ranges of Zeros as “Holes”

A sparse file does not allocate real storage to ranges that run on as zeros and manages them as “holes” instead.⁹ A virtual disk file with a logical size of 42GB using only 500MB on disk is an ordinary occurrence. Read a hole and zeros come back; write to it and exactly that much gets allocated.

flowchart LR
    subgraph L["The logical file (size 1GB)"]
        R1["Data 10MB"]
        H1["Hole (zeros) 500MB"]
        R2["Data 5MB"]
        H2["Hole (zeros) the rest"]
    end
    subgraph P["Allocation on disk (15MB + management information)"]
        A1["Run, the substance of R1"]
        A2["Run, the substance of R2"]
    end
    R1 --> A1
    R2 --> A2

Figure 8: A sparse file. A “hole” has no allocation, so the logical size and the size on disk drift apart

Compression Affects Not Only Space Used but the Behavior of I/O

NTFS compression compresses and stores data one compression unit at a time.¹⁰ It is transparent and convenient, but the cost is not transparent: every read and write runs a decompression or a recompression, and fragmentation advances more easily. And as we saw in Section 5 of Part 2, access to a compressed file never becomes asynchronous (the file system converts it to synchronous). It is one of the places to suspect when “I switched to asynchronous I/O but some files did not get any faster.”

Four Factors to Check When the Sizes Do Not Match

The allocation-based real size can be obtained with GetCompressedFileSize. When “the total of the file sizes” and “the disk space used” do not match, check the following four in order.

Separating the logical length from the allocated space makes the differences in what is displayed much easier to follow.

8. See It for Yourself

This time too, all of it can be observed on your own Windows machine (some of it requires administrator privileges). So that you can judge for yourself whether the results are right, each command comes with a note on where to look and what it tells you.

8.1. Spot an ADS by the Line That Carries a Stream Name

:: Look at alternate data streams
dir /r C:\Users\%USERNAME%\Downloads

Where to look: below the normal file lines, indented lines of the form filename:Zone.Identifier:$DATA are listed with their lengths. If such a line is there, that file carries the Mark of the Web (Section 3). Files downloaded through a browser have it; files you created yourself do not. Running the command in both places and comparing makes the presence or absence of an ADS obvious.

8.2. Look at Resident and Non-Resident Storage, and at the Layout of the Data

:: Look at a file's layout on the MFT (its runs) and its attributes
fsutil file layout C:\path\to\file.dat

:: Look at the extents only (a subcommand described in the official documentation)
fsutil file queryextents C:\path\to\file.dat

Where to look: layout lists, per stream, the size and the allocated size, and for a non-resident stream a list of extents (triples of VCN, LCN, and cluster count). A very small file that shows no extent lines is resident (Section 2.2), and one split across several lines is fragmented. Running it against a text file of a few bytes and against a file of a few hundred megabytes and comparing the two is the quickest way to get a feel for resident against non-resident.

8.3. Compare the Short-Name Generation Setting With the Actual Names

:: The 8.3 short-name generation setting, and existing short names
fsutil 8dot3name query C:
dir /x

Where to look: query returns whether short-name generation is enabled or disabled on that volume (omit the volume and you get the default setting shared by all volumes).⁴ dir /x displays a column of short names next to the long names, so an empty column means no short name has been created. It lets you confirm “there is not necessarily a short name” from Section 4.2 in your own environment.

8.4. Look at the USN Journal’s State and How Recording Progresses

:: The state of the USN journal
fsutil usn queryjournal C:

Where to look: this displays the journal ID, the range of valid USNs (First USN / Next USN), the target maximum size (MaximumSize), and the allocation unit (AllocationDelta).¹² Create a file and run it again and Next USN should have advanced, which is the confirmation that changes are being recorded. MaximumSize is the rough guide to “how far back you can reach” touched on in Section 6. On a volume where the journal is disabled, this returns an error.

8.5. Look at a Reparse Point’s Attribute and Tag

:: Checking reparse points (the target and the tag)
dir /aL C:\Users\%USERNAME%
fsutil reparsepoint query "C:\Users\%USERNAME%\OneDrive"

Where to look: whatever dir /aL lists is a reparse point (something with FILE_ATTRIBUTE_REPARSE_POINT set). Symbolic links and junctions are shown with a type such as <SYMLINKD> or <JUNCTION>. fsutil reparsepoint query displays the value of the reparse tag and, for a name-redirection tag, the target path. Pointing it at something that is not a reparse point returns an error, so getting an error is itself confirmation that this is an ordinary folder.

8.6. Follow Real File Operations With Procmon

Trace file operations with Procmon and this article’s cast of characters flows past under their real names (writes to $LogFile, paths with a stream name attached, reparse processing). See A Practical Guide to Process Monitor (ProcMon) for how to use it.

9. Summary

• NTFS is built around the MFT. Every file is a record in the ledger, and its information sits either inside the record or in the external area the record points to. Small data is resident, large data is referenced by runs, and the slowness of processing huge numbers of small files, along with fragmentation, are consequences of this structure.¹
• A file can hold more than one data stream. Zone.Identifier (Mark of the Web) is nothing but an ADS: it is visible with dir /r, and it is not carried outside NTFS.²¹¹
• Names are attributes, and a file can hold several of them. A hard link is a name of equal standing pointing at the same record; the 8.3 name is one more name kept for compatibility. Deletion means detaching a name, and the entity disappears only once the last name, the last handle, and every reference inside the kernel (a mapped section and so on) are all gone.³⁴
• A reparse point is the official hook into “open”, and symbolic links, junctions, and Files On-Demand are all applications of it. Code that walks a tree needs to keep FILE_ATTRIBUTE_REPARSE_POINT in mind.⁵⁶
• There are two journals. $LogFile recovers structural consistency (so nothing breaks); the USN journal is the change history (what changed). “It is journaling, so the data is safe too” does not follow — data durability is built with the tools from Part 4.⁷⁸
• Logical size and allocation are different things. Sparseness, compression, ADS, and cluster rounding are the four big causes of “the sizes do not match.” Together with the fact that a compressed file never goes asynchronous, this is a drawer worth having in performance investigations.⁹¹⁰

The series concludes next time, in Part 6, Filter Drivers and Minifilters — Why Procmon and Antivirus Scanners Can Intercept I/O. How do the ones standing in between, who have made occasional appearances since Part 1 — antivirus, Procmon, OneDrive, encryption — actually intercept I/O? As the finale of the series, we reveal the true identity of the residents standing in the gaps of the device stack.

Related Articles

• The Depths of Windows I/O (Part 1) — Every Read and Write Becomes an IRP: The Big Picture of the I/O System
• The Depths of Windows I/O (Part 2) — Synchronous and Asynchronous I/O: What OVERLAPPED Really Means
• The Depths of Windows I/O (Part 4) — Cache Manager: When Does Your WriteFile Actually Reach the Disk?
• Why Windows Shows “Windows protected your PC”
• MAX_PATH and Windows Path/Filename Pitfalls — the 260-Character Limit, Reserved Names, Trailing Dots, and Case Sensitivity
• A Practical Guide to FileSystemWatcher - Handling Missed and Duplicate Events
• Pitfalls of Network Drives and UNC Paths — Working With File Servers (Shared Folders) From a Business Application
• A Practical Guide to Process Monitor (ProcMon) — Pinpointing “Settings Not Applied” and “ACCESS DENIED” in 10 Minutes

Related Consulting Areas

KomuraSoft LLC handles the design and investigation of Windows business applications rooted in how NTFS works, including puzzling behavior around file size and copy performance and bugs tangled up with links and streams.

• Windows Application Development
• Bug Investigation and Root Cause Analysis
• Legacy Asset Utilization and Migration Support
• Contact Us

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What is the MFT (Master File Table)?

It is the data structure at the heart of an NTFS volume: a ledger that holds at least one entry (a file record) for every file on the volume, including an entry for the MFT itself. Everything about a file, from its size, timestamps, and access permissions down to the content of the data itself, is stored either inside an MFT entry or in an area outside the MFT that the entry points to. A small file fits entirely inside its MFT entry, data and all (resident), while a large file has only a reference to where its data sits, the run of clusters, recorded in the entry (non-resident). Deleting a file marks its entry as free for reuse, but the size of the MFT itself never shrinks.

What is the invisible "Zone.Identifier" data attached to a file?

It is one of NTFS's multiple data streams (an alternate data stream). In NTFS a single file can hold several byte sequences (streams), and what you normally read and write is the unnamed default stream. Windows records where a file came from, for example that it was downloaded from the internet, in an additional stream that you specify with a colon, as in "file.txt:Zone.Identifier". This is the so-called Mark of the Web, and it is what SmartScreen's warning and Office's Protected View use as evidence for their decisions. Alternate streams do not appear in Explorer's size display; you can check for them with the dir /r command or Sysinternals' streams tool. Note as well that they are not preserved when a file is copied to a file system other than NTFS, such as FAT.

What is the difference between a hard link and a symbolic link?

A hard link means one more name of equal standing is added, pointing at the same file entity, that is, the same MFT record. It can be created only within the same volume, accessing the file through any of its names gives you the same file, and deleting one name does not remove the file as long as another name remains. A symbolic link is a signpost that directs you to a different path, and it is implemented as a reparse point. Since it only holds the target path as a string, it can point to a different volume or even to a remote location, but it becomes a dead end if the target disappears. In practice the basic rule is to use a hard link to share the underlying entity, which changes what deletion means, and a symbolic link to re-point a path, for relocation or redirection.

NTFS is a journaling file system, so does that mean data survives a power failure?

You need an exact understanding of what is protected. What NTFS's transaction log ($LogFile) protects is the consistency of the file system's structure, its metadata. Even if a system failure occurs, NTFS automatically restores consistency using the log at the next startup, preventing the situation where the volume is corrupted and unreadable. But that does not mean the actual content of a file that was mid-write is restored. As we saw in Part 4 of this series, dirty data sitting in the cache is lost in a power failure. So the correct understanding is that the volume will not break but the content of the last write can still vanish, and if you need durability for the data itself you have to build it in with FlushFileBuffers, WRITE_THROUGH, or an application-level write design such as writing to a temporary file and renaming it.

Why does a file's "size" differ from its "size on disk"?

Because NTFS manages a file's logical length and the disk space actually allocated to it separately. Even an ordinary file shows some difference because allocation is rounded up to whole clusters (4KB by default), but sparse files and compressed files are where the gap becomes large. A sparse file does not allocate real storage to ranges that run on as zeros and manages them as holes instead, so a file with a logical size of several gigabytes using only a few megabytes on disk is entirely possible. A compressed file has only its post-compression size allocated. Conversely, if the "size on disk" looks larger, cluster rounding or an alternate data stream is often the cause.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.