The Depths of Windows I/O (Part 4) — Cache Manager: When Does Your WriteFile Actually Reach the Disk?

WriteFile returned success. So where is the data right now?

In an ordinary cache-enabled write, the data is first handed to a cache in the OS’s memory. Success from WriteFile means the data was handed off to the OS, not that it was made durable on disk. Getting it onto the disk is something the OS does afterwards.¹

Once you know that difference, you can explain phenomena such as “I saved it, and a power cut made it disappear anyway” and “the measured file-copy rate is faster than the disk can go.” Databases write out explicitly with fsync and the like for the same reason: to separate accepting a write from making it durable.

This article is Part 4 of the series “The Depths of Windows I/O”. It takes up the Cache Manager that sits between applications and the disk, and works through how the cache is built, when data is written out, and how to save data you cannot afford to lose.

Along the way, it explains Part 2’s “why asynchronous I/O also completes synchronously when the data is in the cache” and Part 1’s “fast I/O, the shortcut that builds no IRP.”

1. The Bottom Line First

• In a default write, the copy into the cache and the write out to disk are two separate things. Windows uses write-behind caching, and the lazy writer writes the data out later. Data that reached the OS cache survives a crash of the application alone, but a power cut or an OS crash loses the dirty pages that have not been written out.¹
• For data you cannot afford to lose, pick the method that matches the granularity of your saves. If you commit at milestones, FlushFileBuffers (FileStream.Flush(true) in .NET) is the basic answer; if you want to remove the delay on every write, FILE_FLAG_WRITE_THROUGH is. FILE_FLAG_NO_BUFFERING is a different tool that bypasses the system cache, it carries alignment requirements, and you still have to think about the cache inside the device and about metadata. For frequent durability, the official documentation names the combination of NO_BUFFERING and WRITE_THROUGH.²³¹
• Read performance and I/O paths follow from how the cache works, too. The cache is really a file mapping, and read-ahead works on reads. Treat coherency with a mapped view and durability as separate questions, and treat fast I/O as a shortcut available when the IRP can be skipped.⁴⁵⁶

You can start from whichever section matches your purpose.

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (32 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

2. What the Cache Really Is — Files Are Mapped into Memory

2.1. 256KB Slots and Memory Copies

The cache is really a view that maps part of a file into memory. The Cache Manager maps a 256KB section of a file into a “slot” in the system address space. Cache-enabled reads and writes are executed as memory copies between that slot and the application’s buffer.¹

flowchart TB
    subgraph U["Application (user mode)"]
        BUF["Application buffer<br/>(the region passed to ReadFile/WriteFile)"]
    end
    subgraph S["System address space"]
        SLOT["System file cache<br/>A slot mapping a 256KB section of the file"]
    end
    DISK[("File on disk")]
    BUF <-->|"ReadFile/WriteFile =<br/>a memory copy to and from the slot"| SLOT
    SLOT <-->|"The first read and the<br/>later write-back go page by page"| DISK

Figure 1: What cache-enabled I/O really looks like. What the application sees as “reading and writing a file” is, in most cases, just a memory copy

256KB Is Not the Unit of Disk I/O

256KB is the granularity of the view (the mapping); it does not mean disk I/O is always done in 256KB units. Pages inside a slot are read in as needed, and the actual amount of I/O varies with the request size and the access pattern.

For a section being read for the first time, disk I/O is issued to fill the cache, and the IRP we saw in Part 1 travels down to the storage stack below. If it is already in the cache, the read completes with nothing but a memory copy.

Even an Asynchronous Request Can Be Handled on the Spot

“An asynchronous request completes synchronously on a cache hit,” from Section 5 of Part 2, is the behavior of completing a request on the spot when it can be answered immediately.

Conversely, even with caching enabled, an asynchronous read can be processed synchronously when the page is not in memory, because page-fault handling has no asynchronous mechanism. Keep immediate completion on a cache hit and synchronous processing caused by a missing page as two separate ideas.⁷

2.2. What “Free Memory Went Down” Really Means

Separate Per-Open Settings from Shared Data

Whether caching is used and the state of read-ahead are managed per open, that is, per file object.¹ The cached data itself, on the other hand, is shared per file (per stream). Opening the same file many times does not create separate caches. Every handle seeing the same cache contents is the foundation of the coherency discussed in Section 6.

The Cache Manager keeps managing the cache continuously for as long as Windows is running.¹ When you copy a large file or read and write a lot, free physical memory is put to use as cache. Much of it is standby memory that can be repurposed fairly promptly once an application asks for memory.

So free memory going down and memory running short are not the same thing. The practice of observing with this distinction in mind is also covered in “Telling a GC Wait from a Memory Leak in .NET.”

On Screen, Watch Standby and Free Change

In Task Manager, under Performance > Memory, the memory composition bar at the bottom shows In use / Modified / Standby / Free. Most of the file cache lands in Standby, and the list on the right totals it as “Cached.” The Memory tab of Resource Monitor shows the same breakdown with sizes.

Copy a single file of a few gigabytes and compare before and after. The behavior you see — Standby goes up and Free goes down while “In use” barely moves — confirms that memory that had been free was put to work as cache.

3. Read-Ahead — Speculating on Reads

The Cache Manager reads ahead the sections it expects to be read next, inferred from past access patterns. This is read-ahead.

When a file is read sequentially from the beginning, the read is faster by however much of the following data is already in the cache before the next request is issued. The amount read ahead is not fixed; it varies with the detected pattern and the request size.

flowchart LR
    A["History of the application's read requests<br/>Reading in order from the beginning"]
    D{"The Cache Manager<br/>detects the pattern"}
    R["Read-ahead — load the following section<br/>before it is requested<br/>(the amount varies with the pattern and request size)"]
    H1["Hint FILE_FLAG_SEQUENTIAL_SCAN<br/>= read ahead aggressively"]
    H2["Hint FILE_FLAG_RANDOM_ACCESS<br/>= read-ahead would be wasted, so hold back"]
    A --> D
    D --> R
    H1 -.-> D
    H2 -.-> D

Figure 2: Read-ahead. On top of detecting the access pattern, you can give hints through CreateFile flags

FileOptions.SequentialScan and RandomAccess, listed in the mapping table in Part 1, are hints to this read-ahead. They tell the OS about the access the application already knows it is going to perform.

What matters is picking the hint that matches what the code does; neither one pins the amount read ahead to a fixed value.

4. Lazy Writing — What “Success” from WriteFile Means

4.1. The Lazy Writer Comes Around Every Second

In a default write, WriteFile returns success as soon as it has copied the data into a cache slot, and writing it out to disk is deferred. This write-behind caching policy is called lazy writing.¹

The write-out is done by the lazy writer, which the Cache Manager starts once per second. It queues one-eighth of the pages that have not been flushed recently, and queues more if there is a lot of data to write.¹

“Once per second” here is the interval at which the work starts. It must not be read as a guarantee that any individual write becomes durable within one second.

The Temporary-File Attribute Is a Hint to Hold Back Write-Back

A temporary file carrying the FILE_ATTRIBUTE_TEMPORARY attribute is assumed to be deleted soon and is excluded from the lazy writer’s flushing.¹ The attribute is only a hint, though. It can still be written back if memory comes under pressure, and it does not apply just because the name looks like a temporary file.

sequenceDiagram
    participant App as Application
    participant C as System cache
    participant LW as lazy writer (starts every second)
    participant D as Disk

    App->>C: WriteFile(data)
    Note over C: Copy into the slot and mark<br/>the page dirty (not yet written)
    C-->>App: TRUE returns immediately
    Note over App,C: From here until the write-back is the danger window<br/>A power cut or OS crash destroys this data
    LW->>C: Select 1/8 of the dirty pages
    LW->>D: Write them back together
    Note over D: Only here does it become durable

Figure 3: Lazy writing. Success from WriteFile means the data was handed off to the OS, not that it was made durable

4.2. What Fails, and How Much Disappears

Between WriteFile succeeding and the write-back finishing, there is a window in which the data is still only in the cache. What happens to the data in that window depends on whether only the application stopped or the whole OS stopped.

flowchart TB
    W["Data right after WriteFile succeeded<br/>(a dirty page in the cache)"]
    Q{"What happened"}
    A1["The application process<br/>crashed or was killed"]
    A2["The whole OS stopped<br/>(power cut, blue screen)"]
    S["The data survives<br/>The cache belongs to the OS, so the<br/>lazy writer writes it back as planned"]
    L["The dirty pages are lost<br/>Only what reached the disk remains"]
    W --> Q
    Q --> A1
    Q --> A2
    A1 --> S
    A2 --> L

Figure 4: The kind of failure decides what survives. The cache belongs to the OS, not to the process

“The application died right after saving, yet the file was fine” happens because the OS holds the data once it has been copied into the cache. On a sudden power loss, by contrast, the official documentation states explicitly that unwritten cache data is lost. Flush frequency is tuned as a trade-off between performance and reliability.¹

What you decide first in design is whether this data may be lost at the instant of a power cut. The last few seconds of a log may be acceptable, while a committed order record is not. For writes you cannot afford to lose, use the methods in the next section.

5. The Toolbox for “Definitely Written”

This section distinguishes three methods: writing out the buffers you have now, removing the delay on writes, and bypassing the system cache. Section 5.4 at the end summarizes the order in which to choose, based on your reliability requirement and the cost you can pay.

5.1. FlushFileBuffers — Write It All Out Now

FlushFileBuffers writes the buffered data of the specified file out to the device. Because file-system metadata is always cached, getting metadata through as well requires a flush or WRITE_THROUGH.¹²

In .NET, Distinguish Flush() from Flush(true)

On Windows, the equivalent of FlushFileBuffers is FileStream.Flush(true). Be aware of how it differs from simply calling Flush().⁸

Consider the Cost of Flushing Every Time

The official documentation points out that calling FlushFileBuffers after each of many writes is inefficient. When frequent writes each need to be made durable, it names the combination of NO_BUFFERING and WRITE_THROUGH described below.²

5.2. FILE_FLAG_WRITE_THROUGH — Remove Only the Delay

When a file is opened with FILE_FLAG_WRITE_THROUGH, a write goes to the cache as well, but also to disk without waiting for the lazy writer.¹

Because it does not take the system cache itself out of the picture, reads can still use the cache. This is the method for “keep the read cache and remove only the write delay.”

5.3. FILE_FLAG_NO_BUFFERING — Not Going Through the Cache

FILE_FLAG_NO_BUFFERING takes the Windows system cache out of reads and writes. Both reads and writes become I/O to the disk device without going through the cache.¹

That said, the write cache inside the device is a separate stage. NO_BUFFERING does not bypass that too, so if you need to survive a power cut, keep considering WRITE_THROUGH alongside it or FlushFileBuffers.

It suits bulk transfers of large amounts of data and database engines that manage their own buffering, but the application has to meet the following alignment requirements.³

• The size and file offset of a read or write must be an exact multiple of the volume’s sector size (512, 1024, 1536 and so on for a 512-byte sector).
• The buffer address must also be aligned to the physical sector size (which also means allowing for “Advanced Format” disks with 4096-byte physical sectors).
• Even then, metadata continues to be cached, so complete durability needs WRITE_THROUGH alongside it or FlushFileBuffers.¹²

Align All Three — Size, Offset, and Address

Just adding the flag does not switch you over to NO_BUFFERING. A read or write that violates the alignment requirements fails with ERROR_INVALID_PARAMETER (87). There are three things to check.³

Checking Buffer Alignment with a Minimal C++ Example

The third one, the buffer address, is especially easy to overlook. Addresses returned by malloc, new, or a C# array carry no guarantee of alignment to a sector boundary.

The following example allocates a region aligned to a page boundary with VirtualAlloc. An ordinary 4096-byte page boundary also satisfies the requirement of an Advanced Format disk with 4096-byte physical sectors. Size and offset advance in multiples of the sector size that was retrieved.

// C++ / Win32. Error handling is kept to a minimum
DWORD sectorsPerCluster = 0, bytesPerSector = 0, freeClusters = 0, totalClusters = 0;
if (!GetDiskFreeSpaceW(L"C:\\", &sectorsPerCluster, &bytesPerSector,
                       &freeClusters, &totalClusters))
{
    return GetLastError();
}

// Make the read/write unit an exact multiple of the sector size (1 MiB or so here)
const DWORD chunk = (1024 * 1024 / bytesPerSector) * bytesPerSector;

// Take a region aligned to a page boundary for the buffer (malloc/new give no guarantee)
BYTE* buffer = static_cast<BYTE*>(
    VirtualAlloc(nullptr, chunk, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE));
if (buffer == nullptr) { return GetLastError(); }

HANDLE h = CreateFileW(L"C:\\temp\\big.bin", GENERIC_READ, FILE_SHARE_READ, nullptr,
                       OPEN_EXISTING, FILE_FLAG_NO_BUFFERING, nullptr);
if (h == INVALID_HANDLE_VALUE)
{
    // GetLastError returns the result of the most recent Win32 call. Calling VirtualFree
    // first would overwrite the real reason CreateFileW failed (access denied, path not
    // found, and so on) with the result of the cleanup, leaving a code that explains nothing
    const DWORD err = GetLastError();
    VirtualFree(buffer, 0, MEM_RELEASE);
    return err;
}

// Advancing chunk bytes at a time keeps both size and offset aligned
DWORD read = 0;
while (ReadFile(h, buffer, chunk, &read, nullptr) && read > 0)
{
    // Process the first read bytes of buffer
    // (at the end of the file read < chunk. That is normal)
}

CloseHandle(h);
VirtualFree(buffer, 0, MEM_RELEASE);

In .NET Too, You Cannot Skip Managing Alignment

.NET’s FileOptions has no value corresponding to FILE_FLAG_NO_BUFFERING. If you need it, call CreateFile directly, and even then you have to meet the alignment requirements yourself.

The order of consideration is not “NO_BUFFERING because I want it fast” but “NO_BUFFERING because I manage my own buffers.”

5.4. Choosing Between Them

flowchart TB
    A["Application buffer"]
    B["System file cache<br/>(dirty pages)"]
    C["Cache inside the disk device"]
    D[("Non-volatile storage medium")]
    A -->|"Default WriteFile — success returns at this point"| B
    B -->|"lazy writer (every second) / WRITE_THROUGH (immediate)"| C
    C -->|"The device's own timing /<br/>FlushFileBuffers demands a complete write"| D
    A -.->|"NO_BUFFERING skips the cache and goes straight here"| C

Figure 5: The layers the data passes through and how far each tool pushes it. Watch out for that last stage, the cache inside the disk device

Decide Reliability First, Then Check the Cost

Start by deciding how many records you can afford to lose in a power cut. Then separate whether what you cannot lose is a “milestone,” such as committing a transaction, or “every single write.”

Next, check how much slower you can afford to be, and whether you can handle your own buffer management and alignment. Do not choose by the name of the method; follow the branches below.

flowchart TB
    S["About to write this data"]
    Q1{"May it be lost at the instant of<br/>a power cut or a blue screen"}
    A0["Leave the default (caching enabled)<br/>The fastest. Most I/O belongs here"]
    Q2{"Is what you cannot lose a<br/>milestone or every single write"}
    A1["FlushFileBuffers at milestones<br/>Flush(true) in .NET<br/>Cost — only the wait at each milestone"]
    Q3{"Can you manage your own buffers and<br/>meet the alignment requirements of 5.3"}
    A2["FILE_FLAG_WRITE_THROUGH<br/>Straight to disk on every write<br/>Reads stay fast on the cache"]
    A3["FILE_FLAG_NO_BUFFERING<br/>+ FILE_FLAG_WRITE_THROUGH<br/>The official shape of frequent durability"]
    S --> Q1
    Q1 -->|"It may<br/>(the last few seconds of a log, say)"| A0
    Q1 -->|"It may not"| Q2
    Q2 -->|"Milestone<br/>(committing a transaction, say)"| A1
    Q2 -->|"Every single write"| Q3
    Q3 -->|"No (an ordinary application)"| A2
    Q3 -->|"Yes (a database engine or similar)"| A3

Figure 6: How to choose a tool. The first branch is the reliability requirement, the second is the cost you can pay. There is no path for “FlushFileBuffers on every single write” because, as Section 5.1 showed, the official documentation calls that inefficient

Turning This into Save Design and Performance Measurement

In practice it is easier to choose if you tie the decision to the following three patterns.

1. “Write to a temporary file, flush, rename” is the standard way to avoid leaving half-broken files behind. Write the contents out completely and then commit by name — this atomic hand-off was covered in detail in “The Basics of Exclusive Control in File-Based Integration.”
2. Leaving it to a database is a perfectly good design. For how SQLite builds durability out of WAL and flushes, see “Using SQLite in Business Applications with C#.” The option of “not writing your own flush strategy” is always there.
3. Suspect the cache in benchmarks. A measurement where “reads are too fast” is usually measuring a cache hit on the second run and after. The discipline of measurement is collected in “How to Correctly Compare the Speed of Program Versions on Windows.”

Think All the Way Down to the Cache Inside the Device

At the end of Figure 5, the cache inside the disk device remains. FlushFileBuffers demands a complete write that includes it.

With USB flash drives and external disks, the device-side disk write caching policy — “Quick removal” and “Better performance” — is involved as well. For handling removable devices, see also “Working with USB Devices from a Windows Application.”

6. Coherency with Memory-Mapped Files

6.1. A Mapped View and the Cache See the Same Data

If the cache is really a file mapping, what is the relationship between a view you created yourself with MapViewOfFile and the cache contents that ReadFile and WriteFile use?

Ordinary cache-enabled I/O and a mapped view share data backed by the same file. When pages of a file mapping object are evicted, the changes are written back to the file. Data is likewise coherent when multiple processes create views of the same local file from the same file mapping object.⁴

flowchart TB
    subgraph P1["Address space of process A"]
        V1["The view from MapViewOfFile"]
    end
    subgraph SYS["System address space"]
        SC["The Cache Manager's view<br/>(the slot ReadFile/WriteFile use)"]
    end
    PAGES["The same set of physical pages<br/>(file-backed memory)"]
    DISK[("File on disk")]
    V1 --> PAGES
    SC --> PAGES
    PAGES --> DISK
    NB["I/O with FILE_FLAG_NO_BUFFERING falls<br/>outside this sharing (straight to disk)"]
    NB -.-> DISK

Figure 7: Both the mapped view and the cache look at the same file-backed pages. The only one outside the frame is NO_BUFFERING

6.2. Check Coherency and Durability Separately

NO_BUFFERING I/O falls outside this coherency. Reads and writes that do not go through the cache are not reconciled with the contents of a mapped view or of cached I/O. If you mix them, the application has to keep them consistent itself.

Also, a change being visible in a mapped view is not the same as that change having been made durable on disk. Persisting a mapped view takes the following order.⁵

The practice of using file mappings as shared memory (named sharing, synchronization, failure patterns) is covered in “Shared Memory Pitfalls and Practical Best Practices.”

7. Fast I/O — Collecting on Part 1’s Homework

Fast I/O is a shortcut that handles synchronous reads and writes to cached files without building an IRP. An IRP is an “I/O request packet,” the structure that holds a request the kernel passes to a driver.⁶

7.1. When the IRP Can Be Skipped, and When It Falls Back to the Normal Path

The statement in Section 5.2 of Part 1 that “not all I/O becomes an IRP” refers to this path.

If a read or write can be handled with nothing but a memory copy to and from the cache, there is no need to build an IRP and send it down the device stack. In fast I/O, the file system’s entry points are called directly and data is exchanged with the Cache Manager.⁶

When fast I/O cannot be used, however, because the data is not in the cache, because locks are involved, because a filter intervenes, and so on, it falls back to the normal IRP path.

Note also that “a cache hit always means fast I/O” is not true. Operations on an asynchronous (FILE_FLAG_OVERLAPPED) handle can be processed on the IRP path even when they complete from the cache on the spot. “Does it complete on the spot,” from Section 5 of Part 2, and “is the IRP skipped,” from this section, are two different questions.

flowchart TB
    REQ["A synchronous read or write on a cache-enabled handle"]
    Q{"Can fast I/O handle it<br/>(is it in the cache, and so on)"}
    FAST["Fast I/O<br/>Copies directly with the cache, building no IRP<br/>Shown as FASTIO_ in Procmon"]
    IRP["Normal path<br/>Build an IRP and send it down the device stack<br/>(the world of Figure 6 in Part 1)"]
    REQ --> Q
    Q -->|yes| FAST
    Q -->|no| IRP

Figure 8: The fast-I/O branch. This is why FASTIO_READ and IRP_MJ_READ appear mixed together in Procmon

7.2. When Observing, Distinguish the Paths a Single Read Can Take

The reason FASTIO_ lines were mixed in during the Procmon observation in Section 7 of Part 1 is that the same read can travel different paths. Read FASTIO_READ and IRP_MJ_READ as the difference between fast I/O and the normal IRP path.

This shortcut also matters for the filter drivers covered in Part 6. Minifilters are designed so that they can intervene in fast I/O as well as in the normal IRP path.

8. Summary

Let us look back over the whole article in the order of mechanism, failure, design decision.

With write-behind, read-ahead, and the lazy writer in mind, first decide whether this data may be lost in a power cut. Then choose how to save it, taking into account the cost of flushing, the metadata that is always cached, and the cache inside the device.¹²³

Mapped-view coherency and completion of the write-out, and a cache hit versus skipping the IRP, are separate judgments too. That distinction is the lead you follow when investigating a save failure or an unnaturally fast benchmark.⁴⁵⁶

Next is Part 5, “NTFS Internals — Understanding the File System Through the MFT.” Up to now we have treated a file as “an offset and a byte sequence,” but behind that, how does NTFS lay the data out? The MFT, multiple data streams, the journal, hard links — we descend into the static structures on the disk.

Related Articles

• The Depths of Windows I/O (Part 1) — Every Read and Write Becomes an IRP: The Big Picture of the I/O System
• The Depths of Windows I/O (Part 2) — Synchronous and Asynchronous I/O: What OVERLAPPED Really Means
• The Depths of Windows I/O (Part 3) — I/O Completion Ports (IOCP) and the .NET Thread Pool: The Basement Under async/await
• The Basics of Exclusive Control in File-Based Integration - Best Practices for File Locks and Atomic Claims
• Shared Memory Pitfalls and Practical Best Practices
• Using SQLite in Business Applications with C# — WAL Mode, Exclusive Control, Corruption Countermeasures, and Choosing Between It and EF Core
• How to Correctly Compare the Speed of Program Versions on Windows
• Working with USB Devices from a Windows Application — Choosing Between Virtual COM, HID, WinUSB, and Vendor SDKs

Related Consulting Areas

KomuraSoft LLC handles the design and failure investigation of file I/O in Windows business applications, for problems such as “data I thought I had saved is gone” and “file writes are slow, or suspiciously fast.”

• Windows Application Development
• Bug Investigation & Root-Cause Analysis
• Legacy Asset Reuse & Migration Support
• Contact

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

Once WriteFile returns success, has the data been written to disk?

By default, no. The Windows file cache is a write-back cache, and WriteFile returns success as soon as it has copied the data into the system file cache. The write to disk is performed afterwards by the lazy writer that the Cache Manager starts once per second. What matters is the difference between kinds of failure. If the application's process crashes, data that reached the cache is not lost, because the OS writes it out later as long as the OS itself is alive. If the whole OS goes down instead, through a power cut or a blue screen, dirty cache pages that have not been written yet are lost. The accurate reading is not that a successful WriteFile means the data has been made durable, but that it has been handed off to the OS.

How do I make sure data actually reaches the disk?

There are three tools. First, FlushFileBuffers, which writes the buffered data and metadata for that file through to the device (in .NET, FileStream.Flush(true) is the equivalent). Second, FILE_FLAG_WRITE_THROUGH, which writes to the cache on every write while also writing to disk immediately. Third, FILE_FLAG_NO_BUFFERING, which does not go through the cache at all. Microsoft's documentation states that calling FlushFileBuffers on every write is inefficient, and that applications needing reliable durability with frequent writes should combine FILE_FLAG_NO_BUFFERING with FILE_FLAG_WRITE_THROUGH. All of them are slower because they give up the benefit of the cache, so the practical instinct is not to apply them everywhere but to reserve them for writes you cannot afford to lose.

What is the difference between FILE_FLAG_WRITE_THROUGH and FILE_FLAG_NO_BUFFERING?

WRITE_THROUGH means the write still goes to the cache, but it also goes to disk before completion. Reads keep benefiting from the cache, and only the lazy-writer delay is removed. NO_BUFFERING means reads and writes do not go through the system cache, so both become device I/O on every call (what it bypasses, though, is only the Windows cache, not the write cache inside the device). In exchange it comes with strict constraints. The size and file offset of every read and write must be an exact multiple of the volume's sector size, and the buffer address has to be aligned to a physical sector boundary. Also, file-system metadata continues to be cached even under NO_BUFFERING, so making metadata durable still requires FlushFileBuffers or WRITE_THROUGH on top. It is typically used by software that manages its own buffering, such as database engines; ordinary applications should normally consider WRITE_THROUGH or FlushFileBuffers first.

Task Manager shows little free memory. Is the file cache to blame?

In most cases yes, and it is normal behavior. Windows actively uses free physical memory as file cache, so a large file copy or a lot of reading and writing swells the cache and makes memory usage look higher. That said, most of the pages the cache is using are of a kind that can be repurposed fairly promptly once an application asks for memory, which has to be distinguished from a state where memory is genuinely exhausted. When you suspect a memory shortage, the practical approach is to look at indicators such as committed memory and hard-fault frequency rather than the apparent amount of free memory alone.

If I touch the same file through a memory-mapped file and through ReadFile/WriteFile, can the contents drift apart?

Not against ordinary cache-enabled I/O. The Windows cache is itself implemented as a file mapping, and a mapped view and the cache for the same local file share the same data, so a change made through one is visible through the other. Data is also coherent when multiple processes create views from the same file mapping object. However, reads and writes on a handle opened with FILE_FLAG_NO_BUFFERING do not go through the cache, so they fall outside this coherency. And to reliably write changes in a mapped view to disk, FlushViewOfFile alone is not enough, because it does not write metadata and does not wait for the hardware cache, so you need to call FlushFileBuffers after FlushViewOfFile.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.