The Depths of Windows I/O (Part 2) — Synchronous and Asynchronous I/O: What OVERLAPPED Really Means

Last time (Part 1) we saw that a Windows I/O request becomes a packet called an IRP that flows through the device stack, and that issuing and completing are separated from the very bottom of the kernel.

This time we dig into the mechanism applications use to work with that separation — asynchronous I/O (overlapped I/O). You add FILE_FLAG_OVERLAPPED and it still comes back synchronously. You reuse an OVERLAPPED and your data gets corrupted. You call CancelIoEx and it doesn’t stop. You cancel and the app crashes with an access violation — all of these “scary stories about asynchronous I/O” come from not holding the mechanism in your head as a single picture. This article builds that picture.

This is Part 2 of the “The Depths of Windows I/O” series. The overall structure is laid out at the top of Part 1.

1. The Bottom Line First

• The line between synchronous and asynchronous I/O isn’t in the kernel — it’s “whether you wait.” Synchronous I/O’s guarantee is that the call does not return before it completes. Only when a request becomes pending does the I/O manager wait for completion, and the thread sleeps in a wait state that consumes no CPU (Section 2).¹
• FILE_FLAG_OVERLAPPED is a mode of the handle (the file object). It’s decided at the moment of CreateFile and cannot be switched per call. Because an asynchronous handle has no system-maintained file pointer, for a file on disk you specify the position every time via the OVERLAPPED structure’s Offset (Section 3).²¹
• The OVERLAPPED structure is “a slip for one operation.” You need one for every operation in flight, and the official documentation explicitly states that sharing or reusing one leads to data corruption. Neither the structure nor the buffer may be touched until completion (Section 3).³⁴
• There are really four ways to receive completion: the handle being signalled (deprecated), an OVERLAPPED event plus GetOverlappedResult, an APC (alertable wait), and I/O completion ports (next time) (Section 4).¹⁵⁶
• Asynchronously issued I/O can still complete synchronously. Being cached, NTFS compression/encryption, and length-extending writes are the classic cases. “Asynchronous” does not mean “will never block” (Section 5).³
• Cancellation is a “request.” Even after calling CancelIoEx, the operation comes back as a completion with ERROR_OPERATION_ABORTED. You must not clean up until you’ve seen that completion (Section 6).⁷⁸
• .NET’s FileOptions.Asynchronous is the direct switch for this mode. A mismatch between the handle’s mode and the API you call produces “fake asynchrony,” where the thread pool stands in, or wasted synchronisation overhead (Section 7).⁹¹⁰

2. Synchronous I/O — Where Does the Thread Sleep?

Let’s start with the default shape. A handle opened without FILE_FLAG_OVERLAPPED is in synchronous mode. Call ReadFile and the function does not return until the I/O completes.¹

As we saw in Part 1, the driver puts the request into a pending state and waits for the hardware to respond. So for synchronous I/O, who is doing the waiting? The I/O manager waits for completion before returning control to the app.

sequenceDiagram
    participant App as Application thread
    participant IOM as I/O manager
    participant DRV as Driver -- stack

    App->>IOM: ReadFile - synchronous handle
    IOM->>DRV: Issue an IRP
    DRV-->>IOM: STATUS_PENDING - waiting on a response
    Note over App,IOM: The thread enters a wait state in the kernel<br/>and sleeps without consuming CPU
    DRV->>IOM: Completion - IoCompleteRequest
    IOM-->>App: Result returned, thread wakes<br/>ReadFile returns TRUE/FALSE

Figure 1: Synchronous I/O (when the request becomes pending). The I/O manager waits for completion before returning to the app

Note that this diagram describes the case where the request becomes pending. If the driver can complete the request on the spot (a cache hit, for example — the “immediate completion” path from Figure 5 in Part 1), no waiting occurs at all, and the call returns with the result straight away. The guarantee of synchronous I/O is “the call does not return before completion,” not “the thread will always sleep.”

Two points are worth keeping in mind.

• Waiting costs no CPU. A thread in a wait state is removed from the scheduler’s set of runnable threads. The case for letting a thread wait rather than strangling itself with polling is made in “Why You Should Prefer Event Waits over Sleep(1) on Windows.”
• On a synchronous-mode handle, the kernel maintains the file pointer (current position). That’s why successive ReadFile calls can read “from where you left off.” This state belongs not to the handle but to the file object, so handles duplicated with DuplicateHandle share that position (Section 3.3 of Part 1).

The weakness of synchronous I/O comes down to one thing: while waiting, that thread can’t do anything else. Do synchronous I/O on a UI thread and the screen freezes; spin up one thread per connection on a server and you end up drowning in threads across a few hundred connections. There’s also an API called CancelSynchronousIo for rescuing, from the outside, a different thread that is stuck in synchronous I/O (Section 6).¹¹

3. Asynchronous I/O — The Handle’s Mode and the Operation’s Slip

3.1. Mode Is Decided per Handle

Pass FILE_FLAG_OVERLAPPED to CreateFile and the file object behind that handle is opened in asynchronous mode.¹ What matters here is that this is a per-handle property. You cannot say “just this one call, asynchronously.” You can open the same file with two separate handles, one for synchronous use and one for asynchronous use (you simply end up with two file objects).

An asynchronous-mode handle has one more crucial difference: the system does not maintain a file pointer.² With multiple operations potentially in flight at once, “the current position” has no meaning. For devices that have a position, such as a file on disk, you must explicitly specify the read/write position every time via the OVERLAPPED structure’s Offset/OffsetHigh. For devices with no notion of a seek position, such as serial ports or named pipes, Offset is not used to specify a position (leave it at zero). Even then, as we’ll see in the next section, an OVERLAPPED structure is still required for every single operation.

3.2. OVERLAPPED Is “a Slip for One Operation”

The role of the OVERLAPPED structure is to identify one operation in flight and carry its state.⁴

flowchart LR
    subgraph H["Handle - file object = mode"]
        M1["Synchronous mode<br/>Kernel manages the current position<br/>ReadFile does not return until completion"]
        M2["Asynchronous mode - FILE_FLAG_OVERLAPPED<br/>Current position not maintained<br/>Issuing and completion are separated"]
    end
    subgraph O["OVERLAPPED structure = a slip for one operation"]
        F1["Offset - where to read"]
        F2["hEvent - how to learn of completion"]
        F3["Internal/InternalHigh -<br/>status and result, written by the system"]
    end
    C["Decided once, at CreateFile"] --> H
    R["One prepared for every ReadFile/WriteFile issued"] --> O

Figure 2: Mode lives on the handle; state lives on the operation (the slip). Confusing this division causes accidents

From here, two prohibitions the official documentation states explicitly follow naturally.³

1. You need one OVERLAPPED for every operation in flight. Issue three, you need three. Reusing one leads to “unpredictable results or data corruption.”
2. Until completion, keep both the OVERLAPPED and the data buffer alive, and don’t touch them. The kernel is going to write into that memory. Issuing with a local-variable OVERLAPPED and then returning from the function is a classic way to have the kernel trample your stack.

3.3. There Are Three Ways an Issue Can Return

A ReadFile on an asynchronous handle can return in three different ways.²

flowchart TB
    A["ReadFile - asynchronous handle, with OVERLAPPED"]
    Q{"What's the return value?"}
    T["TRUE<br/>Completed on the spot - synchronous completion<br/>By default a completion notification still arrives separately"]
    P["FALSE + ERROR_IO_PENDING<br/>Accepted. Completion will be notified later"]
    E["FALSE + some other error<br/>The issue itself failed"]
    W["Wait for the completion notification<br/>- the four methods in Section 4"]
    A --> Q
    Q --> T
    Q --> P
    Q --> E
    P --> W

Figure 3: The three-way branch of an asynchronous issue. Asynchronous I/O only works once you correctly handle both TRUE (immediate completion) and ERROR_IO_PENDING

The check you translate into code is the combination of the return value and GetLastError. This table maps directly onto the branch.

// C++ / Win32
// hFile : a handle opened with FILE_FLAG_OVERLAPPED
// ov    : an OVERLAPPED allocated specifically for this operation (Offset and hEvent already set)
// buf/len: a buffer dedicated to this operation. Do not free it until you receive the completion notification
DWORD IssueRead(HANDLE hFile, OVERLAPPED* ov, BYTE* buf, DWORD len)
{
    // For an asynchronous issue, pass NULL for lpNumberOfBytesRead;
    // the transferred byte count is retrieved later via GetOverlappedResult on completion
    if (ReadFile(hFile, buf, len, nullptr, ov))
    {
        // (1) Synchronous completion. A completion notification will still arrive by default,
        // so don't process the result here
        return ERROR_SUCCESS;
    }

    DWORD err = GetLastError();
    if (err == ERROR_IO_PENDING)
    {
        // (2) Accepted. Wait for the completion notification without touching ov or buf
        return ERROR_IO_PENDING;
    }

    // (3) The issue itself failed. No completion notification will arrive, so the
    // caller cleans up right here
    return err;
}

ERROR_IO_PENDING is not an error — it’s “accepted.” Treating it as an ordinary error, or conversely writing code that never accounts for the TRUE (synchronous completion) case, are the two classic bugs here. Why synchronous completion happens is covered in Section 5.

And there is one important caveat here. By default, a completion notification still arrives separately even for an operation that completed synchronously (TRUE). For a handle associated with an I/O completion port, a completion packet is still queued; with the event method, the event is still signalled. So if you write “process the result on the spot if TRUE, and process it again when the notification arrives,” you end up with an accident where the same operation is processed twice and its slip is freed twice. The safe basic shape is to funnel result handling through the notification path for both the “TRUE (synchronous completion)” and “ERROR_IO_PENDING” routes. The third route is the exception — when the issue itself fails (FALSE + some other error), no completion notification arrives. If you route that case into the wait-for-notification path, you’ll wait forever, so the issuing code must handle the error and clean up the slip on the spot. Only if you want to switch to “skip the notification and handle it on the spot when synchronously completed” should you explicitly enable SetFileCompletionNotificationModes (FILE_SKIP_COMPLETION_PORT_ON_SUCCESS) — but note this only suppresses the packet to the I/O completion port; it does not suppress the signalling of OVERLAPPED.hEvent. It’s an optimisation specific to the IOCP path and cannot be used with the event method (Section 5).¹²

Note that if you pass an OVERLAPPED to a synchronous-mode handle, the read happens from the specified Offset, but the call still blocks until completion.² “I passed an OVERLAPPED, so it’s asynchronous” is not correct — the mode belongs to the handle, full stop.

4. How Do You Learn of Completion? — Four Notification Paths

Once issuing and completion are separated, how you receive “it’s done” becomes central to the design. There are really four paths.¹

flowchart TB
    DONE["I/O completes in the kernel<br/>- IoCompleteRequest, then the result is finalised via an APC"]
    N1["(1) The file handle becomes signalled<br/>Receive via: WaitForSingleObject on the handle"]
    N2["(2) OVERLAPPED's hEvent becomes signalled<br/>Receive via: WaitForSingleObject + GetOverlappedResult"]
    N3["(3) A completion routine is queued onto the issuing thread's APC queue<br/>Receive via: runs during an alertable wait such as SleepEx"]
    N4["(4) A completion packet lands on an I/O completion port<br/>Receive via: GetQueuedCompletionStatus - Part 3"]
    DONE --> N1
    DONE --> N2
    DONE --> N3
    DONE --> N4

Figure 4: The four completion-notification paths. Which one you get depends on how you issued the operation — whether there’s an hEvent, whether you used ReadFileEx, whether the handle is associated with a port

Here’s the overview as a table first; each subsection explains one row.

4.1. Handle Signalling — Don’t Use It

If you issue without setting hEvent, the file handle itself becomes signalled on completion. It looks convenient, but if several operations are in flight on the same handle, you can’t tell which one completed.¹ Barring the special case of “issuing asynchronous I/O one at a time only,” it’s safest not to use this.

4.2. Event + GetOverlappedResult — the Basic Shape

Set a manual-reset event in OVERLAPPED.hEvent when issuing, wait on it with WaitForSingleObject (or WaitForMultipleObjects for several at once), and retrieve the result (success/failure and bytes transferred) with GetOverlappedResult.¹³ Passing TRUE for GetOverlappedResult’s bWait gives you “wait for completion, then retrieve” in one call. If the event is auto-reset, there’s a trap where another wait can consume the signal before you get to it, leaving GetOverlappedResult stuck — which is why a manual-reset event is recommended.¹³⁴

This is the clearest, most solid way to handle a handful of concurrent I/Os. For devices like serial ports where “reading while writing” is mandatory, this shape is still very much in active use (“Serial Communication App Pitfalls”).

4.3. APC — Delivered to the Issuing Thread

ReadFileEx/WriteFileEx take a completion routine (callback) instead of an event. On completion, that routine is queued onto the issuing thread’s APC queue and runs when the thread enters an alertable wait, such as SleepEx or WaitForSingleObjectEx.¹⁴⁵¹⁵

The distinguishing feature of this method is that completion processing always runs on the issuing thread. That means you don’t need locking, but the flip side is that the completion routine never runs at all unless the issuing thread enters an alertable wait. Combining it with a UI thread’s message loop requires MsgWaitForMultipleObjectsEx, and the waiting design gets tricky enough that most people reach for an event or IOCP for general-purpose use.

And “the APC never arrives” is the classic bug with this method. There’s almost always one cause: the wait isn’t alertable.

// C++ / Win32. hFile is a handle opened with FILE_FLAG_OVERLAPPED,
// and ov and buf are assumed to be kept alive until completion (Section 3.2)

// Bad example: the completion routine is never called
ReadFileEx(hFile, buf, len, ov, OnReadCompleted);
Sleep(1000);            // Not an alertable wait. The APC will not be delivered

// Good example: keep alertably waiting until this specific I/O finishes
//
// Set this flag from within the completion routine (e.g. hosted in a struct alongside ov)
volatile bool completed = false;

// Always check whether the issue itself succeeded. A return of 0 means
// no completion routine was ever queued
if (!ReadFileEx(hFile, buf, len, ov, OnReadCompleted))
{
    const DWORD err = GetLastError();   // Retrieve it immediately -- it gets overwritten by the next API call
    ReportError(err);                   // e.g. the device was removed, or the handle is invalid
    return;                             // Do NOT enter the wait loop below
}

while (!completed)
{
    DWORD r = SleepEx(1000, TRUE);   // TRUE as the second argument makes this alertable
    if (r == WAIT_IO_COMPLETION)
    {
        // Some APC ran. It isn't necessarily this one's, though,
        // so check completed and wait again if it isn't set
        continue;
    }
    // Returned on timeout. The I/O is still outstanding, so if we're
    // giving up, cancel with CancelIoEx and wait for the completion to be delivered
    CancelIoEx(hFile, ov);
}

Never enter the wait loop without checking ReadFileEx’s return value. If the issue itself fails — say, the device was just unplugged, or the handle is already invalid — ReadFileEx returns 0 and not a single completion routine gets queued. Enter while (!completed) in that state and completed will never become true: you get a loop that endlessly repeats SleepEx and CancelIoEx against I/O that doesn’t exist. And because it looks exactly like “the device just isn’t responding,” it takes forever to track down the real cause. If you get a 0, take GetLastError() on the spot — even a single intervening API call will overwrite it — and exit without entering the wait.

Calling SleepEx just once is not enough. When it returns on timeout, the thread exits the alertable wait right there. The I/O is still outstanding, so if the scope exits afterward and ov or buf go away, the kernel ends up writing into a buffer it still thinks is alive (Section 3.2). Either keep waiting until the completion routine has recorded that it fired, or cancel with CancelIoEx and then wait for the completion to be delivered — one or the other.

A return of WAIT_IO_COMPLETION only means “at least one APC ran” — it doesn’t necessarily mean it was your I/O’s completion routine. If some other I/O’s APC, or one queued by QueueUserAPC, was pending on the same thread, that’s what you’ll come back on instead. So don’t judge purely by the return value — check the flag you set yourself.

The choice of wait function itself is simple, though: swap Sleep for SleepEx(..., TRUE), and WaitForSingleObject for WaitForSingleObjectEx(..., TRUE). When you’ve written a completion routine and nothing happens, first check whether the wait function’s name ends in Ex and whether the alertable argument is TRUE.⁵

4.4. I/O Completion Ports — the Scalable Answer (Next Time)

The mechanism for handling many concurrent I/Os with a small number of threads is the I/O completion port (IOCP). Associate a handle with a port and completions land in the port’s queue, from which worker threads retrieve them with GetQueuedCompletionStatus.⁶ It’s also where .NET’s async/await I/O ultimately ends up. We’ll spend the whole of next time digging into it.

5. The “Asynchronous, But It Completed Synchronously” Problem

This is where designs for asynchronous I/O most commonly trip up first. Even when you issue correctly in asynchronous mode, an I/O completing synchronously is entirely normal. Microsoft’s troubleshooting documentation explicitly states the typical reasons.³

flowchart TB
    A["Issue ReadFile/WriteFile on an asynchronous handle"]
    Q{"Does it hit a synchronous-completion condition?"}
    C1["A request that can be satisfied right away<br/>- e.g. the data is already in the cache"]
    C2["An NTFS-compressed file<br/>- compressed files never go asynchronous"]
    C3["An NTFS-encrypted - EFS - file"]
    C4["A write that extends the file's length"]
    T["Returns TRUE immediately<br/>= it ran to completion inside the call"]
    P["Returns ERROR_IO_PENDING<br/>= genuinely in progress, asynchronously"]
    A --> Q
    Q --> C1
    Q --> C2
    Q --> C3
    Q --> C4
    C1 --> T
    C2 --> T
    C3 --> T
    C4 --> T
    Q -->|"None of these"| P

Figure 5: The main conditions for synchronous completion. Caching, compression, encryption, and extending writes all “never go asynchronous”

Each has a reason behind it.³

• Cache hits. Many drivers have special-case handling that completes a request on the spot when it can be satisfied immediately — for disk, this means the data is already in the memory-resident cache. It’s fast, so no one complains, but code that assumes “ERROR_IO_PENDING is always returned” breaks right here.
• Conversely, there’s a trap even when the data isn’t cached. Windows’s cache is implemented via file mapping, and because page-fault handling has no asynchronous mechanism when a page is missing, a cache-enabled asynchronous read can end up being processed synchronously. The cache mechanism itself is covered in Part 4.
• NTFS compression / EFS encryption. The file system driver converts access to compressed or encrypted files into synchronous access.
• A write that extends the file’s length. A write that changes the length becomes synchronous.

The practical implications are simple.

1. Always write the “returns TRUE immediately” path. All three branches in Figure 3 are normal operation. That said, since a completion notification still arrives separately for synchronous completions by default, it’s safest to funnel the actual result handling through the notification path (Section 3.3).
2. You cannot rely on this for responsiveness. “It’s asynchronous, so the UI won’t freeze” does not hold. Threads that must never freeze need a design where they simply don’t issue I/O themselves in the first place — offloading to a dedicated thread or a thread pool. This ground is also covered in “A Practical Guide to Getting as Close to Soft Real-Time as Possible on Ordinary Windows.”
3. For high-frequency I/O, synchronous completion is also an optimisation opportunity. There’s an API, SetFileCompletionNotificationModes, that skips the packet to the I/O completion port on synchronous completion, and it pays off when combined with IOCP (Part 3).¹²

6. Cancellation and Cleanup — “Please Stop” Is a Request

The proper way to stop a long-running I/O (an unresponsive network destination, serial data that never arrives) is CancelIoEx.⁷

sequenceDiagram
    participant App as Application
    participant IOM as I/O manager
    participant DRV as Driver

    App->>IOM: CancelIoEx - handle, OVERLAPPED
    Note over IOM: Requests cancellation - marks<br/>the matching outstanding IRP
    IOM->>DRV: Calls the cancel routine
    Note over DRV: Aborts if it's in a cancellable state<br/>may still complete normally if it was already close to done
    DRV->>IOM: IoCompleteRequest<br/>- STATUS_CANCELLED
    IOM-->>App: The completion notification arrives<br/>GetOverlappedResult reports ERROR_OPERATION_ABORTED
    Note over App: Only after seeing this notification<br/>do you free the OVERLAPPED and the buffer

Figure 6: How cancellation actually works. A cancelled operation still comes back as a “completion”

Knowing the mechanism, three consequences follow naturally.

• Cancellation is an asynchronous “request.” Even if CancelIoEx succeeds, all it did was “mark” the operation. One that was already close to completing can still complete normally.⁸
• A cancelled operation still comes back as a completion, with ERROR_OPERATION_ABORTED. Until you receive that notification, the kernel still considers the OVERLAPPED and the buffer in use. Freeing them first causes memory corruption. “It started crashing after I added cancellation” is almost always this.⁷⁸
• Finish off outstanding I/O before you close the handle. As we saw in Part 1, when the last handle closes, cleanup processing does cancel any outstanding IRPs — but code that “closes the handle while an issued I/O is still outstanding” tends to fall apart around managing the completion notification and buffer lifetime. Cancel → see the completion → close is the rule to follow.

Two footnotes. The older CancelIo can only cancel I/O issued by the calling thread itself (a limitation that existed until CancelIoEx arrived in Vista, and there’s no reason to reach for it deliberately today).¹⁶ And for a different thread stuck in synchronous I/O, use CancelSynchronousIo.¹¹ “The OS won’t handle timeouts for you — you design cancellation yourself” is the core of practical asynchronous I/O work.

7. Seen from .NET — a Mode Mismatch Produces “Fake Asynchrony”

Everything so far maps directly onto .NET code. FileStream’s constructor parameter useAsync (or FileOptions.Asynchronous) is exactly the direct switch for FILE_FLAG_OVERLAPPED (see the mapping table in Part 1).

flowchart TB
    A["await fs.ReadAsync(...)"]
    Q{"Is the handle in asynchronous mode -<br/>FileOptions.Asynchronous?"}
    Y["Genuine asynchronous I/O<br/>Issues something equivalent to OVERLAPPED<br/>Completion reaches the thread pool via IOCP - Part 3"]
    N["Fake asynchrony<br/>A thread-pool thread stands in<br/>and waits on a synchronous Read"]
    A --> Q
    Q -->|Yes| Y
    Q -->|No| N

Figure 7: The same ReadAsync call ends up doing something completely different underneath, depending on the handle’s mode

The difference comes down to a single line — where you open the file. The call site for ReadAsync looks identical either way, so you can’t tell just by reading the calling code.

using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.Win32.SafeHandles;

string path = @"C:\temp\data.bin";
byte[] buffer = new byte[4096];

// (A) Fake asynchrony. Omitting useAsync, or setting it false, opens the handle in synchronous mode
using (var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read,
                               bufferSize: 4096, useAsync: false))
{
    // The caller isn't blocked, but behind the scenes one thread-pool thread stands in for a synchronous Read and waits
    await fs.ReadAsync(buffer, 0, buffer.Length);
}

// (B) Genuine asynchrony. useAsync: true maps directly onto FILE_FLAG_OVERLAPPED
using (var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read,
                               bufferSize: 4096, useAsync: true))
{
    // Completion reaches the thread pool via IOCP (Part 3)
    await fs.ReadAsync(buffer, 0, buffer.Length);
}

// (C) .NET 6 and later. A plain style with the mode and offset both explicit
using (SafeFileHandle handle = File.OpenHandle(path, FileMode.Open, FileAccess.Read,
                                               options: FileOptions.Asynchronous))
{
    int read = await RandomAccess.ReadAsync(handle, buffer, fileOffset: 0);
}

The only difference between (A) and (B) is the single word useAsync (writing FileOptions.Asynchronous amounts to the same thing) — and that single word is exactly the reproduction condition for “fake asynchrony.” When auditing existing code, don’t look at the ReadAsync/WriteAsync call site — look for where the FileStream is constructed. Short overloads like File.OpenRead or new FileStream(path, FileMode.Open) all open in synchronous mode. Note also that when constructing a FileStream from a SafeFileHandle, the isAsync argument must match the handle’s actual mode.

• A synchronous-mode handle plus ReadAsync is “fake asynchrony” — a thread-pool thread performs the synchronous read. The caller doesn’t wait, but one thread is asleep behind the scenes. The real-world impact is small when this happens occasionally, but on a server or under high-frequency processing it becomes a source of thread-pool exhaustion.⁹
• An asynchronous-mode handle plus a synchronous Read is the mismatch in the other direction, and incurs internal completion-waiting overhead. Keeping the mode and the API you call in sync is the principle here.¹⁰
• From .NET 6 onward, FileStream’s internals were rewritten wholesale, and an API arrived — File.OpenHandle + RandomAccess — that lets you “read and write with an explicit SafeFileHandle and offset.”⁹ This style, where you pass an offset every single time, is exactly the raw Win32 shape we’ve seen in this article: an asynchronous handle plus OVERLAPPED.Offset.
• As long as the handle is in asynchronous mode, cancelling file I/O via CancellationToken ultimately reaches CancelIoEx internally. Behind a ReadAsync that was given a token and ends in an OperationCanceledException, the diagram from Section 6 is exactly what’s running. The same caveats apply — cancellation is a “request” and immediacy is not guaranteed. On the other hand, with the “fake asynchrony” of a synchronous-mode handle, there’s no overlapped operation to target for cancellation, so this path is not available. Recent .NET runtimes do include a mechanism that attempts cancellation via CancelSynchronousIo for calls executing synchronously like this, but whether it works depends on the runtime version and the type of operation, and it does not guarantee a reliable abort. If cancellation is meant to be a design premise, the right approach is to keep the mode consistent and use genuine asynchronous I/O.

For the layer above this — how you should actually write async/await (ConfigureAwait, the relationship with the UI thread) — see “A Practical Decision Table for C# async/await - Task.Run and ConfigureAwait” and “WPF/WinForms async and the UI Thread on One Sheet.” This article is one floor down from that; next time (IOCP) takes us down another.

8. Summary

• Synchronous and asynchronous I/O are not separate plumbing — the difference is whether the I/O manager waits for completion or returns without waiting. A synchronous I/O’s thread sleeps in a wait state and consumes no CPU.¹
• Mode belongs to the handle (file object); state belongs to the operation (OVERLAPPED). Because an asynchronous handle has no maintained file pointer, files with a position must specify it via Offset every time.²⁴
• Keep the OVERLAPPED and the buffer alive and untouched until the completion notification. Prepare one for every concurrent issue. Reusing one means data corruption.³
• Completion notifications come via four paths: handle, event, APC, IOCP. Don’t use handle signalling with multiple concurrent I/Os; use manual-reset events; APCs require an alertable wait.¹¹³⁵
• Even asynchronously issued I/O completes synchronously for caching, NTFS compression/encryption, and extending writes. Always write the “returns TRUE immediately” path as a normal case, and don’t rely on it for responsiveness.³
• Cancellation is a request. Even after CancelIoEx, see the completion notification (ERROR_OPERATION_ABORTED) through before cleaning up. Cancel → confirm completion → close, in that order.⁷⁸
• .NET’s FileOptions.Asynchronous is the direct line to FILE_FLAG_OVERLAPPED, and a mismatch between mode and API produces “fake asynchrony.” Underneath a CancellationToken, CancelIoEx is what’s actually running.⁹¹⁰

Next up is Part 3, “I/O Completion Ports (IOCP) and the .NET Thread Pool — Down Another Floor from async/await.” We’ll go down into why IOCP — mentioned by name alone in Section 4.4 here — is a design that unifies “a queue of completion notifications” with “control over the number of executing threads,” and even which thread the rest of an await actually runs on.

Related Articles

• The Depths of Windows I/O (Part 1) — Every Read and Write Becomes an IRP: The Big Picture of the I/O System
• A Practical Decision Table for C# async/await - Task.Run and ConfigureAwait
• WPF/WinForms async and the UI Thread on One Sheet
• Serial Communication App Pitfalls - Through Reconnection and Log Design
• Why You Should Prefer Event Waits over Sleep(1) on Windows
• A Practical Guide to Getting as Close to Soft Real-Time as Possible on Ordinary Windows
• The Misconception That TCP Lets You Receive in the Same Units You Send — Designing Reception Around a Byte Stream

Related Consulting Areas

KomuraSoft LLC works on the design of Windows business applications and device-communication applications that use asynchronous I/O, and on investigating the causes of faults such as freezing, crashes on cancellation, and thread-pool exhaustion.

• Windows App Development
• Bug Investigation and Root Cause Analysis
• Soft Real-Time Windows App Development
• Contact Us

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What changes when I add FILE_FLAG_OVERLAPPED?

The file object behind the handle is opened in "asynchronous mode." This is a per-handle property decided at the moment you call CreateFile — you cannot switch between synchronous and asynchronous on a call-by-call basis. On an asynchronous-mode handle you must always pass an OVERLAPPED structure to ReadFile/WriteFile. The system does not maintain a file pointer (current position) for this handle, so for devices that have a position, such as a file on disk, you specify the read/write position every time via the OVERLAPPED structure's Offset (devices with no notion of position, such as serial ports, don't use Offset). An issued operation can return control before it completes, in which case ReadFile returns FALSE and GetLastError reports ERROR_IO_PENDING. You receive completion through a notification mechanism — an event, an APC, an I/O completion port, and so on.

I issued an asynchronous I/O, so why does it come back completed right away?

Because asynchronous mode means "you don't have to wait for completion," not "you will never be made to wait." Microsoft's documentation lists the typical reasons an operation issued asynchronously nonetheless completes synchronously: a request that can be satisfied immediately (for example, the data is already in the cache), an NTFS-compressed file, an NTFS-encrypted (EFS) file, and a write that extends the length of the file. In these cases ReadFile/WriteFile returns TRUE and the result is already final. Code that uses asynchronous I/O must therefore handle both the "returns with ERROR_IO_PENDING" case and the "completes on the spot" case — and responsiveness is never absolutely guaranteed. Note that, by default, a completion notification (an event being signalled, or a packet arriving at an I/O completion port) still arrives separately even for an operation that completed synchronously, so it is safest to funnel result handling through the notification path alone.

Can I reuse an OVERLAPPED structure?

Not for more than one operation at the same time. An OVERLAPPED structure represents "the state of one operation currently in flight," and Microsoft's documentation explicitly states that if you issue three I/O operations you need three OVERLAPPED structures, and reusing one leads to unpredictable results and data corruption. Until an operation completes, keep both the structure and the read/write buffer valid and do not touch their contents. If you plan to reuse a structure after an operation has completed, reinitialise it each time so that leftover data from the previous use has no effect. For hEvent, using a manual-reset event is the safe choice.

How do I cancel an I/O operation that is already in progress?

CancelIoEx lets you request cancellation of an outstanding I/O on a given handle regardless of which thread issued it. Pass an OVERLAPPED as the second argument to target one specific operation, or NULL to target every operation on that handle. The older CancelIo can only cancel operations issued by the calling thread itself. What matters is that cancellation is a "request," not an immediate "guarantee." An operation that was already close to completing can still complete normally, and a cancelled operation is reported as completed with ERROR_OPERATION_ABORTED. Either way, you must not free the OVERLAPPED structure or the buffer until you have received the completion notification. For a thread that is blocked in synchronous I/O on another thread, there is a dedicated API called CancelSynchronousIo.

What happens if I don't specify FileOptions.Asynchronous (useAsync) on .NET's FileStream?

The handle is opened in synchronous mode, so calling ReadAsync/WriteAsync does not produce genuine asynchronous I/O — instead, a thread-pool thread stands in and performs the synchronous read/write, giving you "fake asynchrony." The calling thread isn't blocked, but a different thread is left waiting behind the scenes, which can lead to thread-pool exhaustion and reduced scalability. Conversely, opening in asynchronous mode and then calling the synchronous Read/Write incurs the overhead of waiting for completion internally. The principle is to keep "the handle's mode" and "the API you call" in sync; from .NET 6 onward, File.OpenHandle together with RandomAccess lets you write this out plainly, with the mode and the offset both explicit.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.