The Depths of Windows I/O (Part 2) — Synchronous and Asynchronous I/O: What OVERLAPPED Really Means

Last time (Part 1) we saw that a Windows I/O request becomes a packet called an IRP that flows through the device stack, and that issuing a request and completing it are separate things inside the kernel. This time we take up asynchronous I/O (overlapped I/O), which is how an application puts that separation to use.

You added FILE_FLAG_OVERLAPPED and the call still makes you wait. You reused an OVERLAPPED and the data got corrupted. You freed the buffer right after canceling and the process crashed. The key to understanding all three is a single division of labor: the mode belongs to the handle, the state belongs to each operation, and cleanup happens only after completion has been confirmed.

This article follows the sequence in order: open the file, issue the I/O, receive the result, clean up. Once the Win32 mechanism is clear, we check where .NET’s FileStream, ReadAsync, and CancellationToken connect to it.

This is Part 2 of the series “The Depths of Windows I/O”. The overall structure is laid out at the start of Part 1.

1. The Bottom Line First: Three Distinctions That Are Easy to Confuse

Asynchronous I/O is hard to reason about if you judge it by API names alone. Start by separating what you configure from the point at which completion is decided.

Synchronous I/O never returns to the caller before the operation completes. Asynchronous I/O gives you a path that returns before completion. However, even in asynchronous mode an operation can complete inside the call, so this is not a guarantee that you will never be made to wait.¹²

In an implementation, think in this order: decide the mode, prepare a structure and a buffer dedicated to the operation, judge the result of issuing, receive completion, clean up. Even when you have requested cancellation, you do not skip the stage where completion is received.³⁴

If you already know what you are after, start from the guide below.

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (36 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

2. Synchronous I/O: A Thread Waiting for Completion Sleeps Without Burning CPU

2.1. The I/O Manager Is What Waits for Completion

A handle opened without FILE_FLAG_OVERLAPPED is in synchronous mode. ReadFile does not return until the I/O has completed.¹

When a driver pends the request because it is waiting on the hardware to respond, the I/O Manager waits for completion before returning control to the application. The application’s thread waits inside the kernel for that entire period.

sequenceDiagram
    participant App as Application thread
    participant IOM as I/O Manager
    participant DRV as Driver (stack)

    App->>IOM: ReadFile(synchronous handle)
    IOM->>DRV: Issue an IRP
    DRV-->>IOM: STATUS_PENDING (waiting on the device)
    Note over App,IOM: The thread enters a wait state inside the kernel<br/>and sleeps without consuming CPU
    DRV->>IOM: Completion (IoCompleteRequest)
    IOM-->>App: Return the result and wake the thread<br/>ReadFile returns TRUE or FALSE

Figure 1: Synchronous I/O for a request that was pended. ReadFile returns only after completion has been waited for

That said, synchronous I/O does not always put the thread to sleep. A request that can be satisfied on the spot, such as a cache hit, returns its result with no wait at all (the “immediate completion” path in Figure 5 of Part 1). What is guaranteed is only that the call does not return before completion.

2.2. Not Burning CPU and Being Able to Do Other Work Are Different Things

A thread in a wait state is removed from the scheduler’s set of runnable threads, so it consumes no CPU. Why letting the thread wait beats polling on your own is also explained in “Why You Should Prefer Event Waits over Sleep(1) on Windows.”

A waiting thread, on the other hand, cannot do any other work. If it is the UI thread, the screen freezes; if a server dedicates a thread to each connection, a few hundred connections mean a few hundred threads. The weakness of synchronous I/O is not CPU usage but the fact that the thread is unavailable until completion.

In synchronous mode the kernel also maintains the file pointer (the current position). That is why consecutive ReadFile calls read “from where the last one left off.” The position belongs to the file object behind the handle, so handles duplicated with DuplicateHandle share the same position (Part 1, Section 3.3).

There is also CancelSynchronousIo, which requests cancellation of a synchronous I/O running on another thread. Section 6 sums up when to use it versus the APIs meant for asynchronous I/O.⁵

3. Preparing and Issuing Asynchronous I/O: Separate the Mode, the State, and the Return Value

3.1. Asynchronous Mode Is Decided When You Open the File

Passing FILE_FLAG_OVERLAPPED to CreateFile puts the file object behind the handle into asynchronous mode. The mode is not something you switch per call. You can open the same file twice, with one handle for synchronous use and one for asynchronous use, and in that case there are two file objects as well.¹

In asynchronous mode the system does not maintain a file pointer. Because several operations can be in flight at once, for a file on disk you specify the read/write position every time with OVERLAPPED.Offset / OffsetHigh. For devices with no seek position, such as serial ports and named pipes, this position is not used and should be left at zero. Even when you do not specify a position, a dedicated OVERLAPPED for the operation is still required.⁶

Conversely, passing an OVERLAPPED to a synchronous-mode handle does not make it asynchronous. It reads from the position in Offset, but the behavior of blocking until completion does not change. What matters is not whether you passed the structure, but which mode the handle was opened in.⁶

3.2. Match One OVERLAPPED and One Buffer to One Operation

OVERLAPPED is the structure that identifies an in-flight operation and carries its position, state, and result. Think of it as “the slip for one operation” and the division of labor with the handle becomes clear.³

flowchart LR
    subgraph H["Handle (file object) = mode"]
        M1["Synchronous mode<br/>the kernel maintains the current position<br/>ReadFile does not return until completion"]
        M2["Asynchronous mode (FILE_FLAG_OVERLAPPED)<br/>the current position is not maintained<br/>issuing and completion are separated"]
    end
    subgraph O["OVERLAPPED structure = the slip for one operation"]
        F1["Offset: where to read"]
        F2["hEvent: how you learn of completion"]
        F3["Internal/InternalHigh:<br/>state and result (written by the system)"]
    end
    C["Decided once, at CreateFile time"] --> H
    R["One prepared for every ReadFile/WriteFile issued"] --> O

Figure 2: The handle holds the mode; OVERLAPPED holds the position and state of each operation

Two things must be observed here: count and lifetime. If you issue three I/O operations at once, prepare three OVERLAPPED structures. Sharing one structure across several outstanding operations leads to unpredictable results and data corruption.²

Also, until an operation completes, keep the structure and the data buffer valid: do not modify, reuse, or free them. The kernel is still using that memory. If you issue with an OVERLAPPED that is a local variable and leave the function while the operation is outstanding, you hand the kernel a stack region whose lifetime is over.³²

When you reuse a structure after confirming completion, reinitialize it so no state from the previous operation survives. If you choose the event method, use a manual-reset event for hEvent. How that relates to the way you wait is covered in Section 4.2.³

3.3. Split ReadFile’s Return Value Three Ways and Decide Where Each Is Handled

The result of issuing ReadFile against an asynchronous handle is judged from the combination of the return value and GetLastError(). The important part is not to treat every FALSE as a failure.⁶

flowchart TB
    A["ReadFile(asynchronous handle, with OVERLAPPED)"]
    Q{"What is the return value?"}
    T["TRUE<br/>completed on the spot (synchronous completion)<br/>by default a completion notification also arrives"]
    P["FALSE + ERROR_IO_PENDING<br/>accepted. Completion is notified later"]
    E["FALSE + another error<br/>issuing itself failed"]
    W["Wait for the completion notification<br/>(the four methods in Section 4)"]
    A --> Q
    Q --> T
    Q --> P
    Q --> E
    P --> W

Figure 3: Handling the three-way split of synchronous completion, accepted and in progress, and failure to issue

The function below does nothing but this judgment and returns to the caller. Preparing the handle, the per-operation structure, buffer, and event, and the code that receives completion, are assumed to exist elsewhere.

// C++ / Win32
// hFile : a handle opened with FILE_FLAG_OVERLAPPED
// ov    : an OVERLAPPED allocated for this operation alone (Offset and hEvent already set)
// buf/len: a buffer for this operation alone. Do not free it until the completion notification arrives
DWORD IssueRead(HANDLE hFile, OVERLAPPED* ov, BYTE* buf, DWORD len)
{
    // For an asynchronous issue, pass NULL for lpNumberOfBytesRead and
    // retrieve the transferred byte count with GetOverlappedResult after completion
    if (ReadFile(hFile, buf, len, nullptr, ov))
    {
        // (1) Synchronous completion. A notification also arrives by default, so do not handle the result here
        return ERROR_SUCCESS;
    }

    DWORD err = GetLastError();
    if (err == ERROR_IO_PENDING)
    {
        // (2) Accepted. Wait for the completion notification without touching ov or buf
        return ERROR_IO_PENDING;
    }

    // (3) Issuing itself failed. No notification will arrive, so the caller cleans up here
    return err;
}

ERROR_IO_PENDING is the result “accepted, not yet complete.” It must not be cleaned up as an ordinary error. Synchronous completion returning TRUE is equally a normal path, so always handle it. Why an operation completes synchronously is explained in Section 5.

Handle the result exactly once. By default, even for an operation that completed synchronously, a completion packet is queued if the handle is associated with an IOCP, and the event is signaled if you are using the event method. If you handle the result both right after TRUE and again when the notification arrives, you process the same operation twice and risk freeing the structure twice. The safe baseline is to funnel both the TRUE path and the ERROR_IO_PENDING path into result handling on the notification side.¹

By contrast, the path where issuing itself failed is where the issuer performs the error handling and the cleanup. If you send it off to wait when no notification is coming, you wait forever.

There is also an optimization that skips the IOCP notification on synchronous completion, but that is a different design from the default behavior. Where FILE_SKIP_COMPLETION_PORT_ON_SUCCESS applies is covered separately in Section 5.3.⁷

4. Choosing a Completion Notification: Decide by the Number of I/Os and the Thread That Processes Them

Once issuing and completion are separate, you need a way to receive completion. Start by comparing the four methods along two axes: how many I/Os you handle at once and which thread runs the completion processing.¹

flowchart TB
    DONE["The I/O completes in the kernel<br/>(IoCompleteRequest, then an APC finalizes the result)"]
    N1["(1) The file handle becomes signaled<br/>receiving: WaitForSingleObject(handle)"]
    N2["(2) The OVERLAPPED's hEvent becomes signaled<br/>receiving: WaitForSingleObject + GetOverlappedResult"]
    N3["(3) The completion routine is queued to the issuing thread's APC queue<br/>receiving: it runs during an alertable wait such as SleepEx"]
    N4["(4) A completion packet enters the I/O completion port<br/>receiving: GetQueuedCompletionStatus (Part 3)"]
    DONE --> N1
    DONE --> N2
    DONE --> N3
    DONE --> N4

Figure 4: The four completion-notification paths. How you receive completion depends on how you issued the I/O

4.1. Signaling the Handle: You Cannot Tell the Operations Apart

If you issue without specifying hEvent, the file handle itself becomes signaled at completion. But when several operations are in flight on the same handle, you cannot tell which one completed.¹

Unless you are in the special case of never having more than one asynchronous I/O in flight, it is safest not to use this. Convenient as it looks, it does not give you a way to manage results per operation.

4.2. Events and GetOverlappedResult: The Baseline for a Handful of Concurrent I/Os

You set a manual-reset event in each operation’s OVERLAPPED.hEvent and issue. After waiting with WaitForSingleObject, you retrieve success or failure and the transferred byte count with GetOverlappedResult. To wait on several events together you use WaitForMultipleObjects, but it can wait on at most 64 at a time.¹⁸

Setting GetOverlappedResult’s bWait to TRUE also lets you wait for completion and then take the result. If you use an auto-reset event here, GetOverlappedResult can keep waiting after another wait has consumed the signal. Using a manual-reset event is how you avoid that rendezvous problem.⁸³

For handling a few concurrent I/Os dependably, this is a clear and readable method. It is also used in the “read while writing” logic of serial ports. For a practical example see “Serial Communication App Pitfalls.”

4.3. APC: Keep the Issuing Thread in an Alertable Wait Until Completion

ReadFileEx / WriteFileEx are the method where you specify a completion routine (a callback). When the I/O completes, the routine is queued to the APC queue of the thread that issued it. It runs when that thread enters an alertable wait through SleepEx, WaitForSingleObjectEx, or a similar call.⁹¹⁰¹¹

Because completion processing runs serially on the same thread, logic that stays within a single thread can avoid locks. On the other hand, if the issuing thread never enters an alertable wait, the completion routine never runs. Combining it with a UI message loop requires MsgWaitForMultipleObjectsEx, so designing the wait becomes harder. For general use, events or IOCP are the more common choice.

The three things easiest to overlook with APCs are whether the issue succeeded, whether you are waiting correctly, and whether your own operation has completed. The code below is an excerpt contrasting a bad wait with a good one; it is not an example of running the two in sequence. Preparing the handle and the buffer, and OnReadCompleted, which updates the per-operation completion flag, are assumed to exist elsewhere.

// C++ / Win32. hFile is a handle opened with FILE_FLAG_OVERLAPPED,
// and ov and buf are assumed to stay alive until completion (Section 3.2)

// Bad example: the completion routine is never called
ReadFileEx(hFile, buf, len, ov, OnReadCompleted);
Sleep(1000);            // not an alertable wait. The APC is not delivered

// Good example: keep waiting alertably until this I/O finishes
//
// The completion routine sets this flag (hold it in the structure that hosts ov, for example)
volatile bool completed = false;

// Always check whether the issue succeeded. When 0 is returned, no completion routine was queued
if (!ReadFileEx(hFile, buf, len, ov, OnReadCompleted))
{
    const DWORD err = GetLastError();   // take it immediately. Later APIs overwrite it
    ReportError(err);                   // device removed, invalid handle, and so on
    return;                             // NOTE: you must not enter the wait loop below
}

while (!completed)
{
    DWORD r = SleepEx(1000, TRUE);   // the TRUE in the second argument is alertable
    if (r == WAIT_IO_COMPLETION)
    {
        // Some APC ran. It is not necessarily your own I/O, however,
        // so decide from completed, and wait again if it is not yours
        continue;
    }
    // Returned on timeout. The I/O is still outstanding, so
    // if you are giving up, cancel with CancelIoEx and wait for completion to be delivered
    CancelIoEx(hFile, ov);
}

If issuing fails, do not enter the wait. When ReadFileEx returns 0 because the device was removed, the handle is invalid, or for some other reason, no completion routine has been queued. Take GetLastError() immediately, handle the error, and leave. Miss this and completed never becomes true while you repeat SleepEx and CancelIoEx against an I/O that does not exist.⁹

Do not treat a wait timeout as the end of the I/O. When SleepEx times out it leaves the alertable wait, but the issued I/O can still be outstanding. Do not leave the scope and let ov or buf expire at that point. Either keep waiting for completion, or, if you are giving up, request cancellation and wait until that completion is delivered. The lifetime rule from Section 3.2 is the same after a timeout.

Do not conclude from WAIT_IO_COMPLETION alone that your I/O has finished. That return value means one or more APCs ran. If another I/O or a QueueUserAPC APC is queued to the same thread, it returns for those too. Judge from the flag your own completion routine updates, and wait again if it is not set yet.¹⁰

When “the APC never arrives,” check the wait function as well as whether the issue succeeded. Is it SleepEx(..., TRUE) rather than Sleep, and WaitForSingleObjectEx(..., TRUE) rather than WaitForSingleObject? The trailing Ex and the TRUE in the alertable argument are the two things to check.¹⁰

4.4. IOCP: Serving Many I/Os with a Few Workers

With an I/O completion port (IOCP) you associate the handle with a port. Completion packets enter the port’s queue and worker threads take them out with GetQueuedCompletionStatus. It is the mechanism for processing many concurrent I/Os with a small number of threads.¹²

It is also the path that underpins .NET’s asynchronous I/O. How the queue of completion notifications is combined with control over the number of concurrently running threads is covered in detail next time, in Part 3.

5. The Synchronous-Completion Exception: “Asynchronous” Is Not the Same as “Never Made to Wait”

5.1. The Typical Conditions for Completing Inside the Call

Even when you issue correctly in asynchronous mode, the I/O can complete inside the call. Synchronous completion means the I/O finished before the function returned; it is not a promise that the function returns quickly. Keep the case that finishes fast because of a cache hit separate from the case where you are made to wait inside the call.²

flowchart TB
    A["Issue ReadFile/WriteFile on an asynchronous handle"]
    Q{"Does it hit a synchronous-completion condition?"}
    C1["A request that can be satisfied immediately<br/>(the data is already in the cache, etc.)"]
    C2["An NTFS-compressed file<br/>(compressed files are never accessed asynchronously)"]
    C3["An NTFS-encrypted (EFS) file"]
    C4["A write that extends the length of the file"]
    T["Returns TRUE right away<br/>= it ran to completion inside the call"]
    P["Returns ERROR_IO_PENDING<br/>= genuinely in progress asynchronously"]
    A --> Q
    Q --> C1
    Q --> C2
    Q --> C3
    Q --> C4
    C1 --> T
    C2 --> T
    C3 --> T
    C4 --> T
    Q -->|"none of them"| P

Figure 5: The main conditions under which an asynchronously issued I/O completes synchronously. Keep this separate from how long the call takes to return

Microsoft’s troubleshooting document lists the following reasons.²

The important point is that synchronous processing can happen not only on a cache hit but also when the data is not in the cache. The cache mechanism itself is covered in Part 4 of this series.

5.2. Design the Issue-Result Split and UI Responsiveness Separately

The first requirement is to handle all three branches from Section 3.3. Treat TRUE as a normal result too, and by default funnel result handling to the notification side.

Writing the branches correctly still does not guarantee responsiveness, however. Since you cannot say “the UI will not freeze because the I/O is asynchronous,” you need a design that moves the act of issuing I/O off any thread that must never stall, handing it to a dedicated thread or a thread pool. The related practice is covered in “A Practical Guide to Getting as Close to Soft Real-Time as Possible on Ordinary Windows.”

5.3. Skipping the Notification on Synchronous Completion Is an IOCP-Only Optimization

For high-frequency I/O you can optimize by skipping the notification on synchronous completion. Enabling FILE_SKIP_COMPLETION_PORT_ON_SUCCESS through SetFileCompletionNotificationModes makes the system not queue a completion packet to the IOCP for an I/O that succeeded immediately. It is the setting for when you switch to a design that handles the result on the spot instead of on the notification side.⁷

What is skipped is only the packet to the IOCP: the signaling of OVERLAPPED.hEvent is not suppressed. Do not apply the same optimization to the event method. Mixing the default notification path with the optimized path leads to the double handling from Section 3.3 or to waiting for a notification that never comes. Its combination with IOCP is covered in Part 3.

6. Cancellation and Shutdown: Request, Confirm Completion, Close

6.1. Pick the API That Matches What You Are Canceling

The cancellation APIs are chosen according to the operation and the issuing thread.⁴¹³⁵

CancelIoEx was introduced in Vista. For asynchronous I/O there is no reason today to deliberately use the older CancelIo with its issuing-thread restriction, so make CancelIoEx your default.

6.2. CancelIoEx Succeeding Does Not Mean the I/O Has Ended

CancelIoEx is an API that requests cancellation of outstanding IRPs, not one that waits for the operation to complete. Success only means the cancellation has been requested. An operation that was already on the verge of completing may complete normally because the cancellation did not arrive in time.¹⁴

sequenceDiagram
    participant App as Application
    participant IOM as I/O Manager
    participant DRV as Driver

    App->>IOM: CancelIoEx(handle, OVERLAPPED)
    Note over IOM: Request cancellation of the matching<br/>outstanding IRP (mark it)
    IOM->>DRV: Call the cancel routine
    Note over DRV: Abort if it can still be canceled<br/>if it is nearly done, it may complete normally
    DRV->>IOM: IoCompleteRequest<br/>(STATUS_CANCELLED)
    IOM-->>App: The completion notification arrives<br/>GetOverlappedResult reports ERROR_OPERATION_ABORTED
    Note over App: Free the OVERLAPPED and the buffer<br/>only after seeing this notification

Figure 6: A canceled operation is also reported as a completion. Cleanup happens after that confirmation

An operation that really was canceled comes back in the completion notification as ERROR_OPERATION_ABORTED. Whether it completed normally or was canceled, do not free the structure or the buffer until you have received the notification. Free them first and the kernel loses memory it is still using, which leads to memory corruption. When you get an access violation after canceling, check this lifetime first.⁴¹⁴

6.3. Reclaim Issued Operations Before Closing the Handle

The basic shutdown sequence is request cancellation, see completion through, close the handle.

As we saw in Part 1, closing the last handle runs cleanup processing, which cancels outstanding IRPs. But closing the handle while issued I/O is still outstanding tends to wreck your management of completion notifications and buffer lifetimes. Do not delegate cleanup to the close; finish off the outstanding operations first.

The same applies when you want to abandon processing on a timeout. The OS does not decide your application’s abandonment criteria for you, so design the post-timeout cancellation together with the procedure for receiving completion. With the APC method, keep the alertable wait going until completion is delivered, as in Section 4.3.

7. How This Maps to .NET: Look at Where the File Is Opened, Not Just at ReadAsync

7.1. Keep the Handle’s Mode and the API You Call in Step

FileStream’s useAsync, or FileOptions.Asynchronous, corresponds to Win32’s FILE_FLAG_OVERLAPPED. Just as in the mapping table in Part 1, in .NET too it is the mode chosen when the file is opened that matters.¹⁵¹⁶

flowchart TB
    A["await fs.ReadAsync(...)"]
    Q{"Is the handle in asynchronous mode<br/>(FileOptions.Asynchronous)?"}
    Y["Genuine asynchronous I/O<br/>the equivalent of OVERLAPPED is issued and<br/>completion reaches the thread pool via IOCP (Part 3)"]
    N["Fake asynchrony<br/>a thread-pool thread stands in<br/>and waits on a synchronous Read"]
    A --> Q
    Q -->|Yes| Y
    Q -->|No| N

Figure 7: For one and the same ReadAsync, the handle’s mode changes the path taken on the OS side

Even with fake asynchrony the calling thread is not made to wait, but behind it another thread waits instead. With a small number of operations the real damage is limited, but in a server or in high-frequency processing it causes thread-pool exhaustion and reduced scalability. Keeping the mode and the API in step is the principle.¹⁶¹⁵

7.2. Compare Three Ways of Creating a FileStream

In (A) and (B) below, ReadAsync is called exactly the same way. The only difference is useAsync when the file is opened. (C) is the .NET 6 and later example that makes the handle and the position explicit.

using System;
using System.IO;
using System.Threading.Tasks;
using Microsoft.Win32.SafeHandles;

string path = @"C:\temp\data.bin";
byte[] buffer = new byte[4096];

// (A) Fake asynchrony. Omitting useAsync or setting it to false opens the handle in synchronous mode
using (var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read,
                               bufferSize: 4096, useAsync: false))
{
    // The caller is not blocked, but behind it one thread-pool thread stands in on a synchronous Read and waits
    await fs.ReadAsync(buffer, 0, buffer.Length);
}

// (B) Genuine asynchrony. useAsync: true maps straight onto FILE_FLAG_OVERLAPPED
using (var fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.Read,
                               bufferSize: 4096, useAsync: true))
{
    // Completion reaches the thread pool via IOCP (Part 3)
    await fs.ReadAsync(buffer, 0, buffer.Length);
}

// (C) .NET 6 and later. The straightforward style that makes the mode and the offset explicit
using (SafeFileHandle handle = File.OpenHandle(path, FileMode.Open, FileAccess.Read,
                                               options: FileOptions.Asynchronous))
{
    int read = await RandomAccess.ReadAsync(handle, buffer, fileOffset: 0);
}

When auditing existing code, look not only at the ReadAsync / WriteAsync call sites but at the places where FileStream is created. File.OpenRead and short overloads such as new FileStream(path, FileMode.Open) open in synchronous mode. When you build a FileStream from a SafeFileHandle, match the isAsync argument to the handle’s actual mode as well.

7.3. With RandomAccess You Make the Handle and the Offset Explicit

In .NET 6 the internal implementation of FileStream was rewritten wholesale and File.OpenHandle and RandomAccess were added. They are APIs where you deal with a SafeFileHandle directly and pass the read/write position on every call.¹⁶

The style in (C), which makes the mode and fileOffset explicit, corresponds to the division of labor this article described: an asynchronous handle plus a per-operation OVERLAPPED.Offset.

7.4. Even with CancellationToken, Cancellation Remains a Request

On an asynchronous-mode handle, canceling file I/O through a CancellationToken connects internally to CancelIoEx. When a ReadAsync you passed a token to ends in an OperationCanceledException, the mechanism from Section 6 is what is working behind it. The point that immediate abortion is not guaranteed is the same as well.

Fake asynchrony in synchronous mode has no overlapped operation to cancel, so this path is unavailable. Recent .NET runtimes do include a mechanism that attempts to cancel a call that is executing synchronously via CancelSynchronousIo, but the behavior depends on the runtime version and the kind of operation, and reliable abortion is not guaranteed. If you are designing around cancellation, the proper route is to line the handle’s mode up and use the OS’s asynchronous I/O.

For the practical layer above async/await — ConfigureAwait, the relationship with the UI thread, and so on — see “A Practical Decision Table for C# async/await - Task.Run and ConfigureAwait” and “WPF/WinForms async and the UI Thread on One Sheet.” This article explains how the OS moves the reads and writes along underneath all of that.

8. Summary: Check Issuing Through Cleanup as One Continuous Path

When auditing asynchronous I/O, follow the code in this order.

1. At the place that opens, confirm the synchronous or asynchronous mode. For a file on disk, specify the position on every asynchronous operation.
2. At the place that issues, confirm that there is an OVERLAPPED and a buffer dedicated to the operation, and that all three branches from Section 3.3 are handled.
3. At the place that receives completion, confirm that the way you wait matches the method in use — event, APC, IOCP — and that the same result is not handled twice.
4. At the place that shuts down, confirm that you are not freeing anything on the strength of a timeout or a cancellation request alone.

Synchronous I/O and asynchronous I/O are not separate plumbing. The difference is whether you return after waiting for completion or take a path that returns before it. But since synchronous completion happens even in asynchronous mode, keep the design of the issue-result branches separate from the design for responsiveness.¹²

The mode belongs to the handle, the state belongs to each operation, and cleanup happens only after completion has been confirmed. This division of labor is the same whether you work with Win32’s OVERLAPPED directly or use .NET’s FileOptions.Asynchronous. A canceled operation, too, stays under your management until you receive its completion.³⁴¹⁵

Part 3 is next: “I/O Completion Ports (IOCP) and the .NET Thread Pool — The Basement Under async/await.” It takes up why the IOCP from Section 4.4 unifies the queue of completion notifications with control over the number of executing threads, and which thread the rest of an await runs on.

Related Articles

• The Depths of Windows I/O (Part 1) — Every Read and Write Becomes an IRP: The Big Picture of the I/O System
• A Practical Decision Table for C# async/await - Task.Run and ConfigureAwait
• WPF/WinForms async and the UI Thread on One Sheet
• Serial Communication App Pitfalls - Through Reconnection and Log Design
• Why You Should Prefer Event Waits over Sleep(1) on Windows
• A Practical Guide to Getting as Close to Soft Real-Time as Possible on Ordinary Windows
• The Misconception That TCP Lets You Receive in the Same Units You Send — Designing Reception Around a Byte Stream

Related Consulting Areas

KomuraSoft LLC works on the design of Windows business applications and device-communication applications that use asynchronous I/O, and on investigating the causes of defects such as freezing, crashing on cancellation, and thread-pool exhaustion.

• Windows App Development
• Bug Investigation and Root Cause Analysis
• Soft Real-Time Windows App Development
• Contact Us

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What changes when I add FILE_FLAG_OVERLAPPED?

The file object behind the handle is opened in "asynchronous mode." This is a per-handle property decided at the moment you call CreateFile, and you cannot switch between synchronous and asynchronous on a call-by-call basis. On an asynchronous-mode handle you must always pass an OVERLAPPED structure to ReadFile/WriteFile. The system does not maintain a file pointer (current position) for this handle, so for a device that has a position, such as a file on disk, you specify the read/write position every time through the OVERLAPPED structure's Offset (devices with no notion of position, such as serial ports, do not use Offset). An issued operation can return control before it completes, in which case ReadFile returns FALSE and GetLastError reports ERROR_IO_PENDING. You receive completion through a notification mechanism such as an event, an APC, or an I/O completion port.

I issued an asynchronous I/O, so why does it come back completed right away?

Because asynchronous mode means "you do not have to wait for completion," not "you will never be made to wait." Microsoft's documentation lists the typical reasons an operation issued asynchronously nonetheless completes synchronously: a request that can be satisfied immediately (for example, the data is already in the cache), an NTFS-compressed file, an NTFS-encrypted (EFS) file, and a write that extends the length of the file. In these cases ReadFile/WriteFile returns TRUE and the result is already final. Code that uses asynchronous I/O must therefore handle both the case that returns ERROR_IO_PENDING and the case that completes on the spot, and responsiveness is not absolutely guaranteed either. Note that by default a completion notification (an event being signaled, or a packet queued to an I/O completion port) still arrives separately even for an operation that completed synchronously, so it is safest to funnel result handling through the notification path alone.

Can I reuse an OVERLAPPED structure?

You must not share one across several operations at the same time. An OVERLAPPED structure represents the state of one operation currently in flight, and Microsoft's documentation states explicitly that if you issue three I/O operations you need three OVERLAPPED structures, and that reusing one leads to unpredictable results and data corruption. Until an operation completes, keep both the structure and the read/write buffer valid and do not touch their contents. If you reuse a structure after an operation has completed, reinitialize it each time so that leftover data from the previous use has no effect. For the hEvent that holds the event, a manual-reset event is the safe choice.

How do I cancel an I/O operation that is already in progress?

CancelIoEx lets you request cancellation of an outstanding I/O on a given handle regardless of which thread issued it. Pass an OVERLAPPED as the second argument to target one specific operation, or NULL to target every operation on that handle. The older CancelIo can only cancel operations issued by the calling thread itself. What matters is that cancellation is a request, not an immediate guarantee. An operation that was already close to completing can still complete normally, and a canceled operation is reported as completed with ERROR_OPERATION_ABORTED. In either case you must not free the OVERLAPPED structure or the buffer until you have received the completion notification. For a thread that is stuck in synchronous I/O on another thread, there is a dedicated API called CancelSynchronousIo.

What happens if I don't specify FileOptions.Asynchronous (useAsync) on .NET's FileStream?

The handle is opened in synchronous mode, so calling ReadAsync/WriteAsync does not give you genuine asynchronous I/O. Instead a thread-pool thread stands in and performs the synchronous read or write, which is fake asynchrony. The calling thread is not blocked, but another thread is left waiting behind the scenes, and that causes thread-pool exhaustion and reduced scalability. Conversely, opening in asynchronous mode and then calling the synchronous Read/Write incurs the overhead of waiting for completion internally. The principle is to keep the handle's mode and the API you call in step, and from .NET 6 onward File.OpenHandle together with RandomAccess lets you write this out straightforwardly, with both the mode and the offset explicit.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.