How to Use FileSystemWatcher Safely - Lost Events, Duplicate Notifications, and the Traps Around Completion Detection

1. Short version

• FileSystemWatcher events are not “completion notifications” - they are “hints that something changed”
• Created/Changed/Renamed can be duplicated, arrive out of order, or be lost on overflow
• don’t do heavy work in the event handler - just enqueue a rescan request and you’ll be much more stable
• don’t infer completion - declare it explicitly with temp -> close -> rename or a done/manifest file
• if you have multiple workers, claim files atomically before reading
• tuning InternalBufferSize is only a band-aid - what really saves you is full rescan + idempotency

2. Common misconceptions

2.1 Treating Created as a completion signal

With copies and transfers, Created fires the moment the file appears, and several Changed events follow as the bytes are written.

sender: creates orders.csv
FileSystemWatcher: Created -> receiver starts reading (still mid-copy!)
sender: writes the rest -> Changed -> Changed
result: missing rows / broken JSON / corrupt ZIP

Created only means “a name became visible” - it does not mean “safe to read.”

2.2 Trusting the count and order of Changed events

• Changed is not guaranteed to fire exactly once - it can fire many times
• antivirus and indexers can trigger Changed too
• a Changed can even arrive after a Renamed

2.3 Losing changes to internal buffer overflow

When a lot of changes happen in a short window, the internal buffer fills up and individual notifications get dropped. If the Error event fires, don’t trust the per-event stream - do a full rescan.

3. Anti-patterns

3.1 Doing the work directly in the event handler

// bad
watcher.Created += (_, e) =>
{
    using var stream = File.OpenRead(e.FullPath);
    Import(stream); // might still be copying
};

What’s wrong:

• the file may not be fully written when Created fires
• there’s no recovery for failures or overflow
• the right pattern is: enqueue a rescan request and return immediately

3.2 Trying to reconstruct state from the event stream

A design that says “add on Created, update on Changed, drop on Deleted” falls apart under duplicates, splits, and overflow. Looking at what’s actually on disk each time is far more robust.

3.3 “It’s done if Changed has stopped”

A rule like “if the file size hasn’t changed for 10 seconds, it’s done” breaks in all of these cases:

• a long copy that pauses mid-way
• a sender that saves in multiple stages
• notifications delayed over a network share
• don’t infer completion - declare it

3.4 Logging Error and ignoring it

The Error event tells you the buffer overflowed or the watcher itself is in trouble. At minimum:

• request a full rescan
• recreate the watcher if monitoring looks unhealthy
• design every step to be idempotent so you can safely reprocess

4. Best practices

4.1 Collapse all notifications into “please rescan”

Fold every event (Created/Changed/Deleted/Renamed/Error) into a single signal that just means “go look.”

void OnAnyChange(object? sender, FileSystemEventArgs e)
{
    RequestScan(full: false);
}

void OnError(object? sender, ErrorEventArgs e)
{
    Log(e.GetException());
    RequestScan(full: true);  // full rescan on error
}

4.2 Make the sender declare completion

The textbook pattern:

1. write everything to data.tmp (a temp name)
2. flush and close
3. rename / replace to data.csv on the same filesystem
4. optionally drop a data.done / manifest.json last

FileSystemWatcher is not a tool for deciding when something is done - it’s a tool for noticing quickly when someone else has declared it done.

4.3 The receiver claims atomically

With multiple workers, take ownership by renaming incoming -> processing/<worker>/.

incoming/
  order-123/
    payload.csv
    manifest.json

A single rename of the bundle directory transfers ownership atomically.

4.4 Always full rescan on startup / overflow / reconnect

Make sure a full rescan runs at all of these moments:

• application startup
• when an Error event arrives
• right after recreating the watcher
• on a regular schedule (as a safety net)

The mental model is “the watcher is a hint about diffs - the rescan is what restores consistency.”

4.5 Build on top of idempotency

Accept that you will look at the same item more than once - that’s a feature, not a bug.

• put an IdempotencyKey in the manifest
• skip side effects if the key has already been processed
• aim for at-least-once + idempotency, not exactly-once

5. A correct implementation (excerpt)

private readonly SemaphoreSlim _scanSignal = new(0, int.MaxValue);
private int _scanRequested = 0;
private int _fullRescanRequested = 0;

void RequestScan(bool full)
{
    if (full) Interlocked.Exchange(ref _fullRescanRequested, 1);
    if (Interlocked.Exchange(ref _scanRequested, 1) == 0)
        _scanSignal.Release();
}

async Task ScannerLoopAsync(CancellationToken ct)
{
    RequestScan(full: true); // initial scan on startup
    while (!ct.IsCancellationRequested)
    {
        await _scanSignal.WaitAsync(ct);
        await Task.Delay(200, ct); // coalesce notification bursts

        Interlocked.Exchange(ref _scanRequested, 0);
        bool full = Interlocked.Exchange(ref _fullRescanRequested, 0) == 1;

        foreach (var bundle in EnumerateReadyBundles(incomingDir, full))
        {
            if (!TryClaimByRename(bundle.Path, processingPath)) continue;
            var manifest = ReadManifest(manifestPath);
            if (AlreadyProcessed(manifest.IdempotencyKey))
            {
                MoveToArchive(processingPath, archiveDir);
                continue;
            }
            ProcessBundle(processingPath);
            RecordProcessed(manifest.IdempotencyKey);
            MoveToArchive(processingPath, archiveDir);
        }

        if (Volatile.Read(ref _scanRequested) == 1)
            _scanSignal.Release(); // don't drop notifications that arrived mid-scan
    }
}

6. Wrap-up

• FileSystemWatcher is not a substitute for a completion signal
• the truth is not the event stream - it’s what’s actually on disk right now
• declare completion explicitly with temp -> close -> rename or done/manifest
• take ownership atomically with a claim rename
• collapse notifications into rescan requests, and full-rescan on startup / overflow / reconnect
• let idempotency absorb the duplicates and the re-runs