Mutual Exclusion Fundamentals for File-Based Integration - Best Practices for File Locks and Atomic Claims

Mutual exclusion in file-based integration becomes an issue in almost every setup involving shared folders, nightly batches, or cross-process hand-offs. The questions people search for most are: is a file lock alone enough, how do I stop multiple workers from picking up the same file, and how do I avoid reading files that are still being written?

In this article, we look at mutual exclusion for file integration through the lenses of file locks, atomic claims, temp -> rename, and idempotency.

Getting the Terminology Straight First

This field is full of terms that get thrown around loosely, and leaving their meaning vague makes everything harder to read. So let us pin down what each one means in this article before going further.

Knowledge map for this article

This article lays out an approach that designs file-based integration over shared folders and nightly batches as a handoff protocol rather than leaving it to OS locks. Before reading, the protocol secures the right to process with an atomic claim, keeps a file that is still being produced under a temp name and publishes it by renaming to the final name once it has been closed, and states completion explicitly with a done file or a manifest instead of guessing from size or timestamp. If a lock file is used, it should take a lease form that carries an ownerId and an expiresAt so that stale locks are handled, and after accounting for the fact that a Windows byte-range lock is ignored on memory-mapped files and that an advisory lock has no effect on a party that ignores the convention, the article concludes that a design which ultimately absorbs everything through idempotency, so that reprocessing the same input does not break anything, is what holds up in practice.

flowchart LR
    accTitle: Locking for file-based integration
    accDescr: Diagram showing how a handoff protocol combines atomic claim, publishing by temp file and rename, done and manifest files, lock files turned into leases, and idempotency, and which antipattern each of them answers in order to prevent failures such as double processing or reading a file while it is still being written
    file_handoff_protocol["File Handoff Protocol"]
    atomic_claim["Atomic Claim"]
    temp_then_rename_publish["Temp-Then-Rename Publish Pattern"]
    done_manifest_file["done/manifest File"]
    lock_file_lease["Lease-Based Lock File"]
    idempotent_processing["Idempotent Processing Design"]
    os_file_lock["OS File Lock"]
    atomic_creation["Atomic Creation (CreateNew / O_CREAT|O_EXCL)"]
    duplicate_processing["Duplicate Processing and Lost Updates"]
    exists_then_create_antipattern["Exists-Then-Create Anti-Pattern"]
    direct_final_write_antipattern["Writing directly to the final filename"]
    partial_write_read["Reading a Partially Written File"]
    size_stability_completion_check["File-Size Stability Completion Check"]
    shared_file_mutual_update_antipattern["Shared-File Mutual Update Antipattern"]
    stale_lock["Stale Lock"]
    byte_range_lock["Byte-Range Lock"]
    heterogeneous_system_integration["Heterogeneous System Integration"]
    advisory_lock["Advisory Lock"]
    cross_volume_rename_fallback["Cross-volume rename fallback (copy + delete)"]
    smb_share_file_handoff["File Handoff over SMB Share"]
    rename_fails_on_open_handle["Rename Failure from Open Handles"]
    file_timestamp_unreliability["Unreliable File Timestamps"]
    periodic_directory_listing["Periodic Directory Enumeration"]
    change_notification_loss["Missed Change Notifications"]
    file_handoff_protocol -->|"uses"| atomic_claim
    file_handoff_protocol -->|"uses"| temp_then_rename_publish
    file_handoff_protocol -->|"uses"| done_manifest_file
    file_handoff_protocol -->|"uses"| lock_file_lease
    file_handoff_protocol -->|"uses"| idempotent_processing
    file_handoff_protocol -.->|"uses"| os_file_lock
    atomic_claim -.->|"uses"| atomic_creation
    atomic_claim -->|"prevents"| duplicate_processing
    exists_then_create_antipattern -->|"may cause"| duplicate_processing
    atomic_claim -->|"recommended for"| exists_then_create_antipattern
    atomic_creation -->|"recommended for"| exists_then_create_antipattern
    direct_final_write_antipattern -->|"may cause"| partial_write_read
    temp_then_rename_publish -->|"prevents"| partial_write_read
    size_stability_completion_check -.->|"may cause"| partial_write_read
    done_manifest_file -->|"recommended for"| size_stability_completion_check
    shared_file_mutual_update_antipattern -->|"may cause"| duplicate_processing
    idempotent_processing -->|"recommended for"| duplicate_processing
    lock_file_lease -->|"recommended for"| stale_lock
    lock_file_lease -->|"requires"| atomic_creation
    os_file_lock -->|"uses"| byte_range_lock
    os_file_lock -.->|"mitigates"| duplicate_processing
    os_file_lock -->|"not recommended for"| heterogeneous_system_integration
    advisory_lock -.->|"not recommended for"| heterogeneous_system_integration
    temp_then_rename_publish -.->|"incompatible with"| cross_volume_rename_fallback
    smb_share_file_handoff -.->|"may cause"| cross_volume_rename_fallback
    smb_share_file_handoff -.->|"may cause"| rename_fails_on_open_handle
    done_manifest_file -->|"recommended for"| file_timestamp_unreliability
    periodic_directory_listing -->|"recommended for"| change_notification_loss
    smb_share_file_handoff -.->|"may cause"| change_notification_loss

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (29 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

Table of Contents

1. The Conclusion First (In One Line)
2. Race Patterns That Occur in File Integration (Diagrams)
   • 2.1. Reading a File Mid-Write
   • 2.2. Multiple Workers Picking Up the Same File Simultaneously
   • 2.3. Everyone Stalls on a Stale Lock
3. Anti-Patterns
   • 3.1. The Two-Step Exists -> Create Check
   • 3.2. Writing Directly to the Final File Name
   • 3.3. Treating a File as Done When Its Size Stops Changing
   • 3.4. Everyone Updating a Shared File
   • 3.5. Believing Lock APIs Are Almighty
4. Best Practices
   • 4.1. Publish via temp -> close -> rename / replace
   • 4.2. Make Completeness Explicit With a done File / Manifest
   • 4.3. The Receiver Takes a Claim Atomically
   • 4.4. If You Rely on Lock Files, Make Them Leases
   • 4.5. Assume Idempotency
5. Pseudocode (Excerpts)
6. A Rough Guide to Choosing
7. Conclusion
8. References

File integration is a field where the “hand-off agreement” breaks more easily than the code itself. Things pass in unit tests, yet occasionally break only in the production shared folder or the nightly batch - and the failure is hard to reproduce. This is entirely common.

The cause is usually not the file I/O APIs themselves, but ambiguity in these three things:

• When is it OK to read?
• Who holds the right to process?
• How do we recover when something fails?

In this article, rather than ending the discussion at OS locks, we organize file-integration mutual exclusion as a hand-off protocol.

The code in this article is published on GitHub as a complete buildable and runnable sample set (a library, a demo that demonstrates claim contention between two workers and lease takeover, and unit tests that reproduce contention, corruption, and stale locks).

file-integration-locking-best-practices-komurasoft-style - komurasoft-blog-samples (GitHub)

1. The Conclusion First (In One Line)

• The most important thing in file integration is to ensure that the moment the final file name becomes visible, the file is already safe to read
• Express generating / published / processing / processed states through file names and directories
• If there are multiple workers, take a claim atomically before reading
• Use lock files and OS locks as aids, and let idempotency catch whatever slips through

In short, the real substance of file integration is not so much mutual exclusion as the design of a hand-off protocol. It is never as simple as calling one lock function and being done.

2. Race Patterns That Occur in File Integration (Diagrams)

2.1. Reading a File Mid-Write

If you start writing directly under the final file name, this is exactly what you get. A JSON file is missing its closing brace, a CSV is short on rows, and a ZIP is simply corrupt.

sequenceDiagram
    participant 送信 as Sender
    participant 共有 as Shared folder
    participant 受信 as Receiver

    送信->>共有: Create orders.csv under its final name
    送信->>共有: Writing rows 1 through 5000
    受信->>共有: Detects orders.csv
    受信->>共有: Starts reading right away
    Note over 受信: Still incomplete
    送信->>共有: Writes the rest
    Note over 受信: Missing rows / parse failure / partial processing

2.2. Multiple Workers Picking Up the Same File Simultaneously

With a “list the directory, open anything unprocessed” flow, two workers can grab the same file. This is how double counting and duplicate sends begin.

sequenceDiagram
    participant W1 as Worker 1
    participant W2 as Worker 2
    participant Dir as incoming

    W1->>Dir: Finds a.csv
    W2->>Dir: Finds a.csv
    W1->>Dir: Starts reading
    W2->>Dir: Starts reading
    Note over W1,W2: The same input is processed twice

2.3. Everyone Stalls on a Stale Lock

A design that just drops a lock file tends to jam up after abnormal termination. If you cannot tell whose lock it is, whether the owner is still alive, or how long it is valid, everyone downstream waits forever.

sequenceDiagram
    participant A as Worker A
    participant Lock as lock file
    participant B as Worker B

    A->>Lock: Creates the lock
    Note over A: Crashes here
    B->>Lock: Checks whether the lock exists
    B->>Lock: Holds off on processing
    B->>Lock: Keeps waiting
    Note over B,Lock: Cannot tell if it is stale - everyone stops

3. Anti-Patterns

3.1. The Two-Step Exists -> Create Check

The problem here is that checking and acquiring are separate operations. Another process can squeeze in between them, so this is not mutual exclusion at all.

sequenceDiagram
    participant A as Process A
    participant B as Process B
    participant FS as File system

    A->>FS: Checks that no lock exists
    B->>FS: Checks that no lock exists
    FS-->>A: None
    FS-->>B: None
    A->>FS: Creates the lock
    B->>FS: Creates the lock
    Note over A,B: Both proceed

The typical bad example looks like this.

if (!File.Exists(lockPath))
{
    File.WriteAllText(lockPath, Environment.ProcessId.ToString());
    ProcessFile();
}

What you need is to make “create if absent” a single operation. In .NET that means the FileMode.CreateNew family; on POSIX systems, atomic creation such as O_CREAT | O_EXCL.

3.2. Writing Directly to the Final File Name

If the receiver’s interpretation is “once that name is visible, it is safe to read,” you have already lost the moment you start writing directly under the final name. The basic rule is: do not equate being visible with being safe to read.

flowchart LR
    A[Final name becomes visible] --> B[Receiver detects it]
    B --> C[Sender is still writing]
    C --> D[Incomplete data gets read]

using var writer = OpenForWrite(finalPath); // finalPath becomes visible here
foreach (var row in rows)
{
    writer.WriteLine(row);
}

This approach invites the breakage in 2.1 all on its own.

3.3. Treating a File as Done When Its Size Stops Changing

This looks convenient but is quite precarious. Copies over the network, sender-side pauses, buffering, and retries all make it wobble routinely.

sequenceDiagram
    participant 送信 as Sender
    participant 共有 as Shared folder
    participant 受信 as Receiver

    送信->>共有: Starts copying data.zip
    送信->>共有: Pauses partway
    受信->>共有: Size unchanged for 10 seconds
    Note over 受信: Misjudges it as complete
    受信->>共有: Starts reading
    送信->>共有: Resumes the copy

if (currentLength == lastLength && stableSeconds >= 10)
{
    return Ready;
}

If you determine completion by guessing, shared folders and large files will trip you up. Completion is far more stable when stated explicitly via a manifest or a done file.

3.4. Everyone Updating a Shared File

A design where everyone reads and updates a single status.csv or counter.json usually ends in “last writer wins.” When file integration starts being used as a makeshift database, this is where it starts to hurt.

sequenceDiagram
    participant A as Batch A
    participant B as Batch B
    participant F as status.csv

    A->>F: Reads v1
    B->>F: Reads v1
    A->>F: Writes v2-A
    B->>F: Writes v2-B
    Note over F: The update from A is lost

There is the escape hatch of going append-only, but its semantics wobble depending on the file system and deployment layout. If shared updates are required, it is better not to strain file integration here.

3.5. Believing Lock APIs Are Almighty

Lock APIs matter, but they only work when every participant plays by the same rules. In heterogeneous system integration, it is safer not to over-trust them.

Additional notes:

• flock on Linux is an advisory lock, so a party that ignores the agreement can simply write anyway
• Windows byte-range locks are ignored when memory-mapped files are involved
• In other words, do not make OS locks alone carry the design of completion notification and ownership

The second point is documented as Windows behavior. In Microsoft Learn’s Locking and Unlocking Byte Ranges in Files, immediately after stating that any access by another process to a locked range always fails - that is, Windows byte-range locks are enforced rather than advisory - comes the caveat that byte-range locks are ignored when a memory-mapped file is used. If the other side touches the same file through CreateFileMapping, your lock is simply bypassed.

To take a range lock in .NET, use FileStream.Lock / Unlock (on Windows).

using var stream = new FileStream(
    path, FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite);

// Lock only the first byte exclusively, as an in-progress token
stream.Lock(0, 1);
try
{
    // Read and write the payload here
}
finally
{
    // Always release it before closing
    stream.Unlock(0, 1);
}

This shape works between applications that operate under the same agreement. But as noted above, it has no effect if the other side goes through a memory map, and there is no guarantee that another system will even look at this token in the first place. That is why the hand-off protocol in section 4 is the real substance.

4. Best Practices

Before the details, here is how each practice maps to the anti-patterns in section 3. If you know you are hitting one of them, you can go straight to the matching subsection.

4.1. Publish via temp -> close -> rename / replace

The classic approach. Keep the file under a temp name while it is being generated, and switch it to the final name only after closing it. The receiver watches only final names.

flowchart LR
    A[Create a unique temp name] --> B[Write the full content to temp]
    B --> C[Flush / close]
    C --> D[Rename / replace to the final name in the same directory]
    D --> E[Receiver watches only final names]

Key points:

• Put temp and final in the same directory - at minimum the same volume / file system
• On Windows / .NET, the File.Replace family is worth considering
• Make it the agreed contract that once the final name is visible, the content is complete

If you put temp on a different drive, the rename degrades into the equivalent of a copy, or Replace fails. This prerequisite is unglamorous but very important.

Over a shared folder (SMB), four more things wobble. Since shared folders are precisely what this article is about, they are worth spelling out separately.

• A rename within the same directory of the same share is executed on the server side. So the property that no intermediate name is ever visible still holds. Cross a share boundary, however - say from \\server\shareA to \\server\shareB - and the two paths count as different volumes, at which point Windows MoveFileEx substitutes a copy plus a delete for the move when MOVEFILE_COPY_ALLOWED is specified. The move is no longer atomic and the intermediate state becomes visible. Keeping temp and final, and incoming and processing, inside the same share matters even more here than it does locally
• A rename fails when someone merely has the file open. Shared folders get touched by parties you have no visibility into: antivirus software, search indexers, clients at other sites. The realistic approach is to treat a failed publish or claim rename as a normal branch rather than an error, and retry after a short wait
• Timestamps are not something you can base a decision on. Microsoft Learn’s File Times states that the only guarantee for file times is that they are correctly reflected once the handle that made the change is closed. The last-write time during a write is not fully updated until every write handle is closed. Granularity depends on the file system as well: FAT records last-write time in 2-second units, and NTFS last-access time can lag by up to an hour. Over SMB the timestamp is stamped by the server clock, so if the client and server clocks disagree, a rule like “process it N minutes after the last update” drifts by exactly that much. This is why completion should be determined by the done file / manifest in 4.2 rather than by time or size
• Change notifications get dropped too. Watching a shared folder is more stable when event notifications are paired with periodic directory listing instead of relied on alone. That topic is covered in A Practical Guide to FileSystemWatcher - Handling Missed and Duplicate Events

4.2. Make Completeness Explicit With a done File / Manifest

Beyond the data itself, explicitly stating “what has been completed” in a separate file stabilizes the receiver. This is especially effective in heterogeneous system integration.

flowchart TD
    A[Generate data.tmp] --> B[Publish as data.csv]
    B --> C[Create data.done / manifest.json]
    C --> D[Receiver detects the done file / manifest]
    D --> E[Verify file name, size, and hash]

Items worth putting in the manifest include:

• Target file name
• Size
• Hash
• Record count
• Integration ID / idempotency key
• Generation timestamp

Order matters too. If you place the done file before publishing the payload, it is not a completion notice - it is an advance warning of trouble.

4.3. The Receiver Takes a Claim Atomically

If multiple workers watch the same incoming, “move it into your own area before reading” is the clearest approach. Only the worker whose rename from incoming to processing/<worker>/ succeeds gets to process the file.

sequenceDiagram
    participant W1 as Worker 1
    participant W2 as Worker 2
    participant IN as incoming
    participant PR as processing

    W1->>IN: Finds a.csv
    W2->>IN: Finds a.csv
    W1->>PR: Renames a.csv
    W2->>PR: Renames a.csv
    Note over W1,W2: Only the one that succeeds first takes ownership

Operationally, separating the directories also makes things easier to trace.

flowchart LR
    T[temp] -->|publish| I[incoming]
    I -->|claim| P[processing]
    P -->|success| A[archive]
    P -->|failure| E[error]

The claim rename, too, must happen on the same file system - that is a prerequisite.

4.4. If You Rely on Lock Files, Make Them Leases

If you use lock files, make them ownership records with an expiration rather than mere empty files. A lock whose owner is unknown will cause an argument sooner or later.

flowchart TD
    L[lock.json] --> A[ownerId]
    L --> B[host]
    L --> C[pid]
    L --> D[acquiredAt]
    L --> E[expiresAt]
    L --> F[heartbeatAt]

Key points:

• Create it atomically
• Use a stopped heartbeat as the basis for judging the lease stale
• As a rule, only the creator deletes it
• Assume releases will sometimes be missed, and decide the recovery procedure in advance

A lock file is, in the end, a token for cooperation. Trying to guarantee full consistency with that single token usually gets rough.

4.5. Assume Idempotency

Mutual exclusion matters, but in real operation you can never fully eliminate the occasional duplicate delivery or the mid-way re-run. In the end, a design that does not break when fed the same input again is what saves you.

flowchart LR
    A[Input + idempotency key] --> B{Already processed?}
    B -- Yes --> C[Treat as success without re-executing]
    B -- No --> D[Execute the processing]
    D --> E[Record in the processed ledger]

For example, give each received file an integration ID and record it in a processed ledger. If results are never double-counted even when exclusion breaks once, operations get considerably easier.

5. Pseudocode (Excerpts)

MakeTempPathSameDirectory and TryClaimBundleByRename below are fictional function names, placed there to show the ordering. Implementations that actually run are in the sample set introduced at the top of this article.

The implementation behind this pseudocode (library, a two-worker claim contention demo, unit tests) - komurasoft-blog-samples (GitHub)

5.1. The Typical Failure Pattern

var lockPath = finalPath + ".lock";

if (!File.Exists(lockPath))
{
    File.WriteAllText(lockPath, "");
    using var writer = OpenForWrite(finalPath); // Writes directly to the final name
    WritePayload(writer);

    File.Delete(lockPath);
}

There are three problems.

• Exists and WriteAllText are separate operations
• finalPath becomes visible while it is still being written
• The lock is left behind on abnormal termination

5.2. An Example in the Right Direction (Roughly Sketched)

var tempPath = MakeTempPathSameDirectory(finalPath);
WritePayload(tempPath);
FlushAndClose(tempPath);

PublishByRenameOrReplace(tempPath, finalPath); // Assumes same FS / same volume
PublishDoneFile(finalPath + ".done", new
{
    FileName = Path.GetFileName(finalPath),
    Size = GetFileSize(finalPath),
    Hash = ComputeHash(finalPath),
    IdempotencyKey = integrationId
});

if (!TryClaimBundleByRename(baseName, incomingDir, processingDir))
{
    return; // Another worker claimed it first
}

var manifest = ReadDoneFile(Path.Combine(processingDir, baseName + ".done"));
VerifyPayload(Path.Combine(processingDir, baseName), manifest);

if (AlreadyProcessed(manifest.IdempotencyKey))
{
    MoveBundle(processingDir, archiveDir, baseName);
    return;
}

Process(Path.Combine(processingDir, baseName));
RecordProcessed(manifest.IdempotencyKey);
MoveBundle(processingDir, archiveDir, baseName);

What matters here is the ordering rather than the implementation details. Keeping “write,” “publish,” “take ownership,” and “record as processed” unmixed makes things much harder to break.

6. A Rough Guide to Choosing

• Single writer / single reader / same host: just temp -> rename already gets you quite far
• Multiple consumers: add the incoming -> processing claim rename
• Heterogeneous systems, NAS, shared folders: safer to go all the way to manifest / done files and idempotency
• Multiple writers updating the same logical state: do not over-stretch file integration - also consider a DB or a queue
• OS locks are effective within a homogeneous set of apps sharing the same assumptions, but they are no substitute for a hand-off protocol

That last item is also a signal to walk away from files altogether. Some problems genuinely become painful when done with files.

7. Conclusion

Mutual exclusion in file integration is not about calling a lock function - it is about defining state transitions. That is the backbone of this article. Express generating / published / processing / processed through names and directories, and avoid the two-step Exists -> Create check, direct writes to the final file name, waiting for size stability, mutual updates of shared files, and over-trusting lock APIs. On top of that, combining temp -> close -> rename / replace, done files / manifests, claim renames, leases, and idempotency prevents most of the ways shared-folder integration breaks.

The trick in file integration is to never equate “can be read” with “may be read.” Just separating those two dramatically reduces the kind of failure that only shows up in the middle of the night.

8. References

• Complete sample code for this article (library, demo, unit tests) - komurasoft-blog-samples (GitHub)
• LockFileEx function (Win32)
• Locking and Unlocking Byte Ranges in Files (Win32)
• Moving and Replacing Files (Win32)
• MoveFileEx function (Win32)
• File Times (Win32)
• FileStream.Lock Method (.NET)
• File.Replace Method (.NET)
• rename — POSIX
• open — POSIX (O_CREAT | O_EXCL)
• flock(2) — Linux manual page
• open(2) — Linux manual page

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

Is a lock API alone enough for mutual exclusion in file-based integration?

Usually not. flock on Linux is an advisory lock, so a party that ignores the agreement can simply write anyway, and Windows byte-range locks are ignored when memory-mapped files are involved. OS locks are effective within a homogeneous set of apps that share the same assumptions, but use them as an aid: the real substance is the hand-off protocol - temp -> rename, done files and manifests, atomic claims, and idempotency.

How do I keep a file from being read while it is still being written?

Publishing via temp -> close -> rename/replace is the classic approach. Keep the file under a temp name while it is being generated, switch it to the final name in the same directory after closing it, and have the receiver watch only final names. This assumes temp and final live in the same directory, or at least on the same volume or file system, and it makes the moment the final name appears the contract that the content is complete.

How do I stop multiple workers from processing the same file at the same time?

Take the claim atomically before reading. Concretely, only the worker whose rename from incoming into processing/<worker>/ succeeds gets to process the file. A two-step Exists -> Create check splits checking from securing, so another process can slip into the gap and it provides no mutual exclusion at all. When you need atomic creation, use the FileMode.CreateNew family in .NET or O_CREAT | O_EXCL on POSIX.

Is there anything to watch out for when using lock files?

Make the lock file an expiring lease that carries ownership information - ownerId, host, pid, acquiredAt, expiresAt, heartbeatAt - rather than a bare empty file. Create it atomically, use a stopped heartbeat as the basis for judging the lease stale, let only the creator delete it as a rule, and decide the recovery procedure on the assumption that releases will sometimes be missed. Rather than trying to guarantee full consistency with a single lock file, a design that ultimately catches the remainder with idempotency holds up far better in practice.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.