Investigating Long-Run Crashes of an Industrial Camera App - The Handle Leak (Part 1)

When a Windows app suddenly crashes after running for a long time, the first instinct is very often to suspect a memory leak. In reality, however, it is not uncommon for a handle leak to be the main culprit, finally surfacing weeks later as a secondary failure.

This article presents a case where we investigated a Windows app controlling an industrial camera that suddenly crashed after roughly one month of continuous operation. As we narrowed things down, the cause turned out to be a handle leak occurring on the failure path around camera reconnection.

In this first part, we cover what a handle leak is, how we isolated this incident, and what logs you should keep to prevent recurrence. In the second part, Building a Windows Failure-Path Test Foundation with Application Verifier, we discuss building a failure-path test foundation.

Proper names and some log fields have been redacted, but the way of thinking is broadly shared across Windows equipment control apps in general.

Table of Contents

1. The Conclusion First (In One Line)
2. What Is a Handle Leak?
   • 2.1. What “Handle” Means Here
   • 2.2. Why It Tends to Surface Only After Long-Running Operation
   • 2.3. How It Differs from a Memory Leak
3. Case Study: An Industrial Camera Control App That Suddenly Crashes After One Month
   • 3.1. The Symptoms
   • 3.2. The Metrics We Looked at First
   • 3.3. The Leak That Was the Root Cause
4. How We Isolated It
   • 4.1. Compress Time Instead of Waiting for a Month-Scale Repro
   • 4.2. Read the Slope of Handle Count
   • 4.3. Check the Pairing of create/open and close/dispose
   • 4.4. For Handle Leaks, Find Where It Leaked, Not Where It Crashed
5. The Logs You Need to Prevent Recurrence
   • 5.1. The Minimum Set to Keep First
   • 5.2. The Logs We Actually Strengthened
   • 5.3. At What Granularity to Collect
6. A Rough Decision Guide
7. Summary
8. References

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (18 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

1. The Conclusion First (In One Line)

• In a control app that only crashes after long-running operation, always look at Handle Count, not just Private Bytes
• Handle leaks tend to hide not in the normal path but in the timeout / reconnect / partial-failure / early-return paths
• The line that actually crashes is often the place that could no longer create a new handle later, not the place that leaked it
• The logs you need first are: the operation/session context, the process’s handle count, the open/close pairing of resources, and Win32 / HRESULT / SDK errors
• Rather than waiting for a month-scale repro, it is faster to run the connect-disconnect-reconnect-failure paths thousands of times in a short loop
• Application Verifier, covered in Part 2, is quite effective, but the foundation is being able to trace lifetime breakdowns with your own logs first

In short, the first thing to do on a case like this is not to stare at the fact that “it crashed after a long period,” but to get the growth of resources and the failure paths into an observable form.

By the time a handle leak is found, it usually already wears the face of a secondary failure. So if you only look at the exception at the moment of the crash, you tend to walk off in quite the wrong direction.

flowchart TB
    accTitle: What to tackle first
    accDescr: Shows that looking only at the exception at the moment of the crash leads off in the wrong direction, while first making resource growth and failure paths observable leads to the handle leak that wears the face of a secondary failure.
    crash["Look only at the exception at the crash"] -.-> wrong["Walk off in the wrong direction"]
    obs["Observe resource growth and failure paths"] --> right["Reach the real source of the leak"]

Figure 1: Before staring at the fact that it crashed, get the growth of resources and the failure paths into an observable form.

2. What Is a Handle Leak?

2.1. What “Handle” Means Here

A handle here is the identifier through which a Windows process references OS resources. Examples of what falls under this include:

What tends to become a problem in control apps in particular is the pattern of “forgetting to close a resource that was opened temporarily for some operation, on a partial-failure path”.

The typical flow looks like this.

• Create one event on every reconnect
• Callback registration or acquisition start fails partway through
• The success path closes it, but the failure path does not
• Routine short tests only exercise the success path, so it goes unnoticed

This type slips through quite routinely, both in code review and in production.

flowchart TB
    accTitle: The typical pattern that leaks only on the failure path
    accDescr: Shows that an event is created on every reconnect and closed when callback registration and acquisition start succeed, but is left unclosed on a partial-failure path, and that short tests only exercise the success path so the leak goes unnoticed.
    ev["Create an event on every reconnect"] --> q{"Did registration and start succeed"}
    q -->|"Success"| close["Closed on the success path"]
    q -->|"Partial failure"| leak["Not closed on the failure path"]
    leak -.-> unseen["Short tests run only the success path and miss it"]

Figure 2: Forgetting to close a temporarily opened resource on a partial-failure path. Especially common in control apps.

2.2. Why It Tends to Surface Only After Long-Running Operation

A handle leak does not necessarily break things spectacularly in one shot. What is actually nastier is a small-slope leak, where one failure leaks just one handle.

flowchart LR
    A[Normal operation] --> B[Occasional timeout / reconnect]
    B --> C[Failure path creates an event handle]
    C --> D[CloseHandle is never called]
    D --> E[Handle Count creeps up slightly]
    E --> F[Repeats hundreds of times]
    F --> G[CreateEvent / SDK open fails]
    G --> H[Crash / stall somewhere else]

Figure 3: One small leak per occurrence piles up hundreds of times at the boundary conditions of 24/7 operation and finally surfaces.

If one reconnect leaks just one handle, nothing happens within minutes. But in an equipment control app running 24/7, boundary conditions like timeouts, re-initializations, and disconnect recovery occur over and over. The result is the odd presentation of a problem that only surfaces weeks later.

What matters here is that the handle leak itself is not necessarily the crashing line. The common modes of breakage are these.

• An API that creates a new event / file / thread fails
• The SDK cannot create a resource it needs internally and returns only a generic failure code
• Error handling after the failure is thin, and the app dereferences a null / invalid handle and crashes
• Timeouts increase, and as a result a watchdog or upstream controller kills the process

In other words, the crash site is the “last victim,” not necessarily the “original culprit.”

flowchart TB
    accTitle: The crash site is the last victim
    accDescr: Shows that when handles keep leaking somewhere, an API that creates a new resource eventually fails and the problem surfaces as a crash or stall somewhere else where error handling is thin, so the crashing line is the last victim rather than the original culprit.
    leak2["Handles keep leaking somewhere"] --> fail["An API that creates a new resource fails"]
    fail --> vict["Crash or stall somewhere else"]
    vict -.-> note["The crashing line is only the last victim"]

Figure 4: The leak itself is not necessarily the crashing line. The breakage is almost always a secondary failure.

A naive question comes up here: why does a mere few thousand handles bring the app down?

Going by the raw numbers, the limit looks far away. For kernel object handles, the theoretical ceiling is 2^24 (about 16.77 million) per process. Handles live in the paged pool, though, so the number you can actually create depends on the memory available, and on 32bit Windows it lands far below the theoretical figure.

In short, crashing because you reached the theoretical limit is the minority case. What actually bites first is usually one of the following.

So the reading “we still have headroom to the limit, so we are fine” does not hold up. Judge by whether what should come back has come back, not by whether the limit is within reach. Once the slope stands up, it is safer to treat the app as already abnormal.

flowchart TB
    accTitle: What you hit before the theoretical limit
    accDescr: Shows that crashing on the theoretical kernel handle limit is the minority case and that in practice the GDI object limit, the SDK internal bookkeeping table, or the address space of a 32bit process is hit first, so the judgment should be whether what should come back has come back.
    limit["The kernel theoretical limit is about 16.77 million"] -.-> rare["Reaching it is the minority case"]
    rare -.-> first["What you actually hit first"]
    first --> g1["The GDI limit"]
    first --> g2["The SDK internal bookkeeping table"]
    first --> g3["The 32bit address space"]
    g1 --> see["Judge by whether it comes back or not"]
    g2 --> see
    g3 --> see

Figure 5: Headroom to the limit is no reassurance. Once the slope stands up, treat it as abnormal.

2.3. How It Differs from a Memory Leak

For defects after long-running operation, the first suspicion is a memory leak. That instinct is natural, of course, but handle leaks are sometimes faster to find when viewed along a different axis.

So when isolating long-run issues, looking only at memory is like driving with one eye closed. At minimum, watching Handle Count and Thread Count together makes things considerably easier to sort out.

flowchart TB
    accTitle: Metrics to watch together when isolating long-run issues
    accDescr: Shows that memory leaks and handle leaks are read from different metrics, so watching Handle Count and Thread Count alongside memory metrics such as Private Bytes avoids the state of driving with one eye closed.
    watch["Isolating long-run issues"] --> m1["Memory metrics (Private Bytes and so on)"]
    watch --> m2["Handle Count"]
    watch --> m3["Thread Count"]
    m2 -.-> hint["If it grows and never comes back, it is a handle leak"]

Figure 6: Watching memory alone is driving with one eye closed. Track the handle and thread counts on the same screen.

3. Case Study: An Industrial Camera Control App That Suddenly Crashes After One Month

3.1. The Symptoms

The incident was simple.

• A Windows app controlling an industrial camera runs 24/7
• It runs fine normally
• After roughly one month, one day the app suddenly crashes
• After a restart, it runs fine again for a while

The first difficulty is that it takes a long time to crash. Waiting one month per reproduction attempt is brutal as an investigation.

What made it even nastier was that the crash site was not exactly the same each time. Sometimes it was right after a reconnect started, sometimes at acquisition start, sometimes after a failed SDK call.

With that presentation, at first you can suspect any of the following.

• Instability on the camera SDK side
• Transient failures caused by communication or device disconnects
• A memory leak
• A race around threading
• An initialization failure not showing up in the logs

In other words, we were in a state of too many “vaguely suspicious” candidates.

flowchart TB
    accTitle: The two things that made this case hard to investigate
    accDescr: Shows that the app crashed suddenly after about a month of continuous operation so each reproduction took a month, and that the crash site was not exactly the same every time, leaving too many suspects such as the SDK, communication, and memory.
    sym["Crashes suddenly after about one month"] --> hard1["Each reproduction takes a month"]
    sym --> hard2["The crash site differs slightly each time"]
    hard2 -.-> many["Too many plausible suspects"]

Figure 7: When it takes a long time to crash and the crash site moves around, guesswork gets you nowhere.

3.2. The Metrics We Looked at First

So the first thing we did was look at how the process’s resources as a whole were growing. In this case, the observed trends were roughly as follows.

At this point, our focus had narrowed considerably. It was more natural to read the situation not as “it crashes after one month,” but “something is leaking a little at a time along the way, and as a result it crashes after one month.”

flowchart TB
    accTitle: The reading the first metrics narrowed us down to
    accDescr: Shows that Handle Count alone grew and never came back, the Private Bytes slope was weak, Thread Count was flat, and the crash site differed each time, which narrowed the reading to a slow leak that brings the app down after a month.
    o1["Handle Count grows and never comes back"] --> narrow["The focus narrows"]
    o2["The Private Bytes slope is weak"] --> narrow
    o3["Thread Count is essentially flat"] --> narrow
    narrow --> view["Leaking a little at a time, crashing after a month"]

Figure 8: Line up the shapes of the four metrics and you see that it has been leaking all along, not that it crashes after one month.

3.3. The Leak That Was the Root Cause

The ultimate cause was a missed close of an event handle created on the initialization-failure path during camera reconnection.

Simplified, the flow looks like this.

sequenceDiagram
    participant App as Control app
    participant OS as Windows
    participant SDK as Camera SDK

    App->>OS: CreateEvent
    App->>SDK: Register callback
    SDK-->>App: Partial failure / timeout
    Note over App: Returns on the failure path
    Note over App: CloseHandle is never called

    loop Repeated reconnects
        App->>OS: Handle Count creeps up
    end

    App->>OS: Next CreateEvent / Open
    OS-->>App: Failure
    App-->>App: Crashes as a secondary failure

Figure 9: The root cause was a missed close of an event handle on the reconnect failure path. It piles up until something else falls over.

As a code sketch, the leak looks like this.

handle = CreateEvent(...)

if (!RegisterCallback(handle))
{
    return Error;   // CloseHandle(handle) is missing
}

if (!StartAcquisition())
{
    return Error;   // close is missing here too
}

...
CloseHandle(handle)

The reason this slips past short tests is also quite easy to see.

• A normal startup -> normal shutdown does close it
• Failures only happen partway through a reconnect
• There is no test that hammers that failure path
• In production, it accumulates a little at a time over weeks

In other words, the structure was: “invisible if you only watch the normal path, but it leaks routinely on the failure paths.”

The fix is not flashy.

• Bring the responsibilities of create/open and close/dispose closer together
• Move release into finally / destructors / a session object so it always happens even on partial failure
• Make ownership explicit around callback registration and acquisition start
• Express “who closes it” through the code’s responsibilities, not comments

flowchart TB
    accTitle: The core of the fix
    accDescr: Shows a fix that brings the create and close responsibilities closer together, moves release into finally, a destructor, or the session object so it always happens even on partial failure, and expresses who closes the handle as a code responsibility rather than a comment.
    pol["The fix"] --> p1["Bring create and close responsibilities together"]
    pol --> p2["Move release into finally or a destructor"]
    pol --> p3["Express ownership as a code responsibility"]
    p3 -.-> nc["Do not rely on a comment-based convention"]

Figure 10: Not a flashy fix. Embed the resource lifetime into the structure of the code itself.

Prose alone is hard to follow, so here is the same operation rewritten.

In C++, define one small RAII type that owns the handle, and keep raw HANDLE values out of the function body.

// C++17 / Windows
#include <windows.h>
#include <utility>

class UniqueHandle
{
public:
    UniqueHandle() noexcept = default;
    explicit UniqueHandle(HANDLE h) noexcept : h_(h) {}

    UniqueHandle(const UniqueHandle&) = delete;
    UniqueHandle& operator=(const UniqueHandle&) = delete;

    UniqueHandle(UniqueHandle&& other) noexcept
        : h_(std::exchange(other.h_, nullptr)) {}

    UniqueHandle& operator=(UniqueHandle&& other) noexcept
    {
        if (this != &other)
        {
            reset(std::exchange(other.h_, nullptr));
        }
        return *this;
    }

    ~UniqueHandle() { reset(); }

    HANDLE get() const noexcept { return h_; }
    explicit operator bool() const noexcept { return h_ != nullptr; }

    void reset(HANDLE h = nullptr) noexcept
    {
        if (h_ != nullptr)
        {
            ::CloseHandle(h_);
        }
        h_ = h;
    }

private:
    HANDLE h_ = nullptr;
};

With this in place, you no longer have to add a CloseHandle to every failure path.

// Member of CameraSession: UniqueHandle frameReady_;
bool CameraSession::Reconnect()
{
    UniqueHandle frameReady{ ::CreateEventW(nullptr, TRUE, FALSE, nullptr) };
    if (!frameReady)
    {
        return false;   // Creation itself failed. Nothing to close
    }

    if (!RegisterCallback(frameReady.get()))
    {
        return false;   // Returning here is fine: the destructor closes it
    }

    if (!StartAcquisition())
    {
        // On a failure after registration succeeded, unregister before closing.
        // If we leave without unregistering, the destructor calls CloseHandle
        // while the SDK still holds the handle we passed in. On the next frame
        // it signals an already-released handle value, and if that value has
        // been reused for another resource, the symptom surfaces as an
        // unrelated event firing on its own.
        UnregisterCallback();
        return false;
    }

    // Transfer ownership to the session only on success
    frameReady_ = std::move(frameReady);
    return true;
}

In C#, a single using often will not do the job, so track whether ownership was handed over in a flag and dispose in finally only when it was not. Writing a plain using var would dispose the object even when the operation succeeded.

// C# / .NET 8
// Field of CameraSession: private ManualResetEvent? _frameReady;
public bool Reconnect()
{
    var frameReady = new ManualResetEvent(false);
    var handedOver = false;
    var registered = false;

    try
    {
        if (!RegisterCallback(frameReady))
        {
            return false;
        }

        registered = true;

        if (!StartAcquisition())
        {
            return false;
        }

        _frameReady?.Dispose();
        _frameReady = frameReady;
        handedOver = true;
        return true;
    }
    finally
    {
        if (!handedOver)
        {
            // Detach the reference the outside world holds before disposing.
            // The SDK keeps the handle passed in at registration time,
            // so reversing the order lets it hit an already-released handle
            if (registered)
            {
                UnregisterCallback();
            }

            frameReady.Dispose();
        }
    }
}

Both do the same thing. The structure guarantees that no matter where the function returns, any resource whose owner is not settled is always disposed of. Instead of a human writing “close it if it fails” every single time, the type and the finally block take that over.

flowchart TB
    accTitle: Ownership transfer decides who releases the resource
    accDescr: Shows a structure where, no matter where the function returns, the session takes over release once ownership has been handed to it, the type or the finally block always disposes of the resource when ownership has not been handed over, and the SDK registration is removed before disposal.
    exit["No matter where the function returns"] --> q2{"Was ownership handed over"}
    q2 -->|"Handed over"| keep["The session handles release from here on"]
    q2 -->|"Not handed over"| drop["The type or finally always disposes of it"]
    drop -.-> unreg["Unregister before disposing"]

Figure 11: Instead of writing close it on failure every time, let the destination of ownership decide who releases the resource.

This is not so much a special technique as housekeeping that embeds resource lifetimes into the code.

4. How We Isolated It

From this chapter on, the English investigation vocabulary appears as is. Here is a short glossary first.

4.1. Compress Time Instead of Waiting for a Month-Scale Repro

In this kind of investigation, waiting a month per attempt is a bad approach. What you should do is drive the suspicious paths over and over in a short time.

In this case, we compressed the repro by running a loop like this.

flowchart LR
    A[Start] --> B[Open camera]
    B --> C[Start acquisition]
    C --> D[Simulated timeout / disconnect]
    D --> E[Reconnect]
    E --> F[Resume acquisition]
    F --> G{Repeat N times}
    G -- Yes --> D
    G -- No --> H[Check the deltas at the end]

Figure 12: Instead of waiting for a month-scale repro, drive only the open, disconnect, and reconnect boundaries thousands of times in a short loop.

The point is to spend your time on the lifetime operations at the boundaries, not on the routine “frames are coming in” periods.

Concretely effective scenarios look like these.

• Run open -> start -> stop -> close in large volumes
• Deliberately trigger timeouts and cycle through reconnects
• Force a failure right after callback registration
• Inject disconnect aborts, reconnect aborts, and shutdown races

You do not need to perfectly reproduce a month of real operation. On the contrary, stepping on the suspected lifetime edge thousands of times gets you much closer to the cause.

4.2. Read the Slope of Handle Count

Before that, a word on where you actually look at Handle Count. Without knowing that, nothing in this section is actionable.

The breakdown by type, and how to track the growth of unnamed events, are written up as step-by-step procedures in Process Explorer / Handle / VMMap in Practice.

For long-run investigations, the real workhorse is the last row: the app emitting the numbers itself. Having a person watch Task Manager does not hold up around the clock.

In a handle leak investigation, looking only at absolute values can be confusing. What matters is whether the count comes back down after operations that should return it, and how many handles you gain per how many operations.

Roughly the following order works well.

1. Establish a baseline after warm-up
2. Record Handle Count after each reconnect / start-stop / close
3. Look at the delta per cycle
4. Also look at the slope aggregated over several cycles

For example, a view like this.

leakSlope =
    (currentHandleCount - baselineHandleCount)
    / reconnectCount

Whether an absolute value of 2000 is high or low varies by app. But if it is +1 per reconnect and never comes back, that is quite suspicious.

Here is a rough guide to what the healthy case should look like. The numbers themselves depend on the app, so judge by the shape.

• Right after startup the count climbs. Do not read anything into this stretch
• Once warm-up finishes, the count should rise and fall with the operations while moving in and out of a fixed range
• After one cycle of open -> start -> stop -> close, a healthy app returns to roughly the same value as before the cycle
• If you run 100 cycles and the difference from the baseline stays within a few handles, the app is healthy as far as this goes
• Conversely, if it climbs cleanly in proportion to the cycle count, it leaks by exactly that slope on every cycle

What you read is not “high or low” but whether it comes back. Get that backwards and you will burn time suspecting a healthy app.

flowchart TB
    accTitle: How to read the slope of Handle Count
    accDescr: Shows a reading where a baseline is established after warm-up, the delta per cycle is examined, a return to the starting value after a cycle counts as healthy, and a clean climb proportional to the cycle count means a leak of exactly that slope on every cycle.
    base["Establish a baseline after warm-up"] --> cyc["Look at the delta per cycle"]
    cyc --> q3{"Does it return to the starting value after a cycle"}
    q3 -->|"It returns"| ok["Healthy as far as this goes"]
    q3 -->|"Climbs proportionally"| ng["Leaking by that slope every cycle"]

Figure 13: Judge by the shape of whether it comes back, not by whether the absolute value is high or low.

The trick here is to not watch Handle Count alone, but to record at least the following alongside it.

• Handle Count
• Private Bytes
• Thread Count
• ReconnectCount
• Which phase you are currently in

With this, you can tell quite quickly whether “memory is growing,” “threads are growing,” or “resources are not coming back on every reconnect.”

4.3. Check the Pairing of create/open and close/dispose

Even once you know the process-wide Handle Count is suspicious, that alone does not get you to the leak site. What you need next is logs that show resource lifecycles as pairs.

As an image, structured logs like these.

CameraSession session=421 cameraId=CAM01 phase=ReconnectStart reason=FrameTimeout handleCount=1824 privateBytesMB=418

CameraResource session=421 resourceId=evt-884 kind=Event name=FrameReady action=Create osHandle=0x00000ABC handleCount=1825

CameraResource session=421 resourceId=evt-884 kind=Event name=FrameReady action=Close osHandle=0x00000ABC handleCount=1824

What matters here is to not rely on osHandle alone. Windows handle values can be reused later, so in the logs it is easier to trace if you carry at least the following.

• sessionId
• resourceId
• kind
• action(Create/Open/Register/Close/Dispose/Unregister)
• osHandle
• phase

With this in place, it becomes much easier to spot the lopsided flow where a Create exists but no Close.

flowchart TB
    accTitle: Logs that trace resource lifecycles in pairs
    accDescr: Shows that recording Create and Close as a pair and linking them with sessionId and resourceId reveals the lopsided flow where a Create exists but no Close, and that osHandle alone is not enough because handle values get reused.
    lg["Record Create and Close as a pair"] --> ids["Link them with sessionId and resourceId"]
    ids --> find["Find Creates with no matching Close"]
    lg -.-> reuse["osHandle gets reused, so it cannot stand alone"]

Figure 14: To get from a process-wide number down to the leak site, you need logs that pair up resource lifecycles.

4.4. For Handle Leaks, Find Where It Leaked, Not Where It Crashed

This point is quite important.

A handle leak often presents like this.

• The crashing line: CreateEvent fails
• The real leak: CloseHandle had been missing on a failure path since days earlier

In other words, the API that finally fell over is the exit of the damage, not necessarily the entrance of the cause.

So the investigation order should be:

1. Look at which resource keeps growing
2. Look at which operation boundary it fails to come back at
3. Find where the pairing of create/open and close/dispose is broken
4. Read the crash site last

In this order, you are far less likely to get lost.

flowchart TB
    accTitle: The investigation order that leads back to where it leaked
    accDescr: Shows that looking at which resource keeps growing, then at which operation boundary it fails to come back at, then finding where the create and close pairing is broken, and reading the crash site last, makes it far less likely that you get lost.
    s1["Look at which resource keeps growing"] --> s2["Look at the operation boundary it never returns from"]
    s2 --> s3["Find where the create and close pairing is broken"]
    s3 --> s4["Read the crash site last"]

Figure 15: The crash site is only the exit. Trace back from the entrance, the place that leaked.

5. The Logs You Need to Prevent Recurrence

5.1. The Minimum Set to Keep First

What worked in this investigation was not simply increasing log volume. It was methodically adding “information that lets you reach the cause later.”

At minimum, you want to keep the following.

We are not claiming this is sufficient. But without at least this, you easily end up with logs that record nothing more than the fact that “it crashed.”

5.2. The Logs We Actually Strengthened

In this case, we strengthened the logs in the following directions.

1. Periodic heartbeat
   • Emit Handle Count / Private Bytes / Thread Count / ReconnectCount every 1-5 minutes
2. Boundary logs per camera session
   • OpenStart
   • CallbackRegistered
   • AcquisitionStart
   • TimeoutDetected
   • ReconnectStart
   • ReconnectDone
   • CloseStart
   • CloseDone
3. Resource lifecycle logs
   • Create/Open/Register and Close/Dispose/Unregister for events / threads / files / timers / SDK registration tokens
4. Error normalization
   • Do not stop at the exception message; emit win32Error, HRESULT, sdkError, and phase together

What is important is to not change the shape of the logs between success and failure. If failures get a different format, aggregation later becomes painful.

flowchart TB
    accTitle: The four log tracks we strengthened
    accDescr: Shows that combining a periodic heartbeat, boundary logs per camera session, resource lifecycle logs, and error normalization, while keeping the same shape for success and failure, produces logs that let you reach the cause later.
    hb["Periodic heartbeat (resource values)"] --> trace["Logs that reach the cause"]
    bd["Session boundary logs"] --> trace
    rl["Resource lifecycle logs"] --> trace
    er["Error normalization"] --> trace
    trace -.-> same["Do not change the shape between success and failure"]

Figure 16: Do not increase log volume; line up the four tracks you can cross-reference later.

5.3. At What Granularity to Collect

A common trap here is “just dump everything at INFO.” But if you do that, you end up facing a wall of logs when you read them later. That is quite painful.

In terms of granularity, roughly the following split is realistic.

• Periodic monitoring
  • Handle Count, Private Bytes, Thread Count, ReconnectCount
• Operation boundaries
  • Session start / done / fail
• Resource boundaries
  • create/open/register and close/dispose/unregister
• Failure details
  • Error codes, stacks, dump capture triggers

Detailed per-frame logging is usually unnecessary. For long-run defects, logs that let you read “which responsibility opened it, and which responsibility closed it” are far more effective.

flowchart TB
    accTitle: How to split log granularity
    accDescr: Shows a split where periodic monitoring covers resource counts, operation boundaries cover session start and end, resource boundaries cover create and close pairs, and detail goes deep only on failures, avoiding the wall of logs that comes from dumping everything at INFO.
    lv["Log granularity"] --> l1["Periodic monitoring: counts"]
    lv --> l2["Operation and resource boundaries"]
    lv --> l3["Deep detail only on failures"]
    all["Dump everything at INFO"] -.-> wall["An unreadable wall of logs"]

Figure 17: Rather than per-frame detail, aim for a granularity where you can read who opened it and who closed it.

6. A Rough Decision Guide

• Crashes only after days to weeks
  • First add a heartbeat for Handle Count / Private Bytes / Thread Count
• There are retries / reconnects / shutdowns
  • Build a harness first that hammers just those boundaries in volume
• Heavy use of native SDKs / P/Invoke / Win32
  • Applying Application Verifier (Part 2) is well worth it
• A GUI lives in the same process
  • In addition to Handle Count, also watch GDI Objects / USER Objects
• The exception at the moment of the crash tells you nothing
  • It is faster to first put operation / session / resource lifecycle structured logs in order

That last item is quite important. In bug investigation, what decides the outcome is often not the analysis technique itself, but whether things are in an observable form.

7. Summary

For an app that only crashes after long-running operation, look at Handle Count, not just memory. Handle leaks tend to hide in the failure paths of abnormal flows rather than the normal path, and the crash site is usually the exit of a secondary failure, not the place that leaked. When it comes to reading the symptoms, it ultimately comes down to these three points.

For prevention, bring the responsibilities of create/open and close/dispose closer together, keep logs that carry context per session / operation, and record both process resources and resource lifecycles. In testing, instead of waiting for a month-scale repro, run timeout / reconnect / shutdown in short loops, and make “traceable when it breaks” - not just “does not break” - the acceptance criterion. What worked in this case was this combination. In Part 2, we use Application Verifier to surface hard-to-trigger failure modes such as memory exhaustion and handle anomalies ahead of time.

In control apps, the normal path working matters, but being able to tell “what happened” when things break counts for a lot in long-term operation.

Handle leaks are exactly the type of defect where that difference pays off. If you look at them through growth rates, boundaries, and responsibility pairs, rather than only at the moment they occur, they become considerably easier to chase.

Part 2: Building a Windows Failure-Path Test Foundation with Application Verifier

8. References

• GetProcessHandleCount function (processthreadsapi.h)
• Process.HandleCount Property (System.Diagnostics)
• Kernel Objects - Win32 apps
• GDI Objects - Win32 apps
• typeperf - Windows Commands
• Process Explorer / Handle / VMMap in Practice - Chasing Hangs, Leaks, and “File in Use” from the State Right Now
• Part 2: Building a Windows Failure-Path Test Foundation with Application Verifier

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Related Case Study

This case-study page shows a similar structure for diagnosis, prioritization, or redesign.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What is a handle leak?

It is when a Windows process forgets to close a handle it uses to reference an OS resource - an event, mutex, file, socket, and so on - so the Handle Count keeps climbing. The most common pattern is a resource opened temporarily for one operation and then left unclosed on a partial-failure path such as a timeout, a reconnect, or an early return. Routine short tests only exercise the success path, so the leak is easy to miss.

How do I tell a memory leak from a handle leak?

You watch different metrics. A memory leak shows up as Private Bytes and Commit creeping up, whereas a handle leak shows up as Handle Count creeping up and never coming back down. When isolating long-run issues, watching only memory leaves you driving with one eye closed, so the basic move is to watch Handle Count and Thread Count alongside it. If a GUI lives in the same process, watch GDI Objects and USER Objects too.

Why does a handle leak only crash the app after long-running operation?

A small-slope leak that loses just one handle per failure does nothing within minutes, but in 24/7 operation boundary conditions such as timeouts and reconnects happen over and over, and the leak accumulates over weeks. It finally surfaces as a secondary failure at the moment an API that creates a new event, file, or thread fails. It also matters that the crash site is usually the last victim, not the place that leaked.

How should I investigate a handle leak?

Rather than waiting for a month-scale reproduction, compress the repro by running the suspicious lifetime boundaries - open -> start -> stop -> close, timeouts, reconnects - thousands of times in a short loop. Establish a baseline after warm-up, look at the per-cycle delta and the slope of Handle Count, hunt for the place where create/open and close/dispose stop matching using structured logs that carry sessionId, resourceId, and action, and read the crash site last. That order makes it much harder to get lost.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.