Building a Windows Failure-Path Test Foundation with Application Verifier

Application Verifier is a strong option when you want to surface, ahead of time, the anomalies that occur in Windows native code and at the Win32 boundary. Especially when you want to test handle anomalies, heap corruption, and low-resource failure paths, it brings problems into the open far earlier than normal-path testing alone ever would.

In Part 1, Investigating Long-Run Crashes of an Industrial Camera App - The Handle Leak (Part 1), we covered a case where investigating a control app that crashed after long-running operation revealed a handle leak as the cause. But strengthening the logs is only half the job. What you really want is to be able to test, in advance, whether you are in a state where “you can tell what happened” if an unexpected programming mistake ever causes a memory leak, a handle leak, a partial failure, or a missed release in the future.

That is where we used Application Verifier. It is a tool that lets you insert runtime checks and fault injection into processing that runs in Windows native code and at the Win32 boundary. What is especially convenient in practice is that you can trigger memory-exhaustion-like and resource-exhaustion-like failure modes ahead of time, without actually devouring the machine’s memory.

In this second part, we organize what Application Verifier is, what it can do, and how to build it into a failure-path test foundation, in the context of an industrial camera control app.

Table of Contents

1. The Conclusion First (In One Line)
2. What Is Application Verifier?
   • 2.1. In One Sentence
   • 2.2. Where It Shines
   • 2.3. What You Gain
   • 2.4. Getting It and Enabling It on Your Machine
3. What Application Verifier Can Do
   • 3.1. Basics: Handles / Heaps / Locks / Memory / TLS, etc.
   • 3.2. Low Resource Simulation: Front-Loading Memory and Resource Exhaustion
   • 3.3. Page Heap and the Debugger
   • 3.4. !avrf / !htrace / Logs
4. Why We Introduced It This Time
   • 4.1. The Goal Is Not Just “Finding Bugs”
   • 4.2. Triggering Memory-Exhaustion-Like Phenomena
   • 4.3. Verifying We Can Trace Handle Anomalies When They Occur
5. How to Trigger Memory- and Resource-Exhaustion-Like Phenomena
   • 5.1. The Idea Behind Low Resource Simulation
   • 5.2. What You Can Make Fail
   • 5.3. How to Apply It in Practice
6. How to Look at Handle Anomalies
   • 6.1. The Handles Check
   • 6.2. Viewing Open / Close Stacks with !htrace
   • 6.3. How to Combine It with Your Own Logs
7. How to Build a Failure-Path Test Foundation
   • 7.1. Move the Execution Unit into a Harness
   • 7.2. Split the Test Menu
   • 7.3. What to Collect
   • 7.4. Acceptance Criteria
   • 7.5. Caveats
8. A Rough Decision Guide
9. Summary
10. References

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (17 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

1. The Conclusion First (In One Line)

• Application Verifier is a tool that makes misuse at Windows’ unmanaged / native boundary easier to catch at runtime
• Its value is not only “finding bugs,” but forcing rarely seen failure paths to occur ahead of time
• Handles detects invalid handles, Heaps exposes heap corruption, and Low Resource Simulation performs fault injection of memory-exhaustion-like and resource-exhaustion-like situations
• Delegating the leak investigation of a long-running resident EXE entirely to Application Verifier is an unsound approach; combining it with your own Handle Count and resource-lifecycle logs is the realistic path
• In a failure-path test foundation, it is easier to read results if you run a normal-path verifier run and a fault injection run separately
• Even when you want to test a DLL, what you enable Application Verifier on is the test EXE that actually exercises that DLL

In short, Application Verifier is a tool for dragging the nasty bugs that live around Windows’ native / Win32 boundary out into the open. It is an especially good fit in worlds like equipment control apps, where native SDKs, P/Invoke, and Win32 APIs routinely mix.

flowchart TB
    accTitle: The two roles of Application Verifier
    accDescr: Application Verifier has two roles, detecting misuse at the native boundary and front-loading failure paths that rarely surface, and it can detect invalid handles and heap corruption and inject faults that mimic memory exhaustion.
    av["Application Verifier"] --> detect["Detect misuse at the native boundary"]
    av --> inject["Front-load rarely seen failure paths"]
    detect --> d1["Invalid handles and heap corruption"]
    inject --> d2["Injection of memory-exhaustion-like conditions"]

Figure 1: Application Verifier rests on two pillars, detecting misuse and front-loading failure paths.

2. What Is Application Verifier?

2.1. In One Sentence

Application Verifier is a runtime verification tool for Windows user-mode applications. It watches how a running app uses OS APIs and manages resources, so it can detect questionable usage and deliberately inject failures.

Unlike “static analysis” or “unit testing,” it is a tool for seeing how things break when that code path is actually exercised. That makes it well suited to flushing out failure paths that routine functional testing never reveals.

flowchart LR
    A[Test harness] --> B[Control app / SDK wrapper]
    B --> C[Application Verifier]
    C --> D[Win32 API / native DLL / OS resources]
    C --> E[verifier stop]
    C --> F[debugger output]
    C --> G[AppVerifier logs]
    B --> H[Own structured log]

Figure 2: Application Verifier watches the control app driven from the test harness and leaves the observations behind as verifier stops, debugger output, and logs.

2.2. Where It Shines

It tends to be especially effective in situations like these.

• You call native DLLs or a camera SDK
• You cross P/Invoke or COM boundaries
• You use handles, heaps, locks, and virtual memory heavily, directly or indirectly
• The app hardly ever crashes on the normal path, but lifetime management looks likely to break on the failure paths
• “Occasionally returns strange failures” shows up before “crashes”

Conversely, it is not a tool for tracing object graphs in the purely managed world. So even in a C# app it pays off considerably if the native SDK or Win32 boundary is thick, but this is not a single tool for fully investigating pure managed heap leaks.

flowchart TB
    accTitle: Telling where Application Verifier is effective
    accDescr: It is effective for apps with native DLLs, P/Invoke, and a thick Win32 boundary, but it is not a tool for tracing object graphs in the purely managed world.
    q{"Which layer is the problem in"}
    q -->|"Native SDK or Win32 boundary"| yes["Application Verifier is effective"]
    q -->|"Purely managed object graph"| no["Out of scope (use a different tool)"]

Figure 3: What decides whether it helps is how thick the native / Win32 boundary is; it is not a tool for looking only at the purely managed world.

2.3. What You Gain

In practice, the benefits boil down to roughly these three.

1. Stop native-boundary misuse early
   • invalid handles
   • heap corruption
   • lock misuse
   • virtual memory API misuse, etc.
2. Front-load failure modes that only appear under low resources
   • malloc-equivalents occasionally fail
   • CreateEvent and CreateFile occasionally fail
   • VirtualAlloc fails
3. Easier tracing when combined with a debugger
   • !avrf
   • !htrace
   • !heap -p -a
   • verifier stop logs

What hurts in equipment control apps is not knowing what happened on the failure path. Application Verifier is quite effective at reducing that not-knowing.

flowchart TB
    accTitle: Three practical benefits
    accDescr: Misuse at the native boundary can be stopped early, failure modes that only appear under low resources can be front-loaded, and combining it with a debugger makes tracing easier.
    av["Application Verifier"] --> b1["Stop misuse early"]
    av --> b2["Front-load failure modes"]
    av --> b3["Easier to trace with a debugger"]
    b3 -.-> t["Extensions such as avrf and htrace"]

Figure 4: The practical value boils down to three things, early detection, front-loading failure paths, and debugger integration.

2.4. Getting It and Enabling It on Your Machine

Let us get the tooling out of the way first. Skip this part and everything that follows stays theoretical.

Application Verifier ships with the Windows SDK. It does not come with Windows itself, so launch the SDK installer and tick “Application Verifier” on the feature selection screen. The executable is named appverif.exe.

There are three prerequisites for using it.

• The user running it must be a member of the Administrators group on that machine
• ARM64EC is not supported
• The verification target must be unmanaged (native) code

The relationship between the GUI and the command line is easiest to keep straight like this.

In other words, whichever you use, the work being done is the same. You use the GUI for the first manual pass and the command line for CI and scripts.

flowchart TB
    accTitle: Relationship between the GUI and the command line
    accDescr: Both the GUI and the command line only write the same registry settings, and when the target EXE starts it reads those settings, loads the verifier DLL, and installs the Win32 API hooks.
    gui["GUI (appverif.exe)"] --> reg["Write settings to the registry"]
    cli["Command line"] --> reg
    reg --> boot["Read at target EXE startup"]
    boot --> hook["Verifier DLL load and hooks"]

Figure 5: The GUI and the command line only write the same registry settings, and the hooks go in when the target EXE starts.

In the GUI, the flow is: right-click in the Applications pane on the left, choose “Add Application” to add the target EXE, tick Basics and whatever else you want in the Tests pane on the right, and press “Save”. To undo it, right-click in the same Applications pane, choose “Delete Application”, and press “Save”.

Two important constraints follow from this.

• A process that is already running cannot be enabled after the fact. The hooks go in when the DLL is loaded, so the order is: configure, then launch.
• The settings remain until you explicitly delete them. Leave them behind after “just trying it once” and that EXE will keep starting under the verifier on that machine.

Detection logs are saved by default in binary form under %USERPROFILE%\AppVerifierLogs, and can be converted to XML for aggregation from either the GUI or the command line.

flowchart TB
    accTitle: Enabling order and how settings persist
    accDescr: Hooks are installed when the DLL loads, so a running process cannot be enabled after the fact and the order is configure then launch, and the settings remain until they are explicitly deleted.
    set["Write the settings"] --> launch["Launch the target EXE"]
    launch --> on["Runs under the verifier"]
    on --> keep["Settings stay until deleted"]
    keep -.-> warn["Left alone it always starts under the verifier"]
    running["A running process"] -.-> ng["Cannot be enabled after the fact"]

Figure 6: The configure-then-launch order cannot be broken, and the settings stay on that machine until you explicitly delete them.

3. What Application Verifier Can Do

3.1. Basics: Handles / Heaps / Locks / Memory / TLS, etc.

Application Verifier’s basic set is Basics. The checks you use most in practice are gathered here.

The point is to stop questionable usage on the spot, rather than “read about it after the crash.” For long-run defects, this front-loading pays off considerably.

flowchart TB
    accTitle: How Basics front-loads detection
    accDescr: Instead of reading the logs after the crash and guessing, questionable usage is stopped on the spot, which surfaces long-run defects ahead of time.
    use["Questionable API usage"] --> basics["The Basics checks"]
    basics --> stop["Stop it on the spot"]
    stop --> early["Surface the problem ahead of time"]
    use -.-> later["Previously only readable after the crash"]

Figure 7: The value of Basics is turning “read it after the crash” into “stop it on the spot.”

3.2. Low Resource Simulation: Front-Loading Memory and Resource Exhaustion

This is the genuinely convenient part in practice. That is because you can trigger phenomena close to memory exhaustion and resource exhaustion without actually devouring the RAM.

The idea is simple.

• Take a certain API call
• With a certain probability
• Make it fail on purpose

This lets you exercise error paths that are practically never taken otherwise.

Concretely, it becomes easy to trigger phenomena like these on purpose.

• HeapAlloc and VirtualAlloc fail
• CreateFile fails
• CreateEvent fails
• MapViewOfFile fails
• OLE/COM allocations like SysAllocString fail

This is far more manageable than trying to genuinely exhaust memory and putting the whole machine through the wringer. What is more, you can aim fault injection at specific DLLs only. For configurations like equipment control apps where your own wrappers mix with vendor SDKs, this is quite practical.

flowchart TB
    accTitle: How Low Resource Simulation works
    accDescr: Making a kind of API call fail on purpose with a fixed probability lets you deliberately take error paths that are practically never reached, and the target can be narrowed to specific DLLs.
    call["API call"] --> judge{"Did it hit the probability?"}
    judge -->|"Yes"| fail["Return a deliberate failure"]
    judge -->|"No"| ok["Process as usual"]
    fail --> path["Into an error path never normally taken"]
    path -.-> dll["Can be scoped to specific DLLs"]

Figure 8: Low Resource Simulation is, at bottom, fault injection that fails API calls with a fixed probability.

3.3. Page Heap and the Debugger

For heap corruption, the combination of Heaps and page heap is strong. Full page heap in particular has the advantage of using guard pages to make it easy to stop at the moment of corruption.

However, it is quite heavy. Rather than long brute-force runs, it is more usable to narrow down to scenarios close to the repro and run them under the debugger.

So as an operating practice, a split like this is realistic.

• First apply Basics broadly
• Once the heap looks suspicious, use full page heap
• If it is too heavy, fall back to light page heap
• For production-like long-run testing, rely primarily on your own logs

Ultimately, AppVerifier is not a magic wand but a tool whose blade you swap per situation.

flowchart TB
    accTitle: Flow for choosing page heap settings
    accDescr: First apply Basics broadly, then if the heap looks suspicious use full page heap to stop at the moment of corruption, fall back to light page heap when it is too heavy, and rely mainly on your own logs for production-like long runs.
    s1["Apply Basics broadly"] --> s2{"Does the heap look suspicious?"}
    s2 -->|"Yes"| s3["Stop with full page heap"]
    s2 -->|"No"| s7["Long runs rely on own logs"]
    s3 --> s4{"Too heavy?"}
    s4 -->|"Yes"| s5["Fall back to light page heap"]
    s4 -->|"No"| s6["Reproduce locally under the debugger"]

Figure 9: Page heap is not a tool to leave on all the time; apply Basics broadly first, then swap the blade per situation.

3.4. !avrf / !htrace / Logs

One word on terminology first. The verifier stop that has come up several times is the detection event Application Verifier raises when it decides a particular usage is wrong. It is not merely a log line: if you are running under a debugger, it breaks right there. Stops are numbered and displayed like VERIFIER STOP 00000300. Some stops let you continue and some do not, leaving you no choice but to end the process.

Application Verifier does not just raise a stop and walk away. With its debugger extensions and logs, what happened becomes easier to chase.

• !avrf
  • View the current verifier settings and the stop currently raised
• !htrace
  • View the stacks of a handle’s open / close / invalid references
• !heap -p -a
  • Combined with page heap, trace the corrupted heap block
• AppVerifier logs
  • Logs can be kept for when a stop occurs

It is especially welcome that enabling Handles automatically enables handle tracing. This makes it much easier to trace, after the fact, where this handle was opened and where it was closed.

flowchart TB
    accTitle: From verifier stop to investigation
    accDescr: Detecting bad usage raises a numbered verifier stop, which breaks on the spot under a debugger, and avrf shows the settings and the stop while htrace shows the handle history.
    bad["Bad usage detected"] --> stop["verifier stop (numbered)"]
    stop --> brk["Breaks on the spot under a debugger"]
    brk --> avrf["Check settings and stop with avrf"]
    brk --> ht["Check handle history with htrace"]
    stop -.-> cont["Continuable stops and non-continuable stops"]

Figure 10: A verifier stop is not just a log line; under a debugger it breaks on the spot and becomes the starting point of the investigation.

4. Why We Introduced It This Time

4.1. The Goal Is Not Just “Finding Bugs”

Our goal this time was not simply to find one bug with AppVerifier. Put more practically, what we wanted to confirm was the following.

• When a resource leak happens again on some other failure path in the future
• Will the logs properly retain the context?
• Can we chase it down to the end, together with debugger information?
• Will we avoid ending up in a “no idea what happened” state?

In other words, we used it not only as a detector, but as a test of our observation foundation.

4.2. Triggering Memory-Exhaustion-Like Phenomena

Genuinely causing memory exhaustion on a regular development machine is fairly tedious. Worse, once the whole machine becomes unstable, the test itself fills with noise.

So we used Low Resource Simulation to go in the direction of deliberately stepping on the failure paths that memory or resource exhaustion would likely trigger.

This makes it much easier to answer questions like these.

• If CreateEvent fails, do cameraId and phase remain in the logs?
• After a half-finished initialization, does clean up actually run?
• If VirtualAlloc fails, does the retry avoid corrupting state?
• If CreateFile fails on the save path, does the handle come back?

What we want to emphasize is that causing the anomaly is not the goal; the goal is that the failure mode is readable when the anomaly occurs.

flowchart TB
    accTitle: Testing the observation foundation with fault injection
    accDescr: Low Resource Simulation deliberately triggers failures so you can check whether the log keeps the context, whether cleanup runs, and whether a retry corrupts state. The goal is not to cause anomalies but to make the failure mode readable.
    inject["Deliberately trigger failures"] --> q1["Does the log keep the context?"]
    inject --> q2["Does clean up run?"]
    inject --> q3["Does the retry avoid corruption?"]
    q1 --> goal["A state where the failure mode is readable"]
    q2 --> goal
    q3 --> goal

Figure 11: The purpose of fault injection is not to cause anomalies but to check whether the failure mode is readable when one occurs.

4.3. Verifying We Can Trace Handle Anomalies When They Occur

As with the handle leak in Part 1, with handles the place that finally crashes and the true cause easily drift apart.

So what we wanted to confirm was this.

• When an invalid handle stop is raised, can we trace the open / close with !htrace?
• Does it tie back to the resourceId / sessionId / phase in our own logs?
• Does the handle count come back down after the failure?
• When the harness is a short-lived process, are the leak deltas easy to read?

Once you can see this far, you can go from a mere “a bug appeared” to identifying which responsibility’s lifetime management broke down.

flowchart TB
    accTitle: Checking traceability when handle anomalies occur
    accDescr: When an invalid handle stop is raised, check whether htrace can follow the open and close, whether it ties back to the context in your own logs, and whether the handle count recovers, so that the responsibility whose lifetime management broke down can be identified.
    stop["invalid handle stop"] --> c1["Follow open and close with htrace"]
    stop --> c2["Tie it to the context in own logs"]
    stop --> c3["Check that handle count recovers"]
    c1 --> goal["Identify the responsibility whose lifetime management broke"]
    c2 --> goal
    c3 --> goal

Figure 12: With handle anomalies, do not stop at “a bug appeared”; check whether you can trace which responsibility’s lifetime management broke down.

5. How to Trigger Memory- and Resource-Exhaustion-Like Phenomena

5.1. The Idea Behind Low Resource Simulation

Low Resource Simulation is, in plain terms, fault injection. Rather than faithfully recreating a low-resource environment, the idea is to artificially mix in the representative API failures that occur under low resources.

So its use cases are quite clear-cut.

• Checking cleanup on failure paths
• Checking the robustness of retry / reconnect
• Checking initialization where partial successes and partial failures mix
• Checking that logs remain even for failures that normally never happen

The trick here is to not fail everything from the start. If you turn everything on at once, the logs explode and you lose track of what you are even looking at.

flowchart TB
    accTitle: Narrowing down fault injection
    accDescr: Failing everything from the start makes the logs explode so that you lose track of what you are looking at, so open up only the failures closest to the failure path you want to see.
    all["Fail everything from the start"] --> noise["Logs explode and cannot be read"]
    narrow["Open only the failures you want"] --> clear["Clear about what you are looking at"]

Figure 13: The trick with fault injection is not to turn everything on; start narrow, with the failures closest to the failure path you want to see.

5.2. What You Can Make Fail

With Low Resource Simulation, you can probabilistically fail the following representative classes of APIs.

In practice, rather than opening everything at once, the key is to start narrow, with the classes closest to the failure path you want to look at this time.

5.3. How to Apply It in Practice

As a command-line sketch, it looks like this, for example.

appverif /verify CameraHarness.exe
appverif /verify CameraHarness.exe /faults
appverif -enable lowres -for CameraHarness.exe -with heap_alloc=20000 virtual_alloc=20000 file=20000 event=20000
appverif -query lowres -for CameraHarness.exe

Copying and pasting these is pointless if you cannot read the intent, so here is what each line does.

It is worth pinning down how to read the arguments as well.

• Probabilities are in parts per million. You can specify an integer from 0 to 1,000,000, so 20000 is 20000 / 1,000,000, that is 2%. It does not mean “once in 20,000 times.” Microsoft’s documentation likewise gives -with registry=20000 file=20000 as an example of failing the registry and file APIs at 2%.
• /faults can be followed by a probability, a grace period, and DLL names. The form is /faults [probability [grace-milliseconds [DLL ...]]]. Omit the probability and it becomes 5%; omit the grace period and it becomes 500 milliseconds. The grace period means “inject no faults for this long after process start,” and it exists to keep startup itself from failing and leaving you with nothing to test.
• /n turns it off. Read n as roughly “no verifier.” It is the counterpart command that keeps you from leaving the verifier enabled indefinitely.

The output of -query lowres comes back roughly in this shape. Use it to check that the probabilities you set are in place, and that you have not opened up classes you were not aiming at.

Settings for CameraHarness.exe:
Test [lowres] enabled.

Include = *
Exclude =
TimeOut = 2000 (0x7D0)
WAIT = 0 (0x0)
HEAP_ALLOC = 20000 (0x4E20)
VIRTUAL_ALLOC = 0 (0x0)
REGISTRY = 0 (0x0)
FILE = 20000 (0x4E20)
EVENT = 20000 (0x4E20)
MAP_VIEW = 0 (0x0)
OLE_ALLOC = 0 (0x0)
STACKS = false

Include and Exclude narrow down the target modules, and TimeOut is the window right after startup during which no faults are injected. The defaults change with how the settings were applied, so rather than assuming, confirm them in this output.

The approach goes like this.

1. First run the normal path with Basics alone
2. Then add Low Resource Simulation and run with fault injection
3. If needed, assign probabilities only to the failures you want to see, such as file or event
4. If you want to target a specific DLL, scope the injection to that DLL

The /faults shortcut is convenient, but on its own it is centered on OLE_ALLOC and HEAP_ALLOC. If you want to look at the failure paths of CreateFile or CreateEvent, it is more reliable to spell out -enable lowres -with file=... event=....

In equipment control apps, it is often easier to read results when you scope to the camera wrapper or the save-path DLL, rather than scattering faults across the whole app.

flowchart TB
    accTitle: Order for applying fault injection
    accDescr: First run the normal path with Basics alone, then add Low Resource Simulation and run with fault injection, assign probabilities only to the failures you want to see, and scope the injection to specific DLLs when needed.
    s1["Normal path with Basics alone"] --> s2["Add Low Resource and run"]
    s2 --> s3["Set probabilities only for wanted failures"]
    s3 --> s4["Scope the injection to specific DLLs"]

Figure 14: Do not snipe from the outset; start with the normal path under Basics and narrow the fault injection down in stages.

Here is the concrete way to write that “scope to a DLL” part. The third and later arguments of /faults name the target modules.

appverif /verify CameraHarness.exe /faults 50000 1000 CameraSdkWrapper.dll

With this, when CameraHarness.exe starts, only the operations that originate in CameraSdkWrapper.dll, and only after 1000 milliseconds have elapsed since startup, fail with a probability of 5% (50000 / 1,000,000). Write module names with their extension and no path. Modules other than .dll, such as .ocx, can be named too.

Whether the scoping took effect can be confirmed on the Include and Exclude lines of appverif -query lowres -for CameraHarness.exe. If they still read *, the whole process is still in scope.

flowchart TB
    accTitle: How DLL-scoped fault injection behaves
    accDescr: Giving faults a probability, a grace period, and a target module makes only the operations that start in that DLL fail at the given probability once the grace period has passed, and the Include and Exclude lines of query confirm the scoping.
    arg["Specify probability, grace period, and DLL name"] --> grace["No injection during the grace period"]
    grace --> target["Only operations starting in that DLL fail"]
    target --> check["Confirm with Include and Exclude in query"]

Figure 15: DLL-scoped fault injection fails only the operations that start in the named module, and only after the grace period has elapsed.

If you are running under a debugger, you can change the scope partway through.

!avrf -trg dll CameraSdkWrapper.dll
!avrf -skp dll VendorSdk.dll

-trg means “target this” and -skp means “skip this.” You can also check the current fault injection settings with !avrf -flt, or view the stacks of the most recently injected failures with !avrf -flt stacks 10.

For example, you can build scenarios like these.

• CreateEvent failure right after a reconnect starts
• CreateFile failure at the start of saving
• Temporary buffer allocation failure
• SysAllocString failure during COM conversion
• Checking the failure paths of the wait APIs

These are practically never reached by routine normal-path testing alone. That is exactly why deliberately stepping on them is worth it.

6. How to Look at Handle Anomalies

6.1. The Handles Check

For everything handle-related, start with Handles. This makes the use of invalid handles easier to detect.

These are the kinds of bugs it typically catches.

• Using a handle again after it was closed
• Passing a corrupted handle value
• Using a handle left uninitialized by a partial failure
• A broken lifetime leading to access from another thread

Where long-run operation would only show an odd error now and then, under the verifier it can stop right on the spot. This front-loading helps a great deal.

flowchart TB
    accTitle: Failures the Handles check catches
    accDescr: Reuse of a closed handle, a corrupted handle value, a handle left uninitialized by a partial failure, and misuse from another thread after a broken lifetime can all be stopped on the spot under the verifier.
    a1["Reuse after close"] --> stop["verifier stop on the spot"]
    a2["Corrupted handle value"] --> stop
    a3["Uninitialized handle"] --> stop
    a4["Misuse after a broken lifetime"] --> stop

Figure 16: Handle bugs that long-run operation would report as nothing more than an odd error now and then are stopped on the spot by the Handles check.

6.2. Viewing Open / Close Stacks with !htrace

What makes Handles so welcome is that it pairs well with handle tracing.

From here on we use a debugger, so install WinDbg first. It is distributed as Debugging Tools for Windows, and can be installed from the same Windows SDK installer as Application Verifier.

windbg -xd av -xd ch -xd sov CameraHarness.exe
!avrf
!htrace 0x00000ABC

The options on the first line are not magic incantations. Application Verifier throws three kinds of exceptions when it detects something.

And -xd specifies that the exception is caught on second chance. The reason is that Application Verifier itself handles the first chance to assemble the stop information, so it is inconvenient if the debugger breaks in first. If you are setting this on a debugger that is already running, it is the same as typing sxd av, sxd ch, and sxd sov.

What you want to see with !htrace is roughly this.

• Where that handle was opened
• Where it was closed
• Whether it was referenced as an invalid handle
• Whether opens are piling up more than expected

Here is what it actually looks like. The following are output examples from the official documentation rather than from our own environment, but the shape is exactly as shown.

Stepping on an invalid handle prints this first.

Invalid handle - code c0000008 (first chance)
===================================================
VERIFIER STOP 00000300: pid 0x558: invalid handle exception for current stack trace
        C0000008 : Exception code.
        0012FBF8 : Exception record. Use .exr to display it.
        0012FC0C : Context record. Use .cxr to display it.
        00000000 :
===================================================
This verifier stop is continuable.
After debugging it use 'go' to continue.
===================================================

Following that with !avrf shows what is currently enabled and which stop has occurred. The last line is the summary.

0:000> !avrf
Global flags: 00000100
Application verifier global flag is set.
Application verifier settings (00000004):
   - no heap checking enabled!
   - handle checks
Page heap is not active for this process.
Current stop 00000300 : c0000008 0012fbf8 0012fc0c 00000000 .
    Using an invalid handle (either closed or simply bad).

Viewing that handle’s history with !htrace lists OPEN / CLOSE / BAD REFERENCE, each with its own stack.

0:000> !htrace 7DC

--------------------------------------
Handle 0x7DC - BAD REFERENCE:
0x801902BE: ntoskrnl!NtSetEvent+0x6C
0x010012C1: badhandle!mainCRTStartup+0xE3
--------------------------------------
Handle 0x7DC - CLOSE:
0x801E1EDD: ntoskrnl!NtClose+0x19
0x010012C1: badhandle!mainCRTStartup+0xE3
--------------------------------------
Handle 0x7DC - OPEN:
0x77DE265C: KERNEL32!CreateEventA+0x66
0x010011A0: badhandle!main+0x20
--------------------------------------

Reading it is straightforward: if a BAD REFERENCE comes after a CLOSE, a closed handle is being used again. The OPEN stack also tells you where that handle was created.

What makes handle leaks and handle misuse troublesome is that the API that finally fell over is not the true cause. With !htrace, you can trace that handle’s history quite concretely.

flowchart TB
    accTitle: How to read handle history with htrace
    accDescr: htrace lists OPEN, CLOSE, and BAD REFERENCE for a handle each with its own stack, so a BAD REFERENCE after a CLOSE means a closed handle was reused, and the OPEN stack shows where the handle was created.
    open["OPEN (where it was created)"] --> close["CLOSE (where it was closed)"]
    close --> bad["BAD REFERENCE"]
    bad --> mean["Reuse of a closed handle confirmed"]
    open -.-> stack["Every record carries a stack"]

Figure 17: Reading htrace is straightforward: a BAD REFERENCE listed after a CLOSE means a closed handle was reused.

6.3. How to Combine It with Your Own Logs

That said, Application Verifier alone is not enough. In particular, doing the leak investigation of a long-running resident EXE with it alone is quite painful.

So in practice we combine the following.

• Periodic Handle Count
• sessionId
• resourceId
• phase
• Lifecycle logs of create/open and close/dispose
• Dumps and debugger output at verifier stops

With this, you can chase the problem like so, for example.

1. The heartbeat shows the slope of Handle Count is suspicious
2. The lifecycle logs narrow down the resource that has a Create but no Close
3. A verifier run surfaces the invalid handle or misuse ahead of time
4. !htrace shows the open / close stacks

This combination makes things dramatically easier to chase.

flowchart TB
    accTitle: Steps for combining own logs with the verifier
    accDescr: Notice the slope of Handle Count from the heartbeat, narrow down the resources with no Close from the lifecycle log, surface misuse ahead of time with a verifier run, and view the open and close stacks with htrace.
    s1["Notice the Handle Count slope"] --> s2["Narrow resources with lifecycle log"]
    s2 --> s3["Front-load misuse with a verifier run"]
    s3 --> s4["View the stacks with htrace"]

Figure 18: Own logs detect the slope and the verifier detects the misuse, and this is the order that connects the two.

7. How to Build a Failure-Path Test Foundation

7.1. Move the Execution Unit into a Harness

Application Verifier cannot be enabled retroactively on an already-running process. You configure first, then launch.

Moreover, the settings persist until you explicitly delete them. So in practice, it is easier to handle if you target a test harness EXE rather than the production app itself.

For example, a configuration like this.

flowchart LR
    A[Scenario Runner] --> B[CameraHarness.exe]
    B --> C[CameraSdkWrapper.dll]
    C --> D[Vendor SDK]
    B --> E[Structured Log]
    B --> F[Dump / Debugger]

Figure 19: A harness setup that runs one scenario per process. What the verifier targets is not the DLL but the harness EXE that drives it.

With this, you get the advantages of:

• Running one scenario per process
• Leak deltas being easy to read
• Easy toggling of the AppVerifier settings ON/OFF
• Being able to test DLLs through the EXE side

The commands look like this.

appverif /verify CameraHarness.exe
appverif /n CameraHarness.exe

/verify enables Basics, and /n deletes the settings (see also the table in 5.3). Enable before launch; disable explicitly. Running this with a harness as the premise also makes it easier to avoid configuration mistakes.

7.2. Split the Test Menu

In a failure-path test foundation, it is better not to do everything in one run. Splitting into roughly these three tracks keeps things readable.

1. Normal path + Basics
   • Inject no failures
   • Confirm that no verifier stops occur
2. Fault injection track
   • Low Resource Simulation
   • Target failures at event / file / heap_alloc / virtual_alloc, etc.
3. Heap deep-dive track
   • Heaps
   • full page heap
   • Reproduce locally under the debugger

Splitting these keeps “is it broken under normal usage” and “does it only break under low resources” from getting tangled.

The presence or absence of fault injection in particular changes the code paths taken considerably. So you should run both the no-fault run and the with-fault run.

flowchart TB
    accTitle: The test menu split into three tracks
    accDescr: Split the tests into a normal path plus Basics track that injects nothing, a fault injection track that deliberately fails calls with Low Resource Simulation, and a heap deep-dive track that runs full page heap under the debugger.
    menu["How to run failure-path tests"] --> m1["Normal path and Basics"]
    menu --> m2["Fault injection track"]
    menu --> m3["Heap deep-dive track"]
    m1 -.-> p1["Confirm that no stop occurs"]
    m2 -.-> p2["Inject the targeted failures"]
    m3 -.-> p3["Reproduce locally under the debugger"]

Figure 20: Splitting into three tracks instead of doing everything in one run keeps it clear where things are breaking.

7.3. What to Collect

At minimum, you want to capture these.

If needed, the AppVerifier-side logs can also be exported to XML and aggregated. But the cause rarely closes from those alone, so the practical premise is reading them side by side with your own logs.

A large volume of logs is not, in itself, a virtue. What matters is that the causality can be connected later.

7.4. Acceptance Criteria

“It did not crash” is also too weak as an acceptance criterion. In this context, we needed at least the following.

• No verifier stops in the normal path + Basics run
• Even with fault injection, the expected failures remain in the logs
• Half-initialized resources get cleaned up properly
• After reconnect / retry, Handle Count returns near the baseline
• When a verifier stop occurs, it can be traced via sessionId / phase / stack
• No failure ends up as “no idea what happened”

What matters here is to evaluate not breaking and being traceable when broken as separate things.

flowchart TB
    accTitle: The two axes of the acceptance criteria
    accDescr: Evaluate the acceptance criteria on two axes, not breaking as shown by no verifier stop on the normal path and resources being cleaned up, and being traceable when broken as shown by the expected failures remaining in the logs and being followable by context and stack.
    pass["Acceptance criteria"] --> a["Not breaking"]
    pass --> b["Being traceable when broken"]
    a --> a1["No stop occurs"]
    a --> a2["Resources get cleaned up"]
    b --> b1["Failures remain in the logs"]
    b --> b2["Traceable by stack"]

Figure 21: “It did not crash” is too weak on its own; evaluate not breaking and being traceable as separate axes.

7.5. Caveats

Application Verifier is quite convenient, but it is not magic.

• Code paths not actually exercised are not verified
• Full page heap is heavy
• Stops can also occur inside third-party SDKs
• The code paths taken differ considerably with and without fault injection
• It is not a single tool for investigating pure managed heap leaks

So its position is this.

• Long-run slopes: your own logs and counters
• Native-boundary misuse: Application Verifier
• Reconstructing causality on failure: structured logs + dumps + debugger

This division of labor is the most practical.

flowchart TB
    accTitle: The overall division of labor in an investigation
    accDescr: Long-run slopes are watched with your own logs and counters, misuse at the native boundary with Application Verifier, and reconstruction of causality on failure with structured logs, dumps, and the debugger.
    q1["Long-run slopes"] --> t1["Own logs and counters"]
    q2["Misuse at the native boundary"] --> t2["Application Verifier"]
    q3["Reconstructing causality on failure"] --> t3["Logs, dumps, and the debugger"]

Figure 22: Application Verifier is not a magic wand; divide the work among tools according to what you want to see.

8. A Rough Decision Guide

• Invalid handles or double closes are suspected
  • Handles + !htrace
• Heap corruption / use-after-free is suspected
  • Heaps + full page heap + !heap -p -a
• You want to trigger memory- or resource-exhaustion-like phenomena
  • Low Resource Simulation
• Things break gradually under long-running operation
  • Start with your own Handle Count / Private Bytes / lifecycle logs
• You want to test a DLL
  • Enable Application Verifier on the harness EXE that calls that DLL

Turning everything on from the start usually just produces a fog of logs. Applying the blade closest to the failure path you want to see is far clearer.

9. Summary

Application Verifier’s position is that of a runtime verifier for Windows’ native / Win32 boundary. Using Handles / Heaps / Locks / Memory / TLS / Low Resource Simulation and the rest, you can force rarely seen failure paths to be exercised ahead of time.

What paid off in this context was that handle anomalies became easy to trace with !htrace when they occurred, that memory- and resource-exhaustion-like phenomena could be triggered without wrecking the whole machine, and that we could confirm whether our own logs would genuinely be useful at that moment.

As for how to run it in practice: split the normal path + Basics runs from the fault injection runs, prepare a harness EXE, and cycle scenarios through short-lived processes. On top of that, combine it with your own logs, dumps, and debugger information, and watch the slope of long-run leaks itself with your own counters. That is the division of labor.

Application Verifier is a tool for going out to meet rare anomalies rather than waiting around for them to happen.

In equipment control apps, not breaking matters, but being able to explain what happened when things break matters just as much. In that sense, we think it is a thoroughly practical tool.

Part 1: Investigating Long-Run Crashes of an Industrial Camera App - The Handle Leak (Part 1)

10. References

• Part 1: Investigating Long-Run Crashes of an Industrial Camera App - The Handle Leak (Part 1)
• Application Verifier - Overview
• Application Verifier - Testing Applications
• Application Verifier - Tests within Application Verifier
• Application Verifier - Debugging Application Verifier Stops
• Application Verifier - Features
• !htrace (WinDbg)
• !avrf (WinDbg)
• Download Debugging Tools for Windows
• Windows SDK download
• GetProcessHandleCount function (processthreadsapi.h)

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Related Case Study

This case-study page shows a similar structure for diagnosis, prioritization, or redesign.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What is Application Verifier?

It is a runtime verification tool for Windows user-mode applications. It watches how a running app uses OS APIs and manages resources, so it can detect questionable usage such as invalid handle use or heap corruption, and it can also inject failures deliberately. Unlike static analysis or unit testing, it is a tool for seeing how things break when that code path is actually exercised, which makes it well suited to flushing out failure paths that routine functional testing never reveals.

Can Application Verifier reproduce memory exhaustion?

Low Resource Simulation lets you trigger phenomena close to memory or resource exhaustion ahead of time, without genuinely devouring the machine's RAM. The mechanism is fault injection: API calls such as HeapAlloc, VirtualAlloc, CreateFile, and CreateEvent are made to fail on purpose with a given probability. You can also aim the injection at specific DLLs only, which keeps it manageable even when your own wrappers and a vendor SDK are mixed together. Failing everything from the start makes the logs unreadable, so the trick is to open up only the failures closest to the failure path you want to see.

Can Application Verifier be used to investigate handle leaks?

Enabling the Handles check detects invalid handle use such as reusing a closed handle, and it also turns on handle tracing automatically, so you can follow that handle's open / close stacks with !htrace. That said, delegating the leak investigation of a long-running resident EXE entirely to Application Verifier is not realistic. Combine it with periodic Handle Count recording and your own resource-lifecycle logs: your own logs detect the slope and the verifier detects the misuse. That division of labor is what works in practice.

How do I use Application Verifier to test a DLL?

What you enable Application Verifier on is the test EXE that actually exercises that DLL. It cannot be enabled on an already-running process, so you have to configure first and launch afterward. The settings also persist until you explicitly delete them, which makes a test harness EXE easier to handle than the production app itself. Running one scenario per process also makes leak deltas easier to read and makes toggling the settings on and off easier.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.