Pinpointing "Slow" with PerfView and dotnet-trace — A Practical Introduction to .NET Performance Investigation

The previous article, “An Introduction to Windows Event Log and ETW,” covered what ETW is and how to define custom events with EventSource, but treated analysis with PerfView as “out of scope.” This article picks up where that one left off. It covers the steps for using PerfView and dotnet-trace to pin down the exact function responsible when a business app is “slow,” “pegs the CPU,” or “occasionally freezes.”

Note that this article’s main subject is investigating CPU and response time. Diagnosing memory that keeps growing is covered in “Telling GC Pauses Apart from Memory Leaks in .NET,” and diagnosing crashes is covered in “Reading Crash Dumps with WinDbg + SOS.”

1. The Bottom Line First

• There are two kinds of “slow.” Slow because the CPU is maxed out (CPU bound), and slow because the CPU is idle but the app is waiting (blocked). The former needs CPU sampling, and the latter needs ThreadTime (context switch) collection — looking only at the default CPU collection won’t reveal the cause of the latter. ¹
• When in doubt, start with dotnet-trace. It’s a cross-platform, EventPipe-based tool, and you can collect without administrator privileges as long as you’re the same user who launched the target process. ² The .nettrace files it collects can be opened in PerfView or Visual Studio. ³
• PerfView becomes necessary when you need to see the whole machine, native code, or blocked time. Because it’s ETW-based, it can also handle kernel events and native code stacks. In exchange, starting an ETW session requires administrator privileges. ³
• CPU sampling defaults to once every millisecond (per processor). Read each sample as roughly 1 ms of CPU time, and as a rule of thumb, wait until you’ve collected at least 1,000 samples (ideally around 5,000) before drawing conclusions. The top functions when the sample count is still low could just be down to chance. ¹
• Interpreting the numbers starts entirely with the distinction between inclusive (self + everything called) and exclusive (self only). A function with a large exclusive value is “a place that’s actually consuming CPU”; a function with a large inclusive value is “a place where something further down is heavy.”
• Before you measure, pin down exactly what “slow” means to a single operation. Collecting data while the app is “generally sluggish” won’t be readable. Fix the operation and the timing first — something like “this report takes 40 seconds to generate” — and only then start collecting. The approach to before/after comparison measurements in “How to Correctly Compare Program Speed Across Versions on Windows” is also worth a look here.

Here’s a decision table for where to start based on the symptom.

2. Sorting Out How the Tools Relate — ETW and EventPipe

There look to be a lot of tools involved, but there are really only two underlying foundations.

• ETW (Event Tracing for Windows): a trace infrastructure that runs across the entire OS. It can record everything from the kernel to the app on the same timeline, but starting and stopping a session requires administrator privileges, and it’s Windows-only. ³
• EventPipe: a trace mechanism built into the .NET runtime itself. It doesn’t require administrator privileges and behaves the same way on every OS, but in exchange, what it can see is limited to managed code and the runtime — it can’t capture kernel events or native stacks. ²

Once this mapping is second nature, the natural workflow of “start with a light collection using dotnet-trace, and reach for the full ETW collection in PerfView if that’s not enough” falls into place on its own.

3. Before You Collect a Trace — Spend 10 Minutes with dotnet-counters

Rather than jumping straight into a trace collection, it’s often faster to first glance at the runtime’s key metrics with dotnet-counters. ⁴

dotnet tool install --global dotnet-counters
dotnet-counters ps
dotnet-counters monitor --process-id <PID> --counters System.Runtime

What you’re looking at here is CPU usage, the GC heap size and GC count, the ThreadPool’s queue length and thread count, and the exception count. If you spot a trend at this stage — “GC is running several times a second,” “the ThreadPool queue keeps growing” — it narrows down what kind of trace to collect next (allocation-focused, or blocking-focused). How to read memory-related trends is covered in detail in “Telling GC Pauses Apart from Memory Leaks in .NET.”

4. Collecting a Trace with dotnet-trace

dotnet-trace is an EventPipe-based collection tool. ⁵ Installation and basic collection look like this.

dotnet tool install --global dotnet-trace
dotnet-trace ps
dotnet-trace collect --process-id <PID> --duration 00:00:00:30

If you don’t specify any options, it collects using the default profile configuration, which includes the runtime’s key events plus thread sampling. The profile that used to be called cpu-sampling has been retired; the default today is a combination of dotnet-common and dotnet-sampled-thread-time. ⁵ Depending on what you need, you can also choose --profile gc-verbose (GC and object allocation sampling) or --profile gc-collect (records only GC occurrences, at low overhead).

To collect the custom EventSource provider we built in the previous article alongside everything else, add --providers.

dotnet-trace collect --process-id <PID> --providers KomuraSoft-OrderService

The collected result is a .nettrace file. There are three ways to open it: open it in Visual Studio or PerfView, or convert it to speedscope format and view it in a browser. ⁵

dotnet-trace convert trace.nettrace --format Speedscope

The speedscope format is a lightweight flame graph viewable at speedscope.app, and it’s good for getting an intuitive sense of “where did time go, and under which function’s call tree.” For a more rigorous investigation down to the exact function, PerfView’s stack viewer is more powerful, so let’s move on to the next chapter.

5. Investigating CPU with PerfView — How to Read the Stack Viewer

PerfView is a performance investigation tool that Microsoft publishes for free; it runs as soon as you download the single PerfView.exe file from its GitHub releases page (no installation needed). ⁶

5.1 Collecting

The basic flow through the GUI is to open the dialog via “Collect > Collect,” click Start Collection, reproduce the target operation, then click Stop Collection. From the command line, it looks like this (run as an administrator, since it needs to start an ETW session ³).

PerfView collect /nogui /acceptEULA /maxCollectSec:30

The default collection includes a 1-millisecond CPU sample (with a call stack) on every processor, plus the CLR’s key events. ¹ The overhead of collection is generally reported to be around a few percent with the default configuration. ⁷

5.2 Reading CPU Stacks

Open the collected result (.etl.zip), select the target process under “CPU Stacks,” and the stack viewer opens. The first thing to look at is two columns in the By Name tab.

• Exc (exclusive): the number of samples where that function itself was executing. The place actually consuming CPU.
• Inc (inclusive): the total number of samples for that function plus everything it called. Shows that something further down the call tree is heavy.

The standard way to read this is to sort by Exc and, from the top, sort out “is this my own code, or the runtime/a library?” If your own code is high on the Exc list, that function’s algorithm or loop is the direct target. If something like System.String internals or a JSON serializer is high on Exc instead, follow it back through the Callers tab using Inc, and pin down where in your own code it’s being called so heavily.

Another crucial thing to check is the sample count itself. Since CPU samples are a 1 ms-interval statistic, a low sample count leaves you at the mercy of chance. As a rule of thumb, wait until you’ve collected at least 1,000 samples total (ideally around 5,000) before drawing conclusions, and if you don’t have enough, extend the collection time by repeating the target operation. ¹

One more thing: if a function name from your own module shows up only as an address, its symbols (PDB) haven’t been resolved. Keeping the build artifacts’ PDBs on hand is, in itself, what makes an investigation possible at all. This is covered in detail in “What Is a PDB (Program Database)?”

6. “Slow Even Though the CPU Is Idle” — Investigating Blocked Time with ThreadTime

More than half of real-world “slowness” isn’t about the CPU at all — it’s about waiting. Waiting on a lock, waiting on a DB or HTTP response, file I/O, a deadlock-ish wait via Task.Result. Because this class of problem shows up in CPU samples as “time when the CPU wasn’t being used at all,” the default collection won’t reveal the cause.

In PerfView, enabling the Thread Time option during collection additionally records context-switch events, letting you follow how much time each thread spent “using the CPU” versus “blocked,” combined into one view. ¹

PerfView collect /nogui /acceptEULA /threadTime /maxCollectSec:30

For analysis, open the “Thread Time Stacks” view. It’s the same stack viewer as CPU Stacks, except the samples now include BLOCKED_TIME (time spent blocked) as well as CPU time. Narrow down to the time range of the slow operation, and look at which stack (which wait) accumulates the BLOCKED_TIME of the thread that handled the work — this reveals a breakdown like “35 of the 40 total seconds were spent waiting on this HTTP call.”

Symptoms where the UI freezes (the app stops responding for a few seconds after an action) are almost always, at their core, the UI thread being blocked. Once you’ve used ThreadTime to identify what’s blocking the UI thread, the permanent fix is to redesign the synchronous wait as asynchronous. For the design side of that, see “A One-Page Cheat Sheet for async and the UI Thread in WPF/WinForms.”

One caution: because ThreadTime collection records an event for every single context switch, the data volume is noticeably larger than the default collection. Keep the collection window short, and avoid long-running ThreadTime collections in production.

7. Combining It with EventSource — Slicing “Which Section Is Slow” in Business Terms

Both CPU Stacks and Thread Time are aggregated over the whole process. If you want to slice things at the level of a single business operation — “where inside this one order-processing run is the time going” — the EventSource Start/Stop events from the previous article come into play.

• On the app side, instrument events like OrderProcessingStart / OrderProcessingStop (see the previous article for the implementation)
• Check the timing of that event in PerfView’s “Events” view, and narrow the stack viewer’s time range (the Start/End filter) to that window
• Read the CPU/blocked-time breakdown within that narrowed range

This lets you analyze just “that one slow run” in isolation, without it getting buried in ordinary-time noise. Since custom events can be collected the same way with dotnet-trace, once you’ve added the instrumentation, the same investigation technique works on both a dev machine and in production. In practice, the difficulty of a performance investigation comes down less to how skilled you are with the tools, and more to whether the app has observation points built in.

Summary

• The starting point is separating “slow” into CPU-bound and blocked. The former only shows up with CPU sampling (1 sample ≈ 1 ms); the latter only shows up with ThreadTime collection.
• The basic split is dotnet-trace first, since it needs no administrator privileges, and PerfView when you need the whole machine, native code, or blocked time. Collecting with dotnet-trace and then reading the result in PerfView is also a standard combination.
• The core of reading the stack viewer is the distinction between exclusive and inclusive, and you should always check whether you have enough samples.
• Setting up custom EventSource events that mark off a business operation’s boundaries takes the precision of the investigation up another notch.

Komura Software LLC handles investigating Windows business app performance problems — “slow,” “frozen,” “the CPU is pegged” — as well as consulting on the instrumentation and design work that makes an app easier to measure in the first place. If you have a reproduction procedure and a trace file in hand, we can also take on the analysis alone.

Related Articles

• An Introduction to Windows Event Log and ETW — Putting Business App Logs on the OS’s Standard Mechanism
• Telling GC Pauses Apart from Memory Leaks in .NET — Practical Steps for Observing, Comparing, and Proving Growing Memory
• Reading Crash Dumps with WinDbg + SOS — A Practical Introduction to Analysis After Collection
• An Introduction to Collecting Windows Crash Dumps — WER/ProcDump/WinDbg
• What Is a PDB (Program Database)? — Understanding Debug Info, Symbols, and Source Link
• How to Correctly Compare Program Speed Across Versions on Windows
• A One-Page Cheat Sheet for async and the UI Thread in WPF/WinForms

Related Consulting Areas

• Bug Investigation & Root Cause Analysis
• Windows App Development
• Technical Consulting & Design Review
• Contact

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

Should I use PerfView or dotnet-trace?

Start by trying dotnet-trace first. It works without administrator privileges, lets you target just the process you care about, and the whole operation fits into a single command line. On the other hand, if you need to see native code stacks, the state of the whole machine (the influence of other processes or the kernel), or want to trace blocked time at the level of individual context switches, you need ETW-based PerfView. It's also common practice to collect with dotnet-trace and then open the resulting .nettrace file in PerfView for analysis.

How does this relate to the Visual Studio profiler?

If the problem reproduces on a dev machine, Visual Studio's CPU Usage tool and memory tools are more convenient, and that alone is usually enough. PerfView and dotnet-trace earn their keep when you need to collect on a test or production machine where you can't install Visual Studio, when you want to split collection and analysis across separate machines, or when you need to see ETW events — including custom EventSource events or kernel events. Visual Studio can also open the .nettrace files that dotnet-trace produces.

Is it safe to run these on a production server?

Both tools are designed with short-duration collection on production in mind, but that's not unconditional. Follow this checklist: cap the collection duration at tens of seconds to a few minutes, first try the same command in a staging environment to check the overhead and file size, and confirm there's enough free disk space. ThreadTime (context switch) collection and allocation tracing in particular generate a large volume of events, so files grow quickly. Also, since a trace can include information such as command-line arguments, handle the resulting file with the same care you would any sensitive artifact.

How do WPR/WPA fit into this?

WPR (Windows Performance Recorder) and WPA (Windows Performance Analyzer) are more OS-oriented investigation tools built on the same ETW foundation. For detailed, OS-wide analysis — disk I/O, power, boot time, and so on — WPR/WPA is the stronger choice. For investigating CPU, GC, and blocked time in a .NET app, PerfView is easier to read since its display is optimized for managed code. If your goal is investigating a business app, PerfView and dotnet-trace cover the vast majority of cases.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.