How to Correctly Compare the Speed of Different Program Versions on Windows

You want to compare version A and version B of a program on Windows. The single worst thing you can do is run each once on the same machine and declare “B seems about 8% faster.”

That 8% might genuinely be the code difference. But in reality it turns out to be one of power mode, power plan, thermals, background updates, search indexing, virus scans, affinity, execution order, or cache state – that is the classic Windows benchmarking story. Getting past it is unglamorous work: eliminate the conditions one at a time.

flowchart TB
    accTitle: What the 8% faster claim really is
    accDescr: Diagram showing that a difference produced by running each version once may be a code difference but is often power, thermals, background activity, or cache, so the conditions have to be eliminated one at a time.
    one1["Run each once and compare"] --> dif2["Looks about 8% faster"]
    dif2 -->|"could be"| code1["A genuine code difference"]
    dif2 -->|"common culprit"| env1["Power, thermals, noise, cache"]
    env1 --> crush1["Eliminate the conditions one at a time"]

Figure 1: A one-shot difference is not necessarily a code difference, and you cannot claim it until the conditions are eliminated.

This article summarizes how to compare the execution speed of different versions of a program on Windows in a form as close to the code difference as possible. The main target is Windows 11, but most of it – powercfg, start, and so on – works the same on Windows 10.

Terms to know up front

Some terms appear in English throughout the text. Here they are up front so the first mention does not trip you up.

The Conclusion First

The keys to better reproducibility boil down to these six.

1. Decide first what you want to compare Whether you want to see the code difference or the real user experience changes which environment factors you need to align.

2. Record power mode and power plan as separate things Handle this sloppily on Windows and the comparison tends to become a comparison of the OS power-saving policies.

3. Separate the cold first run from the warmed-up steady state “Only the first run is fast” or “only the later runs are slow” is not unusual.

4. Alternate the runs: A, B, A, B Run all of A first and then all of B, and you take on the skew from thermals and background state.

5. Look at the median and the spread, not just the mean A single outlier badly distorts the whole picture. The mean is more fragile than you think.

6. If the difference is small, dig down to the cause with ETW / WPR Argue from gut feel and the discussion just runs in parallel, with neither side’s claim backed by anything.

Knowledge map for this article

This article takes reproducibility as its theme when comparing the speed of two versions of a program on Windows, and shows that pinning and recording the two layers of power settings, Power mode (the overlay) and Power plan, is the precondition. Differences in Power mode govern PPM behavior such as core parking, and on devices that support Modern Standby the Power plan choices themselves are restricted to the Balanced family. Background work such as the search indexer, Defender scans, and notifications can be a source of variance in the measurements, so reduce it with exclusions, a clean boot, and suppressed notifications. Measure wall-clock time, CPU time, and cycle count with QueryPerformanceCounter, GetProcessTimes, and QueryProcessCycleTime respectively, and when the difference is small and the reason cannot be read from it, the recommendation is to capture an ETW trace with WPR and compare it in WPA.

flowchart LR
    accTitle: Reproducible benchmark comparisons between program versions on Windows
    accDescr: Diagram showing that pinning and recording the two layers of power settings, Power mode and Power plan, is the precondition for benchmark reproducibility, that background work such as the search indexer and Defender scans creates variance in the measurements and can be reduced with exclusions and a clean boot, that wall-clock time, CPU time, and cycle count are each measured with a dedicated API, and how to dig into the cause with WPR and WPA when the difference is small
    benchmark_reproducibility["Benchmark Reproducibility"]
    power_mode["Power Mode (Processor Power Management)"]
    power_plan["Power Plan"]
    powercfg["powercfg"]
    modern_standby["Modern Standby Device"]
    core_parking["Core Parking"]
    search_indexing["Windows Search Indexing"]
    benchmark_result_variance["Benchmark Result Variance"]
    exclusion_setting["Scan Exclusion Settings (Folder Exclusion)"]
    clean_boot["Clean Boot"]
    do_not_disturb["Do Not Disturb (Notification Suppression)"]
    wpr["Windows Performance Recorder (WPR)"]
    etw["ETW (Event Tracing for Windows)"]
    wpa["Windows Performance Analyzer (WPA)"]
    inconclusive_benchmark_result["Inconclusive Benchmark Results"]
    wall_clock_time["Wall-Clock Time"]
    queryperformancecounter["QueryPerformanceCounter (QPC)"]
    cpu_time["CPU time (user + kernel)"]
    getprocesstimes["GetProcessTimes"]
    cycle_count["CPU cycle count"]
    queryprocesscycletime["QueryProcessCycleTime"]
    process_priority["Priority Class"]
    start_command["start Command"]
    processor_affinity["Processor Affinity (Affinity Mask)"]
    processor_groups["Processor Groups"]
    benchmark_reproducibility -->|"requires"| power_mode
    benchmark_reproducibility -->|"requires"| power_plan
    power_plan -->|"configured by"| powercfg
    modern_standby -.->|"incompatible with"| power_plan
    core_parking -.->|"configured by"| power_mode
    search_indexing -.->|"may cause"| benchmark_result_variance
    exclusion_setting -.->|"mitigates"| benchmark_result_variance
    clean_boot -->|"mitigates"| benchmark_result_variance
    do_not_disturb -.->|"mitigates"| benchmark_result_variance
    wpr -->|"uses"| etw
    wpa -->|"uses"| etw
    wpa -.->|"requires"| wpr
    wpr -->|"recommended for"| inconclusive_benchmark_result
    wall_clock_time -->|"verified by"| queryperformancecounter
    cpu_time -->|"verified by"| getprocesstimes
    cycle_count -->|"verified by"| queryprocesscycletime
    process_priority -->|"configured by"| start_command
    processor_affinity -->|"configured by"| start_command
    processor_affinity -.->|"requires"| processor_groups
    benchmark_reproducibility -.->|"uses"| wall_clock_time
    benchmark_reproducibility -.->|"uses"| cpu_time
    benchmark_reproducibility -.->|"uses"| cycle_count
    benchmark_result_variance -->|"incompatible with"| benchmark_reproducibility
    benchmark_reproducibility -.->|"may cause"| inconclusive_benchmark_result
    benchmark_reproducibility -.->|"uses"| start_command

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (25 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

Decide First What You Want to Compare

“Speed comparison” sounds like one thing, but there are actually two kinds.

1. A comparison to see the code difference

You want to know whether the implementation itself got faster through an algorithm change, a data structure change, compiler optimization, a runtime update, and so on.

In this case, cut environmental noise as much as possible. A dedicated benchmarking session, a pinned power mode, notifications off, search indexing and sync suppressed, and a clean boot if necessary.

2. A comparison to see the real user experience

You want to know the speed users will actually feel on their everyday Windows after release.

In this case, you must not erase all the noise that exists in reality. Comparing in a plausible everyday environment that includes OneDrive sync, Defender, notifications, and normal power settings gives results closer to reality.

Mix these two and the conclusions get twisted. Outcomes such as “12% faster in the lab but within noise in the real world” or “faster in the real world but unchanged in CPU time” happen routinely.

flowchart TB
    accTitle: Do not mix the two kinds of comparison
    accDescr: Diagram showing that a comparison aimed at the code difference should cut environmental noise while a comparison aimed at the real user experience should keep the noise of an everyday environment, and that mixing the two twists the conclusion.
    q5["What do you want to compare"] -->|"Code difference"| lab1["Lab environment with noise removed"]
    q5 -->|"Real user experience"| real1["Everyday environment with noise left in"]
    q5 -.->|"if mixed"| twist2["The conclusion gets twisted"]

Figure 2: The environment you need to align is the opposite depending on the goal, so decide which comparison this is before you start.

The Main Causes of Variance on Windows

First, a rough inventory of what makes results wobble.

In short: even the same Windows machine is a different experiment if the conditions are not aligned.

flowchart TB
    accTitle: Unaligned conditions make it a different experiment
    accDescr: Diagram showing that measuring on the same Windows machine is effectively a different experiment unless the multi-layer conditions from hardware through power, thermals, background activity, and build settings are aligned, and that pinning and recording them is what makes it a comparison.
    same1["Measure on the same Windows machine"] -.->|"conditions not aligned"| oth1["Effectively a different experiment"]
    same1 -->|"Pin and record the multi-layer conditions"| cmp1["Only now is it a comparison"]

Figure 3: Even with the same machine, the comparison does not hold unless the conditions across every layer are aligned.

Treat Power Mode and Power Plan Separately

This part matters a lot.

Windows has the Power mode in the Settings app and the traditional Power plan (the power schemes visible via powercfg). They look similar and tend to get lumped together, but handle them sloppily and the comparison conditions become vague, which destroys the reproducibility of the result.

In the Windows Settings app, you choose the power mode from Settings > System > Power & battery. Microsoft’s documentation states you can switch between Best power efficiency, Balanced, and Best performance separately for Plugged in and On Battery. Furthermore, changing the power mode also affects the underlying power-related settings and PPM (Processor Power Management) behavior. In other words, this difference alone can change core parking and performance scaling policy.

The power plan, on the other hand, is the traditional power scheme: Balanced, High performance, and so on. You can check it with powercfg /list and powercfg /getactivescheme.

The confusing part is that Windows has both a power mode overlay and a power plan. Drawn as a diagram, the relationship looks like this.

flowchart TB
    subgraph upper["Upper layer: Power mode - overlay"]
        direction LR
        M1["Best power efficiency"]
        M2["Balanced"]
        M3["Best performance"]
    end

    subgraph lower["Lower layer: Power plan"]
        direction LR
        P1["Balanced"]
        P2["High performance"]
        P3["custom plan"]
    end

    UI["Settings app<br/>Power mode in Power and battery"] --> upper
    CLI["Switch with powercfg /setactive"] --> lower
    upper --> PPM["The power settings that actually take effect<br/>PPM and graphics subgroups"]
    lower --> PPM
    AC["AC power or battery"] --> PPM
    PPM --> RESULT["Frequency ceiling / core parking / performance scaling"]

Figure 4: The power mode overlay, the power plan, and AC/DC combine to determine the power settings that actually take effect.

Looking at only the upper layer or only the lower layer does not tell you the actual behavior. So record at least the following with your benchmark results.

• AC or battery
• Which power mode
• Which active power plan

A benchmark result that does not state these three cannot have its conditions reconstructed when you look back at it later.

flowchart TB
    accTitle: The three things to record at minimum
    accDescr: Diagram showing that unless AC or battery, the power mode, and the active power plan are recorded with the result, the conditions cannot be reconstructed when the result is reviewed later.
    r1["AC or battery"] --> rec2["Record together with the result"]
    r2["Power mode"] --> rec2
    r3["Active power plan"] --> rec2
    rec2 --> rst1["The conditions can be reconstructed later"]

Figure 5: These three power-related items are the minimum record without which the whole result becomes irreproducible.

Power conditions to pin down first

1. Always compare laptops on AC power Battery operation easily introduces unintended limits.

2. Pin the power mode For benchmarking, try Best performance first.

3. Record the active power plan Save the current value with powercfg.

powercfg /list
powercfg /getactivescheme

Microsoft’s documentation shows the output of powercfg /list in the following form. The line for the active plan ends with *. On a Japanese-language system, the heading and the plan names are printed in Japanese.

Existing Power Schemes (* Active)
-----------------------------------
Power Scheme GUID: {guidPlan1}  (Balanced) *
Power Scheme GUID: {guidPlan2}  (Power saver)

Copy the GUID printed here straight into the power_plan field of your result file. The point is to keep the GUID rather than the name, because two plans both called “Balanced” can be different plans that were duplicated or customized.

1. Switch to High performance if needed

# Balanced
powercfg /setactive 381b4222-f694-41f0-9685-ff5bb260df2e

# High performance
powercfg /setactive 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c

Can the power mode be switched from the command line?

This is an easy place to get stuck. The published list of powercfg command-line options contains no option that selects the power mode overlay itself. The supported way to switch it is the Settings app, at Settings > System > Power & battery.

That said, powercfg does support reading and writing the setting values of overlay schemes. The documentation states the following.

• Passing an overlay alias and subgroup to powercfg /q reads the settings on the overlay side
• powercfg /setacvalueindex and /setdcvalueindex can also be used against overlay schemes
• If no scheme is specified, the target is the currently active overlay (or the current power plan when there is no overlay)
• The list of aliases is available from powercfg /aliases

So what the command line can do is read and adjust the contents of the overlay that is currently in effect, not change which overlay is selected. As a reproducible benchmark procedure, the practical approach is to pin the power mode by hand in the Settings app and write that value into the results. State in the procedure document that “Power mode was set to Best performance,” and confirm it on screen before every run.

flowchart TB
    accTitle: What powercfg can and cannot do
    accDescr: Diagram showing that powercfg can read and write the setting values of the overlay currently in effect but has no option to change which overlay is selected, so pinning the power mode by hand in the Settings app and recording the value is the practical approach.
    pc1["powercfg"] -->|"can do"| rw1["Read and adjust the contents of the overlay"]
    pc1 -.->|"cannot do"| sel1["Change which overlay is selected"]
    sel1 --> hand1["Pin it by hand in the Settings app and record it"]

Figure 6: Overlay selection cannot be changed from the command line, so pin it by hand and record it.

“High performance does not show up” is completely normal

This is another stumbling point. Microsoft’s documentation states that on Modern Standby devices, only Balanced, or plans derived from Balanced, are allowed. So instead of “High performance is missing, is it broken?”, the likely answer is that the machine is designed that way.

Microsoft also advises that if the power mode cannot be changed, a custom power plan may be selected, so try choosing Balanced first. When the power mode UI does not respond, that is the quickest thing to suspect.

flowchart TB
    accTitle: How to read a missing High performance plan
    accDescr: Diagram showing that on Modern Standby devices only Balanced or plans derived from it are permitted so a missing High performance plan is by design, and that when the power mode UI cannot be changed the first thing to suspect is that a custom plan is selected.
    nohp1["High performance is nowhere to be found"] --> ms1["By design if the device supports Modern Standby"]
    nomv1["The Power mode UI does not respond"] --> cst1["Suspect that a custom plan is selected"]
    cst1 --> bl1["Try selecting Balanced first"]

Figure 7: A missing plan or an unresponsive UI is not necessarily a fault; look first at the model design and the selected plan.

Kill the Background Noise

Even when you want to measure quietly, Windows keeps running updates, indexing, and scans in the background. Start by reducing how much of that happens.

First, reboot and wait for things to settle

After changing settings, reboot once, and do not start running immediately after login – wait a few minutes. Right after startup, updates, indexing, sync, Defender, and assorted resident processes are still busy.

flowchart TB
    accTitle: Reboot and wait until things settle
    accDescr: Diagram showing the procedure of rebooting once after a settings change and waiting a few minutes after login before measuring, because updates, indexing, sync, and Defender are active right after startup.
    chg1["Change the settings"] --> rb1["Reboot once"]
    rb1 --> wt1["Wait a few minutes after login"]
    wt1 --> ms2["Then start measuring"]
    rb1 -.-> nzz1["Resident processes are still busy right after startup"]

Figure 8: Start measuring only after background activity has settled following the reboot.

For strict comparisons, use a clean boot

Microsoft documents a procedure for reducing the system to a minimal startup configuration with a clean boot: stop non-Microsoft services in msconfig and disable startup apps in Task Manager.

This is powerful for reducing noise. It does move you away from the everyday environment, however, so it suits a lab comparison aimed at seeing the code difference.

Silence notifications

Windows notification banners look harmless but are surprisingly disruptive. Beyond the visual nuisance, they can change execution timing, focus, and background app activity.

Enable Do not disturb manually, or at minimum turn notifications off during the benchmark.

Suppress search indexing and sync

If the benchmark target reads large numbers of files, writes large amounts of output, or rebuilds a source tree repeatedly, search indexing and cloud sync quietly cut into the numbers.

• Exclude the benchmark directory from search indexing
• Stop OneDrive / Dropbox / Google Drive sync
• Close browsers, Teams, Discord, Slack

None of this is dramatic, but when it matters, it matters a lot.

flowchart TB
    accTitle: Noise that hits file-heavy benchmarks
    accDescr: Diagram showing that for benchmarks that read and write large numbers of files, search indexing, cloud sync, and resident applications all cut into the measurement, so exclusions and stopping them reduce the noise.
    ix1["Search indexing"] --> hit1["Hits benchmarks that touch many files"]
    sy1["Cloud sync"] --> hit1
    ap2["Resident applications"] --> hit1
    hit1 --> cutn1["Reduce noise with exclusions and by stopping them"]

Figure 9: The more files a benchmark reads and writes, the more stopping indexing and sync pays off.

A Comparison That Does Not Align Thermals Is Mostly Comparing Thermals

CPUs and GPUs run at different clock speeds when cold and when warmed up. The same code therefore runs under different conditions on every execution. Laptops, thin mini PCs, and small desktops show this most clearly.

flowchart TB
    accTitle: How thermals change the conditions
    accDescr: Diagram showing that CPUs and GPUs change clock speed between the cold state and the warmed-up state so the same code runs under different conditions on every execution, which means a comparison that does not align thermals is mostly comparing thermals.
    cold1["Run while the machine is cold"] --> hot1["The clock changes once it warms up"]
    hot1 --> vary1["Conditions change on every run"]
    vary1 -.-> heatc1["An unaligned comparison is comparing thermals"]

Figure 10: Clocks move with temperature, so without aligned thermal conditions you compare cooling rather than code.

Rules to follow

• Keep room temperature as consistent as possible
• Fix how the laptop is positioned
• Fix the AC adapter, dock, and external display configuration
• Do no heavy work right before the benchmark
• Measure the first run and the steady state separately

Alternate the execution order

Avoid running A 10 times and then B 10 times. The skew from thermals, caches, and background activity rides along with it.

Any of the following works well.

• A B A B A B ...
• A B B A A B B A ...
• Pre-generate a random order and run in that order

flowchart TB
    accTitle: Execution order changes where the skew lands
    accDescr: Diagram showing that running all of A first and then B puts the skew from thermals, caches, and background activity on only one side, while alternating or randomizing the order spreads the skew across both.
    seq1["All of A then all of B"] --> bias1["The skew lands on one side only"]
    alt1["Alternating A B A B or random order"] --> even1["The skew spreads across both"]
    even1 --> fair1["The effect of order can be removed from the difference"]

Figure 11: Grouping the runs means comparing the skew as well, so alternate or randomize.

What You Measure Changes What “Fast” Means

Squash “fast” into a single number and it usually goes wrong. The three representative metrics to look at on Windows are these.

1. Wall-clock time

The time the user waits. It is closest to the end-to-end experience, so this is the first value to look at.

On Windows, QueryPerformanceCounter (QPC) is available for high-resolution timing. In managed code, the Stopwatch family is the standard choice. Eyeballing milliseconds with DateTime.Now is, frankly, a bit defenseless.

2. CPU time (user + kernel time)

The time the process actually used the CPU, obtainable via GetProcessTimes.

This is useful for looking at computational efficiency. For example, if wall-clock improved but CPU time did not change, caches, I/O, wait time, or scheduling may be what is doing the work.

3. Cycle count (CPU cycles)

QueryProcessCycleTime gives you the CPU cycle count for the whole process.

This is also a CPU-work metric, but it shows a different face than wall-clock. It is particularly useful when you want to ask whether the wait time is the same while the computation itself got lighter.

flowchart TB
    accTitle: The different faces the three metrics show
    accDescr: Diagram showing that wall-clock time is the time the user waits, CPU time is the time the process actually used the CPU, and cycle count shows a different face again, so only the combination reveals what the speed difference consists of.
    spd1["Look at what fast consists of"] --> w1["wall-clock: the time spent waiting"]
    spd1 --> u1["CPU time: the CPU time consumed"]
    spd1 --> cy1["cycle: how heavy the computation is"]
    w1 -.-> mixr1["Infer the reason from the combination"]

Figure 12: Do not squash it into one number; read the meaning of speed from the combination of the three metrics.

Priority, Affinity, and NUMA Are Last Resorts

These can have an effect. But reaching for them from the start just because they can have an effect tends to create a different phenomenon.

First, measure normally

If a difference shows up in the default state, that difference itself has value. Throwing in /high or /affinity right away imports conditions that do not occur on real Windows.

flowchart TB
    accTitle: Priority and affinity are last resorts
    accDescr: Diagram showing that measuring in the default state comes first because a difference that appears there has value, that reaching for priority or affinity immediately imports conditions that do not occur on real Windows, and that they should be pinned last with a clear purpose.
    def1["Measure in the default state first"] --> val1["A difference that appears there has value"]
    early1["Reaching for /high or /affinity right away"] -.-> art1["Imports conditions that do not occur in reality"]
    val1 -->|"when needed, with a defined purpose"| lastr1["Pin them as a last resort"]

Figure 13: Use priority and affinity with a purpose, after the default-state measurement is done.

If you use them, be clear about the purpose

• /high: you want fewer disturbances from other processes
• /affinity: you want to pin CPU placement for the comparison
• NUMA control: you want to align memory locality on large machines

The Windows start command can launch a program with a priority class and an affinity mask.

start "" /high /wait myapp.exe --bench case1.json
start "" /affinity F /high /wait myapp.exe --bench case1.json

But skip /realtime

/realtime is available, but you should not use it. It tends to create new problems rather than remove noise.

A Recommended Measurement Procedure

Putting all of the above together, here is a procedure that is easy to run in practice.

Lab-leaning comparison procedure

1. Fix the comparison targets
   • commit hash / build number
   • compiler / runtime version
   • Debug / Release
   • logging, asserts, tracing on or off
2. Fix the machine conditions
   • Windows build
   • BIOS / UEFI version
   • driver version
   • AC power
   • room temperature, physical placement
3. Fix the power conditions
   • Decide the power mode
   • Record the active power plan
4. Reboot
5. Wait a few minutes before benchmarking
6. Clean boot if necessary
7. Include a warm-up
8. Alternate A / B runs
9. Get enough repetitions
10. Keep median, min, max, and p95
11. Save the raw data
12. If the difference is small, capture ETW / WPR

How many repetitions

Step 9, “get enough repetitions,” needs a rule of thumb too. The following is not a statistically rigorous answer but where practice tends to settle.

Estimate the elapsed time as time per run x number of runs x number of versions + warm-up. For a 30-second operation run 30 times each for A and B, that works out to roughly 35 minutes including warm-up. When that is not practical, a sounder approach than cutting the run count is to narrow what you measure by carving out only the heavy stage.

If you are unsure where to stop, a clear method is to increase the run count while watching how the median moves, and stop once adding runs no longer moves it.

flowchart TB
    accTitle: Where to stop adding runs
    accDescr: Diagram showing the practical way to decide the number of runs by increasing the count while watching how the median moves and stopping once additional runs no longer move it.
    add2["Increase the run count"] --> mdz1["Watch how the median moves"]
    mdz1 -->|"still moving"| add2
    mdz1 -->|"no longer moves"| stop1["Stop there"]

Figure 14: Even if you cannot fix the run count in advance, the point where the median settles works as a stopping rule.

Items Worth Recording That Save You Later

In the benchmark CSV or JSON, keeping at least the following puts you in a strong position.

timestamp,version,scenario,elapsed_ms,user_ms,kernel_ms,cycles,power_mode,power_plan,ac_or_dc,room_temp_c,notes

If possible, these are handy as well.

cpu_package_temp_start_c,cpu_package_temp_end_c,affinity_mask,priority_class,windows_build,driver_version

With benchmarks, being interpretable later often matters more than the measuring itself.

Look at the Median and the Distribution, Not Just the Mean

The mean is convenient, but it breaks easily in Windows benchmarks. Defender kicking in just once, a notification appearing, another process hammering the SSD – any of these can carry the mean away.

flowchart TB
    accTitle: Outliers carry the mean away
    accDescr: Diagram showing that a single Defender scan, notification, or burst of I/O from another process is enough to carry the mean away, so the median should be the axis, combined with p95, p99, and min and max to read the distribution.
    once3["Noise appears just once"] --> avg1["The mean gets carried away"]
    med2["Make the median the axis"] --> dist1["Also look at p95 / p99 and min / max"]
    dist1 --> robust1["A reading that resists outliers"]

Figure 15: The mean breaks with a single burst of noise, so read the median together with the distribution.

The recommended combination is this.

• Median: look at this first
• p95 / p99: check whether the tail has gotten worse
• min / max: see how far the outliers stray
• Box plots or scatter plots: useful when the difference is small

How to Read a Difference When You See One

Interpreting results is easiest when you look at combinations.

Only wall-clock is faster

This may be an improvement in I/O, wait time, caches, or scheduling.

CPU time and cycles both dropped

There is a good chance the implementation itself got lighter.

Only the first run is slow or fast

That is the cold versus warm difference. Suspect startup, initialization, cache generation, or JIT.

It gets slower the more runs you do

Suspect thermals, throttling, memory pressure, or background activity.

flowchart TB
    accTitle: Reading the cause from the pattern of the difference
    accDescr: Diagram showing how to read the cause from the pattern of the difference: only wall-clock faster points to waits and I/O, both CPU time and cycles dropping points to a lighter implementation, a difference only on the first run points to cold versus warm, and slowing down over successive runs points to thermals or background activity.
    pt2["Look at how the difference appears"] -->|"wall-clock only"| c1a["Waits and I/O"]
    pt2 -->|"CPU time down too"| c2a["Lighter implementation"]
    pt2 -->|"first run only"| c3a["cold / warm"]
    pt2 -->|"slower later on"| c4a["Thermals and background activity"]

Figure 16: The pattern of the difference, more than the difference itself, points to the cause.

Dig Down to “Why It Is Faster” with ETW / WPR

When the difference is small, or when the reason is unreadable, moving on to the Windows ETW (Event Tracing for Windows) tooling is the standard route.

Microsoft’s Windows Performance Recorder (WPR) is an ETW-based recording tool included in the Windows ADK. It captures CPU, I/O, context switches, page faults, and more in one go.

A minimal capture looks like this.

wpr -start CPU -filemode

REM Run the benchmark here

wpr -stop trace.etl

Once the trace is open in WPA, the graphs to look at first are largely fixed.

For a comparison, the basic approach is to capture one trace each for A and B under the same scenario and view the same graphs side by side. A single trace on its own does not let you judge whether something is slow.

Once you reach this stage, instead of “B is 3% faster” you can say things like “B has less lock waiting and lower ready time” or “A opens more files and has a slower cold start” – that is, you can talk about the difference with a reason attached.

flowchart TB
    accTitle: From a bare number to a difference with a reason
    accDescr: Diagram showing the flow of capturing one WPR trace each for A and B under the same scenario when the difference is small or unreadable, comparing the same graphs side by side in WPA, and thereby being able to talk about why rather than how many percent.
    small2["Difference is small or the reason is unreadable"] --> tr1["Capture WPR traces for A and B"]
    tr1 --> cmp2["View the same graphs side by side in WPA"]
    cmp2 --> rsn1["You can talk about it with a reason"]
    cmp2 -.-> onen1["One trace alone cannot tell you whether it is slow"]

Figure 17: Dig as far as ETW and “how many percent faster” turns into “why it is faster.”

The Whole Checklist on One Page

Finally, here it is in a form you can paste straight into a procedure document.

Pin

• Fixed the comparison targets (commit hash / build number / Debug or Release / build conditions such as PGO / logging and asserts on or off)
• Put the laptop on AC power
• Pinned the power mode in the Settings app
• Checked the active power plan with powercfg /getactivescheme
• Turned off notifications. Stopped search indexing and cloud sync
• Used a clean boot where necessary
• Rebooted and waited a few minutes before starting

Run

• Included a warm-up
• Measured cold (first run) and warm (steady state) separately
• Ran A / B alternately or in random order
• Decided the number of runs and stuck to it (see the table above for guidance)

Record

• Kept raw data with one row per run (elapsed_ms / user_ms / kernel_ms / cycles)
• Kept AC or DC, the power mode, the power plan GUID, the Windows build, and the driver version
• Kept the room temperature and physical placement
• Wrote down the conditions that were not pinned as well

Interpret

• Looked at the median. Did not judge from the mean alone
• Looked at the tail with p95 / p99
• Checked outliers with min / max
• Inferred the reason from the combination of wall-clock, CPU time, and cycles
• Dug as far as ETW / WPR when the difference was small

Summary

When comparing different versions of a program on Windows, what really works is not a flashy trick. What matters is the following unglamorous discipline that pays off in reproducibility.

• Pin and record AC / power mode / power plan
• Separate cold and warm
• Alternate A / B runs
• Look at the median and the distribution
• Clean boot if necessary
• If the difference is small, dig to the reason with ETW / WPR

And most important of all: write down, alongside the results, what you pinned and what you did not. A benchmark is a comparison of speed and, at the same time, a record of experimental conditions.

With a speedup report that states no conditions, nobody else can check whether the same result can be produced. Only the numbers survive; the means of reproducing them does not. Conversely, if the conditions are properly written down, the result has real value even when the difference is small.

flowchart TB
    accTitle: The record of conditions decides the value of the result
    accDescr: Diagram showing that a speedup report without stated conditions leaves only numbers and no means of reproduction while a report with the conditions written down has value even when the difference is small, illustrating that a benchmark is also a record of experimental conditions.
    norec1["Report with no conditions written down"] --> onlyn1["Only the numbers survive"]
    onlyn1 --> norep1["Nobody else can verify it"]
    rec3["Report with the conditions written down"] --> rep1["The means of reproduction survives"]
    rep1 --> worth1["Valuable even when the difference is small"]

Figure 18: The value of a benchmark lies less in the numbers than in the record of what was pinned and what was not.

References

• Microsoft Support: Change the power mode for your Windows PC
• Microsoft Learn: Power Policy Settings
• Microsoft Learn: Customize the Windows performance power slider
• Microsoft Learn: Powercfg command-line options
• Microsoft Support: How to perform a clean boot in Windows
• Microsoft Support: Notifications and Do Not Disturb in Windows
• Microsoft Support: Search indexing in Windows
• Microsoft Learn: Configure custom exclusions for Microsoft Defender Antivirus
• Microsoft Support: Device Security in the Windows Security App
• Microsoft Learn: QueryPerformanceCounter function
• Microsoft Learn: Acquiring high-resolution time stamps
• Microsoft Learn: GetProcessTimes function
• Microsoft Learn: QueryProcessCycleTime function
• Microsoft Learn: start command
• Microsoft Learn: SetPriorityClass function
• Microsoft Learn: SetProcessAffinityMask function
• Microsoft Learn: Processor Groups
• Microsoft Learn: Windows Performance Recorder
• Microsoft Learn: WPR Command-Line Options
• Microsoft Learn: CPU Analysis in Windows Performance Analyzer - which graph to use for what.
• Microsoft Learn: Set the Default Power Plan - example output of powercfg -LIST.

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What are the main causes of unstable benchmark results on Windows?

The factors span several layers: power mode and power plan, thermals, background updates, search indexing, virus scans, priority and affinity, execution order, and cache state. Even on the same Windows machine, if those conditions are not aligned you are effectively running a different experiment. On laptops in particular, behavior changes drastically depending on whether the machine is on AC power or running on battery, so always compare on AC power and record the conditions.

What is the difference between power mode and power plan?

Power mode is the Best power efficiency / Balanced / Best performance selector in the Power and battery page of the Settings app, and it affects the underlying power-related settings and PPM (Processor Power Management) behavior. Power plan is the traditional power scheme such as Balanced or High performance that you can inspect with powercfg. Windows has both, so a benchmark result needs at minimum three things recorded: AC or battery, the power mode, and the active power plan.

Is it a fault if the High performance power plan does not appear?

Most likely it is not a fault. Microsoft's documentation states that on Modern Standby devices, only Balanced or plans derived from Balanced are permitted. In other words, it is perfectly normal for High performance not to appear on a given model by design. And if the power mode UI cannot be changed, a custom power plan may be selected, so the quickest check is to try selecting Balanced first.

In what order should I run the speed comparison between version A and version B?

Avoid running all of A first and then all of B, because the skew from thermals, caches, and background activity then lands on only one side. Run them alternately as A B A B, or execute a randomized order generated in advance. Also measure the cold first run separately from the warmed-up steady state, and look at the median, p95, minimum, and maximum rather than the mean alone so that outliers do not carry the result away.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.