Where to Look When a PowerShell Script Is Slow — Arrays, Pipelines and Matching

“The aggregation script used to finish in a few minutes; now the data has grown and it takes three hours.” This is a common story in places where PowerShell automation has become established. And in most cases, the cause of the slowness is not PowerShell’s raw execution speed but the way the code is written.

PowerShell is a language that prioritises ease of writing, so it contains plenty of ways to write things where “the natural, straightforward approach happens to have terrible complexity”. The flagship example is $array += $item. It looks natural, but internally a full copy of the array runs every time, and it slows down sharply as the item count grows. Knowing these classic pitfalls is what decides whether the same job takes hours or tens of seconds.

This article organises the places to suspect first when you’re looking at a slow script, in order of impact. It also covers how to measure properly, so you don’t rewrite code based on assumptions.

1. The Bottom Line First

• Do not use $array += $item. PowerShell arrays are fixed-length, and += creates a new array and copies every element each time. It gets slower in proportion to the square of the item count.¹
• Use List[T] instead, or assign the output of the whole loop to a variable. The latter is idiomatic PowerShell and fast.
• For matching and lookup, hash tables give the biggest win. A nested-loop linear search (O(n×m)) becomes a key lookup (roughly O(n+m)).²
• The pipeline is convenient, but it has a per-object cost. In inner loops over large volumes the foreach statement is often faster, and it’s worth measuring.³
• Choose your file reading method by purpose. Get-Content -Raw reads it all at once; -ReadCount reads in batches. Sometimes it’s the per-line object creation that’s hurting you.⁴
• Use -Filter with Get-ChildItem. Because filtering happens on the provider side, it is more efficient than fetching everything and discarding with Where-Object — the official documentation says so explicitly.⁵
• Concatenate strings with -join or StringBuilder. $s += "..." is slow for the same reason arrays are.⁶
• Format-* belongs only at the very end, for display. Put it in the middle and you break downstream processing as well as paying pointless formatting costs.⁷
• Measure, then fix. Measure-Command discards output, so it doesn’t include display cost. Never use the first run’s value.⁸
• Parallelisation comes last. Consider it after fixing the algorithm (see “PowerShell Parallel Processing”).

2. Measure First — Using Measure-Command Correctly

Tune by guesswork and you’ll usually fix somewhere that doesn’t matter. The first thing to do is measure.⁸

# The first run includes module loading and JIT, so throw it away
$null = Measure-Command { Invoke-KsAggregate }

# Measure the second and later runs a few times and look at the spread too
1..3 | ForEach-Object {
    (Measure-Command { Invoke-KsAggregate }).TotalSeconds
}

There are two things to watch out for with Measure-Command.

• The script block’s output is discarded. The cost of formatting and drawing to the screen is not included. When the perceived slowness is in the real run, the culprit is sometimes the display side
• Compare under the same conditions. Whether the file cache is warm makes a large difference to measurements of anything involving I/O

When you want to break the work down and find which part is slow, use Stopwatch.

$sw = [System.Diagnostics.Stopwatch]::StartNew()
$rows = Import-Csv $csvPath
Write-Verbose "Load: $($sw.ElapsedMilliseconds) ms"; $sw.Restart()

$index = $master | Group-Object Code -AsHashTable -AsString
Write-Verbose "Build index: $($sw.ElapsedMilliseconds) ms"; $sw.Restart()

$result = foreach ($r in $rows) { Join-KsRecord $r $index }
Write-Verbose "Match: $($sw.ElapsedMilliseconds) ms"; $sw.Stop()

Emit per-section timings like this and facts such as “it turns out loading was 80% of it” become obvious immediately. Write-Verbose is used here so that nothing appears in normal runs and it is only visible during an investigation (see “PowerShell Output Streams and Log Design”).

3. The Biggest Culprit — += on Arrays

PowerShell arrays are fixed-length. You cannot add an element to them, and += expands into “create a new array, copy every element into it, and add the new element at the end”.¹ In other words, a loop that adds n items performs copying proportional to n squared overall.

# [BAD] gets dramatically slower as the item count grows
$result = @()
foreach ($row in $rows) {
    $result += ConvertTo-KsRecord $row     # the whole array is copied every time
}

# [GOOD-1] use List[T] (adding an element is constant time)
$result = [System.Collections.Generic.List[object]]::new()
foreach ($row in $rows) {
    $result.Add((ConvertTo-KsRecord $row))
}

# [GOOD-2] assign the output of the whole loop to a variable (idiomatic PowerShell, and fast)
$result = foreach ($row in $rows) {
    ConvertTo-KsRecord $row                # the output is collected as-is
}

Once you get used to it, the [GOOD-2] style is the most natural form in PowerShell. The output of a foreach statement, ForEach-Object, if and so on can be collected straight into a variable. This property goes hand in hand with understanding “output streams”.

The same logic applies to strings. Strings are immutable, so $s += "line" creates a new string every time.

# [BAD]
$text = ''
foreach ($line in $lines) { $text += "$line`r`n" }

# [GOOD-1] -join (the most concise)
$text = $lines -join "`r`n"

# [GOOD-2] StringBuilder (for complex assembly involving conditionals)
$sb = [System.Text.StringBuilder]::new()
foreach ($line in $lines) { [void]$sb.AppendLine($line) }
$text = $sb.ToString()

4. Getting Rid of Nested Loops — Match With a Hash Table

Next in impact is matching. Write “for each row of order data, look up the product name from the master” the straightforward way and you get this.

# [BAD] 10,000 orders x 10,000 master rows = 100 million comparisons
foreach ($order in $orders) {
    $item = $master | Where-Object { $_.Code -eq $order.Code }   # a full scan every time
    $order | Add-Member NoteProperty ItemName $item.Name
}

Switch to a hash table and the number of comparisons drops dramatically.²

# [GOOD] build the index once, then look up by key
$index = $master | Group-Object -Property Code -AsHashTable -AsString

$result = foreach ($order in $orders) {
    $hit = $index[$order.Code]        # key lookup doesn't depend on the item count
    [pscustomobject]@{
        Code     = $order.Code
        Qty      = $order.Qty
        ItemName = if ($hit) { $hit[0].Name } else { $null }   # unregistered codes are detectable too
    }
}

Group-Object -AsHashTable groups elements sharing a key into an array, so the values are arrays (hence $hit[0] above). If you know the key is unique, there is nothing wrong with building the hash table yourself.

$index = @{}
foreach ($m in $master) { $index[$m.Code] = $m }    # assumes unique keys

When all you need is to test “is it in there?” repeatedly, a HashSet is effective. -contains and -in against an array are linear searches, so they start to hurt when the number of tests is large.

$known = [System.Collections.Generic.HashSet[string]]::new(
    [string[]]$master.Code, [System.StringComparer]::OrdinalIgnoreCase)

$unknown = $orders | Where-Object { -not $known.Contains($_.Code) }

Practical patterns for matching CSV files against each other are also covered in “Automating Excel and CSV Business Processing With PowerShell”.

5. The Pipeline versus the foreach Statement

The pipeline is a central PowerShell feature, but there is a per-object cost to flowing between cmdlets. In inner loops on the scale of hundreds of thousands of items, the foreach statement is often faster.³

# Pipeline: readable, and processes sequentially so intermediate results aren't accumulated
$rows | Where-Object { $_.Status -eq 'OK' } | ForEach-Object { $_.Amount } |
    Measure-Object -Sum

# foreach statement: less per-item overhead, faster on large volumes
$sum = 0
foreach ($r in $rows) { if ($r.Status -eq 'OK') { $sum += $r.Amount } }

Memory is easily misunderstood, so a note on it. The foreach statement evaluates the expression in the parentheses first and then begins the loop, so passing the result of a command — as in foreach ($line in (Get-Content $path)) — puts every item in memory at that point.³ On the other hand, if you pass a lazily enumerating IEnumerable, items are enumerated one at a time. So even with an enormous file, writing it like the following lets you keep using the foreach statement without consuming memory.

# Process with the foreach statement without loading everything into memory (lazy enumeration)
foreach ($line in [System.IO.File]::ReadLines($path)) {
    if ($line.StartsWith('ERROR')) { $errors++ }
}

Be careful, though, that the default character encoding differs. The single-argument ReadLines($path) reads as UTF-8 (preferring a BOM if one is present). Windows PowerShell 5.1’s Get-Content, meanwhile, reads using the current ANSI code page if there is no BOM (Shift_JIS in a Japanese environment). In other words, swapping in this replacement for a Shift_JIS log will produce mojibake for Japanese text. When you make the substitution, use the overload that specifies the encoding explicitly.

# When reading a Shift_JIS (CP932) log
# From PowerShell 6.2 onwards the code page provider is already registered,
# so GetEncoding(932) can be called directly (the same applies on Windows PowerShell 5.1)
$enc = [System.Text.Encoding]::GetEncoding(932)
foreach ($line in [System.IO.File]::ReadLines($path, $enc)) {
    if ($line.StartsWith('ERROR')) { $errors++ }
}

Note that in PowerShell 7 the default for Get-Content is UTF-8 (no BOM), so as long as you are handling UTF-8 files on 7, the single-argument overload gives matching results. Before making the substitution, always check the character encoding of the target files and the version of the execution environment.

The rules of thumb are as follows.

.Where() and .ForEach() are array methods added in PowerShell 4.0, and they are lighter because they bypass the pipeline.¹ They do, however, assume the target is already an in-memory collection.

6. File and Directory I/O

For reading, choose according to purpose. Get-Content creates an object per line by default, so with enormous files this cost dominates.⁴

Get-Content $path                      # line by line (one object per line)
Get-Content $path -Raw                 # read the whole thing at once as a single string
Get-Content $path -ReadCount 1000      # pass arrays of 1000 lines (fewer objects created)
[System.IO.File]::ReadLines($path)     # .NET. Sequential enumeration, the lightest (defaults to UTF-8)

For directory traversal, use -Filter. The official documentation states explicitly that “filters are more efficient than other parameters, because the provider applies them when the objects are retrieved”.⁵

# [BAD] fetch everything, then throw most of it away
Get-ChildItem -Path $root -Recurse | Where-Object { $_.Extension -eq '.log' }

# [GOOD] narrow on the provider side
Get-ChildItem -Path $root -Recurse -File -Filter '*.log'

# If it's still slow at the scale of hundreds of thousands of files, consider .NET enumeration
[System.IO.Directory]::EnumerateFiles($root, '*.log', 'AllDirectories')

For writing, appending on every iteration inside a loop is the classic bottleneck. Add-Content opens and closes the file on every call.

# [BAD] 10,000 lines = 10,000 open/close cycles
foreach ($r in $result) { Add-Content -Path $out -Value ($r -join ',') }

# [GOOD-1] write it all in one go
$result | ForEach-Object { $_ -join ',' } | Set-Content -Path $out -Encoding utf8

# [GOOD-2] if you genuinely need incremental writes, keep a StreamWriter open
$writer = [System.IO.StreamWriter]::new($out, $false, [System.Text.UTF8Encoding]::new($false))
try   { foreach ($r in $result) { $writer.WriteLine($r -join ',') } }
finally { $writer.Dispose() }

For file operations across the network, the sheer number of round trips dominates in the first place. For the caveats specific to UNC paths, see “Pitfalls of Network Drives and UNC Paths”.

7. Small Costs That Get Overlooked

• The update frequency of Write-Progress. Update progress on every iteration and the drawing cost can exceed the processing time. Thin it out — every 100 items, say — or stop it entirely in unattended runs with $ProgressPreference = 'SilentlyContinue'
• Inserting Format-Table / Format-List in the middle. They convert to formatting objects, which breaks downstream processing as well as costing you needlessly. Format only at the end⁷
• Invariant work inside loops. Simply hoisting things like Get-Date format string construction, regular expression compilation and module reloading out of the loop can pay off
• Overuse of Select-Object -Property. It creates a new PSCustomObject, which is far from negligible at high volumes. Keeping the original object until the stage you actually need it can be faster
• Frequent exceptions. Designs that run huge numbers of try/catch cycles (calling Get-Item for a non-existent file every time and swallowing the exception, for instance) are expensive. Branch on Test-Path beforehand instead

8. Standard Practice in the Field (Decision Table)

9. Summary

• Measure first. Be aware that Measure-Command discards output and therefore excludes display cost, and that the first run’s value is unusable.
• $array += gets slower in proportion to the square of the item count. Replacing it with List[T] or with assigning the loop’s output to a variable is the top priority.
• Turning matching into hash table lookups is the most cost-effective improvement available. When you find a nested loop, ask whether you can build an index.
• Choose reading and writing methods for file I/O according to purpose. Provider-side filtering with -Filter, bulk writes, and knowing when to reach for StreamWriter are the basics.
• Don’t insert Format-* in the middle, thin out progress updates, hoist invariant work out of loops — small accumulations add up when the volumes are large.
• Parallelisation is the last resort. Fix the algorithm first, then apply it to work where waiting dominates.

Download the Sample Code

The code covered in this article is packaged in ready-to-run form. It includes three benchmarks and a measurement helper.

Download the sample code (zip)

The samples in this article have been actually executed and verified on PowerShell 7.6 (8 Pester tests). Run the Invoke-SampleTests.ps1 included in the zip and you can reproduce the same verification on your own machine.

# Syntax parsing + static analysis + Pester tests
./Invoke-SampleTests.ps1

The configuration values (paths, server names, tenant IDs and so on) are examples. Do not run them against a production environment as they are — adapt them to your own environment.

Related Articles

• PowerShell Parallel Processing — Choosing Between ForEach-Object -Parallel and Jobs
• PowerShell Output Streams and Log Design — Stop Using Write-Host
• Automating Excel and CSV Business Processing With PowerShell — Practical Recipes for Aggregation, Matching and Report Output
• Practical PowerShell Command Recipes — Growing the Small Tools You Use Every Day
• How to Correctly Compare the Speed of Different Program Versions on Windows
• Identifying “Slow” with PerfView and dotnet-trace

Related Consulting Areas

KomuraSoft LLC handles speeding up aggregation and matching processes that take too long, rebuilding processing designs so they hold up as data volumes grow, and investigating the causes of performance degradation.

• Technical Consulting & Design Review
• Bug Investigation & Root-Cause Analysis
• Migration & Reuse of Legacy Assets
• Contact Us

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

Why is writing $array += $item slow?

Because PowerShell arrays are fixed-length and cannot have elements added to them. The += operator expands into "create a new array, copy every existing element into it, and append at the end". A full copy runs for every single item added, so adding n items costs time proportional to n squared. At 100 items you won't notice; past 10,000 it becomes perceptibly slow; and at 100,000 it is unusable. The fix is to use System.Collections.Generic.List[T] and call Add, or to rewrite so that the output of the whole loop is assigned to a variable.

I measured with Measure-Command, but it doesn't match the actual run time.

Because Measure-Command measures only the execution time of the script block, and discards its output. In a real run you also pay the cost of formatting the results for display and the time spent drawing to the console. In addition, the first run includes module loading and JIT compilation time, so the first measurement is usually unreliable. Stick to two rules: run the same operation several times and look at the second and later values, and always measure things you are comparing under the same conditions.

Matching two CSV files never finishes. What should I fix?

It's highly likely you are doing a linear search with Where-Object inside the inner loop every time. If each file has 10,000 rows, that's 100 million comparisons. Convert one side into a hash table (or use Group-Object -AsHashTable) and look it up by key, and the number of comparisons drops to roughly the sum of the two row counts. The effect grows as the data grows — this is the single most cost-effective improvement available.

I've heard that calling .NET APIs directly is faster than using cmdlets. Should I always do that?

No — it's a judgement call. .NET APIs (System.IO.File's ReadLines or EnumerateFiles, for example) certainly are faster, but you lose the convenience of PowerShell's provider features, wildcards and relative path resolution, and the code becomes harder to read. Measure first, and replace only the places you have actually shown to be bottlenecks. The practical approach is to confine this to situations where it clearly pays off, such as walking hundreds of thousands of files or reading enormous files.

Will parallelising the work make it faster?

It helps when waiting dominates the work, but in terms of ordering it comes last. First reduce the wasted work: narrow down how much you fetch, hoist invariant work out of loops, and turn matching into hash lookups. Parallelising an algorithm that is still O(n^2) only buys you a speed-up proportional to your core count. Conversely, parallelising work where each item is cheap can actually be slower because of the overhead.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.