Parallel Processing in PowerShell — Choosing Between ForEach-Object -Parallel and Jobs

“A script that pings 200 PCs for a liveness check takes 15 minutes per pass.” “Hashing 100,000 files under a shared folder never finishes.” — Once PowerShell automation grows to a certain size, you inevitably hit a wall of processing time. And most of this work is dominated by waiting. The CPU is idle; the script is slow because it’s waiting for network responses and disk I/O one item at a time. That’s where parallelism comes in.

PowerShell 7 introduced ForEach-Object -Parallel, dramatically lowering the barrier to parallel processing. But there’s a catch: careless parallelization not only fails to speed things up, it can make them slower — and corrupt the results as well. “The value disappeared when I updated a shared variable,” “the output comes back in a different order every time,” “the function I defined in the caller can’t be found” — these are all classic symptoms of using parallel processing without understanding how it works.

This article maps out the parallelization options available in PowerShell, then covers how to use ForEach-Object -Parallel and its pitfalls, when to use Start-ThreadJob versus Start-Job, and the cases where you shouldn’t parallelize at all — in a form you can act on in practice.

1. The Bottom Line First

• ForEach-Object -Parallel was added in PowerShell 7.0. It doesn’t exist in Windows PowerShell 5.1.¹
• Each script block runs in a separate runspace (execution environment). Variables and functions from the caller aren’t visible as-is. Variables are passed with the $using: scope modifier.¹
• The default for -ThrottleLimit is 5. From PowerShell 7.1 onwards a runspace pool is reused, and ThrottleLimit becomes that pool’s size. If you want a fresh one each time, use -UseNewRunspace.¹
• What $using: passes is a reference. Reading is safe, but if you’re going to update it you need a thread-safe type from System.Collections.Concurrent. Concurrently updating an ordinary hashtable or List will corrupt it.¹
• Parallelization is not a “it always gets faster” measure. The official documentation explicitly states that parallelizing trivial work can be much slower than normal. It pays off for work with long waits, and for computation that genuinely benefits from multiple cores.¹
• Both output and error ordering are non-deterministic. The order in which things are written to the error stream is random too, and a terminating error inside a script block stops only that iteration, reported as a PSTaskException.¹
• You can dispatch it as a job with -AsJob. Note, though, that -ThrottleLimit is the parallelism per job, so creating multiple jobs multiplies the number of concurrent executions.¹
• For lightweight parallelism use Start-ThreadJob; when you need isolation use Start-Job. Start-Job runs in a separate process, so overhead is high and results come back serialized, having lost their methods.²³
• If you’re running the same work across multiple Windows machines, Invoke-Command’s implicit parallelism is the shortest route. It runs against up to 32 machines concurrently by default.⁴

2. The Four Parallelization Options

Let’s start with a map. There are broadly four ways to “run things at the same time” in PowerShell.

The practical decision is simple. If you’re applying the same work to a large number of targets, use ForEach-Object -Parallel; if the targets are “multiple remote PCs,” remote execution comes first. The latter runs against up to 32 of the specified computers concurrently by default, so you don’t need to write parallelism yourself.⁴ For setting up remote execution itself, see “An Introduction to PowerShell Remoting (WinRM).”

Start-Job is heavyweight because background jobs are launched as a separate process. On top of the process startup cost, results come back serialized, so the objects you receive are “deserialized copies” without methods.³ Start-ThreadJob, by contrast, runs on a thread within the same process and is dramatically lighter.²

3. The Basics of ForEach-Object -Parallel

The minimal form looks like this. $_ is the current input object, and outer variables are referenced with a $using: prefix.¹

$timeout = 2   # A variable in the caller

$result = $computers | ForEach-Object -Parallel {
    $name = $_
    # Outer variables aren't visible without the $using: prefix
    $ok = Test-Connection -TargetName $name -Count 1 -TimeoutSeconds $using:timeout -Quiet

    # The basic form is to "output" rather than update a shared aggregation variable
    [pscustomobject]@{
        Computer = $name
        Alive    = $ok
        CheckedAt = Get-Date
    }
} -ThrottleLimit 20

$result | Where-Object { -not $_.Alive } | Format-Table

There’s one design principle to internalize here. Instead of updating shared variables, have each script block “output” its result and collect them all in the caller. With that shape, thread-safety problems never arise in the first place. Code that causes trouble in parallel processing is almost always trying to mutate shared state.

If you really must collect into a shared collection, use a thread-safe type.¹

# The pattern shown in the official docs: ConcurrentDictionary can be safely updated from multiple threads
$safeDict = [System.Collections.Concurrent.ConcurrentDictionary[string,object]]::new()

Get-Process | ForEach-Object -Parallel {
    $dict = $using:safeDict
    # TryAdd() returns a boolean. Discard it or True/False gets mixed into the output
    $null = $dict.TryAdd($_.ProcessName, $_)
}

# [Dangerous] Passing an ordinary Hashtable or List<T> with $using: and updating it is not safe
# $shared = @{}          # Concurrent updates corrupt the internal structure, causing exceptions or lost values

4. Five Pitfalls

(1) Functions from the caller aren’t visible

Because a parallel script block runs in a separate runspace, functions defined in the caller can’t be called as-is. The correct approach is to move shared logic into a module and import it at the top of the script block.

$items | ForEach-Object -Parallel {
    Import-Module 'D:\Scripts\Modules\KsOps' -ErrorAction Stop   # Load it in each runspace
    Convert-KsRecord -Input $_
} -ThrottleLimit 8

However, the cost of loading the module is incurred in each runspace, once per unit of parallelism. In work that uses a heavy module, this can be the main reason throughput plateaus even as you raise the degree of parallelism. Conventions for modularization are covered in “PowerShell Parameter Design and Modularization.”

(2) Runspaces are reused, so state can carry over

In PowerShell 7.0 a new runspace was created for each iteration, but from 7.1 onwards runspaces are reused from a pool by default.¹ That’s a major performance improvement, but it means that preference variables set, current-directory changes made, or modules loaded during one iteration are visible to subsequent iterations that use the same runspace. If you need complete independence between iterations, specify -UseNewRunspace (at a cost in speed).¹

(3) Neither output nor error ordering is guaranteed

The order of parallel execution is non-deterministic. The order in which things are written to the error stream is random too, and the same applies to the warning, verbose, and information streams.¹

1..5 | ForEach-Object -Parallel {
    if ($_ -eq 3) { throw "Terminating Error: $_" }   # Only this iteration stops
    "Output: $_"
}
# Output: 1 / Output: 4 / Output: 2 / Output: 5 (in any order); Output: 3 never appears

A terminating error inside a script block ends only that iteration, and the other parallel executions continue. The error is written to the error stream as an ErrorRecord whose FullyQualifiedErrorId is PSTaskException. Since the behaviour is not “stop everything if even one fails,” you have to aggregate success and failure across all items yourself.¹

$results = $files | ForEach-Object -Parallel {
    # Inside a catch block $_ changes to the ErrorRecord, so always stash
    # the input object in a separate variable before using it
    $file = $_

    try {
        $hash = Get-FileHash -Path $file.FullName -Algorithm SHA256 -ErrorAction Stop
        [pscustomobject]@{ Path = $file.FullName; Hash = $hash.Hash; Error = $null }
    }
    catch {
        # Return failures as "output" too, and sort them out in the caller
        [pscustomobject]@{ Path = $file.FullName; Hash = $null; Error = $_.Exception.Message }
    }
} -ThrottleLimit 8

$failed = $results | Where-Object Error
if ($failed) { Write-Warning "$($failed.Count) item(s) failed" }

(4) PipelineVariable can’t be used

The common parameter -PipelineVariable isn’t supported in parallel scenarios, even with $using:.¹ It’s a point that trips people up when porting from sequential processing.

(5) Progress display and logs get interleaved

If multiple threads write Write-Progress or their own logs at the same time, lines get interleaved and files contend with each other. Rather than appending directly to a file inside the script block, it’s safer to return logs as result objects and write them from a single place in the caller. Output stream design is covered in “PowerShell Output Streams and Logging Design.”

5. How to Decide ThrottleLimit

-ThrottleLimit is the number of script blocks running at the same time, and the default is 5.¹ Guidelines for setting it break down by the nature of the work.

That last row matters most in practice. Taking down a business server in order to speed up your own script defeats the purpose, so when the other side is an internal API or file server, set your degree of parallelism to “what they can withstand.” For dealing with API rate limits, see “Integrating with REST APIs from PowerShell.”

The documentation is also explicit about a caveat when using -AsJob: ThrottleLimit is the limit per ForEach-Object -Parallel invocation, so if you create 10 jobs, “10 × ThrottleLimit” run concurrently.¹

6. Cases Where Not Parallelizing Is Faster

The official documentation goes further than you might expect — new runspaces carry considerable overhead compared with sequential processing, a trivial parallel script can be much slower than normal, and you should experiment to find where it actually helps.¹ There’s even a sample in the official docs explicitly labelled as an inefficient use of parallelism.

The criteria are simple.

• Short per-item processing time (milliseconds) → don’t parallelize. String processing, hashtable lookups and the like are faster sequentially
• Few items (a few dozen) → don’t parallelize. The overhead is relatively large
• Waits of several hundred milliseconds or more per item → parallelization is worth it
• Long-running computation → multiple cores help

And always measure before deciding. Comparing the sequential and parallel versions with Measure-Command gives you an answer in tens of seconds. Measurement conventions and speeding up PowerShell scripts in general are covered in “Where to Look When a PowerShell Script Is Slow.”

# Compare sequential and parallel with the same input (run it a few times and look at the stable value)
# The parallel side runs in a separate runspace, so functions defined in the caller aren't visible.
# That makes the comparison meaningless, so either import a module or inline the work
$seq = Measure-Command {
    $items | ForEach-Object { Invoke-Work $_ }
}
$par = Measure-Command {
    $items | ForEach-Object -Parallel {
        Import-Module 'D:\Scripts\Modules\KsOps'   # <- without this you get a command-not-found
        Invoke-Work $_
    } -ThrottleLimit 16
}
'{0:N1}s -> {1:N1}s' -f $seq.TotalSeconds, $par.TotalSeconds

7. Choosing Between Start-ThreadJob and Start-Job

ForEach-Object -Parallel fits the shape of “apply the same work to many inputs,” but when you want to run work of different kinds concurrently and collect the results later, jobs are the better fit.

# Run separate tasks concurrently and wait for them together (thread jobs = lightweight)
# Thread jobs also run in a separate runspace, so the caller's functions aren't visible.
# Load the module at the top of each job (or make the script block self-contained)
$jobs = @(
    Start-ThreadJob -Name 'AD User Inventory' -ScriptBlock {
        Import-Module 'D:\Scripts\Modules\KsOps'; Get-KsAdUserReport
    }
    Start-ThreadJob -Name 'File Server Capacity' -ScriptBlock {
        Import-Module 'D:\Scripts\Modules\KsOps'; Get-KsShareUsage
    }
    Start-ThreadJob -Name 'License Summary' -ScriptBlock {
        Import-Module 'D:\Scripts\Modules\KsOps'; Get-KsLicenseSummary
    }
)

# Wait for completion and collect the results. Always capture errors with -ErrorVariable
$reports = $jobs | Receive-Job -Wait -ErrorAction SilentlyContinue -ErrorVariable jobErrors

# The state only becomes Failed when the script block died with a "terminating error".
# A job that ended with a non-terminating error such as Write-Error stays Completed,
# so looking only at the state means "errors occurred but it's treated as a success"
$failed = @($jobs | Where-Object { $_.State -eq 'Failed' })
$jobs | Remove-Job -Force

if ($failed.Count -gt 0 -or @($jobErrors).Count -gt 0) {
    throw "Errors occurred during parallel execution: $(($jobErrors | ForEach-Object { $_.Exception.Message }) -join ' / ')"
}

The reason not to use -AutoRemoveJob here is that the job disappears immediately after collection, leaving you unable to check which one failed. Errors from Receive-Job are non-terminating errors by default, so if you write nothing, you get the worst possible shape: “some of them failed, but only the successful portion came back and it looks like it completed.” In inventory and reporting automation, that kind of silent loss lingers as wrong numbers.

You’d choose Start-Job (the process-isolated kind) in cases like these.³

• Work you don’t want to take the calling process down with it (calling a native DLL that might crash, for example)
• Work that needs a different process environment (you want to run it under a different culture setting or different environment variables)
• Work where you want to separate the execution environment itself, such as 32-bit versus 64-bit

Conversely, if none of those apply, Start-Job is at a disadvantage by exactly the amount of its overhead and serialization constraints. The fact that deserialized objects have no methods also comes back to bite you in downstream processing.³

8. In Environments That Only Have Windows PowerShell 5.1

-Parallel isn’t available on 5.1. There are three realistic options.

1. Start-ThreadJob (install the ThreadJob module from the PowerShell Gallery) — lightweight parallelism is available even on 5.1. For conventions on in-house distribution, see “In-House Distribution and Updating of PowerShell Modules”²
2. Invoke-Command -ComputerName — if the targets are multiple Windows machines, this alone gives you parallelism (32 machines concurrently by default)⁴
3. Install PowerShell 7 — 5.1 and 7 can coexist, so running only the heavy work under 7 is a realistic choice

The third often turns out cheapest in the end, and we recommend “run just that piece of work under 7” over “write elaborate 5.1 code for the sake of parallelism.” Our thinking on coexistence and migration is set out in “The Differences Between Windows PowerShell 5.1 and PowerShell 7.”

9. Practical Rules of Thumb (Decision Table)

10. Summary

• ForEach-Object -Parallel is a PowerShell 7.0+ feature that runs each script block in a separate runspace. The basic form is $using: for the caller’s variables, and modularizing functions and importing them.
• If you design each iteration to output its result and aggregate in the caller instead of updating shared variables, you avoid thread-safety problems almost entirely. If you really must share, use the Concurrent types.
• The default ThrottleLimit is 5. Make it large for work dominated by waiting, around the core count for CPU work, and for work involving another system, their capacity is the ceiling.
• Output and error ordering is non-deterministic, and a terminating error stops only that iteration. You must aggregate success and failure across all items yourself.
• Parallelization is not a cure-all. For light work or small item counts, overhead makes it slower. Always compare against the sequential version with Measure-Command before adopting it.
• On 5.1 environments, Start-ThreadJob; for remote work, Invoke-Command; and if that still isn’t enough, installing PowerShell 7 is the realistic route.

Sample Code Download

The code covered in this article is packaged in a ready-to-run form. It includes practical boilerplate for ForEach-Object -Parallel and Start-ThreadJob.

Download the sample code (zip)

The samples in this article have been actually run and verified on PowerShell 7.6 (8 Pester tests). Run Invoke-SampleTests.ps1 included in the zip and you can reproduce the same verification on your own machine.

# Syntax parsing + static analysis + Pester tests
./Invoke-SampleTests.ps1

Configuration values (paths, server names, tenant IDs and so on) are examples. Do not run them as-is in a production environment — adapt them to your own environment.

Related Articles

• The Differences Between Windows PowerShell 5.1 and PowerShell 7 — A Practical Guide to Migrating In-House Scripts
• An Introduction to PowerShell Remoting (WinRM) — Managing Multiple Windows Machines at Once
• PowerShell Parameter Design and Modularization — From “a Script That Works” to “a Script You Can Hand Over”
• Taking Stock of a File Server with PowerShell — Capacity Surveys and Access Permission (ACL) Audits
• PowerShell Error Handling and Retry Design — From the try/catch Trap to Exit Codes and Retry Best Practices

Related Consulting Areas

KomuraSoft LLC handles speeding up long-running operations scripts, reviewing automation designs that involve parallel processing, and investigating faults such as “the results went wrong after we parallelized it.”

• Technical Consulting & Design Review
• Bug Investigation & Root-Cause Analysis
• Legacy Asset Migration & Reuse
• Contact

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

Can I use ForEach-Object -Parallel in Windows PowerShell 5.1?

No. -Parallel is a parameter set added in PowerShell 7.0 and simply doesn't exist in 5.1. To parallelize on 5.1, your options are Start-ThreadJob from the ThreadJob module available on the PowerShell Gallery, the built-in Start-Job (process-isolated and heavyweight), or building a RunspacePool yourself. If the goal is to speed up in-house scripts, installing PowerShell 7 and using -Parallel is better for maintainability than writing elaborate parallel code for 5.1.

I parallelized my script and it got slower. Why?

Because the overhead of parallel execution is larger than the work itself. ForEach-Object -Parallel runs each script block in a separate runspace, which carries considerable overhead compared with sequential processing, and the official documentation explicitly states that a trivial parallel script can be much slower than normal. If each item finishes in a few milliseconds, or you only have a few dozen items, not parallelizing is usually faster. Parallelism pays off for work with long network or file I/O waits, and for computation that genuinely benefits from multiple cores.

Can I write to a variable passed in with $using: from inside a parallel script block?

Reading the referenced value is safe, but writing is not safe unless the target is a thread-safe type. $using: is a mechanism for passing a variable reference from the calling thread to each script block's thread, and because multiple threads touch it simultaneously, updating an ordinary hashtable or List will corrupt it. If you want to collect aggregated results, use a thread-safe type from the System.Collections.Concurrent namespace such as ConcurrentDictionary or ConcurrentBag — or better, design each script block to output its value and receive it in the caller.

What's the correct value for ThrottleLimit?

It depends on the nature of the work. The default is 5. Work dominated by network or file I/O waits (reachability checks against servers, API calls, and so on) can benefit from values higher than the CPU core count. Computational work that saturates the CPU, on the other hand, just gets slower from contention once you go much beyond the core count. When the other side is a business server or an API, the ceiling isn't only your own convenience — their concurrent connection limits and rate limits are the ceiling too. The safe approach is to measure with the default of 5 first, then double it and check whether it actually helps.

I can't call my own functions from inside a parallel script block.

A parallel script block runs in a runspace separate from the caller, so functions and variables defined in the calling scope aren't visible as-is. There are two fixes: move the shared logic into a module (.psm1) and Import-Module at the top of the script block, or pass the function definition as text with $using: and redefine it inside the script block. For maintainability, the former is recommended. Be aware, though, that loading the module in each runspace has a cost, so if the module is heavy, raising the degree of parallelism will hit a ceiling.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.