How to Fairly Compare the Execution Speed of C#, C++, Java, and Go

“C++ is supposed to be fast.” “Go is lightweight in production.” “Java gets really fast on long-running workloads.” “C# is surprisingly strong too, thanks to the .NET JIT.”

You hear this kind of thing all the time. But the single worst thing you can do here is take numbers measured by different people in different environments, line them up, and declare a winner among languages.

C# and Java are easily affected by JIT and warm-up, while C++ and Go are normally compiled ahead of time. The presence and characteristics of GC differ too. Differences in the standard library and the surrounding library implementations matter quite a lot. And even on the same machine, results wobble easily with power settings, heat, background activity, and skew in the input data. It is an unglamorous corner of the field.

flowchart TB
    accTitle: Sources of variance in results
    accDescr: Diagram showing that JIT and warm-up effects, differences in GC and library implementations, and power settings, heat, background noise and skew in the input data all combine so that measured results wobble easily.
    n1["JIT and warm-up effects"] --> n4["Results wobble easily"]
    n2["GC and library implementation differences"] --> n4
    n3["Power, heat, noise, input skew"] --> n4
    n4 -.-> n5["Do not rank languages from numbers taken elsewhere"]

Figure 1: Precisely because there are so many sources of variance, a comparison assembled from other people’s numbers does not hold.

This article lays out how to compare C# / C++ / Java / Go as fairly as possible. To give away the conclusion first: the most important thing is not to try to settle which language is fastest with a single number.

The subject of this article is, strictly, how to structure the comparison. Lining up environment-dependent numbers gets you nowhere, because a change of conditions flips the ranking easily. For that reason we do not publish a measured ranking here. Instead, we focus on how to design a comparison so that it is actually worth something.

Who this is for

This is written for developers and technical leads who have several candidate languages and want to decide which one to implement in on performance grounds, and for anyone who has to put together measurements solid enough to say we compared the speed in an internal report. It is not a deep dive into one language; the subject is the design of a comparison spanning four languages, so it should read fine even if you only work in one of them.

The code samples use PowerShell for the common runner, and BenchmarkDotNet for C# and JMH for Java as the per-language harnesses. For C++ and Go we only list the conditions to align.

Terms to know up front

A few terms appear in English throughout the article. Here they are up front so the first occurrence does not trip you up.

The Conclusion First

In a C# / C++ / Java / Go speed comparison, these seven things are what really matter.

1. Decide first what kind of speed you want to compare Whether it is startup time, steady-state throughput, p95 latency, or memory efficiency changes how you measure.

2. Never draw a conclusion from a single benchmark CPU computation, memory allocation, parallel processing, and startup time each make different languages and runtimes look strong.

3. Separate cold and warm for C# and Java Mixing comparisons that include the first run with steady-state comparisons after warm-up twists the whole discussion.

4. Measure with the same algorithm, the same input, and the same correctness check “It was not a faster implementation, it was just solving a different problem” is a classic benchmark failure.

5. Separate per-language microbenchmarks from cross-language end-to-end benchmarks Each language’s dedicated harness is convenient, but a cross-language comparison is better run from an outer common runner.

6. Look at the median and the distribution, not just the mean A single GC pause or background task landing on one run is enough to wreck the average.

7. Record the conditions, not just the numbers A benchmark result is a record of the experimental conditions just as much as a record of speed. Results with no documented conditions become quite painful later.

Knowledge map for this article

This article points out how risky it is to compare the execution speed of C#, C++, Java, and Go with a single number, and lays out a measurement design for a fair comparison. C# and Java show a large gap between cold and warm because of JIT compilation and Tiered Compilation, while C++ and Go are normally compiled ahead of time with AOT, so the two groups should not be mixed into the same table. In C++, dead code elimination, in which the compiler removes computation it sees as unused, has to be countered with Google Benchmark and with correctness checks based on a checksum; measurement within a single language should use a dedicated harness such as BenchmarkDotNet, JMH, go test -bench, or Google Benchmark, and comparison across languages is recommended to follow a two-layer structure in which an outer shared runner randomizes the execution order and verifies checksums. Looking beyond the mean to the median and to the distribution of tail latency such as p95 and p99 is set out as a further condition for interpreting the results correctly.

flowchart LR
    accTitle: Cross-language benchmarking of C#, C++, Java, and Go
    accDescr: Diagram showing how the difference between JIT compilation with Tiered Compilation in C# and Java and ahead-of-time compilation in C++ and Go demands a distinction between cold and warm, how Google Benchmark and checksum verification counter dead code elimination in C++, how per-language harnesses and an outer shared runner form a two-layer structure, and how all of it connects to evaluating the distribution of tail latency.
    cross_language_benchmarking["Cross-language benchmark design"]
    csharp["C#"]
    jit_compilation["JIT (Just-In-Time) Compilation"]
    java_lang["Java"]
    cpp["C++"]
    ahead_of_time_compilation["Ahead-of-Time (AOT) Compilation"]
    golang["Go"]
    tiered_compilation["Tiered Compilation"]
    gomaxprocs["GOMAXPROCS"]
    cgo["cgo"]
    benchmarkdotnet["BenchmarkDotNet"]
    jmh["JMH (Java Microbenchmark Harness)"]
    go_benchmark_testing["go test -bench and benchstat"]
    google_benchmark["Google Benchmark"]
    warm_up_measurement["Steady-State (Warm) Measurement"]
    dead_code_elimination["Dead code elimination"]
    checksum_verification["Checksum-Based Correctness Check"]
    tail_latency_percentile["Tail Latency Percentiles (p95/p99)"]
    garbage_collection["Garbage Collection (GC)"]
    csharp -.->|"uses"| jit_compilation
    java_lang -.->|"uses"| jit_compilation
    cpp -->|"uses"| ahead_of_time_compilation
    golang -->|"uses"| ahead_of_time_compilation
    csharp -->|"uses"| tiered_compilation
    golang -->|"uses"| gomaxprocs
    golang -.->|"uses"| cgo
    csharp -.->|"uses"| benchmarkdotnet
    java_lang -.->|"uses"| jmh
    golang -.->|"uses"| go_benchmark_testing
    cpp -.->|"uses"| google_benchmark
    benchmarkdotnet -->|"uses"| warm_up_measurement
    jmh -->|"uses"| warm_up_measurement
    google_benchmark -.->|"prevents"| dead_code_elimination
    cpp -.->|"may cause"| dead_code_elimination
    checksum_verification -->|"prevents"| dead_code_elimination
    cross_language_benchmarking -->|"uses"| tail_latency_percentile
    csharp -->|"uses"| garbage_collection
    java_lang -->|"uses"| garbage_collection
    golang -->|"uses"| garbage_collection
    cross_language_benchmarking -.->|"uses"| warm_up_measurement
    cross_language_benchmarking -->|"uses"| checksum_verification
    cross_language_benchmarking -->|"uses"| jit_compilation
    cross_language_benchmarking -->|"uses"| ahead_of_time_compilation

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (24 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

What to Decide First

If you let “fast” stay a single word, things usually go wrong. Start by deciding what you are going to call fast.

For the same program, what you want to look at can differ considerably.

1. Do you want to look at startup time?

For CLI tools, short-lived batch jobs, and helper tools that start once and exit immediately, cold start and process startup are what matter. On this axis, results change dramatically depending on whether JIT and class-loading initialization costs are included.

2. Do you want to look at long-running throughput?

For servers, resident processes, workers, and long-running conversion jobs, steady-state throughput is what counts. In that case, being slow only on the first run is not the point; the question is how high and how steadily it climbs after warm-up.

3. Do you want to look at tail latency?

For APIs, UI, and near-real-time processing, p95 / p99 can matter more than the mean. Even if the average is fast, occasional long stalls hurt in terms of user experience and SLAs.

4. Do you want to include memory efficiency?

If you look only at CPU time and ignore peak RSS, allocation volume, GC count, and GC pauses, you will misjudge how heavy the thing really is in production. “Fast but eats a lot of memory” and “a bit slower but consistently lightweight” can swap places in the ranking depending on the use case.

In short, the question to settle first is

What this comparison should answer is not which language is fast, but which workload, under which conditions, on which metric, it can process faster.

Start collecting numbers while this is still vague and nothing will come together at the end.

flowchart TB
    accTitle: Decide what fast means before measuring
    accDescr: Diagram showing that how you measure depends on whether you want startup time, steady-state throughput, tail latency or memory efficiency, so before comparing you decide which workload, under which conditions, on which metric you are looking at.
    q1["What does this comparison need to answer"] --> a1["Startup time"]
    q1 --> a2["Steady-state throughput"]
    q1 --> a3["p95 / p99 latency"]
    a3 -.-> a4["Memory efficiency is another axis"]
    a1 -.-> a5["Each axis changes how you measure"]

Figure 2: Do not start collecting numbers until the definition of fast is settled.

Why Comparing Languages Is Hard

Mixing JIT and AOT turns it into a different experiment

C# and Java are normally affected by the JIT. C++ and Go, on the other hand, are normally compiled ahead of time.

That means if you measure the first run, you are measuring not only the speed of the program itself but also runtime startup, class loading, and JIT preparation. Conversely, if you only look at fully warmed-up runs, the comparison becomes how far steady-state optimization can go.

Both are meaningful. But they do not mean the same thing.

flowchart TB
    accTitle: The JIT group versus the AOT group
    accDescr: Diagram showing that C# and Java are normally affected by the JIT so the first run mixes in runtime startup, class loading and JIT preparation, while C++ and Go are normally compiled ahead of time, which makes a first-run comparison and a post-warm-up comparison two different experiments.
    j1["C# and Java (normally JIT)"] --> j2["The first run mixes in startup and JIT preparation"]
    g1["C++ and Go (normally AOT)"] --> g2["They run from an ahead-of-time compiled binary"]
    j2 --> mix["First-run and steady-state comparisons are different experiments"]
    g2 --> mix

Figure 3: Between languages with different execution models, where you start measuring decides what the experiment is about.

Implementation differences routinely outweigh language differences

Even for the same “sort”,

• one side uses the standard library
• one side is hand-rolled
• one side does extra copies
• one side regenerates the input every time

That alone changes the results considerably.

Moreover, once you get into JSON, compression, cryptography, or regular expressions, library implementation differences matter far more than the language itself. Unless you make explicit what you are measuring, what you intended as a “language comparison” becomes a “library comparison”.

flowchart TB
    accTitle: A language comparison quietly becomes a library comparison
    accDescr: Diagram showing that for the same work the results change depending on whether the standard library or a hand-rolled implementation is used and whether there are extra copies or input regeneration, and that for JSON, compression, cryptography and regular expressions the library implementation matters more than the language, so without stating what is measured a language comparison becomes a library comparison.
    d1["Implementations that supposedly do the same work"] --> d2["Implementation and library differences get mixed in"]
    d2 --> d3{"Is it explicit what is being measured"}
    d3 -->|"No"| d4["A language comparison that is really a library comparison"]
    d3 -->|"Yes"| d5["The result can be interpreted as a comparison"]

Figure 4: Naming what you measure correctly clears up most of the misunderstanding on its own.

C++ has a trap where optimization deletes the work

In microbenchmarks especially, when the compiler decides that nobody is using a computed result, it can eliminate the computation entirely. Then it is not that the code is fast; you are measuring the fact that it is doing nothing at all. The typical symptom is the whole loop disappearing and elapsed time dropping to nearly zero.

This problem shows up especially blatantly in C++, so consuming the result, printing a checksum, or using the benchmark framework’s optimization-suppression facilities is quite important.

flowchart TB
    accTitle: The trap where optimization deletes the work
    accDescr: Diagram showing that in microbenchmarks the compiler removes the computation when it decides nobody uses the result, so the measurement records the absence of work rather than speed, which makes consuming the result, printing a checksum and using suppression facilities important.
    o1["Nobody uses the computed result"] --> o2["The compiler deletes the work"]
    o2 --> o3["Elapsed time looks close to zero"]
    o3 -.-> o4["Not fast, simply doing nothing"]
    o3 --> o5["Prevent it with checksum output or suppression facilities"]

Figure 5: When a result looks impossibly fast, first suspect that the work was deleted.

GC is neither a disadvantage nor an advantage; it is a characteristic

C#, Java, and Go have a GC. Reducing that to “it has GC, therefore it is slow” is far too crude.

In practice, what matters more is

• how large numbers of short-lived objects are handled
• heap size configuration
• GC frequency and pauses
• object layout
• the allocation habits of libraries

Conversely, C++ lets you control things finely through manual management and RAII, but that also means design and implementation differences show up more easily. In other words, a difference in memory management model is not by itself a verdict of better or worse.

flowchart TB
    accTitle: GC is a characteristic, not a handicap
    accDescr: Diagram showing that in languages with a GC the handling of short-lived objects, heap configuration, GC frequency and pauses and allocation habits matter more, while C++ offers fine control at the cost of design and implementation differences showing up easily, so a different management model is not a verdict of better or worse.
    gc1["Languages with a GC"] --> gc2["Configuration and allocation habits dominate"]
    gp1["Manual management and RAII in C++"] --> gp2["Controllable, but implementation differences show up easily"]
    gc2 --> gv1["A different management model is not a verdict"]
    gp2 --> gv1

Figure 6: A way of framing this that avoids settling for “it has GC, therefore it is slow”.

What Not to Do in a Comparison

1. Mixing Debug and Release

This is out of the question. Always align the comparison targets on production-grade optimized builds.

2. Not solving the same problem

Different input formats, different output, error handling on only one side, different memory reuse policies. Leave these alone and you end up measuring differences in requirements, not speed.

3. Running once and drawing a conclusion

A single run is mostly noise.

• JIT
• page cache
• CPU boost
• heat
• background tasks
• GC
• first-time file reads

All of this gets mixed into that one run.

flowchart TB
    accTitle: What gets mixed into a single run
    accDescr: Diagram showing that a single run mixes together JIT and first-time file reads, page cache and CPU boost, heat, background tasks and GC, which is why one run is mostly noise.
    x1["JIT and first-time file reads"] --> x4["A single run mixes all of it together"]
    x2["Page cache and CPU boost"] --> x4
    x3["Heat and background tasks"] --> x4
    x4 --> x5["A single run is mostly noise"]

Figure 7: A number from one run carries the largest share of everything you did not want to measure.

4. Blurring warm-up

When measuring C# and Java, being vague about whether the first run is included or only post-warm-up runs count makes the discussion collapse. Treat cold and warm as separate things.

5. Skipping correctness checks

Before “fast”, a benchmark needs “returns the same result”. Always confirm that every implementation under comparison produces the same checksum or the same output from the same input.

6. Building a worldview from a single microbenchmark

Winning a tight loop does not mean winning across a real service. Conversely, losing on startup time can still leave you plenty strong on long-running workloads.

The Basic Approach When Comparing C# / C++ / Java / Go

This part is quite important. The recommendation is a two-layer structure.

flowchart TB
    subgraph outer["Outer layer: the cross-language common runner"]
        direction TB
        R["Common runner<br/>randomized execution order / cold and warm separated<br/>checksum verification / raw data saved"]
        R --> E1["bench executable<br/>C++"]
        R --> E2["bench executable<br/>C#"]
        R --> E3["bench executable<br/>Java"]
        R --> E4["bench executable<br/>Go"]
    end

    subgraph inner["Inner layer: the per-language deep dive"]
        direction TB
        H1["Google Benchmark"]
        H2["BenchmarkDotNet"]
        H3["JMH"]
        H4["go test -bench and benchstat"]
    end

    E1 -.->|"measure the same work in detail inside the language"| H1
    E2 -.->|"measure the same work in detail inside the language"| H2
    E3 -.->|"measure the same work in detail inside the language"| H3
    E4 -.->|"measure the same work in detail inside the language"| H4

Figure 8: Cross-language comparison runs in the outer common runner; per-language digging happens in the dedicated harnesses.

Numbers from the outer layer are numbers you may compare across languages; numbers from the inner layer are numbers for tracking improvement within one language. The key point is not to mix the two into a single table.

1. For per-language measurement, use the harness suited to that language

Each language has a benchmark tool that absorbs that language’s own circumstances.

• C#: BenchmarkDotNet
• Java: JMH
• Go: go test -bench and benchstat
• C++: Google Benchmark

These take care of each runtime’s quirks, the statistical processing, and the common measurement traps to a reasonable degree. They are quite effective for comparisons within a language and for drilling into an implementation.

For instance, measuring “sort 10 million int32 values” with BenchmarkDotNet looks like this in its minimal form.

// C# / .NET 8 + BenchmarkDotNet 0.13.x
//   dotnet add package BenchmarkDotNet
//   dotnet run -c Release
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<SortBench>();

[MemoryDiagnoser]                       // also report allocation volume and GC counts
public class SortBench
{
    private int[] _source = Array.Empty<int>();
    private int[] _work = Array.Empty<int>();

    [GlobalSetup]
    public void Setup()
    {
        var rng = new Random(12345);    // fixed seed so the input is identical every time
        _source = new int[10_000_000];
        for (int i = 0; i < _source.Length; i++)
        {
            _source[i] = rng.Next();
        }
        _work = new int[_source.Length];
    }

    [IterationSetup]                    // restore the unsorted state before every iteration
    public void ResetInput() => Array.Copy(_source, _work, _source.Length);

    [Benchmark]
    public long SortInt32()
    {
        Array.Sort(_work);

        long checksum = 0;
        foreach (int v in _work)
        {
            checksum = checksum * 31 + v;
        }
        return checksum;                // return the result so the optimizer cannot delete it
    }
}

[IterationSetup] comes with a constraint. The official BenchmarkDotNet documentation does not recommend it for microbenchmarks because it pollutes the results, and states that it is useful for macrobenchmarks that take 100ms or more. Sorting 10 million values meets that condition, but when you measure short operations, move the work into [GlobalSetup] and carry the state across instead.

With JMH on Java, the thinking is the same.

// Java / JMH. Add jmh-core and jmh-generator-annprocess to pom.xml
//   mvn clean verify
//   java -jar target/benchmarks.jar SortBench
package bench;

import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Fork;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Measurement;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import org.openjdk.jmh.annotations.Warmup;

import java.util.Arrays;
import java.util.Random;
import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
@Warmup(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS)
@Measurement(iterations = 10, time = 1, timeUnit = TimeUnit.SECONDS)
@Fork(3)                                 // separate JVMs to even out JIT luck
public class SortBench {

    private int[] source;
    private int[] work;

    @Setup(Level.Trial)
    public void setUp() {
        Random rng = new Random(12345);  // fixed seed so the input is identical every time
        source = new int[10_000_000];
        for (int i = 0; i < source.length; i++) {
            source[i] = rng.nextInt();
        }
        work = new int[source.length];
    }

    @Setup(Level.Invocation)             // restore the unsorted state before every invocation
    public void resetInput() {
        System.arraycopy(source, 0, work, 0, source.length);
    }

    @Benchmark
    public long sortInt32() {
        Arrays.sort(work);

        long checksum = 0;
        for (int v : work) {
            checksum = checksum * 31 + v;
        }
        return checksum;                 // JMH consumes the return value, so it is not optimized away
    }
}

Level.Invocation carries the same kind of constraint. The JMH javadoc states explicitly that this level can only be used for benchmarks where a single @Benchmark method call takes more than one millisecond. It takes a timestamp on every invocation, so for short operations the measurement itself becomes the bottleneck.

The @Warmup / @Measurement / @Fork values in BenchmarkDotNet and JMH are experimental conditions in their own right. Always keep them with the results.

2. For cross-language comparison, put a common runner on the outside

On the other hand, placing BenchmarkDotNet results from C# next to JMH results from Java as-is is a bit dangerous. The harnesses themselves follow different conventions.

So for cross-language work, the recommendation is to turn each implementation into an executable that can be invoked through the same CLI contract and drive them all from the outside under identical conditions.

For example, prepare an executable of this shape in each language.

bench --scenario sort_int32 --dataset data/sort_10m.bin --mode warm
bench --scenario group_words --dataset data/words_100mb.txt --mode cold
bench --scenario parallel_hash --dataset data/blob_1gb.bin --threads 8

Make the output a contract as well. If every implementation, in every language, promises to print exactly these two lines on standard output, the runner can parse them with a single regular expression per line.

checksum=[hex string]
inner_ms=[milliseconds as a decimal]

checksum is for the correctness check, and inner_ms is the time the implementation measured for the core work itself. The runner separately measures the wall-clock time of the whole process, so the time including startup and the time for the body alone both survive.

flowchart TB
    accTitle: The CLI contract and the output contract
    accDescr: Diagram showing that turning each language implementation into an executable answering the same CLI contract and printing only the two lines checksum and inner_ms lets the runner measure whole-process time separately, so both the startup-inclusive time and the body-only time are preserved.
    ct1["Executables sharing one CLI contract"] --> ct2["Print the two lines checksum and inner_ms"]
    ct2 --> ct3["The runner measures whole-process wall-clock"]
    ct3 --> ct4["Both startup-inclusive and body-only times survive"]

Figure 9: Once the call and the output are contracts, all four languages can run on the same footing.

Then, on the common runner side,

• randomize the execution order
• separate cold from warm
• pass the same dataset
• verify the checksum
• collect wall-clock time and memory
• keep the raw data in CSV / JSON

is the flow. Written out as a skeleton, it looks like this.

# run-bench.ps1 : the cross-language common runner (skeleton)
# Runs on PowerShell 7.4.
#   pwsh ./run-bench.ps1 -Scenario sort_int32 -Dataset ./data/sort_10m.bin -Runs 15

param(
    [Parameter(Mandatory = $true)][string]$Scenario,
    [Parameter(Mandatory = $true)][string]$Dataset,
    [int]$Runs = 15,
    [int]$WarmupRuns = 3,
    [string]$OutCsv = "./results/raw.csv"
)

# The executable for each language. The CLI contract is identical across all four implementations.
# For Java, place a thin wrapper around java -jar so that the invocation is uniform.
$Implementations = @(
    @{ Language = "cpp";    Exe = "./build/cpp/bench.exe" },
    @{ Language = "csharp"; Exe = "./build/csharp/bench.exe" },
    @{ Language = "java";   Exe = "./build/java/bench.cmd" },
    @{ Language = "go";     Exe = "./build/go/bench.exe" }
)

function Invoke-OneRun {
    param(
        [Parameter(Mandatory = $true)][hashtable]$Impl,
        [Parameter(Mandatory = $true)][string]$Mode,
        [Parameter(Mandatory = $true)][int]$Index
    )

    # $Scenario and $Dataset come from the param block at the top of the script
    $sw = [System.Diagnostics.Stopwatch]::StartNew()
    $stdout = & $Impl.Exe --scenario $Scenario --dataset $Dataset --mode $Mode
    $sw.Stop()
    $exitCode = $LASTEXITCODE

    # The output contract with the implementations is parsed here, in exactly one place
    $checksum = ""
    $innerMs = [double]::NaN
    foreach ($line in $stdout) {
        if ($line -match '^checksum=(\S+)$') { $checksum = $Matches[1] }
        if ($line -match '^inner_ms=(\S+)$') { $innerMs = [double]$Matches[1] }
    }

    if ($exitCode -ne 0 -or [string]::IsNullOrEmpty($checksum)) {
        throw "$($Impl.Language) / $Mode / run $Index failed. exit=$exitCode"
    }
    # An implementation that forgets to emit inner_ms still returns exit code 0 with only a checksum.
    # Without this guard the NaN goes straight into the CSV and the inner-time comparison
    # breaks silently
    if ([double]::IsNaN($innerMs) -or [double]::IsInfinity($innerMs) -or $innerMs -lt 0) {
        throw "$($Impl.Language) / $Mode / run $Index did not return inner_ms (value: $innerMs). " +
              "The output contract is the two lines checksum= and inner_ms="
    }

    return [pscustomobject]@{
        timestamp    = (Get-Date).ToString("o")
        language     = $Impl.Language
        scenario     = $Scenario
        cold_or_warm = $Mode
        run_index    = $Index
        process_ms   = [math]::Round($sw.Elapsed.TotalMilliseconds, 3)
        inner_ms     = $innerMs
        checksum     = $checksum
    }
}

# 1. Correctness first. If the implementations do not all return the same checksum,
#    there is no point measuring speed
$expected = $null
foreach ($impl in $Implementations) {
    for ($i = 1; $i -le $WarmupRuns; $i++) {
        $r = Invoke-OneRun -Impl $impl -Mode "warm" -Index $i
        if ($null -eq $expected) {
            $expected = $r.checksum
        }
        elseif ($r.checksum -ne $expected) {
            throw "checksum mismatch. $($impl.Language) returned $($r.checksum), the baseline is $expected"
        }
    }
}

# 2. The real thing. Shuffle the execution order on every run to even out heat and
#    time-of-day bias. The check in step 1 applied to warm runs that get discarded, so
#    every recorded run is verified against the checksum too. Skip this and the timings
#    of an implementation that started doing different work partway through end up in the
#    CSV, which invalidates the comparison itself
$rows = [System.Collections.Generic.List[object]]::new()
foreach ($mode in @("cold", "warm")) {
    for ($i = 1; $i -le $Runs; $i++) {
        $shuffled = $Implementations | Get-Random -Count $Implementations.Count
        foreach ($impl in $shuffled) {
            $r = Invoke-OneRun -Impl $impl -Mode $mode -Index $i
            if ($r.checksum -ne $expected) {
                throw "checksum mismatch. $($impl.Language) $mode run #$i returned $($r.checksum), the baseline is $expected"
            }
            $rows.Add($r)
        }
    }
}

# 3. Always keep the raw data. Aggregation happens later, against this CSV.
# Given a name with no parent directory such as -OutCsv raw.csv,
# Split-Path -Parent returns an empty string and New-Item rejects it.
# This blows up after every run has finished, so the whole measurement is lost
$outDir = Split-Path -Parent $OutCsv
if ($outDir) { New-Item -ItemType Directory -Force -Path $outDir | Out-Null }
$rows | Export-Csv -Path $OutCsv -NoTypeInformation -Encoding utf8
Write-Host "raw data: $OutCsv / $($rows.Count) rows"

There are three things this deliberately does.

• It puts the correctness check ahead of the speed measurement. Comparing speed while the checksums disagree means nothing
• It shuffles on every single run. The goal is to avoid running all of A and then all of B
• It writes out raw data only, with no aggregation. Means and medians are computed from the CSV afterwards

Doing this makes it much easier to keep best practices inside each language and fairness across languages as separate concerns.

flowchart TB
    accTitle: Three deliberate choices in the common runner
    accDescr: Diagram showing that the common runner deliberately places the checksum correctness check ahead of speed measurement, shuffles the execution order on every run, and writes raw data without aggregating it.
    rn1["Correctness check before speed"] --> rn4["A measurement that can be compared fairly"]
    rn2["Shuffle on every run"] --> rn4
    rn3["Write raw data without aggregating"] --> rn4
    rn2 -.-> rn5["Evens out heat and time-of-day bias"]

Figure 10: The runner’s job is not to run fast but to remove doubt.

A Concrete Example: What Benchmark Scenarios to Prepare

When someone says “I want to compare C# / C++ / Java / Go”, the recommendation is: if you only run one, pick a simple CPU-bound scenario that is hard to misread; if you run several, prepare three or four workloads with different characters.

A recommended lineup

1. sort_int32_10m

Purpose: observe CPU, memory bandwidth, and the use of temporary storage

• Input: 10 million int32 values generated with a fixed seed
• Processing: sort the array and return a checksum
• Caveat: restore the same unsorted input every time

This one is relatively easy to read. It does include differences between standard sort implementations, though, so it is a comparison including the standard library rather than the language itself.

2. hash_group_count

Purpose: observe hash tables, string processing, allocation, and GC tendencies

• Input: a fixed text dataset
• Processing: count the occurrences of each word
• Output: the top N entries plus a checksum

This is close to real work, but differences in string libraries and map implementations also matter a lot. In exchange, it is a more realistic comparison.

3. parallel_sha256

Purpose: observe parallelism, the scheduler, worker pools, and synchronization habits

• Input: a sequence of fixed-size binary chunks
• Processing: hash them in order across N threads and return a final checksum
• Conditions: step the thread count through 1 / 2 / 4 / 8

Compared with a simple tight loop, this makes how well it scales under parallel execution much easier to see.

4. startup_noop or startup_parse_small

Purpose: observe startup time

• noop: start and exit immediately
• parse_small: process one small input and exit

Here the JIT and initialization costs of C# / Java are easy to see, and the picture differs quite a bit from C++ / Go. Put the other way round, a gap here is a separate question from who wins on long-running work.

flowchart TB
    accTitle: Four benchmark scenarios with different characters
    accDescr: Diagram showing a lineup of four scenarios that each reveal something different, where sorting shows CPU and memory bandwidth, word counting shows hashing, strings and allocation, parallel hashing shows scaling under parallel execution, and the startup benchmark shows startup time.
    w1["sort_int32_10m"] -.-> v1["CPU, bandwidth, temporary storage"]
    w2["hash_group_count"] -.-> v2["Strings, maps, GC tendencies"]
    w3["parallel_sha256"] -.-> v3["Scaling under parallel execution"]
    w4["startup scenarios"] -.-> v4["Startup and initialization cost"]
    w1 --> w2
    w2 --> w3
    w3 --> w4

Figure 11: One scenario cannot show everything, so split the scenarios by what you want to see.

What about JSON and HTTP benchmarks?

JSON and HTTP are close to real work, so of course they are meaningful. In that case, though, it becomes a comparison that includes libraries, frameworks, and the ecosystem rather than a comparison of languages.

That is not a bad thing in itself. In practice it is often the more important question. But in an article or a report, stating it explicitly

This is not a comparison of languages but a comparison of typical implementations together with the major libraries

leaves less room for misunderstanding.

Conditions to Align per Language

C++

• Align on optimized builds
• Pin the compiler
• Pin the standard library implementation
• Document conditions such as -O3 / /O2, LTO, and PGO
• Take care that results are not optimized away
• Suspect undefined behavior when something looks suspiciously fast

C++ offers a lot of freedom, which means differences in conditions show up directly. That makes which compiler, which flags, and which STL you measured with quite important.

C#

• Align on Release builds
• Pin the .NET version
• Record conditions such as Server GC / Workstation GC
• Document whether Tiered Compilation, ReadyToRun, and Native AOT are in play
• Separate cold and warm

For C#, differences in .NET configuration change how things look. In particular, JIT-compiled C# and Native AOT C# are different axes even though both are “C#”. Mix them and what you are comparing is no longer the language but the deployment form.

Java

• Pin the JDK vendor and version
• Document the GC
• Pin the warm-up, measurement, and fork settings
• Record the heap size and JVM options
• Separate cold start from steady state

Java benefits readily from the JIT, but in exchange how it looks on the first run varies considerably. That makes separating short-lived process comparisons from long-running comparisons mandatory.

Go

• Pin the Go version
• Pin GOMAXPROCS
• Document CGO_ENABLED
• If you touch GOGC, always record it
• Keep benchmark-format output where possible

Go is relatively easy to handle, but in parallel benchmarks the influence of GOMAXPROCS is large. Whether or not you use cgo also changes the whole picture, so always record that among the conditions.

How to Align the Execution Environment

In any language, a comparison run without aligning the environment is mostly a comparison of environments.

flowchart TB
    accTitle: What an unaligned comparison really measures
    accDescr: Diagram showing that when the CPU, OS, power conditions, input data, process priority and core count are not aligned, the difference in the results is a difference in environments rather than a difference between languages.
    en1["Two measurements on environments that do not match"] --> en2["A difference appears in the results"]
    en2 --> en3{"What is that difference a difference in"}
    en3 -->|"Environment aligned"| en4["Readable as an implementation or language difference"]
    en3 -->|"Not aligned"| en5["Only a comparison of environments"]

Figure 12: You may blame a difference on the language only after the environmental difference has been removed.

Things to align

• Same CPU / memory / storage
• Same OS version
• Same power conditions
• Conditions close to the same room temperature
• Same input data
• Same process priority
• Same core-count conditions
• Same container or bare-metal conditions

Things that matter especially

Power settings and CPU frequency

On a laptop, AC power versus battery alone puts you in a different world. If the CPU governor or power mode is not aligned, comparison results wobble considerably.

Power conditions on Windows, notifications, background noise, heat, and how to align execution order are covered in detail in a separate article, How to Correctly Compare the Speed of Different Program Versions on Windows. If you measure on Windows, that side of things matters a lot.

Heat

If only the first few runs are fast and later ones fall off, suspect heat and throttling. Rather than running all of A and then all of B, alternating like A / B / A / B reduces the bias.

flowchart TB
    accTitle: Heat bias and execution order
    accDescr: Diagram showing that when only the later runs degrade you suspect heat and throttling, and that instead of running all of A and then all of B you alternate A and B to reduce the bias.
    th1["Only the later runs degrade"] --> th2["Suspect heat and throttling"]
    th2 --> th3["Do not run all of A and then move to B"]
    th3 --> th4["Alternate A and B"]

Figure 13: Heat cannot be removed, but a well-chosen order distributes it fairly between both sides.

Background activity

Updates, indexing, sync, virus scans, browsers, chat tools. This is unglamorous stuff, but it interferes all the time.

What to Measure

For language comparisons, the recommendation is to look at at least these four separately.

1. wall-clock time

The real time the user waits. This is the first metric to look at.

2. CPU time

“How much CPU was actually consumed.” If only the wall-clock time improves while CPU time stays the same, the difference may come from waiting or I/O.

3. memory / allocations

• peak RSS
• total allocation volume
• allocation count
• GC count
• GC pauses

Looking at these reveals the cost behind the speed.

4. Distribution

• median
• p95 / p99
• min / max
• standard deviation and spread

Talk in averages only and you never see what is really going on in the runs that occasionally spike.

flowchart TB
    accTitle: Four metrics to look at separately
    accDescr: Diagram showing that a language comparison looks separately at wall-clock time as the real time the user waits, CPU time as the time actually spent on the CPU, memory including peak RSS and allocations, and the distribution including the median and percentiles.
    ms1["wall-clock time"] --> ms5["Look at the four separately"]
    ms2["CPU time"] --> ms5
    ms3["Memory and allocations"] --> ms5
    ms5 -.-> ms4["Distribution (median, p95) in its own frame"]

Figure 14: There is no single number for speed; it only becomes readable once the costs behind it are laid out alongside.

A Recommended Execution Procedure

The flow that works well in practice goes roughly in this order.

flowchart TB
    a1["1. Decide the workload"] --> a2["2. Fix a common dataset"]
    a2 --> a3["3. Pass the correctness check first"]
    a3 --> a4{"Did every implementation<br/>agree on the checksum"}
    a4 -- No --> a3
    a4 -- Yes --> a5["4. Pin the build conditions"]
    a5 --> a6["5. Separate cold and warm"]
    a6 --> a7["6. Run with a randomized order"]
    a7 --> a8{"Reached the required<br/>number of runs"}
    a8 -- No --> a7
    a8 -- Yes --> a9["8. Save the raw data"]
    a9 --> a10{"Did a meaningful<br/>difference appear"}
    a10 -- No --> a11["Record the conditions and counts and stop"]
    a10 -- Yes --> a12["9. Profile and dig into the cause"]

Figure 15: The procedure from deciding the workload through the correctness check and the randomized real run to saving the raw data.

1. Decide the workload

First, make explicit what you want to compare.

• startup time
• steady-state throughput
• tail latency
• memory efficiency
• parallel scaling

2. Fix a common dataset

Align the input data with a fixed seed or a fixed file. If data generation is inside the measurement, that too has to run under the same conditions in every language.

3. Pass the correctness check first

Confirm that every implementation returns the same result on both small and large data. Having them emit a checksum or a hash makes this easy to handle.

4. Pin the build conditions

Produce Release, optimized executables in each language, and record the versions and flags.

5. Separate cold and warm

This is especially important for C# and Java.

• cold: includes the moment right after process startup
• warm: the stable state after several runs

Drawing where the measurement starts and stops makes it obvious that these two are different things.

flowchart LR
    subgraph coldrange["The range measured as cold"]
        direction LR
        s1["Process startup"] --> s2["Runtime initialization<br/>class loading"]
        s2 --> s3["First JIT compilation"] --> s4["First run of the body"]
    end

    s4 --> s5["Second and later runs of the body<br/>Tiered Compilation progresses"]

    subgraph warmrange["The range measured as warm"]
        direction LR
        s6["The body at steady state"]
    end

    s5 --> s6

Figure 16: cold spans process startup through the first JIT compilation, while warm measures only the steady state.

Because C++ and Go are compiled ahead of time, they have no step corresponding to “first JIT compilation”, and their runtime initialization is comparatively light. Much of the cold difference is born right there. That is exactly why these two are cleaner kept out of the same table.

6. Alternate or randomize the execution order

For example:

cpp -> csharp -> java -> go
go -> java -> cpp -> csharp
csharp -> go -> java -> cpp
...

This reduces bias from heat and noise.

7. Secure enough runs

For a lightweight microbenchmark, run a great many; for end-to-end runs, you want at least 10. When the difference is small and the run count is low, the interpretation gets quite shaky.

8. Save the raw data

Keep the raw data for every run, not just the aggregates. Looking back later, you can read outliers and warm-up quirks out of it.

9. Profile when a difference appears

Only when a difference appears do you start digging into the cause.

• CPU profile
• allocation profile
• GC logs
• flame graphs
• OS-level traces

Once you get this far, you can talk about why it happens rather than just “fast or slow”.

How to Read the Results

Even after the numbers are in, reading them wrong is still dangerous.

C# / Java are slow only on the first run

Suspect JIT, class loading, and initialization. In that case it is

• a meaningful difference if startup time matters
• a difference that belongs in a separate table if long-running operation is the subject

C++ is strong in tight loops

Low-level optimization, object layout, and minimal runtime overhead may be paying off. Looking only at that and concluding “therefore it is the fastest in a real service too” is a leap.

Go looks favorable on startup time and ease of distribution

The single binary, the relatively light startup, and the approachable concurrency model can pay off. That does not mean it is favorable on every CPU-bound workload.

C# / Java catch up considerably at steady state, or overtake

JIT optimization may be kicking in. This is not a rare story either. That is why it matters not to mix startup-inclusive comparisons with steady-state comparisons.

Large differences on allocation-heavy work

In this case, more than the language name, what usually matters is

• memory layout
• how strings and maps are handled
• GC behavior
• extra copies

flowchart TB
    accTitle: How to read a difference when one appears
    accDescr: Diagram showing that when C# or Java is slow only on the first run you suspect JIT and initialization, treat it as a meaningful difference if startup time matters and as a difference for a separate table if long-running operation is the subject, and dig into the cause with a profile.
    rd1["A difference appeared"] --> rd2{"Under which condition"}
    rd2 -->|"Startup included"| rd3["Meaningful if startup time is the subject"]
    rd2 -->|"Steady state"| rd4["Treat it as a long-running comparison"]
    rd3 --> rd5["Dig into the cause with a profile"]
    rd4 --> rd5

Figure 17: Before comparing the size of the numbers, confirm which arena the difference belongs to.

A Recording Template

Keeping at least this much with every benchmark result will help you later.

timestamp,language,scenario,run_kind,cold_or_warm,elapsed_ms,cpu_ms,max_rss_mb,alloc_bytes,gc_count,checksum
compiler_or_runtime,compiler_version,flags,os,cpu,threads,input_id,notes

For example, run_kind can be split like this.

• micro
• macro
• startup
• parallel

For cold_or_warm, you definitely want to state which one it is.

• cold
• warm

Decide up front what goes into each column and the entries stay consistent.

The key point is to allow measurement columns to be blank and to keep the blank itself on record. “C++ has no gc_count” is information; delete the column entirely and nobody can tell later.

What you miss by looking only at the mean

When you aggregate down to a single mean, information disappears. The following are made-up numbers used to explain the arithmetic, not measured values for any language. Take them as the elapsed_ms of 10 runs of one implementation, sorted from fastest to slowest.

98, 99, 100, 101, 101, 102, 103, 104, 106, 720

Here are the summary statistics those 10 values produce.

In other words, a table that shows only the mean gives “an implementation that occasionally stalls badly” and “an implementation that is consistently a bit slower” the same face. In terms of how the table is built, these differences are what count.

If you want to publish p95 or p99, you need more runs in the first place. To talk about a distribution you need enough samples for the distribution to be visible is all there is to it.

With benchmarks, being interpretable later often matters more than the act of measuring.

flowchart TB
    accTitle: What a single mean hides
    accDescr: Diagram showing that one large outlier pulls the mean well above the normal range while the median stays closer to experience, and that a large gap between min and max is a signal to investigate the outlier, which makes a mean-only table risky.
    av1["One large outlier appears"] --> av2["The mean drifts above the normal range"]
    av2 --> av3["A mean-only table hides what is happening"]
    av3 --> av4["Report the median with min and max"]
    av4 -.-> av5["An outlier is a signal to investigate the cause"]

Figure 18: The mean does not lie, but it stays silent about the outlier.

Summary

What really matters in a C# / C++ / Java / Go speed comparison is taking the crude question of which language is fastest and turning it into the shape of an experiment: which workload, under which conditions, on which metric are we comparing?

These are the points that are hardest to get wrong.

• Separate startup time from steady state
• Measure with the same algorithm, the same input, and the same correctness check
• Never draw a conclusion from a single benchmark
• Separate per-language benchmarks from cross-language benchmarks
• Look at the median and the distribution rather than the mean
• Keep the conditions and the raw data

And the most important thing of all: do not push too hard to settle winners and losers by language name. Real-world performance is decided by the combination of language, runtime, libraries, build conditions, data, OS, and hardware.

“C++ is fast”, “Java is strong”, “Go is lightweight”, “C# is plenty fast too” are all, in a sense, correct. But once under which conditions you are saying so drops out, the discussion ends without the two sides ever meeting.

Align the conditions, use multiple workloads, separate cold from warm, and look all the way down to the distribution. It is unglamorous, but in the end this is what works best.

flowchart TB
    accTitle: Recasting a crude question as an experiment
    accDescr: Diagram showing that the crude question of which language is fastest is recast as an experiment asking which workload under which conditions on which metric, and that aligning the conditions, using multiple workloads, separating cold from warm and looking at the distribution is what works best.
    sq1["Which language is fastest"] --> sq2["Recast it in the shape of an experiment"]
    sq2 --> sq3["Decide the workload, the conditions, and the metric"]
    sq3 --> sq4["Run cold and warm separately"]
    sq4 --> sq5["Look at the distribution and keep it with the conditions"]

Figure 19: Reframing the question is the single biggest conclusion of this article.

References

• BenchmarkDotNet Getting Started https://benchmarkdotnet.org/articles/guides/getting-started.html

• BenchmarkDotNet Setup and Cleanup (the conditions under which [IterationSetup] may be used) https://benchmarkdotnet.org/articles/features/setup-and-cleanup.html

• OpenJDK JMH Project https://openjdk.org/projects/code-tools/jmh/

• JMH Level javadoc (the constraints and warnings for Level.Invocation) https://javadoc.io/doc/org.openjdk.jmh/jmh-core/latest/org/openjdk/jmh/annotations/Level.html

• JMH GitHub Repository / README https://github.com/openjdk/jmh

• Go testing package https://pkg.go.dev/testing

• Go benchstat https://pkg.go.dev/golang.org/x/perf/cmd/benchstat

• Google Benchmark User Guide https://google.github.io/benchmark/user_guide.html

• How to Correctly Compare the Speed of Different Program Versions on Windows /en/blog/2026/03/16/002-windows-benchmark-comparing-program-versions/

Related Topics

Pages that are easier to understand when read together with this article.

• Technical Topics

Where to Discuss This Topic

Designing performance comparisons, aligning measurement conditions, interpreting results, and digging into root causes are a great fit for the following services.

• Technical Consulting & Design Review
• Bug Investigation & Root Cause Analysis

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

Which of C#, C++, Java, and Go is the fastest?

A single number cannot settle it, because real-world performance is decided by the combination of language, runtime, libraries, build conditions, data, OS, and hardware. What matters is turning the question of which language is fastest into the shape of an experiment: which workload, under which conditions, on which metric are we comparing? On startup time the JIT and initialization costs of C# and Java are easy to see, while at steady state it is not unusual at all for JIT optimization to close most of the gap or even take the lead.

Why separate warm-up in C# and Java benchmarks?

Because C# and Java are normally affected by the JIT, so measuring the first run captures not only the speed of the program itself but also runtime startup, class loading, and JIT preparation. C++ and Go, by contrast, are normally compiled ahead of time. Cold and warm are both meaningful but they do not mean the same thing, so treat cold, which includes the moment right after process startup, and warm, the stable state after several runs, as separate things and keep them out of the same table.

How should a cross-language benchmark be designed?

A two-layer structure works well. For measurement inside a language, use the harness that fits it: BenchmarkDotNet for C#, JMH for Java, go test -bench with benchstat for Go, and Google Benchmark for C++. For cross-language comparison, lining up the output of those harnesses as-is is risky, so the sound approach is to turn each implementation into an executable that answers to the same CLI contract and let an outer common runner randomize the execution order, separate cold from warm, feed the same dataset, verify checksums, and save the raw data.

What should I watch out for in C++ microbenchmarks?

Watch out for the trap where optimization deletes the work. If the compiler decides that nobody uses the computed result, it can remove the computation itself, and the result you measure is not speed but the absence of work. That is why consuming the result, printing a checksum, and using the optimization-suppression facilities of a benchmark framework matter. C++ also exposes differences in conditions very directly, so stating which compiler, which flags such as -O3 or O2, LTO, and PGO, and which STL you measured with is quite important.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.