Security Design for Auto-Update - Why HTTPS Alone Is Not Enough

Table of Contents

1. The Conclusion First
2. Why Auto-Update Is Dangerous Territory
3. Anti-Patterns
4. Best Practices
5. Minimum Safe Configuration
6. How to Think About It in Windows Projects
7. Minimum Checklist
8. Summary
9. References

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (27 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

1. The Conclusion First

Let me start with where this tends to land in practice.

• If the requirements fit, prefer an existing update infrastructure such as MSIX App Installer or ClickOnce first
• If you need a custom updater, the first thing to build is not the UI but signature verification and failure recovery
• Treat update information such as latest.json as signed metadata, not as an unsigned configuration file
• TLS is necessary, but it is not sufficient
• Base the update decision not on “because the server says so” but on “because the client verified it and judged it to be correct”
• Separate development and production signing keys, and protect them with an HSM or a signing service
• On update failure, fail closed rather than fail open
• An updater with no rollback protection is safer to reason about on the assumption that it will be pushed back onto a vulnerable version
• If you are not yet in a position to add signature verification, handing out signed installers manually is safer than auto-update

Let me pin down the terminology first. The rest of the article uses these words in this sense.

In short, the heart of auto-update is not “how to download” but “what to trust, where to verify it, and how to get back when it breaks.”

2. Why Auto-Update Is Dangerous Territory

Ordinary features stay contained inside the app. An updater, by contrast, holds all three of these at once.

1. It fetches files from outside
2. It trusts those files
3. It replaces the executables already installed

In other words, a path to arbitrary code execution is built into the product from day one.

The common misconception here is “it is HTTPS, so it is safe.” TLS is of course necessary. But what it protects is mainly the communication channel and the legitimacy of the endpoint. For a compromised update server, a wrong artifact placed on the legitimate CDN, or an unsigned manifest being swapped out, TLS on its own is not enough.

In fact, looking only at the threats cataloged by TUF (The Update Framework, the specification that defines a trust model for software updates), update systems face all of the following.

• Getting arbitrary malicious software installed
• Rollback, which pushes clients back onto a vulnerable old version
• Freeze, which hides new versions
• Mix-and-match, which combines metadata and artifacts that do not belong together

So auto-update is not “file transfer”; it is “distribution of trust.” Only once you design that part does auto-update start running safely.

2.1 Threats and Countermeasures at a Glance

Threats and countermeasures live in separate chapters of this article, so here is the mapping on one page first. This is enough to frame the design.

The point of this article is that the “Can TLS alone prevent it” column reads either “No” or “Out of scope” all the way down. TLS protects the path; protecting what you ship and who is allowed to ship it takes a different mechanism.

3. Anti-Patterns

First, here is a summary of the dangerous shapes that show up most often in practice.

Let me go through these in a little more detail.

3.1 Stopping at “it is HTTPS, so it is fine”

This is the most common one by far.

• Read latest.json at startup
• Pull out downloadUrl
• Download the zip / exe
• Unpack it and swap the files
• Done

It looks plausible, but the root of trust leans far too heavily on the server’s response. If the update server or the delivery configuration is compromised, a malicious update can be shipped over perfectly valid HTTPS.

TLS is necessary. But TLS alone does not finish the design of an updater.

3.2 Files are signed, but the client never verifies them

Signing files at release time means nothing if the client never looks at the signature.

The shape you see most often is:

• CI does sign the files
• But the updater only checks a hash
• And that hash itself comes from an unsigned manifest

With this shape, the moment the manifest is swapped out, the hash gets swapped along with it. “We check the hash, so it is safe” only holds once the provenance of the hash is protected too.

3.3 The manifest is unsigned

What an update system really has to protect is not just the executable itself. At minimum, the following pieces of information are dangerous if tampered with.

• version / release id
• The URL and file name of the download target
• hash / size
• channel (stable, beta, and so on)
• Whether the update is mandatory
• The applicable OS and architecture
• The metadata expiry
• The minimum required updater version

In other words, the right instinct is to put everything used in the update decision into signed metadata.

3.4 Sloppy handling of signing keys

The security of an update feature is, to a large degree, the security of its key management.

If the production signing key is kept in any of these ways, you are in a dangerous spot.

• Left sitting in the certificate store on a dev PC
• A .pfx uploaded as a CI secret
• The same private key handed out locally to several people
• Development signing and production signing sharing one trust chain

In that state, even a perfectly correct updater cannot stop a “legitimately signed malicious update.”

3.5 Overwrite updates that do not keep the old version

For updates, the design for the failure case matters more than the design for the success case.

• The download was cut off partway
• Unpacking failed
• Power was lost mid-replacement
• The new version launched but died during its first migration

If the old version is gone at that point, recovery gets heavy. In practice, “the app no longer starts on site” is a bigger problem than the fact that “the update failed.”

3.6 Not thinking about rollback

Even a signed, legitimate version can suit an attacker just fine if it is an old, vulnerable one.

For example:

• Version 1.8 has a known vulnerability
• The field has already moved up to 2.3
• The attacker re-delivers 1.8

If that goes through, it is dangerous even though the signature itself is valid.

Checking “is it signed” is not enough. You also have to check “is it acceptable to install this version now.”

3.7 Fail-open

This is the one thing you must never do in production.

• On signature verification failure, show a warning and continue
• A hidden flag that lets you ignore certificate expiry errors
• A debug-only skipVerify=true that survives into production

The worse the outage or the attack, the more these escape hatches become the main path.

4. Best Practices

4.1 Ride on an existing update infrastructure first

It is safest to begin by questioning whether you really need a custom updater at all.

On Windows, as long as the requirements fit, these are the easiest options to consider first.

• MSIX + App Installer
• ClickOnce
• Store / MDM / an internal distribution platform
• MSI + the customer’s own distribution management

The reason is simple: it lets you push a fair share of the responsibility for updating onto the platform. You do give up some freedom, but the update UI, the distribution manifest, package signing, and alignment with operations all become easier to keep consistent.

A custom updater becomes necessary in cases like these.

• You want strict control over multiple channels: stable / beta / preview
• You want staged delivery and rollout percentages
• You need fine-grained control of update timing for business-specific reasons
• You have a configuration that does not fit MSIX or ClickOnce

Even then, you will stay on firmer ground if you frame it not as “we want more freedom” but as “we are taking on the responsibility for updates ourselves.”

4.2 Put the root of trust on the client side

A safe updater does not take the server’s response at face value. The client side needs at least these two things.

1. A trusted public key or certificate chain
2. A mechanism that verifies metadata signed with that key

Put differently, you need to reach a state where the client can confirm not “the server says this is the latest version” but “this metadata is the latest version, issued by a signer we trust.”

Here is how trust runs from the root all the way down to the file.

flowchart TD
    ROOT["root key<br/>the trust anchor embedded in the client"] --> SIGNKEY["metadata signing key<br/>delegated from root and re-issued often"]
    SIGNKEY --> META["signed update metadata<br/>version / url / hash / size / expiry"]
    META --> CHECK1{"Verify signature / expiry / version"}
    CHECK1 -- "Fail" --> STOP["Abort the update<br/>fail-closed"]
    CHECK1 -- "Pass" --> DL["Download the artifact into the staging area"]
    DL --> CHECK2{"Verify size / hash / package signature"}
    CHECK2 -- "Fail" --> STOP
    CHECK2 -- "Pass" --> ACT["Activate while keeping the old version"]
    ACT --> HEALTH{"Health check on first launch"}
    HEALTH -- "Fail" --> RB["Roll back to the old version"]
    HEALTH -- "Pass" --> DONE["Update complete"]

There are two things to take from the diagram. Top to bottom, trust runs as a single unbroken chain. And wherever that chain breaks, the destination is either Abort the update or rollback, never proceed anyway.

4.3 Design around signed metadata

At minimum, put the following into the update metadata and make it part of what gets signed.

The important part here is to consolidate every input to the update decision into signed metadata. Converging on a shape where the logic lives on the client and the authenticity of the information is protected by signatures is what keeps things from going wrong.

Without a concrete shape this does not turn into code, so here is a minimal example. First, the content that gets signed.

{
  "schema_version": 1,
  "channel": "stable",
  "release_version": "2.4.1",
  "published_at": "2026-04-09T01:00:00Z",
  "expires_at": "2026-04-16T01:00:00Z",
  "minimum_updater_version": "2.0.0",
  "minimum_allowed_version": "2.2.0",
  "mandatory": false,
  "artifacts": [
    {
      "os": "windows",
      "arch": "x64",
      "package_type": "msi",
      "file_name": "MyApp-2.4.1-x64.msi",
      "url": "https://updates.example.com/stable/MyApp-2.4.1-x64.msi",
      "size": 48234496,
      "sha256": "5f2c...64 hex characters..."
    }
  ]
}

Then wrap that together with the signature.

{
  "signed": {
    "schema_version": 1,
    "channel": "stable",
    "release_version": "2.4.1",
    "_comment": "the object above, inlined as is"
  },
  "signatures": [
    {
      "keyid": "3f9a...",
      "sig": "MEUCIQ..."
    }
  ]
}

The reason for this shape is that every value used in the update decision sits inside signed. The URL, the version, the channel, and the mandatory-update flag are all on the inside, so swapping out only the outside makes verification fail. Fix the client-side order of operations as “verify signed, and once it passes, use only the values inside it.” An implementation that reads url and starts downloading before verification throws away the whole point of this shape.

There is one implementation detail to watch. In JSON, the byte sequence changes with key ordering and whitespace. Signature verification runs against bytes, so decide up front whether what you sign is a canonicalized representation or the received bytes themselves. Leave that undecided and you end up with the server signing the object it generated while the client verifies against the result of re-serializing it, which makes a perfectly good update fail verification. And if you start working around that mismatch by deciding “let’s just let it through,” you have taken the first step toward fail-open.

4.4 Verify the artifact itself as well

After verifying the metadata, check the following on the downloaded artifact too.

• size
• hash
• Package signature / code signature
• The publisher, or the identifier you expect

If you are dealing with Windows PE / MSI / MSIX, it is safer to assume that Authenticode or package signature verification happens on the client side. On macOS, you stay on firmer ground by assuming Developer ID and notarization on the update path as well.

4.5 Protect keys through operations, not features

Key management shows its differences in operations rather than in implementation.

At minimum, keep these separate.

• The development signing key
• The staging signing key
• The production signing key

And for production, you want the design to cover:

• HSM
• A cloud signing service
• A signing system with an approval flow
• Audit logs
• A key rotation procedure
• Timestamped signatures

“CI signs automatically whenever a production build passes” is convenient, but it also widens the blast radius of a compromise. At the very least, who signed what and when should be traceable.

Once operations have settled in, it is safer still to split the rarely-changed root trust from the key used to re-sign update metadata frequently. Keeping the root mostly offline and using a separate key for update metadata makes it easier to shrink the blast radius of a key compromise.

4.6 Fail-closed and staged updates

The update flow basically runs in this order.

1. Fetch the metadata
2. Verify the signature, the expiry, and the version
3. Download the artifact into a staging area
4. Verify hash / size / signature
5. Prepare activation while keeping the old version
6. Switch over at restart, or through a dedicated helper
7. Health-check the first launch
8. Roll back if something is wrong

What matters here are these two rules: Do not replace anything before verification finishes. Do not proceed once something fails.

4.7 Narrow the updater’s privileges

Avoid running the whole updater with administrator privileges.

The ideal separation is:

• Download and verification: low privilege
• The actual file replacement only: a helper with minimal privileges
• The helper does nothing beyond “place a verified package in the designated location”

The more a design depends on privilege elevation, the more dangerous it becomes unless you draw a clear line around what has already been verified before elevation happens.

4.8 Close off rollback, freeze, and mix-and-match from the start

These are painful to retrofit, so build them in at the beginning.

• Rollback protection The client keeps “the highest metadata version / release version seen so far” and rejects anything older

• Freeze protection Give the metadata an expiry and reject metadata that is too old

• Mix-and-match protection Keep the pieces of metadata consistent with each other. At minimum, pin the target artifact hash, size, and version inside the manifest itself

On top of that, being able to distribute a blocklist of specific builds or a minimum allowed version through signed metadata makes containment much faster when something goes wrong.

Even if you do not adopt TUF wholesale, these three properties matter a great deal.

4.9 Start with full updates

Delta updates are effective for bandwidth, but they are complex as a first implementation.

• Which old version to which new version does a given delta apply
• The precondition hash before the delta is applied
• The final hash after the delta is applied
• Recovery when application fails partway
• Cleaning up partial applications and stale deltas

All of this piles on at once. For the first version, safely swapping in a signed full package is plenty.

5. Minimum Safe Configuration

Even without going as far as full TUF, the minimum safe configuration for a custom updater ends up looking roughly like this.

5.1 What the client holds

What the client holds is the material it needs in order to doubt the server’s response. If this is empty, whether to update or not is decided purely on the server’s word.

• A trusted root public key, or a pinned certificate chain
• The version currently running
• The highest metadata version / release version seen in the past
• The channels it is allowed to accept
• The immediately preceding version, kept for rollback

5.2 What the server returns

What the server returns is not the decision itself but the material for the decision. None of it is trusted on its own; it only means something once it passes verification against the trust anchor from 5.1.

• Signed update metadata
• Artifacts that are signed, or platform-signed
• If needed, blocklist and minimum allowed version information

5.3 A typical flow

Fetch the metadata
  ↓
Verify signature, expiry, version, channel
  ↓
Download the artifact into staging
  ↓
Verify size / hash / package signature
  ↓
Activate while keeping the old version
  ↓
Roll back if the first launch fails

The important point here is that nothing holds up on the update server’s response alone. What makes it hold up is the trust anchor the client carries, and the verification logic.

6. How to Think About It in Windows Projects

For Windows apps, it is easiest to work backward from the distribution method.

• If the requirements fit, MSIX App Installer
• For an internal .NET app where per-user works, ClickOnce
• If you need services, drivers, shell extensions, or custom channel control, MSI plus a custom updater is also worth comparing

Choosing a custom updater, however, does not reduce the work. If anything, it increases it.

• Authenticode / package signature verification
• A signed manifest
• Rollback protection
• Privilege separation for the update helper
• An update strategy for the updater itself

6.1 How to verify an Authenticode signature on the client

Writing “verify Authenticode” does not turn into code by itself, so here are the entry points on Windows.

In PowerShell, the minimum check is just this.

$path = ".\MyApp-2.4.1-x64.msi"
$sig = Get-AuthenticodeSignature -FilePath $path

if ($sig.Status -ne 'Valid') {
    throw "signature check failed: $($sig.Status) / $($sig.StatusMessage)"
}

# A valid signature is not enough on its own. Pin down whose signature it is.
# But Subject (the distinguished name) is not a unique identifier. A certificate
# with the same CN/O/C can be issued by any number of other CAs, and if this
# machine trusts one of them, Status comes back Valid and the Subject comparison
# passes too.
# What to pin is the issuing chain or the public key; Subject is a secondary filter.
$expectedIssuers = @(                      # thumbprints of the issuers (intermediate CA / root)
    '9F86D081884C7D659A2FEAA0C55AD015A3BF4F1B'   # <- replace with the real value
)
$expectedSubject = 'CN=Example Software Inc., O=Example Software Inc., C=JP'

# The chain rebuilt here exists only to walk up to the issuer; whether the file
# can be trusted was already settled by Status = Valid above.
# X509ChainPolicy.VerificationTime defaults to the moment the constructor ran
# (= now), so building as is fails with NotTimeValid the moment the signing
# certificate expires. A timestamped signature stays Valid past expiry, so
# leaving this at the default gives you an updater that rejects every past
# release it verified correctly, the instant the certificate is renewed.
# If the signing time is known, put it in VerificationTime. If it is not, do not
# look at the validity period in this Build, since it was checked above
$chain = [System.Security.Cryptography.X509Certificates.X509Chain]::new()
$chain.ChainPolicy.VerificationFlags = 'IgnoreNotTimeValid'
$chain.ChainPolicy.RevocationMode    = 'NoCheck'
try {
    $built = $chain.Build($sig.SignerCertificate)

    # An expected issuer has to appear somewhere between the signer and the root.
    # When Build fails the chain is only partially populated, so that case
    # is caught by this same check (fail-closed)
    $chainThumbprints = @($chain.ChainElements | ForEach-Object { $_.Certificate.Thumbprint })
    if (-not ($expectedIssuers | Where-Object { $chainThumbprints -contains $_ })) {
        throw "unexpected issuing chain (built=$built): $($chainThumbprints -join ' / ')"
    }
}
finally {
    $chain.Dispose()
}

if ($sig.SignerCertificate.Subject -ne $expectedSubject) {
    throw "unexpected signer: $($sig.SignerCertificate.Subject)"
}

There are five constraints worth keeping in mind here.

• Even when Status is Valid, that only means the signature itself is well-formed and trusted. Whose signature it is has to be checked separately
• A matching Subject is not proof of identity. A distinguished name is not a unique identifier. Both a corporate CA and a public CA can issue a code signing certificate carrying the same distinguished name, CN=Example Software Inc., O=Example Software Inc., C=JP. If the client trusts that CA, a substituted build signed with a different key by a different issuer will pass with Status = Valid and a matching Subject. What you pin is either the issuing chain (that the thumbprint of the expected intermediate CA or root appears on the path from the signer to the root) or the public key, and Subject is then used to narrow things down on top of that
• Once you pin something, decide the procedure for changing it in advance. Otherwise the failure mode is that updates stop on every machine on the day you switch certificates or CAs. Always hold the expected values in an array, and get to a state where old and new are accepted side by side before you switch (three steps: ship an updater whose allowlist already includes the new issuer, then switch the signature, then drop the old value). Shipping the updater first is only possible while you can still update the updater, so design this together with the “update strategy for the updater itself” listed at the top of chapter 6
• A signature without a timestamp stops verifying the moment the certificate’s validity period ends. Always attach a timestamp at release time. That is exactly why timestamped signatures appear in 4.5
• Do not let the X509Chain you build to walk up to the issuer re-evaluate the validity period against the current time. X509ChainPolicy.VerificationTime defaults to the moment the constructor ran, that is, now. The documentation states it explicitly: when verifying a signed message, the signature has to have been valid at the time of signing rather than at the time of verification, which is what makes this property important. Leave it at the default and you reject a package that a timestamp just made Status = Valid, one step later, as having an expired certificate. That wipes out the point of attaching a timestamp, and every past release drops out the instant you renew the certificate. If the signing time is known, put it in VerificationTime; if it is not, add IgnoreNotTimeValid. This is not a weakening. The validity period and the trust decision were already settled by Status = Valid one step above, and this Build exists only to find out who issued the certificate. For the same reason, revocation is not checked here either (a CRL for an expired certificate is not guaranteed to keep being published in the first place). What stops a leaked key is not a CRL but the blocklist and the minimum allowed version (4.8)

One more thing: the verification result depends on that machine’s certificate store and trust settings. In an environment where the client-side trust settings are loose, the meaning of Status = Valid gets loose too. If the updater itself carries the issuers it expects, as above, widening the trust settings on the machine does not affect it.

The classic dangerous shape on Windows is the straight line DownloadFile -> unzip -> kill process -> overwrite -> restart. It can work, but it is weak on both security and recoverability.

Getting users through SmartScreen or UAC warnings with “More info -> Run anyway” is not update design; it is training people to ignore warnings. If you are building a proper update path, converge on a distribution and verification setup that does not trigger warnings in the first place, rather than getting users used to them.

The comparison of distribution methods themselves is covered in this article. Choosing a Windows App Distribution Method - MSI/MSIX/ClickOnce/xcopy/Custom Updater

7. Minimum Checklist

Before shipping a custom updater, you want at least this much confirmed.

• The update metadata is signed
• The metadata contains version / hash / size / channel / expiry
• The client verifies the signature and the version
• The signer is pinned by the issuing chain or the public key, not by the distinguished name (Subject)
• There is a defined procedure for changing the pinned values, including a window in which old and new are both accepted
• The artifact’s hash and platform signature are verified
• The production signing key is separated from the development environment
• Key usage logs and approval records are kept
• Timestamped signatures are used
• Staged updates switch over while keeping the old version
• There are conditions and a procedure for rollback
• On verification failure, the updater fails closed and stops
• There is an update policy for the updater itself
• A blocklist and a minimum allowed version can be distributed
• There is a kill switch to halt a staged rollout
• Failure rate, rollback rate, and signature verification failures are observable

If a lot of these boxes are still empty, tightening up the distribution trust model will do more good than building the updater’s UI first.

8. Summary

In the end, the security of an auto-update feature comes down to this.

Design not the convenience of the update, but whom you trust and how the client verifies that trust.

On top of that, the practical judgment goes roughly like this.

• If an existing platform is enough, ride on it first
• If you build a custom updater, put in signed metadata and key management before you worry about HTTPS
• An updater with no designed failure recovery and rollback will hurt in production
• The updater is not a distribution feature; it is the product’s security boundary itself

If your current setup is close to latest.json + zip swap, the first thing to fix is not the download logic but where you place your trust. Fixing that alone changes the risk profile considerably.

9. References

• CISA Secure by Design Pledge
• NIST: Security Considerations for Code Signing
• NIST Secure Software Development Framework (SSDF)
• The Update Framework Specification
• TUF: Roles and metadata
• TUF: Security
• Microsoft Learn: Authenticode Digital Signatures
• Microsoft Learn: WinVerifyTrust function
• Microsoft Learn: Get-AuthenticodeSignature
• Microsoft Learn: Auto-update and repair apps - MSIX
• Microsoft Learn: ClickOnce Deployment and Security
• Apple Developer: Developer ID
• CA/Browser Forum: Baseline Requirements for the Issuance and Management of Publicly-Trusted Code Signing Certificates

Related Topics

These topic pages sit close to this theme. Starting from the article, you can move on to related services and other articles.

Windows Technical Topics

An entry point that gathers technical topics on Windows development, defect investigation, and making use of existing assets.

Services Related to This Theme

Windows App Development

Auto-update is not just a UI matter. It is a design that spans the distribution method, privileges, recovery, and operations. For new Windows app development or a review of existing software, we can start from sorting out the update approach.

Technical Consulting and Design Review

You can bring us in at the sorting-out stage, with questions like “do we actually need a custom updater,” “is MSIX or ClickOnce enough,” and “where is our current update design risky.”

Author Profile

Go Komura

Representative, KomuraSoft LLC

Working mainly on Windows software development, technical consulting, and defect investigation, with particular strength in projects that carry existing assets and in investigating failures whose cause is hard to see.

• View profile
• Contact us

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

If auto-update runs over HTTPS, isn't that safe?

TLS is necessary, but it is not sufficient. What TLS protects is mainly the communication channel and the legitimacy of the endpoint, so it does nothing about a compromised update server, a wrong artifact placed on the legitimate CDN, or an unsigned manifest being swapped out. The update decision has to rest not on 'because the server says so' but on 'because the client verified signed metadata and judged it to be correct'.

What should go into update metadata and be signed?

The basic rule is to consolidate every input to the update decision into signed metadata. Concretely, that means the release version, the artifact URL and file name, hash and size, the channel (stable, beta, and so on), the target OS and architecture, the minimum updater version, the metadata expiry (expires_at), and a flag for whether the update is mandatory. If only the binary is signed and the manifest is left unsigned, the URL, the version, and the mandatory-update flag remain open to tampering.

What is a rollback attack, and how do you prevent it?

It is an attack in which an attacker re-delivers an old version that carries a known vulnerability, even though that version is legitimately signed, to push clients back onto the vulnerable build. Because the signature itself is valid, signature verification alone cannot stop it. The countermeasure is for the client to keep the highest release version it has ever seen and reject anything older. Alongside that, give the metadata an expiry to block freeze attacks that hide new versions, and pin the artifact hash, size, and version in the manifest to close off mix-and-match attacks.

Should I build my own updater, or use an existing mechanism?

If the requirements fit, the safe move is to prefer an existing update infrastructure such as MSIX App Installer or ClickOnce first, because it lets you push the responsibility for updates onto the platform. A custom updater becomes necessary when you have requirements the existing infrastructure cannot carry, such as strict control over multiple channels or staged delivery. Even then, the first thing to build is not the UI but signature verification and failure recovery: hold the fail-closed line so that nothing proceeds when verification fails, and switch over while keeping the old version in place so you can roll back.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.