Time Travel Debugging — Recording and Rewinding the Bugs That Never Reproduce in Long-Running Apps

“It crashes once a month, only in the middle of the night.” “The same operation never reproduces it on my machine.” “We got a dump, but looking at the crash site does not tell us why the value became what it is.” Among the bug investigations of long-running Windows apps, these are the cases that consume the most time. In an earlier article, “Reading Crash Dumps with WinDbg + SOS,” we looked at how to read a crash dump, which is a single photograph. This article continues from there and covers the tool for the situations where a photograph is not enough: Time Travel Debugging (TTD).

TTD is a WinDbg feature that records the entire execution of a process and lets you replay it later, forward and backward. Instead of trying over and over to reproduce a bug, you can “rewind” the debugger session.¹ The intended readers are developers and maintainers of Windows, .NET, and C++ applications who have already done their share of dump and log investigation and still have cases they cannot get to the root cause of. The prerequisite environment is Windows 10/11 or Windows Server 2016 or later with the current WinDbg, and recording requires administrator privileges.¹² The difficulty is intermediate.

Assumptions of This Article

1. The Bottom Line First

• A dump preserves “state”; TTD preserves the “path.” The official documentation states that dumps tend to miss the state and execution path that led to the failure.¹ If a photograph of the moment of the crash does not tell you the cause, what you need next is not more photographs but a recording.
• Recording is heavy. While recording, the target process runs 5 to 20 times slower or worse, and the trace file grows by 5 to 50 MB per second while the process is active, with no cap.²⁴ It is not a tool you attach unconditionally to a long-running app.
• For long-running apps, design “what to record.” Chapter 5 covers the four entry points TTD.exe provides: -ring/-maxFile (keep only the last N MB), -module (record only while your own module is executing), -recordmode Manual (let the app specify the recording interval), and -monitor (record every launch).²
• Replay revolves around three things: positions (!tt), events (dx @$curprocess.TTD.Events), and queries (TTD.Calls / TTD.Memory). Combine ba (a break-on-access breakpoint) with g- (reverse execution), and the debugger answers “who last wrote this value?” directly.⁵
• A trace contains the contents of memory. It can include personal and confidential information such as file paths, registry data, and the contents of memory and files.¹ Design sharing and storage as you would for a confidential file.
• Know what TTD cannot do before you start. It cannot record kernel mode, it cannot inject into protected processes (PPL), it cannot detach itself once attached, and memory cannot be modified during replay.³²

In the diagram a solid line marks a relation that always holds and a dashed line marks a conditional one (the conditions are given per relation on the detail page). The full list of relations (32 in total, with evidence and certainty) and the definitions of the main concepts are collected on the knowledge map detail page (in Japanese). Data: JSON-LD / Turtle

2. Where a Dump Is Not Enough

A crash dump is a copy of memory and registers at the moment the process crashed (or was stopped). As described in the SOS analysis article, !clrstack tells you where it crashed and !dumpheap -stat tells you what is consuming the heap. What it cannot tell you is what happened on the way to that state.

flowchart TB
    accTitle: What a dump captures and what TTD captures
    accDescr: A crash dump captures only the state at the moment of the crash, not the path that led there. TTD keeps the entire instruction execution from the start of recording to the end, so the path is preserved along with the state
    dump["Crash dump: state at the moment of the crash"] --> q1["Does not show why the value became what it is"]
    ttd["TTD trace: instruction execution of the recorded interval"] --> q2["Can go back to the position where the value was written"]
    q1 -.-> gap["This gap is what drags out long-running investigations"]

Figure 1: A dump is state; TTD is the path. In long-running bugs, what you want is usually the latter.

Typical cases look like these.

• One field of a structure on the heap holds an impossible value. The dump shows the corrupted value, but not who wrote it or when.
• The location of the exception is known, but not why the argument passed to it was invalid. Walking further up the callers, the function that produced the value along the way is no longer on the stack.
• Handles or memory grow over the course of a month. A single dump can only say “it is growing”; the call path that grew it is not there.

flowchart TB
    accTitle: What a dump misses in typical long-running cases
    accDescr: The corrupted value is captured but not who wrote it, the exception position is captured but the function that produced the argument is no longer on the stack, and the resource growth is captured but not the path that grew it; the three kinds of case share this
    c1["Corrupted field"] --> m1["No record of who wrote it or when"]
    c2["Exception on an invalid argument"] --> m2["Function that produced the value is no longer on the stack"]
    c3["Resource that grows over a month"] --> m3["No record of the call path that grew it"]
    m1 --> same["Common point: state exists, but no path"]
    m2 --> same
    m3 --> same

Figure 2: All three cases have “state but no path.” More photographs do not fill in the path.

Microsoft’s documentation compares the strengths and weaknesses of the investigation methods as follows.¹

There is one more important property that the official TTD walkthrough points out. When the debugger stops at the point of failure, that point is often inside error-handling code several steps past the real cause.⁵ A dump is always taken at this “several steps later” position. With TTD, you can step back from there one instruction at a time.

flowchart TB
    accTitle: The gap between the point of failure and the real cause
    accDescr: The point of failure where a dump is taken is often inside error handling several steps past the real cause, and with TTD you can rewind from that point instruction by instruction back to the cause
    cause["Real cause (the instruction that corrupted the value)"] --> steps["Several steps forward"]
    steps --> fail["Point of failure (exception, error handling)"]
    fail -->|"Dump"| photo["Frozen here"]
    fail -->|"TTD"| back["Go back with p- / t- / g-"]
    back --> cause

Figure 3: A dump is frozen at the point of failure. TTD can walk back from the point of failure to the cause.

3. How TTD Works and What It Costs

3.1 What Is Being Recorded

TTD injects a recording engine into the target process and records the executed instructions, instruction by instruction. In the words of the official documentation, it “encodes a complete instruction-level trace at an average of less than one byte per instruction”; in practice it lands somewhere between one bit and one byte per instruction. Programs that execute few kinds of functions and handle little data produce smaller traces; the opposite produces larger ones.²⁴

A recording produces two files.¹

flowchart TB
    accTitle: The flow from TTD recording to replay
    accDescr: A recording engine is injected into the target process and instruction execution is recorded to a .run file; when WinDbg opens the .run file it creates an .idx index and replays forward and backward using positions, events, and queries
    proc["Target process"] --> inj["Recording engine injected (TTDRecordCPU)"]
    inj --> run[".run (record of instruction execution)"]
    run --> open["Open in WinDbg"]
    open --> idx["Generate .idx (index)"]
    idx --> play["Replay forward and backward with positions, events, and queries"]

Figure 4: Recording is done by TTD.exe or WinDbg; reading is done by WinDbg. Sharing only the .run file is enough.

Time inside a .run file is expressed as a “position.” It takes the form of two hexadecimal numbers separated by a colon, such as 12:0 or 1A0:12F; the first half is the sequencing number (corresponding to a sequencing event) and the second half is the approximate number of instructions since that event.⁶ FFFFFFFFFFFFFFFE:0 means the end of the trace.⁷ Positions take center stage in Chapter 6.

flowchart TB
    accTitle: How positions in a trace are expressed
    accDescr: A position is a hexadecimal sequencing number and step count separated by a colon; the start is near 0, the end is expressed as FFFFFFFFFFFFFFFE:0, and you can also move to an approximate position by percentage
    pos["Position xx:yy (hexadecimal)"] --> seq["xx: sequencing number"]
    pos --> step["yy: instructions since that event"]
    seq --> tail["End is FFFFFFFFFFFFFFFE:0"]
    step --> pct["Percentages such as !tt 50 also work"]

Figure 5: A position is “event number:instruction count.” It is not wall-clock time, but it can be converted to wall-clock time as shown in Chapter 6.

3.2 The Cost

The official documentation itself describes TTD as an “invasive technology.”² Here is the cost in numbers.

flowchart TB
    accTitle: The cost of recording and its effect on long-running apps
    accDescr: The slowdown while recording, file growth, the silent wait when the disk runs out, and the inability to detach after attaching are the four costs that make it impossible to attach TTD unconditionally to a long-running app
    cost["TTD recording"] --> slow["5 to 20 times slower"]
    cost --> grow["Grows 5 to 50 MB per second, no cap"]
    cost --> disk["Waits silently when the disk runs out"]
    cost --> stuck["Cannot detach after attaching"]
    slow --> no["Unconditional continuous recording is not viable"]
    grow --> no
    disk --> no
    stuck --> no

Figure 6: Any one of the four costs rules out continuous recording. That is why the “recording design” of Chapter 5 is needed.

3.3 What It Cannot Do

• User mode only. It can record only the user-mode execution of a process; code that runs in kernel mode, such as drivers, cannot be debugged.³
• Protected processes. TTD cannot inject itself into Windows protected processes such as Protected Process Light (PPL).³
• Replay is read-only. You can go back in time, but you cannot change history. Commands that read memory work; commands that modify it do not.³
• Incompatibility with antivirus and memory-monitoring software. Because TTD hooks into the process, it conflicts with software that tracks or shadows system memory calls. If recording produces an error that looks like insufficient permissions, disable such software temporarily to isolate the problem. The Electron framework is a known conflict as well; even if a recording succeeds, the target process may deadlock or crash.³
• UWP apps cannot be launched and recorded (attaching to an already-running UWP app is possible). “Unusual processes” running in another session or a different security context are currently not supported either.⁸

flowchart TB
    accTitle: What TTD cannot record or do
    accDescr: Kernel-mode code, protected processes, launch recording of UWP apps, and processes in another session or security context cannot be recorded; memory cannot be modified during replay; and antivirus software and Electron can conflict
    no["TTD limitations"] --> g1["Cannot be recorded"]
    no --> g2["Limitations and conflicts"]
    g1 --> k["Kernel-mode code (drivers, etc.)"]
    k --> ppl["Protected processes (PPL)"]
    ppl --> uwp["Launch recording of UWP (attach is possible)"]
    uwp --> sess["Other sessions and security contexts"]
    g2 --> ro["Replay is read-only"]
    ro --> av["May conflict with antivirus and Electron"]

Figure 7: The limitations come in two kinds: “cannot record” and “conflicts.” The latter has to be isolated per environment.

The last item trips up anyone who wants to record a service. The TTD.exe documentation describes -attach as intended for “investigating services and long-running apps” and -monitor as recording “every time a program or service starts,”² which does not simply square with the statement on the troubleshooting page. In practice, the safe approach is to verify per environment in this order: confirm that ping.exe or cmd.exe can be recorded in the same configuration as production, then try the target process.⁸

4. Recording — The WinDbg UI and TTD.exe

There are two entry points for recording: the WinDbg UI, or the command-line TTD.exe.

4.1 Recording from the WinDbg UI

Run WinDbg as administrator (elevation is mandatory for TTD¹), choose File > Start debugging > Launch executable (advanced), specify the executable, and check Record with Time Travel Debugging. Choosing Configure and Record lets you set the location of the trace file and Record subset of execution (limit the recorded modules with a comma-separated list such as notepad.exe,kernelbase.dll). For a process that is already running, choose File > Start debugging > Attach to process and likewise check Record Process with Time Travel Debugging.⁹

While recording, a small dialog with “Stop and Debug” and “Cancel” buttons appears. When the app exits (or crashes), the trace is closed and WinDbg opens it automatically and builds the index.⁵

flowchart TB
    accTitle: The flow of recording from the WinDbg UI
    accDescr: In WinDbg started as administrator, choose Launch executable (advanced) or Attach to process, check Record with Time Travel Debugging, set the save location and module filter in Configure and Record, go through the recording dialog, and when the app exits the trace is closed and indexed automatically
    adm["Start WinDbg as administrator"] --> pick["Launch executable (advanced) / Attach to process"]
    pick --> chk["Check Record with Time Travel Debugging"]
    chk --> cfg["Configure and Record: save location, module filter"]
    cfg --> rec["Recording dialog (Stop and Debug)"]
    rec --> fin["App exits, trace is closed and indexed automatically"]

Figure 8: Recording from the UI takes five steps. Skip “start as administrator” and you stop at the first dialog.

4.2 Recording with TTD.exe

When you need to record on a PC where WinDbg cannot be installed, or to automate recording, use TTD.exe on its own. It installs through App Installer from https://aka.ms/ttd/download, and after installation you can verify it with ttd.exe -help. For offline environments, Microsoft also provides an official procedure (with a PowerShell script) for unpacking the MSIX bundle by hand and extracting only the binaries.² Recording requires administrator privileges and is normally run from an administrator command prompt.²

There are three recording modes.²

flowchart TB
    accTitle: The three recording modes of TTD.exe
    accDescr: launch starts a new process with arguments and records it, but it runs with elevated privileges. attach attaches to a running process by PID. monitor records every time the specified program starts, and the launch happens with normal privileges
    m["TTD.exe recording modes"] --> l["-launch: launch and record"]
    m --> a["-attach: attach to a running PID"]
    m --> mo["-monitor: record every launch"]
    l -.-> lp["Launched with administrator privileges"]
    a -.-> ap["Keeps normal privileges"]
    mo -.-> mp["Normal launch path, suited to automation"]

Figure 9: Only -launch can pass arguments, but it runs with elevated privileges. To record behavior close to production, use -attach or -monitor.

:: Launch and record (default mode; -launch can be omitted)
TTD.exe -out C:\traces MyApp.exe --config prod.json

:: Attach to a running process (create the output directory first)
TTD.exe -attach 21440 -out C:\traces\MyApp.run

:: Record every launch (Ctrl+C ends monitoring; -out requires a full path)
TTD.exe -out C:\traces\ -monitor MyApp.exe

• -launch is the only mode that can pass arguments, but the program is launched with the same (administrator) privileges as TTD.exe. For apps whose behavior changes with privileges, record with -attach or -monitor to keep normal privileges.²
• -attach requires the output directory to exist already. If you specify a file name, no file with that name may exist.²
• -monitor installs a process-launch monitoring driver and records every time the specified program (more than one can be given) starts. It stays in effect until reboot and is stopped with Ctrl+C. Its advantages are that you do not have to assemble the launch yourself, the target runs with normal privileges, and it suits scripted automation. Adding -cmdLineFilter "string" records only launches whose command line contains that string.²
• -children records child processes too, but each process gets its own .run file, and WinDbg can open only one at a time.²

While recording, a small UI with two buttons appears: “Tracing Off” (stop recording and let the app continue) and “Exit App” (close the app and end recording). For automation, hide it with -noUI and accept the EULA with -accepteula.² The recording log is kept in an .out file in the same location as the .run file, where you can read the wall-clock start and end times of the recording, the length of the recording session (simulation time), whether it was a launch or an attach, and the OS version. When a recording fails, some error messages appear only in the .out file.²

5. Recording Design for Long-Running Apps

This is the heart of the article. Given the costs in Chapter 3, catching a bug that occurs at an unknown time in a process that runs for days requires a recording design. TTD.exe provides four means, and you choose by the nature of the symptom.

flowchart TB
    accTitle: Choosing the recording scope by the symptom of a long-running app
    accDescr: If you do not know when it will occur, keep only the last part with a ring buffer; if you know which module is suspect, record only that module; if you can modify the app, specify the interval with the manual recording API; if it appears only at startup or on particular launches, record every launch with monitor mode
    q{"Nature of the symptom?"} -->|"Unknown when it occurs"| ring["-ring / -maxFile: keep only the end"]
    q -->|"Only at startup or on particular launches"| mon["-monitor: record every launch"]
    ring -->|"Suspect module is clear"| mod["Add -module"]
    ring -->|"App can be modified"| man["Specify the interval with -recordmode Manual"]

Figure 10: The four are not mutually exclusive. Combining -ring with -module is the most practical combination for long-running apps.

5.1 Ring Buffer — Keep Only the Last N MB

With -ring, the trace is written to a ring buffer of the size given by -maxFile, and the file never grows beyond that limit. What remains is only the last part of the recording that fits in that size.² The unit of -maxFile is MB; in ring-buffer mode the default is 2,048 MB, the minimum 1 MB, and the maximum 32,768 MB (the default for the in-memory ring of a 32-bit process is 256 MB).²

:: Attach to a running monitoring app and keep only the most recent 4 GB of execution
TTD.exe -accepteula -noUI -attach 21440 -ring -maxFile 4096 -out C:\traces\MyApp.run

:: When the symptom is detected, stop recording (the app keeps running)
TTD.exe -stop 21440

-stop accepts a process name, a PID, or all, and stops that recording. -wait <seconds> waits until every recording session on the system has finished (-1 for indefinitely), and is used in automation scripts to enforce the order “stop, then collect the file.”²

flowchart TB
    accTitle: Timeline of a ring-buffer recording
    accDescr: Recording starts on attach, older parts are pushed out of the ring buffer, and when the symptom appears and stop is issued, only the most recent maxFile worth remains as the trace
    s["Start recording with -attach -ring"] --> old["Older intervals are pushed out"]
    old --> sym["Symptom appears"]
    sym --> stop["Stop recording with -stop"]
    stop --> keep["Only the most recent maxFile worth remains"]
    keep -.-> note["Size it so the moments before the symptom fit in the buffer"]

Figure 11: The ring buffer is a device for keeping “the moments before the symptom.” Work -maxFile backward from “time from noticing the symptom to stopping, multiplied by growth per second.”

There are two design points.

1. Make the buffer large enough to absorb the time from noticing to stopping. In an active process the trace grows by 5 to 50 MB per second,⁴ so a 4 GB ring corresponds to one to two minutes under heavy load and a little over ten minutes under light load. You need a mechanism that keeps the lag between detecting the symptom (a specific log line, a counter threshold, an alert from monitoring) and -stop within that window.
2. Stopping does not detach. -stop stops the recording, but TTD does not detach itself from the target process.² Ending the recording completely requires terminating the process, so include “restart at the next maintenance window after collecting the trace” in the operating procedure.

sequenceDiagram
    accTitle: Stopping a ring-buffer recording in coordination with monitoring
    accDescr: When the monitoring side detects the symptom through logs or counters, it calls TTD.exe stop, collects the finalized .run file, and restarts the process at the next maintenance window to remove TTD
    participant W as Monitoring (logs, counters)
    participant T as TTD.exe
    participant P as Target process
    W->>W: Detect the symptom
    W->>T: -stop PID
    T->>P: Stop recording (process continues)
    T-->>W: .run is finalized
    W->>W: Collect .run, encrypt, and store
    W->>P: Restart at the next maintenance window

Figure 12: Only when the lag between detection and -stop fits in the buffer do the moments before the symptom remain in the trace. The restart is part of the operation.

5.2 Module Filter — Record Only While Your Own Code Runs

-module <module name> records only the specified module (the executable itself or a loaded DLL; more than one can be given) and the code that module calls. The target process runs at full speed until code in the specified module executes; recording starts when execution enters the module, stops when it leaves, and the process returns to full speed. Because turning recording on and off is expensive, recording stays on while the specified module is calling other modules in the process.²

:: Record only while our measurement-logic DLL is running (combined with a ring)
TTD.exe -accepteula -noUI -attach 21440 -module MeasureCore.dll -ring -maxFile 2048 -out C:\traces\

A trace made this way simply skips the intervals where recording was off, treating “the next instruction” as the first instruction after recording resumed, and you debug it no differently from a trace of the whole process.² In long-running apps, the UI idle loop and the framework’s internal processing tend to account for most of the executed instructions, so restricting recording to your own module alone reduces overhead and file size substantially.

flowchart TB
    accTitle: How module-filtered recording behaves
    accDescr: The target process runs at full speed outside the specified module; recording starts when code in the specified module is entered, continues while that module calls other modules, and stops when execution leaves the module
    out1["Outside the specified module: full speed"] --> in1["Enter the specified module: recording starts"]
    in1 --> callee["Other modules it calls: recording continues"]
    callee --> out2["Leave the specified module: recording stops"]
    out2 --> out1

Figure 13: Because the Win32 APIs and runtime that your DLL calls are recorded as well, this is enough to trace “what our code handed to the OS.”

5.3 Manual Recording — Let the App Specify the Interval

With -recordmode Manual, the process keeps running at full speed even after TTD is injected, and recording happens only when the program calls TTD’s in-process recording API (the default, Automatic, records from the moment of injection).² The API documentation and samples are in the WinDbg-Samples repository on GitHub.¹⁰

If you can modify the app, this is the least wasteful approach. You can build in logic that starts recording when the app itself knows an anomaly is imminent and stops when things return to normal: “three consecutive communication retries failed,” “the queue backlog exceeded the threshold,” and so on. In a supervisor-process configuration like the one covered in the Job Object article, the supervisor can decide the interval instead.

5.4 Monitor Mode — Record Every Launch

For bugs that appear only right after startup or on particular launches, -monitor is the right fit. The official comparison table also describes monitor mode as intended for “catching intermittent problems and startup problems.”² When recording the same program many times, the default sequential file names (MyApp01.run, MyApp02.run, and so on) become inefficient because existing files have to be scanned, so use -timestampFilename for timestamped names. The number of simultaneous recordings can be capped with -maxConcurrentRecordings.²

flowchart TB
    accTitle: How monitor mode behaves
    accDescr: The monitor option installs a process-launch monitoring driver, narrows the target with the command-line filter every time the specified program starts, records it, creates a separate trace file per launch, and continues until Ctrl+C or a reboot
    drv["Install the launch-monitoring driver"] --> launch["Detect a launch of the specified program"]
    launch --> filt{"Matches -cmdLineFilter?"}
    filt -->|"Yes"| rec["Record that launch (separate file per launch)"]
    filt -->|"No"| skip["Do not record"]
    rec --> next["Wait for the next launch (until Ctrl+C or reboot)"]
    skip --> next

Figure 14: Monitor mode “lies in wait for the launch.” It suits bugs that appear on every launch and bugs that can be narrowed by launch conditions.

5.5 Where to Put the Disk

For long-running recordings, put the traces on a dedicated volume and include the growth of the .run file in your monitoring. As noted in Section 3.2, when the disk fills up, recording just waits silently with no error. The official workaround is equally primitive: “check free space in Explorer” and “check that the .run file is growing regularly.”⁴ Without free-space monitoring, you end up with an incomplete trace in which the moment you wanted most was never written.

flowchart TB
    accTitle: How running out of disk produces an incomplete trace
    accDescr: When the disk runs out during recording, TTD writes the last page and waits silently with neither an error nor a warning, so the symptom that occurs afterward is not recorded, leaving an incomplete trace that opens but lacks the crucial part. Prevent it with a dedicated volume and monitoring of .run growth
    full["Disk runs out"] --> wait["Writes the last page and waits silently"]
    wait --> none["No error, no warning"]
    none --> sym["Symptom occurs afterward"]
    sym --> inc["Incomplete trace without the symptom"]
    guard["Dedicated volume + monitoring of .run growth"] -.->|"prevents"| full

Figure 15: A trace that “opens but lacks the crucial part” is born from a gap in disk monitoring.

6. Replaying — Positions, Events, and Rewinding

6.1 Opening

When you open a .run file in WinDbg, if there is no index, !index runs automatically and builds the .idx file while counting keyframes (positions in the trace generated automatically for indexing; larger traces have more of them). The larger the trace, the longer it takes.⁵ The state of the index can be checked with !index -status; if it reports anything other than “Index file loaded,” rebuild it with !index -force. If that still fails, close the debugger, delete the .idx file, and reopen the .run file. Rebuilding the index does not modify the .run file, so no data is lost.⁸

One caveat: when indexing of large traces was improved in TTD 1.11.611, the index format changed, and existing traces need to be re-indexed.¹¹ If you carry old .idx files around, this is where you stumble.

flowchart TB
    accTitle: Checking and rebuilding the index
    accDescr: After opening the trace, check the state with !index -status; if it is anything other than Index file loaded, rebuild with !index -force, and if that still fails, close the debugger, delete the .idx file, and reopen the .run file. Rebuilding does not modify the .run file
    open["Open the trace"] --> st["!index -status"]
    st -->|"Index file loaded"| ok["Proceed to analysis"]
    st -->|"Anything else"| force["Rebuild with !index -force"]
    force -->|"Fails"| del["Close, delete .idx, reopen .run"]
    del --> ok
    force -->|"Succeeds"| ok

Figure 16: Rebuilding the index does not touch the .run file. When in doubt, delete it and reopen.

6.2 Moving by Position

Passing a position to !tt moves to that point in time.⁶

!tt 0          ; start of the trace
!tt 50         ; approximately the 50% position
!tt 100        ; end of the trace
!tt 1A0:12F    ; to position 1A0:12F

A position is a pair of hexadecimal numbers, sequencing number:step count.⁶ The Position object has the properties Percent (the fraction of the trace), Sequence, and Steps, plus SeekTo(), which moves to that position, and ToSystemTime(), which returns the approximate wall-clock time (UTC).⁷ In long-running investigations, ToSystemTime() is what makes the difference, because it lets you match the timestamps in the app’s log against positions in the trace. TTD.Calls results also carry SystemTimeStart / SystemTimeEnd.¹²

flowchart TB
    accTitle: Matching positions against log timestamps
    accDescr: Starting from a timestamp in the app's log, follow the approximate wall-clock time of the Position object to identify the position in the trace, move there with SeekTo, and read the execution around it
    log["App log: time of the anomaly"] --> match["Find a position whose ToSystemTime is close"]
    match --> seek["Move there with SeekTo"]
    seek --> read["Read the surrounding calls and values"]

Figure 17: “What was it doing just before this log line?” can be looked up through the correspondence between positions and wall-clock time.

Positions are also useful for sharing. When you hand a trace to a colleague, attach the !tt x:y position and they can start looking from the same point in time. Writing position ranges in bug reports is an officially recommended practice as well.²

6.3 Rewinding

Appending - to the usual stepping commands moves backward in time.¹³

The events that stop g- are the same ones that stop a forward g.¹³ In other words, set a ba (break-on-access breakpoint) or bp and run g-, and you go straight back to “the last position where that condition held.” This is the basic move of Chapter 7.

flowchart TB
    accTitle: Choosing among the reverse commands
    accDescr: p- goes back one step across function calls, t- goes back one instruction into functions, and g- goes straight back to a breakpoint, an event, or the start of the trace. Whatever stops a forward g also stops g-
    cur["Current position"] -->|"p-"| over["Go back one step across calls"]
    cur -->|"t-"| into["Go back one instruction into functions"]
    cur -->|"g-"| run["Go straight back to the next stopping condition"]
    run -.-> stop["ba / bp / event / start of trace"]

Figure 18: To look carefully nearby, use t-; to jump to a distant cause, use ba + g-.

6.4 Entering Through Events

If you are unsure where to start reading, begin with the event list. @$curprocess.TTD.Events lists thread creation and termination, module load and unload, and exceptions as events.¹⁴

dx -g @$curprocess.TTD.Events
dx @$curprocess.TTD.Events.Where(t => t.Type == "Exception").Select(e => e.Exception)

An exception event contains the position, the type (Software / Hardware), the exception code, and the program counter at the time, and clicking the [Time Travel] link in the output moves to that position.¹⁵ The official walkthrough follows this flow: it jumps to the position of an access violation (0xc0000005), suspects stack corruption because the stack pointer and base pointer disagree, and steps back three instructions with t- to check the values.⁵ WinDbg’s Timelines window visualizes exceptions, breakpoints, memory accesses, and function calls as a timeline, and double-clicking an exception issues the same SeekTo().¹⁶

6.5 Threads and Positions

!positions shows every active thread at the current position together with each thread’s position in the trace.¹⁷ There is one pitfall here. Switching threads with ~<number>s does not move the position in the trace. The position the debugger uses to read memory does not change, so to look at another thread’s memory “at that point in time,” move there with the position link in the !positions output or with !tt x:y.¹³

flowchart TB
    accTitle: The basic path through a replay
    accDescr: Open the trace and build the index, move from the event list to the position of the exception, step backward to the cause, and if needed move to another thread's position with positions
    open["Open the trace and index it"] --> ev["Find the exception in TTD.Events"]
    ev --> seek["Move to the position with [Time Travel]"]
    seek --> back["Go back with t- / p- / g-"]
    back --> th["Check other threads' positions with !positions"]
    th -.-> caution["~s does not move the trace position"]

Figure 19: “Enter through an event, walk backward” is the basic path through a replay. Switching threads is not moving positions.

7. Finding “When” with Queries — TTD.Calls and TTD.Memory

The real strength of replay is that you can query the whole trace. TTD’s objects are exposed through the debugger data model (the dx command), and you can filter, sort, and aggregate them LINQ-style.¹⁸

7.1 TTD.Calls — Searching for Function Calls

@$cursession.TTD.Calls("module!symbol") collects the calls to the specified function from the whole trace. Wildcards are allowed, and each call carries its start and end positions (TimeStart / TimeEnd), the thread ID (with a UniqueThreadId that is never reused), the arguments (Parameters[]), the return value (ReturnValue), and the return address (ReturnAddress).¹²

The example in the official documentation is GetLastError. Aggregate the calls whose return value is nonzero by error code, and you get a list of which errors occurred how many times during the trace.¹⁸

dx -g @$cursession.TTD.Calls("kernelbase!GetLastError").Where(x => x.ReturnValue != 0).GroupBy(x => x.ReturnValue).Select(x => new { ErrorNumber = x.First().ReturnValue, ErrorCount = x.Count() }).OrderByDescending(p => p.ErrorCount),d

For “where was the last MessageBox called from?”, take the last call with OrderBy(c => c.TimeStart).Last() and move there through the [Time Travel] link of its TimeStart.¹⁸

flowchart TB
    accTitle: Building a TTD.Calls query
    accDescr: Collect calls by function name, filter by return value or arguments, aggregate by error code and the like, sort by time, and move to the position of the call you want through the Time Travel link
    calls["TTD.Calls (function name, wildcards)"] --> where["Where: filter by return value or arguments"]
    where --> group["GroupBy: aggregate by error code, etc."]
    group --> order["OrderBy: sort by time"]
    order --> jump["Move through [Time Travel] on TimeStart"]

Figure 20: The way to use queries is to start not from “where” but from “when, how many times, and with which arguments.”

A word on symbols. TTD determines the number and types of a function’s arguments, its return type, and its calling convention from the PDB symbol information. With private symbols you get the function name and the correct arguments. With public symbols only, you get the function name and default arguments (four 64-bit unsigned integers). A module with no symbols at all gets the function name UnknownOrMissingSymbols.¹²¹⁸ For the kinds of PDB and how to store them, see “What Is a PDB?”.

flowchart TB
    accTitle: Symbol availability and TTD.Calls results
    accDescr: With private symbols you get the function name and the correct arguments; with public symbols only you get the function name and the default four 64-bit integer arguments; with no symbols the function name becomes UnknownOrMissingSymbols
    sym{"Symbols for the module?"} -->|"private symbols"| full["Function name + correct arguments and return value"]
    sym -->|"public symbols"| pub["Function name + default arguments (4 x 64-bit integers)"]
    sym -->|"none"| unk["UnknownOrMissingSymbols"]

Figure 21: Whether you have kept the PDBs of your own modules directly determines how useful the queries are.

Calls involves computation, so the larger the trace, the longer it takes and the higher the CPU usage. Results are cached in memory, so the second and later queries for the same function are faster.¹² When a query returns nothing, there are four causes: how the call is written (check the module name with the x command; if it comes back in uppercase, use that), the target DLL is not yet loaded at that position (move to a position after the load and run it again), the function is inlined (cannot be tracked), or the wildcard is too broad (narrow it).¹⁸

7.2 TTD.Memory — Searching for Memory Accesses

@$cursession.TTD.Memory(start address, end address, "access type") collects accesses to the specified memory range from the whole trace. The types are r (read), w (write), rw, e (execute), rwe, and ec (execute/change).¹⁹ “Who last wrote this variable?” is answered by collecting with "w", taking .Last(), and moving to that position.⁵

dx -g @$cursession.TTD.Memory(0x00a4fca0, 0x00a4fca4, "w")
dx @$cursession.TTD.Memory(0x00a4fca0, 0x00a4fca4, "w").Last().TimeStart.SeekTo()

If you only need to search forward or backward from the current position, @$curprocess.TTD.PrevMemoryAccess("w", address, size) / NextMemoryAccess are lighter and accept multiple ranges at once. The position where a register changed can be found with @$curthread.TTD.PrevRegisterWrite("rcx").⁶

7.3 ba + g- — Making the Debugger Answer “Who Corrupted It?”

Condensing the procedure shown in the official walkthrough into a form usable for long-running investigations gives the following.⁵

1. Move to the position of the exception with TTD.Events
2. Step back with t- and hypothesize which variable holds the corrupted value
3. Get the address of that variable with dx &variable
4. Set a write breakpoint with ba w4 <address>
5. Run g- to go straight back to the position where that variable was last written
6. Check whether that point (or a few instructions earlier) is the cause. If the written value came from another variable, set ba on that variable and run g- again
7. Repeat until you reach the instruction that corrupted it

flowchart TB
    accTitle: Tracing the origin of a value with ba and g-
    accDescr: Set a write breakpoint on the address of the corrupted value and execute in reverse, stop at the instruction that last wrote it, and if that value came from another variable repeat the same steps until you reach the instruction that corrupted it
    bad["Identify the address of the corrupted value"] --> ba["Set a write breakpoint with ba w"]
    ba --> gb["Execute in reverse with g-"]
    gb --> writer["Stop at the instruction that last wrote it"]
    writer --> q{"Did the value come from another variable?"}
    q -->|"Yes"| bad
    q -->|"No"| found["The corrupting instruction = the cause"]

Figure 22: An investigation that ends at “it is corrupted” with a dump proceeds mechanically to “who corrupted it” with TTD.

TTD 1.11.553 and later also add @$curframe.TTD.VariableHistory(), which returns the history of a frame’s local variable values. It shows a table of the variable names and which values each variable held over which position ranges.¹¹ In situations like stack corruption where you want to know “since when has the value been wrong,” it is useful for narrowing down where to set ba first.

8. Applying This to Long-Running Bugs

Now we apply these tools to the three kinds of case listed at the beginning.

flowchart TB
    accTitle: Types of long-running bugs and the TTD entry point for each
    accDescr: For intermittent exceptions and data corruption, go back from the event with ba and g-; for resource growth, match acquisition and release calls with TTD.Calls; for not responding and chains of waits, follow positions and the wall-clock times of wait API calls
    t1["Intermittent exceptions, data corruption"] --> a1["TTD.Events, then ba + g-"]
    t2["Handle and memory growth"] --> a2["Match acquisition and release with TTD.Calls"]
    t3["Not responding, chains of waits"] --> a3["!positions and wall-clock times of wait APIs"]
    a2 -.-> pre["Record with -module limited to your own DLL"]

Figure 23: The entry point differs by type. What they share is narrowing the recording scope before you start reading.

Type 1: Intermittent exceptions and data corruption. Keep the moments before the symptom with the ring buffer of Section 5.1, jump to the position of the exception from the event list of Section 6.4, and trace the origin of the value with the ba + g- of Section 7.3. The difference from a dump investigation is that the investigation does not end when you find “the corrupted variable”; from there you proceed mechanically backward.

Type 2: Handle and memory growth. This is the kind of case dissected in “A Handle Leak That Crashes After a Month.” You cannot record a month’s worth, so restrict recording to your own DLL with the -module of Section 5.2, combine it with -ring, and keep “the few minutes while it is growing.” On the trace, collect TTD.Calls("kernelbase!CreateFileW") and TTD.Calls("kernelbase!CloseHandle") and match the return value of CreateFileW (the handle value) against the first argument of CloseHandle (Parameters[0]). The ReturnValue of CloseHandle is a Boolean success flag, so it is useless for matching. If a handle value is left unclosed, the ReturnAddress (the caller) of that CreateFileW call gives you the caller that is leaking.¹² That said, observing whether a leak exists and how large it is is cheaper with Application Verifier or handle counts; the correct division of labor is to bring in TTD at the stage of pinning down which path is leaking. Note that recording with Application Verifier enabled noticeably degrades replay performance because of how memory is used, so disable it while recording.⁸

Type 3: “Not responding” and chains of waits. As written in “What “Not Responding” Really Is,” a hang is a question of who is waiting for whom. A trace does not grow while idle,⁴ so the waiting time of a thread that is waiting in the kernel does not appear as instructions. What does appear is the call just before entering the wait and the instructions after it returned. Look at each thread’s position with !positions,¹⁷ and line up “which thread started waiting for what, and when” from the SystemTimeStart / SystemTimeEnd of TTD.Calls("kernelbase!WaitForSingleObject") and the like; matched against the log timestamps, that lets you reconstruct the chain of waits.¹² “Order-dependent bugs” like DllMain and the loader lock or spurious wakeups of condition variables are an area where TTD, which records the order itself, is a good fit.

9. TTD with .NET Apps

The official TTD documentation states that managed code can be debugged with TTD in WinDbg using the SOS extension (sos.dll) running in 64-bit mode.¹ Loading it is the same as in Chapter 3 of the SOS analysis article (.loadby sos coreclr or automatic loading), and !clrstack and !pe work at each position in the trace. The basic path is to move to the position of the exception with TTD.Events and then read the managed stack with !clrstack.

flowchart TB
    accTitle: Setup for reading a TTD trace of a .NET app
    accDescr: Open the TTD trace in WinDbg, load the 64-bit SOS extension, move to the position of the exception event, and read the managed state with !clrstack and !pe. Use TTD.Calls for calls across native boundaries
    run["TTD trace (.run)"] --> wd["WinDbg"]
    wd --> sos["SOS extension (64-bit)"]
    sos --> ev["Move to the exception position with TTD.Events"]
    ev --> clr["Read with !clrstack / !pe"]
    wd -.-> calls["TTD.Calls for calls across native boundaries"]

Figure 24: Managed state through SOS, call searches at native boundaries. Separate the roles and you will not get lost.

Two caveats. First, what Microsoft guarantees is “SOS in 64-bit mode.” Nothing is stated about .NET apps built for x86, so run the investigation target as x64 if you can. Second, TTD.Calls relies on PDB symbol information.¹² Do not count on searching JIT-compiled managed methods by name; the reliable use is to follow calls across native boundaries such as P/Invoke targets in native DLLs, COM, and Win32 APIs. Managed-heap problems like those in “Telling GC Lag from a Memory Leak in .NET” are pursued first with dotnet-counters / dotnet-gcdump / !gcroot on a dump, and TTD comes out at the stage where a native boundary is involved.

10. Choosing Among Dumps, Logs, ETW, and TTD

TTD is not a cure-all, and it does not replace the existing tools. Here it is lined up against the tools covered in our articles.

flowchart TB
    accTitle: Choosing an investigation method
    accDescr: First grasp the state with lightweight dumps and logs, investigate failing APIs with ProcMon and performance with ETW, and move on to TTD only when you still need who passed what and when
    start["Symptom"] --> light["Grasp the state with dumps and logs (lightweight)"]
    light --> api["Failing APIs: ProcMon"]
    light --> perf["Performance: WPR/WPA, PerfView"]
    light --> need{"Need who, when, and what was passed?"}
    need -->|"Yes"| ttd["Design the recording scope and record with TTD"]
    need -->|"No"| done["Settled with lightweight tools"]

Figure 25: Grasp the “result” with lightweight tools first, and bring out TTD only for the cases where the “path” turns out to be needed.

The order is “lightest first.” Dumps and logs cost almost nothing to collect, so keep them in place permanently; bring out TTD, after the design of Chapter 5, for the cases where reading the dump showed that the path is needed. Given TTD’s overhead and the fact that traces contain confidential information, there is no reason to reverse this order.

11. Operational Notes

• Treat traces as confidential files. A recording contains memory contents and can include personal and security-related information such as file paths, registry data, and the contents of memory and files.¹² When recording in a customer environment, understand in advance what may be included (connection strings, tokens, customer data) and decide on an encrypted transfer channel, a storage location, and a retention period.
• Share only the .run file. The .idx file is about as large as the .run file and is generated automatically when WinDbg opens it. .run files compress well. When reporting a bug in TTD itself, attach the .out file too.²
• Keep versions aligned. TTD keeps being updated together with WinDbg; 1.11.611 includes a fix for recording crashes in programs that use AVX/AVX512 and a change in the index format.¹¹ Mixing an old version on the recording side or the replay side means re-indexing or re-recording.
• Check -replayCpuSupport when replaying on a different CPU. The default favors portability, and MostConservative is provided for cases where the recording CPU and the replay CPU differ (such as replaying an Intel trace on arm64). Conversely, if you know the replay CPU is equal or better, you can choose a smaller, faster recording.²
• Windows Server can be recorded too. TTD.exe supports Windows Server 2016/2019/2022/2025.²
• Isolating a failure to record. First try whether ping.exe or cmd.exe can be recorded; if not, suspect a conflict with invasive software such as antivirus or application virtualization.⁸

flowchart TB
    accTitle: Procedure for handling traces
    accDescr: Because a recorded trace contains memory contents, understand what information it may contain, compress only the .run file and hand it over through an encrypted channel, decide on the storage location and retention period, and on the analysis side open it with the same version of WinDbg and build the index
    rec["Recording complete (.run / .idx / .out)"] --> know["Understand what information it may contain"]
    know --> share["Compress only .run, encrypt, and hand over"]
    share --> keep["Decide the storage location and retention period"]
    keep --> open["Open with the same version of WinDbg and generate .idx"]

Figure 26: Handing over a trace is handing over a confidential file. Decide the procedure before you record.

12. Summary

• A dump is “state”; TTD is the “path.” In cases that need the origin of a corrupted value or the order of calls, TTD turns the investigation into mechanical work
• Recording is 5 to 20 times slower, grows 5 to 50 MB per second, and cannot detach. For long-running apps, design the recording scope first with -ring/-maxFile, -module, -recordmode Manual, and -monitor
• Replay is “enter through an event, walk backward”: TTD.Events, then [Time Travel], then t-/g-, and ba + g- to make the debugger answer “who corrupted it”
• TTD.Calls and TTD.Memory are queries over the whole trace. Match against log timestamps with ToSystemTime() and SystemTimeStart
• For .NET, read the state with 64-bit SOS and use TTD.Calls at native boundaries
• A trace is a confidential file. Share only the .run file, and decide the channel and storage first

Cases where a dump was captured but the cause is out of reach, or where work has stalled because the bug does not reproduce, are quite often settled with TTD once the recording scope can be designed. We can also take on everything from the recording design to the analysis of the .run file, so feel free to get in touch with your dumps and logs.

Related Articles

• Reading Crash Dumps with WinDbg + SOS — A Practical Guide to Analysis After Collection
• An Introduction to Collecting Windows Crash Dumps - WER/ProcDump/WinDbg
• What Is a PDB (Program Database)? — Understanding Debug Information, Symbols, and Source Link
• Investigating Long-Run Crashes of an Industrial Camera App - The Handle Leak (Part 1)
• What “Not Responding” Really Is — How Windows Decides an App Has Hung, and How to Design Apps That Don’t
• A Practical Guide to Process Monitor (ProcMon)
• WPR/WPA in Practice — An Introduction to System-Wide Performance Investigation for “the Whole PC Is Slow”
• Pinpointing “Slow” with PerfView and dotnet-trace
• What Remains After the Parent Dies — Keeping Child Processes on a Leash with Job Objects

Related Consulting Areas

KomuraSoft LLC handles root-cause investigation of Windows app bugs that appear only after long runs or intermittently, combining crash dumps, logs, and TTD traces; building an investigation setup including the design of the recording scope; and isolating failures that involve native boundaries (COM, P/Invoke, device SDKs). Get in touch as early as the stage of “we have a dump but cannot reach the cause.”

• Bug Investigation and Root Cause Analysis
• Technical Consulting and Design Review
• Windows App Development
• Contact

References

Related Articles

Recent articles sharing the same tags. Deepen your understanding with closely related topics.

Related Topics

These topic pages place the article in a broader service and decision context.

Where This Topic Connects

This article connects naturally to the following service pages.

Frequently Asked Questions

Common questions about the topic of this article.

What is the difference between Time Travel Debugging (TTD) and a crash dump?

A crash dump is a photograph of memory at the moment of the crash: it shows the state at that instant, but not the path that led there. A TTD trace is a complete recording of the process's instruction execution that can later be replayed forward and backward, so you can rewind and directly check who last wrote to a variable or what was called just before an exception. Microsoft's official documentation also states that dumps tend to miss the state and execution path that led to the failure. The price is that the process runs 5 to 20 times slower while recording, and the trace file grows by roughly 5 to 50 MB per second while the process is active.

Can I leave TTD attached to a production app that runs for days?

Not as is. TTD slows the process considerably while recording, the trace grows by 5 to 50 MB per second in an active process, and there is no cap on the file size. Using it on a long-running app presupposes a recording design: TTD.exe's -ring (ring buffer) with -maxFile to keep only the last N MB, -module to record only while your own module is running, or -recordmode Manual to let the app choose the recording interval. Also, once attached, TTD cannot detach itself, so you must decide in advance how you will end the recording, which means terminating (restarting) the process.

Can services and processes in other sessions be recorded?

The TTD.exe documentation describes -attach as intended for investigating services and long-running apps, and -monitor as recording every time a program or service starts. On the other hand, the troubleshooting page states that unusual processes running in another session or a different security context are currently not supported for recording. Whether it actually works depends on the environment, so first record a simple process such as ping.exe or cmd.exe in the same configuration as production, then try the target process, and only then build it into your operation.

Can TTD be used on traces of .NET apps?

Yes. The official documentation states that the SOS extension (sos.dll) running in 64-bit mode can be used on a TTD trace in WinDbg to debug managed code. The basic pattern is to run SOS commands such as !clrstack and !pe at each position in the trace, moving to the position of an exception event and then reading the managed stack. The TTD.Calls query, which searches for calls by symbol name, relies on PDB symbol information, so in .NET apps the reliable use is to follow calls across native boundaries such as P/Invoke, COM, and Win32 APIs.

Is it safe to send a trace file (.run) to another company or outside the organization?

Not as is. A TTD recording contains the process's memory contents, and the official documentation states explicitly that it can include personal or confidential information such as file paths, registry data, and the contents of memory and files. If you send one, first understand what was recorded (connection strings, tokens, customer data, and so on), then decide on an encrypted transfer channel and storage location. Sharing only the .run file is sufficient; the index file (.idx) is generated automatically when WinDbg opens it.

TTD.Calls returns nothing when I search for a function. Why?

There are four main causes. First, symbols: functions in a module without a PDB are named UnknownOrMissingSymbols, and the module name may be uppercase, so check the actual symbol name with the x command. Second, the target DLL may not be loaded yet at that position; move to a position after the DLL was loaded and query again. Third, if the function is inlined, the query engine cannot track it. Fourth, the wildcard may be too broad and match too many functions; narrow the pattern.

Author Profile

Profile page for the article author.

Go Komura

Representative of KomuraSoft LLC

Focused on Windows software development, technical consulting, and investigations into failures that are difficult to reproduce.