natsdotnet/dottrace.md

# dotTrace Command-Line Profiler

## Installation

Installed as a .NET global tool:

```bash
dotnet tool install --global JetBrains.dotTrace.GlobalTools
```

Update to latest:

```bash
dotnet tool update --global JetBrains.dotTrace.GlobalTools
```

Current version: **2025.3.3**

## Quick Start

### Profile the NATS server (sampling, 30 seconds)

```bash
dottrace start --framework=NetCore --profiling-type=Sampling \
  --timeout=30s --save-to=./snapshots/nats-sampling.dtp \
  -- dotnet run --project src/NATS.Server.Host -- -p 14222
```

### Profile the NATS server (timeline, with async/TPL info)

```bash
dottrace start --framework=NetCore --profiling-type=Timeline \
  --timeout=30s --save-to=./snapshots/nats-timeline.dtt \
  -- dotnet run --project src/NATS.Server.Host -- -p 14222
```

### Attach to a running server by PID

```bash
dottrace attach <PID> --profiling-type=Sampling \
  --timeout=30s --save-to=./snapshots/nats-attach.dtp
```

### Attach by process name

```bash
dottrace attach NATS.Server.Host --profiling-type=Sampling \
  --timeout=30s --save-to=./snapshots/nats-attach.dtp
```

## Profiling Types

| Type | Flag | Snapshot Extension | Use Case |
|------|------|--------------------|----------|
| Sampling | `--profiling-type=Sampling` | `.dtp` | Low overhead, CPU hotspots (default) |
| Timeline | `--profiling-type=Timeline` | `.dtt` | Thread activity, async/await, TPL tasks |
| Tracing | `--profiling-type=Tracing` | `.dtp` | Exact call counts, higher overhead |
| Line-by-Line | `--profiling-type=LineByLine` | `.dtp` | Per-line timing (not available for attach) |

### Sampling options

```bash
# Use thread time instead of CPU instructions
--time-measurement=ThreadTime

# Default (CPU instruction count)
--time-measurement=CpuInstruction
```

### Timeline options

```bash
# Disable TPL data collection for better performance
--disable-tpl
```

## Common Options

| Option | Description |
|--------|-------------|
| `--framework=NetCore` | Required for .NET Core / .NET 5+ apps |
| `--save-to=<path>` | Snapshot output path (file or directory) |
| `--overwrite` | Overwrite existing snapshot files |
| `--timeout=<duration>` | Auto-stop after duration (e.g., `30s`, `5m`, `1h`) |
| `--propagate-exit-code` | Return the profiled app's exit code instead of dotTrace's |
| `--profile-child` | Also profile child processes |
| `--profile-child=<mask>` | Profile matching child processes (e.g., `dotnet`) |
| `--work-dir=<path>` | Set working directory for the profiled app |
| `--collect-data-from-start=off` | Don't collect until explicitly started via service messages |

## Interactive Profiling with Service Messages

For fine-grained control over when data is collected, use `--service-input=stdin`:

```bash
dottrace start --framework=NetCore --service-input=stdin \
  --save-to=./snapshots/nats-interactive.dtp \
  -- dotnet run --project src/NATS.Server.Host -- -p 14222
```

Then type these commands into stdin (each must start on a new line and end with a carriage return):

| Command | Effect |
|---------|--------|
| `##dotTrace["start"]` | Start collecting performance data |
| `##dotTrace["get-snapshot"]` | Save snapshot and stop collecting |
| `##dotTrace["drop"]` | Discard collected data and stop |
| `##dotTrace["disconnect"]` | Detach/stop profiler |

Stdout will emit status messages like:

```
##dotTrace["ready"]
##dotTrace["connected", {pid: 1234, path:"dotnet"}]
##dotTrace["started", {pid: 1234, path:"dotnet"}]
##dotTrace["snapshot-saved", {pid: 1234, filename:"./snapshots/nats-interactive.dtp"}]
```

## Example Workflows

### Profile a benchmark run

```bash
dottrace start --framework=NetCore --profiling-type=Sampling \
  --save-to=./snapshots/bench.dtp \
  -- dotnet run --project tests/NATS.Server.Benchmarks -c Release
```

### Profile tests

```bash
dottrace start --framework=NetCore --profiling-type=Sampling \
  --timeout=2m --save-to=./snapshots/tests.dtp \
  -- dotnet test tests/NATS.Server.Core.Tests --filter "FullyQualifiedName~PubSub"
```

### Profile with child processes (e.g., server spawns workers)

```bash
dottrace start --framework=NetCore --profile-child \
  --timeout=30s --save-to=./snapshots/nats-children.dtp \
  -- dotnet run --project src/NATS.Server.Host
```

## Exporting Reports

dotTrace's XML report tool (Reporter.exe) is Windows-only. On macOS, use `dotnet-trace` for profiling with exportable formats:

```bash
# Install dotnet-trace
dotnet tool install --global dotnet-trace

# Collect a trace from a running process (nettrace format)
dotnet-trace collect --process-id <PID> --duration 00:00:30

# Collect directly in speedscope format
dotnet-trace collect --process-id <PID> --format speedscope --duration 00:00:30

# Convert an existing .nettrace file to speedscope
dotnet-trace convert --format speedscope trace.nettrace
```

Speedscope files can be visualized at [speedscope.app](https://www.speedscope.app) — a web-based flame graph viewer that works on any platform.

#### dotnet-trace output formats

| Format | Extension | Viewer |
|--------|-----------|--------|
| `nettrace` (default) | `.nettrace` | PerfView, Visual Studio, Rider |
| `speedscope` | `.speedscope.json` | [speedscope.app](https://www.speedscope.app) |
| `chromium` | `.chromium.json` | Chrome DevTools (`chrome://tracing`) |

#### Example: profile NATS server and export flame graph

```bash
# Start the server
dotnet run --project src/NATS.Server.Host -- -p 14222 &
SERVER_PID=$!

# Collect a 30-second trace in speedscope format
dotnet-trace collect --process-id $SERVER_PID --format speedscope \
  --duration 00:00:30 --output ./snapshots/nats-trace

# Open the flame graph
open ./snapshots/nats-trace.speedscope.json  # opens in default browser at speedscope.app
```

## Viewing Snapshots

Open `.dtp` / `.dtt` snapshot files in:

- **dotTrace GUI** (`/Users/dohertj2/Applications/dotTrace.app`)
- **JetBrains Rider** (built-in profiler viewer)

```bash
open /Users/dohertj2/Applications/dotTrace.app --args ./snapshots/nats-sampling.dtp
```

## Parsing Raw `.dtp` Snapshots To JSON

The repository includes a Python-first parser for raw dotTrace sampling and tracing snapshots:

- Python entrypoint: [tools/dtp_parse.py](/Users/dohertj2/Desktop/natsdotnet/tools/dtp_parse.py)
- .NET helper: [tools/DtpSnapshotExtractor/Program.cs](/Users/dohertj2/Desktop/natsdotnet/tools/DtpSnapshotExtractor/Program.cs)

The parser starts from the raw `.dtp` snapshot family and emits machine-readable JSON for call-tree and hotspot analysis. It uses the locally installed dotTrace assemblies to decode the snapshot format.

### Prerequisites

- `python3`
- `.NET 10 SDK`
- dotTrace installed at `/Users/dohertj2/Applications/dotTrace.app`

If dotTrace is installed elsewhere, set `DOTTRACE_APP_DIR` to the `Contents/DotFiles` directory:

```bash
export DOTTRACE_APP_DIR="/path/to/dotTrace.app/Contents/DotFiles"
```

### Print JSON to stdout

```bash
python3 tools/dtp_parse.py snapshots/js-ordered-consume.dtp --stdout
```

Useful flags:

- `--top N` limits hotspot and flat-path output. Default: `200`
- `--filter TEXT` keeps only call-tree paths and hotspots whose method names match `TEXT`
- `--flat` or `--paths` adds a `hotPaths` section with the heaviest flat call chains
- `--include-idle` keeps idle and wait methods in hotspot/path rankings. Idle exclusion is on by default.

### Write JSON to a file

```bash
python3 tools/dtp_parse.py snapshots/js-ordered-consume.dtp \
  --out /tmp/js-ordered-consume-calltree.json
```

```bash
python3 tools/dtp_parse.py snapshots/js-ordered-consume.dtp \
  --filter Microsoft.DotNet.Cli.Program \
  --flat \
  --top 25 \
  --out /tmp/js-ordered-consume-calltree.json
```

### Output shape

The generated JSON contains:

- `snapshot` — source path, payload type, time unit, thread count, node count, and reader diagnostics
- `summary` — wall time, active time, total samples, and top exclusive method summary
- `threadRoots` — top-level thread roots with inclusive time
- `callTree` — nested call tree rooted at a synthetic `<root>`
- `hotspots` — flat `inclusive` and `exclusive` method lists
- `hotPaths` — optional flat call-path list when `--flat` is used

Hotspot entries are method-first. Synthetic frames such as thread roots are excluded from hotspot rankings, and idle wait frames are excluded by default so the output is easier to feed into an LLM for slowdown analysis.

### Typical analysis workflow

1. Capture a snapshot with `dottrace`.
2. Convert the raw `.dtp` snapshot to JSON:

```bash
python3 tools/dtp_parse.py snapshots/nats-sampling.dtp \
  --out /tmp/nats-sampling-calltree.json
```

3. Inspect the top hotspots:

```bash
python3 - <<'PY'
import json
with open('/tmp/nats-sampling-calltree.json') as f:
    data = json.load(f)
print('Top inclusive:', data['hotspots']['inclusive'][0]['name'])
print('Top exclusive:', data['hotspots']['exclusive'][0]['name'])
PY
```

4. Feed the JSON into downstream tooling or an LLM to walk the call tree and identify expensive paths.

### Verification

Run the parser test with:

```bash
python3 -m unittest tools.tests.test_dtp_parser -v
```

## Exit Codes

| Code | Meaning |
|------|---------|
| 0 | Success |
| 65 | Profiling failure |

## Notes

- Snapshots consist of multiple files: `*.dtp`, `*.dtp.0000`, `*.dtp.0001`, etc. Keep them together.
- Attach on macOS requires .NET 5 or later.
- Use `--` before the executable path if arguments start with `-`.
- The `snapshots/` directory is not tracked in git. Create it before profiling:
  ```bash
  mkdir -p snapshots
  ```
- The parser currently targets raw `.dtp` snapshots. Timeline `.dtt` snapshots are still intended for the GUI viewer.