natsdotnet/dottrace.md

# dotTrace Command-Line Profiler

## Installation

Installed as a .NET global tool:

```bash
dotnet tool install --global JetBrains.dotTrace.GlobalTools
```

Update to latest:

```bash
dotnet tool update --global JetBrains.dotTrace.GlobalTools
```

Current version: **2025.3.3**

## Quick Start

### Profile the NATS server (sampling, 30 seconds)

```bash
dottrace start --framework=NetCore --profiling-type=Sampling \
  --timeout=30s --save-to=./snapshots/nats-sampling.dtp \
  -- dotnet run --project src/NATS.Server.Host -- -p 14222
```

### Profile the NATS server (timeline, with async/TPL info)

```bash
dottrace start --framework=NetCore --profiling-type=Timeline \
  --timeout=30s --save-to=./snapshots/nats-timeline.dtt \
  -- dotnet run --project src/NATS.Server.Host -- -p 14222
```

### Attach to a running server by PID

```bash
dottrace attach <PID> --profiling-type=Sampling \
  --timeout=30s --save-to=./snapshots/nats-attach.dtp
```

### Attach by process name

```bash
dottrace attach NATS.Server.Host --profiling-type=Sampling \
  --timeout=30s --save-to=./snapshots/nats-attach.dtp
```

## Profiling Types

| Type | Flag | Snapshot Extension | Use Case |
|------|------|--------------------|----------|
| Sampling | `--profiling-type=Sampling` | `.dtp` | Low overhead, CPU hotspots (default) |
| Timeline | `--profiling-type=Timeline` | `.dtt` | Thread activity, async/await, TPL tasks |
| Tracing | `--profiling-type=Tracing` | `.dtp` | Exact call counts, higher overhead |
| Line-by-Line | `--profiling-type=LineByLine` | `.dtp` | Per-line timing (not available for attach) |

### Sampling options

```bash
# Use thread time instead of CPU instructions
--time-measurement=ThreadTime

# Default (CPU instruction count)
--time-measurement=CpuInstruction
```

### Timeline options

```bash
# Disable TPL data collection for better performance
--disable-tpl
```

## Common Options

| Option | Description |
|--------|-------------|
| `--framework=NetCore` | Required for .NET Core / .NET 5+ apps |
| `--save-to=<path>` | Snapshot output path (file or directory) |
| `--overwrite` | Overwrite existing snapshot files |
| `--timeout=<duration>` | Auto-stop after duration (e.g., `30s`, `5m`, `1h`) |
| `--propagate-exit-code` | Return the profiled app's exit code instead of dotTrace's |
| `--profile-child` | Also profile child processes |
| `--profile-child=<mask>` | Profile matching child processes (e.g., `dotnet`) |
| `--work-dir=<path>` | Set working directory for the profiled app |
| `--collect-data-from-start=off` | Don't collect until explicitly started via service messages |

## Interactive Profiling with Service Messages

For fine-grained control over when data is collected, use `--service-input=stdin`:

```bash
dottrace start --framework=NetCore --service-input=stdin \
  --save-to=./snapshots/nats-interactive.dtp \
  -- dotnet run --project src/NATS.Server.Host -- -p 14222
```

Then type these commands into stdin (each must start on a new line and end with a carriage return):

| Command | Effect |
|---------|--------|
| `##dotTrace["start"]` | Start collecting performance data |
| `##dotTrace["get-snapshot"]` | Save snapshot and stop collecting |
| `##dotTrace["drop"]` | Discard collected data and stop |
| `##dotTrace["disconnect"]` | Detach/stop profiler |

Stdout will emit status messages like:

```
##dotTrace["ready"]
##dotTrace["connected", {pid: 1234, path:"dotnet"}]
##dotTrace["started", {pid: 1234, path:"dotnet"}]
##dotTrace["snapshot-saved", {pid: 1234, filename:"./snapshots/nats-interactive.dtp"}]
```

## Example Workflows

### Profile a benchmark run

```bash
dottrace start --framework=NetCore --profiling-type=Sampling \
  --save-to=./snapshots/bench.dtp \
  -- dotnet run --project tests/NATS.Server.Benchmarks -c Release
```

### Profile tests

```bash
dottrace start --framework=NetCore --profiling-type=Sampling \
  --timeout=2m --save-to=./snapshots/tests.dtp \
  -- dotnet test tests/NATS.Server.Core.Tests --filter "FullyQualifiedName~PubSub"
```

### Profile with child processes (e.g., server spawns workers)

```bash
dottrace start --framework=NetCore --profile-child \
  --timeout=30s --save-to=./snapshots/nats-children.dtp \
  -- dotnet run --project src/NATS.Server.Host
```

## Exporting Reports

dotTrace's XML report tool (Reporter.exe) is Windows-only. On macOS, use `dotnet-trace` for profiling with exportable formats:

```bash
# Install dotnet-trace
dotnet tool install --global dotnet-trace

# Collect a trace from a running process (nettrace format)
dotnet-trace collect --process-id <PID> --duration 00:00:30

# Collect directly in speedscope format
dotnet-trace collect --process-id <PID> --format speedscope --duration 00:00:30

# Convert an existing .nettrace file to speedscope
dotnet-trace convert --format speedscope trace.nettrace
```

Speedscope files can be visualized at [speedscope.app](https://www.speedscope.app) — a web-based flame graph viewer that works on any platform.

#### dotnet-trace output formats

| Format | Extension | Viewer |
|--------|-----------|--------|
| `nettrace` (default) | `.nettrace` | PerfView, Visual Studio, Rider |
| `speedscope` | `.speedscope.json` | [speedscope.app](https://www.speedscope.app) |
| `chromium` | `.chromium.json` | Chrome DevTools (`chrome://tracing`) |

#### Example: profile NATS server and export flame graph

```bash
# Start the server
dotnet run --project src/NATS.Server.Host -- -p 14222 &
SERVER_PID=$!

# Collect a 30-second trace in speedscope format
dotnet-trace collect --process-id $SERVER_PID --format speedscope \
  --duration 00:00:30 --output ./snapshots/nats-trace

# Open the flame graph
open ./snapshots/nats-trace.speedscope.json  # opens in default browser at speedscope.app
```

## Viewing Snapshots

Open `.dtp` / `.dtt` snapshot files in:

- **dotTrace GUI** (`/Users/dohertj2/Applications/dotTrace.app`)
- **JetBrains Rider** (built-in profiler viewer)

```bash
open /Users/dohertj2/Applications/dotTrace.app --args ./snapshots/nats-sampling.dtp
```

## Parsing Raw `.dtp` Snapshots To JSON

The repository includes a Python-first parser for raw dotTrace sampling and tracing snapshots:

- Python entrypoint: [tools/dtp_parse.py](/Users/dohertj2/Desktop/natsdotnet/tools/dtp_parse.py)
- .NET helper: [tools/DtpSnapshotExtractor/Program.cs](/Users/dohertj2/Desktop/natsdotnet/tools/DtpSnapshotExtractor/Program.cs)

The parser starts from the raw `.dtp` snapshot family and emits machine-readable JSON for call-tree and hotspot analysis. It uses the locally installed dotTrace assemblies to decode the snapshot format.

### Prerequisites

- `python3`
- `.NET 10 SDK`
- dotTrace installed at `/Users/dohertj2/Applications/dotTrace.app`

If dotTrace is installed elsewhere, set `DOTTRACE_APP_DIR` to the `Contents/DotFiles` directory:

```bash
export DOTTRACE_APP_DIR="/path/to/dotTrace.app/Contents/DotFiles"
```

### Print JSON to stdout

```bash
python3 tools/dtp_parse.py snapshots/js-ordered-consume.dtp --stdout
```

### Write JSON to a file

```bash
python3 tools/dtp_parse.py snapshots/js-ordered-consume.dtp \
  --out /tmp/js-ordered-consume-calltree.json
```

### Output shape

The generated JSON contains:

- `snapshot` — source path, payload type, thread count, node count
- `threadRoots` — top-level thread roots with inclusive time
- `callTree` — nested call tree rooted at a synthetic `<root>`
- `hotspots` — flat `inclusive` and `exclusive` method lists

Hotspot entries are method-first. Synthetic frames such as thread roots are excluded from the hotspot lists so the output is easier to feed into an LLM for slowdown analysis.

### Typical analysis workflow

1. Capture a snapshot with `dottrace`.
2. Convert the raw `.dtp` snapshot to JSON:

```bash
python3 tools/dtp_parse.py snapshots/nats-sampling.dtp \
  --out /tmp/nats-sampling-calltree.json
```

3. Inspect the top hotspots:

```bash
python3 - <<'PY'
import json
with open('/tmp/nats-sampling-calltree.json') as f:
    data = json.load(f)
print('Top inclusive:', data['hotspots']['inclusive'][0]['name'])
print('Top exclusive:', data['hotspots']['exclusive'][0]['name'])
PY
```

4. Feed the JSON into downstream tooling or an LLM to walk the call tree and identify expensive paths.

### Verification

Run the parser test with:

```bash
python3 -m unittest tools.tests.test_dtp_parser -v
```

## Exit Codes

| Code | Meaning |
|------|---------|
| 0 | Success |
| 65 | Profiling failure |

## Notes

- Snapshots consist of multiple files: `*.dtp`, `*.dtp.0000`, `*.dtp.0001`, etc. Keep them together.
- Attach on macOS requires .NET 5 or later.
- Use `--` before the executable path if arguments start with `-`.
- The `snapshots/` directory is not tracked in git. Create it before profiling:
  ```bash
  mkdir -p snapshots
  ```
- The parser currently targets raw `.dtp` snapshots. Timeline `.dtt` snapshots are still intended for the GUI viewer.