feat(dsight): add CLI-only inference trace exploration

[nv-yna]

User impact

Generate a self-contained inference dashboard from a preserved AgentX/AIPerf or AgentPerf run. Select a time range, inspect client sessions and requests, and compare available worker metrics, hardware samples, iteration logs, and Nsight activity on the same timeline.

Generation is an explicit command run manually on a cluster login node. It is independent of job submission, benchmark execution, cleanup, and upload.

Run on a login node

Use a Bash shell with uv on PATH, Python 3.10+, and a writable srt-slurm checkout that includes this change. The preserved run directory must be readable and the report's parent directory writable on that node. No Slurm allocation, GPU, container, running deployment, or browser is needed for generation.

Replace the quoted placeholders with your paths. Relative paths resolve from the current working directory. uv run --no-dev prepares the checkout's Python environment without development dependencies; the first invocation needs package access or a populated cache.

cd "<path_to_srt_slurm_checkout>"
uv run --no-dev srtctl dsight build "<path_to_run_directory>" \
  --output "<path_to_report_directory>"

OTel is optional and is imported automatically when available. To skip reading OTel files entirely:

uv run --no-dev srtctl dsight build "<path_to_run_directory>" \
  --output "<path_to_report_directory>" --no-otel

Optionally include existing Nsight SQLite exports and the IANA timezone of timezone-free TRT-LLM iteration logs. These options can also be combined with --no-otel:

uv run --no-dev srtctl dsight build "<path_to_run_directory>" \
  --output "<path_to_report_directory>" \
  --nsys-sqlite "<path_to_nsys_sqlite_exports>" \
  --iteration-timezone "<iteration_log_timezone>"

Query the report on the login node without opening the UI:

uv run --no-dev srtctl dsight query "<path_to_report_directory>" summary
uv run --no-dev srtctl dsight query "<path_to_report_directory>" requests \
  --from 10 --to 20 --limit 10

The output directory contains index.html, trace-data.json.gz, and manifest.json. Copy or publish <path_to_report_directory>/index.html to view it in a browser. Pass either a run directory containing logs/ or the log directory itself; use --client "<path_to_client_export>" when multiple client exports exist.

Large captures can require substantial CPU, memory, and filesystem reads. Follow the site's login-node resource limits and use a CPU job when needed.

Optional data and UI behavior

• Client data is sufficient to build a report. OTel, worker logs, metrics, and Nsight exports add the corresponding recorded detail.
• Requests without usable correlated OTel activity have no lifecycle expansion control, stage rows, or OTel source-measurement table. This applies to missing/empty traces, unjoined requests, and --no-otel, including untraced requests within a partially traced run.
• Client request bars, measured TTFT, time selection, and independent worker/metric/Nsight data remain available. Empty request paths and identity bridges are omitted.
• Traced requests retain the cumulative lifecycle rows. Browser actions and restored view links respect per-request availability.
• CLI/Python, read-only MCP query_trace, and window.traceExplorer expose the same dataset. Unavailable lifecycle queries return available: false with empty stage/activity lists.

Implementation and browser assets live in src/srtctl/dsight/. Nsight capture/export is outside this command; it reads already exported SQLite files. Shared NVTX/iteration activity is not attributed as a request's exclusive compute time.

Validation

• Executed all three documented build examples directly on a real cluster login node from this revision, using preserved real data and a fresh Python environment: default OTel import, --no-otel, and existing Nsight SQLite exports with an explicit iteration-log timezone. Each produced 8,590 requests; the Nsight example imported 33 profiles.

• All nine login-node CLI queries passed. HTML/data SHA-256 hashes matched their manifests, and --no-otel produced zero joined OTel spans and zero request lifecycle rows. No Slurm allocation or serving container was used.

• make check: 2,942 passed, 2 skipped, 6 integration tests deselected.

• 28 focused DSight tests, including missing, empty, unjoined, unsupported, and disabled OTel. Explicit opt-out succeeds with a malformed OTel file because it is not read.

• Six browser scenarios cover mixed traced/untraced, missing, empty, unjoined, disabled, and client-only inputs; lifecycle controls, native range selection, saved views, metrics, iterations, and Nsight inspection pass. No JavaScript errors or external requests.

• DSight type checking, JavaScript syntax checking, and pre-commit on every changed file pass. The full pre-commit run reports 31 existing findings outside this change; unrelated automatic formatting was discarded.

See DSight documentation for inputs, timing definitions, query APIs, and capture limits.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 84.29036% with 145 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@85086d3). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
src/srtctl/dsight/nsys.py	53.84%	48 Missing ⚠️
src/srtctl/dsight/metrics.py	80.68%	28 Missing ⚠️
src/srtctl/dsight/query.py	77.57%	24 Missing ⚠️
src/srtctl/dsight/importer.py	91.95%	23 Missing ⚠️
src/srtctl/dsight/build.py	86.20%	8 Missing ⚠️
src/srtctl/dsight/cli.py	86.27%	7 Missing ⚠️
src/srtctl/dsight/__main__.py	0.00%	3 Missing ⚠️
src/srtctl/dsight/clients.py	94.82%	3 Missing ⚠️
src/srtctl/dsight/model.py	98.79%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #479   +/-   ##
=======================================
  Coverage        ?   81.41%           
=======================================
  Files           ?      137           
  Lines           ?    20318           
  Branches        ?        0           
=======================================
  Hits            ?    16541           
  Misses          ?     3777           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.