Skip to content

Fix GPU task metric unit inference - #2139

Merged
amahussein merged 2 commits into
NVIDIA:devfrom
amahussein:cudf-tools-2136
Sep 4, 2026
Merged

Fix GPU task metric unit inference#2139
amahussein merged 2 commits into
NVIDIA:devfrom
amahussein:cudf-tools-2136

Conversation

@amahussein

Copy link
Copy Markdown
Collaborator

Fixes #2136

Summary

AppSparkMetricsAnalyzer decided a GPU metric's unit, its scale and its aggregation by testing substrings of the metric's own name. That information belongs to the accumulator emitting the metric, not to the string naming it, so the guesses were wrong in five ways. On a profiled application whose stage carries 17 GPU accumulables, gpu_stage_level_aggregated_task_metrics.csv emitted 9 rows.

The worst of it was silent. Every timing metric was divided by a million a second time, because parseAccumFieldToLong had already converted the plugin's "00:00:01.773" form to milliseconds; the values floored to zero and the rows were then deleted by the zero-signal filter. "Wait" matched inside "Waiting", so three counters were scaled away the same way. gpuMaxTaskFootprint was labelled count because its name lacks the word Bytes. avg was hardcoded empty for max-aggregated metrics, discarding the mean across tasks of each task's own peak. And gpuOnGpuTasksWaitingGPUAvgCount, a Double-valued accumulator no parse branch could read, did not merely go missing: the stage path substituted a zero and published it, so the file reported an empty GPU wait queue on stages whose true peak was 7.

This replaces the substring rules and the hardcoded max-aggregate set with a declarative catalog, configs/metrics/metricCatalog.yaml, carrying one entry per metric with its family, unit, value form, storage scale and aggregation. Adding a metric is a YAML edit. convertValue is deleted rather than reworked, since the parser already normalises every serialised form to a canonical unit and nothing downstream should convert again. A metric absent from the catalog is still discovered by name prefix and falls back to the old heuristic for its unit label only, which is now harmless because no value is scaled by it.

The decimal metric is stored as fixed-point thousandths and divided out at render, so avg survives the integer store. Rounding it instead leaves the mean reading zero, and widening StatisticsMetrics to Double changes 44 of 83 median cells in an existing expectation file.

What changes in the output

The stage file goes from 9 rows to 17. gpuTime reappears at 417,903 ms summed with a 29,623 ms worst task, gpuMaxTaskFootprint becomes bytes, and gpuOnGpuTasksWaitingGPUAvgCount reads max=2.5, avg=0.72 where stage_level_all_metrics.csv previously showed 0,0,0,0. The SQL and app rollups inherit the same corrections. Row counts are unchanged, so no stage entries are fabricated, and a CPU event log still emits no gpu_* file.

Because one metric now renders fractional cells, the report contract retypes ten columns from Long to Double. This is a Python-side declaration only; values remain Long throughout the Scala. sum stays Long, since the only decimal metric is max-aggregated and its sum is always empty.

Qualification is unaffected. I built the parent commit and diffed both tools' output on a CPU log and a Photon log: byte-identical apart from one new warning naming an accumulable no parser can read, which was previously dropped in silence.

Testing

883 tests, 0 failures. Scalastyle and the Scala 2.13 cross-build are clean.

parseAccumFieldToLong had no unit test at all. There are now cases pinning each parse branch's output unit, the fixed-point scaling, the overflow boundary on both signs, and the inputs Double.parseDouble wrongly accepts. AnalysisSuite no longer re-implements the unit rule as its own expectation nor keeps a private copy of the max-aggregate set, and it asserts that every GPU accumulable carrying a non-zero value produces a row, which is the assertion whose absence let eight metrics vanish unnoticed.

Fixes NVIDIA#2136

- add metric registry to configure how to handle metrics.

Reads unit, scale and aggregation from the metric catalog instead of inferring them from substrings of a metric's name.

- Remove the second unit conversion. `parseAccumFieldToLong` already yields milliseconds, and `convertValue` divided by 1e6 again, flooring every timing metric to zero so its row was dropped.
- Stop matching `Wait` inside `Waiting`, which labelled three counters as durations.
- Label `gpuMaxTaskFootprint` as bytes; the old rule required `Bytes` in the name.
- Populate `avg` for max-aggregated metrics. `sum` stays empty, since per-task peaks never coexist.
- Read `gpuOnGpuTasksWaitingGPUAvgCount`, a Double no parse branch could read. It published a fabricated zero; now stored as fixed-point thousandths and divided out at render.
- Gate GPU reporting on the catalog's `family`, so a non-GPU metric declared later cannot leak into the `gpu_*` files. Report an unreadable accumulable once per process.

Testing

- Full core suite: 846 tests, 0 failures. Scalastyle and the Scala 2.13 cross-build clean.
- New `parseAccumFieldToLong` coverage: per-branch units, fixed-point scaling, and rejection of `NaN`/`Infinity`/`1e9`/`3d`/`5f` including the overflow boundary. `AnalysisSuite` now reads the catalog rather than re-implementing the rule, and asserts every non-zero GPU accumulable produces a row.
- On a mirrored event log the stage file goes from 9 to 17 rows, and `gpuOnGpuTasksWaitingGPUAvgCount` reads `max=2.5, avg=0.714` where it previously showed `0,0,0,0`. Row counts unchanged; a CPU log still emits no `gpu_*` files.

Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
@amahussein amahussein self-assigned this Sep 3, 2026
@amahussein amahussein added bug Something isn't working core_tools Scope the core module (scala) labels Sep 3, 2026
@amahussein
amahussein requested a review from parthosa September 3, 2026 15:24
Comment thread core/src/main/resources/configs/reports/coreRawMetricsReport.yaml Outdated
The rollup weighted stage means by the stage task count, which includes
tasks that never reported the metric. Pool the totals and counts instead.

Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>

@parthosa parthosa left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @amahussein. LGTME.

@amahussein
amahussein merged commit e7212f8 into NVIDIA:dev Sep 4, 2026
17 checks passed
@amahussein
amahussein deleted the cudf-tools-2136 branch September 4, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working core_tools Scope the core module (scala)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] GPU task metric units are inferred from name substrings, dropping 8 of 17 metrics and publishing a false zero

3 participants