Fix GPU task metric unit inference - #2139
Merged
Merged
Conversation
Fixes NVIDIA#2136 - add metric registry to configure how to handle metrics. Reads unit, scale and aggregation from the metric catalog instead of inferring them from substrings of a metric's name. - Remove the second unit conversion. `parseAccumFieldToLong` already yields milliseconds, and `convertValue` divided by 1e6 again, flooring every timing metric to zero so its row was dropped. - Stop matching `Wait` inside `Waiting`, which labelled three counters as durations. - Label `gpuMaxTaskFootprint` as bytes; the old rule required `Bytes` in the name. - Populate `avg` for max-aggregated metrics. `sum` stays empty, since per-task peaks never coexist. - Read `gpuOnGpuTasksWaitingGPUAvgCount`, a Double no parse branch could read. It published a fabricated zero; now stored as fixed-point thousandths and divided out at render. - Gate GPU reporting on the catalog's `family`, so a non-GPU metric declared later cannot leak into the `gpu_*` files. Report an unreadable accumulable once per process. Testing - Full core suite: 846 tests, 0 failures. Scalastyle and the Scala 2.13 cross-build clean. - New `parseAccumFieldToLong` coverage: per-branch units, fixed-point scaling, and rejection of `NaN`/`Infinity`/`1e9`/`3d`/`5f` including the overflow boundary. `AnalysisSuite` now reads the catalog rather than re-implementing the rule, and asserts every non-zero GPU accumulable produces a row. - On a mirrored event log the stage file goes from 9 to 17 rows, and `gpuOnGpuTasksWaitingGPUAvgCount` reads `max=2.5, avg=0.714` where it previously showed `0,0,0,0`. Row counts unchanged; a CPU log still emits no `gpu_*` files. Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
parthosa
reviewed
Sep 4, 2026
The rollup weighted stage means by the stage task count, which includes tasks that never reported the metric. Pool the totals and counts instead. Signed-off-by: Ahmed Hussein (amahussein) <a@ahussein.me>
parthosa
approved these changes
Sep 4, 2026
parthosa
left a comment
Collaborator
There was a problem hiding this comment.
Thanks @amahussein. LGTME.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2136
Summary
AppSparkMetricsAnalyzerdecided a GPU metric's unit, its scale and its aggregation by testing substrings of the metric's own name. That information belongs to the accumulator emitting the metric, not to the string naming it, so the guesses were wrong in five ways. On a profiled application whose stage carries 17 GPU accumulables,gpu_stage_level_aggregated_task_metrics.csvemitted 9 rows.The worst of it was silent. Every timing metric was divided by a million a second time, because
parseAccumFieldToLonghad already converted the plugin's"00:00:01.773"form to milliseconds; the values floored to zero and the rows were then deleted by the zero-signal filter."Wait"matched inside"Waiting", so three counters were scaled away the same way.gpuMaxTaskFootprintwas labelledcountbecause its name lacks the wordBytes.avgwas hardcoded empty for max-aggregated metrics, discarding the mean across tasks of each task's own peak. AndgpuOnGpuTasksWaitingGPUAvgCount, aDouble-valued accumulator no parse branch could read, did not merely go missing: the stage path substituted a zero and published it, so the file reported an empty GPU wait queue on stages whose true peak was 7.This replaces the substring rules and the hardcoded max-aggregate set with a declarative catalog,
configs/metrics/metricCatalog.yaml, carrying one entry per metric with its family, unit, value form, storage scale and aggregation. Adding a metric is a YAML edit.convertValueis deleted rather than reworked, since the parser already normalises every serialised form to a canonical unit and nothing downstream should convert again. A metric absent from the catalog is still discovered by name prefix and falls back to the old heuristic for its unit label only, which is now harmless because no value is scaled by it.The decimal metric is stored as fixed-point thousandths and divided out at render, so
avgsurvives the integer store. Rounding it instead leaves the mean reading zero, and wideningStatisticsMetricstoDoublechanges 44 of 83 median cells in an existing expectation file.What changes in the output
The stage file goes from 9 rows to 17.
gpuTimereappears at 417,903 ms summed with a 29,623 ms worst task,gpuMaxTaskFootprintbecomesbytes, andgpuOnGpuTasksWaitingGPUAvgCountreadsmax=2.5, avg=0.72wherestage_level_all_metrics.csvpreviously showed0,0,0,0. The SQL and app rollups inherit the same corrections. Row counts are unchanged, so no stage entries are fabricated, and a CPU event log still emits nogpu_*file.Because one metric now renders fractional cells, the report contract retypes ten columns from
LongtoDouble. This is a Python-side declaration only; values remainLongthroughout the Scala.sumstaysLong, since the only decimal metric is max-aggregated and its sum is always empty.Qualification is unaffected. I built the parent commit and diffed both tools' output on a CPU log and a Photon log: byte-identical apart from one new warning naming an accumulable no parser can read, which was previously dropped in silence.
Testing
883 tests, 0 failures. Scalastyle and the Scala 2.13 cross-build are clean.
parseAccumFieldToLonghad no unit test at all. There are now cases pinning each parse branch's output unit, the fixed-point scaling, the overflow boundary on both signs, and the inputsDouble.parseDoublewrongly accepts.AnalysisSuiteno longer re-implements the unit rule as its own expectation nor keeps a private copy of the max-aggregate set, and it asserts that every GPU accumulable carrying a non-zero value produces a row, which is the assertion whose absence let eight metrics vanish unnoticed.