Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/data-pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -442,3 +442,14 @@ All normalizer logic lives in `packages/db/src/etl/normalizers.ts`. The function
- **v2 (2025-12-19+)**: Separate `prefill_tp` / `decode_tp` / `prefill_ep` / `decode_ep` / `prefill_dp_attention` / `decode_dp_attention` / `prefill_num_workers` / `decode_num_workers` / `num_prefill_gpu` / `num_decode_gpu` fields are present. These map directly; `num_prefill_gpu` / `num_decode_gpu` fall back to `tp * ep` if absent.

Detection is a single `'prefill_tp' in row` check — no version field is required in the artifact.

### Power Audit Provenance (`power_invalid_reasons`, `power_audit`)

Producers (`aggregate_power.py`) annotate every aggregate result row with two optional provenance fields alongside the `power_valid` verdict:

- **`power_invalid_reasons`** — array of snake_case reason-code strings explaining a withheld verdict (emitted when `power_valid == 0`), e.g. `sampling_gap_exceeded`, `expected_gpu_count_mismatch`.
- **`power_audit`** — compact measurement-window audit object with a fixed 8-key shape: `window_start_unix`, `window_end_unix`, `expected_gpu_count`, `observed_gpu_count`, `sample_count`, `max_sample_gap_s`, `producer_sha`, `exporter_image_sha256`. Present on valid and invalid rows alike.

`mapBenchmarkRow()` narrows them defensively (`extractPowerInvalidReasons` / `extractPowerAudit`): reason codes must match `/^[a-z][a-z0-9_]*$/` (≤ 64 chars, deduplicated, capped at 32), audit numerics must be finite (counts: non-negative safe integers), shas collapse to `null` unless a non-empty string ≤ 128 chars, and unknown audit keys are dropped. An empty result maps to `undefined`, so the dedicated `benchmark_results.power_invalid_reasons` / `power_audit` JSONB columns (migration 014, mirroring the `workers` precedent from migration 006) store SQL NULL — never `[]` or `{}`. Legacy artifacts without the fields flow through every layer as NULL/undefined.

Reads are **permanently tolerant**: `queries/benchmarks.ts` selects the columns as `to_jsonb(br) -> 'power_invalid_reasons'` (and `lb` on the matview branch) rather than bare column references. A bare reference fails during query planning until the next ingest workflow applies the migration, because migrations run in the ingest workflows rather than at Vercel deploy. The key lookup degrades to NULL while the column is missing and is byte-identical once it exists, making deploy order irrelevant.
34 changes: 34 additions & 0 deletions packages/app/src/app/api/unofficial-run/route.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,40 @@ describe('normalizeArtifactRows', () => {
},
);

it('carries power audit provenance on overlay rows', () => {
const audit = {
window_start_unix: 1756174800,
window_end_unix: 1756175400,
expected_gpu_count: 8,
observed_gpu_count: 8,
sample_count: 4800,
max_sample_gap_s: 1.013,
producer_sha: null,
exporter_image_sha256: null,
};
const [row] = normalizeArtifactRows(
[
rawRow({
power_valid: 0,
power_invalid_reasons: ['sampling_gap_exceeded'],
power_audit: audit,
}),
],
'2026-03-01',
);

expect(row.power_invalid_reasons).toEqual(['sampling_gap_exceeded']);
expect(row.power_audit).toEqual(audit);
expect(row.metrics).not.toHaveProperty('power_invalid_reasons');
expect(row.metrics).not.toHaveProperty('power_audit');
});

it('leaves provenance keys undefined for rows without the contract fields', () => {
const [row] = normalizeArtifactRows([rawRow()], '2026-03-01');
expect(row.power_invalid_reasons).toBeUndefined();
expect(row.power_audit).toBeUndefined();
});

it('preserves recipe identity for unofficial overlays', () => {
const rows = normalizeArtifactRows(
[
Expand Down
2 changes: 2 additions & 0 deletions packages/app/src/app/api/unofficial-run/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,8 @@ export function normalizeArtifactRows(
// Surface the same per-worker payload the DB path emits so unofficial
// overlays carry the multinode measured-power breakdown too.
workers: params.workers,
power_invalid_reasons: params.powerInvalidReasons,
power_audit: params.powerAudit,
date,
run_url: runUrl,
});
Expand Down
1 change: 1 addition & 0 deletions packages/app/src/components/inference/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ export interface AggDataEntry {
// Measured GPU telemetry (emitted by runner's aggregate_power.py).
// Optional because historical runs predate the fields.
power_valid?: number;
power_invalid_reasons?: string[];
power_metric_schema_version?: number;
/**
* Certification tier for the measured power telemetry, derived by
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -798,6 +798,82 @@ describe('generateGPUGraphTooltipContent', () => {
});
});

describe('measured-power withheld tooltip line', () => {
const reasons = ['sampling_gap_exceeded', 'expected_gpu_count_mismatch'];

it('renders the withheld line with humanized codes (en)', () => {
const html = generateTooltipContent(
tooltipConfig({ data: pt({ power_valid: 0, power_invalid_reasons: reasons }) }),
);
expect(html).toContain('Measured power withheld');
expect(html).toContain('sampling gap exceeded');
expect(html).toContain('expected gpu count mismatch');
});

it('renders the withheld line in Chinese on /zh surfaces', () => {
const html = generateTooltipContent(
tooltipConfig({
data: pt({ power_valid: 0, power_invalid_reasons: reasons }),
locale: 'zh',
}),
);
expect(html).toContain('实测功耗未采信');
expect(html).toContain('sampling gap exceeded');
});

it('filters malformed codes before HTML interpolation (defense in depth)', () => {
const html = generateTooltipContent(
tooltipConfig({
data: pt({
power_valid: 0,
power_invalid_reasons: ['<img src=x>', 'sampling_gap_exceeded', 'UPPER'],
}),
}),
);
expect(html).not.toContain('<img src=x>');
expect(html).not.toContain('UPPER');
expect(html).toContain('sampling gap exceeded');
});

it('omits the line entirely when every code is malformed', () => {
const html = generateTooltipContent(
tooltipConfig({ data: pt({ power_valid: 0, power_invalid_reasons: ['<img src=x>'] }) }),
);
expect(html).not.toContain('Measured power withheld');
});

it.each([
['absent reasons', pt({ power_valid: 0 })],
['empty reasons', pt({ power_valid: 0, power_invalid_reasons: [] })],
['valid row', pt({ power_valid: 1 })],
])('omits the line for %s', (_name, data) => {
const html = generateTooltipContent(tooltipConfig({ data }));
expect(html).not.toContain('Measured power withheld');
});

it('gives unofficial overlay tooltips the same line', () => {
const html = generateOverlayTooltipContent({
...tooltipConfig({ data: pt({ power_valid: 0, power_invalid_reasons: reasons }) }),
overlayData: {
label: 'feature-branch',
hardwareConfig: mockHardwareConfig,
data: [],
runUrl: 'https://example.com',
} as any,
} as OverlayTooltipConfig);
expect(html).toContain('Measured power withheld');
expect(html).toContain('sampling gap exceeded');
});

it('gives GPU comparison tooltips the same line', () => {
const html = generateGPUGraphTooltipContent(
tooltipConfig({ data: pt({ power_valid: 0, power_invalid_reasons: reasons }) }),
);
expect(html).toContain('Measured power withheld');
expect(html).toContain('sampling gap exceeded');
});
});

describe('worker power drilldown', () => {
const workers = [
{ role: 'frontend', worker_idx: 0, hosts: ['fe0'], num_gpus: 0, avg_power_w: 120 },
Expand Down
19 changes: 19 additions & 0 deletions packages/app/src/components/inference/utils/tooltipUtils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ const TOOLTIP_STRINGS = {
powerData: 'Power Measurement',
powerCertified: 'Validated (current PowerX method)',
powerLegacy: 'Historical (not validated under the current method)',
powerWithheld: 'Measured power withheld',
},
zh: {
dismiss: '点击其他区域关闭',
Expand All @@ -164,6 +165,7 @@ const TOOLTIP_STRINGS = {
powerData: '功耗测量',
powerCertified: '已验证(采用当前 PowerX 方法)',
powerLegacy: '历史测量(尚未按当前方法验证)',
powerWithheld: '实测功耗未采信',
},
} as const;

Expand Down Expand Up @@ -519,6 +521,20 @@ const generateParallelismHTML = (d: InferenceData, locale: Locale = 'en'): strin
${tooltipLine(t.dpAttention, d.dp_attention ? t.yes : t.no)}`;
};

const POWER_REASON_CODE_RE = /^[a-z][a-z0-9_]*$/u;

/** Revalidate producer codes before interpolating them into raw tooltip HTML. */
const powerWithheldHTML = (d: InferenceData, locale: Locale): string => {
if (!Array.isArray(d.power_invalid_reasons) || d.power_invalid_reasons.length === 0) return '';
const codes = d.power_invalid_reasons
.filter(
(code) => typeof code === 'string' && code.length <= 64 && POWER_REASON_CODE_RE.test(code),
)
.map((code) => code.replaceAll('_', ' '));
if (codes.length === 0) return '';
return tooltipLine(TOOLTIP_STRINGS[locale].powerWithheld, codes.join(', '));
};

/**
* Generates HTML content for official data point tooltips.
*
Expand Down Expand Up @@ -572,6 +588,7 @@ export const generateTooltipContent = (config: TooltipConfig): string => {
${tooltipLine(t.concurrency, `${d.conc}`)}
${tooltipLine(t.precision, `${d.precision.toUpperCase()}`)}
${generateCacheMetadataHTML(d, locale)}
${powerWithheldHTML(d, locale)}
${generateAgenticHTML(d, locale)}
${generateWorkerPowerHTML(d, isPinned, locale)}
${runLinkHTML(runUrl, locale)}
Expand Down Expand Up @@ -614,6 +631,7 @@ export const generateOverlayTooltipContent = (config: OverlayTooltipConfig): str
${tooltipLine(t.concurrency, `${d.conc}`)}
${tooltipLine(t.precision, `${d.precision.toUpperCase()}`)}
${generateCacheMetadataHTML(d, locale)}
${powerWithheldHTML(d, locale)}
Comment thread
cursor[bot] marked this conversation as resolved.
${generateAgenticHTML(d, locale)}
${generateWorkerPowerHTML(d, isPinned, locale)}
</div>
Expand Down Expand Up @@ -673,6 +691,7 @@ export const generateGPUGraphTooltipContent = (config: TooltipConfig): string =>
${tooltipLine(t.concurrency, `${d.conc}`)}
${tooltipLine(t.precision, `${d.precision.toUpperCase()}`)}
${generateCacheMetadataHTML(d, locale)}
${powerWithheldHTML(d, locale)}
${generateAgenticHTML(d, locale)}
${generateWorkerPowerHTML(d, isPinned, locale)}
${runLinkHTML(runUrl, locale)}
Expand Down
5 changes: 3 additions & 2 deletions packages/app/src/lib/api-documentation.power.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,14 @@ describe('measured-power API documentation', () => {
}
});

it('reserves optional power_invalid_reasons and power_audit row fields', () => {
it('documents optional power_invalid_reasons and power_audit row fields', () => {
const reasons = benchmarkRowSchema?.properties?.power_invalid_reasons;
expect(reasons?.type).toBe('array');
expect(reasons?.items).toEqual({ type: 'string' });
expect(reasons?.description).not.toMatch(/reserved|forthcoming/iu);

const audit = benchmarkRowSchema?.properties?.power_audit;
expect(audit?.description).not.toMatch(/reserved|forthcoming/iu);
expect(Object.keys(audit?.properties ?? {}).toSorted()).toEqual(
[
'window_start_unix',
Expand All @@ -41,7 +43,6 @@ describe('measured-power API documentation', () => {
'exporter_image_sha256',
].toSorted(),
);
// Producers may emit partial audits, so individual audit fields remain optional.
expect(audit?.required).toBeUndefined();

expect(benchmarkRowSchema?.required).not.toContain('power_invalid_reasons');
Expand Down
8 changes: 4 additions & 4 deletions packages/app/src/lib/api-documentation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ const powerAuditSchema: ApiSchema = {
},
additionalProperties: true,
description:
'Optional power measurement-window audit. Individual fields may be absent; legacy rows omit the object.',
'Compact power measurement-window audit emitted alongside the power_valid verdict: window bounds, expected vs. observed GPU counts, sample statistics, and producer identity (producer_sha / exporter_image_sha256 are null for single-node telemetry without an srt-slurm producer). Present on valid and invalid rows from provenance-aware producers; absent on legacy rows.',
};
const workerPowerSchema = objectSchemaWithOptional(
{
Expand Down Expand Up @@ -297,7 +297,7 @@ const benchmarkRowSchema = objectSchemaWithOptional(
power_invalid_reasons: {
...arraySchema(stringSchema),
description:
'Optional snake_case validation reason codes when metrics.power_valid == 0. Absent on legacy rows.',
'Producer snake_case reason codes explaining a withheld measurement, present when metrics.power_valid == 0. Absent on legacy rows and validated rows.',
},
power_audit: powerAuditSchema,
date: { type: 'string', format: 'date' },
Expand Down Expand Up @@ -2678,8 +2678,8 @@ const overview = {
id: 'measured-power',
title: text('Measured power', '实测功率'),
description: text(
'Benchmark rows may carry measured power, energy, and GPU-telemetry metric keys (avg_power_w, joules_per_*, avg_temp_c, peak_temp_c, avg_util_pct, avg_mem_used_mb). power_valid is tri-state: 1 means the measurement window was validated; 0 means validation failed and measured values are withheld end-to-end (the producer strips them and ingest scrubs them — treat any that remain as unreliable); absent marks a legacy row predating validation. power_metric_schema_version == 2 defines every unprefixed joules_per_* field as whole-deployment energy — unversioned disaggregated joules are ambiguous because those fields previously carried role-local values. workers[] carries the per-worker power/telemetry breakdown on multinode and disaggregated runs. power_invalid_reasons and power_audit provide optional producer validation details. For measured-power requests, use powerValid=strictV2 to require power_valid == 1 and power_metric_schema_version == 2. It is the only supported power filter. Omit powerValid for general benchmark requests so results remain available even when they lack valid power measurements.',
'基准测试数据行可能包含实测功率、能耗和 GPU 遥测指标(avg_power_w、joules_per_*、avg_temp_c、peak_temp_c、avg_util_pct、avg_mem_used_mb)。power_valid 有三种状态:1 表示测量窗口已通过验证;0 表示验证失败,生产端会移除实测值,摄取端也会再次清除,若仍有残留,应视为不可靠;缺失表示该数据行早于验证机制。power_metric_schema_version == 2 规定所有无前缀的 joules_per_* 字段均按整个部署统计能耗。未标注版本的分离式部署数据中,这些字段曾记录单个角色的能耗,因此其统计口径不明确。多节点和分离式运行中,各 worker 的功率和遥测明细位于 workers[]。power_invalid_reasons power_audit 可包含生产端的验证详情。查询实测功率时,使用 powerValid=strictV2,仅保留 power_valid == 1 且 power_metric_schema_version == 2 的行。这是唯一支持的功率筛选值。常规基准测试请求应省略 powerValid,以保留缺少有效功率测量的结果。',
'Benchmark rows may carry measured power, energy, and GPU-telemetry metric keys (avg_power_w, joules_per_*, avg_temp_c, peak_temp_c, avg_util_pct, avg_mem_used_mb). power_valid is tri-state: 1 means the measurement window was validated; 0 means validation failed and measured values are withheld end-to-end (the producer strips them and ingest scrubs them — treat any that remain as unreliable); absent marks a legacy row predating validation. power_metric_schema_version == 2 defines every unprefixed joules_per_* field as whole-deployment energy — unversioned disaggregated joules are ambiguous because those fields previously carried role-local values. workers[] carries the per-worker power/telemetry breakdown on multinode and disaggregated runs. power_invalid_reasons lists the producer reason codes (snake_case) on withheld rows (power_valid == 0), and power_audit carries the compact measurement-window audit (window bounds, GPU counts, sample statistics, producer identity) on valid and invalid rows alike; both are absent on rows ingested before the provenance contract. For measured-power requests, use powerValid=strictV2 to require power_valid == 1 and power_metric_schema_version == 2. It is the only supported power filter. Omit powerValid for general benchmark requests so results remain available even when they lack valid power measurements.',
'基准测试数据行可能包含实测功率、能耗和 GPU 遥测指标(avg_power_w、joules_per_*、avg_temp_c、peak_temp_c、avg_util_pct、avg_mem_used_mb)。power_valid 有三种状态:1 表示测量窗口已通过验证;0 表示验证失败,生产端会移除实测值,摄取端也会再次清除,若仍有残留,应视为不可靠;缺失表示该数据行早于验证机制。power_metric_schema_version == 2 规定所有无前缀的 joules_per_* 字段均按整个部署统计能耗。未标注版本的分离式部署数据中,这些字段曾记录单个角色的能耗,因此其统计口径不明确。多节点和分离式运行中,各 worker 的功率和遥测明细位于 workers[]。power_invalid_reasons 在实测值被扣留的行(power_valid == 0)上列出生产端的 snake_case 原因码;power_audit 在有效与无效行上都携带精简的测量窗口审计信息(窗口边界、GPU 数量、采样统计、生产者标识);早于溯源契约摄取的行均不含这两个字段。查询实测功率时,使用 powerValid=strictV2,仅保留 power_valid == 1 且 power_metric_schema_version == 2 的行。这是唯一支持的功率筛选值。常规基准测试请求应省略 powerValid,以保留缺少有效功率测量的结果。',
),
shape: 'BenchmarkRows',
example: {
Expand Down
4 changes: 2 additions & 2 deletions packages/app/src/lib/api-route-catalog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ export const apiRouteCatalog = [
en: 'UI-only overlay for unofficial workflow artifacts; upstream artifact availability and shape are not stable.',
zh: '仅供界面叠加非官方工作流制品;上游制品的可用性和结构并不稳定。',
},
sourceSha256: '4a3f3da8399c741c26f0f502d44b1870a8ccdc05775edfd6ea3dee4e020df25c',
sourceSha256: '6973369b5ee4ded847754b7d38ec7b9db7c30f895c9b0e54c7aaf9b2bf7eb45d',
},
{
source: 'src/app/api/v1/agentic-aggregates/route.ts',
Expand Down Expand Up @@ -731,7 +731,7 @@ export const apiContractSourceDigests = [
},
{
source: '../db/src/queries/benchmarks.ts',
sourceSha256: '486e34d55275170c7e0544af24c151199752eb5628191d38606b1ecec289dfdf',
sourceSha256: '0793d7901630d74301493a34436c8ad868cc4c938d46ba40256b19fc76b17332',
reviewArea: {
en: 'Benchmark row fields and latest, exact-run, history, and TCO query semantics.',
zh: '基准行字段以及最新、精确运行、历史和 TCO 查询语义。',
Expand Down
9 changes: 9 additions & 0 deletions packages/app/src/lib/benchmark-api-view.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ const rows = [
unused_debug_metric: 99,
},
workers: [{ rank: 0, avg_power_w: 700 }],
power_invalid_reasons: ['sampling_gap_exceeded'],
power_audit: { sample_count: 4800, producer_sha: null, exporter_image_sha256: null },
},
{
benchmark_type: 'single_turn',
Expand Down Expand Up @@ -46,6 +48,13 @@ describe('toCalculatorBenchmarkRows', () => {
]);
});

it('strips workers and the power audit provenance from the payload-trimmed view', () => {
const [row] = toCalculatorBenchmarkRows(rows, '1k/1k');
expect(row).not.toHaveProperty('workers');
expect(row).not.toHaveProperty('power_invalid_reasons');
expect(row).not.toHaveProperty('power_audit');
});

it('keeps all three cache tiers — the trim cannot know which one a row will use', () => {
// `measuredCacheHitRate` picks between external and CPU per row, so the allowlist
// has to pass all three through or the choice is made for it by the trim.
Expand Down
Loading
Loading