BenchmarkReporter can recommend publication while Tier 2 is NOT RUN

[holgerroth]

Description

BenchmarkReporter can emit an overall PASS — Recommended for publication verdict while its Tier Status table reports:

Tier 2 | Semantic deduplication | NOT RUN | No result was recorded

This occurs when the reporter receives Tier 1 and valid Tier 3 results but no Tier 2 results, as can happen when CI executes tiers separately and publishes a tier-local BENCHMARK.md.

The card is internally contradictory: it recommends publication despite displaying missing evaluation evidence.

Public examples

The same mismatch appears in signed benchmark cards for unrelated skills:

• nvflare-autofl
• nemo-relay-integrate-upstream
• omniverse-usd-performance-tuning

Each card recommends publication while reporting Tier 2 as NOT RUN. These examples span different repositories and skills, so the behavior is not skill-specific.

Reproduction

At public SkillEvaluator commit 5a909a8:

git clone https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
git checkout 5a909a8af48e072537a83d9f8ca0c44476e0e7a3

PYTHONPATH=src python - <<'PY'
import runpy

from skillevaluator.reporting import BenchmarkReporter

fixtures = runpy.run_path("tests/golden/test_benchmark_card.py")
results = [
    fixtures["_deterministic_results"]()[0],
    fixtures["_live_tier3_result"](),
]

report = BenchmarkReporter(include_timestamp=False).render_all(results)

print(report.splitlines()[2])
print(next(line for line in report.splitlines() if line.startswith("| Tier 2")))
PY

Observed output:

> ✅ **Overall verdict: PASS — Recommended for publication**
| Tier 2 | Semantic deduplication | **NOT RUN** | No result was recorded |

Relevant contract

BenchmarkReporter correctly reports NOT RUN when its Tier 2 result group is empty:

• _render_tier_status
• _tier_status
• Tier classification

However, the overall-status calculation does not treat the absence of a required Tier 2 result as incomplete.

Expected behavior

If Tier 2 is required for the publication profile and no Tier 2 result is present, the benchmark should:

• report Overall verdict: INCOMPLETE;
• omit the publication recommendation; and
• retain Tier 2: NOT RUN with its existing explanation.

Alternatively, the reporter could accept an explicit required-tier policy and only fail closed for tiers required by that policy.

Suggested regression test

Add a test covering valid Tier 1 plus valid Tier 3 with no Tier 2 result:

def test_benchmark_is_incomplete_when_required_tier2_is_missing() -> None:
    results = [_deterministic_results()[0], _live_tier3_result()]

    rendered = BenchmarkReporter(include_timestamp=False).render_all(results)

    assert "Overall verdict: INCOMPLETE" in rendered
    assert "Recommended for publication" not in rendered
    assert "| Tier 2 | Semantic deduplication | **NOT RUN** |" in rendered

A complementary integration check should ensure that split-tier publication regenerates BENCHMARK.md from the same merged result collection used for the combined HTML and JSON reports.