Skip to content

Add PR-description staleness guardrails (PROF-16592) - #980

Open
jbachorik wants to merge 4 commits into
mainfrom
cosmic-minty-quokka
Open

jbachorik wants to merge 4 commits into
mainfrom
cosmic-minty-quokka

Conversation

@jbachorik

@jbachorik jbachorik commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:

Agents evolve a branch after its PR is open — review fixes, rebase replays, scope changes — without re-verifying the PR title and description. This adds a warn-level guardrail set that keeps descriptions in sync:

  • pr-writing skill rule plus an AGENTS.md routing line: whenever commits are added to a branch with an open PR, re-verify the title and description against the new diff and update what no longer holds.
  • .githooks/pre-push: every branch push that passes the marker, copyright and CLAUDE.md gates and targets a branch with an open PR warns (never blocks) to re-verify the description.
  • The Claude Code / Pi review-freshness hook: the PR-ready subcommand is blocked when the PR description was last edited before the PR's head commit (one gh pr view probe; comments also bump updatedAt, so a fresh value allows). The probe fails open.

Motivation:

The pre-PR review gate certifies the code but nothing watches the description afterwards, so stale descriptions (wrong scope, dead numbers, dropped accepted costs) reach reviewers and merge. The pre-push path warns because the common write-description-then-commit-then-push flow has the description edit predating the head commit — a timestamp gate there would false-fire; the ready path blocks because marking ready is an explicit freshness claim.

Additional Notes:

Accepted costs, flagged for reviewer acknowledgment:

  • pre-push now makes one gh pr list call on every branch push (previously skipped for a fresh marker); offline pushes fail open, at up to a 10s timeout.
  • A rebase or --force-with-lease replay rewrites commit dates, so marking ready blocks until the description is re-saved — a no-op edit clears it (documented in utils/README.md known limits).

Agentic review (sphinx, quick effort): 0 CRITICAL/HIGH; 4 LOW/INFO findings fixed via the address run (naive-timestamp fail-open, three fail-open tests, README rebase note).

Commit 1222e4e82 (Fix spotless violation in JvmtiEventTimingTest) repairs a base-commit formatting break: #947 landed that file unformatted on main, failing check-formatting for every branch based on it (main's own CI run at 4cc58ca39 failed). Content is byte-for-byte spotlessApply output, verified by a green :ddprof-test:spotlessCheck (log build/logs/20261111-183236); re-review waived by the user for this mechanical commit. Commit 5545c352e addresses the CodeQL code-quality comment on stale_description (implicit adjacent string literals): explicit +, jq query text byte-identical, re-review waived.

How to test the change?:

bash utils/tests/test_review_marker.sh — 260 tests, 0 failed (adds description-staleness block/equal/fresh, unparseable timestamp, missing UTC designator, no-commits fail-open, and pre-push warning cases).

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles
    credentials of any kind, I've requested a security review (run the dd:platform-security-review
    skill, or file a request via the PSEC review form).
    bewaire also runs automatically on every PR.
  • This PR doesn't touch any of that.
  • JIRA: PROF-16592

pre-push warns on every open-PR push to re-verify the description;
the PR-ready subcommand blocks when the description predates the PR
head commit, failing open; pr-writing skill and AGENTS.md route the
re-verification.
@jbachorik jbachorik added the AI label Nov 11, 2026
@dd-octo-sts

dd-octo-sts Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #43071213716 | Commit: b30ff70 | Duration: 20m 1s (longest job)

✅ All 32 test jobs passed

Status Overview

JDK glibc-aarch64/debug glibc-amd64/debug musl-aarch64/debug musl-amd64/debug
8 - ✅ - -
8-ibm - ✅ - -
8-j9 ✅ ✅ - -
8-librca - - ✅ ✅
8-orcl - ✅ - -
11 - ✅ - -
11-j9 ✅ ✅ - -
11-librca - - ✅ ✅
17 ✅ ✅ - -
17-graal ✅ ✅ - -
17-j9 ✅ ✅ - -
17-librca - - ✅ ✅
21 ✅ ✅ - -
21-graal ✅ ✅ - -
21-librca - - ✅ ✅
25 ✅ ✅ - -
25-graal ✅ ✅ - -
25-librca - - ✅ ✅

Legend: ✅ passed | ❌ failed | ⚪ skipped | 🚫 cancelled

Summary: Total: 32 | Passed: 32 | Failed: 0


Updated: 2026-11-11 18:08:46 UTC

@dd-octo-sts

dd-octo-sts Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Scan-Build Report

User:runner@runnervmwuk1x
Working Directory:/home/runner/work/java-profiler/java-profiler/ddprof-lib/src/test/make
Command Line:make -j4 all
Clang Version:Ubuntu clang version 21.1.8 (6ubuntu1)
Date:Wed Nov 11 17:47:33 2026

Bug Summary

Bug TypeQuantityDisplay?
All Bugs2
Logic error
Dereference of null pointer1
Unused code
Dead initialization1

Filter Results by File

Reports

Bug Group Bug Type ▾ File Function/Method Line Path Length
Unused codeDead initializationreferenceChains.cppapplyOomRamp4971
Logic errorDereference of null pointerperfEvents_linux.cppunregisterThread72026

Formatting-only: the file landed on main unformatted via #947
(PROF-16527); spotlessCheck fails for every branch based on it.
@dd-octo-sts

dd-octo-sts Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

❌ 39 passed, 1 failed out of 40 expected configurations

Test Matrix

Platform JDK 8 JDK 11 JDK 17 JDK 21 JDK 25
glibc-x64-hotspot ✅ ✅ ✅ ✅ ✅
glibc-x64-openj9 ✅ ✅ ✅ ✅ ✅
glibc-arm64-hotspot ✅ ✅ ✅ ✅ ✅
glibc-arm64-openj9 ✅ ✅ ✅ ✅ ✅
musl-x64-hotspot ✅ ✅ ✅ ✅ ✅
musl-x64-openj9 ✅ ✅ ✅ ✅ ✅
musl-arm64-hotspot ✅ ✅ ✅ ✅ ✅
musl-arm64-openj9 ✅ ✅ ✅ ❌ ✅

Failure Details

musl-arm64-openj9-jdk21

Profiler-only:

"  Skipped (tracer not expected in this configuration)"
""
"[7/8] Checking for unexpected events..."
"  ✓ No unexpected events found"
""
"[8/8] Scenario-specific validation (ddprof_only)..."
"  Validating profiler-only scenario..."
"  ✓ Profiler-only scenario checks passed"
""
"=== Validation Summary ==="
"ExecutionSample:            0.0 events (datadog.ExecutionSample)"
"Stack traces:               0.0 samples"
"Thread diversity:           0.0 threads"
"Allocation samples:         379 events (datadog.ObjectSample)"
"ThreadAllocationStatistics: 44 events"
""
"VALIDATION_FAILED: One or more checks did not pass"

Script executed successfully: 425 commands
VALIDATION_FAILED: Validation checks did not pass

Links

CodeQL code-quality flags implicit adjacent string concatenation;
same string, explicit +.
@jbachorik
jbachorik marked this pull request as ready for review November 11, 2026 17:45
@jbachorik
jbachorik requested a review from a team as a code owner November 11, 2026 17:45
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T17:57:49.923074Z b45dc07 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b45dc07d8f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +249 to +251
"--json", "updatedAt,commits", "-q",
('.updatedAt + " " + (.commits | if length > 0 then .[-1].committedDate'
+ ' else "" end)')], cwd)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Compare against the description's edit time, not PR activity

When new commits are pushed to an existing PR, GitHub also advances the PR's updatedAt, even if its description remains unchanged. That timestamp will normally be later than the newly pushed commit's committedDate, so this check allows gh pr ready in precisely the stale-description scenario it is intended to detect. This is broader than the acknowledged comment-related bypass: the code push itself makes the description appear fresh. GitHub defines updatedAt as when “the object was last updated,” not when its body was edited (GraphQL reference). Query the description's edit timestamp, with an appropriate creation-time fallback, instead.

Useful? React with 👍 / 👎.

@datadog-datadog-us1-prod datadog-datadog-us1-prod Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

On PRs with more than 100 commits, the ready-time guard can compare freshness against commit 100 instead of the head, allowing stale descriptions through.

Open Bits AI session

🤖 Bits Code Review · Commit 5545c35

None (fail open) when gh fails, the PR has no commits, or a timestamp is unparseable."""
out = gh_query(["pr", "view", *pr_selector(args)[:1], *( ["-R", repo] if repo else [] ),
"--json", "updatedAt,commits", "-q",
('.updatedAt + " " + (.commits | if length > 0 then .[-1].committedDate'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Compare description freshness against the actual head commit

For PRs with more than 100 commits, gh pr view --json commits returns only the first 100 commits, so .[-1] selects commit 100 rather than the head. When updatedAt falls between those commit timestamps, the hook allows gh pr ready despite the description predating the actual head. Query the head commit's timestamp directly and update the gh mock and regression coverage accordingly.

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means
  • 👍 Useful: correct, and you're fixing it or will.
  • 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
  • 😕 Unclear: you can't tell what it means or why it matters.
  • 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

🤖 Bits Code Review · Open Bits AI session

@jbachorik
jbachorik marked this pull request as draft November 11, 2026 17:55
@jbachorik
jbachorik marked this pull request as ready for review November 11, 2026 17:55
@datadog-datadog-us1-prod

datadog-datadog-us1-prod Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Pipelines

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 4 Pipeline jobs failed

DDCI - DataDog/java-profiler | post-pr-comment — 🔧 Needs a code fix, caused by this PR

View more details · View in DDCI

DataDog/java-profiler | integration-test-arm64-musl: [openj9, 21] — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

DataDog/java-profiler | post-pr-comment — 🔧 Needs a code fix, caused by this PR

View more details · View in GitLab

View all 4 failed jobs.

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b45dc07 | Docs | View more details | Give us feedback!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant