Skip to content

fix(wall): address sphinx review findings for #663 - #890

Open
jbachorik wants to merge 8 commits into
paul.fournillon/wallclock-all-threadsfrom
sphinx/fix-663-review-findings
Open

jbachorik wants to merge 8 commits into
paul.fournillon/wallclock-all-threadsfrom
sphinx/fix-663-review-findings

Conversation

@jbachorik

@jbachorik jbachorik commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?:
Hardens the wall-clock precheck and thread registry from #663, and completes reusable unowned wall-clock root preservation, retained context, recording/context restarts, and monotonic timer cadence. This is the stacked follow-up to #663: its base remains paul.fournillon/wallclock-all-threads; #663 itself is unchanged.

  • Preserve exact reusable trace IDs across storage rotations using isolated liveness collectors; include zero-weight/out-of-context cached roots. Recoverable preservation failure marks only matching cached IDs dropped while retaining weight, state and cadence.
  • Keep context-window membership as owner truth across recordings, validate per-recording registry projections, and retain the original span/root IDs and configured tags with deferred unowned tails.
  • Keep ordinary samples at weight 1 with actual serialization timestamps. Advance a monotonic, unchanged-dither opportunity deadline without catch-up bursts or elapsed-time weight compensation; add opt-in fixed-size timer-local phase diagnostics.
  • Retain the earlier TLS teardown, index-lifetime, recording-epoch, retryable timer-join, failed-start teardown and DEBUG-only test-seam fixes already on this PR.
  • Preserve the PR-only active-batch shutdown-drain fix through the stronger bounded awaitBatch/joinUntil/stopGracefully implementation: every worker must drain before success; interruption, stalls, expired/submillisecond budgets and scheduler cleanup cannot silently report success. The 14-case suite exercises real independently gated workers and both disabled commands.

Motivation:
Cached wall stacks must remain live and keep their factual identity/context across context changes, dumps and fresh recordings. Timer work/wake lateness should not reduce a nominal one-second sampling opportunity window, and shutdown must not leave hook-emitting workers behind. Signal handlers retain their no-allocation/no-locking constraints; no tracer/backend changes or production rollout are included.

Additional Notes:

  • Upstream formatter prerequisite: carry the exact Spotless plugin content from merged fix(build): register Java formatting lazily and update formatter [PROF-16537] #955 / e8c3dfbd6d719bc654fa46b55e9912c248191781 (blob 56a38829f0e95b6e2ee1e360ff6597e89595f16d, GJF 1.30.0 and lazy Java-plugin registration). fix(wall): address sphinx review findings for #663 #890's stacked feat(wall): support signal suppression for unfiltered threads #663 base still has GJF 1.7, so this upstream prerequisite is carried here to make the branch's own JVM25 formatting gates work; main already contains it. The former local GJF 1.36.1 prerequisite and unpublished formatter-only TestKit are not included. This is the canonical upstream fix, not a competing formatter implementation.
  • Linux qualification is PENDING: integrated Linux x64 HotSpot8 shutdown14, HotSpot21 JVM, native ASan/UBSan/TSan and Linux-only stress cells have not run on this candidate. They remain queued for the exclusive heavy-VM slot; guest project write/remove probes and actual VM-headroom verification are required before execution. Host passes are not Linux/sanitizer passes. No skip or zero-case binary is counted as a pass.
  • Inherited unresolved host baseline: CPU-bound VirtualThreadWallClockTest, cstack=vmx, still fails to observe ForkJoinWorkerThread carrier frames on macOS ARM64 / HotSpot25. Debug: 48 passed / 22 skipped / 1 failed; production Release: 46 passed / 24 skipped / 1 failed (71 reached per configuration). This pre-existing diagnostic repeats the preserved predecessor outcomes; it was not filtered out or fixed here. Follow-up is routed to the JVM/stack-walker domain (feat: emit TaskBlock events for JVM blocking states #665 / CORE), and the broad host wall package is not reported passing.
  • Collapsing-oracle integrity: the pre-reconciliation integration produced 1010 timer sends = 1010 actual serialized weight-1 events. After adopting the canonical upstream formatter, the fresh final-candidate run produced 1000 sends = 1000 weight-1 events, one factual OS/Java thread identity, no missing stacks, and maximum decoded gap 1,607,750ns. Both retain the original one-second workload, totalWeight > 700 and count > 9 predicates. No weighting compensation, fabricated stack/thread identity, scheduled timestamp or weakened oracle was used.
  • Evidence carry-forward: historical collapsing 670/669, earlier formatter/setup failures and the original read-only Linux transport failure remain failed evidence. Later cells are new source-bound runs, not reclassifications. Original host evidence is retained; the PR-head integration and upstream reconciliation each received fresh host qualification. Final reviewed tree: 5ac5c5bbfbfe2b1520317d946b0d656a4dc22692; source-set SHA256: 189782eacee8320d802d6f5c3d6159d666f81cea01f6eaf2a2506ef4138cc81b. Evidence/design/scratch files are not part of the change.

How to test the change?:
All Gradle execution uses ./.claude/commands/build-and-summarize.

Fresh final-candidate macOS ARM64 / HotSpot25 results:

Coverage Actual result
Native Debug 1,038 passed; 8 genuine skips; 14 zero-case binaries; 0 failures
Native Release UNIT_TEST 1,032 passed; 9 genuine skips; 14 zero-case binaries; 0 failures (not a production JVM Release substitute)
Timer/tracker subset All 12 timer + 50 tracker cases reached within the full Debug run
Collapsing + shutdown 1 unchanged collapsing test + exact archived 14-case shutdown name multiset; 0 failures/errors/skips
Lifecycle Original 64-worker × 500-cycle test; 0 failures/skips
Performance execution All 6 enabled/disabled × 100/500/1,000-worker populations; 18 real forks, 54 scored iterations, 18 live mapped profiler workers and 18 retained JFRs; no invented performance threshold or claimed established speedup/regression
Formatting Actual candidate-scoped spotlessApply + spotlessCheck passed with the upstream formatter; no source bytes changed; not an aggregate full-repository style claim
Broad wall Debug/production Release Failed only at the inherited carrier-frame assertion disclosed above; all reached XML retained

Reproduction tasks: :ddprof-lib:gtestDebug, :ddprof-lib:gtestRelease; :ddprof-stresstest:testOwnedBlockPrecheck --tests com.datadoghq.profiler.chaos.ParkBlockChurnAntagonistTest; :ddprof-test:testDebug -Ptests=com.datadoghq.profiler.lifecycle.StartWithChurningThreadsTest; and the unchanged WallClockPrecheckOverheadBenchmark JMH include. Retain recordings with -PkeepJFRs for tests and JAVA_TOOL_OPTIONS=-Dddprof_test.keep_jfrs=true for JMH. The broad com.datadoghq.profiler.wallclock.* Debug/Release commands reproduce the open diagnostic. Source/JDK/compiler/wrapper/XML/JFR/native-image identities and owned-process cleanup are bound in the qualification manifest; two independent reviews approved the final reconciled tree.

For Datadog employees:

  • If this PR touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a security review.
  • This PR doesn't touch any of that.
  • JIRA: N/A

…me safety

Fixes: mutex acquisition from the pthread TLS destructor (retireThreadByTid
lock-free path), TidIndex use-after-free for in-flight lock-free readers in
~ThreadFilter, capacityExhausted() hint/gate contract, and context-window
state update races.
…cker state fixes

Timer loops re-read recordingEpoch() each tick instead of caching it;
stop() keeps the pthread_t when pthread_join fails; flushUnownedBlockedTail
resets only on valid payloads; owned-run admission preserves unflushed
unowned-blocked weight; test-only exitBlockedRun overload gated behind
UNIT_TEST; epoch publish ordering tightened. Adds lazyBackfill timer-path
coverage and positive-path fuzz assertions for owned-block suppression.
…aProfilerTestSupport to test sources

The registry-activity accessor now returns false in release builds; the
hooks live on JavaProfiler (DEBUG-gated natives) instead of a public
support class shipped in the main jar.
…haos threads

stopProfiler() no longer wipes trace context unconditionally; the
force-wall-start-failure toggle is disarmed in @AfterEach; wall-precheck
counter reads use getOrDefault to avoid NPEs; the parked-worker spin loop
yields; the churn antagonist checks join results and reuses one interruptor
thread.
@jbachorik jbachorik added the AI label Oct 23, 2026
@jbachorik
jbachorik marked this pull request as ready for review October 23, 2026 13:15
@jbachorik
jbachorik requested a review from a team as a code owner October 23, 2026 13:15
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T13:20:22.326641Z 3f77b78 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f77b7836b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The completed static review identified no reportable defects in deferred thread retirement, wall-clock restart handling, or blocked-time accounting. Runtime behavior was not tested.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 3f77b78

@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

CI Test Results

Run: #43021271121 | Commit: 993639c | Duration: 0s (longest job)

✅ All 0 test jobs passed

No CI outcome reports were found for this run.
Summary: Total: 0 | Passed: 0 | Failed: 0


Updated: 2026-11-11 12:34:40 UTC

@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

✅ All 40 integration tests passed

📊 Dashboard · 👷 Pipeline · 📦 c78cbf64

jbachorik and others added 3 commits November 2, 2026 11:01
Carry the exact SpotlessConventionPlugin content from upstream #955 (e8c3dfb): register Java formatting lazily and use GJF 1.30.0. The stacked #663 base predates that fix; no local formatter variant or formatter-only TestKit is included.
Preserve reusable exact-ID roots and retained context across rotations and recording/context restarts. Keep ordinary sample weight one while accounting for timer work and wake lateness with monotonic deadlines. Add fixed-size phase diagnostics and bounded shutdown/drain regressions without weakening the one-second collapsing oracle.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants