Skip to content

feat: Phase 5b: Add ledger/peer/tx spans + expand Grafana dashboards - #6433

Closed
pratikmankawde wants to merge 31 commits into
pratik/otel-phase5-docs-deploymentfrom
pratik/otel-phase5b-spans
Closed

feat: Phase 5b: Add ledger/peer/tx spans + expand Grafana dashboards#6433
pratikmankawde wants to merge 31 commits into
pratik/otel-phase5-docs-deploymentfrom
pratik/otel-phase5b-spans

Conversation

@pratikmankawde

@pratikmankawde pratikmankawde commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

PR Chain: #6436#6437#6438#6424#6425#6426#6427#6433 (this PR) / #6439#6493#6494#6513
Base: pratik/otel-phase5-docs-deployment

High Level Overview of Change

Add 6 new OpenTelemetry spans to instrument ledger building, ledger validation,
ledger storage, transaction application, and peer proposal/validation message
receipt. Expand all Grafana dashboards from 4 to 8 panels each and add 2 new
dashboards (Ledger Operations, Peer Network), bringing the total to 32 panels
across 5 dashboards. Each panel now includes a description explaining what it
shows and how the PromQL query calculates the metric.

Context of Change

Phase 5a (the base branch `pratik/otel-phase5-docs-deployment`) delivered
consensus spans, the observability stack (otel-collector, Jaeger, Tempo, Prometheus,
Grafana), and integration tests. During review, we identified that the
`trace_ledger` config category existed with its `XRPL_TRACE_LEDGER` macro but
had zero spans using it, and `trace_peer` had only 1 span (`tx.receive`).
Dashboard panels were limited to 4 per dashboard with no panel descriptions.

This PR fills those gaps:

New spans (6, bringing total from 10 to 16):

Span File Category Key Attributes
`ledger.build` BuildLedger.cpp:31 `trace_ledger` `xrpl.ledger.seq`
`ledger.validate` LedgerMaster.cpp:915 `trace_ledger` `xrpl.ledger.seq`, `xrpl.ledger.validations`
`ledger.store` LedgerMaster.cpp:409 `trace_ledger` `xrpl.ledger.seq`
`tx.apply` BuildLedger.cpp:88 `trace_transactions` `xrpl.ledger.tx_count`, `xrpl.ledger.tx_failed`
`peer.proposal.receive` PeerImp.cpp:1667 `trace_peer` `xrpl.peer.id`, `xrpl.peer.proposal.trusted`
`peer.validation.receive` PeerImp.cpp:2264 `trace_peer` `xrpl.peer.id`, `xrpl.peer.validation.trusted`

Dashboard changes (12 → 32 panels):

  • RPC Performance: 4 → 8 panels (+ throughput, success/error, top commands, WebSocket)
  • Transaction Overview: 4 → 8 panels (+ heatmap, tx.apply duration, peer receive, failed rate)
  • Consensus Health: 4 → 8 panels (+ mode tracking, accept vs close, validation vs close, heatmap)
  • Ledger Operations: NEW, 8 panels (build/validate/store rates and durations)
  • Peer Network: NEW, 4 panels (proposal/validation receive, trusted vs untrusted)

All traces viewable in both Jaeger and Grafana Tempo via OTel Collector fan-out.

Type of Change

  • New feature (non-breaking change which adds functionality)
  • Tests (you added tests for code that already exists, or your new feature included in this PR)
  • Documentation update

API Impact

None. This change only adds internal observability instrumentation (OpenTelemetry
spans) and Docker-based Grafana dashboards. No public API, libxrpl, or peer
protocol changes.

Test Plan

Integration test (`bash docker/telemetry/integration-test.sh`) passes 24/24
with all 16 span types verified across a 6-node consensus network:

  • Phase 2 (RPC): `rpc.request`, `rpc.process`, `rpc.command.*` — 5 traces each
  • Phase 3 (TX): `tx.process`, `tx.receive`, `tx.apply` — 5 traces each
  • Phase 4 (Consensus): `consensus.proposal.send`, `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send` — 5 traces each
  • Phase 5 (Ledger): `ledger.build`, `ledger.validate`, `ledger.store` — 5 traces each
  • Phase 5 (Peer): `peer.proposal.receive`, `peer.validation.receive` — 5 traces each
  • Spanmetrics: 22 Prometheus series with duration histograms
  • Grafana: healthy with 5 dashboards, 32 panels

@pratikmankawde pratikmankawde added the DraftRunCI Normally CI does not run on draft PRs. This opts in. label Feb 26, 2026
@pratikmankawde pratikmankawde changed the title Pratik/otel phase5b spans Phase 5b: Add ledger/peer/tx spans + expand Grafana dashboards Feb 26, 2026
@codecov

codecov Bot commented Feb 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.3%. Comparing base (0fc3f6b) to head (f0d8ed2).

Additional details and impacted files

Impacted file tree graph

@@                         Coverage Diff                          @@
##           pratik/otel-phase5-docs-deployment   #6433     +/-   ##
====================================================================
- Coverage                                81.3%   81.3%   -0.0%     
====================================================================
  Files                                    1000    1000             
  Lines                                   74764   74764             
  Branches                                 7631    7644     +13     
====================================================================
- Hits                                    60779   60771      -8     
- Misses                                  13985   13993      +8     
Files with missing lines Coverage Δ
src/xrpld/app/ledger/detail/BuildLedger.cpp 92.4% <ø> (ø)
src/xrpld/app/ledger/detail/LedgerMaster.cpp 42.6% <ø> (ø)
src/xrpld/overlay/detail/PeerImp.cpp 5.7% <ø> (ø)

... and 8 files with indirect coverage changes

Impacted file tree graph

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from fc1ed3c to 3581839 Compare February 27, 2026 18:01
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5b-spans branch from 2194022 to 261caa2 Compare February 27, 2026 18:03
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from 3581839 to 56cc5e6 Compare February 27, 2026 18:06
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5b-spans branch from 261caa2 to 4fce0da Compare February 27, 2026 18:06
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from 56cc5e6 to 85f583f Compare February 27, 2026 18:16
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5b-spans branch from 4fce0da to e2b2589 Compare February 27, 2026 18:16
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from 85f583f to 93aa588 Compare March 6, 2026 16:23
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5b-spans branch from e2b2589 to 31761c5 Compare March 6, 2026 16:23
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from 93aa588 to 87f4abe Compare March 6, 2026 17:37
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5b-spans branch 2 times, most recently from 6d0b533 to 2d9b5c7 Compare March 6, 2026 17:48
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from ecade9c to 3c3fb44 Compare March 8, 2026 18:28
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5b-spans branch 2 times, most recently from 294f01a to fbecd74 Compare March 8, 2026 18:59
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from 0c29397 to 27d208a Compare March 8, 2026 19:05
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5b-spans branch from fbecd74 to e95ae38 Compare March 8, 2026 19:05
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from 27d208a to cf2a9f3 Compare March 8, 2026 19:30
pratikmankawde and others added 21 commits March 31, 2026 22:28
…nfig

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…iring

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nfig

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nfig

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rd parity

Adds node health context (amendment_blocked, server_state) to rpc.command.*
spans, inspired by the community xrpl-validator-dashboard.

Part of the external dashboard parity initiative across phases 2-11.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add amendment_blocked and server_state span attributes to every
rpc.command.* span so operators can correlate RPC behavior with node state.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…sk lists

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… NetworkOPs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ard parity

Adds xrpl.peer.version attribute to tx.receive spans for version-mismatch
correlation during network upgrades.

Part of the external dashboard parity initiative across phases 2-11.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… 3.7)

Tag transaction receive spans with the relaying peer's rippled version
to enable version-mismatch correlation during network upgrades.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… close time

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…dashboard parity

Adds ledger_hash, validation.full to validation send/receive spans,
and validation_quorum, proposers_validated to consensus.accept spans.
Foundation for Phase 7 ValidationTracker agreement computation.

Part of the external dashboard parity initiative across phases 2-11.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add validation ledger hash and full-validation flag to
consensus.validation.send spans, plus quorum and proposer count to
consensus.accept spans for trace-level agreement analysis.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…span

Move validation_quorum and proposers_validated attributes from
consensus.accept.apply to consensus.accept span to match the design
spec. Both values are available in onAccept() scope.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ucture

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move xrpld data paths from ./data/ to docker/telemetry/data/ so runtime
files stay within the docker telemetry directory. Add .gitignore to
exclude the data directory from version control.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… Tempo API

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… span (Task 4.8)

Add ledger hash and full-validation flag to peer.validation.receive
spans for trace-level agreement analysis across validators.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The xrpld/app/misc/CanonicalTXSet.h header doesn't exist — it was
incorrectly added during a rebase conflict resolution. The correct
include xrpl/ledger/CanonicalTXSet.h is already present.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from 58aa308 to 87ed778 Compare April 1, 2026 10:02
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5b-spans branch from 4f8197e to cfa57cd Compare April 1, 2026 10:02
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase5-docs-deployment branch from 87ed778 to 360ecbd Compare April 24, 2026 22:17
@github-actions

Copy link
Copy Markdown

This PR has conflicts, please resolve them in order for the PR to be reviewed.

@pratikmankawde

Copy link
Copy Markdown
Contributor Author

Phase 5b content has been absorbed into Phase 6 (#6439) during the rebase of the PR chain onto the updated Phase 3 base. All spans (ledger.build, tx.apply, ledger.store, ledger.validate, peer.proposal.receive, peer.validation.receive), dashboards (ledger-operations, peer-network), TESTING.md updates, and codecov exclusions are now part of the Phase 6 commit. Closing as merged-by-absorption.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant