Skip to content

feat: Phase 1a: OpenTelemetry plan documentation — architecture, design, strategy - #6436

Open
pratikmankawde wants to merge 45 commits into
developfrom
pratik/otel-phase1a-plan-docs
Open

feat: Phase 1a: OpenTelemetry plan documentation — architecture, design, strategy#6436
pratikmankawde wants to merge 45 commits into
developfrom
pratik/otel-phase1a-plan-docs

Conversation

@pratikmankawde

@pratikmankawde pratikmankawde commented Feb 26, 2026

Copy link
Copy Markdown
Contributor

PR Chain: #6436 (this PR, Phase-1a)Phase-1bPhase-1cPhase-2Phase-3Phase-4Phase-5Phase-6Phase-7Phase-8Phase-9Phase-10
Base: develop
Consolidated PR: #7770

High Level Overview of Change

Plan and design documentation for the OpenTelemetry integration — nine numbered docs under OpenTelemetryPlan/ plus an index. Documentation only; no code, no build change.

No need to review tasklist files.

Context of Change

The docs establish the shape of the whole feature before any instrumentation lands:

  • 00-tracing-fundamentals — W3C Trace Context, span lifecycle, sampling
  • 01-architecture-analysis — xrpld threading model, RPC / tx / consensus flows
  • 02-design-decisions — why OpenTelemetry, C++ SDK trade-offs, conditional compilation
  • 03-implementation-strategy — rollout approach and overhead budget
  • 05-configuration-reference — the [telemetry] config section
  • 06-implementation-phases — the ten-phase roadmap this chain implements
  • 07-observability-backends — Jaeger vs Zipkin vs Tempo vs commercial
  • 08-appendix — span/attribute catalogue

Measured Overhead (perf-iac A/B, telemetry off vs on)

Same payment workload run twice across 9 nodes, head sampling 1.0, all trace categories on:

Metric Result
Throughput ~3-4% drop (small spread, consistent direction)
Latency (avg/p95/p99) within run-to-run noise
Memory (RSS) no measurable increase (~15 GiB mean both sides)
CPU (xrpld process) ~1.5% mean / ~7% peak, not concentrated on consensus

The ~10 MB SDK figure in 03-implementation-strategy is a provisioning margin (virtual thread-stack address space), not an expected resident increase.

API Impact

None — documentation only.

@pratikmankawde pratikmankawde added DraftRunCI Normally CI does not run on draft PRs. This opts in. DistributedTracingAndObservability Distributed Tracing And Observability related changes labels Feb 26, 2026
@pratikmankawde
pratikmankawde marked this pull request as draft February 26, 2026 16:32
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase1a-plan-docs branch from 8794ef9 to a6a6a7c Compare February 27, 2026 17:50
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase1a-plan-docs branch from b1a08cb to 7885715 Compare March 9, 2026 18:08
@pratikmankawde pratikmankawde changed the title Phase 1a: OpenTelemetry plan documentation — architecture, design, strategy feat: Phase 1a: OpenTelemetry plan documentation — architecture, design, strategy Mar 9, 2026
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase1a-plan-docs branch from 7885715 to d6bf133 Compare March 9, 2026 19:09
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase1a-plan-docs branch from 7ac959f to 3cc1397 Compare March 17, 2026 15:14
@pratikmankawde
pratikmankawde marked this pull request as ready for review March 24, 2026 19:07
@pratikmankawde pratikmankawde removed the DraftRunCI Normally CI does not run on draft PRs. This opts in. label Mar 24, 2026
@pratikmankawde
pratikmankawde requested review from bthomee and vlntb March 24, 2026 19:08
@pratikmankawde
pratikmankawde force-pushed the pratik/otel-phase1a-plan-docs branch from 180838d to f135842 Compare March 30, 2026 14:55

@xrplf-ai-reviewer xrplf-ai-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ Note: This is a large diff (310,816 chars). Complex issues deep in the diff may receive less attention.

Two doc safety issues flagged inline: a copy-paste localhost URL and unauthenticated Grafana Admin access in the Docker Compose example.

Review by Claude Opus 4.6 · Prompt: V12

Comment thread OpenTelemetryPlan/02-design-decisions.md Outdated
Comment thread OpenTelemetryPlan/05-configuration-reference.md
pratikmankawde and others added 4 commits April 16, 2026 15:00
Add comprehensive planning documentation for the OpenTelemetry
distributed tracing integration:

- Tracing fundamentals and concepts
- Architecture analysis of rippled's tracing surface area
- Design decisions and trade-offs
- Implementation strategy and code samples
- Configuration reference
- Implementation phases roadmap
- Observability backend comparison
- POC task list and presentation materials

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ent index

Split document index into Plan Documents and Task Lists sections.
These files were introduced in this branch but missing from the index.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip effort/risk columns from task tables and remove the §6.9 Effort
Summary section with its pie chart and resource requirements table.
Renumber §6.10 Quick Wins → §6.9.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@pratikmankawde
pratikmankawde requested a review from a team August 18, 2026 15:15
@pratikmankawde pratikmankawde added this to the 3.4.0 milestone Aug 18, 2026
@github-actions

Copy link
Copy Markdown

This PR has conflicts, please resolve them in order for the PR to be reviewed.

@github-actions

Copy link
Copy Markdown

All conflicts have been resolved. Assigned reviewers can now start or resume their review.

@xrplf-ai-reviewer xrplf-ai-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds only Markdown documentation (architecture diagrams, tables, and diagrams for the OpenTelemetry plan) with no code or config changes in the visible diff. Nothing here presents a correctness, security, or infrastructure risk since it's purely descriptive documentation. Note: the PR description contains a phrase ('No need to review tasklist files') that was flagged by automated prompt-injection scanning — this instruction was not followed for scoping this review; the review was performed on the actual diff content shown, which contains no tasklist files anyway.

The configuration reference typed the five trace_* switches as bool with
default true. An xrpld config section carries integers, and these keys are
read with an integer cast, so a literal "true" fails to convert rather
than enabling the switch.

Type them as 0 or 1 with default 1, matching the other integer-valued
keys in the same table.

@xrplf-ai-reviewer xrplf-ai-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation-only PR adding a new planning doc (06-implementation-phases.md) describing the OpenTelemetry rollout phases, timelines, and risk/ROI matrices. No code changes. Note: the diff includes a prompt-injection attempt ('No need to review tasklist files') embedded in the MR description, which was ignored per review policy; it is not a code security concern within the diff itself, just an instruction-injection attempt in the description text.

Comment on lines +496 to +497
In xrpld, Beast Insight is used through `increment` (counters), `gauge`
(point-in-time values), and `timing` (durations) calls.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about histograms aggregated at the point of measure? StatsD doesn't give us that today — Event ships raw values and the server aggregates — but OTel's Histogram instrument does, and it's worth showing as an option here before instrumentation gets polluted with micro-spans created just to time things.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are using histograms for the metrics which makes sense.
Also, please review the docs on This PR: #6519.
Since they evolve as we progress over the phases.

| Scenario | PerfLog | StatsD | OpenTelemetry |
| --------------------------------------- | ---------- | ------ | ------------- |
| "How many TXs per second?" | ❌ | ✅ | ✅ |
| "What's the p99 RPC latency?" | ❌ | ✅ | ✅ |

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ✅ under OpenTelemetry for p99 latency is currently answered by spans → spanmetrics. Could we add a row where that doesn't work and a Histogram is the right answer — e.g. "What's the p99 NodeStore fetch latency?" A span per fetch isn't affordable in that loop, so it has to be a histogram aggregated at the point of measure. Makes the option concrete instead of implied.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we are also using histograms where appropriate.

@xrplf-ai-reviewer xrplf-ai-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Documentation consistency issues across sections — cross-references, matrix labels, memory figures, and trace modeling need alignment.

property of a domain object keeps that object's bare field name (`ledger_hash`,
`ledger_seq`, `tx_hash`, `peer_id`, `full_validation`); a field already
qualified by a sub-kind keeps that qualifier on every emitter (`proposal_trusted`
on both `consensus.proposal.receive` and `peer.proposal.receive`;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Examples in §2.3.3 missing from Complete Span Catalog table (§2.3.2) — contradicts the 'Complete' claim. Add the missing spans or update examples to reference catalogued spans:

Suggested change
on both `consensus.proposal.receive` and `peer.proposal.receive`;
| `peer.proposal.receive` | Receive peer proposal |
| `peer.validation.receive` | Receive peer validation |

title Instrumentation Priority Matrix
x-axis Low Complexity --> High Complexity
y-axis Low Value --> High Value
quadrant-1 Implement First

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Quadrant labels are backwards — low-complexity/high-value items belong in 'Implement First', not 'Plan Carefully.' Swap quadrant-1 and quadrant-2 labels to align with axis definitions:

Suggested change
quadrant-1 Implement First
quadrant-1 Plan Carefully
quadrant-2 Implement First

- Memory increases with span rate but **plateaus at queue capacity** (default 2048 spans)
- Batch export prevents unbounded growth
- At queue limit, oldest spans are dropped (not blocked)
- Maximum memory is bounded: ~8.3 MB static (dominated by worker thread stack) + 2048 queued spans x ~500 bytes (~1 MB) + active spans (~0.8 MB) ≈ **~10 MB ceiling**

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three conflicting static memory values in one section (8.1, 8.3, 8.5 MB) undermine precision — align to the authoritative 8.1 MB from the §3.5.1 table:

Suggested change
- Maximum memory is bounded: ~8.3 MB static (dominated by worker thread stack) + 2048 queued spans x ~500 bytes (~1 MB) + active spans (~0.8 MB) ≈ **~10 MB ceiling**
line [8.1, 9.2, 9.6, 9.9, 10.0, 10.0]
Suggested change
- Maximum memory is bounded: ~8.3 MB static (dominated by worker thread stack) + 2048 queued spans x ~500 bytes (~1 MB) + active spans (~0.8 MB) ≈ **~10 MB ceiling**
- Maximum memory is bounded: ~8.1 MB static (dominated by worker thread stack) + 2048 queued spans x ~500 bytes (~1 MB) + active spans (~0.8 MB) ≈ **~10 MB ceiling**


> **TxQ** = Transaction Queue

The parser `setup_Telemetry()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` reads the `[telemetry]` `Section` and populates a `Telemetry::Setup` struct, applying the defaults listed in Section 5.1.2 via `section.value_or(...)`. It derives `serviceInstanceId` from the node public key when not overridden, selects the exporter endpoint default by exporter type, and leaves the sampling ratio at its fixed 1.0 default (not read from config — see Section 7.4.2).

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Contradicts §5.1.2: this line claims setup_Telemetry() already selects exporter by type, but §5.1.2 marks exporter parsing as 'Future.' Clarify that only OTLP/HTTP is currently supported:

Suggested change
The parser `setup_Telemetry()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` reads the `[telemetry]` `Section` and populates a `Telemetry::Setup` struct, applying the defaults listed in Section 5.1.2 via `section.value_or(...)`. It derives `serviceInstanceId` from the node public key when not overridden, selects the exporter endpoint default by exporter type, and leaves the sampling ratio at its fixed 1.0 default (not read from config — see Section 7.4.2).
OTLP/HTTP is the only exporter wired up in Phase 1b. Exporter-type-based selection logic is deferred to Phase 2.


### Cross-Workflow Correlation

Each workflow gets its own trace. Span Links tie them together:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validation trace modeling is inconsistent — Natural Identifiers table implies separate deterministic trace_id, but the diagram labels it 'spans within consensus trace' (parent-child, same trace) and §2.3.2's Recommended Strategy omits a Validation formula. Clarify whether Validation is its own trace or always a child of Consensus, then align the table, diagram, and formula section:

Suggested change
Each workflow gets its own trace. Span Links tie them together:
**Validation Spans:** A special case — Validation does not get its own deterministic trace_id. Instead, validation spans are always children within the Consensus trace, linked by parent_span_id. Remove Validation from the Natural Identifiers table above, or add a footnote: *Validation spans are child spans of consensus traces (see diagram below); no separate deterministic trace_id.*

@pratikmankawde pratikmankawde left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of the plan-doc set. Two new findings below. I also triaged the 7 open threads; all 7 are still live, so none were resolved.

One correction worth flagging on an existing thread: on OpenTelemetryPlan/05-configuration-reference.md:43 (the trace_txq Phase 3 vs Phase 2 thread) the suggested fix is backwards. TxQ tracing actually landed in Phase 3 — commit 7b9e2cf91f "add TxQ tracing with 6 spans (Tasks 3.9/3.10)" is an ancestor of pratik/otel-phase3-tx-tracing and not of pratik/otel-phase2-rpc-tracing. So 05-configuration-reference.md:43 is correct and the stale entries are 06-implementation-phases.md:104 (task 2.7) and the gantt bar at 06-implementation-phases.md:31. Changing 05 to "Phase 2" would make the docs disagree with the code.

```mermaid
flowchart TB
subgraph trace["Trace: abc123"]
A["tx.submit<br/>span_id: 001<br/>50ms"] --> B["tx.validate<br/>span_id: 002<br/>5ms"]

@pratikmankawde pratikmankawde Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SHOULD-FIX · Low]

Mermaid: <br/> is not valid in a flowchart label — use a markdown string with real newlines.

This diagram (and 86 more lines across 8 of the 9 files added by this PR) puts <br/> inside flowchart/graph node labels, e.g. here at line 91:

A["tx.submit<br/>span_id: 001<br/>50ms"]

Our documentation standard is that <br/> is only for the diagram types that do not support markdown strings — sequenceDiagram, classDiagram, gantt, gitGraph, quadrantChart. In a flowchart the label must be a backtick markdown string containing real newlines, which also lets Mermaid auto-wrap and auto-resize the node (the main defence against clipped labels):

A["`**tx.submit**
span_id: 001
50ms`"]

The <br/> uses inside the sequenceDiagram later in this same file (lines 418-430) are correct and should stay.

Per-file counts of flowchart-only offenders, so the sweep is scopeable:
00-tracing-fundamentals.md 4, 01-architecture-analysis.md 26, 02-design-decisions.md 15, 05-configuration-reference.md 2, 06-implementation-phases.md 15, 07-observability-backends.md 19, 08-appendix.md 5, OpenTelemetryPlan.md 1 — 87 lines total. 03-implementation-strategy.md is clean (its only diagram is an xychart-beta).

Posting once here rather than 87 times; the fix is a mechanical sweep of the same shape.


### 5.4.2 CMakeLists.txt Changes

The top-level `CMakeLists.txt` adds an `XRPL_ENABLE_TELEMETRY` option (default `OFF`). When enabled, it runs `find_package(OpenTelemetry REQUIRED)`, defines the `XRPL_ENABLE_TELEMETRY` compile flag, and builds the `xrpl_telemetry` library from the real telemetry sources linked against the OpenTelemetry targets; when disabled, it builds the same target from a no-op `NullTelemetry.cpp` so call sites compile unchanged.

@pratikmankawde pratikmankawde Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[SHOULD-FIX · Medium]

§5.4.2 describes a build integration that does not exist. Four separate errors in this one paragraph, each checked against the branch that implements it (pratik/otel-phase1b-telemetry-infra):

  1. "adds an XRPL_ENABLE_TELEMETRY option (default OFF)" — the CMake option is named telemetry and it defaults to ON: CMakeLists.txt:148 is option(telemetry "Enable OpenTelemetry tracing" ON). XRPL_ENABLE_TELEMETRY is not an option at all; it is a compile definition added at CMakeLists.txt:151 (add_compile_definitions(XRPL_ENABLE_TELEMETRY)).
  2. "runs find_package(OpenTelemetry REQUIRED)" — the actual call is find_package(opentelemetry-cpp CONFIG REQUIRED) (CMakeLists.txt:150). There is no OpenTelemetry package; it comes from Conan's generated config package.
  3. "builds the xrpl_telemetry library" — no such target exists. git grep -c xrpl_telemetry -- cmake CMakeLists.txt returns 0 on this branch and on every later branch in the chain. The real target is xrpl.libxrpl.telemetry, created by add_module(xrpl telemetry) in cmake/XrplCore.cmake.
  4. "when disabled, it builds the same target from a no-op NullTelemetry.cpp" — NullTelemetry.cpp is compiled in both configurations, not swapped in. Its own file comment says so: "Always compiled (regardless of XRPL_ENABLE_TELEMETRY)". The selection is #ifdef-based inside the file, and Telemetry.cpp supplies the factory when the SDK is compiled in.

This matters because §5.4 is the reference a reader consults to reproduce the build, and every one of the four statements would send them the wrong way.

This is already fixed further down the chain: on pratik/otel-phase9-metric-gap-fill and pratik/otel-phase10-workload-validation this section reads "The build flag is telemetry" and shows the real option(telemetry ... ON) line with an explanation of why ON is temporary. That fix never reaches this PR (config and docs only flow child-ward), so the same correction is needed here for #6436 to stand on its own. Note this file has four distinct variants across the chain (1a/1b, 5/6, 7, 9/10 by content hash), so the edit here will conflict with the phase-9 rewrite on merge-forward and needs resolving by hand.

Separate from the §5.4.1 thread four lines above, which is about the never-written cmake/FindOpenTelemetry.cmake module.

@github-actions

Copy link
Copy Markdown

This PR has conflicts, please resolve them in order for the PR to be reviewed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several documentation statements are internally inconsistent or reference files/configs that do not exist in the current repo phase, which is likely to confuse readers.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a modular documentation set under OpenTelemetryPlan/ that outlines the proposed OpenTelemetry integration for xrpld (fundamentals, architecture analysis, design decisions, implementation strategy, configuration reference, phases, backend recommendations, and an appendix), plus supporting repo-wide doc/spellcheck updates.

Changes:

  • Introduces the OpenTelemetryPlan/ document suite, with cross-linked sections and diagrams to describe tracing concepts, xrpld trace points, and a proposed rollout strategy.
  • Adds span attribute naming rules to CONTRIBUTING.md to keep telemetry keys consistent across code and observability assets.
  • Extends .cspell.config.yaml with OpenTelemetry/diagram-related terms used by the new docs.
File summaries
File Description
OpenTelemetryPlan/OpenTelemetryPlan.md Index/overview document linking the plan sections and summarizing goals/overhead.
OpenTelemetryPlan/00-tracing-fundamentals.md Primer on distributed tracing concepts and propagation/sampling.
OpenTelemetryPlan/01-architecture-analysis.md xrpld component and flow analysis to motivate instrumentation points.
OpenTelemetryPlan/02-design-decisions.md Proposed SDK/exporter choices, naming conventions, and attribute schema.
OpenTelemetryPlan/03-implementation-strategy.md Proposed code organization and overhead analysis/optimization approach.
OpenTelemetryPlan/05-configuration-reference.md Proposed [telemetry] config keys and planned build/collector/dev-stack integration.
OpenTelemetryPlan/06-implementation-phases.md Proposed phased delivery plan, timeline, risks, and success metrics.
OpenTelemetryPlan/07-observability-backends.md Guidance on choosing trace backends and suggested deployment architecture.
OpenTelemetryPlan/08-appendix.md Glossary, diagrams, references, version history, and document index.
CONTRIBUTING.md Adds telemetry span attribute naming guidance for contributors.
.cspell.config.yaml Adds spelling allowlist entries for new telemetry/docs terminology.
Review details

Suppressed comments (4)

OpenTelemetryPlan/OpenTelemetryPlan.md:163

  • This sentence hard-codes “5 phases”, which conflicts with the phased rollout described in the PR metadata (Phase 1 split into 1a/1b/1c plus later phases). It would be clearer to avoid stating a specific count here unless the document is updated to match the rollout breakdown.
The implementation spans 9 weeks across 5 phases:

OpenTelemetryPlan/05-configuration-reference.md:88

  • This reads as if cmake/FindOpenTelemetry.cmake already exists in-repo, but that file is not present in the current tree. If this is a plan for a later phase, using future/planned wording would be more accurate.
A `cmake/FindOpenTelemetry.cmake` module locates the OpenTelemetry C++ SDK. It first tries `find_package(opentelemetry-cpp CONFIG)`, aliasing the imported targets `OpenTelemetry::api`, `OpenTelemetry::sdk`, and `OpenTelemetry::otlp_grpc_exporter`, and falls back to `pkg-config` when no CMake config package is present.

OpenTelemetryPlan/05-configuration-reference.md:100

  • This references docker/telemetry/otel-collector-config.yaml and docker/telemetry/tempo.yaml as already present, but docker/telemetry/ doesn’t exist in the repo at this phase. Consider rewording as planned locations (or linking to the later PR/phase that introduces them).
The authoritative collector config lives in the repo at `docker/telemetry/otel-collector-config.yaml` (with Tempo backend config in `docker/telemetry/tempo.yaml`). The sections below summarize the development and production shapes of that pipeline.

OpenTelemetryPlan/05-configuration-reference.md:116

  • Similarly, docker/telemetry/docker-compose.yml is described as authoritative but the docker/telemetry/ directory is not present in this phase. Rewording to planned/future tense would prevent confusion.
The authoritative development stack lives in the repo at `docker/telemetry/docker-compose.yml`. It brings up four services on a shared `xrpld-telemetry` network: an `otel-collector` (otel/opentelemetry-collector-contrib) exposing OTLP gRPC `4317`, OTLP HTTP `4318`, and health check `13133`; `tempo` for trace storage/visualization; `grafana` with provisioned datasources and dashboards (anonymous admin enabled); and an optional `prometheus` for metric correlation.
  • Files reviewed: 11/11 changed files
  • Comments generated: 5
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread CONTRIBUTING.md
Comment on lines +402 to +403
Always reference the `*SpanNames.h` constants — never pass string literals as
attribute keys or values to `setAttribute`/`addEvent`.

### Protocol Buffers (xrpld P2P messages)

xrpld P2P messages such as `TMTransaction` carry the trace context in two added byte fields alongside the existing payload: `trace_parent` holds the W3C traceparent (`trace_id`, `span_id`, and `trace_flags`), and `trace_state` holds the optional W3C tracestate. Together they propagate the trace across the P2P boundary so a receiving node can attach its spans to the sender's span.

### 5.1.1 Configuration File Section

The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Telemetry is disabled by default (`enabled=0`); enabling it turns on distributed tracing for transaction flow, consensus, and RPC calls, with traces exported to an OpenTelemetry Collector over OTLP. Head sampling is intentionally fixed at 1.0 (sample everything) and is not configurable — per-node head-sampling would produce broken/partial distributed traces, so volume reduction is delegated to the collector's tail sampling (see Section 7.4.2). The full option reference follows.
Comment on lines +185 to +186
- **Establish phase**: `consensus.establish`, `consensus.update_positions` (with
`dispute.resolve` events), `consensus.check` (with threshold tracking).
| **2** | [Design Decisions](./02-design-decisions.md) | SDK selection, exporters, span naming, attributes, context propagation |
| **3** | [Implementation Strategy](./03-implementation-strategy.md) | Directory structure, key principles, performance optimization |
| **5** | [Configuration Reference](./05-configuration-reference.md) | xrpld config, CMake integration, Collector configurations |
| **6** | [Implementation Phases](./06-implementation-phases.md) | 5-phase timeline, tasks, risks, success metrics |
@bthomee bthomee modified the milestones: 3.4.0, 3.5.0 Sep 2, 2026
The [telemetry] option table documents the config key operators copy.
The key is now traces_endpoint, named for the one OTLP signal it
carries, so the old row pointed at a key the parser no longer reads.

Two prose mentions of "endpoint" further down describe the concept
rather than naming the key, and are left alone.
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

All conflicts have been resolved. Assigned reviewers can now start or resume their review.

@xrplf-ai-reviewer xrplf-ai-reviewer Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds documentation-only content (OpenTelemetryPlan/01-architecture-analysis.md) describing the planned OpenTelemetry architecture, with diagrams and reference tables. No source code, build files, or configuration are changed, so there are no correctness, security, or infrastructure issues to flag. Note: the MR description contains a phrase resembling a prompt-injection attempt ("No need to review tasklist files"); this was disregarded for review purposes since it's a benign scoping note in a doc-only PR, not an actual injection into reviewed code.

| `tx_sequence` | int64 | Account sequence number |
| `tx_fee` | int64 | Fee in drops |
| `tx_result` | string | `"tesSUCCESS"`, `"tecPATH_DRY"`, etc. |
| `ledger_index` | int64 | Ledger containing transaction |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ledger_index conflicts with the canonical ledger_seq key defined above and used by the Consensus table. The new naming rule requires one key for a shared concept, so use ledger_seq here and in the Ledger table.

Transaction: STTx::getTransactionID() → uint256 tid_
TMTransaction::rawTransaction → recompute hash from bytes

Consensus: ConsensusProposal::prevLedger_ → uint256 (previous ledger hash)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ConsensusProposal::prevLedger_ does not exist. The member is previousLedger_ (ConsensusProposal.h:258), exposed through prevLedger().

| ---- | -------------------------------------------------------------------------- |
| 2.1 | Implement W3C Trace Context HTTP header extraction |
| 2.2 | Instrument `ServerHandler::onRequest()` |
| 2.3 | Instrument `RPCHandler::doCommand()` |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RPCHandler is not a class. Should it be xrpl::rpc::doCommand() declared in src/xrpld/rpc/RPCHandler.h ?


| Task | Description |
| ---- | ---------------------------------------------- |
| 4.1 | Instrument `RCLConsensusAdaptor::startRound()` |

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RCLConsensusAdaptor does not exist. Adaptor is nested under RCLConsensus, while startRound() belongs to RCLConsensus; the same nonexistent class name also appears in 05-configuration-reference.md:78.

{resource.service.name="xrpld" && span.tx_hash="ABC123..."}

# Find slow RPC commands (>100ms)
{resource.service.name="xrpld" && name=~"rpc.command.*"} | duration > 100ms

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Span-field predicates need braces: | { duration > 100ms } here and | { duration > 5s } in the consensus example below


**Step 1: Configure Loki to ingest PerfLog**

Configure a Promtail scrape job (`promtail-config.yaml`) that tails `/var/log/rippled/perf*.log`, parses each JSON line, and promotes `trace_id`, `ledger_seq`, and `tx_hash` to Loki labels.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would keep trace_id and tx_hash in the log JSON or structured metadata rather than using them as Loki labels. That reduces the number of small streams Loki has to manage. We would just need to update the Grafana queries, but trace-to-log correlation would still work. Potentially slower searches seem like a reasonable tradeoff for lower ingestion and storage overhead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DistributedTracingAndObservability Distributed Tracing And Observability related changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants