feat: Phase 6: Integrate beast::insight StatsD metrics into telemetry pipeline

[pratikmankawde]

PR Chain: Phase-1a → Phase-1b → Phase-1c → Phase-2 → Phase-3 → Phase-4 → Phase-5 → #6439 (this PR, Phase-6) → Phase-7 → Phase-8 → Phase-9 → Phase-10
Base: pratik/otel-phase5-docs-deployment
Consolidated PR: #7770

High Level Overview of Change

Bridges xrpld's existing beast::insight StatsD metrics into the telemetry pipeline, adds the first ten Grafana dashboards, fills the empty ledger/peer span categories, and extends the integration test to cover metrics.

No need to review tasklist files.

Context of Change

StatsD → Collector → Prometheus — the Collector gains a statsd receiver on UDP 8125 with timer_histogram_mapping, and the metrics pipeline becomes receivers: [spanmetrics, statsd] behind a single Prometheus endpoint on :8889. Node configs set [insight] server=statsd. This exposes the ~300 metrics xrpld already computes without touching any call site.

Gauge fix — StatsDGaugeImpl started with dirty_{false}, so a gauge whose value never moved off 0 was never flushed and never appeared in Prometheus. It now starts dirty, so the initial value is emitted on the first flush.

Spans added — fills the previously empty trace_ledger category and extends peer tracing:

• ledger.build, ledger.validate, ledger.store (BuildLedger.cpp, LedgerMaster.cpp)
• tx.apply — batch transaction application per ledger, attrs tx_count / tx_failed
• peer.proposal.receive, peer.validation.receive (off by default)

Dashboards (10, all new) — consensus-health, ledger-operations, peer-network, rpc-performance, transaction-overview (span-derived) and statsd-node-health, statsd-network-traffic, statsd-rpc-pathfinding, statsd-ledger-data-sync, statsd-overlay-traffic-detail (StatsD-derived), with validate_dashboards.py as a structural gate and provisioned Prometheus/Tempo datasources.

Docs — OpenTelemetryPlan/09-data-collection-reference.md is introduced here as the signal inventory; integration-test.sh verifies StatsD metrics reach Prometheus.

Known limitation — Resource Manager warn / drop use the non-standard |m StatsD type, which the Collector's receiver silently drops. Changing it to |c is a wire-format change for existing StatsD consumers and is deferred.

API Impact

• Public API changes
• No API change (internal instrumentation + ops assets)

Test Plan

• python3 -m json.tool on every dashboard, plus validate_dashboards.py.
• docker compose up exposes 8125/udp; StatsD metrics appear in Prometheus once [insight] is set.
• ledger.build / ledger.validate / ledger.store / tx.apply visible in Tempo.
• integration-test.sh passes its metric step.

[codecov]

Codecov Report

❌ Patch coverage is 24.61538% with 49 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/xrpld/app/ledger/detail/LedgerMaster.cpp	8.8%	31 Missing ⚠️
src/xrpld/overlay/detail/PeerImp.cpp	0.0%	18 Missing ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

This PR has conflicts, please resolve them in order for the PR to be reviewed.

[github-actions]

All conflicts have been resolved. Assigned reviewers can now start or resume their review.

[xrplf-ai-reviewer]

Unit scaling mismatch in dashboard and missing span in documentation inventory — see inline.

[xrplf-ai-reviewer]

Operating Mode Duration unit mismatch — divide metric by 1000000 (microseconds to seconds):

Suggested change

	"expr": "rippled_State_Accounting_Full_duration{exported_instance=~\"$node\"}",
	// Divide by 1000000 to convert microseconds to seconds:
	rippled_State_Accounting_Full_duration / 1000000

Suggested change

	"expr": "rippled_State_Accounting_Full_duration{exported_instance=~\"$node\"}",
	"expr": "rippled_State_Accounting_Full_duration{exported_instance=~\"$node\"} / 1000000",

[xrplf-ai-reviewer]

Documentation accuracy issues: telemetry attributes swapped between span rows, phase-9 implementation tense inconsistent. See inline comments.

[xrplf-ai-reviewer]

Present tense contradicts the immediately-following caveat about phase-9. Change 'are created'/'assigned' to future tense so the sentence doesn't mislead.

Suggested change

	For each of the same 35 non-special job types, three gauges are created in the `JobTypeData`
	For each of the same 35 non-special job types, three gauges will be created in the `JobTypeData`

[pratikmankawde]

Review summary

3 BLOCKING · 2 SHOULD-FIX · 0 NIT posted inline, plus the notes below.

Scope of this pass: the ten dashboards, validate_dashboards.py, the Grafana provisioning, the collector config, docker-compose.yml, the StatsD C++ change, and the ledger/peer span additions. I read every file with git show pratik/otel-phase6-statsd:<path> at 6283a1eb12.

How the runtime evidence was obtained. For anything that depends on what actually reaches Prometheus, I ran this PR's own otel-collector-config.yaml in the pinned otel/opentelemetry-collector-contrib:0.158.0 alongside prom/prometheus:v3.13.2 with this PR's own prometheus.yml, on throwaway ports, and fed it real OTLP spans and StatsD datagrams in the exact format StatsDCollector emits. My collector config differed from the committed one by exactly one line, in the traces exporter list (exporters: [debug, otlp/tempo, spanmetrics] -> [spanmetrics], because there was no Tempo container), which cannot affect metric naming or labels. Containers were removed afterwards. Two useful side results: the committed config starts cleanly on the pinned image, and the StatsD receiver is wired (otel-collector-config.yaml:22 defines it, :149 is receivers: [spanmetrics, statsd]) — so any claim that this branch has no StatsD receiver is wrong.

Already raised in threads that were resolved without the fix landing

Not re-posting these, but they are all still live at this tip and I verified each independently:

1. The metric-name prefix still does not match, so all five StatsD dashboards return exactly zero series. docker/telemetry/xrpld-telemetry.cfg:67 is prefix=xrpld; docker/telemetry/integration-test.sh:347 writes prefix=rippled and its assertions at :590-605 check rippled_*; and the five dashboards contain 167 rippled_* references and zero xrpld_*. The prefix really does become the metric-name prefix — StatsDCollector.cpp:524, 565, 613, 688 all emit impl_->prefix() << "." << name_ — and I confirmed the whole path live: xrpld.total.Bytes_In:12345|g arrives as xrpld_total_Bytes_In. The fix direction is forced, and three of the resolved threads propose the wrong one. xrpld is the only committable value in a .cfg: .github/scripts/rename/docs.sh:33 globs *.cfg and :39 applies s@([^/+-])rippled@\1xrpld@g, and .github/workflows/reusable-check-rename.yml:37 runs that script with :49 failing the job on a dirty tree. I proved the rewrite by running that glob and sed over throwaway .cfg/.sh/.json files: the .cfg line prefix=rippled became prefix=xrpld while the .sh and .json copies were untouched — which is also why the legacy name survived in the script and the dashboards in the first place. So the script and dashboards must move to xrpld_*. Thread PRRT_kwDOACmRR86TlukF proposed exactly that and was resolved without being applied; PRRT_kwDOACmRR86TjAo7, PRRT_kwDOACmRR86XGo6r and PRRT_kwDOACmRR86ZbdYl propose changing the cfg to rippled, which check-rename would revert. The still-open thread on OpenTelemetryPlan/09-data-collection-reference.md:481 is the live tracker; note its neighbouring text at :483 asserts prefix=xrpld is "matching the integration test and Grafana dashboards", which is the part that is factually wrong today.

2. The job-queue metrics are all missing the jobq_ group segment (thread PRRT_kwDOACmRR86TjAo2, resolved). src/xrpld/app/main/Application.cpp:375 passes collectorManager_->group("jobq") and src/libxrpl/beast/insight/Groups.cpp:44 is return name_ + "." + name;, so the group comes first. Confirmed live: xrpld.jobq.job_count:4|g arrives as xrpld_jobq_job_count. The dashboards ask for rippled_job_count and the per-job-type names without the segment, across "Job Queue Depth", "Key Jobs Execution Time", "Key Jobs Dequeue Wait Time" and the two __name__=~ detail panels. Because these five files exist on phase-6 only, "it is fixed downstream" is not available here — the downstream boards are different files.

3. Resource Warnings Rate and Resource Drops Rate can never plot (thread PRRT_kwDOACmRR86CFc1s, resolved). include/xrpl/resource/detail/Logic.h creates these with makeMeter, and StatsDCollector.cpp:688 emits meters as |m, which the OTel StatsD receiver does not implement. I sent xrpld.warn:1|m and xrpld.drop:1|m and got zero series and, notably, zero diagnostics in the collector log. Two stat tiles that will read a reassuring flat zero forever.

4. The PR adds a linter and 558 violations of it in the same commit (also raised in PRRT_kwDOACmRR86TjAo2). Running the committed validate_dashboards.py unmodified against the committed dashboards gives exit code 1 and FAIL: 558 violation(s) — 149 in ledger-data-sync, 140 in overlay-traffic-detail, 138 in node-health, 86 in network-traffic, 45 in rpc-pathfinding, and 0 in the five span-derived boards. By category: 486 missing tier template variables, 72 cumulative-raw. It is not wired into CI on this branch (grep -rn validate_dashboards .github/ finds nothing), so the failure is latent — but it leaves reviewers no baseline.

5. The three raw-counter traffic panels (statsd-network-traffic.json:210, :263, :323) are still unwrapped; I left the line-210 thread open as the single tracker and resolved the other two as duplicates. One correction for whoever fixes them: these are registered as beast insight gauges (OverlayImpl.h:583-586) whose value is a cumulative total copied from the monotonic atomics in TrafficCount.h:45,47 (OverlayImpl.h:640-643), so they carry counter semantics without a _total suffix — rate() is still the right function, and the absence of _total is correct rather than a bug. I verified the _total rule live: |c counters get it (xrpld.rpc_requests:7|c -> xrpld_rpc_requests_total), |g gauges do not.

Two convention items, reported here rather than as inline comments

• Every panel is gridPos.h: 8; the project convention is 10. All 111 panels across all ten dashboards: timeseries 82, stat 14, bargauge 4, heatmap 4, piechart 4, barchart 1, gauge 1, state-timeline 1 — every one at height 8. .claude/instructions.md fixes these at 10 (12 for table/logs, of which there are none here). This is already fixed for the span-derived boards on phase-7, so only phase-6 and the five phase-6-only StatsD boards need it.
• No panel or target declares a datasource uid — every panel is "datasource": null and every target is {"type": "prometheus"} or {"type": "tempo"}, while provisioning/datasources/prometheus.yaml gives Prometheus uid: prometheus and isDefault: true and tempo.yaml gives Tempo uid: tempo with no isDefault. The six TraceQL targets in consensus-health.json are the exposed ones. I am flagging this as an observation rather than a defect because I could not exercise Grafana's frontend datasource resolution, so the impact is unverified; what is verified is that no uid is present, that only Prometheus is the default, and that phase-9 rewrote exactly these targets to "uid": "${DS_PROMETHEUS}" / "${DS_TEMPO}".

[pratikmankawde]

[BLOCKING · High] — The $node dropdown is empty on all five span-derived dashboards, because target_info has no service_instance_id label.

This variable is "query": "label_values(target_info, service_instance_id)". I ran the PR's own collector config (otel/opentelemetry-collector-contrib:0.158.0, the tag docker-compose.yml:24 pins) plus prom/prometheus:v3.13.2 with the PR's own prometheus.yml, sent one OTLP span whose resource had service.instance.id="Node-1", and queried Prometheus:

target_info{deployment_environment="local", exported_instance="Node-1", exported_job="xrpld",
            instance="otel-collector:8889", job="otel-collector", service_version="2.4.0",
            xrpl_network_id="0", xrpl_network_type="mainnet"} 1

label_values(target_info, service_instance_id) -> []
label_values(target_info, exported_instance)   -> ['Node-1']

service.instance.id is translated to instance on target_info, and Prometheus renames it to exported_instance because prometheus.yml sets no honor_labels. So the label the variable asks for does not exist there and the dropdown lists nothing but "All".

The panel selectors are fine — service_instance_id genuinely does exist on the span metrics, because otel-collector-config.yaml:186-188 enables resource_to_telemetry_conversion. Same run:

traces_span_metrics_calls_total{..., service_instance_id="Node-1", span_name="tx.apply", ...}

So only the variable query is wrong. Nothing looks broken — panels still draw with $node falling back to .* — you simply can never isolate a node on Consensus Health, Transaction Overview, RPC Performance, Ledger Operations or Peer Network.

Same line in the other four: transaction-overview.json:929, rpc-performance.json:499, ledger-operations.json:398, peer-network.json:226.

Fix: label_values(traces_span_metrics_calls_total, service_instance_id), which is where the label actually lives. Not fixed downstream — git show pratik/otel-phase10-workload-validation:docker/telemetry/grafana/dashboards/consensus-health.json | grep -c 'label_values(target_info, service_instance_id)' still returns 1, so the same fix is needed there separately.

[pratikmankawde]

[BLOCKING · High] — Choosing a node blanks every panel on all five StatsD dashboards: exported_instance does not exist on StatsD series at all.

The StatsD receiver produces metrics with no service resource, so there is no service.instance.id for the exporter to map to instance, and nothing for Prometheus to rename. In the live run described in my comment on consensus-health.json:1081 I sent a datagram in the exact format StatsDCollector emits (src/libxrpl/beast/insight/StatsDCollector.cpp:613 is ss << impl_->prefix() << "." << name_ << ":" << value_ << "|g") and got the complete label set:

xrpld_Peer_Finder_Active_Inbound_Peers{
  deployment_environment="local", instance="otel-collector:8889", job="otel-collector",
  metric_type="gauge", otel_scope_name=".../receiver/statsdreceiver",
  otel_scope_version="0.158.0", xrpl_network_type="mainnet"} 5

No exported_instance, no service_instance_id. Queried both ways:

xrpld_..._Active_Inbound_Peers{exported_instance=~"Node-1"}  -> result count: 0
xrpld_..._Active_Inbound_Peers{exported_instance=~".*"}      -> result count: 1

This is worse than a filter that plainly does not work. label_values(exported_instance) here is unscoped, so on a live stack it returns the node names coming from the span pipeline and the dropdown looks populated — then the moment an operator picks one, all 162 targets across these five boards go to "No data". Affected target counts: node-health 46, ledger-data-sync 39, overlay-traffic-detail 36, network-traffic 26, rpc-pathfinding 15.

There is no per-node dimension on the StatsD path to filter on, so this cannot be fixed in the dashboard alone. Either add a collector-side processor that stamps a node identity onto StatsD metrics, or drop the node variable and the exported_instance=~"$node" clauses and state in the dashboard description that these boards are single-node.

Worth knowing for scoping: these five files exist on phase-6 only — git ls-tree -r --name-only <branch> -- docker/telemetry/grafana/dashboards/ | grep -c statsd- gives 5 here and 0 on phase-7, phase-9 and phase-10. So there is no downstream copy that inherits a fix, and equally no downstream copy that is already correct.

[pratikmankawde]

[BLOCKING · High] — This panel does not render a wrong number, it returns a Prometheus execution error.

Panel "All Traffic Categories (Detail)". rate() drops __name__, and StatsD series carry no per-category labels (see my comment on statsd-node-health.json:892 for the full label dump), so once the name is gone every traffic category collapses to the same label set and the result is rejected. Reproduced against the live stack with two categories present, with a control to isolate the cause:

topk(15, rate({__name__=~"xrpld_.*_Bytes_In", __name__!~"xrpld_total_.*"}[5m]))
  -> status=error  errorType=execution
     error="vector cannot contain metrics with the same labelset"

topk(15, {__name__=~"xrpld_.*_Bytes_In", __name__!~"xrpld_total_.*"})     <- same selector, no rate()
  -> status=success
       xrpld_proposals_Bytes_In     333
       xrpld_transactions_Bytes_In  222

The control shows the selector itself is fine; rate() is what breaks it. Two categories is not an edge case — src/xrpld/overlay/detail/TrafficCount.h defines 57 of them, so any real node hits this immediately.

legendFormat: "{{__name__}}" on this target would also render empty for the same reason, even if the query ran.

This is the only site: grep -nE '(rate|irate|increase)\(\{__name__' docker/telemetry/grafana/dashboards/*.json returns line 754 and nothing else.

Fix: aggregate by name so the label set stays unique — topk(15, sum by (__name__) (rate({...}[$__rate_interval]))).

[pratikmankawde]

[SHOULD-FIX · Medium] — This list is lowercase and the comparison is case-sensitive, so the linter is blind to the metric names this branch actually produces.

CUMULATIVE_PREFIXES here is all-lowercase ("total_bytes_", "transactions_messages_", …), STATE_DURATION at line 43 is re.compile(r"state_accounting_\w+_duration") with no re.I, and the test is a plain m in expr substring match. But StatsD metric names on this branch are case-preserved: src/libxrpl/beast/insight/StatsDCollector.cpp:613 passes the beast name through verbatim, and src/xrpld/overlay/detail/OverlayImpl.h:583-586 registers those names as "Bytes_In" / "Messages_In". I confirmed the end result live through the PR's own collector: xrpld.total.Bytes_In:12345|g arrives as xrpld_total_Bytes_In.

I mutation-tested the linter on copies in /tmp (the repo files were not touched), with a control:

rate() present, lowercase name   (control, should pass)      exit=0  cumulative-raw violations=0
raw + LOWERCASE name             (should be caught)          exit=1  cumulative-raw violations=1
raw + REAL casing Bytes_In       (same defect)               exit=0  cumulative-raw violations=0

Identical defect, only the case differs, and the real-world spelling slips through.

Why it matters concretely: this is why the linter does not flag the "Transaction Traffic" panel that the still-open thread on statsd-network-traffic.json:210 is about (rippled_transactions_Messages_In, plotted raw), nor "Peer Disconnects" (rippled_Overlay_Peer_Disconnects), nor "Operating Mode Duration". "Proposal Traffic" and "Validation Traffic" are caught only by luck, because proposals_ and validations_ happen to be lowercase inside the real names.

Fix: lower-case both sides — compare against expr.lower(), and add re.I to STATE_DURATION. Two lines. Note validate_dashboards.py has a second variant later in the chain (143 lines here, 196 on phase-9/10), so the same fix is needed there too.

[pratikmankawde]

[SHOULD-FIX · Low] — Five of the 22 display-name overrides on this panel target field names that can never exist, so the one panel meant to be human-readable is only partly readable.

Panel "Overlay Traffic by Category (Bytes In)". The authoritative category list is the Category -> string table in src/xrpld/overlay/detail/TrafficCount.h (57 names, starting at line 240 with {Category::Base, "overhead"}). I checked all 22 byName matchers against it programmatically; 17 match a real category and these five do not:

rippled_ping_Bytes_In        -> "ping"        not a category   (display "Ping")
rippled_status_Bytes_In      -> "status"      not a category   (display "Status")
rippled_getObject_Bytes_In   -> "getObject"   not a category   (display "Get Object")
rippled_haveTxSet_Bytes_In   -> "haveTxSet"   not a category   (display "Have Tx Set")
rippled_ledgerData_Bytes_In  -> "ledgerData"  not a category   (display "Ledger Data")

Absence check — grep -c '"ping"' src/xrpld/overlay/detail/TrafficCount.h and the same for the other four all return 0. The nearest real names are overhead, getobject_* (lowercase o, and always with a sub-type suffix such as getobject_get), have_transactions, and ledger_data_get / ledger_data_share.

The other five matchers I initially suspected — the mixed-case ledger_data_Account_State_Node_get family — are genuine; that was an artifact of my first grep excluding uppercase, and they do appear in the table.

Fix: delete the five dead matchers, or repoint them at the real names if those message classes were the intent.

[xrplf-ai-reviewer]

Unit inconsistency flagged in dashboard—see inline comment.

[xrplf-ai-reviewer]

Unit mismatch between panels—this shows 's' but another divides by 1M implying µs. Confirm unit and sync:

Verify whether State_Accounting_*_duration is emitted in microseconds or seconds, then:

1. If microseconds: change this panel's unit to 'µs' and remove /1000000 from the 'State Duration Rate' panel
2. If seconds: add /1000000 scaling to this panel's metric expressions