feat: Phase 10: Synthetic workload generation & telemetry validation suite by pratikmankawde · Pull Request #6519 · XRPLF/rippled

pratikmankawde · 2026-03-10T18:25:40Z

PR Chain: Phase-1a → Phase-1b → Phase-1c → Phase-2 → Phase-3 → Phase-4 → Phase-5 → Phase-6 → Phase-7 → Phase-8 → Phase-9 → #6519 (this PR, Phase-10)
Base: pratik/otel-phase9-metric-gap-fill
Consolidated PR: #7770

High Level Overview of Change

The validation harness for everything above: drives a multi-node validator cluster under synthetic RPC and transaction load, asserts every expected span, attribute and metric actually fires, then gates timing against a committed baseline. Test and CI assets only — no src/ or include/ changes.

No need to review tasklist files.

Context of Change

Up to here, "the telemetry works" rested on reading dashboards by hand. This adds a repeatable check.

Cluster + load — docker-compose.workload.yaml, generate-validator-keys.sh, xrpld-validator.cfg.template; rpc_load_generator.py (async WebSocket, configurable rate/mix) and tx_submitter.py (10 transaction types, configurable TPS, periodic sequence refresh), sequenced by workload_orchestrator.py.
Assertions — validate_telemetry.py checks the inventories in expected_spans.json (41 span types, 62 unique attributes, plus parent-child relationships) and expected_metrics.json (26 metric groups) against live Tempo and Prometheus.
Regression gate — capture_timings.py queries histogram_quantile over span_duration_milliseconds_bucket and job_{queued,running}_us_bucket, producing 28 keys: 8 span names × p50/p95/p99, plus 2 job types × queued/running at p95. compare_to_baseline.py diffs them against baselines/baseline-timings.json (populated, captured 2026-06-05) and flags a regression only when both the percentage and absolute bounds are breached.
Excluded on purpose — rpc.process is not gated: it is created only on the HTTP JSON-RPC path, which the WebSocket-only generators never reach, so its quantiles were always null. There is no rpc_methods group either (FU-4).
Overhead benchmark — benchmark.sh runs the workload with telemetry off and on.
CI — .github/workflows/telemetry-validation.yml: build → cluster → load → validate → regression gate → report.

Follow-ups

FU-2: publish the baseline as a develop artifact instead of committing it
FU-4: replace benchmark.sh proxy measurements with PromQL, and gate RPC method timings
FU-6: historical trend dashboard

API Impact

None — test, CI and documentation assets only.

Test Plan

docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld — cluster reaches consensus, workload completes.
Validation reports zero missing spans, attributes and metrics.
Regression gate passes against the committed baseline.

pratikmankawde

CI Build Performance Issues

The Telemetry Validation workflow is timing out because:

Conan packages are being built from source (~45 min) instead of pulled as pre-built binaries from the remote
No ccache for the xrpld compilation step (~45+ min cold build)

Combined, these exceed the 90-minute timeout. See inline comments for specific fixes.

pratikmankawde

Build Performance Fixes

4 suggestions to fix the timeout. Apply all of them together:

Add ccache env vars
Install ccache package
Authenticate to Conan remote (this is why deps build from source for 45 min)
Use ccache in CMake configure

pratikmankawde

Fix broken YAML from previous suggestions

The 4 suggestions below fix the duplicate/missing lines. Apply all as a batch.

codecov · 2026-03-16T13:23:27Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

The committed baseline was captured 2026-08-26, before the account-funding race was detectable. Phases whose funding silently failed submitted no transactions, so the capture recorded artificially low ledger and transaction timings, and job.transaction.queued.p95 and job.transaction.running.p95 could not be captured at all. Once funding worked, span.ledger.build.p99 read 29.00 ms against a 9.11 ms baseline and turned the gate red on a run whose 200 span and metric checks all passed. Refresh every value to the median of CI runs 34495527952, 34505215266 and 34507425933, the first three with the fix in place, and re-derive each absolute bound as hi_next - baseline from that median. Exclude span.ledger.build.p99. Across those three runs it read 29.00, 7.06 and 8.94 ms, a 4.11x spread whose maximum is 1.16x its 25 ms trip point, so a healthy run reddens CI. Widening cannot fix it: a bound tolerating 29.00 ms would reach into the bucket above and restore the single-crossing false positive the derivation rule removes. span.ledger.build.p95 stays gated at 0.48 of its trip point, so ledger construction keeps coverage. The other 19 keys sit between 0.17 and 0.76 of their trip points. span.tx.process.p95 is the tightest and is the first to re-measure if the gate reddens again. Repoint one bounds-checker test at span.ledger.build.p95, since it mutated the p99 override this commit removes.

xrplf-ai-reviewer

Baseline docs mismatch and validator directory naming bugs — see inline.

xrplf-ai-reviewer · 2026-09-10T18:19:21Z

+
+## Current state: 20 metrics gate, on a baseline captured 2026-08-26
+
+`baseline-timings.json` holds real captured values for the 20 keys the harness gates, from CI run


README narrative doesn't match baseline-timings.json (dates, SHA, run IDs differ) — regenerate against the actual file.

…ase10-workload-validation

Merges pratik/otel-phase9-metric-gap-fill into pratik/otel-phase10-workload-validation. Auto-merged. Carries the weak_ptr gauges_ change into OTelCollector, the Test 1 standalone-store fix in TESTING.md, and the collection-lifecycle calls in the StatsD test that phase-7 requires.

…m phase9-metric-gap-fill

…ric-gap-fill

xrplf-ai-reviewer

Baseline README contradicts baseline-timings.json; pkill patterns target wrong node directories — see inline.

…c-gap-fill

…-metric-gap-fill

xrplf-ai-reviewer

Path mismatches prevent stale process cleanup and error detection.

…c-gap-fill

The comment measured date +%s%N cost 'on a dev box'; say 'on one Linux host' instead. The number is the point, not where it was taken.

…ase10-workload-validation

xrplf-ai-reviewer

Directory pattern mismatches in pkill commands prevent cleanup of stray xrpld processes — see inline.

…ase10-workload-validation

xrplf-ai-reviewer

Process cleanup uses incorrect directory names (node/nodeN instead of validator-N/bench-node-N), leaking processes and disabling crash detection — see inline.

…ase10-workload-validation

xrplf-ai-reviewer

Process cleanup regex patterns don't match actual directory names — see inline for fixes.

…ase10-workload-validation

xrplf-ai-reviewer

Two pkill regex patterns don't match directory naming — see inline.

…workload harness The harness killed and probed node directories named `node<N>`, but the directories it creates are `validator-<N>` in run-full-validation.sh and `bench-node-<N>` in benchmark.sh. Verified with pgrep against processes whose command lines mimic the real ones: the pattern matched nothing either script produces. Three consequences, all live: - `--cleanup` deleted the workdir and left the xrpld processes running. They are host processes, so the compose teardown does not reach them. - The pre-run cleanup could not free the previous run's RPC, WS and peer ports, which surfaces much later as a cluster that never reaches consensus. - The startup crash fast-fail read a pid path that never exists, so its `stopped > 0` branch was unreachable and a dead node waited out the full 120-attempt window. Rather than patch four literals, derive every node path, kill pattern and log glob from one NODE_PREFIX per script. The directory name is also the node's identity: the collector's file_log receiver lifts that segment into service.instance.id, so the directory and the [telemetry] service_instance_id must agree. Deriving both from one value is what stops them drifting again. Also in the same files, each confirmed by test rather than inspection: - The collector readiness probe could never fail. curl -w '%{http_code}' prints 000 on a refused connection and then exits non-zero, so the `|| echo 000` inside the substitution appended a second 000 and the "not ready" comparison never matched. Move the fallback outside. - The generated config wrote [ips], the starter-list section. A loopback mesh that must reach quorum is the [ips_fixed] case, which is what the variable, the comment and the sibling cfg template already said. - benchmark.sh returned exit 1 for a row it could not measure, though the exit-code table reserves 1 for "every metric was measured and one breached". Report 2 there instead. - Five bc computations fell back to 0, which clears every threshold. The guards beside them already fall back to the inconclusive token; these now do too. - A comment claimed a `|| guard` after a heredoc lands in the heredoc, and that claim had removed a real guard from the config write. It does not: the guard runs, and fires when cat fails. - The EXIT trap was installed 88 lines before stop_workload was defined. If it fired in that window, errexit aborted the handler on "command not found" and the cluster reap never ran. Install it below both handlers. - jq exits 5 on malformed JSON, outside this script's documented codes, so read_metric now routes that through cannot_measure. - --nodes and --duration were unvalidated, and --nodes 0 made the pid-count guard compare 0 with 0 and pass, handing the sampler no pids at all. - --cleanup now passes -v so the named tempo-data volume goes with it. Otherwise the next run's Tempo still serves the previous run's traces and a span assertion can be satisfied by them. - Five messages reported an attempt count as seconds, though each attempt is a sleep plus every node's probe. The baselines README and the two regression JSON files had gone stale when the baseline was refreshed to a three-run median: they described 20 gated keys and five exclusions, against an actual 19 and six, and cited the superseded run, date and commit. Re-derive every affected figure from the committed files. The detection floors are recomputed (2.00x to 7.41x, so a 10x regression is now caught on all 19 keys), the newly excluded span.ledger.build.p99 is documented, and figures that no committed artifact can verify are either replaced with derivable ones or labelled with their numerator. No baseline value, threshold bound or derivation entry changes.

xrplf-ai-reviewer

Regression bound lookup ignores metric unit — see inline.

xrplf-ai-reviewer · 2026-09-17T14:18:07Z

+            f"max_abs_increase_{unit} and max_pct_increase -- fix it in {THRESHOLDS}"
+        ]
+
+    bound = rule.get("max_abs_increase_ms", rule.get("max_abs_increase_us"))


Hardcoded max_abs_increase_ms as primary fallback ignores metric unit — use unit-aware lookup:

Suggested change

bound = rule.get("max_abs_increase_ms", rule.get("max_abs_increase_us"))

bound = rule.get(f"max_abs_increase_{unit}")

Suggested change

bound = rule.get("max_abs_increase_ms", rule.get("max_abs_increase_us"))

bound = rule.get(f"max_abs_increase_{unit}")

The validation owned its own push trigger and paths filter, which made it the one build-and-test workflow outside the on-pr.yml plus reusable-*.yml pair the rest of CI uses. Rename it to reusable-telemetry-validation.yml, take workflow_call, and let on-pr.yml decide when it runs. on-pr.yml gains a second changed-files list for the telemetry paths. Its existing `go` flag is true for nearly every pull request, so gating on `go` alone would run a self-hosted build and a 30-minute cluster on all of them. The configure step no longer repeats the telemetry option. A command-line define is written to the cache before the toolchain runs and wins over it, so it could turn telemetry on with the package never fetched. build-xrpld reads the value the build actually configured out of CMakeCache.txt and publishes it as a job output; validate-telemetry runs only when that says telemetry is on, and every value CMake does not accept as a boolean fails the job rather than skipping silently.

The previous commit switched the generated node config from [ips] to [ips_fixed] on the grounds that the variable, the comment and the sibling cfg template all named ips_fixed, and that ips_fixed is the section whose documented meaning fits a private cluster. Both of those are still true. The switch is reverted anyway, because it is a workload change rather than a naming fix. Measured on CI, parent commit against this branch's previous tip, one functional config line apart: span.consensus.ledger_close.p95 0.57 ms -> 6.43 ms (tripped the gate) span.consensus.ledger_close.p99 0.94 ms -> 9.50 ms span.consensus.accept.p50 0.97 ms -> 2.63 ms span.tx.process.p50 0.36 ms -> 0.18 ms (faster) job.acceptLedger.running.p95 21157 us -> 10938 us (faster) Every consensus-path span rose and every transaction-path metric fell, which is the shape a denser always-connected mesh produces and not the shape of run-to-run variance. [ips_fixed] holds connections open to all four peers instead of treating the list as a discovery hint, so each node processes proposals and validations from the full mesh every round. Nothing else in that commit touches the consensus path: the emitted config differed in exactly three lines, of which one is a die message and one expands to an identical string. The committed baseline describes the [ips] topology. Adopting [ips_fixed] therefore needs a refreshed baseline and re-derived bounds, which is the process baselines/README.md already documents for a workload change. Left as its own work item rather than smuggled in behind a section rename, and the reason is now recorded beside the line so it is not repeated. This also falsified a claim the previous commit had written into baselines/README.md and regression-thresholds.json: that none of the six weakly-guarded keys fires on any observed run. Corrected in both, and the measurement above is cited in place of the absolute.

…ase10-workload-validation

xrplf-ai-reviewer

JSON type/range validation and Docker cleanup issues flagged inline.

xrplf-ai-reviewer · 2026-09-17T16:31:31Z

+# one here surfaces much later as a cluster that never reaches consensus.
+pkill -f "$WORKDIR/$NODE_PREFIX-[0-9]+/xrpld\.cfg" 2>/dev/null || true
+sleep 2
+rm -rf "$WORKDIR" || die "Could not remove the previous run's workdir $WORKDIR"


Auto-cleanup kills node processes but skips docker compose down, leaving stale containers bound to old directory inodes—causes false pass/fail on reruns via stale traces. Add Docker teardown before filesystem cleanup:

docker compose -f "${COMPOSE_FILE}" down -v 2>/dev/null || true rm -rf "${WORKDIR}"

xrplf-ai-reviewer · 2026-09-17T16:31:32Z

+    weights = DEFAULT_TX_WEIGHTS.copy()
+    if args.weights:
+        try:
+            custom = json.loads(args.weights)


JSON parsing assumes dict; crashes on array/scalar with unhandled AttributeError. Also, negative weights pass validation but crash in random.choices mid-run. Validate type and range upfront:

Suggested change

custom = json.loads(args.weights)

if not isinstance(custom, dict):

sys.exit("--weights must be a JSON object, not array or scalar")

for name, weight in custom.items():

if not isinstance(weight, (int, float)) or weight <= 0:

sys.exit(f"weight {name}={weight} must be positive")

…ase10-workload-validation

tx.receive no longer carries a suppressed attribute: the span is created only once the node has decided to process the transaction, so there is no dropped copy for the attribute to describe. The validator fails a span that is missing a required attribute, so leaving it listed turns the telemetry-validation leg red.

xrplf-ai-reviewer

Off-by-one in expected_spans.json; missing argument validation and port-query robustness in workload scripts.

xrplf-ai-reviewer · 2026-09-17T22:30:32Z

+    }
+  ],
+  "total_span_types": 41,
+  "total_unique_attributes": 62


total_unique_attributes is 62, but union of all required_attributes arrays yields only 61. Recount programmatically and correct:

Suggested change

"total_unique_attributes": 62

"total_unique_attributes": 61,

Suggested change

"total_unique_attributes": 62

"total_unique_attributes": 61

xrplf-ai-reviewer · 2026-09-17T22:30:32Z

+    done
+
+    # Record current validated ledger seq.
+    for port in "${RPC_PORTS[@]}"; do


Only queries first RPC port; records zero on failure. Apply max-across-ports robustness from INITIAL_SEQ, skipping failed ports:

Suggested change

for port in "${RPC_PORTS[@]}"; do

for port in ${RPC_PORTS}; do

seq=$(curl -s -w '%{http_code}' http://localhost:"$port" 2>/dev/null | jq -r '.result.ledger_index // empty')

if [ -n "$seq" ] && [ "$seq" -gt 0 ]; then

echo "$seq" >> "$LEDGER_FILE"

break

fi

done

xrplf-ai-reviewer · 2026-09-17T22:30:32Z

+    exit 0
+}
+
+while [ $# -gt 0 ]; do


Unbound variable on missing flag argument under set -u. Validate argument count before accessing $2:

Suggested change

while [ $# -gt 0 ]; do

[ $# -ge 2 ] || die "--xrpld requires a value"

XRPLD="$2"

shift 2

pratikmankawde marked this pull request as draft March 10, 2026 18:28

pratikmankawde added DraftRunCI Normally CI does not run on draft PRs. This opts in. DistributedTracingAndObservability Distributed Tracing And Observability related changes labels Mar 10, 2026

semgrep-companion-app Bot reviewed Mar 10, 2026

View reviewed changes

Comment thread .github/workflows/reusable-telemetry-validation.yml

pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from dc53faa to d3d2325 Compare March 10, 2026 20:08

semgrep-companion-app Bot reviewed Mar 10, 2026

View reviewed changes

Comment thread .github/workflows/reusable-telemetry-validation.yml

pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch 3 times, most recently from 7fb4b09 to 0a44290 Compare March 11, 2026 14:54

pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch 3 times, most recently from 3faf309 to ea60b46 Compare March 11, 2026 16:58

semgrep-companion-app Bot reviewed Mar 11, 2026

View reviewed changes

Comment thread .github/workflows/reusable-telemetry-validation.yml

pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from 252aceb to 42a46a3 Compare March 12, 2026 10:54

pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from 42a46a3 to 0314b25 Compare March 12, 2026 12:00

pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from 0314b25 to fb71866 Compare March 12, 2026 13:04

pratikmankawde commented Mar 12, 2026

View reviewed changes

Comment thread .github/workflows/telemetry-validation.yml

Comment thread .github/workflows/telemetry-validation.yml Outdated

Comment thread .github/workflows/telemetry-validation.yml Outdated

Comment thread .github/workflows/telemetry-validation.yml

pratikmankawde commented Mar 12, 2026

View reviewed changes

pratikmankawde force-pushed the pratik/otel-phase10-workload-validation branch from 5dfef07 to 6367ca1 Compare March 12, 2026 22:14

xrplf-ai-reviewer Bot suggested changes Sep 10, 2026

View reviewed changes

pratikmankawde added 5 commits September 14, 2026 20:40

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-ph…

6601697

…ase10-workload-validation

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-ph…

fa9e021

…ase10-workload-validation

merge: bring the stop() recording gate and identity tests forward fro…

379bd5a

…m phase9-metric-gap-fill

merge: bring the noopMeter() clang-tidy fixes forward from phase9-met…

fe7d973

…ric-gap-fill

xrplf-ai-reviewer Bot suggested changes Sep 14, 2026

View reviewed changes

Comment thread docker/telemetry/workload/baselines/README.md Outdated

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/benchmark.sh Outdated

pratikmankawde added 2 commits September 15, 2026 11:24

merge: bring the SecretKey test-compare fix forward from phase9-metri…

904d4da

…c-gap-fill

merge: bring the clang-tidy include-cleaner fixes forward from phase9…

d87ca87

…-metric-gap-fill

xrplf-ai-reviewer Bot suggested changes Sep 15, 2026

View reviewed changes

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/run-full-validation.sh

Comment thread docker/telemetry/workload/run-full-validation.sh

pratikmankawde added 3 commits September 15, 2026 14:22

merge: bring the optional-access NOLINT fix forward from phase9-metri…

a305b10

…c-gap-fill

docs(telemetry): drop the host name from the sampling-clock comment

50eff17

The comment measured date +%s%N cost 'on a dev box'; say 'on one Linux host' instead. The number is the point, not where it was taken.

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-ph…

8efc8e9

…ase10-workload-validation

xrplf-ai-reviewer Bot suggested changes Sep 15, 2026

View reviewed changes

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/benchmark.sh Outdated

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-ph…

5277ef4

…ase10-workload-validation

xrplf-ai-reviewer Bot suggested changes Sep 15, 2026

View reviewed changes

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/benchmark.sh Outdated

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-ph…

5f03e41

…ase10-workload-validation

xrplf-ai-reviewer Bot suggested changes Sep 16, 2026

View reviewed changes

Comment thread docker/telemetry/workload/benchmark.sh Outdated

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-ph…

3b674d8

…ase10-workload-validation

xrplf-ai-reviewer Bot suggested changes Sep 17, 2026

View reviewed changes

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

Comment thread docker/telemetry/workload/run-full-validation.sh Outdated

xrplf-ai-reviewer Bot reviewed Sep 17, 2026

View reviewed changes

pratikmankawde added 3 commits September 17, 2026 16:55

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-ph…

e0520b7

…ase10-workload-validation

xrplf-ai-reviewer Bot reviewed Sep 17, 2026

View reviewed changes

pratikmankawde added 2 commits September 17, 2026 23:01

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-ph…

04786c4

…ase10-workload-validation

xrplf-ai-reviewer Bot reviewed Sep 17, 2026

View reviewed changes


		## Current state: 20 metrics gate, on a baseline captured 2026-08-26

		`baseline-timings.json` holds real captured values for the 20 keys the harness gates, from CI run

	bound = rule.get("max_abs_increase_ms", rule.get("max_abs_increase_us"))
	bound = rule.get(f"max_abs_increase_{unit}")

-            custom = json.loads(args.weights)
+if not isinstance(custom, dict):
+    sys.exit("--weights must be a JSON object, not array or scalar")
+for name, weight in custom.items():
+    if not isinstance(weight, (int, float)) or weight <= 0:
+        sys.exit(f"weight {name}={weight} must be positive")

-    for port in "${RPC_PORTS[@]}"; do
+for port in ${RPC_PORTS}; do
+  seq=$(curl -s -w '%{http_code}' http://localhost:"$port" 2>/dev/null | jq -r '.result.ledger_index // empty')
+  if [ -n "$seq" ] && [ "$seq" -gt 0 ]; then
+    echo "$seq" >> "$LEDGER_FILE"
+    break
+  fi
+done

-while [ $# -gt 0 ]; do
+[ $# -ge 2 ] || die "--xrpld requires a value"
+XRPLD="$2"
+shift 2

Conversation

pratikmankawde commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

High Level Overview of Change

Context of Change

Follow-ups

API Impact

Test Plan

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pratikmankawde left a comment

Choose a reason for hiding this comment

CI Build Performance Issues

Uh oh!

pratikmankawde left a comment

Choose a reason for hiding this comment

Build Performance Fixes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pratikmankawde left a comment

Choose a reason for hiding this comment

Fix broken YAML from previous suggestions

Uh oh!

codecov Bot commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

xrplf-ai-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xrplf-ai-reviewer Bot Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xrplf-ai-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xrplf-ai-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xrplf-ai-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xrplf-ai-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xrplf-ai-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xrplf-ai-reviewer Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

xrplf-ai-reviewer Bot left a comment

pratikmankawde commented Mar 10, 2026 •

edited

Loading

codecov Bot commented Mar 16, 2026 •

edited

Loading