feat: add actor and kv metrics #3952

MasterPtato · 2026-01-17T00:29:50Z

No description provided.

railway-app · 2026-01-17T00:29:58Z

🚅 Deployed to the rivet-pr-3952 environment in rivet-frontend

Service	Status	Web	Updated (UTC)
website	🕒 Building (View Logs)	Web	Jan 27, 2026 at 3:17 am
frontend-inspector	🕒 Building (View Logs)	Web	Jan 27, 2026 at 3:17 am
frontend-cloud	🕒 Building (View Logs)	Web	Jan 27, 2026 at 3:17 am

MasterPtato · 2026-01-17T00:30:07Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

fix: move actor metrics to separate workflow #4044 : 2 dependent PRs (#4014 , #4047 )
chore: clean up pools, remove unused crates #4013
feat: add actor and kv metrics #3952 👈 (View in Graphite)
fix: remove attributes from last metric push metric #4043
fix(pb): always check actor queue on init #3982
main

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

claude · 2026-01-20T18:50:43Z

Pull Request Review: Add Actor and KV Metrics

Summary

This PR adds comprehensive metrics tracking for Rivet Actors and KV operations, including actor uptime, KV storage usage, network traffic, and request counts. The implementation includes database keys, metrics recording in workflows, and Grafana dashboards for visualization.

Code Quality & Best Practices

✅ Strengths

Well-structured metric system: The new keys::ns::metric module provides a clean enum-based metric system with proper serialization/deserialization.
Consistent error handling: Uses the repository's custom error system correctly with anyhow::Result.
Proper use of atomic operations: Uses MutationType::Add for incrementing metrics, which is appropriate for concurrent updates.
Comprehensive test coverage: Added test files kv_list_edge_cases.rs and kv_operations.rs.
Documentation updates: Added Grafana dashboards and updated documentation in docs/engine/GASOLINE/.

⚠️ Issues & Concerns

1. Inconsistent Byte Encoding (Critical)

Location: engine/packages/pegboard/src/keys/ns/metric.rs:85-90

Issue: This file uses little-endian (LE) encoding for metrics, while engine/packages/pegboard/src/keys/actor.rs:36-41 uses big-endian (BE) encoding.

// In metric.rs - Uses LE
fn deserialize(&self, raw: &[u8]) -> Result<Self::Value> {
    Ok(i64::from_le_bytes(raw.try_into()?))
}

// In actor.rs - Uses BE
fn deserialize(&self, raw: &[u8]) -> Result<Self::Value> {
    Ok(i64::from_be_bytes(raw.try_into()?))
}

Why this matters: Mixing endianness can lead to difficult debugging, potential bugs, and confusion for future developers.

Recommendation:

Document WHY LE encoding was chosen for metrics (is it for atomic operations compatibility?)
Consider standardizing on one endianness across the entire codebase
Add a comment explaining the trade-offs

2. Potential Integer Overflow (Medium)

Location: engine/packages/pegboard/src/actor_kv/mod.rs:124-130, 236-242, 276-283, 347-354

Issue: Multiple locations round up byte sizes without overflow protection:

let total_size_chunked = (total_size as u64).div_ceil(util::metric::KV_BILLABLE_CHUNK)
    * util::metric::KV_BILLABLE_CHUNK;
keys::ns::metric::inc(
    &tx.with_subspace(keys::subspace()),
    recipient.namespace_id,
    keys::ns::metric::Metric::KvRead(recipient.name.clone()),
    total_size_chunked.try_into().unwrap_or_default(),
);

Problem:

total_size_chunked could theoretically overflow when multiplied back
try_into().unwrap_or_default() silently converts overflow to 0, which would undercount metrics

Recommendation: Use saturating arithmetic:

let total_size_chunked = (total_size as u64)
    .div_ceil(util::metric::KV_BILLABLE_CHUNK)
    .saturating_mul(util::metric::KV_BILLABLE_CHUNK);
let metric_value = total_size_chunked.try_into().unwrap_or(i64::MAX);

3. Unbounded Metrics Growth (Critical)

Location: engine/packages/pegboard/src/keys/ns/metric.rs

Issue: Metrics are stored as database keys per actor name, with no apparent cleanup mechanism. This could lead to unbounded database growth if:

Actor names are dynamically generated
Old actor names are never cleaned up
Metrics accumulate indefinitely

Recommendation:

Implement a metric cleanup/archival strategy
Add documentation about metric lifecycle
Consider TTL or periodic aggregation

4. Atomic Load Ordering (Performance)

Location: engine/packages/pegboard-gateway/src/metrics_task.rs:61-62

Issue: Uses Acquire ordering for atomic loads:

let new_ingress_bytes = ingress_bytes.load(std::sync::atomic::Ordering::Acquire);
let new_egress_bytes = egress_bytes.load(std::sync::atomic::Ordering::Acquire);

Analysis: Acquire ordering is stronger than necessary for simple counter reads. Relaxed would suffice and be more efficient.

Recommendation:

let new_ingress_bytes = ingress_bytes.load(std::sync::atomic::Ordering::Relaxed);
let new_egress_bytes = egress_bytes.load(std::sync::atomic::Ordering::Relaxed);

5. Missing Error Context (Minor)

Location: engine/packages/pegboard/src/actor_kv/entry.rs:39, 44

Issue: Errors lack context about which key failed:

ensure\!(\!self.value.is_empty(), "empty value at key");
self.metadata.context("no metadata for key")?

Recommendation: Include the key in error messages:

ensure\!(\!self.value.is_empty(), "empty value at key {:?}", self.key.0);
self.metadata.with_context(|| format\!("no metadata for key {:?}", self.key.0))?

Security Concerns

✅ No Critical Security Issues Found

Input validation: KV operations properly validate keys and sizes
CORS handling: Proper CORS implementation with origin echoing and credentials support
No SQL injection: Uses parameterized database operations

Performance Considerations

⚠️ Potential Performance Issues

Frequent Metric Updates
- Metrics are updated every 15 seconds via METRICS_INTERVAL_MS
- Each update performs a full KV size estimation
- Consider batching or reducing frequency for high-traffic scenarios
KV Size Estimation (engine/packages/pegboard/src/actor_kv/mod.rs:32-37)
- Called during deallocation and periodic metrics recording
- Could be expensive for large KV stores
- Recommendation: Cache estimates between intervals or use incremental tracking
Stream Processing (engine/packages/pegboard/src/actor_kv/mod.rs:286-327)
- Uses .buffer_unordered(32) for parallel processing
- Could cause memory pressure with large batches
- Recommendation: Consider reducing concurrency for memory-constrained environments

Test Coverage

✅ Good Test Infrastructure

Added test files for KV operations
Tests cover edge cases for listing operations

⚠️ Missing Test Coverage

No tests for metric recording accuracy
No tests for overflow scenarios in metric calculations
No tests for concurrent metric updates
No tests for metric cleanup/archival

Recommendation: Add unit tests for edge cases and concurrent operations.

Documentation

✅ Strengths

Updated Grafana dashboards for all Docker environments
Added documentation in docs/engine/GASOLINE/

⚠️ Areas for Improvement

No inline documentation explaining metric types and their meaning
Missing documentation on metric retention/cleanup policies
No examples of querying metrics

Recommendation: Add module-level documentation explaining metric types, lifecycle, and usage patterns.

Recommendations Summary

Must Fix (Before Merge)

⚠️ Document or fix LE/BE encoding inconsistency - This could cause subtle bugs
⚠️ Add overflow protection for metric calculations - Prevents silent data loss
⚠️ Add metric lifecycle documentation and cleanup strategy - Prevents unbounded growth

Should Fix (Soon After Merge)

🔧 Improve error messages with key context
⚡ Optimize atomic operations (Acquire → Relaxed)
🧪 Add comprehensive metric tests

Nice to Have

📚 Add module-level documentation with examples
⚡ Consider caching KV size estimates

Overall Assessment

Rating: 🟡 Approve with Minor Changes

This PR adds valuable observability to Rivet Actors with a well-structured metrics system. The implementation follows most repository conventions and includes proper Grafana dashboards. However, there are a few concerns that should be addressed:

Critical: The LE/BE encoding inconsistency should be documented or resolved
Important: Integer overflow handling needs improvement
Important: Missing strategy for metric cleanup/growth management

The code quality is generally good, and the issues identified are addressable with minor changes. The PR significantly improves observability without introducing major security vulnerabilities.

Reviewed with attention to Rivet conventions from CLAUDE.md

pkg-pr-new · 2026-01-20T18:51:32Z

More templates

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@3952

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@3952

@rivetkit/db

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/db@3952

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@3952

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@3952

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@3952

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@3952

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@3952

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@3952

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@3952

commit: 46b43db

engine/packages/pegboard/src/workflows/actor/setup.rs

graphite-app · 2026-01-27T02:49:38Z

Merge activity

Jan 27, 2:49 AM UTC: MasterPtato added this pull request to the Graphite merge queue.
Jan 27, 2:50 AM UTC: The Graphite merge queue removed this pull request due to downstack failures on PR #3982.
Jan 27, 2:50 AM UTC: The Graphite merge queue removed this pull request due to downstack failures on PR #3982.
Jan 27, 2:59 AM UTC: MasterPtato added this pull request to the Graphite merge queue.
Jan 27, 3:00 AM UTC: The Graphite merge queue removed this pull request due to downstack failures on PR #4043.
Jan 27, 3:00 AM UTC: The Graphite merge queue removed this pull request due to downstack failures on PR #4043.
Jan 27, 3:12 AM UTC: MasterPtato added this pull request to the Graphite merge queue.
Jan 27, 3:12 AM UTC: The Graphite merge queue removed this pull request due to downstack failures on PR #4043.
Jan 27, 3:12 AM UTC: The Graphite merge queue removed this pull request due to downstack failures on PR #4043.
Jan 27, 3:17 AM UTC: MasterPtato added this pull request to the Graphite merge queue.
Jan 27, 3:18 AM UTC: CI is running for this pull request on a draft pull request (#4048) due to your merge queue CI optimization settings.
Jan 27, 3:19 AM UTC: Merged by the Graphite merge queue via draft PR: #4048.

railway-app bot temporarily deployed to rivet-frontend / rivet-pr-3952 January 17, 2026 00:30 Destroyed

MasterPtato changed the title ~~fix: day 1 patch, add dashboards~~ feat: add actor and kv metrics Jan 17, 2026

MasterPtato force-pushed the 01-15-feat_add_actor_and_kv_metrics2 branch from 67c4a86 to 8b7ad2b Compare January 20, 2026 18:48

railway-app bot temporarily deployed to rivet-frontend / rivet-pr-3952 January 20, 2026 18:48 Destroyed

MasterPtato force-pushed the 01-15-feat_add_actor_and_kv_metrics2 branch from 8b7ad2b to b47e44c Compare January 20, 2026 20:53

railway-app bot temporarily deployed to rivet-frontend / rivet-pr-3952 January 20, 2026 20:53 Destroyed

MasterPtato force-pushed the 01-15-feat_add_actor_and_kv_metrics2 branch from b47e44c to fc05f9d Compare January 21, 2026 21:33

railway-app bot temporarily deployed to rivet-frontend / rivet-pr-3952 January 21, 2026 21:33 Destroyed

MasterPtato marked this pull request as ready for review January 21, 2026 21:33

MasterPtato requested a review from NathanFlurry January 21, 2026 21:33

graphite-app bot reviewed Jan 21, 2026

View reviewed changes

engine/packages/pegboard/src/workflows/actor/setup.rs Show resolved Hide resolved

This was referenced Jan 23, 2026

chore: clean up pools, remove unused crates #4013

Closed

DO NOT MERGE: add metrics exporter for ee #4014

Closed

MasterPtato force-pushed the 01-15-feat_add_actor_and_kv_metrics2 branch from fc05f9d to 8246e45 Compare January 23, 2026 01:09

railway-app bot temporarily deployed to rivet-frontend / rivet-pr-3952 January 23, 2026 01:09 Destroyed

MasterPtato changed the base branch from main to graphite-base/3952 January 26, 2026 21:23

MasterPtato force-pushed the 01-15-feat_add_actor_and_kv_metrics2 branch from 8246e45 to d1cdcc5 Compare January 26, 2026 21:23

railway-app bot temporarily deployed to rivet-frontend / rivet-pr-3952 January 26, 2026 21:23 Destroyed

MasterPtato changed the base branch from graphite-base/3952 to 01-26-fix_remove_attributes_from_last_metric_push_metric January 26, 2026 21:23

This was referenced Jan 26, 2026

fix(pb): always check actor queue on init #3982

Closed

fix: remove attributes from last metric push metric #4043

Closed

fix: move actor metrics to separate workflow #4044

Closed

NathanFlurry approved these changes Jan 27, 2026

View reviewed changes

graphite-app bot force-pushed the 01-26-fix_remove_attributes_from_last_metric_push_metric branch 2 times, most recently from 0bd45c1 to e0d8017 Compare January 27, 2026 03:02

graphite-app bot changed the base branch from 01-26-fix_remove_attributes_from_last_metric_push_metric to graphite-base/3952 January 27, 2026 03:14

feat: add actor and kv metrics

46b43db

MasterPtato force-pushed the graphite-base/3952 branch from e0d8017 to be53522 Compare January 27, 2026 03:16

MasterPtato force-pushed the 01-15-feat_add_actor_and_kv_metrics2 branch from d1cdcc5 to 46b43db Compare January 27, 2026 03:16

railway-app bot temporarily deployed to rivet-frontend / rivet-pr-3952 January 27, 2026 03:16 Destroyed

MasterPtato changed the base branch from graphite-base/3952 to main January 27, 2026 03:16

MasterPtato mentioned this pull request Jan 27, 2026

chore: remove ch provision users #4047

Closed

graphite-app bot pushed a commit that referenced this pull request Jan 27, 2026

feat: add actor and kv metrics (#3952)

f534c17

graphite-app bot closed this Jan 27, 2026

graphite-app bot deleted the 01-15-feat_add_actor_and_kv_metrics2 branch January 27, 2026 03:19

This was referenced Jan 27, 2026

fix: fix active req caculation, add hws metrics #4053

Closed

chore(guard): clean up guard #4057

Closed

fix(gateway): configure cors for error responses #4058

Closed

fix(guard): cache custom serve routes #4059

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add actor and kv metrics #3952

feat: add actor and kv metrics #3952

Uh oh!

MasterPtato commented Jan 17, 2026 •

edited

Loading

Uh oh!

railway-app bot commented Jan 17, 2026 •

edited

Loading

Uh oh!

MasterPtato commented Jan 17, 2026 •

edited

Loading

Uh oh!

claude bot commented Jan 20, 2026 •

edited

Loading

Uh oh!

pkg-pr-new bot commented Jan 20, 2026 •

edited

Loading

Uh oh!

Uh oh!

graphite-app bot commented Jan 27, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

feat: add actor and kv metrics #3952

feat: add actor and kv metrics #3952

Uh oh!

Conversation

MasterPtato commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

railway-app bot commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MasterPtato commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

claude bot commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Review: Add Actor and KV Metrics

Summary

Code Quality & Best Practices

✅ Strengths

⚠️ Issues & Concerns

1. Inconsistent Byte Encoding (Critical)

2. Potential Integer Overflow (Medium)

3. Unbounded Metrics Growth (Critical)

4. Atomic Load Ordering (Performance)

5. Missing Error Context (Minor)

Security Concerns

✅ No Critical Security Issues Found

Performance Considerations

⚠️ Potential Performance Issues

Test Coverage

✅ Good Test Infrastructure

⚠️ Missing Test Coverage

Documentation

✅ Strengths

⚠️ Areas for Improvement

Recommendations Summary

Must Fix (Before Merge)

Should Fix (Soon After Merge)

Nice to Have

Overall Assessment

Uh oh!

pkg-pr-new bot commented Jan 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

graphite-app bot commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge activity

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MasterPtato commented Jan 17, 2026 •

edited

Loading

railway-app bot commented Jan 17, 2026 •

edited

Loading

MasterPtato commented Jan 17, 2026 •

edited

Loading

claude bot commented Jan 20, 2026 •

edited

Loading

pkg-pr-new bot commented Jan 20, 2026 •

edited

Loading

graphite-app bot commented Jan 27, 2026 •

edited

Loading