Status Aware Error Sampling for OpenTelemetry

[vishalpolley-1mg]

Description

This PR implements Status Aware Error Sampling natively in the catalyst-core OpenTelemetry integration (src/otel.js).

In high-throughput environments operating at low head-sampling rates (e.g. 1%), standard head-sampling discards 99% of all traces, resulting in the loss of 99% of error traces (4xx and 5xx). This feature introduces a two-stage sampling flow that defers the export decision until a request finishes, allowing the system to selectively promote and export error traces at independent rates (typically 100% for 5xx and 10% for 4xx) without inflating successful trace volume or incurring excessive memory overhead.

The implementation is fully backwards-compatible and operates as an opt-in feature under the errorSampling configuration block.

---

Key Features & Components

1. Custom Sampler: StatusAwareSampler

• Upstream Propagation: Respects parent/upstream sampling decisions.
• Record Stage: If a trace fails the initial head-sampling check, it returns RECORD (instead of NOT_RECORD). This instructs the SDK to build spans in memory so they can be analyzed upon request completion.
• Independent Bit-Ranges: Uses characters 0–12 (first 48 bits) of the traceId hex string to evaluate the head-sampling probability.

2. Custom Processor: PromotingSpanProcessor

• Pass-through: Automatically routes head-sampled spans (SAMPLED bit set) straight to the internal BatchSpanProcessor to preserve standard low-latency batch exports.
• Trace Buffer: Holds non-sampled spans (RECORD only) in an in-memory Map keyed by traceId.
• Bot Exclusion: Root spans carrying http.response.is_bot: true are excluded from every promotion rule below by default (PROMOTE_BOT_TRAFFIC), so bot-triggered errors can't inflate promoted volume unless explicitly opted in.
• Deferred Promotion Evaluation: When the root span of a request ends, the processor evaluates the HTTP status code:
  • 5xx Errors / Exceptions: Promoted at probability RATE_5XX (typically 100%). Evaluated using characters 12–24 of the traceId string. SKIP_PROMOTION_CODES is now consulted here too, not just for 4xx — gateway timeouts (504/524/598/599) are correctly excluded instead of always promoting anyway.
  • 4xx Client Errors: Promoted at probability RATE_4XX (excluding configured timeouts/gateway noise).
  • Handled Errors: If enabled, promotes 2xx root spans that contain errored child spans underneath.
• Backpressure-Safe Export: Promoted spans are handed to the same internal BatchSpanProcessor used for head-sampled traffic (batchProcessor.onEnd()) rather than exported directly. A burst of promotions during an incident queues and batches like any other traffic instead of opening a flood of individual export calls against the collector. Trade-off: promoted spans export on the processor's next batch flush (scheduledDelayMillis, default ~5s) rather than instantly — tunable via batchProcessorConfig.
• Late-Arriving Spans: Employs a short-lived cache of recently promoted traceIds so that asynchronous child spans completing after the root span are still successfully tagged and routed for export.
• Export Failure Visibility: init() registers a global OTEL error handler (setGlobalErrorHandler) that logs export failures — collector down, batch rejected, etc. — through the app logger, for both normal and promoted traffic, instead of failing silently.
• Memory Safeguards: Features a 30-second interval sweep enforcing a 5-minute TTL on buffered traces, alongside a hard limit of 1024 traces in the buffer to prevent OOM errors.

---

Configuration Reference

You can activate the feature by passing the errorSampling block to Otel.init:

"ERROR_SAMPLING": {
  "ENABLED": true,
  "RATE_4XX": 0.1,
  "RATE_5XX": 1.0,
  "EXPORT_FULL_TRACE_ON_ERROR": true,
  "PROMOTE_HANDLED_ERRORS": true,
  "REPORT_ACTUAL_PROMOTION_RATE": false,
  "SKIP_PROMOTION_CODES": [408, 504, 524, 598, 599],
  "PROMOTE_BOT_TRAFFIC": false
}

Full config reference, including all defaults, is now documented via JSDoc on init() in src/otel.js, and in a new "Observability" section in README.md.

---

Fixes since the initial implementation

A follow-up audit surfaced several issues, all addressed in this PR:

• Restored the OTEL_ENABLE opt-in guard in init(), which had been temporarily commented out during development — without it, the SDK would initialize unconditionally in every environment.
• Fixed SKIP_PROMOTION_CODES: the skip check previously only gated the 4xx branch, so 4 of its 5 default entries (504/524/598/599) were unreachable and got promoted anyway despite being configured to skip.
• Removed the unbounded, un-backpressured direct export path for promoted spans (see Backpressure-Safe Export above).
• Export failures are now logged instead of silently discarded.
• Added PROMOTE_BOT_TRAFFIC to keep bot traffic from inflating promoted volume by default.
• Documented the full config surface (README + JSDoc), which previously had zero mentions outside the source.

---

Testing & Verification Done

• Unit & Integration Tests: Validated ratio-based head sampling decisions, span buffering, 4xx/5xx promotion criteria, handled error routing, and cache cleanup routines.
• Local Verification: Verified trace propagation using a local test script. Spans are correctly recorded, set with traceFlags: 1 (SAMPLED), and exported to Jaeger.
• Application E2E Test: Deployed and verified in local mweb and dweb apps, confirming successful traces are exported according to the sampling rate, while 4xx and 5xx errors are successfully promoted and tagged with the "promoted": true attribute.

[deputydev-agent]

DeputyDev will no longer review pull requests automatically.To request a review, simply comment #review on your pull request—this will trigger an on-demand review whenever you need it.