Distributed Tracing

Overview

Hoist-core provides distributed tracing built on OpenTelemetry, using the OTel SDK directly for span processing and export. The system supports OTLP export and is configured dynamically via soft config.

Tracing is disabled by default and has negligible overhead when disabled — all public methods delegate to no-op implementations, so no null checks are needed in application code.

Key capabilities

Central service — TraceService manages the OpenTelemetry SDK lifecycle, exporter pipeline, and provides the withSpan span creation API.
Combined observability — ObservedRun is a composable builder that wraps a closure with any combination of tracing, logging, and metrics. Typically started via BaseService.span(), with BaseService.observe() available for the rare case where no span is wanted.
Automatic request spans — HoistFilter creates a SERVER span for each request, extracting any inbound W3C traceparent so the span joins an existing trace. Pings, version checks, and websocket handshakes are excluded.
Export — OTLP (HTTP/protobuf) export configured via soft config. Applications can register additional exporters (e.g. Zipkin) via addExporter().
W3C trace propagation — incoming traceparent headers are honored; outbound HTTP calls via JSONClient and BaseProxyService inject trace context automatically.
Cluster context propagation — ClusterTask captures and restores trace context across Hazelcast remote execution, maintaining parent-child span relationships.
Thread context propagation — Grails task {} calls automatically carry the current trace context to worker threads via HoistPromiseFactory.
Client span relay — Browser-generated spans are submitted to xh/submitSpans and exported through the same server-side pipeline, producing end-to-end client-to-server traces. Client spans are pre-sampled in the browser — only sampled spans are relayed.
Rule-based sampling — Configurable sampleRules match span tags (with glob patterns) to determine per-span sample rates.
Log correlation — traceId is captured on log markers when inside a traced context, enabling log-to-trace correlation.

Source Files

File	Location	Role
`TraceService.groovy`	`grails-app/services/io/xh/hoist/telemetry/trace/`	Central tracing service — SDK lifecycle, exporter pipeline, span API
`JdbcTraceService.groovy`	`grails-app/services/io/xh/hoist/telemetry/trace/impl/`	Internal service that installs/uninstalls JDBC DataSource instrumentation in sync with the trace config
`TraceContextService.groovy`	`grails-app/services/io/xh/hoist/telemetry/trace/`	Internal service hosting W3C context propagation — inbound filter, outbound HTTP, cluster tasks
`HoistFilter.groovy`	`src/main/groovy/io/xh/hoist/`	Wraps every request — restores trace context, enforces auth, creates the SERVER span for tracing, captures any exception, and stamps HTTP semantic-convention attributes
`SpanRef.groovy`	`src/main/groovy/io/xh/hoist/telemetry/trace/`	Wrapper around an active Span + Scope with tag/status/error helpers
`ObservedRun.groovy`	`src/main/groovy/io/xh/hoist/telemetry/`	Composable builder for combined tracing + logging + metrics
`TraceConfig.groovy`	`src/main/groovy/io/xh/hoist/telemetry/trace/`	Typed wrapper around `xhTraceConfig`
`HoistPromiseFactory.groovy`	`src/main/groovy/io/xh/hoist/`	Wraps the Grails PromiseFactory to propagate framework thread context — including OTel trace context — to `task {}` worker threads
`DelegatingOpenTelemetry.groovy`	`src/main/groovy/io/xh/hoist/telemetry/trace/`	Stable `OpenTelemetry` facade that resolves to the current SDK on every tracer/span lookup — lets library instrumentation (e.g. `opentelemetry-jdbc`) capture a reference once and follow SDK rebuilds
`TagSpanProcessor`	Inner class of `TraceService.groovy`	Stamps cross-cutting attributes (e.g. `user.name`) on every span at start time, regardless of whether the span was created by `TraceService` or by a library instrumenter
`ManualRateSampler.groovy`	`src/main/groovy/io/xh/hoist/telemetry/trace/`	Per-span thread-local sampler driven by `sampleRules` and the fallback `sampleRate`
`ClientSpanData.groovy`	`src/main/groovy/io/xh/hoist/telemetry/trace/`	Adapts client-relayed span JSON into OTel `SpanData` for export through the server pipeline

TraceService

File: grails-app/services/io/xh/hoist/telemetry/trace/TraceService.groovy

The central service for distributed tracing. Initialized early in the bootstrap sequence — immediately after ConfigService, so the remainder of startup (including MetricsService initialization) runs inside a trace span. Manages the OpenTelemetry SDK and exporter pipeline, and provides the span creation API.

`withSpan(args, closure)`

Execute a closure within a new trace span. Creates a child span if a parent context exists, or a root span otherwise. Exceptions are recorded on the span and re-thrown. The closure may optionally accept a SpanRef parameter, which may be further enhanced with tags, or information about errors. Note that a NoOp span is passed even if tracing is disabled.

For combined tracing + logging + metrics, use ObservedRun via BaseService.span() instead.

Arguments (passed as named params):

Key	Type	Description
`name`	String	Span name (required).
`kind`	SpanKind	`INTERNAL` (default), `SERVER`, or `CLIENT`.
`tags`	Map	Key-value attributes to set on the span.
`caller`	Object	Object making the call, auto-sets the `code.namespace` attribute.
`startTime`	Instant	Optional backdated start; defaults to now.

traceService.withSpan(name: 'fetchData', kind: SpanKind.CLIENT, tags: [url: endpoint]) { SpanRef span ->
    def result = httpClient.get(endpoint)
    span.setHttpStatusAndErrorStatus(result.statusCode)
    result
}

`createSpan(args)` — framework-internal

Creates and starts a span with a manually managed lifecycle, returning a SpanRef the caller must close. Private to TraceService — not callable from application code. Its parameters are the source of truth for the withSpan arguments documented above, which passes them straight through.

`addExporter(exporter)` / `removeExporter(exporter)`

Register additional SpanExporter instances to receive both server-generated and client-relayed spans. Triggers a pipeline rebuild. Use this to add custom export destinations (e.g. Zipkin):

traceService.addExporter(
    ZipkinSpanExporter.builder()
        .setEndpoint('http://zipkin:9411/api/v2/spans')
        .build()
)

SpanRef

File: src/main/groovy/io/xh/hoist/telemetry/trace/SpanRef.groovy

A wrapper around an active OTel Span and its Scope, passed to the closures run by withSpan and ObservedRun.run. Use it to enrich a span in flight with tags, a revised name, or the outcome of the work.

A shared no-op instance (SpanRef.NOOP) is passed when tracing is disabled, so closure bodies can call these methods unconditionally without null-checking.

Method	Description
`setTag(key, value)`	Set a single attribute. Coerces `Integer`/`Long` → long, `Boolean` → boolean, `Double` → double, anything else via `toString()`.
`setTags(map)`	Set multiple attributes, with the same coercion rules.
`updateName(name)`	Revise the span's display name — useful once a generic operation resolves to something more specific.
`setErrorStatus(description?)`	Mark ERROR with an optional description, no exception required.
`recordException(t)`	Record an exception event. Leaves span status untouched. No-op for `RoutineException`.
`recordExceptionAndErrorStatus(t)`	Record an exception event and mark ERROR, with a description derived from the throwable. No-op for `RoutineException`.
`setHttpStatusAndErrorStatus(code)`	Set the `http.response.status_code` tag and mark ERROR per OTel HTTP conventions: CLIENT spans at ≥400, SERVER spans at ≥500.
`close()`	Close the scope and end the span. Handled for you by `withSpan` and `ObservedRun.run`.

Flagging failures without an exception

Not every failure arrives as a Throwable or an HTTP status — a validation rejection, a well-formed but unusable payload from a dependency, or a batch that completes with unrecoverable records are all real failures with nothing to catch. Use setErrorStatus for these:

span(name: 'importOrders').run { SpanRef span ->
    def result = orderImporter.run()
    span.setTag('rejectedCount', result.rejected.size())
    if (result.rejected) {
        span.setErrorStatus("Rejected ${result.rejected.size()} of ${result.total} orders")
    }
    result
}

Do not synthesize a throwable just to flag a span. Beyond the fabricated stack trace, Datadog's OTLP intake maps any exception event onto error.* tags, so an invented exception yields an invented error.type and error.stack.

Note that setErrorStatus is unconditional — unlike the recordException methods, it does not skip RoutineException cases, since the caller is explicitly asking for the status. Also note that span status is last-call-wins, so a later recordExceptionAndErrorStatus on the same span will overwrite an earlier description.

ObservedRun

File: src/main/groovy/io/xh/hoist/telemetry/ObservedRun.groovy

A composable builder for wrapping a closure with any combination of tracing, logging, and metrics. Each concern is opt-in via dedicated builder methods, then executed with run(). The closure is wrapped in an onion from outermost to innermost: span → log → metrics → closure.

Access via BaseService.span(name, kind?, tags?), which creates a builder pre-configured with the service as the caller (used for span code.namespace and log context) and an initial span. Additional builder methods (logInfo, timer, counter, etc.) can be chained as needed. BaseService.observe() returns the same builder without an initial span — use it only when no span is wanted.

Builder methods

Method	Description
`.span(name, kind?, tags?)`	Configure a trace span.
`.logInfo(msg)`	Log at INFO via `LogSupport.withInfo`.
`.logDebug(msg)`	Log at DEBUG via `LogSupport.withDebug`.
`.logTrace(msg)`	Log at TRACE via `LogSupport.withTrace`.
`.timer(name, tags?)`	Record elapsed time on the named Micrometer Timer.
`.counter(name, tags?)`	Increment the named Micrometer Counter.
`.run(closure)`	Terminal — execute with all configured observability.

Multi-level logging

When multiple log levels are configured, ObservedRun selects the finest enabled level at run() time. This allows callers to specify both a coarse and fine message — the finest enabled level wins:

span('importData')
    .logInfo('Importing data')
    .logDebug(['Importing data', [source: url, batchSize: n]])
    .run {
        // If DEBUG is enabled: logs with the detailed debug message
        // Otherwise: logs with the shorter info message
    }

Examples

Span + log + timer (most common pattern):

class PortfolioService extends BaseService {

    private Portfolio generatePortfolio() {
        span('generatePortfolio')
            .logInfo('Generating Portfolio')
            .timer('generatePortfolio')
            .run {
                // business logic
            }
    }
}

Both .timer() and .counter() add an xh.outcome tag of success or failure on completion, based on whether the closure threw. Metric names are prefixed with BaseService.telemetryPrefix when set on the owning service — pass useNamePrefix: false to opt out.

Span + log only:

span('generateOrders')
    .logDebug("Generating ${count} orders")
    .run {
        // business logic
    }

Span only (with SpanRef access):

For the common case of a span without logging or metrics, BaseService.span() is a shortcut for observe().span(...):

span(name: 'processOrder', tags: [orderId: id])
    .run { SpanRef span ->
        def result = doWork()
        span.setTag('resultCount', result.size())
        result
    }

Standalone (no BaseService):

ObservedRun.observe(this)
    .span('myOp')
    .logDebug('Working')
    .run {
        // works from any LogSupport implementor
    }

Configuration

`xhTraceConfig`

Property	Value
Type	`json`
Default	See below
Client Visible	Yes (client reads `enabled`, `sampleRate`, and `sampleRules` for browser tracing)
Purpose	Distributed tracing infrastructure configuration.

Default value:

{
    "enabled": false,
    "sampleRate": 1.0,
    "sampleRules": [],
    "jdbcTracingEnabled": false,
    "otlpEnabled": false,
    "otlpConfig": {}
}

Key	Type	Description
`enabled`	Boolean	Master switch for tracing. When false, all tracing is no-op. Dynamic.
`sampleRate`	Double	Fallback sampling rate (0.0–1.0) applied when no sampling rule matches. Dynamic.
`sampleRules`	List<Map>	Ordered rules for per-span sampling. Each rule has a `match` map of tag patterns (plus the reserved `name` key that matches the span's name) and a `sampleRate`. First match wins; unmatched spans use the fallback `sampleRate`. See Sampling Rules below. Dynamic.
`jdbcTracingEnabled`	Boolean	Emit CLIENT spans for all JDBC `DataSource` operations — applies to every pool (primary + any additional Grails datasources). Defaults to `false`. Dynamic. See JDBC below.
`otlpEnabled`	Boolean	Enable OTLP span export (HTTP/protobuf). Dynamic. In local development, additionally gated — see Local-development gating.
`otlpConfig`	Map	OTLP exporter config (e.g. `{"endpoint": "http://localhost:4318/v1/traces"}`).

When xhTraceConfig is updated, the exporter pipeline is torn down and recreated. This is handled by clearCaches() responding to the xhConfigChanged event.

Export Configuration

OTLP

When otlpEnabled: true, spans are exported via HTTP/protobuf to an OTLP-compatible backend (e.g. Jaeger, Grafana Tempo, Datadog). Configure the endpoint via otlpConfig:

{
    "enabled": true,
    "sampleRate": 1.0,
    "otlpEnabled": true,
    "otlpConfig": {
        "endpoint": "http://localhost:4318/v1/traces",
        "timeout": "30000"
    }
}

Local-development gating

OTLP export is suppressed by default when the app is running in local development, even when otlpEnabled: true in xhTraceConfig. This avoids polluting a shared OTLP backend with developer-machine spans during routine work. The same gating applies to metrics export — see metrics.md.

To opt in, set the otlpEnabledInLocalDev instance config to 'true'. Local-development detection follows Utils.isLocalDevelopment, which reflects the Grails runtime mode (Environment.isDevelopmentMode() — true when started via bootRun, false in a deployed war). This is independent of the configured appEnvironment, so a deployed instance configured as Development is not affected by this flag.

When OTLP export runs in local dev, the deployment.environment.name resource attribute is suffixed with the OS username (e.g. Development-johndoe) so per-developer data can be distinguished in a shared backend. Override ClusterConfig.getOtelResourceAttributes() if your backend prefers a different scheme.

Custom exporters

Applications can register additional exporters via traceService.addExporter(). These receive both server-generated and client-relayed spans.

Sampling Rules

Sampling rules provide fine-grained, tag-based control over which spans are sampled. Rules are evaluated at span creation time (head-based sampling) on both client and server. Each rule has a match map of tag patterns and a sampleRate — the first matching rule wins.

The reserved key name matches against the span's name (not a tag), using the same glob syntax as tag-value patterns.

name matches the name at creation time, not the final name. Request spans are created as the bare HTTP method (GET) and only renamed to {METHOD} {controller}/{action} after routing — so a rule like {"name": "GET xh/health*"} never matches. To target requests by route, match the url.path tag, which is set before the sampling decision.

Configuration

Add rules to the sampleRules array in xhTraceConfig:

{
    "enabled": true,
    "sampleRate": 0.1,
    "sampleRules": [
        {"match": {"url.path": "/xh/health*"}, "sampleRate": 0},
        {"match": {"xh.source": "hoist"}, "sampleRate": 0.01},
        {"match": {"user.name": "jsmith"}, "sampleRate": 1.0}
    ]
}

In this example: health-check requests are dropped entirely, framework-generated spans are sampled at 1%, spans from user jsmith are always sampled, and everything else falls back to the 10% default.

Pattern matching

String tag values support simple glob patterns:

Pattern	Matches
`*`	Any value
`foo*`	Values starting with `foo`
`*foo`	Values ending with `foo`
`foo`	Values containing `foo`
`foo`	Exact match

Non-string values (numbers, booleans) use strict equality.

Sampling flow

Tags are assembled on the span before the sampling decision.
If a valid parent context exists, the child inherits its decision — including an unsampled parent, which drops the child. Rules are not consulted.
Otherwise, sampleRules are evaluated against the span's name and tags. The first rule whose match entries all match produces the sampleRate for a probabilistic decision.
Unmatched spans use the fallback sampleRate.
Unsampled spans are dropped — not recorded, and never exported.

The client-side TraceService in hoist-react evaluates the same sampleRules config, so sampling decisions are consistent across client and server spans.

Built-in Instrumentation

Request spans (HoistFilter)

HoistFilter extracts incoming W3C traceparent headers from the request, restoring the client's trace context. It then wraps the whole of request handling — cluster readiness check, authentication, and chain.doFilter — in a SERVER span, which becomes a child of the client span when a traceparent was present. Because the span encloses auth, rejected requests are traced too.

Pings (/ping, /xh/ping), /xh/version, and websocket handshakes are not traced.

Name: created as the bare HTTP method (GET), the only information available pre-routing. HoistInterceptor then renames it to {METHOD} {controller}/{action} (e.g. GET portfolio/positions) and adds the http.route tag once routing resolves. See the note under Sampling Rules — the sampling decision sees only the pre-routing name.
Attributes (at creation): http.request.method, url.path, url.scheme, server.address, server.port, client.address, user_agent.original, xh.source=hoist. Then http.route from HoistInterceptor, and http.response.status_code when the span closes.
Exceptions escaping dispatch are rendered by Hoist's ExceptionHandler and then recorded on the span via recordException. Exceptions handled inside a controller action are recorded by BaseController.

Outbound HTTP (JSONClient)

All outbound HTTP calls via JSONClient automatically:

Create a CLIENT span named with the HTTP method (e.g. POST)
Inject W3C traceparent headers onto the outbound request

Attributes: http.request.method, url.full, server.address, server.port, http.response.status_code, xh.source=hoist

Proxy requests (BaseProxyService)

Proxied requests via BaseProxyService automatically:

Create a CLIENT span named with the HTTP method (e.g. GET)
Inject W3C trace context onto the proxied request

Attributes: http.request.method, url.full, server.address, http.response.status_code, xh.source=hoist

Outbound JDBC

At startup, JdbcTraceService.init() walks down each DataSource's proxy chain to the underlying raw pool and swaps it for an OpenTelemetryDataSource wrap. Wrapping at the bottom of the chain means every consumer sharing the chain — Spring DI, direct JDBC, Hibernate/GORM — is instrumented by a single wrap. The wrap is a no-op unless jdbcTracingEnabled is true on xhTraceConfig, so toggling JDBC tracing is a runtime config change — no restart or re-wrapping needed.

When enabled, CLIENT spans are emitted for each connection acquire and statement execution, parented under whatever span is active at query time — typically the request span created by HoistFilter, but also timer tasks, withSpan blocks, and cluster tasks.

Name: {operation} {schema}.{table} where derivable (e.g. SELECT xh_app_config), otherwise the bare operation.
Attributes: standard OTel DB semconv — db.system, db.namespace, db.statement, server.address, server.port, etc.

Enabling.

{
    "enabled": true,
    "jdbcTracingEnabled": true
}

jdbcTracingEnabled is inert unless the master enabled flag is also true.

Multi-datasource apps. Grails apps configured with multiple datasources (dataSource_reporting, sessionFactory_reporting, etc.) are handled automatically — JdbcTraceService iterates every Spring DataSource bean and every Hibernate SessionFactory, covering both direct-JDBC and GORM paths. The flag applies uniformly to all pools; there's no per-pool gating.

Context Propagation

Cluster tasks

ClusterTask captures the current traceparent string at construction time and restores it on the remote instance before execution. This means runOnAllInstances(...) calls from within a traced request produce child spans on remote instances.

The traceparent is a plain String field that serializes naturally with Kryo. When no active trace context exists at construction time, traceparent is null and context restoration is skipped.

Thread context propagation

At startup, HoistCoreGrailsPlugin installs a HoistPromiseFactory that wraps the default Grails PromiseFactory. This ensures that every task {} call — the primary async dispatch mechanism in Hoist — automatically carries the calling thread's OTel trace context to the worker thread, alongside the other framework context it propagates.

This covers all Grails task {} usage across the codebase (e.g. asyncEach, LdapService, MonitorEvalService, TrackService) without any per-call-site changes.

Client-to-server propagation

The client-side TraceService sends a traceparent header on every fetch request. HoistFilter extracts this header and restores the trace context, so the SERVER span it creates becomes a child of the client's span — producing end-to-end traces from browser interaction through server processing.

Client Span Relay

Browser-generated spans are batched and posted to the xh/submitSpans endpoint, which routes to TraceService.submitClientSpans(). That method converts each entry into an OTel SpanData (via ClientSpanData) — preserving the original trace/span IDs — and exports it through the same pipeline as server-generated spans. Client and server spans appear as a coherent distributed trace in the collector.

Client spans are pre-sampled in the browser using the shared sampleRules config — only sampled spans are relayed to the server.

Log Correlation

LogSupportMarker captures the active traceId at logging time. The default LogSupportConverter appends traceId=... to log output for ERROR-level and above, enabling correlation between logs and traces in observability platforms.

Resource Attributes

All spans include these resource attributes identifying the source instance:

Attribute	Value
`service.name`	Application code (e.g. `myApp`)
`service.instance.id`	Cluster instance name (e.g. `inst1`)
`deployment.environment.name`	Hoist AppEnvironment (e.g. `PRODUCTION`)
`service.version`	Application version

FilesExpand file tree

tracing.md

Latest commit

History

tracing.md

File metadata and controls

Distributed Tracing

Overview

Key capabilities

Source Files

TraceService

withSpan(args, closure)

createSpan(args) — framework-internal

addExporter(exporter) / removeExporter(exporter)

SpanRef

Flagging failures without an exception

ObservedRun

Builder methods

Multi-level logging

Examples

Configuration

xhTraceConfig

Export Configuration

OTLP

Local-development gating

Custom exporters

Sampling Rules

Configuration

Pattern matching

Sampling flow

Built-in Instrumentation

Request spans (HoistFilter)

Outbound HTTP (JSONClient)

Proxy requests (BaseProxyService)

Outbound JDBC

Context Propagation

Cluster tasks

Thread context propagation

Client-to-server propagation

Client Span Relay

Log Correlation

Resource Attributes

`withSpan(args, closure)`

`createSpan(args)` — framework-internal

`addExporter(exporter)` / `removeExporter(exporter)`

`xhTraceConfig`