Ahmadki/lens gym callsites r0.6.0 - #2850
Open
oyilmaz-nvidia wants to merge 1 commit into
Open
Conversation
Contributor
Author
|
/ok to test 4ecf3c3 |
ananthsub
approved these changes
Aug 28, 2026
…esses (#2647) ## What does this PR do? Stacked on #<base PR>. **Please review and merge that one first** — this PR targets `ahmadki/lens-gym-base`, so its diff is only the call sites. Attaches the instrumentation call sites to the base layer, so a rollout produces a single distributed trace spanning Gym's independent server processes. Everything here is a no-op unless the `telemetry` extra is installed and telemetry is enabled, so the default path is unchanged. ### Cross-process propagation is the substance; the spans hang off it A rollout crosses several Gym processes — the agent server calls the model server, both call the resources server — and nothing in a plain HTTP request tells the receiving server which rollout it belongs to. W3C Trace Context fixes that in two halves: - **Egress.** `server_utils.request()` is Gym's only outbound HTTP path — `AGENTS.md` bans httpx for async precisely to keep it that way — so it is the one place `traceparent` is injected. The retry loop moves into `_request_with_retries` so the gated wrapper can place one CLIENT span around the whole logical request rather than one per attempt. - **Ingress.** `SimpleServer.run_webserver` initialises telemetry once per process and applies OTel FastAPI auto-instrumentation, which extracts the inbound header and parents that server's SERVER span to the caller's. It also flushes on the way out, since a batch processor otherwise drops whatever is pending at exit. The result is a continuous parent chain across process boundaries, with the CLIENT/SERVER pairing a backend needs to draw the edge between two services. ### Spans at the seams Handlers are wrapped where routes are registered, which instruments every Gym server without touching any of them individually: `gym.rollout` on the agent's `/run`, `gym.agent.responses`, `gym.model.*`, `gym.verify`, `gym.sandbox.*`, and `gym.job` around rollout collection. Gym's existing `current_rollout_id()` is **bridged**, not replaced — it lands on spans as `nemo.gym.rollout.id`, so a trace, Gym's own logs and a captured trajectory still line up. The span starts inside `rollout_context` so the id is set by the time it is read. Every site checks its span group before doing any work, including building attributes, so a user who leaves telemetry off pays for one frozenset membership test per site. ### Getting telemetry into the server processes Server venvs install `nemo-gym[dev]`, not `nemo-gym[telemetry]`, so `head_server_deps` now carries the telemetry packages when telemetry is enabled. The requirement is derived from the nemo-lens distribution installed in the orchestrator rather than restated: the base PR's `[tool.uv.sources]` entry governs only dependencies resolved *through* the local project, so a bare `nemo-lens[sdk]` named on the uv command line would resolve from PyPI and silently skew the servers against the orchestrator. The container image gains the telemetry extra — a small set of pure-Python packages, without which tracing cannot be switched on with an environment variable. ### Deliberate omissions, in case they look like gaps - **`gym.server.request_duration_ms` is unused.** nemo-lens records it without attributes, so it would merge every endpoint of every server type into one histogram. The FastAPI instrumentor's `http.server.request.duration`, dimensioned by route/method/status, is used instead. - **`gym.servers.active` is written only by the CLI orchestrator.** It is a gauge, so a value written by every process would report whichever wrote last. - **Sandbox commands and URL query strings are never recorded**, since they carry model output, task content and credentials. Provider, exit code and duration are. - **Reward and accuracy are not mirrored into OpenTelemetry.** They are experiment telemetry and stay in `AggregateMetricsMixin`. ### Docs Adds an `observability` section under `fern/versions/latest/pages/` covering configuration, distributed tracing, span groups, metrics, and how to instrument your own server. Frozen `fern/versions/v0.*` snapshots are untouched. ## Checklist - [x] I have read the [contributing guidelines](https://docs.nvidia.com/nemo/gym/latest/contribute/development-setup). - [x] The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs. - [x] Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable). - [x] Pre-commit checks pass locally (`pre-commit run --all-files`) (so CI lint/format/copyright pass). - [x] All commits have DCO sign-off (`git commit -s`) (so the DCO check passes). --------- Signed-off-by: Ahmad Kiswani <kiswani.ahmad@gmail.com> Signed-off-by: Onur Yilmaz <oyilmaz@nvidia.com> Co-authored-by: Onur Yilmaz <oyilmaz@nvidia.com> Co-authored-by: Onur Yilmaz <35306097+oyilmaz-nvidia@users.noreply.github.com>
ananthsub
force-pushed
the
ahmadki/lens-gym-callsites-r0.6.0
branch
from
August 28, 2026 21:44
4ecf3c3 to
3061648
Compare
Contributor
|
🌿 Preview your docs: https://nvidia-preview-ahmadki-lens-gym-callsites-r0-6-0.docs.buildwithfern.com/nemo/gym Here are the markdown pages you've updated: |
kajalj22
approved these changes
Aug 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Checklist
pre-commit run --all-files) (so CI lint/format/copyright pass).git commit -s) (so the DCO check passes).