AsyncGRPO: train loop-owning agents on the engine's own token ids, with a Harbor/OpenEnv example

.github/workflows/tests-experimental.yml

@@ -7,6 +7,7 @@ on:
       - ".github/workflows/tests-experimental.yml"
       - "trl/experimental/**"
       - "tests/experimental/**"
+      - "examples/async_grpo_harbor/**"
 
 env:
   TQDM_DISABLE: 1
@@ -22,6 +23,37 @@ concurrency:
 permissions: {}
 
 jobs:
+  openenv_contract:
+    permissions:
+      contents: read
+    name: OpenEnv capture contract (CPU)
+    runs-on: ubuntu-latest
+    env:
+      PYTHONPATH: temp/openenv/envs
+      CUDA_VISIBLE_DEVICES: ""
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          repository: huggingface/OpenEnv
+          ref: 34825a772ae54760fb6bf7a8b2073a4a85714004
+          path: temp/openenv
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97  # v7.0.0
+        with:
+          python-version: '3.12'
+      - uses: astral-sh/setup-uv@bec219d24cd3e171d82865faccec33120bb574f4  # v10.1.0
+      - name: Install CPU test dependencies
+        run: |
+          uv venv
+          uv pip install torch --index-url https://download.pytorch.org/whl/cpu
+          uv pip install ".[test]" ./temp/openenv
+      - name: Check pinned producer and consumer
+        run: |
+          source .venv/bin/activate
+          python -c "import harbor_env.harness; import trl.experimental.async_grpo.openenv_harness"
+          python -m pytest -q tests/experimental/test_openenv_tito.py tests/experimental/test_harbor_example.py \
+            tests/experimental/test_async_grpo_trainer.py::TestReconciler tests/test_vllm_control_requests.py
+
   tests:
     permissions:
       contents: read

docs/source/example_overview.md

@@ -24,6 +24,7 @@ Check for additional optional dependencies [here](https://github.com/huggingface
 | Example | Description | Open in Colab |
 | --- | --- | --- |
 | [`async_distillation_math`](https://github.com/huggingface/trl/tree/main/examples/async_distillation_math) | Async on-policy distillation on GSM8K with [`experimental.async_distillation.AsyncDistillationTrainer`]: the teacher is served over HTTP with vLLM, including a multi-teacher (MOPD) math + code variant. | |
+| [`async_grpo_harbor`](https://github.com/huggingface/trl/tree/main/examples/async_grpo_harbor) | AsyncGRPO training on a [Harbor](harbor) task suite through any installed coding agent (`mini-swe-agent`, `opencode`, `claude-code`, ...) served over [OpenEnv](openenv): the agent owns its own loop in a sandbox and TRL trains on the token ids and logprobs recorded by the capture proxy. | |
 | [`async_grpo_math`](https://github.com/huggingface/trl/tree/main/examples/async_grpo_math) | Asynchronous GRPO on GSM8K with [`experimental.async_grpo.AsyncGRPOTrainer`], decoupling generation (vLLM server) from training, including a LoRA variant that syncs only the adapter to the vLLM server. | |
 | [`async_grpo_opencode`](https://github.com/huggingface/trl/tree/main/examples/async_grpo_opencode) | AsyncGRPO training of the real `opencode` coding agent on an [OpenEnv](openenv) environment (loop-owning: the external agent runs its own tool loop and TRL trains on its captured proxy trace), with a local subprocess sandbox or remote Hugging Face sandboxes. | |
 | [`dpo_reduce_hallucinations`](https://github.com/huggingface/trl/tree/main/examples/dpo_reduce_hallucinations) | DPO fine-tuning of a Vision Language Model to reduce hallucinations using the [openbmb/RLAIF-V-Dataset](https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset) dataset. | |

docs/source/openenv.md

@@ -627,11 +627,13 @@ Each rollout runs in its own isolated session. In the example that means one san
 You pass a `HarnessRolloutWorker` to [`experimental.async_grpo.AsyncGRPOTrainer`] with `harness_adapter=None` to select loop-owning mode. Besides the usual training arguments, you provide three functions (`rollout_reward_fn`, `train_turn_fn`, and `agent_turn_fn`) that tell TRL how to score, filter, and read the agent's rollouts. They are described in [What you need to define](#what-you-need-to-define).
 
 ```python
+from functools import partial
+
 from trl.experimental.async_grpo import AsyncGRPOConfig, AsyncGRPOTrainer
 from trl.experimental.async_grpo.openenv_harness import HarnessRolloutWorker, has_tool_call
 
 worker = HarnessRolloutWorker(
-    harness_session_factory=build_factory(...),  # your OpenEnv ResourceSessionFactory
+    harness_session_factory=partial(build_factory, ...),  # accepts sampling= and returns a ResourceSessionFactory
     harness_adapter=None,                         # loop-owning: the agent runs its own loop
     rollout_reward_fn=my_reward,                  # outcome -> float | None
     train_turn_fn=has_tool_call,                  # reinforce only action turns

examples/async_grpo_harbor/README.md

@@ -0,0 +1,61 @@
+# AsyncGRPO with Harbor
+
+An installed agent runs each task through OpenEnv. TRL trains on the engine's captured token IDs,
+logprobs, and loss masks, then updates the same vLLM instance.
+
+## Local setup
+
+Use Python 3.12 and this TRL checkout. Pin OpenEnv to the merge of
+[OpenEnv #1036](https://github.com/huggingface/OpenEnv/pull/1036):
+
+```sh
+git clone https://github.com/huggingface/OpenEnv.git OpenEnv
+git -C OpenEnv checkout --detach 34825a772ae54760fb6bf7a8b2073a4a85714004
+pip install -e . trackio datasets ./OpenEnv
+export PYTHONPATH="$PWD/OpenEnv/envs${PYTHONPATH:+:$PYTHONPATH}"
+```
+
+The source path is required because this revision's environment wheel does not package
+`harbor_env.harness`. Both the core package and the harness above use the same revision.
+
+Run the OpenEnv server with its sandbox credentials, then start vLLM and the trainer using the commands
+in [async_grpo_harbor.py](async_grpo_harbor.py). The server additionally needs `pip install './OpenEnv[harbor]'`;
+the trainer can use an existing server. Use the same model and vLLM endpoint for rollouts and weight updates.
+Set the server's `MAX_CONCURRENT_ENVS` to at least `--max-inflight + 1` (9 for the defaults); the factory
+keeps one connection for task metadata. The Jobs launcher sets this capacity automatically.
+
+Training saves a checkpoint every 50 steps, keeps the latest three, and saves the final model and
+tokenizer in the output directory on successful completion, including runs shorter than 50 steps.
+
+## Hugging Face Jobs
+
+[launcher.py](launcher.py) starts the server, vLLM, and trainer on one two-GPU job. It installs the pinned
+OpenEnv runtime and checks out the matching harness source. Pass `HF_TOKEN` and the sandbox credential
+as job secrets. A bucket mounted at `/data` preserves checkpoints and logs.
+
+Use `--train-script-url` with the raw URL of the TRL revision being tested before the example is merged.
+Use `--tunnel-check-s 0` to disable tunnel supervision. A server restart interrupts existing sessions;
+subsequent sessions receive the new proxy URL.
+
+## Capture contract
+
+- Prompt and completion IDs come from the inference engine; TRL does not re-tokenize captured prompts.
+- `loss_mask` covers prompt plus completion. Prompt positions are zero; completion positions may mix
+  zeros and ones. TRL maps the completion span to its generic `TurnRecord.output_mask`.
+- For example, prompt `[10, 11]`, completion `[12, 13, 14]`, and mask `[0, 0, 1, 0, 1]` train on tokens
+  `12` and `14`, while retaining `13` as context. A whole-turn filter cannot express this selection.
+- The worker constructs the session factory with its sampling policy. Pass a callable such as
+  `partial(HarborSessionFactory, server_url, ...)`; do not configure sampling separately on the factory.
+  OpenEnv applies the policy to captured requests.
+- Invalid captures stop the worker. Transport failures remain unscorable; an agent timeout retains
+  valid captured turns and the verifier's score.
+
+Rewritten histories can produce multiple training rows. Token retention does not guarantee equal
+rollout weighting or a suitable memory budget for every harness.
+
+Run the CPU contract and launcher checks with the pinned checkout on `PYTHONPATH`:
+
+```sh
+python -m pytest tests/experimental/test_openenv_tito.py tests/experimental/test_harbor_example.py \
+    tests/experimental/test_async_grpo_trainer.py::TestReconciler tests/test_vllm_control_requests.py -q
+```