Skip to content

WIP: k8s lineages#230

Open
TrevorBasinger wants to merge 31 commits into
mainfrom
feature/k8s-lineage
Open

WIP: k8s lineages#230
TrevorBasinger wants to merge 31 commits into
mainfrom
feature/k8s-lineage

Conversation

@TrevorBasinger

@TrevorBasinger TrevorBasinger commented Jul 10, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Makes Kubernetes training workloads a managed roar lineage path. A user (or a
cluster admin, via the webhook) runs an unmodified training manifest and gets
file- and object-store-level lineage reconstituted into their local
.roar/roar.db — no changes to training code.

roar run kubectl apply -f train-job.yaml

plans the submit: pre-registers an encrypted fragment session with GLaaS,
rewrites the manifest so each container bootstraps the roar runtime around the
original command, submits, waits for completion, then fetches/decrypts/merges
the streamed fragments. Fire-and-forget submissions recover later with
roar k8s attach WORKLOAD.

Supported workloads: batch/v1 Job (plain and Indexed), JobSet, PyTorchJob
(training-operator v1), TrainJob (trainer v2), and RayJob (delegated to the
existing Ray backend). All validated live against the real controllers on a
KIND harness.

What's in the box

Capture (per pod, one fragment per container run)

  • File I/O via the preload tracer, child processes included.
  • Object-store I/O via botocore/aiobotocore hooks (S3 data ops as s3://
    refs with etag hashes; ranged GetObject recorded as byte_ranges).
  • Opt-in roar-s3-proxy sidecar for clients the hooks can't see (Go/Rust/CLI
    S3 clients) — requires image-staged runtime; re-signs with the workload's
    AWS credentials; refs deduped against hook captures.
  • FUSE mount-map rewriting (gcsfuse / Mountpoint CSI + [k8s.mount_map]
    config) so mounted-object I/O resolves to gs:///s3:// URIs; PVC mounts
    get pvc:// identity tags.

Identity & retries

  • Task identity pod_uid:container:completion_index:restart_attempt; node
    rank chain JOB_COMPLETION_INDEX → PET_NODE_RANK → pod RANK. Job
    retries land as attempt-distinct jobs, never conflated (chaos-tested).

Transport

  • Streaming-first: AES-GCM-encrypted fragment batches to GLaaS fragment
    sessions; Secret-delivered credentials (never inline in manifests; the
    prepared manifest is 0600 and deleted after kubectl reads it).
  • Session TTL renewal: the in-pod streamer renews on 403 and retries once
    (long jobs outlive TTLs); reconstituters renew on fetch, so late attach
    rescues expired sessions. Purely reactive — no client-side expiry clock.
  • Honest delivery: emit_fragment_dicts reports "streamed" only when every
    batch landed; partial/total failures fall through to the bundle fallback
    (k8s.bundle_dir + roar k8s ingest-bundles) with identity-dedup
    absorbing overlap.
  • Oversized fragments split after HTTP 413 are re-merged per path at
    reconstitution — splitting never discards refs.

Runtime staging

  • install mode: pip at container start.
  • image mode: hermetic per-ABI trees (cp310–cp313) staged by a
    roar-runtime init container — no network at pod start (~11s).
  • Wrapper is fail-open by design: missing python3, failed install, absent
    staged tree, and (via an import probe just before exec) broken installs or
    ABI mismatches all fall back to running the original command
    uninstrumented. An x86-64-only runtime image on an arm64 node degrades
    gracefully rather than failing training.

Zero-touch

  • Mutating admission webhook: namespaces opt in with
    roar.glaas.ai/lineage=enabled; plain kubectl apply by a roar-unaware
    client gets instrumented; roar k8s attach recovers the lineage.
    failurePolicy Ignore + allow-with-warning on any internal failure —
    lineage never blocks admission. The manifest rewrite runs before any
    external resource is created, so rejected admissions orphan nothing.
  • Helm chart (deploy/charts/roar-lineage-webhook): Deployment/Service,
    create-only Secret RBAC, MutatingWebhookConfiguration, TLS via supplied
    secret+caBundle or cert-manager, and a secretCleanup CronJob (own
    ServiceAccount, list/delete only, roar-fragment-* prefix-guarded) that
    ages out credential Secrets — ownerReferences are impossible at CREATE
    admission (no workload UID yet). The e2e harness deploys via this chart,
    so the webhook tests exercise the packaged artifact.

Ray fixes that fell out of this work

  • roar run ray job submit no longer self-conflicts on Ray's strict
    Job/driver runtime-env merge (duplicated ROAR_NO_TELEMETRY killed every
    instrumented driver on ray ≥ 2.4); driver-side additions now consult the
    injected job config and add only what the Job env lacks. Native compose
    harness went from dead-on-arrival to 57 passing.
  • Worker pathlib I/O (io.open) is captured; the earlier "ray 2.46 can't be
    wrapped" diagnosis was disproven and replaced with detect-and-warn.
  • RayJob-delegated ray tasks are parented to the recorded k8s submit job
    (fragment-level parent_job_uid via ROAR_DRIVER_JOB_UID), asserted live.

Design decisions worth reviewer attention

  1. Lineage never blocks training. Registration failure → uninstrumented
    submit with a warning. Wrapper failures → fallback exec. Webhook failures
    → allow-with-warning. The one residual window (crash inside pod_entry
    after the import probe passes) is documented, not overclaimed.
  2. Capture-first, filter/reconcile at reconstitution. Raw fragments stay
    immutable; mount-map rewriting, staging-noise filtering, split-part
    merging, and capture-method dedup all happen at merge time.
  3. Credentials via Secrets only. Never inline in CRs (Custom Resources) or prepared
    manifests that persist; webhook creates Secrets through the API.
  4. GLaaS-required at submit (not local-first). A deliberate v1 scope
    decision; a bundle-first/offline session mode is tracked as a
    design item since it touches the session trust model.
  5. kubectl matching contract. Verb anywhere + -f file + exactly one
    supported workload. Stdin/URLs/kustomize pass through uninstrumented.

@TrevorBasinger
TrevorBasinger marked this pull request as ready for review July 15, 2026 17:31
TrevorBasinger and others added 28 commits July 23, 2026 18:39
@TrevorBasinger @claude
test(k8s): add Tier-1 KIND lineage smoke harness
cafe0f5 
Phase-0 spike for the Kubernetes training lineage integration: a KIND
harness that proves the existing roar runtime works in vanilla training
pods before any roar.backends.k8s code exists.

- bootstrap_k8s.sh: pinned kind/kubectl download, 1cp+2worker cluster,
  dist/ wheel hostPath mounts, in-cluster glaas Service->host endpoint
  wiring, image pre-pull, pod->glaas preflight probe; destroy_k8s.sh
- hand-wrapped single-pod Job template modeling the future
  `roar k8s prepare` output: wheel staging, `roar run --tracer preload`
  command wrap, Secret-delivered fragment-session credentials,
  downward-API identity env
- roar-unaware synthetic trainer + wrapper that exports the recorded
  job as an ExecutionFragment (backend=k8s, task_id contract
  pod_uid:container:index:attempt) and streams it via the shared
  fragment transport
- smoke tests (k8s_e2e marker): traced reads/writes with hashes,
  identity metadata, and host-side merge into .roar/roar.db through
  the shared fragment lineage engine

Verified live: 3/3 passing repeatedly (~16s warm); default gate,
ruff, and mypy green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): add Kubernetes Job execution backend
7bde87d 
Phase 1 of the k8s training lineage integration: a `roar.backends.k8s`
backend that instruments plain batch/v1 Jobs submitted through
`roar run kubectl apply|create -f <manifest>`, gated by k8s.enabled.

- submit planning: pre-registers a GLaaS fragment session, rewrites the
  manifest (0600 prepared file + context sidecar), returns session_id so
  the shared submit finalizer reconstitutes after completion; degrades
  to an uninstrumented submit with a warning when GLaaS is unavailable
- manifest rewriter: wraps explicit container commands via a /bin/sh
  bootstrap that pip-installs the roar runtime and execs pod_entry,
  falling back to the original command on any bootstrap failure;
  injects the env/identity contract (Secret-backed session creds,
  downward-API pod identity, cluster-visible GLAAS_URL); appends the
  Secret document; fails actionably on missing explicit commands,
  generateName, or multiple Jobs
- pod_entry: roar init + roar run (preload) around the original
  command, then exports the recorded job as an ExecutionFragment with
  task identity pod_uid:container:completion_index:restart_attempt and
  streams it via the shared fragment transport (best-effort; training
  exit code always wins)
- host execution: kubectl submit, prepared-manifest cleanup, Job
  terminal-condition wait, submit job recorded with the ORIGINAL
  command and plan-generated parent uid so reproduce re-enters the
  backend and pod fragments link to the submit node
- reconstituter + K8S_FRAGMENT_LINEAGE_BACKEND merge k8s_task child
  jobs through the shared fragment lineage engine
- `roar k8s prepare` CLI for inspectable rewrites (no Secret embedded)
- shared roar.execution.fragments.export extracted from the OSMO bundle
  exporter (OSMO delegates, behavior unchanged)
- registry: k8s builtin (priority 95), config section, job-env marker

Verified: product-path e2e green on the KIND harness (4 tests: rewrite,
streaming, reconstitution into .roar/roar.db, secret hygiene) plus the
Phase-0 runtime smoke (7 total); 23 new unit tests; default gate 1993
passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): add distributed workload adapters and roar k8s attach
89cc0d7 
Phase 2 of the k8s training lineage integration on top of the Job
backend: operator kind adapters, multi-pod distributed capture, and an
attach flow for fire-and-forget submits.

- manifest rewriter generalized to a WORKLOAD_KINDS adapter registry:
  batch/v1 Job, jobset.x-k8s.io JobSet (nested replicatedJobs pod
  templates), kubeflow.org/v1 PyTorchJob (per-role replica specs), and
  trainer.kubeflow.org TrainJob (wraps the spec.trainer command/env
  override; requires an explicit trainer command). Task names carry the
  replica role; terminal wait conditions unioned across kinds in the
  new workload_wait module
- pod identity node-index chain extended: JOB_COMPLETION_INDEX ->
  PET_NODE_RANK -> pod-level RANK (PyTorchJob v1)
- roar k8s attach WORKLOAD: recovers the parent job uid and Secret name
  from the cluster object's own env contract, resolves credentials
  (local key -> cluster Secret -> --session-file), optionally waits,
  records a local attach job under the recovered parent uid, and
  reconstitutes streamed fragments — the CI flow
- harness: bootstrap --with-jobset installs the JobSet controller
  (v0.12.0); new distributed e2e covers a 2-pod Indexed Job (shared
  fragment session, distinct completion-index identities, child-process
  capture, cross-pod artifact edge with topological step ordering),
  attach from a fresh project via the cluster Secret, and the same
  worker as a JobSet through the real controller

Verified live on KIND: 12/12 e2e (5 new distributed + 4 product path +
3 smoke); 21 new unit tests (65 total k8s-related); default gate 2003
passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): add bundle-mode fallback and in-pod object-store I/O capture
c15c67c 
Completes the remaining Phase-2 transport/capture slices.

Bundle fallback (GLaaS-less pods):
- k8s.bundle_dir config names a mounted shared volume, injected as
  ROAR_K8S_BUNDLE_DIR; pod_entry probes GLaaS reachability upfront and
  writes roar-fragments-<pod>.json bundles there when streaming is
  unavailable (the streamer swallows per-batch POST failures and
  reports "streamed" regardless — hence the probe; surfacing streamer
  failure counts is a noted follow-up). Training never fails on
  lineage-transport outages
- roar k8s ingest-bundles DIR merges a host-visible copy of the bundle
  volume through the shared fragment lineage engine (bundles.py)

Object-store I/O hooks (direct S3 is HTTP, invisible to the tracer):
- the k8s backend now ships a RuntimeImportAdapter; the existing
  roar_inject.pth/sitecustomize dispatch (activated by ROAR_WRAP=1,
  which pod_entry sets for the traced tree) patches
  botocore.client.BaseClient._make_api_call plus aiobotocore's async
  variant (covers s3fs/fsspec) to append S3 data-op events to
  ROAR_K8S_OBJECT_IO_FILE (object_io.py); hooks only record after the
  real call succeeds, never raise into user code, and no-op when the
  env is unset
- pod_entry folds the events into the exported fragment as s3:// refs
  with etag hashes, deduped last-wins per (mode, path)

Verified live on KIND: 14/14 e2e — new bundle-fallback test (black-hole
cluster GLaaS URL, bundle pulled off the node with docker cp, ingested;
note /var/tmp hostPath because KIND nodes mount /tmp as tmpfs which
docker cp cannot read) and MinIO S3 test (host-staged dataset, pod
get->put captured as job inputs/outputs with etag artifact hashes).
10 new unit tests; default gate 2013 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): capture ranged S3 reads as byte_ranges through fragments
9f1aa74 
Closes the ranged-read fidelity gap that was previously the roar-proxy's
main advantage over the in-process hooks (the proxy itself is now
explicitly a Phase-3 opt-in webhook-injected sidecar for non-Python S3
clients — see the design doc's resolved open question 5).

- object_io: GetObject Range headers are parsed (fully-specified
  bytes=start-end pairs only) and recorded on events; the loader
  accumulates deduped ranges per (mode, path) across events while
  etag/size stay last-wins
- shared fragment path (additive): ArtifactRef gains an optional
  byte_ranges field with validating hydration, and the fragment merge
  engine now carries it into job_inputs/job_outputs.byte_ranges —
  previously fragments dropped ranges entirely (Ray/OSMO fragments
  gain the capability for free)

Verified live on KIND: MinIO e2e extended with a ranged GetObject,
asserting byte_ranges [[0,9]] on the s3:// input row; 14/14 e2e,
3 new unit tests, default gate 2015 passed, ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): add mount-map rewriting and retry-chaos coverage
3e4c711 
Completes Phase 2 of the k8s training lineage integration.

Mount-map rewriting (mount_map.py):
- per-container mount maps derived at rewrite time from inline CSI
  volumes the pod spec can name (GCS FUSE, Mountpoint-for-S3 ->
  gs://bucket / s3://bucket URIs, subPath-aware) plus an explicit
  [k8s.mount_map] config table for PVC-backed FUSE drivers whose bucket
  lives in the cluster-side PV; PVC mounts get pvc://claim identity
  tags and are never rewritten (their cross-pod edges connect through
  content hashes)
- injected as ROAR_K8S_MOUNT_MAP; pod_entry stamps the map into
  fragment metadata so capture stays raw and the mapping is auditable;
  collect_k8s_fragments rewrites ref paths at reconstitution with
  longest-prefix wins (streaming, bundle, and attach paths all covered)

Retry chaos e2e:
- a backoffLimit-1 Job whose first pod fails mid-run yields two
  attempt-distinct k8s_task jobs keyed by pod UID — one exit 1 with
  partial outputs, one exit 0 with final outputs, never conflated —
  proving the identity contract under real k8s pod recreation

Verified live on KIND: 16/16 e2e (chaos + mount-map new; mount-map via
config-declared map over a hostPath volume, asserting rewritten s3://
inputs/outputs and no raw mount paths leaking); 8 new unit tests;
default gate 2021 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): validate operators live and delegate RayJob to the Ray bac…
9a4d034 
…kend

Live operator validation:
- bootstrap --with-kubeflow installs training-operator v1 (v1.9.2) and
  trainer v2 (v2.2.1) + runtimes; --with-kuberay installs KubeRay v1.4.0
- PyTorchJob e2e proves the per-role adapter against the real
  controller (Master+Worker, RANK-chain node indices 0/1); TrainJob e2e
  runs the real TrainJob->JobSet pipeline via a slim-image clone of the
  shipped torch-distributed ClusterTrainingRuntime

RayJob delegation (rayjob.py, rewrite_style="rayjob"):
- entrypoint wrapped through the Ray driver entrypoint; runtimeEnvYAML
  merged with the roar pip requirement, worker setup hook, and the Ray
  env contract (ROAR_JOB_ID = the k8s parent uid so fragments link to
  the submit node); credentials only as Secret refs in the RayCluster
  pod templates; status.jobStatus terminal handling; reconstitution and
  attach delegate to the Ray backend's reconstituter (fragments are Ray
  TaskFragments merging as ray_task jobs)
- two live-found env-contract fixes: the merged local-proxy redirect
  (AWS_ENDPOINT_URL/ROAR_PROXY_PORT) is stripped (no proxy runs in the
  pods; user S3 traffic would hit a dead localhost port), and ROAR_WRAP
  stays off (the roar_inject.pth fires in Ray pip virtualenvs before
  site-packages are importable and the sitecustomize ABI-repair path
  blows the 60s worker registration timeout — observed as an infinite
  supervisor start/kill/retry loop)

Registry robustness (shared, found by the live smoke): backend plugin
imports that fail during early sitecustomize discovery were cached
forever ("unknown execution backend: ray" in worker setup hooks even
though deps import fine moments later); get_execution_backend now
retries skipped builtin imports on a lookup miss.

Known gap documented in the e2e and docs: per-task I/O fidelity inside
KubeRay workers under setup-hook-only bootstrap (fragments arrive as
ray_task shells without file refs) — a Ray-backend follow-up; the live
smoke asserts the delegation transport scope.

Verified live on KIND: 19/19 e2e (PyTorchJob + TrainJob + RayJob new);
10 new unit tests; default gate 2029 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(ray): capture pathlib I/O in workers and pin KubeRay e2e to ray 2.54
44a11a7 
Closes the per-task capture-fidelity gap on KubeRay found by the
delegation live smoke, with two root causes isolated by probing a real
worker (startup, open patch, executor patch, and task identity were all
verified engaged — yet fragments arrived empty):

- pathlib (Path.open/read_bytes/write_bytes) resolves io.open by module
  attribute, not the builtin, so the builtins-only open patch missed it;
  without the preload tracer (KubeRay pods) that I/O was invisible.
  roar_worker._startup now patches io.open alongside builtins.open. On
  preload-equipped native workers the duplicate python-level captures
  are absorbed by the existing native > python reconstitution dedup
- ray 2.46 does not route task execution through the patched
  FunctionActorManager.get_execution_info, so the per-task flush never
  engaged; the KubeRay e2e image is pinned to the native harness's
  ray 2.54 (rayproject/ray:2.54.0-py312-cpu) where the roar_worker
  internals are supported

The RayJob e2e's strict assertion (ray task file write present in
ray_task outputs) is restored and green.

Native-path regression check: identical compose-harness results with
and without this change (5F/1P parity runs; the failures are a
pre-existing runtime-env merge conflict on this dev box — job and
driver ray.init envs both carrying ROAR_NO_TELEMETRY under a host ray
2.54 CLI — tracked as a separate follow-up). Ray + k8s unit suites and
the full default gate are green.

Verified live on KIND: 19/19 e2e including the strict RayJob smoke;
default gate 2037 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): add roar-runtime image and image-staged runtime mode
2170433 
Phase 3 slice 1: hermetic runtime staging (closes the Phase-0 "wheel is
not hermetic" finding — pods no longer need network installs).

- deploy/roar-runtime/Dockerfile: per-ABI dependency trees (cp310-cp313,
  resolved by uv --python-version without needing each interpreter),
  each with a generated top-level sitecustomize.py mirroring
  roar_inject.pth (PYTHONPATH staging doesn't process .pth files); the
  image doubles as the staging init container (default CMD copies
  /opt/roar-runtime into the shared mount) and the webhook server base
- k8s.runtime_source=image + k8s.runtime_image config: the rewriter
  injects a roar-runtime-staging init container + emptyDir volume
  (idempotent by name for webhook reinvocation) and the wrapper
  activates the ABI-matched tree via PYTHONPATH instead of pip;
  TrainJob keeps the install path (no inline pod template for an init
  container); RayJob keeps its runtime-env pip mechanism
- scripts/build_runtime_image.sh; .dockerignore wheel exception

Verified live on KIND: image-mode e2e green in ~11s (vs ~16s+ for pip
mode) — staging init container present, no pip in the wrapper, lineage
captured; 4 new unit tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): add mutating webhook injector for zero-touch lineage
80ddc0a 
Phase 3 slice 2: the platform-install path. A stdlib HTTPS
AdmissionReview server (roar/backends/k8s/webhook.py, served by the
roar-runtime image) intercepts CREATE of all five workload kinds in
namespaces labeled roar.glaas.ai/lineage=enabled and applies the same
manifest rewrite as the CLI path — clients need no roar at all.

- per admission: mints the fragment session against GLaaS, creates the
  credentials Secret through the k8s API (never embedded in the
  object), rewrites via the shared rewriter (image-staged runtime by
  default), annotates the workload (roar.glaas.ai/parent-uid,
  session-id, fragment-secret), returns a JSONPatch
- never blocks admission: every internal failure returns allowed with
  a warning, paired with failurePolicy Ignore; dry-run admissions are
  side-effect-free (sideEffects NoneOnDryRun); idempotent under
  reinvocationPolicy IfNeeded via the parent-uid annotation
- reconstitution is client-driven via the existing roar k8s attach
  (reads the webhook-created Secret)
- harness: bootstrap --with-webhook builds/loads roar-runtime:dev and
  deploys via deploy_webhook.sh (self-signed CA, RBAC for secret
  creation, Deployment/Service/MutatingWebhookConfiguration); the
  image is always rebuilt (layer cache) — a stale image silently
  breaks the webhook
- fix found live: the API server posts /mutate?timeout=10s — path
  matching must strip the query string

Verified live on KIND: 22/22 e2e — zero-touch test does a plain
kubectl apply in a labeled namespace (no roar client), asserts
injection (annotations, staging init container, wrapped command), job
completion, and attach-recovered lineage keyed to the webhook-minted
parent uid; unlabeled-namespace control untouched. 7 new unit tests;
default gate 2040 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(ray): correct the 2.46 capture diagnosis and warn on unpatchable …
a32f2a8 
…executors

Closes the ray 2.46 circle-back — by disproving it. A local probe
(worker_process_setup_hook wrapping FunctionActorManager methods)
showed ray 2.46 DOES route task execution through get_execution_info;
the empty task fragments previously blamed on 2.46 were entirely the
pathlib/io.open capture gap, confounded because that fix landed
together with the 2.54 image pin, which took the credit.

- strict KubeRay e2e verified green on BOTH 2.46.0 and 2.54.0 with the
  final code; the e2e image is now overridable via ROAR_E2E_RAY_IMAGE
  (default stays the native harness's 2.54 pin) and rayVersion in the
  RayJob template tracks the image tag
- detect-and-warn insurance: _patch_ray_task_execution_for_native_flush
  now prints a loud stderr warning (with the ray version) when the
  function manager cannot be imported or exposes neither
  get_execution_info nor _make_actor_method_executor, instead of
  silently degrading to task fragments without identity or file refs
- developer doc and design doc corrected

Verified: strict RayJob e2e 1/1 on 2.46 and 1/1 on 2.54; ray + k8s
unit suites green; default gate 2040 passed; ruff + mypy clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): opt-in roar-proxy S3 sidecar for hook-invisible clients
e7342fc 
Add k8s.proxy_sidecar / k8s.proxy_upstream (and webhook
ROAR_WEBHOOK_PROXY_SIDECAR / ROAR_WEBHOOK_PROXY_UPSTREAM): the rewriter
appends a roar-s3-proxy native sidecar (init container with
restartPolicy Always) running the roar-proxy binary from the staged
runtime tree, so S3 traffic from clients the botocore hooks cannot see
(non-Python binaries, raw HTTP) still lands as s3:// lineage refs.

- Requires image staging; install mode warns and skips.
- roar-proxy re-signs upstream requests, so the sidecar inherits the
  workload container's AWS_* credential env entries.
- The proxy binds loopback only; the startup probe is an exec probe
  (kubelet tcpSocket probes dial the pod IP and never succeed).
- Wrapped containers get AWS_ENDPOINT_URL=http://127.0.0.1:19191 only
  when the user has not set their own (explicit endpoints win).
- pod_entry parses the proxy log (shared emptyDir) and folds refs into
  the fragment with capture_method="proxy", deduping objects the hooks
  already captured.
- Image-mode fragments no longer report the staged /roar-runtime/**
  tree as inputs (filtered at reconstitution; capture stays raw).

Live-validated on the KIND harness with a hook-invisible raw-urllib S3
client against MinIO (23/23 k8s e2e).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): package the lineage webhook as a Helm chart
a43bb3e 
Add deploy/charts/roar-lineage-webhook: webhook Deployment/Service,
RBAC (Secret-create ClusterRole), and the MutatingWebhookConfiguration
for all five workload kinds. TLS comes either from a supplied secret +
tls.caBundle (required-value guarded) or from an optional cert-manager
self-signed Issuer/Certificate with CA injection. injection.* values
map 1:1 onto the ROAR_WEBHOOK_* env contract, including the proxy
sidecar knobs.

The harness deploy script now generates self-signed TLS material and
installs this chart (helm downloaded into .tools/bin like kind/kubectl),
so the zero-touch webhook e2e proves the packaged artifact end-to-end;
the e2e webhook-presence check selects the configuration by chart labels
instead of the old raw-manifest name.

Validated: helm lint clean, cert-manager and manual-TLS renders, and
23/23 k8s e2e against the chart-deployed webhook.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(fragments): renew expired fragment sessions on 403
652fcd0 
Long training jobs routinely outlive the fragment-session TTL, and an
expired session 403s on both append and read — lineage was lost with no
recovery path. Pair with the glaas-api renewal endpoint
(POST /api/v1/fragments/sessions/:id/renew, token-authenticated and
allowed even after expiry):

- GlaasFragmentStreamer renews on a 403 batch POST and retries the
  chunk once; a second 403 gives up normally. Renewal is purely
  reactive — no client-side expiry clock.
- Both fragment reconstituters (k8s, ray) renew-and-retry on a 403
  fetch, so a late `roar k8s attach` recovers an expired session too.
- GlaasClient.renew_fragment_session for host-side callers.
- Streamer exposes delivered/failed/pending counters and close()
  reports flush success (used by the follow-up transport change).

Degrades gracefully against a server without the endpoint (renew 404s
-> previous behavior). Live-proven on the KIND harness: 60s-TTL session,
75s job, streamer renews mid-run and the finalizer reconstitutes
(tests/backends/k8s/e2e/test_k8s_ttl_renewal.py; skips when the local
glaas-api lacks the endpoint).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(fragments): report undelivered batches instead of claiming streamed
951e1d4 
emit_fragment_dicts returned "streamed" whenever credentials were
present, even when every batch POST failed — mid-run streaming failures
were silently lost because callers (k8s pod_entry bundle fallback, ray
local merge) only fall back on a non-"streamed" result. It now returns
"streamed" only when the streamer delivered everything; partial or
total failures log an undelivered-count warning and fall through to
local_merge / local_fallback with all fragments (downstream merges
dedupe by task identity, so overlap with delivered batches is
absorbed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(ray): stop duplicating roar env keys between Job and driver runti…
487180b 
…me envs

Every `roar run ray job submit` whose driver called ray.init() died with
"Failed to merge the Job's runtime env ... because of a conflict": the
submit rewrite delivers ROAR_NO_TELEMETRY (plus the rest of the env
contract) in the Job-level runtime env, and the patched driver-side
ray.init re-added ROAR_NO_TELEMETRY — ray >= 2.4 rejects the Job/driver
merge on ANY duplicated env-var key, even with identical values. The
failure was roar conflicting with itself, not a ray version issue.

The instrumented driver path now reads the job config ray injects
(RAY_JOB_CONFIG_JSON_ENV_VAR) and only adds what the Job env lacks
(same for py_executable — a job-level value wins with a warning). The
non-instrumented path gets a generic guard that drops roar-added env
keys the Job env already carries; user-supplied keys are left alone so
genuine conflicts still surface through ray's own error. The merged
runtime env that reaches workers is unchanged.

Live-validated on the compose harness: the previously-failing
runtime-env-conflict and native-tracing e2e pass, and the suite goes
from dead-on-arrival instrumented submits to 57 passing (remaining
failures are pre-existing capture-fidelity gaps newly reachable now
that submits run, tracked separately).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(k8s): merge 413-split fragment parts at reconstitution
0e1cdfb 
The streamer splits an oversized fragment into parts sharing one task
identity after an HTTP 413; the keep-last-per-identity dedup then
silently discarded every part but the last, losing most of a large
task's reads/writes. Retries are already attempt-distinct in the
identity contract (pod uid + restart attempt), so same-identity
fragments can only be duplicate deliveries or split parts — dedup now
unions reads/writes per path across them (last ref wins, last
fragment's scalars win). No wire-format change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(k8s,ray): resolve plan-time .roar at the project root; match kube…
25d7f96 
…ctl global flags

Both submit planners saved the fragment-session key (and k8s the
prepared manifest) under a bare cwd/.roar, while the run finalizer loads
the key from the context-resolved project root — submitting from a
project subdirectory created a stray nested .roar and lineage silently
never reconstituted locally. Planners now resolve the project .roar via
the same upward walk the CLI context uses (ROAR_PROJECT_DIR override,
bounded by the git root), shared as
roar.execution.fragments.sessions.resolve_project_roar_dir. Covered by
unit tests for both backends and a KIND e2e that submits from a nested
directory.

The kubectl matcher also accepted the verb only at argv[1], so common
forms like `kubectl --context prod apply -f job.yaml` silently bypassed
instrumentation. The verb now matches anywhere; the existing
-f-manifest and single-supported-workload guards keep false positives
out.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(k8s): parent RayJob ray tasks to the recorded submit job
c843de4 
The RayJob rewrite set ROAR_JOB_ID to the k8s parent uid but never
passed driver_job_uid to merge_worker_bootstrap_env, so workers had no
ROAR_DRIVER_JOB_UID and their fragments merged without a parent —
ray_task rows carried no DAG edge back to the k8s submit job (the merge
prefers the fragment-level parent, which was empty, and the host-side
fallback env is unset in this flow). The live RayJob e2e now asserts
every ray_task row parents to the recorded submit job, which is exactly
the assertion whose absence let this slip.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): webhook creates no resources before the rewrite; aged-Secr…
e631e50 
…et cleanup

The admission handler registered the GLaaS session and created the
credentials Secret before attempting the manifest rewrite, so a
rewrite failure (the step most likely to fail) orphaned both. The
rewrite — a pure function of the admitted object — now runs first;
external resources are only created once it succeeds.

Credential Secrets can never carry an ownerReference (the workload has
no UID at CREATE admission), so the chart gains a secretCleanup
CronJob (roar.backends.k8s.secret_cleanup, runs on the runtime image)
deleting roar-fragment-* Secrets older than the maximum session TTL.
It uses its own ServiceAccount with list/delete only; the webhook's
stays create-only. Running pods are unaffected (env resolves at pod
start) — only late attach credential recovery ages out with the
session itself.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
feat(k8s): pod-wrapper import probe; rayjob attach aliases
80c29c2 
The pod wrappers guarded python3/pip/staged-tree presence but exec'd
pod_entry unconditionally — an import failure past exec (broken
install, ABI mismatch such as the x86-64 staged tree on an arm64 node,
unsupported interpreter) killed the container with no way back. An
import probe now runs as the last step before exec and falls back to
the original command uninstrumented, which also makes non-x86-64
platforms degrade gracefully instead of failing training. The
remaining unisolated window (a crash inside pod_entry after the probe)
is documented instead of overclaimed.

Also adds the missing rayjob/rayjobs attach aliases (a
WORKLOAD_KINDS-completeness test keeps future kinds honest) and
mentions rayjob in the attach help text.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(k8s): preserve user RayJob runtime env during instrumentation
1378e2f 
The RayJob rewrite destroyed three kinds of user runtime configuration:
pip dicts were flattened to package lists (losing pip_check/pip_version),
a user worker_process_setup_hook was silently replaced with roar's, and
a user-supplied AWS_ENDPOINT_URL (MinIO, LocalStack, private S3) was
stripped along with roar's dead local-proxy redirect.

- keep the pip dict shape and append the roar requirement to packages;
  reject pip requirements-file references with an actionable error
- carry a displaced user setup hook as ROAR_USER_SETUP_HOOK; roar's
  worker_bootstrap.startup chains it after capture installs, and user
  hook failures propagate exactly as they would uninstrumented
- drop the local-proxy redirect from the merge source instead of popping
  the merged result, so user endpoints survive the rewrite

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(k8s): fall back to uninstrumented run when roar fails preflight
f01145b 
pod_entry returned roar run's exit code directly, so a tracer/runtime
failure before the workload launched failed the training job — breaking
the documented lineage-never-blocks-training contract (ARM nodes and
unsupported native boundaries were the likely triggers).

roar run already distinguishes pre-launch setup failures from workload
exits (ExecutionReport.setup_error: nonzero exit with no job created).
Expose it across the subprocess boundary: when ROAR_RUN_REPORT_FILE is
set, the run service writes {exit_code, setup_error, tracer_backend}
before finalizers run. pod_entry reruns the original command
uninstrumented only on a positively reported setup failure — a missing
or unreadable report is ambiguous (roar may have died after the
workload ran) and must propagate rather than risk double-running
non-idempotent training.

New KIND chaos e2e forces the failure with k8s.tracer=ebpf (valid mode,
cannot pass preflight in an unprivileged pod) and proves the Job still
completes with the workload's own outcome and no fabricated lineage.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(k8s): per-container bundle identity and multi-manifest submit guard
01dd9ff 
Bundle fallback derived its filename from the pod name alone, so two
wrapped containers in one pod (or an in-place restart) overwrote each
other's fragments last-writer-wins. Bundle names now carry
pod-container-attempt identity (attempt resolution mirrors the task_id
contract) and are written atomically via os.replace so ingest never
reads a torn bundle.

The submit matcher also accepted commands with several -f/--filename
arguments but rewrote only the first: one workload got lineage and
completion polling while the rest were applied unrewritten and
unwaited. Multiple manifest arguments now decline k8s instrumentation
with an actionable warning and leave the kubectl command untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(fragments): owner-only session keys; drop query-string renewal
72f6df9 
Session .key files carry the raw fragment token but were written with
write_text, inheriting the umask (0664 observed) — group/other could
read the credential. Keys are now created 0600 from the first byte and
swapped in with os.replace so a rewrite of an existing over-permissive
key also ends owner-only.

Also removes the unused GlaasClient.renew_fragment_session, which put
the raw token in the URL query string where access logs and proxies
capture it. The live renewal path (fragment_streamer) authenticates
with the x-roar-fragment-token header and stays.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(charts): ship the Secret-cleanup CronJob disabled by default
41cae18 
The cleanup CronJob defaulted on, granting its ServiceAccount
cluster-wide list/delete on Secrets — list responses include Secret
data, so a default install was one compromised cleanup image away from
cluster-wide credential read. Its fixed-age policy also conflicts with
session renewal: a renewed workload outliving maxAgeSeconds loses the
Secret its retry/scale-up pods reference, and attach recovery with it.

The CronJob is now opt-in, with both trade-offs documented in
values.yaml, the module docstring, and the developer doc. Liveness-aware
(controller-grade) cleanup remains deliberately deferred.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(k8s): parse the kubectl subcommand structurally
bd5bb07 
The verb-anywhere matcher accepted global-flag VALUES as the verb:
'kubectl --context apply delete -f job.yaml' and
'kubectl delete -f job.yaml --namespace create' both matched the submit
backend, which would register a session, rewrite a manifest being
deleted, and wait on a workload that no longer exists.

The matcher now finds the first non-flag token after skipping known
value-taking kubectl global flags (space and = forms); both review
commands are regression tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(k8s): make roar k8s prepare mirror the managed rewrite config
274871c 
prepare passed only requirement, GLaaS URL, tracer, Secret name, and
parent UID to the rewriter, so image-staged, bundle-enabled,
mount-mapped, proxy, or namespace-overridden configurations previewed
differently from what the managed submit would actually apply.

Both paths now share manifest_rewrite_config_kwargs (single source of
truth for config-derived rewrite kwargs) and the proxy-sidecar
cannot-inject warning; prepare gains -n/--namespace mirroring
kubectl apply -n semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
TrevorBasinger and others added 3 commits July 23, 2026 18:39
@TrevorBasinger @claude
fix(k8s): require exactly one wheel when building the runtime image
f4ced18 
The Dockerfile globs dist/, so multiple wheels silently produced an
image containing whichever sorted first — typically a stale build. The
build script now fails with the wheel list and prints which wheel a
successful build used.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
docs(k8s): state the local-first boundary and structural matcher
0e7ad68 
Managed capture requires host-side GLaaS at submit time; bundle_dir
covers pod-side outages only. Documented explicitly (with the prepare
escape hatch) until the deferred local-first design lands, and the
match description updated for structural verb parsing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@TrevorBasinger @claude
fix(k8s): isolate in-pod roar state per pod/container/attempt
86ad381 
pod_entry kept .roar/roar.db, object-io events, and the run report in
the workload's workdir. On a shared or persistent volume (emptyDir
shared by co-located wrapped containers, PVC surviving retries) every
participant shared one sqlite db and one events file — racing writes
and cross-attributing lineage between containers and attempts.

roar state now lives in pod-local storage keyed by the identity
contract (pod_uid:container:completion_index:restart_attempt); the
workload keeps its own cwd. RoarContext.create now honors an existing
ROAR_PROJECT_DIR — the contract config loaders and fragment planners
already follow — gated on the directory existing so Ray-propagated host
paths inside pods never resolve to a phantom project. An unusable state
dir falls back to an uninstrumented run rather than failing training.

New shared-workdir KIND e2e: two wrapped containers on one emptyDir
volume reconstitute as two distinct k8s_tasks, each owning only its own
outputs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@TrevorBasinger