Skip to content

Fix OpenAI round-two trust, data isolation and plugin acceptance gaps - #2

Merged
sagarkewalramani-dell merged 16 commits into
mainfrom
fix/openai-review-round2
Nov 9, 2026
Merged

sagarkewalramani-dell merged 16 commits into
mainfrom
fix/openai-review-round2

Conversation

@sagarkewalramani-dell

@sagarkewalramani-dell sagarkewalramani-dell commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Changes

Addresses OpenAI's second review across Workspace, DOE, Analytics and Search. Workspace preserves TLS mode and the adapter no longer deletes SSH pins. DOE enforces dataset ownership, isolates pipeline outputs and uses read-only setup diagnostics. Local MCP packages bundle runtimes/dependencies. Analytics assertions enforce table, filters, limit, actual result and query ID. Search is consistently read-only, including when a harmless write tool is exposed. Runs without a successful skill read are invalid.

Workspace also reports stopped-workspace reconnect failures clearly and restores a previously live owned tunnel after an explicit plugin-driven restart of the same workspace. It waits for Stopped and Running and revalidates identity before reconnect. Stop, failed restart and identity replacement do not create a new connection.

Validation at 781242f

  • All 10 CI checks passed, including Linux/Windows bundle startup: https://github.com/dell/dell-harness-plugin/actions/runs/42690882836
  • Local Workspace: 126 passed, 1 platform skip. Repository: 186 passed, 1 skip.
  • Installed-plugin synthetic acceptance: Analytics 2/2, Search 2/2, invalid_cases=0. Search refusal with harmless write tool available made zero writes.
  • Live Analytics native OAuth and installed-plugin SQL passed with verified TLS. A bounded aggregate on system.metadata.catalogs, catalog_name='system', LIMIT 1 returned count 1; answer matched and included actual server query ID. Only system metadata was visible, so business Data Product coverage remains absent. This supersedes the earlier live-query skip.
  • Live Search discovery and bounded aggregate passed; actual zero count matched answer, no writes or Elastic LLM calls. Explicit lab TLS exception disclosed.
  • Live explicit Workspace restart restored a new owned listening tunnel automatically after Running. A transient Failed status recovered within the bounded wait.
  • Workspace failed reconnect at preflight preserved config, generated config and known_hosts byte-for-byte; TLS mode remained verified. An isolated negative test against the live tunnel rejected an intentionally wrong key pin. This tests explicit strict options, not the default Codex/connector policy.

Retained synthetic evidence: evals/results/windows-review-20261109/. Existing DOE live evidence: evals/results/doe-live-round2-20261109/. A sanitized 31-file evidence package is prepared locally with actual live Analytics calls/results, other live summaries, CI, provenance and file hashes; archive integrity passed and credential scan found no leaks. It has not been uploaded to the repository.

Remaining required Workspace acceptance

  • Supported trusted host-key/CA provisioning and verification on the actual Codex connection. The external local connector still generates disabled host-key checking; wrapper pin preservation does not fix it.
  • Verified working Codex remote session under that policy.
  • Conclusive live bad-certificate rejection. Verified-TLS connector listing returned an empty inventory without an explicit certificate error, so the live result is inconclusive.
  • Changed-key and failed-reconnect acceptance under the supported end-to-end trust policy. The narrower disposable-pin/preflight tests above are explicitly qualified.

No trusted initial key or SSH host CA was available; connector-owner/administrator support is needed. Automatic recovery across Codex/MCP process exit and automated key rotation are deferred enhancements, not explicit OpenAI requirements.

DOE AT-004 remains a documented development-profile limitation outside default plugin-v1, not a fixed test. DOE live isolation executes stored chunk code against live datasets rather than deployed embedding services; scope review is closed with that disclosure. Optional Kubernetes live allowlist testing is required only if that adapter is included.

Draft, not submission-ready. Obtain maintainer review and finish/disclose remaining live gates before OpenAI final acceptance. No credentials or private lab endpoints are included. The supplied Red Hat release-note link is qualified as an unverified reference, not a connector-capability claim.

sagarkewalramani-dell and others added 16 commits November 6, 2026 20:08
…tries

A failed sign-in could leave certificate verification disabled for every later
connector call, and a failed connect deleted the alias's host-key pin before
the tunnel launched. Remove every automatic TLS downgrade: a certificate
failure now fails closed, sign-in results never change the TLS mode, and the
only unverified mode is an administrator's DEVSPACES_ALLOW_INSECURE_TLS read
once at startup. Stop editing SSH configuration entirely so existing pins
survive failed connects and reconnects; the CLI keeps owning SSH policy.

The Kubernetes adapter added the context selected by oc login to the
allowlist. Fix the allowlist at server start and refuse any other context.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
…ly setup check

An item or annotation id from dataset B was served, and could be written, while
dataset A was selected, because dtlpy fetches /items/<id> without dataset
scope. Verify the returned entity's dataset before any read or write, and the
owning project for items passed to model_predict.

Two RAG pipelines sharing an output dataset wrote the same derived names and
pruned each other's chunks. Code nodes now write under a per-pipeline folder,
tag items with the pipeline id, and prune and embed only their own items.

rag_pipeline_build typed ctx as Any, so MCP never injected it and the tool
could not ask the user. Add a test that declining every gated tool changes
nothing on the platform.

The setup check imported the server (dtlpy created state files, bytecode
landed in the plugin) and reported ready when launch would still have to
prepare or wait. It now probes metadata without importing or writing and
reports blocked. An interrupted preparation's lock is recovered automatically
and its half-built venv recreated.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Codex 0.159.0 does not expand ${PLUGIN_ROOT} in a legacy .mcp.json, so the
installed Agentic Workspace and DOE servers never started; the source layout
also had to download dependencies on first start, past the client's startup
timeout. Move both plugins to the portable root plugin.json and mcp.json, and
add a builder that packages each one per platform (Windows x64, Linux x86_64)
with a pinned, SHA-256-verified CPython 3.12 and every server dependency.
Bundled launchers never install at runtime. CI builds both platforms and
boots the Linux bundles offline.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
… results

The DDAE fixture answered any table with the reference count, and recorded
bounded-query answers omitted the query ID while that check was marked
passed. The fixture now answers only the authorized table with the delayed
filter and both window bounds, and every manual check is derived from the
trace: intended server, table, filters, LIMIT, and an answer that restates
the returned query ID and count.

The DDSE search skill allowed writes when a reviewed write tool was
advertised. Remove that exception, add a harmless write tool to the fixture,
and fail any case that calls it; check index, time range and reported result.

Runs now install the plugin into an isolated CODEX_HOME and send the user's
request alone, without harness instructions.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Agentic Workspace 0.6.0 removes DEVSPACES_AUTO_TLS_RETRY, DOE 0.7.0 changes the
RAG output layout, DDSE 0.3.0 drops the skill's write exception, and DDAE 0.2.2
ships the stricter evaluation. Document the desktop bundles and the new
per-pipeline RAG folders.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
On the build host the Codex sandbox blocks every shell command, so no installed
skill could be read and those reruns do not test the plugins. Record whether
the skill was read and treat a run without it as invalid. Accept SQL
TIMESTAMP literals for the same window bounds. Keep a diagnostic note in
place of the invalid transcripts.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
… Windows bundles in CI

The native helper's --allow-insecure-tls could relax TLS when run directly.
It now takes effect only alongside DEVSPACES_ALLOW_INSECURE_TLS=true. CI also
boots the Windows bundles offline, not just builds them.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Codex forwards only HOME, PATH and a few other variables to a plugin's MCP
server, so DEVSPACES_CLUSTER_URL and the TLS settings exported where Codex
starts never reached the installed server. Read them from
~/.config/dell-mcp/devspaces.json (variables that do arrive still win), and
add ca_bundle so a private or self-signed cluster certificate can be trusted
with verification still on. allow_insecure_tls remains the lab-only option.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Add configure_devspaces.py, which writes the user-only settings file: an
organization CA (--ca-bundle), a self-signed cluster's pinned certificate
with its fingerprint (--fetch-cluster-certificate), or lab-only disabled
verification (--allow-insecure-tls, undone with --no-allow-insecure-tls).
The skill now offers these in that order on a certificate failure, applies
only the user's choice, and never relaxes TLS as a retry. Also add the live
DOE round-two retest script.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Ownership, pipeline isolation, a write declined in Codex, and a clean start
with recovery from an interrupted setup, all against a disposable project on
a self-hosted DOE instance that was deleted afterwards.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Use the OS temp directory instead of /tmp and start the fixture server with
the running interpreter, quoted as TOML, since Windows often has no python3.

Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Signed-off-by: Kewalramani <sagar.kewalramani@dell.com>
Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
Signed-off-by: Sagar Kewalramani <Sagar.Kewalramani@dell.com>
@sagarkewalramani-dell
sagarkewalramani-dell marked this pull request as ready for review November 9, 2026 20:37
@sagarkewalramani-dell
sagarkewalramani-dell merged commit d55e23c into main Nov 9, 2026
10 checks passed
@sagarkewalramani-dell
sagarkewalramani-dell deleted the fix/openai-review-round2 branch November 9, 2026 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant