fix(CubeAPI): propagate create-time env vars to envd-backed commands by zyl1121 · Pull Request #566 · TencentCloud/CubeSandbox

zyl1121 · 2026-06-15T06:56:01Z

Motivation

For envd-backed command execution, create-time environment variables should be propagated through the sandbox create path and initialized inside the sandbox runtime.

On the current CubeAPI sandbox creation path, this propagation is incomplete:

the E2B SDK sends create-time environment variables as envs, while CubeAPI only accepted envVars
create-time environment variables are not forwarded end-to-end to the cubelet/envd initialization path used by envd-backed commands

As a result, envd-backed execution such as commands.run cannot see environment variables passed at sandbox creation time.

This PR does not introduce a generic sandbox startup-time environment refresh model across all runtime paths.

What this PR changes

This PR fixes the create-time env propagation chain for envd-backed command execution:

accepts envs as an alias of envVars in NewSandbox, so both E2B SDK and CubeSandbox SDK callers are supported
validates create-time env var names and rejects dangerous control-character values before sandbox creation
forwards create-time env vars from CubeAPI into CubeMaster as create_time_env_vars
serializes the same env map into an internal annotation for cubelet
lets cubelet trigger envd /init after sandbox startup and probe success
uses the propagated envd support annotation, cube.master.components.envd.version, when present
falls back to probing the default envd init endpoint with the same bounded retry when that annotation is missing, so older templates remain backward compatible
fails sandbox creation if cubelet cannot complete envd initialization after startup
adds focused tests for alias deserialization, env forwarding, annotation serialization, cubelet-side envd /init, bounded retry, and the missing-annotation fallback path

On the main CubeMaster-managed create path, failures after sandbox allocation are still covered by CubeMaster's async destroy compensation. The additional cubelet-side cleanup keeps the runtime-local fail-fast path self-contained and also protects direct cubelet callers.

Scope

This PR restores create-time environment variables for envd-backed execution, such as commands.run.

It does not change run_code. run_code goes through the bundled lightweight code interpreter instead of envd-backed process execution, and that interpreter does not currently read envd /envs.

It also does not claim generic startup-time env refresh support for every sandbox runtime path.

Testing

Unit tests:

cargo check --manifest-path CubeAPI/Cargo.toml --bin cube-api
(cd CubeMaster && go test -vet=off ./pkg/service/sandbox -run TestSetCreateTimeEnvVarsAnnotation -count=1)

Cluster validation:

from e2b_code_interpreter import Sandbox

with Sandbox.create(
    template="tpl-xxxx",
    envs={"CUBE_TEST_CREATE": "from-e2b-create"},
) as sandbox:
    print("commands.run:")
    print(sandbox.commands.run("echo $CUBE_TEST_CREATE").stdout)

    print("run_code:")
    result = sandbox.run_code(
        "import os; print(os.environ.get('CUBE_TEST_CREATE', '<NOT SET>'))"
    )
    print(result.logs.stdout)

Before this PR, on current origin/master:

commands.run:


run_code:
['<NOT SET>\n']

After this PR:

commands.run:
from-e2b-create

run_code:
['<NOT SET>\n']

Verified behavior:

verified the E2B SDK envs path
verified the CubeSandbox SDK native envVars path
verified that commands.run sees create-time env vars after sandbox creation
verified that templates without envd support annotation still probe the default envd init endpoint instead of being rejected upfront
verified that sandbox creation returns an explicit error if that fallback envd init probe still fails
verified that envd-backed templates still initialize envd successfully and expose create-time env vars to commands.run
verified that per-call command env vars override sandbox-level env vars
verified that after a per-call override, the next commands.run falls back to the sandbox-level env vars
verified that run_code remains <NOT SET> on the current runtime image
verified on a real cluster that fail-fast after sandbox allocation does not leave a zombie sandbox on the main CubeMaster-managed path

Compatibility

This change is backward compatible for existing create requests:

existing envVars requests continue to work
E2B SDK envs requests now work
sandbox creation without create-time env vars continues to work for legacy templates
sandbox creation with create-time env vars now falls back to probing the default envd init endpoint when the template does not carry the envd support annotation
per-call command environment variable precedence is unchanged

Migration / Compatibility Note

When present, cubelet uses the propagated envd support annotation:

cube.master.components.envd.version

Templates created before envd capability reporting was introduced may not have this annotation yet. For those templates:

sandbox creation without create-time env vars continues to work as before
sandbox creation with create-time env vars falls back to the default envd init endpoint with bounded retry instead of being rejected upfront
if fallback init still fails, sandbox creation returns an explicit error instead of silently dropping the requested env vars
templates rebuilt through the full BOOT -> SNAPSHOT flow will carry explicit envd support metadata and avoid relying on the compatibility fallback
support for non-default envd init endpoints remains a separate follow-up; the current fallback assumes the default endpoint

For envd-backed command execution, precedence remains:

sandbox-level env vars from create-time init < per-call command env vars

Documentation

This PR includes a small quickstart visibility update in examples/code-sandbox-quickstart:

README.md
README_zh.md
create_with_envs.py

The behavior change is intentionally narrow: the sandbox create path now propagates create-time environment variables through CubeAPI, CubeMaster, and cubelet, and treats failed envd initialization as a create failure for envd-backed command execution.

The remaining run_code behavior is tracked in the linked issue and requires runtime/image-side support in the bundled lightweight code interpreter.

Related issue

Refs #565

cubesandboxbot · 2026-06-15T07:02:07Z

+            return Ok(());
+        };
+        let url = build_envd_init_url(base_url, sandbox_id);
+        let resp = self


Missing HTTP timeout on envd init call – The shared reqwest::Client in state.rs (line 46-50) is built without .timeout(), which means reqwest::Client defaults to no timeout. If the envd proxy is unresponsive, this POST will hang indefinitely, blocking the sandbox creation endpoint for that handler. Since this call is best-effort (failure is logged and swallowed), it should have a tight timeout so it fails fast.

Suggested change

let resp = self

let resp = self

.http_client

.post(url)

.json(&EnvdInitRequest { env_vars })

.timeout(std::time::Duration::from_secs(10))

.send()

.await

.map_err(|e| AppError::Internal(anyhow::anyhow!("envd init request failed: {}", e)))?;

cubesandboxbot · 2026-06-15T07:02:15Z

+        let url = build_envd_init_url(base_url, sandbox_id);
+        let resp = self
+            .http_client
+            .post(url)


Missing auth header on envd init request – Every other call to the sandbox proxy (e.g. run_envd_command in agenthub.rs:2162) sends Authorization: Basic cm9vdDo=. This new init_sandbox_env_vars method sends no auth header. If the downstream /init endpoint enforces the same proxy-level auth, this will silently fail on every request (logged at warn, caller sees success). If it does not enforce auth, this is an inconsistency that may widen the attack surface.

Suggested change

.post(url)

.http_client

.post(url)

.header("Authorization", "Basic cm9vdDo=")

.json(&EnvdInitRequest { env_vars })

cubesandboxbot · 2026-06-15T07:02:23Z

+
+    async fn init_sandbox_env_vars(
+        &self,
+        sandbox_id: &str,


Silent skip when env vars are provided but no proxy URL is configured – When sandbox_proxy_base_url is None and env_vars is non-empty, the init call is skipped silently. The caller (create_sandbox) only logs a warning when the HTTP call itself fails, not when it's skipped. This makes it hard for operators to debug why client-provided env vars are being silently ignored. Consider adding a tracing::warn! here when env_vars is non-empty but the proxy URL is unset.

cubesandboxbot · 2026-06-15T07:02:38Z

+    env_vars: &'a HashMap<String, String>,
+}
+
+pub(crate) fn default_sandbox_proxy_base_url() -> Option<String> {


pub(crate) function should have a doc comment – default_sandbox_proxy_base_url is visible outside this module (called from services/mod.rs:104) but has no doc comment. It reads three environment variables with a fallback chain. A doc comment explaining the env vars (AGENTHUB_SANDBOX_PROXY_URL, CUBE_SANDBOX_NODE_IP, CUBE_PROXY_HTTP_PORT), their meanings, and the fallback priority would help future maintainers.

cubesandboxbot · 2026-06-15T07:02:45Z

+            Path((sandbox_id, port)): Path<(String, u16)>,
+            Json(body): Json<Value>,
+        ) -> StatusCode {
+            assert_eq!(sandbox_id, "sb-123");


Assertions inside spawned test handler are invisible on failure – These assert_eq! calls run inside a tokio::spawn-ed axum handler. If they fail (say from a code change that alters the URL path), the panic is caught by tokio and logged to stderr, but does not cause the test to fail. The test would silently pass because the handler returning an error response triggers the best-effort fallback. Consider returning the received values in the response and asserting from the caller side, or using a tokio::sync::oneshot channel to propagate assertion failures.

kinwin-ustc · 2026-06-15T07:15:40Z

Are you doing the same thing as PR #554?

zyl1121 · 2026-06-15T07:28:47Z

Are you doing the same thing as PR #554?

Thanks for pointing this out. It is related to #554, but the fix is applied at a different layer.

#554 calls envd /init from the CubeSandbox SDKs after Sandbox.create(env_vars=...) succeeds. This PR moves that propagation into CubeAPI itself: CubeAPI accepts both envs and envVars, and initializes envd after CubeMaster creates the sandbox.

The main reason is compatibility with clients that do not use the CubeSandbox SDK, especially E2B SDK users, since the E2B SDK sends envs instead of envVars. With this PR, any client creating sandboxes through CubeAPI gets the same create-time env propagation for envd-backed command execution, without implementing the /init call client-side.

cubesandboxbot · 2026-06-15T08:03:11Z

Review: PR #566 — Propagate create-time env vars to envd-backed commands

Overall: This is a well-structured PR with clear data flow (CubeAPI -> CubeMaster -> Cubelet -> envd), thorough input validation, and backward compatibility. The design is sound and the happy-path test coverage is solid. Below are findings that merit attention before merge.

1. Zombie sandbox on envd init failure (High)

Cubelet/services/cubebox/cube_container_create.go:339

When doCreateTimeEnvdInit fails, the function returns an error but containers have already been created, probed, and post-create hooks have run (lines 331-337). There is no cleanup. The sandbox becomes a zombie consuming resources.

Fix: tear down containers on envd init failure, or at minimum mark the sandbox as failed for garbage collection.

2. Synchronous envd init blocks create path (Medium)

Cubelet/services/cubebox/cube_container_create.go:339

The envd init HTTP call blocks sandbox creation for up to 10s. This directly inflates p95/p99 create latency. Consider a shorter dial timeout or asynchronous init.

3. No aggregate env var payload size limit (Medium)

CubeMaster/pkg/service/sandbox/util.go:692

Individual values are capped at 4KB, but there is no limit on total payload size. Hundreds of large env vars could exceed the approx 256KB Kubernetes annotation limit. Add a cap after marshaling.

4. Documentation: Missing files in directory listing (Medium)

examples/code-sandbox-quickstart/README.md and README_zh.md

The directory tree is missing create_with_envs.py (the new file) and env_utils.py (shared loader). Both should be added to the listing.

5. Documentation: envs vs env_vars parameter name (Low)

examples/code-sandbox-quickstart/create_with_envs.py:13

The example uses envs=... but the SDK Sandbox.create() uses env_vars=. It works via **kwargs but wont appear in IDE autocompletion. Consider using env_vars= for consistency.

Summary

Priority	Issue
High	Zombie sandbox on envd init failure - no resource cleanup
Medium	Synchronous envd init blocks create path (10s timeout)
Medium	No aggregate env var payload size limit
Medium	Missing file entries in README directory listings
Low	Example uses envs= instead of documented env_vars=

Inline comments cover the HTTP client redirect behavior, io.ReadAll error discard, mutable global port variable, and test race conditions in more detail.

Overall the approach is sound and core functionality works correctly. The main concerns are operational reliability (zombie cleanup, aggregate size limits).

kinwin-ustc · 2026-06-29T06:43:55Z

That would move the main readiness guarantee to template construction time, instead of rediscovering the same race on every sandbox creation. The create-time retry can still cover restore jitter, but the heavier validation should happen on the lower-frequency template creation path to keep hot-start fast.

Yes, we can probe envd's /health at the stage of creating templates to ensure that envd has started and is ready

zyl1121 · 2026-06-29T06:44:12Z

@kinwin-ustc Added a short bounded retry for envd init.

The current behavior is:

3 attempts max
150ms per attempt
25ms between attempts
retry only for transient readiness cases such as transport errors and 502/503/504

This keeps fail-fast semantics while tolerating a small restore/readiness jitter window.

zyl1121 · 2026-06-29T06:53:42Z

Yes, we can probe envd's /health at the stage of creating templates to ensure that envd has started and is ready

Sounds good. Later I can work on a follow-up to probe envd /health during template creation and make envd readiness part of the envd-capable template contract.

kinwin-ustc · 2026-06-29T11:48:52Z

Another issue worth discussing is forward compatibility. The sandbox.create with envs request would successfully create a sandbox in previous versions, but the envs would not take effect. After merging this PR, the same request would cause the sandbox creation to fail because the template was not rebuilt without envd information and was rejected?

zyl1121 · 2026-06-29T12:06:31Z

Another issue worth discussing is forward compatibility. The sandbox.create with envs request would successfully create a sandbox in previous versions, but the envs would not take effect. After merging this PR, the same request would cause the sandbox creation to fail because the template was not rebuilt without envd information and was rejected?

I agree this changes behavior for legacy templates.

I kept the explicit rejection mainly to align with the direction of using the #650 envd annotation as the capability signal. My understanding is that if create-time env vars are requested, a missing cube.master.components.envd.version means we cannot guarantee envd-backed init will work.

If we silently fall back to probing the default envd endpoint when the annotation is missing, the capability check becomes more like runtime guessing again, which weakens the #650 signal for this path.

So the current behavior is intentionally stricter: ordinary sandbox creation without env vars remains backward compatible, but create-time env injection requires a template carrying the envd capability annotation.

I agree this should be clearly documented as a compatibility / migration note. If we want legacy-template fallback, I think that should be a separate policy decision or follow-up.

kinwin-ustc · 2026-06-30T01:48:29Z

I believe backward compatibility is very important. Imagine that someone has already built thousands of templates, and redoing the templates would be a huge amount of work. Perhaps we can judge the availability of envd through the probe itself. If envd is not installed, then 49983 is likely not listening, and it will immediately return reset. If 49983 is being listened to by non-envd, it will not respond correctly to our requests and will immediately fail in very quick retries. This can ensure that users who have already deployed it can enjoy the new features very lightly, and it will not have any impact on the semantics of the function itself

zyl1121 · 2026-06-30T03:05:47Z

@kinwin-ustc Updated for backward compatibility.

If create_time_env_vars are requested but the template does not carry the envd support annotation, cubelet now falls back to probing the default envd init endpoint with the same bounded retry. If init still fails, it returns an explicit error.

Also added focused tests for the missing-annotation fallback path.

kinwin-ustc · 2026-06-30T03:58:07Z

LGTM

ls-ggg · 2026-06-30T07:38:03Z

@liciazhu Could you please help test this pr?

liciazhu · 2026-06-30T07:39:51Z

@liciazhu Could you please help test this pr?

OK

liciazhu · 2026-06-30T10:08:52Z

TestCleanupAfterEnvdInitFailurePassesExpectedContextAndRequest constructs CubeBox with Metadata.Namespace = "ns-cleanup", but cleanupAfterEnvdInitFailure reads sandBox.Namespace (the top-level field on CubeBox), which shadows Metadata.Namespace due to Go's field promotion rules. As a result, the top-level Namespace is empty, namespaces.WithNamespace is never called, and namespaces.NamespaceRequired(ctx) fails.

Fix: set CubeBox.Namespace directly instead of CubeBox.Metadata.Namespace.

zyl1121 · 2026-06-30T11:17:57Z

TestCleanupAfterEnvdInitFailurePassesExpectedContextAndRequest constructs CubeBox with Metadata.Namespace = "ns-cleanup", but cleanupAfterEnvdInitFailure reads sandBox.Namespace (the top-level field on CubeBox), which shadows Metadata.Namespace due to Go's field promotion rules. As a result, the top-level Namespace is empty, namespaces.WithNamespace is never called, and namespaces.NamespaceRequired(ctx) fails.

Fix: set CubeBox.Namespace directly instead of CubeBox.Metadata.Namespace.

Fixed, I reran the focused tests, including the new TestSetCreateTimeEnvVarsAnnotation coverage, and they all passed.

Create-time env vars were dropped during sandbox creation, so envd-backed command execution (e.g. commands.run) could not see them. Forward env vars from CubeAPI into CubeMaster, serialize them into an internal annotation, and have cubelet initialize envd via its /init data-plane endpoint after sandbox startup. Gate on the propagated envd capability signal (cube.master.components.envd.version) and fail fast when injection is requested on a template without it. Scope is limited to the envd data-plane init path; container-level startup env refresh is intentionally not included. Add focused tests across CubeAPI/CubeMaster/cubelet and a quickstart example. Signed-off-by: zhengyilei <zheng_yilei@qq.com> Co-authored-by: jinlong <jinlong@tencent.com>

kinwin-ustc · 2026-07-01T04:36:17Z

+	})
+}
+
+func (l *local) destroySandbox(ctx context.Context, opts *workflow.DestroyContext) error {


This function name can be more strictly limited to cleaning up after envd init fails; the current name is easily mistaken for the main entrance to destroy

This function name can be more strictly limited to cleaning up after envd init fails; the current name is easily mistaken for the main entrance to destroy

Good point. I narrowed the helper name to destroySandboxAfterEnvdInitFailure.

Keep create-time env var injection backward-compatible for templates built before envd capability propagation. When create_time_env_vars are requested and the template lacks the envd support annotation, probe the default envd init endpoint with bounded retry instead of rejecting upfront. If init still fails, return an explicit error instead of silently dropping the env vars. Also surface missing-annotation context in failure messages and add focused tests for the fallback path. Signed-off-by: zhengyilei <zheng_yilei@qq.com>

ls-ggg · 2026-07-01T06:25:50Z

@zyl1121 I will merge this PR. Thanks for your contribution! If you're interested, feel free to open another PR to update the env-related documentation

zyl1121 requested review from chenhengqi, fslongjin, ls-ggg, tinklone and up2wing as code owners June 15, 2026 06:56

zyl1121 force-pushed the fix/cubeapi-envd-init branch from 273018b to 2779e07 Compare June 15, 2026 06:58

zyl1121 mentioned this pull request Jun 15, 2026

[Bug Report] Create-time envs are not fully propagated to sandbox runtime execution #565

Open