ci(kubernetes): add server pool e2e to nightly build, trim pause/resume matrix - #1583
ci(kubernetes): add server pool e2e to nightly build, trim pause/resume matrix#1583Pangjiping wants to merge 6 commits into
Conversation
Pause/resume only uses stable Pod/Job/CRD APIs across the whole 1.21-1.34 range; the matrix was reduced from 8 versions to 1.22.4 (minimum supported), 1.28.6, and 1.34.2 (newest containerd).
Adds a k8s-pool-e2e job (ingress-header variant) running the server pool allocation e2e: a kubectl-installed Pool CR whose pod template runs execd as PID 1, then SDK sandboxes created with extensions.poolRef are allocated from the pre-warmed buffer and run commands through execd. Release returns the pod to the pool. The pool suite in scripts/common/kubernetes-e2e.sh now also runs the new test_server_pool_e2e_sync.py (sync SDK pool tests stay as the SDK-side counterpart). kubectl namespace is exported to the test env, and the pool test files are added to the kubernetes-mini-e2e change detector.
github-actions
Bot
added
the
size/M
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ef4d687da4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| path === "tests/python/tests/test_sandbox_pool_e2e_sync.py" || | ||
| path === "tests/python/tests/test_sandbox_pool_e2e_async.py" || |
There was a problem hiding this comment.
The kubernetes-mini-e2e matcher adds the existing SDK-side pool tests but omits both newly introduced inputs, tests/python/tests/test_server_pool_e2e_sync.py and tests/python/tests/support/server-pool.yaml. A pull request that changes only the new server-pool test or manifest therefore reports this area as irrelevant and skips the job that exercises those changes; include both new paths in this matcher.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 281c1cf: both new paths (test_server_pool_e2e_sync.py and support/server-pool.yaml) are now in the kubernetes-mini-e2e matcher.
| run: | | ||
| kind delete cluster --name "${KIND_CLUSTER}" || true | ||
|
|
||
| k8s-pool-e2e: |
There was a problem hiding this comment.
For scheduled and manually dispatched runs, the new job is not included in publish-nightly-latest.needs, which still lists only k8s-mini-e2e and execd-init-e2e. As a result, the workflow can push latest images while this pool test is still running, and can publish successfully even when the pool validation fails; the pool job is gated only through the pull-request-only required job. Add k8s-pool-e2e to the publication dependency so failed validation cannot release the nightly images.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 281c1cf: publish-nightly-latest now depends on k8s-pool-e2e, so failed pool validation blocks nightly image publication on scheduled/manual runs.
| kind: Pool | ||
| metadata: | ||
| name: e2e-server-pool | ||
| namespace: opensandbox-e2e |
There was a problem hiding this comment.
When either Kubernetes E2E script is run with a non-default E2E_NAMESPACE, it exports that value as OPENSANDBOX_E2E_NAMESPACE, so the test queries and deletes the pool in the configured namespace, but kubectl apply still creates this manifest in hard-coded opensandbox-e2e. The fixture then times out polling a pool that does not exist in the selected namespace and leaves the created resource behind; render or override this namespace from the configured value before applying the manifest.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 281c1cf: the fixture now renders the namespace into the manifest before applying (kubectl apply -f -), honoring OPENSANDBOX_E2E_NAMESPACE instead of the hard-coded opensandbox-e2e.
test_03 waited for pool allocation to reach zero, but the sandbox from test_02 was still allocated; assert the drop by one instead. The Pool CR also now pins recycleStrategy=Noop so a released pod returns to the buffer instead of being deleted (the CRD default is Delete).
- detect-changes: include the new server pool test file and manifest in the kubernetes-mini-e2e area so pool-only PRs trigger the job - nightly publish: gate 'latest' image publication on k8s-pool-e2e so a failing pool validation cannot release nightly images - server-pool.yaml: render the namespace from OPENSANDBOX_E2E_NAMESPACE when applying instead of hard-coding opensandbox-e2e
github-actions
Bot
added
size/L
Fixes the spotlessKotlinCheck failure on main (introduced by the lifecycle-allocation-core merge): SandboxAllocation must sort before SandboxCreateResponse.
1 participant
Summary
Two related CI changes in the Kubernetes e2e area:
1. Add server pool e2e to the nightly build
New
k8s-pool-e2ejob (ingress-header variant) inkubernetes-nightly-build.ymlthat runs the server-managed pool allocation path:tests/python/tests/support/server-pool.yaml) installed via kubectl; pod template runs execd as PID 1 (/bootstrap.sh sleep 3600+EXECD_INIT=1), so pooled sandboxes can execute SDK commands through execdtest_server_pool_e2e_sync.py: pool warms buffer → SDK sandbox created withextensions.poolRefallocated from the pre-warmed pool →commands.run()works via execd → destroy releases the pod back to the poolscripts/common/kubernetes-e2e.sh:E2E_TEST_SUITE=poolnow runs the SDK-side pool suite (sync) plus the new server pool testscripts/python-k8s-e2e*.sh: exportOPENSANDBOX_E2E_NAMESPACEdetect-changes.yml: pool e2e files trigger thekubernetes-mini-e2eareaThis complements the SDK-side pool tests (
test_sandbox_pool_e2e_sync.py), which never create a Pool CR.2. Trim pause/resume e2e k8s version matrix
Pause/resume uses only stable Pod/Job/CRD APIs across 1.21–1.34; the 8-version matrix was reduced to 1.22.4 (minimum supported), 1.28.6, 1.34.2 (newest containerd). The core e2e matrix is unchanged.
Verification
ruff checkpasses on the new test file