Skip to content

Add KillCaller feature to test server for CLI crash recovery testing #4146

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
varundeepsaini wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Add KillCaller feature to test server for CLI crash recovery testing #4146

varundeepsaini wants to merge 5 commits into from

Conversation

@varundeepsaini
Copy link
Contributor

@varundeepsaini varundeepsaini commented Dec 15, 2025 •
edited
Loading

Closes: #4140

Changes

Add KillCaller option to test server that terminates the CLI process when specific endpoints are hit. This enables testing crash recovery scenarios like bundle deploy interrupted mid-execution.

KillCaller specifies how many times to kill before allowing normal responses, enabling retry/recovery testing.

[[Server]]
Pattern = "GET /api/2.0/workspace/list"
KillCaller = 2  # Kill first 2 calls, then allow

Why

Enables testing CLI crash recovery scenarios (e.g., bundle deploy interrupted mid-execution).

Tests

Added acceptance/selftest/kill_caller/ tests.

@varundeepsaini varundeepsaini force-pushed the ft/server-crash-acceptance branch from 89e1011 to 3a3b6f5 Compare December 15, 2025 15:20
@varundeepsaini
Copy link
Contributor Author

varundeepsaini commented Dec 16, 2025

@denik could you please review this pr

@varundeepsaini varundeepsaini force-pushed the ft/server-crash-acceptance branch 2 times, most recently from debc6c8 to bcc288c Compare December 17, 2025 08:02
@varundeepsaini varundeepsaini force-pushed the ft/server-crash-acceptance branch from bcc288c to ba053c3 Compare December 17, 2025 08:04
@varundeepsaini
Copy link
Contributor Author

varundeepsaini commented Dec 17, 2025 •
edited
Loading

@denik can you please queue the integration tests

@varundeepsaini
Copy link
Contributor Author

varundeepsaini commented Dec 30, 2025

@denik bump ^^

denik
denik reviewed Dec 30, 2025
View reviewed changes
@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Dec 30, 2025 •
edited
Loading

Commit: ba053c3

Run: 20599356933

Env ❌​FAIL 🟨​KNOWN 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 12 5 2 381 668 189:31
❌​ aws windows 2 17 2 381 666 64:26
🟨​ aws-ucws linux 8 9 2 532 545 191:47
🟨​ aws-ucws windows 13 4 2 534 543 204:29
🟨​ azure linux 7 5 3 381 667 220:11
🟨​ azure windows 12 3 383 665 236:03
🟨​ azure-ucws linux 7 5 3 528 544 237:56
🟨​ azure-ucws windows 12 3 530 542 241:11
🟨​ gcp linux 7 5 3 370 673 30:20
🟨​ gcp windows 12 3 372 671 27:50
21 interesting tests: 18 KNOWN, 2 FAIL, 1 SKIP
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/deployment/bind/alert 🙈​S 🙈​S 🙈​S 🙈​S 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/alerts/basic 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/alerts/basic/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 🟨​K 🟨​K 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
❌​ TestAccept/bundle/resources/secret_scopes/permissions ✅​p ❌​F ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p 🙈​s 🙈​s
❌​ TestAccept/bundle/resources/secret_scopes/permissions/DATABRICKS_BUNDLE_ENGINE=terraform ✅​p ❌​F ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🟨​ TestExport 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestExportWithFileFlag 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
🟨​ TestImportDir 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportDirDoesNotOverwrite 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportDirWithOverwriteFlag 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportFileFormatAuto 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
🟨​ TestImportFileFormatSource 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K 💚​R 🟨​K
Top 25 slowest tests (at least 2 minutes):
duration env testname
8:44 azure-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
8:24 azure-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
7:09 azure-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
7:05 azure-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
6:47 aws-ucws linux TestAccept/bundle/resources/synced_database_tables/basic
6:28 aws-ucws windows TestAccept/bundle/resources/synced_database_tables/basic
6:11 azure-ucws linux TestAccept/bundle/resources/synced_database_tables/basic
5:43 aws-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:40 aws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:28 aws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:25 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:21 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:12 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:56 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:57 azure windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:50 azure windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:45 azure-ucws windows TestAccept/bundle/resources/synced_database_tables/basic
2:32 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:28 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:26 aws-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
2:20 gcp linux TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.9
2:20 aws-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:19 aws-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:16 gcp windows TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=yes/READPLAN=
2:13 azure linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform

@varundeepsaini
Copy link
Contributor Author

varundeepsaini commented Dec 30, 2025

@denik
I changed KillCaller from bool to int to support retry scenarios. Now, we can specify how many times to kill before allowing requests through (e.g., KillCaller = 3 kills the first 3 attempts, 4th succeeds).

Had a few questions,

  1. Should I add thread safety (via atomics)?
  2. Would a binary string be useful for granular control (e.g., "1011" = kill, allow, kill, kill)?

@varundeepsaini
cleanup
1ad08e2 
Signed-off-by: Varun Deep Saini <[email protected]>
@github-actions
Copy link

github-actions bot commented Dec 30, 2025

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 4146
  • Commit SHA: 1ad08e2d42e3849d6bc05461c0e376fc9ca3b996

Checks will be approved automatically on success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@denik denik denik left review comments

@jefferycheng1 jefferycheng1 Awaiting requested review from jefferycheng1 jefferycheng1 is a code owner

@kanterov kanterov Awaiting requested review from kanterov kanterov is a code owner

@lennartkats-db lennartkats-db Awaiting requested review from lennartkats-db lennartkats-db is a code owner

@andrewnester andrewnester Awaiting requested review from andrewnester andrewnester is a code owner

@anton-107 anton-107 Awaiting requested review from anton-107 anton-107 is a code owner

@pietern pietern Awaiting requested review from pietern pietern is a code owner

@shreyas-goenka shreyas-goenka Awaiting requested review from shreyas-goenka shreyas-goenka is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add ability to kill CLI process from test server to test crash recovery scenarios

3 participants

@varundeepsaini @eng-dev-ecosystem-bot @denik