Skip to content

[Feature-18695][Spark] Add --py-files, --archives, --principal/--keytab fields to Spark task - #18781

Open
chovy-3012 wants to merge 4 commits into
apache:devfrom
chovy-3012:feature-spark-params
Open

chovy-3012 wants to merge 4 commits into
apache:devfrom
chovy-3012:feature-spark-params

Conversation

@chovy-3012

Copy link
Copy Markdown
Contributor

Was this PR generated or assisted by AI?

NO.

Purpose of the pull request

Closes #18695.

Adds dedicated form fields for four high-frequency spark-submit options — --py-files, --archives, --principal, and --keytab — to the Spark task, so users no longer type them into the free-form option box. PyFiles/Archives/Keytab select resources from the resource center; the worker downloads them and SparkTask appends the options to the generated spark-submit command.

Brief change log

  • SparkConstants: add the four option constants. SparkParameters: add pyFiles/archives (List<ResourceInfo>), principal (String), keytab (ResourceInfo); validate principal/keytab as a pair in checkParameters(); register the three resource fields in getResourceFilesList() so the worker downloads them.
  • SparkTask.populateSparkOptions(): append --py-files/--archives (comma-joined local paths), --principal, and --keytab before the main jar argument.
  • UI: PyFiles (shown for PYTHON only), Archives, Principal, and Keytab fields in use-spark.ts; taskParams serialization and edit-dialog backfill in format-data.ts; frontend pair validation with en/zh messages.
  • Docs: add the four parameters to docs/docs/en/guide/task/spark.md and docs/docs/zh/guide/task/spark.md.

Verify this pull request

Unit tests cover parameter-pair validation, resource registration, and command construction. Also verified end to end on the standalone server: a Spark (PYTHON) task with all four fields ran, the worker log shows the four resources downloaded and the spark-submit line contains all four options, and the edit dialog backfills the four fields from the saved definition.

Passed on Java 8:

./mvnw -pl dolphinscheduler-task-plugin/dolphinscheduler-task-spark test -Dtest='SparkParametersTest,SparkTaskTest' -Dspotless.check.skip=true -Djacoco.skip=true
./mvnw -pl dolphinscheduler-task-plugin/dolphinscheduler-task-spark spotless:check

SparkParametersTest and SparkTaskTest: 11 tests, 0 failures, 0 errors, 0 skipped. spotless:check passes, and eslint reports no issues on the changed UI files.

@github-actions github-actions Bot added UI ui and front end related backend test document labels Nov 12, 2026
@SbloodyS SbloodyS added this to the 3.5.0 milestone Nov 13, 2026
@SbloodyS SbloodyS added the feature new feature label Nov 13, 2026

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Inference] Avoid loading the resource tree twice

[Fact] The new getResources() function registers an onMounted callback that requests the complete resource tree. The existing useResources(...) call later in the same function registers another callback for the same request.

[Inference] Because the first callback awaits the API response before updating taskStore.resources, both callbacks can observe an empty cache and concurrently call /resources/list.

[Fact] Each request recursively enumerates all resource files for the tenant.

[Inference] This duplicates an expensive storage traversal whenever the Spark form is opened with an empty cache, increasing latency and backend load for tenants with large resource trees.

Please share a single resource-loading result or an in-flight promise between these selectors.

@chovy-3012

Copy link
Copy Markdown
Contributor Author

Thanks @SbloodyS for the review! The issue is addressed in 87ee6a1.

Duplicate resource-tree loading
Removed the standalone getResources()/onMounted loader from useSpark in use-spark.ts. The pyFiles, archives, and keytab fields now consume the options ref of the existing useResources(...) field, so opening the Spark form issues a single resource-tree request shared by all resource selectors in the form.

Verified with vue-tsc --noEmit (0 errors) and eslint (clean) on the changed file; no backend change in this round.

@SbloodyS SbloodyS left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Escape resource paths before generating the shell command

dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java:184

The new PyFiles, Archives, and Keytab paths are appended without shell escaping. Resource names may contain spaces, so a path such as /resources/python libs/deps.zip is split into multiple arguments when AbstractYarnTask executes the generated script, causing spark-submit to receive an invalid command. The added tests only exercise paths without spaces.

Escape each generated argument while preserving the comma-separated PyFiles/Archives format, and add regressions using resource paths containing spaces for all three options.

Validate the dedicated resources instead of only sharing their options

dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-spark.ts:203

The PyFiles, Archives, and Keytab selectors reuse resourcesField.options, but not its validateResourceExist validation. Consequently, when a referenced resource is deleted or renamed, formatModel() retains the stale path and formatParams() saves it again. Backend validation only checks the principal/keytab pairing, so the definition is accepted and later fails in the worker when getResourceMetaData() cannot resolve the resource.

Apply resource-existence validation to each dedicated selector, including the single Keytab value, and cover editing a definition after one of its selected resources has been removed or renamed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend document feature new feature test UI ui and front end related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][Spark] Add high-frequency parameters to Spark task: --py-files, --archives, and Kerberos --principal/--keytab

2 participants