Repository navigation
[Feature-18695][Spark] Add --py-files, --archives, --principal/--keytab fields to Spark task - #18781
chovy-3012 wants to merge 4 commits into
Conversation
…ab fields to Spark task
SbloodyS
left a comment
There was a problem hiding this comment.
[Inference] Avoid loading the resource tree twice
[Fact] The new getResources() function registers an onMounted callback that requests the complete resource tree. The existing useResources(...) call later in the same function registers another callback for the same request.
[Inference] Because the first callback awaits the API response before updating taskStore.resources, both callbacks can observe an empty cache and concurrently call /resources/list.
[Fact] Each request recursively enumerates all resource files for the tenant.
[Inference] This duplicates an expensive storage traversal whenever the Spark form is opened with an empty cache, increasing latency and backend load for tenants with large resource trees.
Please share a single resource-loading result or an in-flight promise between these selectors.
|
Thanks @SbloodyS for the review! The issue is addressed in 87ee6a1. Duplicate resource-tree loading Verified with |
SbloodyS
left a comment
There was a problem hiding this comment.
Escape resource paths before generating the shell command
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkTask.java:184
The new PyFiles, Archives, and Keytab paths are appended without shell escaping. Resource names may contain spaces, so a path such as /resources/python libs/deps.zip is split into multiple arguments when AbstractYarnTask executes the generated script, causing spark-submit to receive an invalid command. The added tests only exercise paths without spaces.
Escape each generated argument while preserving the comma-separated PyFiles/Archives format, and add regressions using resource paths containing spaces for all three options.
Validate the dedicated resources instead of only sharing their options
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-spark.ts:203
The PyFiles, Archives, and Keytab selectors reuse resourcesField.options, but not its validateResourceExist validation. Consequently, when a referenced resource is deleted or renamed, formatModel() retains the stale path and formatParams() saves it again. Backend validation only checks the principal/keytab pairing, so the definition is accepted and later fails in the worker when getResourceMetaData() cannot resolve the resource.
Apply resource-existence validation to each dedicated selector, including the single Keytab value, and cover editing a definition after one of its selected resources has been removed or renamed.
…e validation for new spark options
Was this PR generated or assisted by AI?
NO.
Purpose of the pull request
Closes #18695.
Adds dedicated form fields for four high-frequency spark-submit options —
--py-files,--archives,--principal, and--keytab— to the Spark task, so users no longer type them into the free-form option box.PyFiles/Archives/Keytabselect resources from the resource center; the worker downloads them andSparkTaskappends the options to the generated spark-submit command.Brief change log
SparkConstants: add the four option constants.SparkParameters: addpyFiles/archives(List<ResourceInfo>),principal(String),keytab(ResourceInfo); validate principal/keytab as a pair incheckParameters(); register the three resource fields ingetResourceFilesList()so the worker downloads them.SparkTask.populateSparkOptions(): append--py-files/--archives(comma-joined local paths),--principal, and--keytabbefore the main jar argument.use-spark.ts; taskParams serialization and edit-dialog backfill informat-data.ts; frontend pair validation with en/zh messages.docs/docs/en/guide/task/spark.mdanddocs/docs/zh/guide/task/spark.md.Verify this pull request
Unit tests cover parameter-pair validation, resource registration, and command construction. Also verified end to end on the standalone server: a Spark (PYTHON) task with all four fields ran, the worker log shows the four resources downloaded and the spark-submit line contains all four options, and the edit dialog backfills the four fields from the saved definition.
Passed on Java 8:
SparkParametersTestandSparkTaskTest: 11 tests, 0 failures, 0 errors, 0 skipped.spotless:checkpasses, and eslint reports no issues on the changed UI files.