[LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue#529
[LIVY-1059] Add livy.spark.yarn.queue configuration to set default YARN queue#529nileshrathi345 wants to merge 2 commits into
Conversation
|
@ArnavBalyan, @gyogal and @roczei could you please take a look when possible thanks! |
ArnavBalyan
left a comment
ArnavBalyan
left a comment
There was a problem hiding this comment.
Thanks for the changes @nileshrathi345, could you pls look into the CI failures, overall LGTM
| appTag: String, | ||
| owner: String, | ||
| proxyUser: Option[String], | ||
| queue: Option[String], |
There was a problem hiding this comment.
Why does the recovery state store save the queue the batch has been started with? Do we want to be able to show it after a Livy server restart? I think this may be independent new functionality not covered by LIVY-1059 (which is only about allowing the user to set a default YARN queue)
| state: String, | ||
| appId: Option[String], | ||
| appInfo: AppInfo, | ||
| queue: Option[String], |
There was a problem hiding this comment.
I think this change is only needed if we want to be able to return which queue the batch job has been submitted into. If so, the Batch object may need to be updated at https://livy.apache.org/docs/latest/rest-api.html
But this is something LIVY-1059 does not currently mention, which is only about defaulting the value for the queue parameter.
gyogal
left a comment
gyogal
left a comment
There was a problem hiding this comment.
Thanks for submitting a PR @nileshrathi345 ! I left two minor comments, could you please check and let me know your thoughts?
3 participants
What changes were proposed in this pull request?
This PR introduces support for configuring a default YARN queue at the Livy server level via a new configuration property:
livy.spark.yarn.queue.Previously, if a client did not explicitly specify a YARN queue parameter (queue) in their session creation payload, the session would fall back to the global Hadoop cluster's default YARN queue. This change allows cluster administrators to isolate Livy-generated workloads into a specific default queue without requiring end-users to pass it manually in every API request.
How was this patch tested?
This patch has been thoroughly validated using both newly added automated unit tests and manual API validation: