SparkJobProperties Class
The properties of the Spark job.
All required parameters must be populated in order to send to Azure.
- Inheritance
-
Constructor
SparkJobProperties(*, file: str, driver_memory: str, driver_cores: int, executor_memory: str, executor_cores: int, num_executors: int, additional_properties: Optional[Dict[str, object]] = None, name: Optional[str] = None, class_name: Optional[str] = None, conf: Optional[object] = None, args: Optional[List[str]] = None, jars: Optional[List[str]] = None, files: Optional[List[str]] = None, archives: Optional[List[str]] = None, **kwargs)
Parameters
- additional_properties
-
dict[str, object]
Required
Unmatched properties from the message are deserialized to this
collection.
- name
-
str
Required
The name of the job.
- file
-
str
Required
Required. File containing the application to execute.
- class_name
-
str
Required
Main class for Java/Scala application.
- conf
-
object
Required
Spark configuration properties.
- args
-
list[str]
Required
Command line arguments for the application.
- jars
-
list[str]
Required
Jars to be used in this job.
- files
-
list[str]
Required
files to be used in this job.
- archives
-
list[str]
Required
Archives to be used in this job.
- driver_memory
-
str
Required
Required. Amount of memory to use for the driver process.
- driver_cores
-
int
Required
Required. Number of cores to use for the driver.
- executor_memory
-
str
Required
Required. Amount of memory to use per executor process.
- executor_cores
-
int
Required
Required. Number of cores to use for each executor.
- num_executors
-
int
Required
Required. Number of executors to launch for this job.