Jobs API

• 09/07/2021
• 43 minutes to read
• • m
  • a
  • l
  • m

The Jobs API allows you to create, edit, and delete jobs. The maximum allowed size of a request to the Jobs API is 10MB. See Create a High Concurrency cluster for a how-to guide on this API.

For details about updates to the Jobs API that support orchestration of multiple tasks with Azure Databricks jobs, see Jobs API updates.

Create

Create a new job.

Example

This example creates a job that runs a JAR task at 10:15pm each night.

Request

curl --netrc --request POST \
https://<databricks-instance>/api/2.0/jobs/create \
--data @create-job.json \
| jq .

create-job.json:

{
  "name": "Nightly model training",
  "new_cluster": {
    "spark_version": "7.3.x-scala2.12",
    "node_type_id": "Standard_D3_v2",
    "num_workers": 10
  },
  "libraries": [
    {
      "jar": "dbfs:/my-jar.jar"
    },
    {
      "maven": {
        "coordinates": "org.jsoup:jsoup:1.7.2"
      }
    }
  ],
  "timeout_seconds": 3600,
  "max_retries": 1,
  "schedule": {
    "quartz_cron_expression": "0 15 22 * * ?",
    "timezone_id": "America/Los_Angeles"
  },
  "spark_jar_task": {
    "main_class_name": "com.databricks.ComputeModels"
  }
}

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• The contents of create-job.json with fields that are appropriate for your solution.

This example uses a .netrc file and jq.

Response

{
  "job_id": 1
}

Request structure

Response structure

List

List all jobs.

Example

Request

curl --netrc --request GET \
https://<databricks-instance>/api/2.0/jobs/list \
| jq .

Replace <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.

This example uses a .netrc file and jq.

Response

{
  "jobs": [
    {
      "job_id": 1,
      "settings": {
        "name": "Nightly model training",
        "new_cluster": {
          "spark_version": "7.3.x-scala2.12",
          "node_type_id": "Standard_D3_v2",
          "num_workers": 10
        },
        "libraries": [
          {
            "jar": "dbfs:/my-jar.jar"
          },
          {
            "maven": {
              "coordinates": "org.jsoup:jsoup:1.7.2"
            }
          }
        ],
        "timeout_seconds": 100000000,
        "max_retries": 1,
        "schedule": {
          "quartz_cron_expression": "0 15 22 * * ?",
          "timezone_id": "America/Los_Angeles",
          "pause_status": "UNPAUSED"
        },
        "spark_jar_task": {
          "main_class_name": "com.databricks.ComputeModels"
        }
      },
      "created_time": 1457570074236
    }
  ]
}

Response structure

Delete

Delete a job and send an email to the addresses specified in JobSettings.email_notifications. No action occurs if the job has already been removed. After the job is removed, neither its details nor its run history is visible in the Jobs UI or API. The job is guaranteed to be removed upon completion of this request. However, runs that were active before the receipt of this request may still be active. They will be terminated asynchronously.

Example

curl --netrc --request POST \
https://<databricks-instance>/api/2.0/jobs/delete \
--data '{ "job_id": <job-id> }'

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• <job-id> with the ID of the job, for example 123.

This example uses a .netrc file.

Request structure

Get

Retrieve information about a single job.

Example

Request

curl --netrc --request GET \
'https://<databricks-instance>/api/2.0/jobs/get?job_id=<job-id>' \
| jq .

Or:

curl --netrc --get \
https://<databricks-instance>/api/2.0/jobs/get \
--data job_id=<job-id> \
| jq .

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• <job-id> with the ID of the job, for example 123.

This example uses a .netrc file and jq.

Response

{
  "job_id": 1,
  "settings": {
    "name": "Nightly model training",
    "new_cluster": {
      "spark_version": "7.3.x-scala2.12",
      "node_type_id": "Standard_D3_v2",
      "num_workers": 10
    },
    "libraries": [
      {
        "jar": "dbfs:/my-jar.jar"
      },
      {
        "maven": {
          "coordinates": "org.jsoup:jsoup:1.7.2"
        }
      }
    ],
    "timeout_seconds": 100000000,
    "max_retries": 1,
    "schedule": {
      "quartz_cron_expression": "0 15 22 * * ?",
      "timezone_id": "America/Los_Angeles",
      "pause_status": "UNPAUSED"
    },
    "spark_jar_task": {
      "main_class_name": "com.databricks.ComputeModels"
    }
  },
  "created_time": 1457570074236
}

Request structure

Response structure

Reset

Overwrite all settings for a specific job. Use the Update endpoint to update job settings partially.

Example

This example request makes job 2 identical to job 1 in the create example.

curl --netrc --request POST \
https://<databricks-instance>/api/2.0/jobs/reset \
--data @reset-job.json \
| jq .

reset-job.json:

{
  "job_id": 2,
  "new_settings": {
    "name": "Nightly model training",
    "new_cluster": {
      "spark_version": "7.3.x-scala2.12",
      "node_type_id": "Standard_D3_v2",
      "num_workers": 10
    },
    "libraries": [
      {
        "jar": "dbfs:/my-jar.jar"
      },
      {
        "maven": {
          "coordinates": "org.jsoup:jsoup:1.7.2"
        }
      }
    ],
    "timeout_seconds": 100000000,
    "max_retries": 1,
    "schedule": {
      "quartz_cron_expression": "0 15 22 * * ?",
      "timezone_id": "America/Los_Angeles",
      "pause_status": "UNPAUSED"
    },
    "spark_jar_task": {
      "main_class_name": "com.databricks.ComputeModels"
    }
  }
}

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• The contents of reset-job.json with fields that are appropriate for your solution.

This example uses a .netrc file and jq.

Request structure

Update

Add, change, or remove specific settings of an existing job. Use the Reset endpoint to overwrite all job settings.

Example

This example request removes libraries and adds email notification settings to job 1 defined in the create example.

curl --netrc --request POST \
https://<databricks-instance>/api/2.0/jobs/update \
--data @update-job.json \
| jq .

update-job.json:

{
  "job_id": 1,
  "new_settings": {
    "existing_cluster_id": "1201-my-cluster",
    "email_notifications": {
      "on_start": [ "someone@example.com" ],
      "on_success": [],
      "on_failure": []
    }
  },
  "fields_to_remove": ["libraries"]
}

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• The contents of update-job.json with fields that are appropriate for your solution.

This example uses a .netrc file and jq.

Request structure

Run now

Run a job now and return the run_id of the triggered run.

Example

curl --netrc --request POST \
https://<databricks-instance>/api/2.0/jobs/run-now \
--data @run-job.json \
| jq .

run-job.json:

An example request for a notebook job:

{
  "job_id": 1,
  "notebook_params": {
    "name": "john doe",
    "age": "35"
  }
}

An example request for a JAR job:

{
  "job_id": 2,
  "jar_params": [ "john doe", "35" ]
}

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• The contents of run-job.json with fields that are appropriate for your solution.

This example uses a .netrc file and jq.

Request structure

Response structure

Runs submit

Submit a one-time run. This endpoint allows you to submit a workload directly without creating a job. Runs submitted using this endpoint don’t display in the UI. Use the jobs/runs/get API to check the run state after the job is submitted.

Example

Request

curl --netrc --request POST \
https://<databricks-instance>/api/2.0/jobs/runs/submit \
--data @submit-job.json \
| jq .

submit-job.json:

{
  "run_name": "my spark task",
  "new_cluster": {
    "spark_version": "7.3.x-scala2.12",
    "node_type_id": "Standard_D3_v2",
    "num_workers": 10
  },
  "libraries": [
    {
      "jar": "dbfs:/my-jar.jar"
    },
    {
      "maven": {
        "coordinates": "org.jsoup:jsoup:1.7.2"
      }
    }
  ],
  "spark_jar_task": {
    "main_class_name": "com.databricks.ComputeModels"
  }
}

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• The contents of submit-job.json with fields that are appropriate for your solution.

This example uses a .netrc file and jq.

Response

{
  "run_id": 123
}

Request structure

Response structure

Runs list

List runs in descending order by start time.

Example

Request

curl --netrc --request GET \
'https://<databricks-instance>/api/2.0/jobs/runs/list?job_id=<job-id>&active_only=<true-false>&offset=<offset>&limit=<limit>&run_type=<run-type>' \
| jq .

Or:

curl --netrc --get \
https://<databricks-instance>/api/2.0/jobs/runs/list \
--data 'job_id=<job-id>&active_only=<true-false>&offset=<offset>&limit=<limit>&run_type=<run-type>' \
| jq .

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• <job-id> with the ID of the job, for example 123.
• <true-false> with true or false.
• <offset> with the offset value.
• <limit> with the limit value.
• <run-type> with the run_type value.

This example uses a .netrc file and jq.

Response

{
  "runs": [
    {
      "job_id": 1,
      "run_id": 452,
      "number_in_job": 5,
      "state": {
        "life_cycle_state": "RUNNING",
        "state_message": "Performing action"
      },
      "task": {
        "notebook_task": {
          "notebook_path": "/Users/donald@duck.com/my-notebook"
        }
      },
      "cluster_spec": {
        "existing_cluster_id": "1201-my-cluster"
      },
      "cluster_instance": {
        "cluster_id": "1201-my-cluster",
        "spark_context_id": "1102398-spark-context-id"
      },
      "overriding_parameters": {
        "jar_params": ["param1", "param2"]
      },
      "start_time": 1457570074236,
      "end_time": 1457570075149,
      "setup_duration": 259754,
      "execution_duration": 3589020,
      "cleanup_duration": 31038,
      "trigger": "PERIODIC"
    }
  ],
  "has_more": true
}

Request structure

Response structure

Runs get

Retrieve the metadata of a run.

Example

Request

curl --netrc --request GET \
'https://<databricks-instance>/api/2.0/jobs/runs/get?run_id=<run-id>' \
| jq .

Or:

curl --netrc --get \
https://<databricks-instance>/api/2.0/jobs/runs/get \
--data run_id=<run-id> \
| jq .

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• <run-id> with the ID of the run, for example 123.

This example uses a .netrc file and jq.

Response

{
  "job_id": 1,
  "run_id": 452,
  "number_in_job": 5,
  "state": {
    "life_cycle_state": "RUNNING",
    "state_message": "Performing action"
  },
  "task": {
    "notebook_task": {
      "notebook_path": "/Users/someone@example.com/my-notebook"
    }
  },
  "cluster_spec": {
    "existing_cluster_id": "1201-my-cluster"
  },
  "cluster_instance": {
    "cluster_id": "1201-my-cluster",
    "spark_context_id": "1102398-spark-context-id"
  },
  "overriding_parameters": {
    "jar_params": ["param1", "param2"]
  },
  "start_time": 1457570074236,
  "end_time": 1457570075149,
  "setup_duration": 259754,
  "execution_duration": 3589020,
  "cleanup_duration": 31038,
  "trigger": "PERIODIC"
}

Request structure

Response structure

Runs export

Export and retrieve the job run task.

Example

Request

curl --netrc --request GET \
'https://<databricks-instance>/api/2.0/jobs/runs/export?run_id=<run-id>' \
| jq .

Or:

curl --netrc --get \
https://<databricks-instance>/api/2.0/jobs/runs/export \
--data run_id=<run-id> \
| jq .

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• <run-id> with the ID of the run, for example 123.

This example uses a .netrc file and jq.

Response

{
  "views": [ {
    "content": "<!DOCTYPE html><html><head>Head</head><body>Body</body></html>",
    "name": "my-notebook",
    "type": "NOTEBOOK"
  } ]
}

To extract the HTML notebook from the JSON response, download and run this Python script.

Request structure

Response structure

Runs cancel

Cancel a run. The run is canceled asynchronously, so when this request completes, the run may still be running. The run will be terminated shortly. If the run is already in a terminal life_cycle_state, this method is a no-op.

This endpoint validates that the run_id parameter is valid and for invalid parameters returns HTTP status code 400.

Example

curl --netrc --request POST \
https://<databricks-instance>/api/2.0/jobs/runs/cancel \
--data '{ "run_id": <run-id> }'

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• <run-id> with the ID of the run, for example 123.

This example uses a .netrc file.

Request structure

Runs get output

Retrieve the output and metadata of a run. When a notebook task returns a value through the dbutils.notebook.exit() call, you can use this endpoint to retrieve that value. Azure Databricks restricts this API to return the first 5 MB of the output. For returning a larger result, you can store job results in a cloud storage service.

This endpoint validates that the run_id parameter is valid and for invalid parameters returns HTTP status code 400.

Runs are automatically removed after 60 days. If you to want to reference them beyond 60 days, you should save old run results before they expire. To export using the UI, see Export job run results. To export using the Jobs API, see Runs export.

Example

Request

curl --netrc --request GET \
'https://<databricks-instance>/api/2.0/jobs/runs/get-output?run_id=<run-id>' \
| jq .

Or:

curl --netrc --get \
https://<databricks-instance>/api/2.0/jobs/runs/get-output \
--data run_id=<run-id> \
| jq .

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• <run-id> with the ID of the run, for example 123.

This example uses a .netrc file and jq.

Response

{
  "metadata": {
    "job_id": 1,
    "run_id": 452,
    "number_in_job": 5,
    "state": {
      "life_cycle_state": "TERMINATED",
      "result_state": "SUCCESS",
      "state_message": ""
    },
    "task": {
      "notebook_task": {
        "notebook_path": "/Users/someone@example.com/my-notebook"
      }
    },
    "cluster_spec": {
      "existing_cluster_id": "1201-my-cluster"
    },
    "cluster_instance": {
      "cluster_id": "1201-my-cluster",
      "spark_context_id": "1102398-spark-context-id"
    },
    "overriding_parameters": {
      "jar_params": ["param1", "param2"]
    },
    "start_time": 1457570074236,
    "setup_duration": 259754,
    "execution_duration": 3589020,
    "cleanup_duration": 31038,
    "trigger": "PERIODIC"
  },
  "notebook_output": {
    "result": "the maybe truncated string passed to dbutils.notebook.exit()"
  }
}

Request structure

Response structure

Runs delete

Delete a non-active run. Returns an error if the run is active.

Example

curl --netrc --request POST \
https://<databricks-instance>/api/2.0/jobs/runs/delete \
--data '{ "run_id": <run-id> }'

Replace:

• <databricks-instance> with the Azure Databricks workspace instance name, for example adb-1234567890123456.7.azuredatabricks.net.
• <run-id> with the ID of the run, for example 123.

This example uses a .netrc file.

Request structure

Data structures

In this section:

• ClusterInstance
• ClusterSpec
• CronSchedule
• Job
• JobEmailNotifications
• JobSettings
• JobTask
• NewCluster
• NotebookOutput
• NotebookTask
• ParamPair
• Run
• RunParameters
• RunState
• SparkJarTask
• SparkPythonTask
• SparkSubmitTask
• ViewItem
• RunLifeCycleState
• RunResultState
• TriggerType
• ViewType
• ViewsToExport

ClusterInstance

Identifiers for the cluster and Spark context used by a run. These two values together identify an execution context across all time.

ClusterSpec

CronSchedule

Job

JobEmailNotifications

JobSettings

Settings for a job. These settings can be updated using the resetJob method.

JobTask

NewCluster

NotebookOutput

NotebookTask

All the output cells are subject to the size of 8MB. If the output of a cell has a larger size, the rest of the run will be cancelled and the run will be marked as failed. In that case, some of the content output from other cells may also be missing.

If you need help finding the cell that is beyond the limit, run the notebook against an all-purpose cluster and use this notebook autosave technique.

ParamPair

Name-based parameters for jobs running notebook tasks.

Run

All the information about a run except for its output. The output can be retrieved separately with the getRunOutput method.

RunParameters

Parameters for this run. Only one of jar_params, python_params, or notebook_params should be specified in the run-now request, depending on the type of job task. Jobs with Spark JAR task or Python task take a list of position-based parameters, and jobs with notebook tasks take a key value map.

RunState

SparkJarTask

SparkPythonTask

SparkSubmitTask

For example, assuming the JAR is uploaded to DBFS, you can run SparkPi by setting the following parameters.

{
  "parameters": [
    "--class",
    "org.apache.spark.examples.SparkPi",
    "dbfs:/path/to/examples.jar",
    "10"
  ]
}

ViewItem

The exported content is in HTML format. For example, if the view to export is dashboards, one HTML string is returned for every dashboard.

RunLifeCycleState

The life cycle state of a run. Allowed state transitions are:

• PENDING -> RUNNING -> TERMINATING -> TERMINATED
• PENDING -> SKIPPED
• PENDING -> INTERNAL_ERROR
• RUNNING -> INTERNAL_ERROR
• TERMINATING -> INTERNAL_ERROR

RunResultState

The result state of the run.

• If life_cycle_state = TERMINATED: if the run had a task, the result is guaranteed to be available, and it indicates the result of the task.
• If life_cycle_state = PENDING, RUNNING, or SKIPPED, the result state is not available.
• If life_cycle_state = TERMINATING or lifecyclestate = INTERNAL_ERROR: the result state is available if the run had a task and managed to start it.

Once available, the result state never changes.

TriggerType

These are the type of triggers that can fire a run.

ViewType

ViewsToExport

View to export: either code, all dashboards, or all.