Jobs API

• 12/04/2019
• 36 minutes to read
• • 

The Jobs API allows you to create, edit, and delete jobs. The maximum allowed size of a request to the Jobs API is 10MB. See Jobs API examples for a how-to guide on this API.

Create

Create a new job.

An example request for a job that runs at 10:15pm each night:

{
  "name": "Nightly model training",
  "new_cluster": {
    "spark_version": "5.3.x-scala2.11",
    "node_type_id": "Standard_D3_v2",
    "num_workers": 10
  },
  "libraries": [
    {
      "jar": "dbfs:/my-jar.jar"
    },
    {
      "maven": {
        "coordinates": "org.jsoup:jsoup:1.7.2"
      }
    }
  ],
  "timeout_seconds": 3600,
  "max_retries": 1,
  "schedule": {
    "quartz_cron_expression": "0 15 22 ? * *",
    "timezone_id": "America/Los_Angeles"
  },
  "spark_jar_task": {
    "main_class_name": "com.databricks.ComputeModels"
  }
}

And response:

{
  "job_id": 1
}

Request Structure

Create a new job.

Response Structure

List

List all jobs. An example response:

{
  "jobs": [
    {
      "job_id": 1,
      "settings": {
        "name": "Nightly model training",
        "new_cluster": {
          "spark_version": "5.3.x-scala2.11",
          "node_type_id": "Standard_D3_v2",
          "num_workers": 10
        },
        "libraries": [
          {
            "jar": "dbfs:/my-jar.jar"
          },
          {
            "maven": {
              "coordinates": "org.jsoup:jsoup:1.7.2"
            }
          }
        ],
        "timeout_seconds": 100000000,
        "max_retries": 1,
        "schedule": {
          "quartz_cron_expression": "0 15 22 ? * *",
          "timezone_id": "America/Los_Angeles"
        },
        "spark_jar_task": {
          "main_class_name": "com.databricks.ComputeModels"
        }
      },
      "created_time": 1457570074236
    }
  ]
}

Response Structure

Delete

Delete the job and send an email to the addresses specified in JobSettings.email_notifications. No action occurs if the job has already been removed. After the job is removed, neither its details or its run history is visible via the Jobs UI or API. The job is guaranteed to be removed upon completion of this request. However, runs that were active before the receipt of this request may still be active. They will be terminated asynchronously.

An example request:

{
  "job_id": 1
}

Request Structure

Delete a job and send an email to the addresses specified in JobSettings.email_notifications. No action occurs if the job has already been removed. After the job is removed, neither its details nor its run history is visible via the Jobs UI or API. The job is guaranteed to be removed upon completion of this request. However, runs that were active before the receipt of this request may still be active. They will be terminated asynchronously.

Get

Retrieves information about a single job. An example request:

/jobs/get?job_id=1

An example response:

{
  "job_id": 1,
  "settings": {
    "name": "Nightly model training",
    "new_cluster": {
      "spark_version": "5.3.x-scala2.11",
      "node_type_id": "Standard_D3_v2",
      "num_workers": 10
    },
    "libraries": [
      {
        "jar": "dbfs:/my-jar.jar"
      },
      {
        "maven": {
          "coordinates": "org.jsoup:jsoup:1.7.2"
        }
      }
    ],
    "timeout_seconds": 100000000,
    "max_retries": 1,
    "schedule": {
      "quartz_cron_expression": "0 15 22 ? * *",
      "timezone_id": "America/Los_Angeles"
    },
    "spark_jar_task": {
      "main_class_name": "com.databricks.ComputeModels"
    }
  },
  "created_time": 1457570074236
}

Request Structure

Retrieve information about a single job.

Response Structure

Reset

Overwrite job settings.

An example request that makes job 2 look like job 1 (from the create_job example):

{
  "job_id": 2,
  "new_settings": {
    "name": "Nightly model training",
    "new_cluster": {
      "spark_version": "5.3.x-scala2.11",
      "node_type_id": "Standard_D3_v2",
      "num_workers": 10
    },
    "libraries": [
      {
        "jar": "dbfs:/my-jar.jar"
      },
      {
        "maven": {
          "coordinates": "org.jsoup:jsoup:1.7.2"
        }
      }
    ],
    "timeout_seconds": 100000000,
    "max_retries": 1,
    "schedule": {
      "quartz_cron_expression": "0 15 22 ? * *",
      "timezone_id": "America/Los_Angeles"
    },
    "spark_jar_task": {
      "main_class_name": "com.databricks.ComputeModels"
    }
  }
}

Request Structure

Overwrite job settings.

Run Now

Run a job now and return the run_id of the triggered run.

An example request for a notebook job:

{
  "job_id": 1,
  "notebook_params": {
    "dry-run": "true",
    "oldest-time-to-consider": "1457570074236"
  }
}

An example request for a JAR job:

{
  "job_id": 2,
  "jar_params": ["param1", "param2"]
}

Request Structure

Run a job now and return the run_id of the triggered run.

Response Structure

Runs Submit

Submit a one-time run. This endpoint doesn’t require a Databricks job to be created. You can directly submit your workload. Runs submitted via this endpoint don’t display in the UI. Once the run is submitted, use the jobs/runs/get API to check the run state.

An example request:

{
  "run_name": "my spark task",
  "new_cluster": {
    "spark_version": "5.3.x-scala2.11",
    "node_type_id": "Standard_D3_v2",
    "num_workers": 10
  },
  "libraries": [
    {
      "jar": "dbfs:/my-jar.jar"
    },
    {
      "maven": {
        "coordinates": "org.jsoup:jsoup:1.7.2"
      }
    }
  ],
  "spark_jar_task": {
    "main_class_name": "com.databricks.ComputeModels"
  }
}

And response:

{
  "run_id": 123
}

Request Structure

Submit a new run with the provided settings.

Response Structure

Runs List

List runs from most recently started to least.

An example request:

/jobs/runs/list?job_id=1&active_only=false&offset=1&limit=1

And response:

{
  "runs": [
    {
      "job_id": 1,
      "run_id": 452,
      "number_in_job": 5,
      "state": {
        "life_cycle_state": "RUNNING",
        "state_message": "Performing action"
      },
      "task": {
        "notebook_task": {
          "notebook_path": "/Users/donald@duck.com/my-notebook"
        }
      },
      "cluster_spec": {
        "existing_cluster_id": "1201-my-cluster"
      },
      "cluster_instance": {
        "cluster_id": "1201-my-cluster",
        "spark_context_id": "1102398-spark-context-id"
      },
      "overriding_parameters": {
        "jar_params": ["param1", "param2"]
      },
      "start_time": 1457570074236,
      "setup_duration": 259754,
      "execution_duration": 3589020,
      "cleanup_duration": 31038,
      "trigger": "PERIODIC"
    }
  ],
  "has_more": true
}

Request Structure

List runs from most recently started to least.

Response Structure

Runs Get

Retrieve the metadata of a run.

An example request:

/jobs/runs/get?run_id=452

An example response:

{
  "job_id": 1,
  "run_id": 452,
  "number_in_job": 5,
  "state": {
    "life_cycle_state": "RUNNING",
    "state_message": "Performing action"
  },
  "task": {
    "notebook_task": {
      "notebook_path": "/Users/donald@duck.com/my-notebook"
    }
  },
  "cluster_spec": {
    "existing_cluster_id": "1201-my-cluster"
  },
  "cluster_instance": {
    "cluster_id": "1201-my-cluster",
    "spark_context_id": "1102398-spark-context-id"
  },
  "overriding_parameters": {
    "jar_params": ["param1", "param2"]
  },
  "start_time": 1457570074236,
  "setup_duration": 259754,
  "execution_duration": 3589020,
  "cleanup_duration": 31038,
  "trigger": "PERIODIC"
}

Request Structure

Retrieve the metadata of a run without any output.

Response Structure

Runs Export

Export and retrieve the job run task.

An example request:

/jobs/runs/export?run_id=452

An example response:

{
  "views": [ {
    "content": "<!DOCTYPE html><html><head>Head</head><body>Body</body></html>",
    "name": "my-notebook",
    "type": "NOTEBOOK"
  } ]
}

To extract the HTML notebook from the JSON response, download and run this Python script.

Request Structure

Retrieve the export of a job run task.

Response Structure

Runs Cancel

Cancel a run. The run is canceled asynchronously, so when this request completes, the run may still be running. The run will be terminated shortly. If the run is already in a terminal life_cycle_state, this method is a no-op.

An example request:

{
  "run_id": 453
}

Request Structure

Cancel a run. The run is canceled asynchronously, so when this request completes the run may be still be active. The run will be terminated as soon as possible.

Runs Get Output

Retrieve the output of a run. When a notebook task returns a value through the dbutils.notebook.exit() call, you can use this endpoint to retrieve that value. Azure Databricks restricts this API to return the first 5 MB of the output. For returning a larger result, you can store job results in a cloud storage service.

Runs are automatically removed after 60 days. If you to want to reference them beyond 60 days, you should save old run results before they expire. To export using the UI, see Export job run results. To export using the Job API, see Runs Export.

An example request:

/jobs/runs/get-output?run_id=453

And response:

{
  "metadata": {
    "job_id": 1,
    "run_id": 452,
    "number_in_job": 5,
    "state": {
      "life_cycle_state": "TERMINATED",
      "result_state": "SUCCESS",
      "state_message": ""
    },
    "task": {
      "notebook_task": {
        "notebook_path": "/Users/donald@duck.com/my-notebook"
      }
    },
    "cluster_spec": {
      "existing_cluster_id": "1201-my-cluster"
    },
    "cluster_instance": {
      "cluster_id": "1201-my-cluster",
      "spark_context_id": "1102398-spark-context-id"
    },
    "overriding_parameters": {
      "jar_params": ["param1", "param2"]
    },
    "start_time": 1457570074236,
    "setup_duration": 259754,
    "execution_duration": 3589020,
    "cleanup_duration": 31038,
    "trigger": "PERIODIC"
  },
  "notebook_output": {
    "result": "the maybe truncated string passed to dbutils.notebook.exit()"
  }
}

Request Structure

Retrieves both the output and the metadata of a run.

Response Structure

Runs Delete

Delete a non-active run. Returns an error if the run is active.

An example request:

{
  "run_id": 42
}

Request Structure

Retrieve the metadata of a run without any output.

Data Structures

ClusterInstance

Identifiers for the cluster and Spark context used by a run. These two values together identify an execution context across all time.

ClusterSpec

CronSchedule

Job

JobEmailNotifications

JobSettings

Settings for a job. These settings can be updated using the resetJob method.

JobTask

NewCluster

NotebookOutput

NotebookTask

All the output cells are subject to the size of 8MB. If the output of a cell has a larger size, the rest of the run will be cancelled and the run will be marked as failed. In that case, some of the content output from other cells may also be missing. If you need help finding the cell that is beyond the limit, run the notebook against an interactive cluster and use this notebook autosave technique.

ParamPair

Name-based parameters for jobs running notebook tasks.

Run

All the information about a run except for its output. The output can be retrieved separately with the getRunOutput method.

RunParameters

Parameters for this run. Only one of jar_params, python_params, or notebook_params should be specified in the run-now request, depending on the type of job task. Jobs with Spark JAR task or Python task take a list of position-based parameters, and jobs with notebook tasks take a key value map.

RunState

SparkJarTask

SparkPythonTask

SparkSubmitTask

For example, assuming the JAR is uploaded to DBFS, you can run SparkPi by setting the following parameters.

{
  "parameters": [
    "--class",
    "org.apache.spark.examples.SparkPi",
    "dbfs:/path/to/examples.jar",
    "10"
  ]
}

ViewItem

The exported content is in HTML format. For example, if the view to export is dashboards, one HTML string is returned for every dashboard.

RunLifeCycleState

The life cycle state of a run. Allowed state transitions are:

• PENDING -> RUNNING -> TERMINATING -> TERMINATED
• PENDING -> SKIPPED
• PENDING -> INTERNAL_ERROR
• RUNNING -> INTERNAL_ERROR
• TERMINATING -> INTERNAL_ERROR

RunResultState

The result state of the run.

• If life_cycle_state = TERMINATED: if the run had a task, the result is guaranteed to be available, and it indicates the result of the task.
• If life_cycle_state = PENDING, RUNNING, or SKIPPED, the result state is not available.
• If life_cycle_state = TERMINATING or lifecyclestate = INTERNAL_ERROR: the result state is available if the run had a task and managed to start it.

Once available, the result state never changes.

TriggerType

These are the type of triggers that can fire a run.

ViewType

ViewsToExport

View to export: either code, all dashboards, or all.