The Databricks REST API 2.0 supports services to manage your workspace, DBFS, clusters, instance pools, jobs, libraries, users and groups, tokens, and MLflow experiments and models.
This article provides an overview of how to use the REST API. Links to each API reference, authentication options, and examples are listed at the end of the article.
For information about authenticating to the REST API using personal access tokens, see Authentication using Azure Databricks personal access tokens. For API examples, see API examples.
For information about authenticating to the REST API using Azure Active Directory tokens, see Authenticate using Azure Active Directory tokens. For examples, see Use an Azure AD access token for a user and Use an Azure AD access token for a service principal.
To ensure high quality of service under heavy load, Databricks enforces rate limits for all REST API calls. Limits are set per endpoint and per workspace to ensure fair usage and high availability. To request a limit increase, contact your Databricks representative.
Requests that exceed the rate limit return a 429 response status code.
It can be useful to parse out parts of the JSON output. In these cases, we recommend that you to use the utility
jq. For more information, see the jq Manual. You can install
jq on MacOS using Homebrew by running
brew install jq.
STRING fields (which contain error/descriptive messaging intended to be consumed by the UI) are unstructured, and you should not depend on the format of these fields in programmatic workflows.
Invoke a GET using a query string
While most API calls require that you specify a JSON body, for
GET calls you can specify a query string.
In the following examples, replace
<databricks-instance> with the workspace URL of your Azure Databricks deployment.
<databricks-instance> should start with
adb-. Do not use the deprecated regional URL starting with
<azure-region-name>. It may not work for new workspaces, will be less reliable, and will exhibit lower performance than per-workspace URLs.
To get the details for a cluster, run:
curl ... https://<databricks-instance>/api/2.0/clusters/get?cluster_id=<cluster-id>
To list the contents of the DBFS root, run:
curl ... https://<databricks-instance>/api/2.0/dbfs/list?path=/
Many API calls require you to specify a Databricks runtime version string. This section describes the structure of a version string in the Databricks REST API.
M- Databricks Runtime major release
F- Databricks Runtime feature release
cpu- CPU version (with
ml- Machine learning
conda- with Conda (no longer available)
scala-version- version of Scala used to compile Spark: 2.10, 2.11, or 2.12
M- Apache Spark major release
F- Apache Spark feature release
scala-version- version of Scala used to compile Spark: 2.10 or 2.11
- API examples
- Authenticate using Azure Active Directory tokens
- Authentication using Azure Databricks personal access tokens
- Clusters API
- Cluster Policies APIs
- DBFS API
- Global Init Scripts API
- Groups API
- Instance Pools API
- IP Access List API
- Jobs API
- Libraries API
- MLflow API
- Permissions API
- SCIM API
- Secrets API
- Token API
- Token Management API
- Workspace API