This article describes how to import data into Azure Databricks using the UI, read imported data using the Spark and local APIs, and modify imported data using Databricks File System (DBFS) commands.
If you have small data files on your local machine that you want to analyze with Azure Databricks, you can import them to DBFS using the UI.
This feature may be disabled by admin users. To enable or disable this setting, see Manage data upload.
There are two ways to upload data to DBFS with the UI:
Upload files to the FileStore in the Upload Data UI.
Files imported to DBFS using these methods are stored in FileStore.
You can also use a wide variety of data sources to access data.
Read data on cluster nodes using Spark APIs
You read data imported to DBFS into Apache Spark DataFrames using Spark APIs. For example, if you import a CSV file, you can read the data using one of these examples.
For easier access, we recommend that you create a table. See Databases and tables for more information.
sparkDF = spark.read.csv('/FileStore/tables/state_income-9f7c5.csv', header="true", inferSchema="true")
sparkDF <- read.df(source = "csv", path = "/FileStore/tables/state_income-9f7c5.csv", header="true", inferSchema = "true")
val sparkDF = spark.read.format("csv") .option("header", "true") .option("inferSchema", "true") .load("/FileStore/tables/state_income-9f7c5.csv")
Read data on cluster nodes using local APIs
You can also read data imported to DBFS in programs running on the Spark driver node using local file APIs. For example:
pandas_df = pd.read_csv("/dbfs/FileStore/tables/state_income-9f7c5.csv", header='infer')
df = read.csv("/dbfs/FileStore/tables/state_income-9f7c5.csv", header = TRUE)
Modify uploaded data
To delete data from DBFS, use the same APIs and tools. For example, you can use the Databricks Utilities command
Deleted data cannot be recovered.