I am trying to load a csv file around 800 mb into databricks delta table. i have exported the csv file into DBFS using import data at databricks workspace and then i have a python notebook to write into delta table.
when i am using below command to write
The databricks is throwing me below error.
Py4JJavaError: An error occurred while calling o466.save.
: org.apache.spark.SparkException: Job aborted.
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 10.0 failed 4 times, most recent failure: Lost task 4.3 in stage 10.0 (TID 90, 10.139.64.6, executor 1): org.apache.spark.SparkException: Task failed while writing rows.
Caused by: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: One of the request inputs is out of range.
when i am limiting the data in my dataframe like below it worked fine.
ConsumerDataDF.filter("Company == 'CITIBANK, N.A.'").write.mode("overwrite").format("delta").partitionBy("Company").save("/FileStore/delta/")
I would like to know if there is any size limitation on delta tables. if not what could be issue for this error