question

rahulsharma-4167 avatar image
0 Votes"
rahulsharma-4167 asked PRADEEPCHEEKATLA-MSFT commented

Storage exception in Azure databricks for delta lake table

Hi Friends,
I am trying to load a csv file around 800 mb into databricks delta table. i have exported the csv file into DBFS using import data at databricks workspace and then i have a python notebook to write into delta table.
when i am using below command to write

ConsumerDataDF.write.mode("overwrite").format("delta").partitionBy("Company").save("/FileStore/delta/")

The databricks is throwing me below error.

Py4JJavaError: An error occurred while calling o466.save.
: org.apache.spark.SparkException: Job aborted.
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:201)
.
.
.

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 4 in stage 10.0 failed 4 times, most recent failure: Lost task 4.3 in stage 10.0 (TID 90, 10.139.64.6, executor 1): org.apache.spark.SparkException: Task failed while writing rows.
.
.
.

Caused by: shaded.databricks.org.apache.hadoop.fs.azure.AzureException: com.microsoft.azure.storage.StorageException: One of the request inputs is out of range.

when i am limiting the data in my dataframe like below it worked fine.

ConsumerDataDF.filter("Company == 'CITIBANK, N.A.'").write.mode("overwrite").format("delta").partitionBy("Company").save("/FileStore/delta/")


I would like to know if there is any size limitation on delta tables. if not what could be issue for this error

azure-databricks
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @rahulsharma-4167,

Welcome to the Microsoft Q&A platform.

Thanks for pointing this issue. I’m working with the product team and get back to you when I have more information.

0 Votes 0 ·

1 Answer

PRADEEPCHEEKATLA-MSFT avatar image
0 Votes"
PRADEEPCHEEKATLA-MSFT answered PRADEEPCHEEKATLA-MSFT commented

Hello @rahulsharma-4167,

There are no limits on Delta tables.

Could you please checkout the location /FileStore/delta/. I would suggest you to try using mounting a blob storage container of your own and then try and write to it.

Hope this helps. Do let us know if you any further queries.


Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @rahulsharma-4167,
Just checking in to see if the above answer helped. If this answers your query, do click “Accept Answer” and Up-Vote for the same. And, if you have any further query do let us know.

0 Votes 0 ·

Hello @rahulsharma-4167,

Following up to see if the above suggestion was helpful. And, if you have any further query do let us know.

0 Votes 0 ·

Hi, I am facing the same issue. I have mounted to my own storage account. Could you please help to let me know what could be the issue.

Error: Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 12) (10.180.0.69 executor 0): org.apache.spark.SparkException: Task failed while writing rows.


Trying to write to a delta file as below:

def writeResultToParquet(inputDF,directory,filename):



save_location = directory

inputDF.write \
.format("delta") \
.mode('overwrite') \
.save(save_location)


0 Votes 0 ·
Show more comments

I try to create the delta in mount store but it seems the execution is still stuck. it didn't throw any error on notebook execution and keep running but in the driver log i found the below error

Caused by: com.microsoft.azure.storage.StorageException: One of the request inputs is out of range.
at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:87)
....
at shaded.databricks.org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.exists(StorageInterfaceImpl.java:332)
at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.retrieveMetadata(AzureNativeFileSystemStore.java:2049)
... 53 more

which is same as earlier. my input file in around 800 MB . but this should not cause any problem.
Can someone let me know what is the meaning of this error to debug this. as i told you if i filter the data it works fine. so something is related to data in my file not sure if it is bad data or data size.

0 Votes 0 ·