question

QuinnKatie-0010 avatar image
0 Votes"
QuinnKatie-0010 asked KranthiPakala-MSFT commented

Data Factory - Internal Server Error and AzureResourceProviderThrottling errors

We had multiple Data Factory pipelines fail over the last few days (4/11-4/13) with several intermittent errors.

One example of the Internal Server Error we saw:
87880-image.png

The internal server errors were on pipeline steps that were transforming the data and not moving data from a source to a sink. Is there a way to diagnose Internal Server Errors within Data Factory?

Another error we were seeing was specific to an Azure Resource Provider Throttling error. The following error message appeared:

Unexpected failure while waiting for the cluster (0412-081827-waxen351) to be ready.Cause Unexpected state for cluster (0412-081827-waxen351): AZURE_RESOURCE_PROVIDER_THROTTLING(CLOUD_FAILURE): azure_error_code:AzureResourceProviderThrottling,azure_error_message:Encountered Azure Resource Provider throttling. Please try again later. Details: ,databricks_error_message:Error code: AzureResourceProviderThrottling, error message: Encountered Azure Resource Provider throttling. Please try again later.

Is there a reason why we would be seeing an Azure Resource Provider Throttling error message? Was there an outage or planned maintenance with databricks?

The errors were intermittent, however, we saw enough instances of these errors across our pipelines to be concerned.


azure-data-factory
image.png (137.1 KiB)
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

1 Answer

KranthiPakala-MSFT avatar image
0 Votes"
KranthiPakala-MSFT answered KranthiPakala-MSFT commented

Hi @QuinnKatie-0010,

Welcome to Microsoft Q&A forum and sorry for your experience.

We usually notice the internal server error when there is an issue with the ADF dependent service Databricks.
As per my conversation with internal team, there was an outage reported by Databricks service on 4/13 probably that could be the reason you are seeing these errors. But the issue is resolved now.

In case if you still continue to receive these errors please do share the latest pipeline and activity runID's for the failed ones so that we can escalate to product team to have a deeper analysis.

You can also check the status of databricks from the status page here: https://status.azuredatabricks.net/
This page also contains info about the planned maintenance.

Hope this info helps. Please do share the pipeline and activity runID's if you ever notice these errors.

Thank you



Please don’t forget to Accept Answer and Up-Vote wherever the information provided helps you, this can be beneficial to other community members.


· 4
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hi @KranthiPakala-MSFT

I just saw another instance of an Internal Server Error. I did not expect there to be another Databricks outage based on their status page. Would you be able to provide some insight into this issue?

0 Votes 0 ·
image.png (25.8 KiB)

Hi @QuinnKatie-0010,

Sorry for your experience and thanks for getting back. Could you please share the failed pipeline and activity runID so that I can escalate to internal team to check on the backend logs to figure out the root cause.

We look forward to your response.

Thank you

0 Votes 0 ·

Hi @KranthiPakala-MSFT,

I believe the pipeline runID is 58619738-03b6-4279-b8d2-df95448dcad7 and the activity runID is d167158a-7071-4b05-9f81-d92d8c2e3176.

0 Votes 0 ·
Show more comments