Getting GraphApiJsonThrottlePerAppAndTenantThresholdExceeded Error in Data Factory

Question

question

dineshkumarrajan-1021 asked Jul 19, '21 Levisyyds-7133 edited Apr 29, '22

Getting GraphApiJsonThrottlePerAppAndTenantThresholdExceeded Error in Data Factory

Hi Team,

I am reading Office 365 Data using graph api in Azure Data Factory. i am reading data in JSON format and stored in ADLS Gen 2.

i am getting this 'GraphApiJsonThrottlePerAppAndTenantThresholdExceeded' error while reading data. pasted the complete error message for reference. generating access token through WEB activity by passing Service principal. clientid etc.

In Copy Activity -- > created rest api linked service to read office data with Authorization header and passing value to the Pagination Rules parameter value as AbsoluteUrl: $['@odata.nextLink'].

i am getting the JSON file when the volume of JSON is less. i executed the same URL in PostMan by providing the access token, it shows the JSON data without any error.

Failure happened on 'Source' side. ErrorCode=RestSourceCallFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The HttpStatusCode 429 indicates failure. Request URL: https://graph.microsoft.com/beta/reports/getOffice365ActiveUserDetail(period='D7')?$format=application%2fjson&$top=1000&$skiptoken=MDoxNzc6ZXlKMGIydGxiaUk2SW Response payload:{"error":{"code":"UnknownError","message":"{\"error\":{\"code\":\"GraphApiJsonThrottlePerAppAndTenantThresholdExceeded\",\"message\":\"Please retry later\"}}","innerError":{"date":"2021-07-19T09:51:26","request-id":"a70da1c8-ec1d-4ff3-XXXX-a6a99cXXXXXX","client-request-id":""}}},Source=Microsoft.DataTransfer.ClientLibrary,'

could you please help us to resolve this. kindly provide if any other implementation approach is to be incorporated.

azure-data-factory microsoft-graph-reports

· 5

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ShaikMaheer-MSFT · Jul 20, 2021 at 12:26 PM

Hi @dineshkumarrajan-1021 ,

Welcome to Microsoft Q&A Platform. Thank you for posting your query here.

Could you please help on below details to troubleshoot issue.

What is the Volume of your json in case of failures? You said from postman its working right. So try failure cases from postman and try to check size of your response.
When you say its working fine when json volume is low, You mean when response size is low? If yes, what is the size of it?

0 ·

dineshkumarrajan-1021 ShaikMaheer-MSFT · Jul 20, 2021 at 03:49 PM

Working fine indicates the volume is just 10 records.
I removed the pagination rules parameter and tried client side paging by mentioning ?top=10000, then i got the response JSON and pipeline execution completed.

Pipeline failed when the JSON response volume is huge, executed the same request in postman and get the data as text format. Record Count is: 315308

I tried the failure case in postman, got the response json first page and it had odata.nextLink URL for the next page also. the same case not worked in ADF pipeline.

this is json response from postman

{
"@odata.context": "https://graph.microsoft.com/beta/$metadata#Collection(office365ActiveUserDetail)",
"@odata.nextLink": "https://graph.microsoft.com/beta/reports/getOffice365ActiveUserDetail(period='D7')?$format=application%2fjson&$skiptoken="kZSRFV4TmtKRk5UZERJbjE5WFE",
"value": { "@odata.type": "#microsoft.graph.office365ActiveUserDetail", "reportRefreshDate": "",![116317-adf-1.png
},

0 ·

adf-1.png (30.8 KiB)

adf-2.png (21.2 KiB)

Levisyyds-7133 dineshkumarrajan-1021 · Apr 29 at 12:05 PM

Try to remove the "$" you have in the value. If you review your source code for the Copy Data activity, you can see that a "$" is already automatically added to the pagination rules.

0 ·

image.png (7.6 KiB)

ShaikMaheer-MSFT · Jul 27, 2021 at 06:37 AM

Hi @dineshkumarrajan-1021 ,

Following up to check is below provided answer helped. If yes, Please Accept Answer. Accepting answer helps community too. Please let us know if any further queries.

0 ·

ShaikMaheer-MSFT · Jul 30, 2021 at 05:40 AM

Hi @dineshkumarrajan-1021 ,

Just checking is below provided answer helped. If yes, Please Accept Answer. Accepting answer helps community too. Please let us know if any further queries.

0 ·

Answer 1 · 2021-07-22T09:39:40.75Z

ShaikMaheer-MSFT answered Jul 22, '21 Levisyyds-7133 edited Apr 29, '22

Hi @dineshkumarrajan-1021 ,

Thank you for posting query in Microsoft Q&A Platform.

When copying data from REST APIs, while to return large amount of data, API splits the result into multiple pages and requires callers to send consecutive requests to get next page of the result.

Usually, the request for one page is dynamic and composed by the information returned from the response of previous page.

Check below link to know more about how pagination works in REST connector in Azure data factory.
https://docs.microsoft.com/en-us/azure/data-factory/connector-rest#pagination-support

In above link example also next page URL was present in previous page response and there as using syntax as $.paging.next to invoke same as AbsoluteUrl. Kindly try to follow steps and details mentioned in above documentation link to resolve your issue.

Hope this will help. Thank you.

Please accept an answer if correct. Original posters help the community find answers faster by identifying the correct answer. Here is how.
Want a reminder to come back and check responses? Here is how to subscribe to a notification.

· 1

5 |1600 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Levisyyds-7133 · Apr 29 at 12:07 PM

Unfortunately, the examples in the documentation you shared about pagination are obsolete. The syntax for example 8 is not correct. Hopefully it can be updated soon.

0 ·

question