question

RaphaelKoning-2501 avatar image
0 Votes"
RaphaelKoning-2501 asked HimanshuSinha-MSFT commented

How to copy all and update API data

I'm trying to copy data from an API to blob storage but because of performence reasons it is not possible to return all the data at once. It only gives 10 elements and it is possible to get other pages using the 'page' parameter. In the first screenshot you can see the data it returns when copying 'meters'. Column _embedded row meters contains the 10 rows. How is it possible to copy and afterwards update all the data to blob storage and maybe combine different dataframes in the process(just copying it all at once every few minutes might not be possible because one of the data sources contains 10252096 elements)? I'm new to azure so I'm having some difficulties with the basics like adding parameters(like 'page') and that kind of stuff. I also put some screenshots of how I copy the data to make it a bit more clear what i did.

Feel free to ask me some more information if anything is unclear!
Thanks :)

78635-image.png
78741-image.png
78722-image.png
78626-image.png
78687-image.png


azure-synapse-analytics
image.png (21.7 KiB)
image.png (22.9 KiB)
image.png (16.0 KiB)
image.png (29.7 KiB)
image.png (28.3 KiB)
· 2
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @RaphaelKoning-2501,
Thanks for the ask and using the Microsoft Q&A platform .
As I understand you have the data in the blob in many different JSON and I understand the intent is to aggregare all the JSON in one as if you were response of a single request .
I am not sure if you have already read this . I think this read , can help you


Please do let me know how it goes .
Thanks
Himanshu


0 Votes 0 ·

Hello @RaphaelKoning-2501 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, will respond back with the more details and we will try to help .
Thanks
Himanshu

0 Votes 0 ·

0 Answers