question

bitshift-8252 avatar image
0 Votes"
bitshift-8252 asked ·

Azure functions, web jobs or Azure Data Factory ADF - example scenarios ?

Trying to choose between Azure functions, web jobs or looking to something like Azure Data Factory
Anyone

Which Azure services fit this need - moving 1-10MB data files with transformations
- Data files must be imported, read and the data "shaped" into a hieararchical JSON structure to feed another import process
- During the process of reading/extracting the data in each file, lookups must be performed to fill in missing/required values
- The transformations will also involve shifting different collections around within each record/object
- This activity needs to be scueduled
- As suspected, one or more steps might take more than the 5 min limit imposed on Azure functions

Since I am a .net coder, I tend to look for solutions that are code-centric. Hence, I am drawn to Azure functions but im not sure this is the right choice.
I have done a fair amount of work with SSIS and have started looking at ADF (Azure Data Factory) but I would tend to lean toward the code-centric approach.
The thing with functions that cause me to pause a bit is when I consider just how much processing must be done with each file, which will contain 100+ orders and their details.

azure-data-factoryazure-functions
· 2
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Hello @bitshift-8252 ,

We haven’t received any email from you as requested in the last response and was just checking back to see if you have a resolution.In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, please respond back with the more details as requested and we will try to help .

Thanks
Himanshu

0 Votes 0 ·

Hello @bitshift-8252 ,

We haven’t received any email from you as requested in the last response and was just checking back to see if you have a resolution.In case if you have any resolution please do share that same with the community as it can be helpful to others . Otherwise, please respond back with the more details as requested and we will try to help .

Thanks
Himanshu

0 Votes 0 ·

1 Answer

HimanshuSinha-MSFT avatar image
1 Vote"
HimanshuSinha-MSFT answered ·

Hello @bitshift-8252 ,

Thanks for the ask and using the Microsoft Q&A platform .
From all in the inputs which you have provided I see that you are more inclined towards Azure Functions . I am pretty confident that you will be able to achieve the ask with that also . Since you are more code centric , this may be more helpful.

I could suggest you to evaluate the Mapping data flow (MDF ) as it does provide the the transformation option and its also lets you play with the JSON object . For MDF you will not need any coding ( just some expression ) as most of the thing are drag and drop .

Please do let me know how it goes .
Thanks
Himanshu
Please do consider to click on "Accept Answer" and "Up-vote" on the post that helps you, as it can be beneficial to other community members

·
10 |1000 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.