Trying to choose between Azure functions, web jobs or looking to something like Azure Data Factory
Anyone
Which Azure services fit this need - moving 1-10MB data files with transformations
- Data files must be imported, read and the data "shaped" into a hieararchical JSON structure to feed another import process
- During the process of reading/extracting the data in each file, lookups must be performed to fill in missing/required values
- The transformations will also involve shifting different collections around within each record/object
- This activity needs to be scueduled
- As suspected, one or more steps might take more than the 5 min limit imposed on Azure functions
Since I am a .net coder, I tend to look for solutions that are code-centric. Hence, I am drawn to Azure functions but im not sure this is the right choice.
I have done a fair amount of work with SSIS and have started looking at ADF (Azure Data Factory) but I would tend to lean toward the code-centric approach.
The thing with functions that cause me to pause a bit is when I consider just how much processing must be done with each file, which will contain 100+ orders and their details.
