question

AnirudThapliyal-9057 avatar image
0 Votes"
AnirudThapliyal-9057 asked AnirudThapliyal-9057 edited

Implementing parallel processing of sequential tasks

I am having a asp.net core web application hosted as app service on Azure. This application takes the files uploaded by users as input. The records in this file (which can be in thousands) has to be processed by 3 separate processes. These 3 processes are time taking processes which will run in background as windows services as these services needs to access on-premise file system and they must run in sequence for each user to process the records in the uploaded file.

Let's say process from file upload till last process(Process1, Process2, Process3) as a work. So, inside single user the processes needs to run in sequential order but works should run in parallel.

User1 ----> File Upload > Process1(if File Upload is success) > Process2(if Process1 is success) > Process3(if Process2 is success) [In sequential order]
User2 ----> File Upload > Process1(if File Upload is success) > Process2(if Process1 is success) > Process3(if Process2 is success) [In sequential order]
User3 ----> File Upload > Process1(if File Upload is success) > Process2(if Process1 is success) > Process3(if Process2 is success) [In sequential order]

User1, User1 and User3 can be parallel.

When user uploads a file with 1 record, this web application will insert 1 message in azure service bus queue(which is session enabled queue). Each message in queue must be processed by all 3 windows services(Process1, Process2 and Process3) in sequential order.
But for different users the execution should be in parallel so that execution process of user1 must not delay the execution for user2. For that I am trying to use concurrent sessions of Azure service bus.

Not sure if this is correct use case for Azure service bus or not, hence I need if anyone could put me in right direction.
I am looking help for design solution, appreciate . Any ideas and implementation suggestions are welcome.
I can also provide code which I am using to send and receive concurrent messages of Azure service bus session enabled queue.

dotnet-csharpazure-webappsazure-service-bus
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ASP.NET Core requests are handled in separate threads. Are the files uploaded and processed at a later time?

0 Votes 0 ·

1 Answer

BruceBarker-8516 avatar image
0 Votes"
BruceBarker-8516 answered

azure durable functions support your case:

durable-functions-overview


5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.