We have code that uses parallelism (Parallel for loops) that is mostly in our services and scheduled tasks. We also have api code using concurrency (await/async). Ideally we would like to be able to share common Business logic between the two types of apps but when combined it has a tendency to create intermittent locking errors at various points so I am looking at ways to learn how to do better.
One thought was to look at TPL Dataflow. I am looking at TPL Dataflow tutorials and am trying to determine if this is a good fit for a web api and if so where would it get inserted into the pipeline? Obviously not in the controller but but beyond that I am struggling to return data to the client. One of the tutorials I found showed a TransformBlock<Request,Response> followed by an ActionBlock<Response> (here is one I followed) in a class which is fine but it writes to the console but I am not sure how use this pattern in the context of an api. https://www.blinkingcaret.com/2019/06/05/tpl-dataflow-in-net-core-in-depth-part-2/.
Essentially I am wondering if TPL Dataflow is a tool that I would consider using instead of a Parallel.ForEach in a task/service because using only concurrency it might cut down on our async/await method errors called by the Parallel threads but isn't appropriate. We didn't have much of a problem until a developer tried to do bulk operations (looping over our atomic actions) that were slow and then tried to mitigate that using a Parallel For loop which seems to cause timeouts.
Any direction as it relates to TPL Dataflow as a replacement for Parallel foreach, if they play better with aysnc/await and lastly in the case where we had bulk processing inside an API is this a bad idea?
Thank you.