Hello,
I'm developing a data analysis tool for the processing of data from Hydrogen-Deuterium exchange mass spectrometry. We would like to accompany our publication with a deployment of the code on Microsoft Azure so that other researchers can quickly and easily try our software without needing to install it.
The application is written in Python and we use the package Panel to create a web interface. Although I have not yet done so myself I've found instructions on how to deploy these panel dashboards on microsoft azure, so I think this step won't be a problem.
The main difficulty is that the webapp has two computationally intensive tasks, which have high peak CPU usage but are sparse, as I expect that if 3 users use the app simultaneously it will be a rare high amount of users while mostly no user will be using the app. So I'm wondering how to properly deploy this on Azure, where we can pay per CPU/GPU cycle for the computationally intensive tasks and keep the hardware requirements for the web interface part minimal.
The first computationally intensive task is a normal CPU fitting procedure. I've already separated this from the main web interface by using Dask, which according to their docs I can deploy on microsoft azure using Kubernetes and Helm. I this plan a sensible approach to doing this? Is it indeed necessary to separate both or can I configure Azure differently to deal with this?
The second computationally intensive task is a tensorflow fit, where I've repurposed the neural network architecture to solve my specific problem. The memory requirement for this step is quite low and I run this on the GPU of my laptop (2060) which then completes in a reasonable time (typ. 5 minutes). How can I best do tensorflow (GPU) jobs and submit then remotely? Should I also schedule this using Dask?
Perhaps I should use Databricks?
This is my first time playing around with Azure, so any general pointers are welcome. So far the plethora of options is quite overwhelming.