@Tim Jaszdziewski Thank you for your question.
Yes, you are going down the right path in terms of using Azure DevOps and Pipelines. This sample here talks about deploying your code to multiple apps.
The good news is you can run multiple apps on a single App Service Plan (think VM), which can help with cost savings.
Please let me know if there are any more specific questions that you might have about web apps or always feel free to create a new post. We're here to help you.