@Filip Hultgren , Firstly, please accept my apologies for the delayed response from over the weekend.
Based on my understanding, just to clarify – You want clients that arrive with some certain IP range to not require AAD, but then you also want to have a different group of clients who do get forced to authenticate.
You may try this approach based on your requirement:
All of the clients would need to be whitelisted in IP access restrictions, otherwise they simply cannot arrive at the site. You can add some customization of the AAD auth. It's not IP access restriction and AAD as separate methods, rather -it's IP restrictions for everyone + AAD for some of users both happening together. You could customize access with AAD based on your needs.
Kindly checkout these docs for more info:
Set up Azure App Service access restrictions
Configure your App Service or Azure Functions app to use Azure AD login
Tutorial: Add authentication to your web app running on Azure App Service