Your app is sharing infrastructure with other apps being deployed in that same region, so there are effectively multiple apps sharing a pool of IP addresses.
When a request comes in to an IP address with no hostname, the system will not know where to route it.
The domain is needed (<yourappname>.azurewebsites.net) so that is can be routed to the right VM hosting it.
IF you NEED to access the app over an IP instead of a hostname you will need to get a dedicated IP address. To do this you will need a custom domain + certificate and create an IP SSL binding. For this to work you app will need to be hosted in a basic of above App Service Plan