question

Enable diagnostics logging for apps in Azure App Service, specifically sending logs to Azure Monitor, is a doc covering the various methods for capturing logging information. Sending logs to Azure Monitor will allow you to query AppServiceHTTPLogs. Application Insights Auto-instrumentation on the other hand, currently works with select frameworks out of the box and captures certain telemetry. Either method should capture failed requests.

Having said that, there isn't any feature in the Azure Portal that will allow you to easily collect login information. Since you're specifically looking for logon information, that type logging will be determined by the authentication method you selected. It's possible that the login information isn't in the header and furthermore, that particular header is not captured in the raw web server logs.

Capturing this information is to log it yourself. I would add console log tracing when a user successfully logs in, logs out, or fails to login. This would require adding the necessary code and redeploying the app. I'm not sure if your concern about not redeploying the app is centered around not being able to alter the code, but this would be the best course of action.