Is it possible to store logs and metrics of an Azure Machine Learning AKS online endpoint?

G Cocci 211 Reputation points Microsoft Employee
2024-04-05T08:42:34.88+00:00

Hi,

I need to store some logs and metrics coming from an AKS online endpoint created inside an Azure Machine Learning Workspace.

I configure two different diagnostic settings both at workspace and endpoint level, setting that all the logs and metrics will be sent to a Log Analytics Workspace and also stored inside an Azure Storage Account.

Despite this configuration, the three log tables available at endpoint level (AmlOnlineEndpointTrafficLog, AmlOnlineEndpointConsoleLog, AmlOnlineEndpointEventLog) remain empty in both the Log Analytics Workspace and the Storage Account (I have already waited a few days, but nothing has changed).

As far as metrics are concerned, only a subset is saved in the Storage Account (RequestPerMinute, Latency) and none of the metrics available for deployment (e.g. CPU, GPU and Memory Utilization).

Are the three log tables mentioned above also available for AKS online endpoints or only for managed online endpoints? Is it possible to historize metrics at the deployment level or can they only be viewed within Azure Monitor?

Thanks,

Giorgio

Azure Machine Learning
Azure Machine Learning
An Azure machine learning service for building and deploying models.
2,572 questions
Azure Kubernetes Service (AKS)
Azure Kubernetes Service (AKS)
An Azure service that provides serverless Kubernetes, an integrated continuous integration and continuous delivery experience, and enterprise-grade security and governance.
1,867 questions

1 answer

Sort by: Most helpful
Most helpful Newest Oldest
  1. YutongTie-MSFT 46,986 Reputation points
    2024-04-06T06:04:26.8966667+00:00

    @G Cocci

    Thanks for reaching out to us.

    Azure Kubernetes Service (AKS) online endpoints in Azure Machine Learning do support logging and metrics, but the availability of specific logs and metrics may depend on the exact configuration and setup of your service. The three log tables you mentioned (AmlOnlineEndpointTrafficLog, AmlOnlineEndpointConsoleLog, AmlOnlineEndpointEventLog) should be available for AKS online endpoints. However, please ensure that you've enabled diagnostics for your online endpoints and that you've correctly set up the integration with Log Analytics.

    As for metrics, Azure Monitor should be able to collect a variety of metrics, including CPU utilization, memory utilization, and network traffic. If you're not seeing these metrics, it may be due to a configuration issue.

    Please also note that there can be a delay between when data is generated and when it appears in Log Analytics, usually around a few minutes, but it can be longer.

    I hope this helps.

    Regards,

    Yutong

    -Please kindly accept the answer if you feel helpful to support the community, thanks a lot.

    0 comments