ML Studio endpoint problem after Aks/VM Restart

Luciano 6

After restarting VM's associated to AKS created with to ML Studio deployed the ws endpoint gets unreachable.

Try several time to restart but it doesn't work.

Any idea? Any workaround besides re-create the cluster?

romungi-MSFT 41,841 Reputation points Microsoft Employee

2020-11-17T08:24:36.107+00:00

@Luciano Could you post any logs of your endpoint deployment? Are you using an existing AKS cluster and restarting it from Azure portal or is this a inference cluster created from Azure ML portal ml.azure.com?

Luciano 6

Hi!.

   I started the vm  this morning again and it start it to work. I don't have logs beacause the last one is the one of this "normal" startup. Is there any other place to see log details?, besides that part, if it's help to diagnose it,  we created the AKS Cluster with Azure ML Portal automatically and we shutdown VM's from the azure portal (as any vm). Is this a correct approach?

romungi-MSFT 41,841 Reputation points Microsoft Employee

2020-11-17T13:56:34.303+00:00

It is preferable to detach the cluster and then shut it down instead of shutting down the VMs directly.
Luciano 6 Reputation points

2020-11-17T14:22:25.067+00:00

Is preferable or the correct way to do it?

And the other part, after dettach i have to shutdown manually the VM's and then start it and re-attach it to the AKS, Right?

Thanks.
romungi-MSFT 41,841 Reputation points Microsoft Employee

2020-11-18T06:45:12.573+00:00

@Luciano It is preferable and correct way to do it from the ML portal.
You can use scaling option to set the minimum number of VMs so you don't get billed when there are no requests for compute and your cluster will scale up/down automatically without the need to manually start/shut your VMs.
Luciano 6 Reputation points

2020-11-26T02:08:56.14+00:00

We are having again the same issue. And this time we tried to detacha before shutdown VM and it doesn't work either. We startup the VM, attach again and we see the message "Provisionan State = Failed".
romungi-MSFT 41,841 Reputation points Microsoft Employee

2020-11-30T10:08:06.457+00:00

@Luciano Is it possible to provide details of your subscription and compute which are failing to start to our team from the feedback option on the top right corner of ML portal ml.azure.com?

Please select the last option to email you on the feedback so they can advise you on how to avoid such failures for future scenarios.
Luciano 6 Reputation points

2020-11-30T12:02:20.74+00:00

Yes. I've done that