question

HazelErickson-8845 avatar image
0 Votes"
HazelErickson-8845 asked romungi-MSFT answered

Custom Vision Stuck on 'Training'

I cannot train any models on Custom Vision. They stay stuck on 'Training' regardless if I choose quick or advanced training, and it stays stuck even after the maximum training time. This problem started a couple days ago, and I haven't had any problems in the past training with the same data/ methods.

This is an example of the iteration details while the training job is in progress:

Iteration status: Training
Iteration Name: Iteration 1
Iteration training time (mins): 0
Iteration reserved budget time (hours): 12
Iteration training type: Advanced
Iteration classification type: None
Iteration error details: None

Yesterday I set a training job for 12 hours and it still had 0 mins for training time after 12 hours, so I had to manually delete the iteration.

I am used both the python SDK and azure custom vision GUI to start the training job, with the same results.

azure-machine-learningazure-custom-vision
· 1
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

@HazelErickson-8845 Is it possible to share details of the region in which your resource is based? I am checking internally for any known issues, if more details are required we will reach out to you for more information.

0 Votes 0 ·

1 Answer

romungi-MSFT avatar image
0 Votes"
romungi-MSFT answered

@HazelErickson-8845 A backend issue caused training jobs to fail or they got stuck. This issue is now fixed.
All jobs that were pending should be in failed or success state now.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.