We have a 500.000 image labeling project in progress. The team was receiving some errors that indicated no work was queued even though the project was only 3% complete. We paused the project and attempted to restart to see if that would clear out the issue but when trying to restart an error is being thrown "Failed" with no additional information.

The issue has been resolved. There was some corruption in the database that was corrected by the product team.

@Dave Hartman Thanks for the question. Please share details of your experiment and issue from the ml.azure.com portal for a service engineer to lookup the issue from the back-end? This option is available from the top right hand corner of the portal by clicking the smiley face, Please select the option Microsoft can email you about the feedback along with a screen shot so our service team can lookup and advise through email.

My labeling project is also now frozen after an attempt to resume from paused... it has been trying to resume for >2 hours now....

Issue starting labeling project

Dave Hartman 1

We have a 500.000 image labeling project in progress. The team was receiving some errors that indicated no work was queued even though the project was only 3% complete.

132586-242115502-983432182220136-1351421978890006110-n.png

We paused the project and attempted to restart to see if that would clear out the issue but when trying to restart an error is being thrown "Failed" with no additional information.

3 answers

Dave Hartman 6 Reputation points

2021-09-20T07:42:01.74+00:00

The issue has been resolved. There was some corruption in the database that was corrected by the product team.
Please sign in to rate this answer.

1 person found this answer helpful.

0 comments No comments
Sign in to comment
Ramr-msft 17,611 Reputation points

2021-09-17T12:50:14.3+00:00

@Dave Hartman Thanks for the question. Please share details of your experiment and issue from the ml.azure.com portal for a service engineer to lookup the issue from the back-end? This option is available from the top right hand corner of the portal by clicking the smiley face, Please select the option Microsoft can email you about the feedback along with a screen shot so our service team can lookup and advise through email.
Please sign in to rate this answer.

0 comments No comments
Sign in to comment
eem 1 Reputation point

2021-09-27T17:43:47.913+00:00

My labeling project is also now frozen after an attempt to resume from paused... it has been trying to resume for >2 hours now....
Please sign in to rate this answer.
eem 1 Reputation point

2021-09-27T23:07:59.237+00:00

After about 3-4 hours, I receieved a notification in the portal stating 'Project Failed' and the labling project went back to the 'paused' state'. I attempted to start the labeling project and it is now 'frozen' again.

Thanks in Advance,
eem

Shu 1 Reputation point

2021-09-28T16:58:52.433+00:00

As mentioned in ramr-msft 's reply, please use the smiley face in the UI to provide feedback or report an issue, so that we know how to reach out and and provide update on your issue.

Besides a screenshot, if you can provide the project ID, that will be really helpful. The project ID is a GUID string showing in your URL, like "b025d4ff-3205-6144-19f6-fdad88269c3f" in the case of "https://ml.azure.com/labeling/project/b025d4ff-3205-6144-19f6-fdad88269c3f/...".

Thanks!

eem 1 Reputation point

2021-09-28T17:53:14.173+00:00

Hello,

I submitted the 'Feedback' form with a screenshot not long ago... I can submit another. I will include the project id in the form... thanks... the project is currently in 'failed/paused' state. I think it is possibly related to a frozen inference job state record in backend... or some other ML assist RI issue...

eem 1 Reputation point

2021-10-01T11:42:44.863+00:00

Hello Shu,

I responded as well through e-mail...

Hello,

After the 'backend' fix to this project, I was able to resume and label now by also turning off the ML Assist which appeared to be the inference job was holding the lock and was stuck. A warning icon appeared on the project's lower section listing ML Assist jobs and stated the inference job could no longer find it's compute. Disabling ML Assist had no impact on being able to resume prior to the backend change as the project would remain in the paused state after attempting to resume for a couple of hours...assuming this was some sort of time out.. anyway.. The project is now running again.

On a related note...You (or anyone) wouldn't happen to have a pointer to or access to C# code using ML.NET to pre-process and score images and then render the mask points on the results using when using the Mask rCNN models generated from the AutoML job processing the label data would you please? The fastcnn ML.NET code is being generated by Model Builder in VS2022, but I can't seem to find C# code with ML.NET for the MaskCNN model being generated by AutoML jobs from this 'mask' oriented label project.

I am able to use the fastcnn generated ONNX models from the labeled data no issues with ML.NET and .NET6...

Thanks Again!
eem

eem 1 Reputation point

2021-10-01T12:27:20.897+00:00

Hello Again.... now another label project in the same workspace with ML Assist 'Training' job is failing at the very end when attempting to start the Inferencing job...

Encountered an internal AutoML error. Error Message/Code: UserErrorException. Additional Info: UserErrorException:
Message: Request failed (400): Invalid dataflow specified. Execution failed with exception. InternalServerError,
InnerException None
ErrorResponse
{
"error": {
"code": "UserError",
"message": "Request failed (400): Invalid dataflow specified. Execution failed with exception. InternalServerError, "
}
}
The training job appears to run ok as the output is produced into the blobstore correctly.

Thanks,
eem
Sign in to comment