question

DaveHartman-9746 avatar image
0 Votes"
DaveHartman-9746 asked eem-0564 commented

Issue starting labeling project

We have a 500.000 image labeling project in progress. The team was receiving some errors that indicated no work was queued even though the project was only 3% complete.

132586-242115502-983432182220136-1351421978890006110-n.png

We paused the project and attempted to restart to see if that would clear out the issue but when trying to restart an error is being thrown "Failed" with no additional information.

132616-screenshot-2021-09-16-102056.png


azure-machine-learning
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

ramr-msft avatar image
0 Votes"
ramr-msft answered DaveHartman-8849 commented

@DaveHartman-9746 Thanks for the question. Please share details of your experiment and issue from the ml.azure.com portal for a service engineer to lookup the issue from the back-end? This option is available from the top right hand corner of the portal by clicking the smiley face, Please select the option Microsoft can email you about the feedback along with a screen shot so our service team can lookup and advise through email.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

DaveHartman-8849 avatar image
1 Vote"
DaveHartman-8849 answered

The issue has been resolved. There was some corruption in the database that was corrected by the product team.

5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

eem-0564 avatar image
0 Votes"
eem-0564 answered eem-0564 commented

My labeling project is also now frozen after an attempt to resume from paused... it has been trying to resume for >2 hours now....135600-stuck-labeler-capture1.png



· 5
5 |1600 characters needed characters left characters exceeded

Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

After about 3-4 hours, I receieved a notification in the portal stating 'Project Failed' and the labling project went back to the 'paused' state'. I attempted to start the labeling project and it is now 'frozen' again.

Thanks in Advance,
eem135667-failed-labeler-capture1.png


0 Votes 0 ·

As mentioned in ramr-msft 's reply, please use the smiley face in the UI to provide feedback or report an issue, so that we know how to reach out and and provide update on your issue.

Besides a screenshot, if you can provide the project ID, that will be really helpful. The project ID is a GUID string showing in your URL, like "b025d4ff-3205-6144-19f6-fdad88269c3f" in the case of "https://ml.azure.com/labeling/project/b025d4ff-3205-6144-19f6-fdad88269c3f/...".

Thanks!

0 Votes 0 ·

Hello,

I submitted the 'Feedback' form with a screenshot not long ago... I can submit another. I will include the project id in the form... thanks... the project is currently in 'failed/paused' state. I think it is possibly related to a frozen inference job state record in backend... or some other ML assist RI issue...

0 Votes 0 ·

Hello Shu,

I responded as well through e-mail...

Hello,

After the 'backend' fix to this project, I was able to resume and label now by also turning off the ML Assist which appeared to be the inference job was holding the lock and was stuck. A warning icon appeared on the project's lower section listing ML Assist jobs and stated the inference job could no longer find it's compute. Disabling ML Assist had no impact on being able to resume prior to the backend change as the project would remain in the paused state after attempting to resume for a couple of hours...assuming this was some sort of time out.. anyway.. The project is now running again.

On a related note...You (or anyone) wouldn't happen to have a pointer to or access to C# code using ML.NET to pre-process and score images and then render the mask points on the results using when using the Mask rCNN models generated from the AutoML job processing the label data would you please? The fastcnn ML.NET code is being generated by Model Builder in VS2022, but I can't seem to find C# code with ML.NET for the MaskCNN model being generated by AutoML jobs from this 'mask' oriented label project.

I am able to use the fastcnn generated ONNX models from the labeled data no issues with ML.NET and .NET6...

Thanks Again!
eem

0 Votes 0 ·
Show more comments