Azure language service custom text classification training error: Bad Gateway, content: upstream connect error or disconnect/reset before headers. reset reason: protocol error

Question

I am training/testing a custom classification model. I have 26 document types. each type has about 200 text files. total is 5200 files. all files are stored in azure storage. Training inside language studio.

50/50 split:

I see this error three times, very repeatable:

Show error details__

1. Evaluation Job Failed, failure msg: Failed to predict documents for the following reason: Http client call failed, message: reasonPhrase: Bad Gateway, content: upstream connect error or disconnect/reset before headers. reset reason: protocol error.., projectName: alltypeslanguage, trainedModel: alltypes100

Training progressProgress: 100% completeTime elapsed: 1 hours, 21 minutes, 46 seconds

Evaluation progressProgress: 95% completeTime elapsed: 0 hours, 19 minutes, 55 seconds

Output modelalltypes100

I saw this error once, but those file exists and can be opened, I don't see any issues :

1. Evaluation Job Failed, failure msg: Failed to predict documents for the following reason: Failed to read documents in the following locations: tra/832002277400.txt,tra/831003445645.txt,tra/832002284866.txt,tra/831004631374.txt,tra/832002498238.txt,tra/832002590405.txt,tra/832002677260.txt,tra/832002578446.txt,tra/832004710463.txt,tra/832005972829.txt,tra/832000697026.txt,tra/832002306569.txt.., projectName: alltypeslanguage, trainedModel: alltypes0116new

Training progressProgress: 100% completeTime elapsed: 1 hours, 31 minutes, 51 seconds

Evaluation progressProgress: 0% completeTime elapsed: 0 hours, 42 minutes, 49 seconds

Output modelalltypes0116new

Answer 1

Hello @chris.c.zhao chris.c.zhao !

Welcome to Microsoft QnA!

It seems that you are facing two seperate issues

1. Bad Gateway , meaning network issues , be careful if you are using a VPN or a Proxy from your end , also check Azure Status Page if there is any problem in the region of your resources
2. Read Documents Error: This is pointing at first glance to permissions or acess rights in general. It can be a also and very probable a tjhrotlling issue . Can you post the tier of yous Service ? Is it a Free tier F0 ? Make sure you are not hitting the limits
3. Also the first issue may affect the second , so if Network is lost for some seconds , files become unreadable

Kinldy check these details , post if possible how you have your resources configured and some logs if the issue persists. Try another tier like S0 and try again

---

I hope this helps!

Kindly mark the answer as Accepted and Upvote in case it helped! Regards