[Errno 10054] An existing connection was forcibly closed by the remote host while requesting images from URL

Question

question

NamLy-3299 asked · Mar 07 at 01:25 PM

[Errno 10054] An existing connection was forcibly closed by the remote host while requesting images from URL

Hi,
I tried to download the tagged images using Custom Vision API and in Python. I set the limit to 256 images. However, after 60-90 images, this error would show up:
[Errno 10054] An existing connection was forcibly closed by the remote host

Can I receive further help?

This is my code. I have edited out the project id and training key as I think this is private information.

https://pastebin.com/yVfG85hM

Update:
I added conn.close after data = response.read().decode(encoding). It is the later processing that is the issue.

I did a google search regarding urlretrieve() and [Errno 10054] An existing connection was forcibly closed by the remote host. There is a solution that add edthe proxy to urllib.request. This works most of the time. When it get [Errno 10054], the images downloaded are around 100 images, which improved compared to without proxy. I do not know why this is the case though. After adding the proxy, I get another type of error urlopen error [WinError 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Resources:
https://stackoverflow.com/questions/27732075/python-3-windows-7-connectionreseterror-winerror-10054-an-existing-connect/27732243
https://stackoverflow.com/questions/22967084/urllib-request-urlretrieve-with-proxy

azure-custom-vision

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

Answer 1 · 2021-03-08T08:10:38.787Z

romungi-MSFT answered · Mar 08 at 08:10 AM

@NamLy-3299 The above code seems to be keeping the connection open until the processing is done for all the images on the client side. Could you try to close the connection after the response from the API i.e after data = response.read().decode(encoding)?

I think since the connection is open until all your images or items are processed on client side in the for loop the API is closing the connection after the default limit.
The initial response object should contain all the image details from the API, could you consider to print this response and check if the later processing is the issue?

Increasing the time out on your http.client call is an option but that can still keep the connection open for a long time until all images in the response object are processed.

· 2 ·

10 |1000 characters needed characters left characters exceeded

Attachments: Up to 10 attachments (including images) can be used with a maximum of 3.0 MiB each and 30.0 MiB total.

NamLy-3299 · Mar 09 at 05:20 AM

I added conn.close after data = response.read().decode(encoding). It is the later processing that is the issue.

I did a google search regarding urlretrieve() and [Errno 10054] An existing connection was forcibly closed by the remote host. There is a solution that add edthe proxy to urllib.request. That seems to work for now. I do not know why this is the case though.

0 ·

romungi-MSFT NamLy-3299 · Mar 09 at 10:02 AM

Thanks for the updates. The solution for proxy in the SO threads seem to be around setting up their proxy server because the corp network used by user needed one. If you are using a proxy network then you can also set one instead of localhost.

For the new error though, i think it would be easier to debug by printing the URL being called to check if the cleanup of URL retrieve is working as expected or you could try using urllib.request.urlcleanup() to ensure there is no stale data between calls for different images.

1 ·

question