System.Net: Intial auto proxy delay resulting in requests failing with "the operation has timed out" exception

Recently I came across an interesting issue.

A windows service application (originally developed with VS2003/.NET Framework 1.1) is designed to make several HTTP(s) requests in multiple threads simultaneously. With upgrading to VS2008 and recompiling the code, the app started seeing several requests failing with "The operation has timed out" exceptions.

Taking a System.Net trace (http://msdn2.microsoft.com/en-us/library/ty48b824(VS.80).aspx) showed several threads failing as follows.

System.Net Verbose: 0 : [6044] WebRequest::Create(http://www.someserver.com/)

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.3396033Z

System.Net Verbose: 0 : [6044] HttpWebRequest#38203409::HttpWebRequest(http://www.someserver.com/)

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.3396033Z

System.Net Verbose: 0 : [6044] Exiting HttpWebRequest#38203409::HttpWebRequest()

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.3396033Z

System.Net Verbose: 0 : [6044] Exiting WebRequest::Create() -> HttpWebRequest#38203409

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.3396033Z

System.Net Verbose: 0 : [6044] HttpWebRequest#38203409::GetResponse()

    ProcessId=9616

    DateTime=2008-02-08T21:45:52.4021029Z

System.Net Information: 0 : [6044] Associating HttpWebRequest#38203409 with ServicePoint#58391824

    ProcessId=9616

    DateTime=2008-02-08T21:46:12.4957243Z

System.Net Error: 0 : [6044] Exception in the HttpWebRequest#38203409::EndGetResponse - The operation has timed out

    ProcessId=9616

    DateTime=2008-02-08T21:46:12.4957243Z

From the above we can see that the request failed almost immediately with the exception. Taking a network trace I confirmed that the specific request above was not sent over the wire.

Checking the trace at the beginning I could see that the application started making numerous requests. However before these requests were sent on wire, .NET Framework was trying to detect auto proxy for the urls submitted. This took about 5-7 seconds, by which time there were numerous requests in queue. The timeout set for each request was 20 seconds, however most of them started failing as shown above.

Since auto proxy detection was added only from 2.0 and above, the application did not see the same problem with 1.1 .NET Framework.

In this case, the following were the possible solutions.

1. Disabling auto proxy and configure static proxy in the app.config for the application.
2. Change code to wait for the  the first request to complete (including auto proxy). Then start sending the requests from the application.

In our case, the issue was resolved by disabling auto proxy and setting static proxy in app.config. More information on doing this is available at http://msdn2.microsoft.com/en-us/library/sa91de1e(VS.80).aspx