New information has been added to this article since publication.
Refer to the Editor's Update below.

Proxy Detection

Take the Burden Off Users with Automatic Configuration in .NET

Durgaprasad Gorti

This article is based on a prerelease version of the .NET Framework 2.0. All information herein is subject to change

This article discusses:

  • Proxy setting configuration and policy
  • Using automatic proxy configuration
  • New features in the .NET Framework 2.0
This article uses the following technologies:
.NET Framework, C#

Code download available at:AutomaticProxyDetection.exe(113 KB)

Contents

Your Proxy Settings
Proxy Support in the .NET Framework 1.1
Proxy Support in the .NET Framework 2.0
Proxy Support for ASP.NET Applications
Conclusion

Internet Proxy servers are deployed in most environments for security, performance, and usage tracking purposes. When your application makes a request to a resource on the Internet, the request is often routed through a proxy. The proxy server then makes the request on your behalf and forwards the response back to you. This means that your application needs to know what proxy server to use when requesting a resource on the Internet. You can either use an explicit proxy server name or take advantage of the automatic proxy detection capabilities.

In this article, I will discuss how automatic proxy detection works and how it is supported in the Microsoft® .NET Framework 2.0. I will also discuss how the .NET Framework 2.0 supports connection-specific proxy settings and how to configure your ASP.NET applications to take advantage of the automatic proxy detection found there. I'll show that proxy support in the .NET Framework 2.0 is substantially improved and, in most cases, proxy configuration becomes much easier than ever.

Your Proxy Settings

The .NET Framework gets the proxy settings from the same location in the registry that Microsoft Internet Explorer uses to store proxy settings. Proxy settings are usually per user and thus are stored under the user-specific registry hive (HKCU). If you look under the registry key HKEY_CURRENT_USER\Software\Microsoft\Windows\ CurrentVersion\Internet Settings\Connections, you will find one or more values of type REG_BINARY. The value name indicates the connection name and the binary value contains the proxy settings to be used for that connection. If you take a look at Figure 1 you'll see the relevant portion of the registry on my computer. This information is all editable by a user through the dialog exposed in Internet Explorer from Tools | Internet Options | Connections.

Figure 1 Proxy Settings Registry Key

Figure 1** Proxy Settings Registry Key **

Although proxy settings are usually set per user, you can set them per machine by setting a policy. (This would, of course, make the settings the same for all users on that machine.) You can find all the details for this policy at HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\ CurrentVersion\Internet Settings\ProxySettingsPerUser. The type of the value is DWORD. By setting the value to 0 you can then force the proxy settings to be applied per machine. In this case, the proxy settings come from HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\ CurrentVersion\Internet Settings\Connections, just like its HKCU counterpart.

Automatic configuration is a two-step process. First, the user agent (Internet Explorer or the .NET Framework) tries to discover the URL of a configuration script. Typically, the configuration script is a JavaScript file that contains logic to determine whether to use a proxy and what proxy to use for a given target resource. The protocol for automatic detection is documented at Web Proxy Auto-Discovery Protocol. The user agent first tries a DHCPINFORM message. The reply to this message contains the URL of the configuration script. If that doesn't work, a DNS lookup for a well-known alias, WPAD, is issued. Then the configuration script's location is determined as http://<machine>/wpad.dat where the machine name is the result of the DNS lookup. You can try this in your own network. If you open Internet Explorer and go to http://wpad/wpad.dat, you might see the configuration script. If you don't see this, it might be because the network administrator uses custom configuration files with the Internet Explorer Administration Kit (IEAK) or some other nonstandard configuration.

The second step is to download the configuration script from the URL and run the script. The specification of what the configuration script looks like is at Navigator Proxy Auto-Config File Format. Basically, the script has one entry point function: FindProxyForURL. You pass the URL into the function as well as the host to which you would like to connect, and the function returns one or more proxy servers to use. The user agent should try the proxy servers in the order returned. Note that the user agent can get different script content each time the script is downloaded. This might happen because the network administrator has updated the script to reflect the new proxy servers she has added to the pool of proxy servers. The script can contain logic to determine what proxy to use based on the day, time, subnet of the client machine, and the target URL. For example, the following script returns DIRECT (bypass proxy) for the 198.95.*.* subnet, proxy1 for connecting to www.example.com, and proxy2 and proxy3 for connecting to any other resource:

FindProxyForURL(url, host) { if (isInNet(host, "198.95.0.0", "255.255.0.0")) { return "DIRECT"; } if(host == "www.example.com") { return "PROXY proxy1;DIRECT"; } return "PROXY proxy2:8080;PROXY proxy3:8081" }

By selecting automatic configuration, user agents detect the configuration script URL, download and run the script to automatically determine what proxy to use. If you already know the configuration script URL and don't want auto-detection, in the LAN Settings dialog you can enter the URL and check "Automatic Configuration Script." If you check both the "Automatically Detect Settings" and "Automatic Configuration Script" boxes, auto-detection takes precedence. You can see that a lot of stuff happens behind the scenes to make automatic configuration work. The benefit is that no manual proxy configuration of clients is required.

Proxy Support in the .NET Framework 1.1

The .NET Framework 1.1 shipped with basic proxy support through the System.Net.WebProxy and System.Net.GlobalProxySelection classes. Configuration support is provided through the <defaultProxy> section under <system.net> node in the configuration file. When you install the .NET Framework 1.1, the Machine.config file has the following configuration:

<system.net> <defaultProxy> <proxy usesystemdefault="true" /> </defaultProxy> </system.net>

This means that when you access the GlobalProxySelection.Select property, the proxy instance returned will contain the settings from the registry. As I mentioned earlier, the registry location is the same location as the one Internet Explorer uses and the settings can come from either HKCU or HKLM (for the local machine) depending on the policy.

One of the issues in the .NET Framework 1.1 proxy support is that when the registry is read, the automatic configuration part of the proxy settings is ignored. Only the static portion of the proxy settings are read. However, if the default proxy settings in Internet Explorer are configured for automatic detection and static settings are not populated, the result is that Internet Explorer seems to download the resources correctly, while applications using the .NET Framework 1.1 get an exception that reads, "The underlying connection was closed: The remote name could not be resolved." What is really happening is that the GlobalProxySelection returns an empty proxy due to missing information in the static portion of the proxy settings. The WebRequest class then tries to connect to the resource directly (bypassing any proxy) to the target resource. Since external names can't be resolved in a protected network, you get a "name not resolved" exception thrown back at you.

A lot of users observed this issue indirectly through the Add Web Reference dialog in Visual Studio® .NET 2003 as shown in Figure 2. On the left side is an Internet Explorer browser control that can automatically detect proxy settings and is able to display the Web methods properly. On the right side, the .NET Framework 1.1 code uses only the static settings and isn't able to retrieve the Web Services Description Language (WSDL) for the Web Service, so it fails to list the Web methods. In order to fix this issue, you have three options. The first is to create a WebProxy instance with a specific proxy name and assign it to the WebRequest.Proxy property. The second option is to enter the static settings in Internet Explorer, so the .NET Framework will read those settings. The third option is to use the configuration (either Machine.config or App.config) file to specify a proxy.

Figure 2 Add Web Reference Dialog Doesn't Show Web Methods

Figure 2** Add Web Reference Dialog Doesn't Show Web Methods **

The other issue with .NET Framework 1.1 proxy support is that it always read the default LAN connection proxy settings and ignored the dial-up and Virtual Private Network (VPN) connection settings. In other words, only the DefaultConnectionSettings value is read irrespective of the current active connection. Imagine a scenario in which you are using a remote access connection to your corporate network from home. When you are connected to the corporate network, your requests should be routed through the proxy in your corporate network. But since the .NET Framework 1.1 always uses the default LAN connection settings, proxy settings for the VPN connection are not picked up and the app will fail to connect to the target resource.

Long-running applications have another issue with .NET Framework 1.1. Imagine that the static proxy you entered in Internet Explorer is down and your friendly helpdesk person gave you another proxy name to try. After you enter the new proxy name, you would expect your managed app to pick up the new proxy settings. But it doesn't. Once the GlobalProxySelection class reads the proxy settings, they are cached per app domain, and they are not refreshed. The only solution is to reload the app domain, which in most applications means you need to restart the app.

Many users wonder why their code that calls a Web service works fine on a console but doesn't work when they drop it in an .aspx page. The answer is that the ASP.NET account runs under a different user account than the interactive user. Since there is no easy way to edit the proxy settings for the ASP.NET user in the registry, you need to enter the proxy settings explicitly in the Web.config file to make it work properly.

Proxy Support in the .NET Framework 2.0

Fear not, however—the .NET Framework 2.0 significantly enhances proxy support. All of the issues I've mentioned with the .NET Framework 1.1 version are resolved in the new version. In most of the cases, you don't need to think about the proxy configuration any more, which is always a good thing. Let's drill down a little further, though.

Automatic proxy configuration is now fully supported. The .NET Framework 2.0 reads the complete settings from the registry, including the automatic configuration flags and any explicitly provided script location. Automatic detection happens by first using the DHCPINFORM message and then using the DNS resolution techniques already described. Once the script location is identified, the script is downloaded, compiled into an in-memory assembly, and then executed in a sandbox app domain where minimal permissions are granted.

In fact, no code access security (CAS) permissions are granted to the script other than the execution permission. Minimal permissions mean that rogue scripts can't do anything significant other than call the few functions exposed to the script. For a detailed sample of how to host and run your own scripts in a sandbox app domain, see the sidebar, "Hosting the Proxy Script."

Failover is also built in. If auto-detection fails and if you have entered an explicit script location, then that script is downloaded and run. If both fail and if you have a static proxy, then the static proxy is used as a fallback. If all fail, a direct connection to the target URL is attempted.Hosting the Proxy Script

The Microsoft Active Scripting architecture provided a very successful extensibility platform for applications. An application could expose an object model and host a script engine, which would enable end users to write scripts to customize and extend the application functionality. The Active Scripting architecture is COM-based. For the .NET Framework, script hosting is provided through Visual Studio for Applications (VSA). VSA provides a set of classes and interfaces to host managed scripting languages like JScript® .NET and Visual Basic® .NET.

The .NET Framework 2.0 uses VSA and the JScript .NET engine to host proxy scripts. Since this is a managed framework, scripts are compiled into an in-memory assembly. A script is executed every time an application tries to connect to a resource and needs to determine what proxy to use. The downloadable code that accompanies this article contains a sample of how the proxy script is hosted and run, but I will walk you through the relevant portions of the code now:

m_JScriptEngine = new Microsoft.JScript.Vsa.VsaEngine(); m_JScriptEngine.RootMoniker = "com.example.wpad://ScriptSample"; m_JScriptEngine.Site = new VsaEngineSite(); m_JScriptEngine.InitNew(); m_JScriptEngine.RootNamespace = "__ProxyScriptSample";

The first thing to do is to create a scripting engine. Once a script engine is created, set the RootMoniker. The RootMoniker must be in the form <protocol>://<path> where the <protocol> string is guaranteed to be unique to the host, and <path> is the unique sequence of characters recognized by the host (the <protocol> element can't be a registered protocol handler such as http: or ftp:). Once the engine is created, you can set the implementation of the IVsaSite using the Site property. I'll take a look at the IVsaSite implementation shortly:

//Turn off print statements m_JScriptEngine.SetOption("print", false); //Turn off optimizations m_JScriptEngine.SetOption("fast", false); //Turn off auto references m_JScriptEngine.SetOption("autoref", false);

Once the engine is created and initialized, you can set specific options through the SetOption methods. Here you turn off print statements and optimizations. The autoref option is also turned off to prevent JScript .NET from automatically referencing assemblies based on import statements or type annotations. It is important that you do this because the proxy configuration script is not supposed to access any objects or assemblies other than a handful of functions exposed to the script:

IVsaCodeItem codeItem = m_JScriptEngine.Items.CreateItem("SourceText", VsaItemType.Code, VsaItemFlag.None) as IVsaCodeItem; codeItem.SourceText = code.ToString(); //Add the object model m_JScriptEngine.Items.CreateItem("__ScriptObjectModel", VsaItemType.AppGlobal, VsaItemFlag.None);

These lines add the script code to the engine and add a global object to the script. The engine will use the IVsaSite implementation to get a reference to the global object when it is first referenced by the script code. The corresponding portion of the IVsaSite implementation is shown here:

public class VsaEngineSite : IVsaSite { ... public object GetGlobalInstance(string name) { //When the engine asks for __ScriptObjectModel, return an instance //of the ScriptObjectModel if (name == "__ScriptObjectModel") { return new ScriptObjectModel(); } throw new VsaException(VsaError.GlobalInstanceInvalid); } ... }

The ScriptObjectModel is shown here:

public class ScriptObjectModel { public bool isPlainHostName(string hostName) { if (hostName == null) { throw new ArgumentNullException("hostName"); } return hostName.IndexOf('.') == -1; } }

I implemented only one of the functions exposed to the script, although the others are similarly easy to implement. The function isPlainHostName simply returns true if there are no "." characters, or false otherwise. This function can be used to test if the target is an intranet host or not.

In order to ensure that you truly run the script in a sandbox environment, you should run the script in its own app domain with a restricted set of permissions. In fact, the script needs no permission other than execution. Figure A shows the code that sets the policy for the AppDomain. This policy grants only execution permission to any assembly that includes SandboxEvidence and full trust to all other assemblies. The SandboxEvidence is nothing more than an empty class that serves the purpose of tagging the assembly produced by compiling the script:

private class SandboxEvidence { }

The following lines of code show how to include the SandboxEvidence to the assembly produced by compiling the script:

Evidence engineEvidence = new Evidence(); //Required because the appdomain policy is intersected with the other //levels. Assembly also gets execute permissions at all other levels engineEvidence.AddHost(new Zone(SecurityZone.Intranet)); //Add SandboxEvidence to force execute only at the AppDomain level engineEvidence.AddHost(new SandboxEvidence()); m_JScriptEngine.Evidence = engineEvidence;

If you don't include the SandboxEvidence, the assembly will get full trust as per the app domain policy, so it is important to add this evidence.

With the tightened security policy on the app domain and hosting the script in that app domain, you can make sure that the script runs in a least-privileged environment. Download the code for this article and see how this is all put together. Then save the following sample proxy script to a file and run the code to see the script host in action:

function FindProxyForURL(url, host) { if(isPlainHostName(host)) { return "DIRECT"; } else { return "PROXY LOCALHOST:8080"; } }

[Editor's Update - 8/19/2005: DefaultWebProxy is a new property on the WebRequest class, not on the WebProxy class.]

This property provides functionality that is similar to the GlobalProxySelection.Select property in version 1.1. If you don't explicitly assign any proxy on your Web request, the proxy obtained from the WebProxy.DefaultWebProxy is used. The GlobalProxySelection class is deprecated, as the WebProxy.DefaultWebProxy is more intuitive and discoverable than the GlobalProxySelection.Select. In the .NET Framework 2.0, GlobalProxySelection.Select is rewired to return the same result as WebRequest.DefaultWebProxy property. This means that in the .NET Framework 2.0, GlobalProxySelection.Select can return a proxy that supports automatic configuration. If you run applications from version 1.1 on version 2.0, by default you get the automatic proxy configuration support without having to change your code.

Let's look at an example. Save the following JScript code as wpad.js under the wwwroot directory:

function FindProxyForURL(url, host) { if(host == "www.example.com") return "PROXY MyProxy:9090"; else return "DIRECT"; }

Make sure that you can download this file from the URL http://localhost/wpad.js. This is your proxy configuration script. Now configure the proxy settings. For this example, enter http://localhost/wpad.js into the address textbox and select Use Automatic Configuration Script. Also, make sure that you uncheck the Automatically Detect Settings box in your browser's settings. If you keep it checked, auto-detection takes precedence and your sample configuration script will not be used. Then you need to open the Internet Options window and select use automatic configuration script. In addition, select manual proxy settings and enter "MyProxy" for proxy address and 80 for port.

Now save the following C# code as GetProxy.cs and compile and run it on the .NET Framework in both versions 1.1 and 2.0:

using System; using System.Net; public class Test { public static void Main(string[] args) { IWebProxy iwp11 = GlobalProxySelection.Select; Console.WriteLine(iwp11.GetProxy(new Uri(args[0])).ToString()); //comment the following 2 lines when compiling on 1.1 IWebProxy iwp20 = WebRequest.DefaultWebProxy; Console.WriteLine(iwp20.GetProxy(new Uri(args[0])).ToString()); } }

It helps if you have a side-by-side installation of both versions so that you don't need to switch machines. Also, make sure that you comment out the lines that use DefaultWebProxy when you are compiling against the .NET Framework 1.1.

Figure 3 Using Automatic Configuration

Figure 3** Using Automatic Configuration **

When you run the previous code on the .NET Framework 2.0 using the settings shown in Figure 3

GetProxy.exe http://www.example.com

prints the following:

http://myproxy:9090 http://myproxy:9090

This is because the proxy script returns http://myproxy:9090 for the host www.example.com. Likewise

GetProxy.exe http://www.microsoft.com

would print the following:

http://www.microsoft.com/ http://www.microsoft.com/

In this case, the proxy script is returning "DIRECT," which is another way of saying "bypass proxy." The .NET Framework 1.1 returns the same URL for the proxy as the target resource when bypassing proxy, so this behavior is continued in the .NET Framework 2.0 for backward compatibility.

When you run this code on the .NET Framework 1.1 (after commenting out the lines specific to version 2.0) the program returns http://myproxy for all hosts, indicating that the automatic configuration is not used in the .NET Framework 1.1.

The .NET Framework 2.0 supports connection-specific proxy settings. Whenever proxy settings are read, the currently active connection on that machine is detected and proxy settings for that specific connection are read. For example, when you're using a remote access connection to the corporate network, the proxy settings specific to that remote access connection are used. When you disconnect the remote access connection upon completion, the proxy settings for the default LAN connection are used. In addition, a very cool feature in the .NET Framework 2.0 is that you don't need to restart the app to get the proxy settings refreshed when you establish a new connection or disconnect. The .NET Framework automatically detects remote access connect-and-disconnect events and refreshes the internal proxy settings, making your applications more robust without you having to write additional code.

For your long-running applications, I have good news too. Whenever the proxy settings for a connection change, you automatically refresh the settings internally by detecting a change in the registry values. What this means is that if you update the proxy settings in Internet Explorer, the new settings are picked up automatically. Once again, you don't need to restart the application.

Proxy Support for ASP.NET Applications

If you have written any code in ASP.NET to connect to an external resource or invoke an external Web service, you've probably seen an application that works perfectly outside of ASP.NET yet fails inside ASP.NET. You might check the code again and again and get the same poor result before you realize that the ASP.NET code is running under a different user than you, the interactive user. For Windows® XP, by default, the ASP.NET worker process runs under the user ASPNET, and on Windows Server™ 2003, by default, the Network Service account is used. When you try to call an external Web service from your .aspx pages, which proxy is used? You can't just open Internet Explorer and look at the proxy settings since that displays the settings stored under your HKCU node, not the information relevant to the ASPNET or Network Service account. If you're curious, you can open the following key to look at the Network Service proxy settings:

HKEY_USERS\S-1-5-20\Software\Microsoft\Windows\CurrentVersion\ Internet Settings\Connections

For ASP.NET users, there is no well-known security identifier (SID). You can run the following code (requiring the .NET Framework 2.0) to determine the SID for the ASPNET account:

using System; using System.Security.Principal; class Test { static void Main(string[] args) { NTAccount account = new NTAccount(args[0]); Console.WriteLine(account); SecurityIdentifier sid = (SecurityIdentifier) account.Translate(typeof(SecurityIdentifier)); Console.WriteLine(sid); } }

On my machine, if I call this with the argument, %COMPUTERNAME%\aspnet, it prints:

S-1-5-21-1957994488-1085031214-725345543-1003

Once I know the SID, I can look up the proxy settings under the path, HKEY_USERS\<SID>\ Software\Microsoft\Windows\CurrentVersion\Internet Settings\Connections. Even if you can peek at the proxy settings for these users, you can't edit them easily because they're binary values.

One simple solution is to set the policy on the machine to make the proxy settings per machine instead of per user. In this case, any user who opens the Internet Options dialog will see and edit the machine-wide proxy settings (provided they have permissions to read and write to the HKLM). Your ASP.NET application will pick up the machine-wide proxy settings and things should work consistently. This simple solution may not work if you must support user-specific proxy settings or when you don't have control over where your application eventually gets deployed.

A better solution may be to use the Web.config file to specify the proxy settings for your ASP.NET applications. Using the <system.net><defaultProxy> section, you can specify a proxy server to be used. This is supported in the .NET Framework 1.1 and 2.0. But what if you don't want to specify a specific proxy but rather want to take advantage of the automatic detection? In the final release of the .NET Framework 2.0, configuration for support for proxy is enhanced in order to support automatic detection. (Please note that this is not part of the .NET Framework Beta 2 release.)

Of course, none of this touches on impersonation. What if you have an ASP.NET application that impersonates authenticated users? As I mentioned before, if no proxy is specified, the default proxy used is the WebRequest.DefaultWebProxy. This property returns the proxy for the user that the process is running under (assuming a default configuration), specifically ignoring the current thread token. This ensures that the default proxy returns consistent settings irrespective of the current thread token.

What if you want the proxy settings for the currently impersonated user? Let's just say, I hope you don't need this. When a user is impersonated, the profile for that user is not typically loaded. That means the HKCU for that user is not available. In a service-oriented application, you don't want to load profiles for each impersonated user because it can become very expensive very quickly. I strongly encourage you to look at other design options. If you must access the proxy settings of the currently impersonated user you can use the WebProxy.GetSystemWebProxy method. This method internally calls the RegOpenCurrentUser API to load the user-specific registry hive. It does not load the user profile, however, and there is currently no managed API to load user profiles. Instead you'll need to use P/Invoke to load the user profile.

If you successfully load the user profile, the proxy settings for that user are read. If the profile is not loaded, the RegOpenCurrentUser returns a handle to HKEY_USERS\.Default hive.

Conclusion

Automatic proxy configuration makes applications more robust by easing the burden put on end users to make sure their Internet settings are correct. The .NET Framework 2.0 fully supports automatic proxy configuration. In addition, by supporting connection-specific proxy settings and automatically refreshing proxy settings whenever the active connection changes, the .NET Framework 2.0 offers a solution to all the major proxy issues that many developers encountered with .NET Framework 1.1.

Durgaprasad Gorti is a test lead at Microsoft for the managed network programming APIs. He can be reached at dgorti@microsoft.com.