ISA Server Port Scan Alerts

By Jim Harrison
ISA Server Sustained Engineering, Microsoft Corporation

See other Security Tip of the Month columns

Overview

Since the dawn of ISA Server time (2000, if you haven’t been watching), ISA Server administrators have received practical but often confusing notifications of “all port scan” and “port scan” intrusion attempt alerts.

Although the ability to notify administrators when potentially malicious traffic is detected is a useful feature of any firewall, these alerts in particular seem to cause more confusion than do other ISA Server alerts. It’s this confusion that we’ll try to eliminate today.

To keep things simple (and short), we’ll limit our examples to ISA Server 2004. The same general principles apply to ISA Server 2000, but the ISA Server user interface and log review examples differ greatly.

Some Basics

How does ISA Server handle connections?

ISA Server maintains a list of connections so that traffic from either end of the connection can be properly handled. Among other things, the connection list includes such useful information as:

• Source and destination IP addresses

• IP protocol

• Source and destination ports (for TCP/UDP protocols)

As long as the connection remains “alive” and the traffic is considered appropriate according to the connection state and protocol details understood by ISA Server, it will pass traffic between the two hosts. Generally, if no traffic is received from either end of the connection within two minutes, ISA Server closes the connection and forgets it. This behavior can be changed, but it involves some scripting and ISA COM skills and that’s not part of today’s discussion. This behavior is also dependent on protocol-specific settings, such as the Web proxy connection timeout, which can override this default if it’s set to a larger or smaller value.

How does ISA Server detect intrusions?

In general, ISA Server treats any unexpected packet as an intrusion attempt. Exactly how ISA Server responds to this attempt depends on how it categorizes the intrusion.

The short story is that when ISA Server receives a packet for which it has no related connection list data, it will drop (ignore) the packet and enter this fact and all available data into the Firewall log.

What is a port scan?

The basic goal of any port scan is to determine what, if any, network services are listening on a host. The most common form of port scan is a targeted scan, in which the scanner probes common service ports to see if the remote host is providing common network services such as Web services and mail. The variations on this theme are many, but they all share the goal of gathering information from a remote host. What can be accomplished with this information depends on a combination of the scanner’s skills and the network and application protection mechanisms in place for the scanned host. Fortunately for ISA Server administrators, the latter is frequently far greater than the former.

ISA Server Port Scan Intrusion Alerts

ISA Server divides port scans into two categories: a well-known port scan and an all port scan. Well-known port scan alerts specifically reference improper connection attempts to ports from 1 through 1023. All port scan alerts include the remaining ports from 1024 through 65535.

How ISA Server determines that a port scan is being attempted is partly controlled by settings configured in ISA Server Management. To view the relevant Intrusion Detection dialog box, in the console tree of ISA Server Management, click to expand the Configuration node, and then click General. In the details pane, click Enable Intrusion Detection and DNS Attack Detection.

Specifically, the following settings affect how long ISA Server waits before triggering port scan attack alerts:

• Port Scan. Specifies that ISA Server will generate an event if an attempt is made to count the services running on a computer by probing each port for a response.

• Detect after attacks on number_of_ports well-known ports. The maximum number of well-known ports that can be scanned before generating an event when a port scan attack is detected. A well-known port is any port in the range from 1 through 1023.

• Detect after attacks on number_of_ports ports. The maximum number of ports that can be scanned before generating an event when a port scan attack is detected.

For instructions on configuring this dialog box, see the ISA Server Help topic “Enable intrusion detection of common attacks.”

What this boils down to is that after n unexpected packets, ISA Server will signal a port scan alert, using the criteria specified in the Intrusion Detection dialog box to determine whether to log an all port scan alert or a well-known port scan alert.

Note: this aspect of ISA scan alerting behavior is too involved to delve into here, but the core point of this is that ISA will only trigger a scan alert when the above settings are satisfied.

Known Scenarios for False Alerts

TCP/IP network communication is clearly defined in several dozen RFCs. However, because some client and server Web applications misbehave “on the wire” and some servers are just overworked, it’s common for ISA Server to trigger an erroneous port scan alert for what is best described as “late packets.”

To explain this further, when ISA Server closes a connection, it suffers from self-imposed “connection amnesia.” As far as ISA Server is concerned, the connection does not exist and any host trying to communicate over that connection is considered hostile. Thus, if a packet arrives from either host involved in the unrecognized connection, ISA Server views this as a port scan and triggers an alert in accordance with the configuration settings specified for intrusion detection.

So 99.444 percent of ISA Server “false” alerts are due to packet timing and fall into two main categories:

• “Impatient applications”: The client closed the connection before the response was received from the server or vice versa.

• “Delayed responses”: The server or client failed to respond before ISA Server timed out the connection.

The following two illustrations depict these common scenarios.

In the “Impatient Applications” scenario, the client has connected to the Web server, requested some content, and then immediately closed the connection before the Web server has a chance to respond. If you ask, “When would this happen?” The answer is every time you say, “It’s late; close that browser and get up to your bed!” Even though your spouse will likely exact some form of revenge later, he or she just clicks the Close button in the browser window.

When Internet Explorer is shut down, it sends a “connection reset” packet to any server with which it has a session. ISA Server will send a “close session” packet to the Web server. If the Web server has already sent a content response packet for the now closed Internet Explorer session before it receives the ISA Server “close session” packet, the content response packet is dropped and logged by ISA Server as a port scan attack.

In the “Delayed Packet” scenario, bedtime has yet to pass and traffic is flowing relatively normally between the client and server. The spousal unit has discovered an incredibly amusing piece of content on this site and has encouraged a couple of friends to join in.

In this scenario, the Web site is running on a Commodore 64 with a 1200 BPS modem and is quickly overwhelmed by a second connection, slowing its responses dramatically. When a third friend accesses it, the Web server goes into a momentary “kill me now” loop and fails to respond. During this moment of revered silence, ISA Server decides that the server has gone “beddy-bye” and closes the connection. When the third friend disconnects in frustration, the Web server can then respond and starts pumping out response data at a blistering 1200 BPS. Because ISA Server has now closed the relevant session and is blissfully unaware that this connection ever existed, this response traffic is viewed as a port scan attack.

Another common cause of port scan alerts is late responses from overworked or weak DNS servers. The basic scenario is identical to the “Delayed Packet” Web server scenario, but the IP protocol and ports are different, and this traffic won’t be found in the Web Proxy logs. In either case, the investigatory techniques used are identical.

How Can I Tell Real Alerts from False Ones?

This is the core question that everyone has been ever so patiently waiting to have answered. The good news is that the process is relatively simple, once you get a handle on how this event is logged and what log data is interesting to you. The bad news is that this process involves a bit of sleuthing on your part and requires a bit of basic TCP/IP knowledge.

What is a “connection”?

TCP/IP provides for the concept of a connection between two (or more) hosts, depending on the functionality and limitations provided by the protocols involved. There are literally thousands of books and articles discussing the finer points of TCP/IP networking, but we’re only interested in the basics of TCP and UDP traffic here, since these are the only IP protocols for which ISA Server will trigger a scan alert. Of the two, only TCP actually uses the concept of a connection within the protocol itself. UDP is known as a “connectionless protocol”—a fact that ISA Server gets around through the connection list it maintains.

TCP and UDP protocols both include the concept of “ports” to assist the hosts with managing these connections. Ports are numeric representations applied to the two ends of the connection to provide for distinction between multiple connections to or from that host. For each connection, five data points distinguish one from another:

• Source IP Address

• Destination IP Address

• IP Protocol (TCP or UDP)

• Source Port

• Destination Port

The term “5-tuple” is often used to describe this collection of data points.

In the Web server example, a single Web server Ted is serving content to two different clients, Bob & Carol (Alice’s ISP cut her off for spamming). The traffic from the clients to the server is described as:

Traffic from the server to the clients is described as:

So you can see that the concepts of “source” and “destination” depend on the direction of traffic flow between the hosts. You may also have noticed that we used the same IP protocol (TCP) for this example. We could have configured the Web server to listen on UDP:80, but since no browser would understand that, the connections would fail and our example would be broken.

What do the ISA Server alerts give me?

If you open the ISA Server Management console tree, expand ArrayName, select Monitoring, and then click the Alerts tab, you may see something like this:

This ISA Server has triggered an all port scan alert at 8:07:43 PM on 10/19/2005 from a host operating on IP address 207.115.64.2. While it may seem at first glance that this is pretty skimpy information given what we’ve discussed up to now, it actually gives us all we need to begin our investigation into the mystery of this alert.

How do I find the log entry for that alert?

ISA Server dumps everything it knows about every connection into the logs as soon as it can (almost in real time). All traffic is logged in the Firewall log, while only the traffic handled by the Web proxy is logged in the Web Proxy log. This is an important distinction that you’ll want to bear in mind for your own investigations.

In order to make use of the ISA logging query mechanism, you have to:

1. Configure ISA Server logging to use MSDE logging (for details, see the ISA Server Help topic “Configure logging to an MSDE database”).

2. Create a query filter that fits the event details that you already know.

In the right pane, if you click the Tasks tab and then click Edit Filter, you’ll be presented with a dialog that allows you to create a query filter using any of the ISA Server log fields. The default filter allows you to see ISA Server traffic as soon as ISA Server logs it. Since we’re interested in “life in the past lane,” we’ll want to modify the filter settings as shown in the Edit Filter dialog.

One thing to bear in mind about log query filters is that all of the items you add use only “and logic.” In other words, the results of any query filter will be any log entries that match all of the filter criteria. The filter example here dictates that we only want to see log entries:

1. Log Record Type = Firewall

2. Log Time = between 10/19/2005 8:07:00 PM and 10/19/2005 8:08:00 PM

3. Action = Denied Connection (because we know ISA Server blocked it)

4. Client IP = 207.115.64.2

When we run the query, the results appear in the results pane.

For details on organizing your results pane, see Note 1.

How do I determine if this ISA Server alert is true or false?

What we derive from this query result is that the packet triggering the scan alert is identified by the following 5-tuple:

• Client IP = 207.115.64.2

• Source Port = 53 (see Note 2)

• Transport (AKA IP protocol) = TCP

• Destination IP = 66.114.140.242

• Destination Port = 11002

Now we have the same data ISA Server used to determine if this packet was part of the active connection list. To decide if this particular alert represents a true or false alert, we need to find out if ISA Server ever had tried to make a similar connection with that host. To do this, we need to devise a query filter that can seek out the log entries of interest. First, we’ll apply the connections and traffic flow concepts discussed earlier to produce the following 5-tuple that represents the reverse traffic flow:

• Client IP = 66.114.140.242

• Source Port = 11002

• Transport (AKA IP protocol) = TCP

• Destination IP = 207.115.64.2

• Destination Port = 53

Then we’ll apply some of that data to the query filter.

1. Log Record Type = from the firewall service

2. Log Time = between 10/19/2005 8:07:00 PM and 10/19/2005 8:08:00 PM

3. Action = Not Equal Connection Status (see Note 3)

4. Destination IP = 207.115.64.2

When we run this filter, ISA Server produces the following results:

You’ll notice that we eliminated the source port from this filter. What this gives us is a “trend” picture of ISA Server traffic behavior surrounding this time frame as it applies to this traffic profile.

Logical Conclusion Time

So we’ve learned oodles and gobs about tracking down alerts and log entries and about customizing the log query results pane. How do we use this mass of skills and knowledge?

If we assemble the basic facts as we have them in timeline order:

• 8:07:12 PM:
  ISA Server created a connection (Action = “Initiated”) and sent a TCP packet from 66.114.140.242:11002 to 207.115.64.2:53.

• 8:07:42 PM:
  ISA Server closed the connection (Action = “Closed”) and promptly forgot the connection ever existed.

• 8:07:44 PM:
  The remote host sent a packet from 207.115.64.2:53 to 66.114.140.242:11002. ISA Server logged an all port scan alert because the connection list had no entry relevant to that packet’s 5-tuple.

Based on this information, we can determine that ISA Server logged this alert properly because it had closed the connection and deleted it from the list before the remote host sent the response packet.

For other protocols like HTTP and FTP, we would have to search through the logs for similar traffic denials from that host within the near time frame (+- 2 minutes, usually) we used in the previous query filters. Because your ISA Server log query skills are now so impressive, we’ll leave that as an exercise for the student.

Notes

Note 1 – Customizing your log query results pane display

The columns list in the query results examples will likely differ from your own results. This is because I’ve selected and organized the interesting data to be displayed first. You too can show off your incredible ISA Server Management skills by modifying this pane to suit your own needs and tastes. All you have to do is right-click on any column header in the results pane and then select Add/Remove columns. You’ll be presented with the following dialog:

You can select from all the ISA Server log fields and organize them to suit your immediate needs.

Note 2 – Notes about DNS traffic

Domain Name System (DNS) traffic operates using port 53 on IP protocols (transports) TCP and UDP at the server. Like many other protocols, DNS traffic has its own set of behavior rules. ISA Server includes a DNS attack detection filter that enforces this behavior for DNS traffic being handled by ISA Server. In particular, the default 2-minute connection timeout mentioned before is overridden by a 30-second timeout imposed by the ISA Server DNS intrusion detection filter. This point will be critical to the analysis example.

Note 3 – What does “Not Equal Connection Status” mean?

ISA Server log query filters can match on many points in the log fields. The “Action” field is of particular interest because we often seek out traffic that ISA Server was actively blocking or allowing. If we’re unsure of what action ISA Server may have taken, we can tell the log query mechanism to ignore this field by specifying this setting. We can accomplish the same by deleting the “Action” field from the query filter as well.