Understanding HTTP Flow with Netmon 3 - By Yuri Diogenes
One of the most common protocols that we need to deal with these days is the HTTP Protocol. This is not only a privilege of Internet users, there are a lot of Intranet users that also use this protocol for internal transactions.
This post will show how to use Network Monitor 3 to better understand HTTP traffic and also to help you troubleshooting HTTP traffic.
2. HTTP Components
On HTTP we pretty much have two messages: HTTP Request and HTTP Response. The picture below shows an example of these messages:
Figure 1 – HTTP Messages
Here is a brief explanation of the main components of a message:
- HTTP Version: the http version in use by the message (ex.: HTTP 1.0 or HTTP 1.1).
- Method: this is about the action that the client is requesting to the server (ex.: GET and POST).
- Status Code: this code describes what happened on that transaction (ex.: 200, 301 and 407).
- Reason: complement to the status code (ex.: OK, NOT OK).
- Headers: the content of the header will depend on the version of the HTTP. For instance, HTTP/1.1. has some headers that needs to be present for the method in use (request or response).
- Body: some messages will contain the body, which is the data itself. Some other messages will have a blank line.
Based on this brief explanation about the main components of a message, let’s see how NetMon 3 can help us tracking down a HTTP conversation.
3. Understanding HTTP Messages using Netmon3
On this example the server is trying to access the website www.sysinternals.com. This server (Windows Server 2003) is behind a Proxy (ISA Server 2004) and using Integrated Authentication. All the traffic was captured from this server while was accessing this web site.
To help understand the HTTP conversations add the columns “HTTP is Request” and “HTTP is Response” . Those columns will have a number 1 in the column if this sentence is TRUE. This will help to identify what HTTP message was in use at that time.
Figure 2 – Choosing Columns.
For this example is quiet easy to identify the traffic, however on a real world scenario it might be difficult to locate the packet that has the URL request that you want. You might say, “Well, let’s create a filter for this request.” The thing is, if you create a filter for this request you will see only one packet requesting for this URL and this is not what we want here.
There is one cool feature on Netmon3 that allows you to use a filter to find a packet. To use this feature you need to click on the menu Frames than click in Find (or click Ctrl+F). The following window will appear:
Figure 3 – Find Packet based on a filter.
In this case I want to find a packet that matches with the following criteria:
After typing this and clicking Find the packet that matches with this request is selected as the current frame.
To make even easier to read the trace we can also change the color for the HTTP packets. This will allow you to quickly identify the HTTP traffic. For this example we will set the HTTP Request in Red and the HTTP Response in blue. Follow the steps below to configure that:
- Click in Filter.
- Click in Color Filters.
- Click in Add and type the query specified on the figure below:
Figure 4 – Color Filter feature.
- After type this query click, the color (red) and the Bold style click OK.
- Click in Add again and now type the following query: protocol.HTTP.response
- Choose the color blue and leave it bold
- Click OK again.
Here an example of how it will look after you apply the color filter:
Figure 5 – Frame summary after applies the filter.
Now we can close the Find Dialog window and look at the packet. Here is the HTTP part of the packet:
- Http: Request, GET http://www.sysinternals.com/ - Request: Command: GET - URI: http://www.sysinternals.com/ + Uri: ProtocolVersion: HTTP/1.0 Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, */* Accept-Language: en-us UA-CPU: x86 Proxy-Connection: Keep-Alive UserAgent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; SV1; .NET CLR 1.1.4322) Host: www.sysinternals.com HeaderEnd: CRLF
As you can see, this is a HTTP Request message and some of the components of the messages previously explained appear on this packet. Let’s check the answer for this packet:
- Http: Response, HTTP/1.1, Status Code = 301 - Response: ProtocolVersion: HTTP/1.1 StatusCode: 301, Moved permanently Reason: Moved Permanently Via: 1.1 SRVISA Connection: Keep-Alive Proxy-Connection: Keep-Alive ContentLength: 31 Date: Sun, 26 Aug 2007 15:05:10 GMT Location: http://www.microsoft.com/technet/sysinternals ContentType: text/html Server: Microsoft-IIS/6.0 XPoweredBy: ASP.NET Set-Cookie: ASPSESSIONIDCCRASDTB=OKKIMCCDOMFAEPIPJCLNPEBN; path=/ Cache-control: private HeaderEnd: CRLF + payload: HttpContentType = text/html
Note: it is important to mention that in my lab there were no multiple streams involved. Which make it easier to track it down the answer, since it is the next packet in the sequence.
This HTTP Response message is really important to emphasize one particular point, which is the Status Code.
The status code on this answer is 301. This number itself already says what is going on in this answer. It is important to know at least the meaning of status code based on the number range. The ranges are:
|200 – 299||Success|
|300 – 399||Redirection|
|400 – 499||Error on the client side|
|500 – 599||Error on the server side|
The netmon3 parser for HTTP has the main codes already defined. If you click on the Parser Tab, click on Protocols and HTTP, you will see on the right panel those definitions.
Figure 6 – Netmon3 HTTP Parser.
You also can view this code on the Table object on the Parser tab, as showed below:
Figure 7 – Table View.
Since this is a redirection answer, the field “location” has the place where the page is now located. This is presented to the client (requester) that based on that will send another HTTP Request for this URL.
4. HTTP with Netmon3 Conversation
The conversation feature on netmon3 allows you to view the frames aggregated on the same conversation. For this next example, let’s see the frames aggregated for the HTTP request for the URL www.microsoft.com:
Figure 8 – Filtering by conversation.
Clicking on the conversation tree filters out packets based on the HTTP traffic automatically. This can help to understand the whole conversation that client and server are having during this access. Another way to customize this filter is right clicking on the conversation and chooses the option Copy Conversation Filter to Clipboard” as showed in figure 8. Remember that all filters are applied in combination with the current node that is selected in the Conversation Tree. Be sure to click on the root of the tree if you don’t want the frames to be qualified further by the conversation tree.
Looking at this conversation we can see another status code that means there was an error on the client side:
- Http: Response, HTTP/1.1, Status Code = 407 - Response: ProtocolVersion: HTTP/1.1 StatusCode: 407, Proxy authentication required Reason: Proxy Authentication Required ( The ISA Server requires authorization to fulfill the request. Access to the Web Proxy filter is denied. ) Via: 1.1 SRVISA - ProxyAuthenticate: Negotiate WhiteSpace: AuthenticateData: Negotiate - ProxyAuthenticate: Kerberos WhiteSpace: AuthenticateData: Kerberos - ProxyAuthenticate: NTLM WhiteSpace: AuthenticateData: NTLM Connection: Keep-Alive Proxy-Connection: Keep-Alive Pragma: no-cache Cache-Control: no-cache ContentType: text/html ContentLength: 4106 HeaderEnd: CRLF + payload: HttpContentType = text/html
The reason why this request was considered an error on a client side is because the ISA requires authentication and the Internet Explorer on the first attempt to access the web site did not send the user credentials. After the response from the server, and depending on the browser and on the configuration, the client will use either NTLM or Kerberos to send another packet with the credentials.
5. General Information
There are many commands that you can use to obtain more information about your HTTP traffic using netmon 3. Let’s see some of them:
|contains(Http.Response.StatusCode,"301")||Show all HTTP packets where the Status Code is 301|
|Property.HttpIsRequest||Show all HTTP Request packets|
|Property.HttpPragma||Show all HTTP messages that cannot be cached. More information about the Pragma Field, see the HTTP Field definition.|