Volume 4, Number 2

Copyright (C) 2002 Mark Russinovich

August 12 - In this issue:


- TCPView v2.21
- LiveKd v2.0
- PsInfo v1.33|
- PsList v1.21
- BgInfo v3.0
- PageDefrag v2.2
- Regmon v4.34 for Win64/Itanium
- Sysinternals in PC World Magazine
- Sysinternals at Microsoft

- Mark and David Solomon Teach Internals in London
- Inside Windows NT Development
- When a Change Doesn't Count
- .NET Server RC1
- Loads of WMI Scripts
- Using BootVis to Profile the Boot Process

A Quick Introduction to Crash Dump Analysis

The Sysinternals Newsletter is sponsored by Winternals Software, on the
Web at http://www.winternals.com. Winternals Software is the leading
developer and provider of advanced systems tools for Windows NT/2K/XP.
Winternals Software products include FAT32 for Windows NT 4.0, NTFSDOS
Professional Edition (a read/write NTFS driver for DOS), and Remote
Winternals is proud to announce Defrag Manager version 2.0, the fastest,
most thorough enterprise-defragmenter available. Now you can manage
defragmentation schedules across your entire Windows enterprise from a
simple MMC snapin - without even having to install any client software
on your NT or Windows 2000 systems. Visit http://www.winternals.com/es
for more information or to request a free 30-day trial version.

Hello everyone,

Welcome to the Sysinternals newsletter. The newsletter currently has
35,000 subscribers.

You have probably heard of AMD's upcoming Hammer processor (now formally
announced as "Opteron"), to be released late this year. With Hammer, AMD
is taking its war with Intel from the 32-bit field to the 64-bit field.
I think Intel should be scared, and I'll tell you why.

The Itanium is Intel's entry into the 64-bit space and includes x86
binary compatibility as a way to ease migration. The Itanium has been in
the marketplace for almost two years, but has failed to make inroads,
even with HP-UX, Linux, Windows XP 64-bit Edition, and Windows Advanced
Server 64-bit Edition supporting it. Industry estimates project Itaniums
as having only sold a few thousand units and my guess is that the
recently released Itanium II won't improve the rate of deployment.

What are the reasons for Itanium's depressing showing thus far? I
believe that the main factor is that Itanium and Intel's entire 64-bit
strategy is based on new RISC-type architecture, and with the exception
of x86 binary compatibility bolted on to it, Itanium is just another
64-bit processor among Alpha, PA-RISC, Sparc64, and PowerPC. And
although Itanium supports existing x86 32-bit applications, those
applications suffer significant performance degradation when compared to
native 64-bit applications. This means that you need to make sure that
any applications that you want to run with reasonable performance on
Itanium are ported to 64 bits.

Hammer, on the other hand, is an evolution that naturally extends the
x86 architecture and instruction set to 64 bits. It's capable of running
32-bit applications and 64-bit applications side-by-side at virtually
the same performance levels. In addition, you can boot Hammer into full
32-bit mode, making it appear to be a standard x86 system capable of
running a 32-bit OS. Not only that, but Hammer supports x86 16-bit modes
and can use standard 32-bit x86 BIOS code even when you boot into a
64-bit OS. This means that you can have a dual boot system with a 32-bit
OS as one selection and a 64-bit OS as another.

I believe that AMD's approach offers a huge advantage over the direction
that Intel took with Itanium. AMD can begin selling Hammer as soon as
it's available to customers that might not be quite ready to go to 64
bits, or that have only one application that can take advantage of a
64-bit machine. 32-bit applications can take full advantage of the
performance innovations that AMD puts into Hammer, and if AMD prices
Hammer close to 32-bit CPU prices the purchase decision could be a
no-brainer. This should allow AMD to infiltrate the marketplace with its
64-bit processors from the low end. Contrast this to Intel's high-end
approach- a strategy that's likely to be far riskier and take much
longer. I believe that Hammer gives AMD an opportunity to sneak in and
become the de facto 64-bit platform.

What about the performance edge that Intel might claim the Itanium's
"clean" Explicitly Parallel Instruction Computing (EPIC) architecture
provides over AMD's x86-64? We'll have to wait and see, but AMD has thus
far given Intel a run for its money in the 32-bit arena. And, if you
look at how Itanium II compares to the best x86 processors (see
www.spec.org) you'll find them pretty evenly matched on integer
performance, with Itanium having a 50-75% lead in floating point. This
is despite that fact that Itanium is roughly twice as expensive. For
example, the low-end Itanium II system from Hewlett Packard costs $5865
and a Hewlett Packard Pentium IV system running at 2GHz in an equivalent
configuration costs just $2385.

Where did AMD get the idea to extend the x86 architecture rather than
define a new one with x86 compatibility as a feature? When Intel moved
into the 32-bit world, it did so by extending the 16-bit x86
architecture and instruction set to make 32-bit version that could also
run 16-bit applications. As a result, 32-bit x86 processors can run
16-bit and 32-bit operating systems. However, Intel didn't use this
approach with their 64-bit strategy. This may be because in the mid-90's
it saw the ceiling fast approaching for the x86 in terms of performance
given the design's complexity, and felt a fresh design could go farther
into the future. Today both Intel and AMD have proven this assumption
wrong with their astonishing x86 clock speeds - Intel has demonstrated a
Pentium IV running at 4 GHz. Intel could pay the price for their

Please pass the newsletter to friends you think might be interested in
its content.



========== WHAT'S NEW AT SYSINTERNALS ==========
TCPView is an application that's been on the Sysinternals site for
several years, but until the recent 2.0 update it wasn't much more than
a GUI version of the Windows netstat command-line tool. This major
revision brings a slew of powerful features, the most significant of
which is the display on NT 4, Windows 2000 and Windows XP of the process
that owns each TCP and UDP endpoint. You can also view a process'
version data and executable path, as well as terminate rogue processes.
A feature you'll find useful if you want to stop an application from
using a particular port is the ability to close connections on the fly
without terminating the process.

In the UI area the latest version introduces automatic refresh at
selectable intervals and difference highlighting. Difference
highlighting, which helps you to zoom in on activity as its happening,
describes the color highlighting of new and deleted items, something I
carried over from Process Explorer.

Download TCPView v2.21 at

One of the tools David Solomon and I included on the CD for our book
Inside Windows 2000 (see http://www.sysinternals.com/insidew2k.shtml )
is LiveKd. LiveKd is a front end to the Microsoft kernel debuggers that
enables you to run the debuggers to interact with a live system. Using a
debugger to look at a system without having to use a second computer and
a serial debugging connection makes exploring internals easy and can
even serve as a troubleshooting aid. The first release of LiveKd only
supported NT 4 and Windows 2000-LiveKd version 2.0 works on Windows XP.

Although Microsoft followed my lead and added live kernel debugging
capabilities to Windows XP, a few debugging commands don't work when run
on a live system, which is why I went to the trouble of updating LiveKd
to work on Windows XP.

One example is the stack output from the !process or !thread commands.
LiveKd shows the complete kernel stack, whereas using the built-in XP
live debugging capability does not. Another example is the !stacks
command, which you can use to see where threads are currently executing
and where they might have started executing. The !stacks command is
useful when you are debugging a problem that involves system threads and
want determine their purpose - knowing the driver in which a thread is
executing can be the clue you need to fix a problem.

Download the LiveKd v2.0 patch at
Note that you'll need a copy of LiveKd from the book CD to which to
apply the patch.

PsInfo, one of the dozen or so tools that make up the PsTools suite of
command-line, remotely capable, Windows NT/2000/XP administrative tools,
continues to evolve based on user feedback. Version 1.33 adds the
ability to list installed hotfixes and applications, as well as to
report label, size and usage data for a computer's disk volumes.

Download PsInfo v1.33 at
Download the entire PsTools package at

If you like using the command-line or have ever wished you could get a
process list for a computer (local or remote) with CPU usage data then
you'll be interested in the latest version of PsList, another member of
the PsTools suite. PsList v1.2 introduces command-line switches that run
it in a task-manager mode in a command window, updating at a
configurable interval and displaying CPU usage. Another new switch has
PsList use indenting to show process parent-child relationships, which
is sometimes useful in understanding the purpose of processes you're
having trouble identifying.

Download PsList v1.21 at
Download the entire PsTools package at

Managing more than one system usually means having to deal with the
annoyance of using sticky notes or opening system information utilities
to determine a system's specifications and configuration. BgInfo is a
utility that displays standard and custom system information you specify
right on the desktop, giving you at a glance all the data you consider

BgInfo is already versatile, allowing you to specify any of a
pre-defined set of values or direct BgInfo to obtain values from the
Registry or from files. BgInfo v3.0 adds even more customization
capability by introducing support for Visual Basic scripts so that you
can craft scripts that capture custom configuration settings and report
them to BgInfo for presentation.

Download BgInfo v3.0 at

Have you found your paging file, Registry files or event log files
becoming more and more fragmented? The built-in defragmenter doesn't
handle these files because they must be defragmented at boot time before
the operating system is running. PageDefrag is a light-weight
defragmenter with the single purpose of defragmenting these critical
system files.

PageDefrag version 2.2 includes Windows XP support and allows you to
specify a countdown timer that PageDefrag displays before defragmenting.
The timer works like the one chkdsk implements where pressing any key
during the countdown aborts the defrag.

Download PageDefrag v2.2 at

Last year Microsoft lent me an Itanium system so that I could port
Sysinternals tools to Win64. Not all tools require porting since most
32-bit applications run without change on an Itanium running Windows XP
64-bit Edition by using the Itanium's 32-bit binary compatibility
support. However, Windows XP 64-bit Edition will not load 32-bit
drivers, which means that any Sysinternals tool that has a driver
component must at least have the driver ported to 64-bits.

The first utility I ported was Filemon. Filemon's port was relatively
simple because all it required was changing the driver and GUI source to
conform to 64-bit data type rules and then recompiling. I decribed the
porting process in the last newsletter. Regmon's port included the same
data type issues, but included other issues that made it much more
difficult to port.

Regmon relies on a driver that patches the system call table in the
kernel with Regmon interception routines so that it can monitor Registry
activity. To make a long story short, the system call interface on the
Itanium has processor-dependent characteristics that make the approach I
used in the 32-bit driver unusable. I therefore had to use some
difficult coding to get system call interception to work on 64-bit XP.

Fortunately, Microsoft has implemented a Registry activity callback
interface in the Windows .NET Server kernel that makes the technique
Regmon uses to watch Registry activity unnecessary. Look for a Regmon
update in the near future that demonstrates the new Windows .NET Server
API and a description of the API in the newsletter.

Download Regmon v4.34 at

Regmon and Filemon both got nice plugs in the August issue of PC World,
a high-circulation computer magazine, as recommended troubleshooting
tools.

Here's the latest installment of Sysinternals references in Microsoft
Knowledge Base (KB) articles released since the last newsletter. This
brings to 38 the total number of KB references to Sysinternals.

HOW TO: Create Custom Administrative Templates in Windows 2000

Logon Delays Over Slow Links if Oplock Is Not Granted for the Policy

XCON: The MTA Process Is Leaking Memory with a High Handle Count

In addition, Microsoft has honored Sysinternals as the currently
featured Windows XP Embedded community site! Check out

================== INTERNALS INFORMATION =============

Hear me and David Solomon present our 3-day Windows 2000/XP/.NET Server
internals class in London September 23-25 (the next class is December
9-11 in Austin, TX). Based on "Inside Windows 2000, 3rd Edition", it
covers the kernel architecture & interrelationship of key system
components & mechanisms such as system threads, system call dispatching,
interrupt handling, & startup & shutdown. The internals of key
subsystems explained include processes & threads, thread scheduling,
memory management, security, the I/O system, NTFS, and the cache
manager. By understanding the inner workings of the OS, you can take
advantage of the platform more effectively and more effectively debug
and troubleshoot problems.

For more information see http://www.sysinternals.com/seminar.shtml

This presentation is a couple of years old, but is nevertheless an
interesting look inside the development history of Windows NT 3.1
through Windows 2000. Mark Lucovsky, one of the original NT software
engineers, delivered the presentation at the USENIX Annual Technical
Conference in 2000. He describes the size of the development teams,
defect rates (of the code they wrote, not the members), the source
control system, and team development challenges.

See Mark Lucovsky's talk on NT development history at

On a related note, here's Microsoft's own history of Windows, from
Window 1.0 through Windows .NET Server:

Windows keeps track of several timestamps for files and directories.
They include create, last access, and last modified. The last modified
timestamp is documented as recording the last time that a file was
written to. Unfortunately, an application that relies on this being true
will be in for a nasty surprise.

If an application opens a file using the CreateFile API and then writes
to it via WriteFile or its variants, then the last modified time does
reflect the time of the file's most recent modification. If the
application opens the file for memory-mapped I/O using CreateFileMapping
and MapViewOfFile, then modifies the file by changing the mapped
contents, the data gets eventually written back to the file by the
Memory Manager and the on-disk version is modified. The problem is that
file system drivers don't update a file's last modified time when a
modification occurs as a result of a memory-mapped write.

Why not? I assume the developers cut the corner in favor of performance,
but the behavior isn't documented so I can only speculate. I guess all
writes aren't created equal.

Windows .NET Server RC 1 has shipped and there's a chance that Microsoft
will cut it's RTM (Release to Manufacturing) build of the new server OS
by the end of the year (about the same chance as getting bit by a snake
and a shark in the same year). From a kernel perspective there's not
major visible change from Windows XP. There aren't any major new kernel
subsystems, Win32 or kernel APIs - just tweaks that enhance performance
on large systems and a handful of APIs that round out existing API sets.

There are, however, big changes in user-space. The biggest is of course
the .NET framework, but there are also new server-oriented services. The
Volume Shadow Copy service, introduced in Windows XP for backup, is
utilized to provide the ability for client users to rollback files on
server shares to previous versions. No administrator interaction is
required because a user can simply select the desired entry in a file
history list shown in a new Explorer file property tab.

A server networking enhancement is Winsock Direct, an infrastructure
that allows application on a server in a multi-tiered configuration with
specialized network interconnection, a configuration called a System
Area Network (SAN), to talk directly to the networking hardware,
bypassing the TCP/IP stack and the kernel.

Look for an upcoming article by David Solomon and I that describes the
.NET Server system-level changes. We are also covering .NET Server in
the next revision of Inside Windows 2000, to be called Windows
Internals, due for publication next spring.

If you like the PsTools suite from Sysinternals, then you are probably a
scripting junky, and if your systems are running Windows 2000 or XP, or
have the NT 4 WMI core components installed, then you should head to
Microsoft's TechNet site and grab the free WMI sample scripts they've
posted there. There's a large collection and they are divided into
categories like Computer Management and Networking and you can download
them individually or all at once.

Get free sample WMI scripts at

Speaking of freebies from Microsoft, you can find a subset of the
Windows 2000 Resource Kit tools posted for download. Visit

I periodically get requests to develop a tool for Sysinternals that
monitors the boot process so that a user can determine what causes
delays they might experience. Fortunately, I waited long enough that
Microsoft beat me to it. The tool they wrote is called BootVis and it
served an important role in the boot performance optimizations they
implemented for Windows XP. You can download it from
. BootVis, which only runs on Windows XP, uses XP's WMI kernel tracing
facility to collect a detailed trace of the boot process, including CPU
usage, disk I/Os, process startup and driver I/Os. Calling the help
file that comes with BootVis terse is being kind, so I'll walk you
through how to use the tool.

If you want to be able to see precisely what functions with drivers and
modules account for CPU time then open the Tools|Options dialog and
point BootVis at your installed Windows XP symbols. Next, have BootVis
capture a trace of the next boot, including driver delays, by selecting
Trace|Next Boot+Driver Delays. BootVis will countdown and then reboot
your system.

After the system has booted log in and run BootVis again. Select
Trace|Stop Trace and it will extract the trace that's being collected
from the kernel and save it to a file, which it then loads for display.
You'll see four charts. In Cpu Usage you see a track of how much CPU
processing was consumed through the boot. The Disk I/O bar chart shows
the number of disk reads (red) and writes (blue) that occurred during
each second. In the Driver Delay graph you'll see rectangles that
represent driver I/Os that had durations larger than the default BootVis
cutoff of 100 milliseconds - such delays are considered significant.
Finally, the Process Creates chart shows labeled rectangles where
processes started.

BootVis doesn't necessarily show all the trace data collected when you
start it, so use the zoom-out button until you start to see the right
edge of graphed data. Now take a look at each of the graphs. If you see
a CPU or disk spike or want to zoom in on process startup activity use
the zoom-in toolbar button. Note that the starting time shown will
remain 0 as you zoom in, so after zooming manually enter a time near the
activity in which you're interested in the Display Time edit box.

The CPU Usage, Disk I/O and Driver Delay graphs allow you to view more
information than is immediately visible. Select a region in the graph of
interest by moving the cursor, pressing the left mouse button and
holding it down, and then dragging. Then right-click to get a menu. The
interesting menu option in the CPU Usage graph is Show Summary, which
pops open a dialog listing a breakdown by process of CPU usage for the
selected time interval. If you double-click on one of the processes
another dialog opens that shows a breakdown of CPU usage by module (e.g.
the DLLs in the process). If you have symbols configured you can even
double-click on a module and obtain a dialog showing a breakdown of that
module's CPU usage by function.

Two menu items appear when you right click on a selected region in the
Disk I/O graph: Show Summary Table and Show Detail Graph. Select Show
Summary Table to see an accounting, broken down by module, of all the
I/Os that occurred during the interval. Choose Show Detail Graph to
display a line chart representing the actual sectors within the disk
that were accessed over the interval. The detail graph even associates
the I/Os with the processes that performed them and lets you filter out
processes by deselecting them in the list on the left. You can select a
region within the detail graph and access another summary chart for that
subinterval by right-clicking and selecting the resulting menu item.

Choosing Show Summary Table from the Driver Delay graph's context menu
brings up a list of the I/Os shown in the selected region that includes
the name of the driver that performed the I/O, the begin, end, and total
time of each I/O, and the I/O function performed.

That brings me to the end of my BootVis tutorial. Although it's not the
most polished tool, after playing with BootVis for a while you'll no
doubt agree with me that the level of detail viewable with its graphs is
amazing. The next time you have a long boot and want to know what's
responsible you'll know where to turn for the answer.

================ WHAT'S COMING UP =================
Microsoft's recent enhancements to its kernel debugging tools have made
crash dump analysis virtually painless. Next time I'll show you how to
become a dump analysis pro in under an hour.

Thank you for reading the Sysinternals Newsletter.