Get started with debugging Hyper-V Virtual Switch 1

This is the first post about the Hyper-V Virtual Switch from debugging point of view.

The Hyper-V Virtual Switch is a software-based layer-2 Ethernet network switch that is available in Hyper-V Manager when you install the Hyper-V server role. The switch includes programmatically managed and extensible capabilities to connect virtual machines to both virtual networks and the physical network. In addition, Hyper-V Virtual Switch provides policy enforcement for security, isolation, and service levels. Hyper-V Virtual Switch starts from windows server 2008 through Windows server 2012 R2. Although it keeps evolving, the fundamental architecture and concepts remain the same all the same. In a series of posts here, I will focus on debugging Windows server 2012 R2 Hyper-V Virtual Switch with real samples and troubleshooting best practices.

In the first place, we should have some basic ideas about the Virtual Switch in Windows server 2012 R2(and legacy OS too)

  • Virtual Switch is not standalone or out-of-box, it runs inside the host OS of Hyper-V for it is built on top of existing Windows OS libraries like NDIS. From windows point of view, Virtual Switch is merely a kernel driver (vmswitch.sys) running on Hyper-V host OS. So it actually shares the prominent resources like CPU/Memory etc. with the other components running on the same OS.
    A handful of offloading technologies are developed to mitigate the impact and resource dependency on Hyper-V OS. For example, VMQ is a typical solution to speed up the data transmit that bypasses the Virtual Switch to some extent or SR-IOV that almost bypasses the virtual switch completely to save the precious CPU/memory resources on the host OS.
  • Virtual Switch is built on top of NDIS library. So we must get acquainted with the NDIS before digging into the virtual switch operation. Here is a great post about debugging NDIS In addition, we also uses quite a few native Windows Debugger commands that facilitate the debugging purpose. If you are new to Windows kernel debugging, check out Ilias’s thorough tutorial  You should follow that tutorial to get your kernel debugger attached to another computer.  While you can use either Windbg.exe or Kd.exe, I highly suggest using Windbg

Okay, let’s unfold the first post. When you do connect to a Hyper-V host via live kernel debugging or open a memory dump via Windbg for debugging, you can double-check that the NDIS symbols are correctly loaded.  Use .reload /f vmswitch.sys to force the debugger to download the vmswitch.pdb symbol file.  Then use the !lmi vmswitch and look for the magic phrase “Symbols loaded successfully”.  If everything went according to plan, you should get output similar to this:

26: kd> .sympath
Symbol search path is:
Expanded Symbol search path is:

************* Symbol Path validation summary **************
Response                         Time (ms)     Location

26: kd> .reload /f vmswitch.sys

26: kd> !lmi vmswitch
Loaded Module Info: [vmswitch]
         Module: vmswitch
   Base Address: fffff8007607c000
     Image Name: vmswitch.sys
   Machine Type: 34404 (X64)
     Time Stamp: 53905875 Thu Jun 05 19:45:57 2014
           Size: ae000
       CheckSum: af005
Characteristics: 22 
Debug Data Dirs: Type  Size     VA  Pointer
             CODEVIEW    25, 6ae34,   6a034 RSDS - GUID: {5C41936C-CB8A-4FDD-A551-AAB433B8A341}
               Age: 1, Pdb: vmswitch.pdb
                CLSID     4, 6ae70,   6a070 [Data not mapped]
     Image Type: MEMORY   - Image read successfully from loaded memory.
    Symbol Type: PDB - Symbols loaded successfully from image header.
    Load Report: public symbols , not source indexed

Now that your debugger is set up correctly, we can start using nvkd.  The first command we’ll try is !nvkd.vswitch

2: kd> !
Commands for C:\DevKit\wdk81\Debuggers\x64\WINXP\nvkd.dll:
  !help    - Displays information on available extension commands
  !proc    - Displays information about a given processor
  !qosline - Displays information about a given QoS Line
  !vmq     - Displays information about a given VMQ (Virtual Machine Queue)
  !vnbl    - Displays Extensible Virtual Switch specific data of an NBL
  !vnic    - Displays information about one or more vnic(s)
  !vport   - Displays information about the given vport
  !vswitch - Displays information about one of more vswitch(es) and child

Then you can run command, !nvkd.vswitch to list out all virtual switches running on the Hyper-V host. It is what is displayed in debugger.

2: kd> !nvkd.vswitch

Type        Address             Flags       FriendlyName
VSWITCH fffffa80035d2000 () External Demo
VSWITCH fffffa8009a32000 () Internal Demo
VSWITCH fffffa8007c4c000 () Private Demo

It is how it looks like in Hyper-V virtual switch management console:

So the debug extension !nvkd.vswitch depicts the virtual switches on the Hyper-V host and you can drill down with the virtual switch object address as below:

2: kd> !nvkd.vswitch fffffa80035d2000

VSWITCH fffffa80035d2000  
    Name: AC2DA35C-661A-40AE-82F2-1F62F70DBB2A
    Friendly Name: External Demo
    RefCount: 9

    ProtocolState: 3 (VmsExtPtStateRunning)
    MiniportState: 3 (VmsExtMpStateRunning)
    DeviceId: {95141247-9BD1-4449-9223-C402789D94DD}
    Miniport: fffffa8003fb01a0   !ndiskd.miniport fffffa8003fb01a0
    Protocol Open: fffffa80043af010   !ndiskd.mopen fffffa80043af010
    IovPreferred: 0

    vPort List: Count (6)
    Type   Address           Id      Type      IsValidation  MonitorMode  FriendlyName
    VPORT  fffffa8003d80000  0x0000  Default   N             None         Internal
    VPORT  fffffa8003d7c000  0x0001  Internal  N             None         External Demo
    VPORT  fffffa8003d7a000  0x0002  External  N             None         Realtek PCI GBE Family Controller - Virtual Switch_External
    VPORT  fffffa8008b35000  0x0003  Synthetic Y             None         Dynamic Ethernet Switch Port
    VPORT  fffffa800416e000  0x0005  Synthetic N             None         Dynamic Ethernet Switch Port
    VPORT  fffffa800808c000  0x0004  Synthetic Y             None         Dynamic Ethernet Switch Port

2: kd> !nvkd.vswitch fffffa8009a32000

VSWITCH fffffa8009a32000  
    Name: E0BB29CA-1F43-4CAC-8025-BDFDA3FD3825
    Friendly Name: Internal Demo
    RefCount: 5

    ProtocolState: 3 (VmsExtPtStateRunning)
    MiniportState: 3 (VmsExtMpStateRunning)
    DeviceId: {4E700BA5-09D3-41D3-8EA3-B2FCB3AC0741}
    Miniport: fffffa80099b81a0   !ndiskd.miniport fffffa80099b81a0
    Protocol Open: fffffa80091ec7d0   !ndiskd.mopen fffffa80091ec7d0
    IovPreferred: 0

    vPort List: Count (2)
    Type   Address           Id      Type      IsValidation  MonitorMode  FriendlyName
    VPORT  fffffa80098cb000  0x0000  Default   N             None         Internal Demo
    VPORT  fffffa8009542000  0x0001  Internal  N             None         Internal Demo

2: kd> !nvkd.vswitch fffffa8007c4c000

VSWITCH fffffa8007c4c000  
    Name: D48F7DC3-D15D-4A3E-BF98-7E46FB8A1E4D
    Friendly Name: Private Demo
    RefCount: 4

    ProtocolState: 3 (VmsExtPtStateRunning)
    MiniportState: 3 (VmsExtMpStateRunning)
    DeviceId: {C264D298-D597-4BEE-BB6E-3DF414901E6B}
    Miniport: fffffa80089571a0   !ndiskd.miniport fffffa80089571a0
    Protocol Open: fffffa8008483330   !ndiskd.mopen fffffa8008483330
    IovPreferred: 0

    vPort List: Count (1)
    Type   Address           Id      Type      IsValidation  MonitorMode  FriendlyName
    VPORT  fffffa8008944000  0x0000  Default   N             None         Private Demo

The beauty of debugger extension is it provides a neat and coherent view of all important attributes of virtual switch. This is just the beginning of our journey, Please stay tuned for more walk-through in next posts.

[Update] Unfortunately, vmswitch.sys public symbol doesn't include the necessary information for public debugger to work with. nvkd.dll will work partially to print out the vmswitch data now.