Visual Studio 2005 - Extensibility Test Specifications
Test Specification for VS Extensibility
Have you ever wondered what we test on VSIP & Extensibility features in the VS QA team when we release Community Tech Previews? We run a small number of automated tests on most of our features against each new build of VS & VSIP. If the results from tests are good, we declare the build usable, Self-host, for further testing/usage. We call this subset of tests our Acceptance tests or Nightlies. They basically verify build quality. This is the same set of tests we run when we release Community Tech Previews. We have many more automated and manual tests that are run on an ongoing basis during the product cycle. We would like to share this set of our Acceptance tests with our customers and would like to hear your thoughts. Is this too much testing, is it good test coverage, do you see a big hole in a most commonly used feature area? Any other feedback you would like us to evaluate and incorporate? Is this useful information if you also had the results of these included when we post the Community Tech Previews?
Extensibility consists of 2 major parts, features that ships in the VS Box(namely add-ins, macros, policy, object model) and features/tools that ships in VSIP. Note, there are many other QA teams doing testing in other feature areas, what we present below is what is applicable to the Extensibility user community.
VS
Add-Ins
Walk through the VS Add-in wizard for all lanugages supported, verify add-in builds
Verify add-in loads successfully
Verify unload of add-in from the Add-in manager works
Removing of .add-in file (new xml registration for add-ins) updates the add-in manager and does not show the add-in.
Verify the add-in manager respects the Tools options setting for add-in “Allow Add-in Components to load”
Verify add-in with wizard option to create a Tools menu item works.
Walk through the Shared Add-in wizard for all languages supported, verify add-in builds.
Macros
View the Macro Explorer, verify default Macro project created
Verify Can add a new module and a new macro to the MyMacros project, can run the macro
Verify creation of a new Macro project and unloading of the same.
Recording of a set of actions in the VS Editor, Stop recording, verify the code emitted,
Playback of macro above results in expected behavior
Record of new VB project creation, stop recording, verify code emitted. Record a new macro to unload the VB project. Verify Temporary macro code previously emitted is overwritten. Verify new temporary macro runs as expected.
Verify a new macro behaves like a VS menu command that can be invoked from the Command Window
Add a breakpoint to the macro in the Macros IDE, verify running the macro from Macro Explorer hits the breakpoint and then run to execution in debug mode
Macros with event handlers does disable events when user clicks on Disable event handlers. Verify no event handlers from the Macro project are executed.
Macros with event handler, enabled for running event handlers does work correctly when a macro project with event handlers is loaded. Verify events do fire.
Verify Tools options setting for Enabling/Disabling macros works as expected from the VS IDE macro explorer.
Policy
Verify a policy action file can be applied to different language projects
Verify can apply and remove policy action files to Solution folders and projects within the solution folder
Verify ElementSet Exclude works when System.Windows.Forms.dll reference is excluded. Apply this policy file to a Winform application, verify a warning is shown in the Errorlist window. Apply the same policy file to a Console Application project, verify no warning is shown
Verify a policy file having Context links when applied shows the Contexts in the Dynamic Help window and Dynamic help window is updated when the policy file is removed.
verify can apply policy with reference for System.Windows.Forms.dll defined and excluded. Verify for a windows application a warning is shown in the error list, verify with a console app you can add the reference and a warning shows up
Verify an errorneous policy file results in an error message and does not apply policy
Verify Property constraints, ToolboxItem constraints and Command constraints work
Verify adding and applying code type policy rules work for a Winform application, where policy file has code type for winform defined and excluded. Verify for a winform application it shows a warning and for a console application it does not
Verify with solution folders and projects, Policy conflicts and propogation are handled correctly in a solution folder hieararchy
Enum constraints work correctly
Verify the correct warnings show up in the error list window
Object Model
· Events
DTE startup/shutdown event fires
TaskList,Documents, Window events fire
VC Project and ProjectItem events fire
VB,C#,J# Project and ProjectItem events fire
Build events fire
· Solution, Project & Build Configuration
On the Project object – Verify you can get to CofigurationManager, CodeModel, ProjectItems, Properties, Project.Object returns VSProject object.
On the ProjectItem object – verify you can get to ContainingProject, Document object, FileCodeModel, VSProjectItem, Properties, SubProject, Save
On the ProjectItems collection – you can get the ContainingProject, can iterate through the items and AddFromTemplate, AddFromFileCopy, AddFromFile, AddFromDirectory work
On the Solution Object – can get to the Projects, TemplatePath, AddFromTemplate, FindProjectItem, Close, Remove, Saved work
Solution Build object – can get the ActiveConfiguration, iterate through solution configurations, BuildDependencies, BuildState and Solution building works
VSProject object on VB.C#, J# projects – Verify we get can get to the BuildManager, VSProjectEvents, References, Imports and DTE Project object, PublishManager and VSProjectEvents2
Can get to the VSProjectItem from the DTE’s ProjectItem object.
· DTE, Tasklist, Outputwindow, Document, Text Editor, Toolbox, Window
Verify objects returned with DTE – ActiveWindow, ActiveDocument, ExecuteCommand
Get the tasklist object, iterate taskitems, can get to the taskitem description and add a new taskitem, set the priority on the new taskitem
Get the outputwindow object, add a new pane, can get to the TextDocument object on a outputwindowpane, add an item to the errorlist window.
Can get to the Document object, can iterate through documents collection, can get to the TextDocument object, selection returned is correct
create each object in the text editor object model and verifies creation – TextPoint, TextRange, TextSelection, VirtualPoint, EditPoint, TextWidnow, TextPane, UndoContext, HTMLWindow
Can iterate through Windows collection, can get to Document, Project, Projectitem objects on a Window object, returns the toolwindow object where supported
Get the Toolbox object, add a new toolboxtabs, new toolboxitem, can delete toolboxtab
· Globals
Verify can get to DTE globals, can add a new global, verify name-value pair returned.
· Code Model
Test CodeNameSpace methods and properties on C++, C#, Projects – AddClass, AddDelegate,AddEnum,AddStruct,AddInterface,AddNameSpace, DocComment
Test FileCodemodel & FileCodeModel2 methods and properties for VB project – AddClass, AddNamespace, AddStruct, AddEnum, AddVariable, AddFunction
VSIP
Setup:
Verify VSIP SDK can be setup successfully after installing VS.
Verify the Experimental hive is correctly setup after installing VSIP
Interop Assemblies
A Primary Interop Assembly is a unique assembly that contains type definitions (as metadata) of types implemented with COM. VSIP Primary Interop Assemblies refer to the managed definitions for the Visual Studio Integration Partner (VSIP) interfaces that clients can use to write packages using managed code (C#, VB.NET and Managed C++) targeting Whidbey IDE.
· Type Verification. Verify number of types and signatures.
Where a type can be an interface, struct or enum. The comparisons are first made between the source (a type library) and the corresponding target (an interop assembly) and then they are compared against last known good interop assembly. The second comparison is a conservative test to track, any changes or new types added, to the IA. There are some types in type libraries that are purposely omitted from the IAs.
PIA nightly test verifies following scenario for each IA that shipped in VSIP Everett Extra’s (as we call them baseline IAs) plus the new IAs that are part of VSWhidbey VSIP
1. interface number in IA matches the number in the corresponding type library.
2. enum number in IA matches the number in the corresponding type library.
3. struct number in IA matches the number in the corresponding type library.
4. interface signatures in IA matches the signature in the corresponding type library.
Besides, test stores type information of the last known good (LKG) IA build and verifies 4 scenarios above against LKG IA. Any unexpected change from LKG is logged as a bug.
· Round-trip testing between Managed and Unmanaged code
Calling Unmanaged code from a Managed package and vice-versa for 4-5 different Interfaces
Wizards
· Package Wizard: For both the languages supported for the package wizard (C# and C++):
Create new Integration package with ToolWindow, MenuItem and Editor options checked.
Verify that the project gets built without errors
Start Exp shell and verify package loads
· Language Wizard: Test that the wizard operates correctly on the default settings (using MyC) with symbols loaded from the default lexer & parser files
Verify that service.cpp & service.h were created
Verify that the g_languageProperties table was generated according to the specifications given in the wizard
Verify that the tokenInfoTable was generated according to the specifications given in the wizard
Verify that the lexer & parser files were added to the project
Verify that the project compiles
Verify that the language exists in the registry & has the correct settings in all registry keys
Verify that the default features work in the Exp environment
Samples For each Managed and Unmanaged sample:
Verify that the sample builds without errors.
Verify that the sample registers without errors.
Verify that the Environment loads the sample without errors under both the product and experimental hive.
Verify some very basic functionality of each sample.
Managed Package Framework
Verify correct operation of RegPkg on a Whidbey package
Verified basic functionality of the OleMenuCommandService class
Verify OleMenuCommand class
Verify the GetService method in the Package class
Verify that all attributes are registering properly with RegPkg
Verify that the Package class functionality is available to derived packages
Package Loading: This is Visual Studio’s package load behavior testing with various switches.
Verify that internal package loads correctly while in safemode
verify that the external package is not skipped after having failed to load and being selected not to skip.
verify that the external package is skipped after having failed to load and being selected to skip.
verify that one previously skipped package is reset when using the /resetskippkg flag
Verify that /novsip causes invalid PLK packages to fail silently to load
Verify that /novsip does not cause an internal package to fail to load
Verify that rerunning devenv w/o the /novsip flag puts the invalid PLK pkgs back the way they were
Verify that /novsip does not cause a package with a valid PLK to fail to load
You can send us feedback at vsextend@microsoft.com