Software Testing Problems 5: Starting to Test Connected Applications

If no one has completely solved issues surrounding the testing of Web Sites or traditional
Win32 applications then you can assume that there are problems abound testing applications
that connect to either other networked applications or web services.   Advancements
such as Indigo will
only make these types of applications more prevalent moving forward so you’re going
to have to get used to it.  Having tested
the Start Page in the VS 2002/2003 releases and now leading the team responsible for
testing what will be a very connected help system I can offer some advice…

"urn:schemas-microsoft-com:office:office" />


Know Thy Dependencies: You need to know
all of the services that are required by your application and you’ll need to know
the services and data stores they depend on.  It
helps to make sure you have a detailed map of your service architecture both in and
out of your span of control.


Control Dependencies:   Strict
control must be placed on the data going back and forth to each service.  If
you have a shared Schema you should put a lot of effort into making sure that this
schema is bulletproof and as restrained as possible given the needs of the application.  If
a service is meant to return you a set of contact information for customers that match
search criteria you have to make sure you have restrictions on each part of the contact
information.  The phone number should
only be acceptable phone numbers, the name only valid characters, etc.  The
data store needs to have checks in place to verify this is the case, the service needs
to make sure that’s all it returns, and the application requesting the information
needs to be sure it only uses data in the correct form. 


Your dependencies have to lock down early and stay that way.  Once
you have defined the communication arrangement you have to be sure that the call remains
the same throughout the expected life of the software unless you are prepared to test
and deliver updates to your client app as often as a web site could change.  It
would be bad if, for VS users, all of a sudden online help searches were returned
in a different format than the UI expected and was unable to display. 


Think about your edge cases here… What is the max number of records that could be
returned? What is the max space allotted for each field?  How
long will it take this full dataset to return over a 28.8 modem?


Goals need to be set early and monitored often at every level when it comes to the
service level your application expects.  If
you expect a user to be able to see their search results in under X seconds the UI
must be able to display returned results in a fraction of X, the data store must be
able to fetch the results in Y and the service must do its job and bundle the results
in Z.  Each step must be tracked both
independently and with the combined round trip to ensure the proper user experience. 


“Trust No one”: So you’ve setup testing
and contracts with your dependencies.  This
doesn’t mean your application can assume it always gets data in the right form.  This
means you’ll need a hefty set of negative tests that attempt to bring your application
to a halt at each level with invalid data.  What
if you get back more than the expected max?  What
if the name contains invalid characters?  Can
you sneak an XML tag into the results with hidden script calls? Be sure your application
ignores or reports any data it’s not supposed to use.  The
VS Start Page simply chooses to ignore data it doesn’t know how to display so, hopefully,
a poor return set won’t result in a completely blank set.  Since
it’s not a mission critical VS component we don’t mind the loss in fidelity. 


Expect the worst from your transporter :
I don’t care if Jason Statham is
moving your data around; he’s going to get stuck in network traffic, pulled over by
your users’ proxy servers, and potentially tripped by aggressive anti-virus software.   So
this means your app has to know how to handle all of these situations appropriately.  For
example, the VS Start Page used the MSXML object to fetch data from online XML stores.  Unfortunately
there is a bug in the MSXML object we didn’t know about that allows it to be interfered
with by some proxy servers even when a standard HTTP request would work to access
the data.  We could have made the call
simpler and not relied on the MSXML component to avoid this.  In
this case the transporter pulled over. I unfortunately have example bugs that fit
each of these categories and more so make sure you have given your transporter a good
work out in conditions you expect your users to encounter. 


Test with Overlap: If your dependency
chain is A->B->C->D you could test D and call it a day.  I’d
recommend that in addition to testing D you work on testing the various combinations
as well.   You should also ensure
that even if A is tested by someone else that this person tries testing through D
on occasion.  It will help the A tester
understand how their component is being used up the chain as well as give you another
set of eyes on D.  Yes, fair trade and
all, it also means the D tester should overlap and test individually down the line
as well.


That’s all for now I probably missed some important things and may not have gotten
specific enough.


I’d love your feedback on the content and depth of these testing entries.