Scenario Scoreboards for Testing Your Product Success
Article
Here is an example of a simple, but surprisingly effective way to test your product success. Around patterns & practices, a few of us affectionately called this a “consumer reports” view. Whether your product is prescriptive guidance or code, the concept is the same – you’re testing your product against user scenarios.
To create a Scenario Scoreboard, simply put together a matrix of scenarios. Next, add criteria. Next, test against the scenarios and rate the criteria. Note – There are two keys to making this successful: 1) have a useful collection of scenarios, and 2) get agreement on the criteria. You want agreement on the criteria so that people can stand behind the results. Tuning your criteria can help you tune your product success. You’re basically creating a feedback loop to measure the success of your product against user scenarios.
Example of a Scenario Scoreboard
What’s important in the example is the simple frame: scenarios and criteria. The scenarios are grouped into buckets.
Category
Scenarios / Tasks
Criteria
Best Practice Compliance
Implementation Complexity
Implementer Competence
Time to Implement
General
How to push peak loads to the cloud to reduce the size of an on premise data center.
How to decide if you application is right for the cloud (on-premise vs. cloud, advantages/disadvantages)
How to run your own VM in the cloud.
How to run an on-premise app in the cloud.
How to create a small-to-medium Web app.
How to create a large Web application.
How to create a parallel processing application.
How to store blog data for an on premise application.
Arch / Design
How to structure your application.
How to partition based on latency.
How to scale horizontally.
How to scale vertically.
How to manage state.
How to design for Hybrid (i.e. part of the application runs on premise, part in the cloud)
Auditing and Logging
How to log information.
How to throttle your logging.
How to determine your log destination
How to view logs
Authentication
How to authenticate callers.
How to perform single sign on (Federation).
How to build an STS.
How to identify callers.
How to manage personally identifying information / sensitive data in the cloud.
Authorization
How to use roles to protect resources.
How to leverage claims.
How to turn your application into a claims aware application.
How to turn claims into roles.
Caching
How to use local storage.
How to leverage a distributed cache.
How to swap out cache providers.
How to cache data effectively.
How to expire the cache.
Configuration
How to cache configuration data.
How to decide what settings you should use.
Data Access
How to design an extensible schema that will never need to be updated.
How to choose a partition key for different entities.
How not to get too much data into one partition.
How to repartition your live data.
How to load initial/domain data.
How to do BI in the cloud.
Monitoring / Health
How to monitor web roles.
How to monitor worker roles.
How to alert/alarm.
Performance and Scalability
How to simulate load tests.
How to compare blog storage against local files.
How to measure performance against CRUD.
How to do capacity planning.
How to access/view Performance Counters
State
How to design statelessness.
Storage
How to decide which storage to use.
How to organize your containers and blobs efficiently.
How to track/retrieve additional information/properties about blobs
How to authorize access to containers/blobs
Transactions
How to roll back.
How to update multiple pieces of data at the same time.
How to implement 2-phase commit.
How to lock effectively.
Worker Model
How to schedule work.
How to group different types of work.
How to communicate between different types of worker roles.
How to determine the number of worker roles.
How to determine if multiple threads should be used
Workflow
How to design for asynchronous work.
How to design for integration (custom cloud applications / finished services / on premise / ESB.
Deployment
How to deploy seamlessly with no downtime
How to architect your application to allow data schema deployment to happen independent from app deployment
Ratings for the Evaluation Criteria Here are some examples of ratings for the criteria: