Giving feedback on Visual Studio - some best practices
Over the past several years, the Visual Studio team has been actively working on ways to hear from you (our customer) and continue to improve our developer tools and frameworks. The Developer Division even built an entire portal (Microsoft Connect ) exclusively for gathering feedback. We’ve run numerous early adopter programs, expanded the breadth of our Beta program, and aggressively disclose our plans. While we have had our growing pains along the way, the goal has remained consistent. We want to continue to create better products through better conversations with our customers.
For those of you that love details, read on; I've included much of the way we handle things in the balance of this post. For those that just want the bottom line info, here it is in a nutshell:
We are adjusting the workflow of Connect bugs with the specific goal of communicating back to you how we plan to handle it. Some bugs are marked to be reviewed; others are deferred to the next release, and sometimes we have to make the hard business decision not to fix something. While this is the reality of all software shops, we owe you the courtesy of letting you know the plans for your submission.
We have somewhere around a million customers, and a team of about 1000 developers. The sheer volume of feedback can be quite daunting. To make matters worse, the amount of feedback is disproportionate to some teams. For example the team that builds the IDE’s shell receives many times more feedback than some other teams because it is a part of the tool that every customer uses every time they open the product. In the Visual Studio 2010 lifecycle, we received almost 500,000 pieces of feedback. That does not even count people voting on an existing piece of feedback!
We take every piece of feedback, and individually evaluate it for a way to improve the product. In every case, multiple people and multiple teams spend time working on each feedback item.
The feedback process for Connect
When a piece of feedback arrives through Connect, a triage team does the initial evaluation to determine of the item is a real request and not blank/spam. They then determine if it can be reproduced (if the feedback is about a behavior). I’m sure many of you who have submitted feedback have received requests for more information; that is part of this phase. From there, the feedback is handed off to the team that owns the feature, and they determine how it should fit into future builds of the product.
As time goes by and more people learn about Connect, we’ve somewhat become a victim of our own success. We set the bar very high for feedback, in that we respond and have subsequent conversations. In the past few years, we’ve further opened the feedback window on Connect to continue to capture feedback after we release the product, even though it was originally intended to capture only pre-release feedback. This has led to situations where some teams simply do not have the bandwidth to address the massive amount of feedback, while trying to simultaneously build the next version of the product. While every bug is evaluated, sometimes the teams have not been able to respond on an individual basis.
The developer teams loosely operate in three *modes*.
1) Implementing New Features – a time when the team primarily focuses on building the next version of the product.
2) Stabilizing – a time when the team primarily focuses on reducing the number of open feedback items, resolving and fixing as many as possible. Our Betas occur during this mode
3) Endgame – the final weeks before a release, when timelines are compressed and only the most critical bugs are addressed immediately.
Common complaints – and their causes:
All this transparency doesn’t come without its own issues; here are some of the most common issues we hear:
- I submitted a bug to Connect, and I never heard back, or stopped hearing back. What happened? - One of the most frustrating things for developers is to submit a piece of feedback and not hear back. Generally, this is due to either a bug arriving during a coding sprint, or a team that is being overwhelmed with feedback. The bug will be looked at by the triage team within a few days. However, the routing or debug teams, in some situations, simply cannot respond to all bugs immediately. We are in the process of adding a new item to the workflow, which will give you an additional response about the plans for the feedback. It may not be earth-shattering news, but at least you can get an idea of the plans for your submission .
I submitted a bug to Connect, and you tell me you are not going to fix it. Why? – Again, we have the fortune of having developers for our customer base, so they understand this better than many other folks, because they live it. Visual Studio has many layers, and sometimes we may be shown a bug that is deep in the architecture. While ideally we would like to fix it, the ramifications of changing core code would require extensive testing up through the layers, and we may simply decide that it is better to live with the *artifact* than move something in the foundation. I know that’s not a perfect answer, but it is the reality of almost every large software solution.
I submitted a bug to Connect, and it says it is fixed. Well, where is it? – Sometimes people submit feedback that turns out to be a bug, and we fix it. Naturally, they want the fix, but in most cases this fix will not be available until the next release. Typically this is because we are working with a build of the product that won't be ready to unveil for 12+ months, and that means that even though we fix it, you would need a QFE, a service pack, or even a new version of the product to get the fix. Ideally, we would like to go back and patch all previous releases, but this is not always possible.
How do we make this better?
There are really three things that will help make the experience better. First, we need to do a much better job letting you know what is going on (this blog entry is a first attempt at that). Second, we need to clearly communicate the best way to submit feedback, as in what is best handled where, etc. Finally, we need you to help us by sending highly actionable feedback to the most appropriate channel, as described below.
Where to submit your feedback
- Coding issues - If you are trying to get help with a coding issue (i.e. *Do you see anything wrong with this code?*), that is best answered in our forums. Forums are regularly monitored by both Microsoft and some of our most knowledgeable developers in the community.
- How do I? – If you are trying to learn how to do something (i.e. *How do I set up an RSS feed for my web application*), the best place to start is on the Learn tab on MSDN.
- Bugs and suggestions in the most recent release/pre-release of our product – To file a bug on Connect, go to http://connect.microsoft.com/visualstudio and set up an account. There is a direct link in Visual Studio 2010, under the Help->Report a bug menu.
- Bugs in previous releases of our product – If you are having an issue with an older version of our product, you can find options on the support page on MSDN.
How to send highly actionable feedback
For starters, it really helps to know what happened, if it is affiliated with some action in the product. When we get a bug submission along the lines of *this sucks*, we can’t do a whole lot to understand where this is happening, or how to fix it. And the time spent emailing back and forth to get details ultimately slows down the whole process and takes additional time from both us and you.
We’re looking into providing some tooling to help capture more actionable feedback, but until that happens, you can really help us move things along more quickly by providing detailed steps to reproduce the issue, if possible. We want to continue to make Visual Studio even better, and clearly your input is central to that. Please partner with us by sending in feedback in a way that helps us build a better developer tool for you. And if you have ideas on how we could make feedback better, let us know.