Pardon the GDN Dust

A few years ago, I read a book about the rise of AOL (it was before any mergers, so the fall wasn't documented yet). In it, the book covered the evolution of the service and recounted a time when AOL's services completely fell apart. Steve Case, the head of AOL at that time, needed to handle the customers that were let down by the service. At a time when he needed to be contrite and apologetic, Case instead thumped his chest in pride. After all, people wouldn't be complaining if they didn't care. He took a lot of flack for gloating about how significant his failure was and managed to upset all of the customers that were let down by the services. He eventually apologized and people moved on. The next 3.5 years were magic...

Well, over the last few weeks, I had a potentially Steve Case-type moment--hopefully without the foot-in-mouth, but with the follow-up success after a tough lesson. We tried the biggest upgrade ever with GotDotNet, upgrading servers (64-bit), Workspaces source control machines, .NET Frameworks (we're on VS 2005!), and a brand new UI (people seem to either love it or hate it, but everyone agrees we were do for a refresh). This was a major migration and done in little over a month. If you've ever been through a major migration, you are undoubtedly listening to my story and rubbing your scars.

Well, we are just about finished! If you haven't seen it, you should check it out. If you saw it during the tough deployment, please go and take a second look. And to you, we apologize for the troubles you've run into. The bugs we suffered through last week were painful to all of us, most of all Betsy Aoki and Pete Coupland, the resident superheroes that were tasked with the impossible mission of doing a monster job in a crazy timeframe and nearly pulled it off. But wouldn't you know it, things didn't work out and we got burned. I can't remember a tougher 2-3 weeks. People were not happy, least of all, the GDN team. So, learning from Steve Case, I will not boast about people's dependence on GDN (in fact, we're probably made a lot of people go elsewhere and that's just horrible). We let you down and we all feel terrible about it. Heck, I will borrow Case's apology from nearly a decade ago. But I will throw in one line of defense. The defense: people accuse GDN of continually being down, but this was our first real downtime in the last six months! We've worked hard to fix things and, while we're still not perfect, I hate to see everybody treat last week like a common instance. The goal is to keep getting better and we will have some tricks up our sleeve, both with GotDotNet as well as some other community resources that were touched upon at the September CodeSlam. Code doesn't magically change its DNA, so we didn't expect the perfect site. Still, no one on our side saw this coming. Needless to say, just as we were getting smug in our performance, we learned our lesson. It's just unfortunate that many of you had to suffer through it as well. I feel like when Slammer shut down SQL boxes a couple of years ago after Microsoft seemed to be on a roll regarding security issues. It was as if all the months of great work and progress were flushed down the toilet. Well, same with GDN and the improved performance of the past months. Hopefully, we can still get back on the right track in the eyes of users...

Again, we'll accept the blame as full-on downtime is something we are committed to avoiding as we attempt to win back your trust. We need your help to make it happen. Please give us feedback when you hit an error. We get hundreds of thousands of users that engage with us in use cases we may or may not have predicted and we benefit from the real-time feedback. If the site is still having problems, please let us know at the GDN feedback alias. Plus, there may be one more deployment downtime before we lock in for a while. We're hoping this one is simple, but we thought the last ones would be easier as well. Please stick with us. We'll try to do a better job communicating when this stuff happens and minimizing the time we're down.