Longest remote home folder test … 600 miles…
Sorry it’s been so long since my last post, but I’ve been quite busy fixing bugs which are either old, new, borrowed and quite far away. In fact some are so far away it takes driving all the way to Burbank, CA to verify the fix.
I’ve spent a good number of weeks working on fixing a variety of Remote Home Folder bugs. If you are not sure what I am talking about you can read about it here: Apple Server Desktop Management. The basic idea is that all user directories live on a remote server. This allows anyone to log into any Apple Macintosh connected to the business or university network and have access to their home directory. The idea has been around since UNIX was developed. Unfortunately, when the age and design of your application pre-dates UNIX … okay, not really, but when the application wasn’t designed for this functionality you are guaranteed to run into a few problems along the way.
So why send me? Because I worked on the fix, silly, why else would you send such a shy and demure sort of guy who really doesn’t like people. (If you believe that I’ve got this bridge in Brooklyn that I just put on the market.) Seriously though, sending me all the way to Burbank to test a fix, and make me drive there to boot? What’s up with that?
The reason I went was because one of the known issues for enterprise and schools was the performance impact of running the Office applications when a user had a Remote Home Folder (RHF). Having spent my time working on emulators, video games and lots of other stuff that I can’t talk about, performance is my middle name. I was tasked, as well as a few other people with focusing on Remote Home Folder issues for this next version and getting things put right.
Reproducing the RHF bugs in our lab proved quite the challenge. The configurations we were trying to test with either differed in network bandwidth, network hardware, Mac OS software versions or even computer hardware versions. On top of that, some customers had a few users or 10,000 users. Trying to narrow down the configuration so we could reproduce the problem involved a great deal of creativity on our part. So when we have a customer close enough to our team that can try out our fixes it’s a great benefit to us and them!
So, back to the bug. Everyone out there that remembers Get1Resource and PBGetCatInfo, go get yourself a cookie … on me. Everyone who doesn’t know what those are, well I am about to explain why these two functions caused us a great deal of pain. Older applications for the Mac OS, think pre-OS X, used resource forks and resource files. These resource forks/files contained icons, data, code, window layouts, strings, etc. The way you could scan these files is by using the combination of these two old Apple APIs.
When Office launches, we scan all of the office applications, reading resource forks and collecting information to shove in the User Preferences for the suite, essentially building up caches. When you install Mac Office 2004, there are over 3000 files. On your local hard drive, this scan can take anywhere from 25 seconds to over a minute (depending completely on hardware). See where I’m going with this? Remember what I said about RHF and application design. Networks are not as fast as hard drives, no matter what anyone tells you, reading/scanning 3000+ files over the network is a bit of a slow thing to do. Some customers were reporting it would take up to 10 minutes to launch our application using RHF.
Now in Mac Office 2008, we have done a great deal of modernization across the board. No longer are we using resource forks and files. We have created application bundles, library frameworks and are completely Mach-O. Unfortunately with doing this, the number of files is now upwards of 20,000. That might seem like a lot, but considering that Safari and iTunes install about 4700 & 5800 files respectively, 20,000 files is not too bad for something as large as Office. (If you are interested in checking, try the following: Open Terminal; type cd /Applications; type find ./iTunes.app –name “*” | wc –l)
The reason behind this is that resource forks and files are now broken up into a single file per resource. Needless to say, if launching Office 2004 with RHF was slow it got much worse in Office 2008. If you were reading carefully, you would have noticed I said, “When Office launches.” Yeah, we discovered every time Office is launched it scans all the files, to verify the integrity of the caches it has build. That problem on top of the fact that we now had 20,000 files caused our performance to drop even further. This is where the team I work with started attacking the problem. It took a lot of digging, using Shark, our own internal debugging tools and really trying to understand what was going on, since the people who worked on this code I think left the company when ADA was created. After a great deal of work, many hours spent with stop watches and testers configuring lab machines, we were able to get the launch times down to 20 seconds or so depending on your machine, no matter if you are using RHF or not. Of course that was in our lab, which is why we needed to make a trip to a customer to test out all our hard work.
But why take a road trip? Burbank is a quick flight from our offices. Simply put, pregnant women don’t fly. My wife and I were expecting our first child and she had to be in San Diego for the week. Being the diligent husband, and actually really wanting to check out the San Diego office (which by the way doesn’t suck), I decided to drive down with her and along the way; I could stop by our valuable customer and verify our fix. This leads me back to why I drove all the way down to Burbank.
Watching the fix work in front of customers is a great feeling. I could say more about the visit, but there’s that implant in the back of my neck that hurts when I talk/type too much about things I’m not allowed to … ouch! They were quite happy with the results, and even happier that I was able to get another serious remote home folder bug fixed after I saw a problem they were having because we hadn’t seen it in our lab. That’s the great benefit of testing in a real world environment.
It was quite satisfying for me and the team: Stephen Shaw, Steven Splinter and Kirk Engelmeier to get this fixed for our customers.