The Mysteries of Search Crawlers and Finding Stuff on the Intranet

As the managing editor for the Office Developer Center, I spend a lot of time trying to determine effective search optimization strategies to help our content get discovered by search crawlers or robots. My focus is generally on the big Internet search engines, but I know that this can be even more critical in an enterprise environment where you want the vast knowledge stored "somewhere" to be available in under five minutes of searching. Nothing is more irritating than resorting to the native search on your local intranet, knowing the content you need is there (you saw it three months ago...where was it???)....and having it not show up in the results. So I take search personally. As a result, I'm glad to highlight two pieces of content that will help all you enterprise search folks create more relevant, custom search scopes for people in the enterprise like, well, me.

The SharePoint folks just published some content that I think will be of interest to those who want to delve into the mysterious world of enterprise search crawler. First up is an article by the prolific, Joel Krist from Akona, called Creating an Enterprise Search Crawl Log Viewer for SharePoint Server 2007. In this article, Joel discusses using the UI to view crawl details, including error messages. Then, he demonstrates how to access the same data programmatically. A code sample accompanies this article, which you can also download. The other article to talk about, also by Joel, is how to use Enterprise Search Property Filters in SharePoint Server 2007. This article talks about how to create targeted search scopes to access some of that obscure data that lives out in the enterprise "wilds" through things like managed properties, including custom metadata in doc libraries, and modified queries.

Enjoy - and please do add your comments to the articles. Your feedback helps us determine what other content we should present.