Search scopes in FAST Search for SharePoint (Part 1)

Article
02/26/2010

Introduction

When using search scopes in a FAST Search Server for SharePoint 2010 (FAST Search) installation there are some things that are nice to know to leverage the full potential of scopes. In this post I’ll try to shed some light on how scopes are used with FAST Search as the back-end.

FAST Search supports SharePoint Search Scopes

FAST Search supports all scopes that SharePoint Search exposes. These are shared scopes, site scopes and contextual scopes (e.g. scope:”this site”).

Search scopes with FAST Search for SharePoint are FQL filters

In FAST Search a search scope is just a pre-defined property filter specified in FAST Query Language (FQL) syntax that can be configured via PowerShell only OOB – the scope object itself is public though, so there’s nothing in the way of creating custom scope filter tools should one want to do that. The filter property is attached to the same search scope entity used by SharePoint Search. This means that the scopes you define to be used with FAST Search will show up in the UI in either central admin or site collection admin, but all you can do in terms of scopes administration is either to create or remove the scope itself, more about this a little later. To add, change or remove filters you need to use PowerShell.

Creating a search scope for use with FAST Search

As mentioned, there is no UI to administer scopes in an FAST Search setting, you need to use PowerShell. Let’s start with ways of creating a scope filter to be used with FAST Search. All examples use an SSA named “Search Service Application”, make sure you’re using the correct name for your SSA when creating scopes.

Creating a scope setting the filter on creation

In the SharePoint 2010 Management Shell type the following:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name MySimpleScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘title:vista’

This will create a scope named “MySimpleScope” with a FQL filter on title containing the word ‘vista’. ExtendedSearchFilter is the parameter to make a note of here.

Please note that the scope must be compiled before it can be used in your queries.

Creating a scope and setting the filter after creation

In the SharePoint 2010 Management Shell type the following:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name MySimpleScope –Description "My Simple Scope" -DisplayInAdminUI 1

$scope = Get-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Identity "MySimpleScope”

$scope.Filter = "title:vista"

$scope.Update()

Creating a site scope

To create a site scope, add the –OwningSiteUrl switch to your command line:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name MySimpleScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘title:vista’ –OwningSiteUrl “https://myhost/sites/mysite/”

You can also create a scope through the UI, but you must use PowerShell to set the filter after creating the scope.

Reviewing your scope filters after creation

To print the filter of a shared scope, do the following:

$scope = Get-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Identity "MySimpleScope"

$scope.Filter

The same for a site scope, just add a –Url switch:

$scope = Get-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Identity "MySimpleScope" –Url “https://myhost/sites/mysite/”

$scope.Filter

Update status=empty/Items=empty?

If you look in the UI after creating the FAST Search scope filters, you’ll notice that the scopes you have created are reported as empty with the number of items reported as empty as well.

The reason these scopes are reported as empty is because of the nature of how scopes are implemented. Remember that a scope filter is attached to the same scope entity as used by SharePoint Search, so what you’re seeing in the UI are the metrics of the SharePoint Search scopes, not the FAST Search filter. I know this is a little confusing, but it’s important to understand that there is (probably) nothing wrong with your scope filter even if the scope is reported as empty.

Scope rules have no influence on FAST Search queries

So what happens if you add scope rules to the empty scopes in the UI?

Scope rules will be used, but only for SharePoint Search queries. If you have setup FAST Search you will likely have two configured Search Service Applications (SSA): One query SSA with a connection to the FAST Search back-end as well as SharePoint (people) search and one ExtendedConnector SSA for feeding to FAST Search. For more information on Search Service Applications see Arnt’s blog post The two types of Search Service Applications (in a SharePoint 2010 deployment with FAST Search Server).

You can of course add content sources to your query SSA, in which the crawled content from these content sources will go into a SharePoint Search index. That’s when your scope rules come into play.

If you choose to have a SharePoint index on the SSA that is configured with FAST Search please note the following:

1. Numbers reported in the UI reflect the items in the SharePoint Search index

2. If you aim for using the same scopes with both SharePoint Search and FAST Search remember to update the scopes you crate with filters using PowerShell. My next blog post will go more in depth on how you can convert scope rules into FQL filters both manually and programmatically.

Specifying which fulltext index to search

FAST Search scope filters have one additional capability; you can in the filter specify which non-default fulltext index (=composite field if you’re familiar with FAST ESP terminology) you want to query. To specify the fulltext index as part of, or the whole scope filter use “FullTextIndex=” as part of the filter. Note that the you separate the FullTextIndex parameter and the value with equals (=).

Create a scope named “MySimpleScope” with a FQL filter on titles containing the word ‘vista’ searching the fulltext index named ‘mycustomfulltextindex’.

Using the scope in a query:

microsoft scope:mysimplescope

Fulltext index can also be a managed property

The value behind the FullTextIndex= can also be a managed property, i.e. if you specify say ‘title’ as the fulltext index your queries will be limited to this managed property.

This is best demonstrated with these two scopes:

Scope1:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name TitleVistaScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘title:vista’

Scope2:

New-SPEnterpriseSearchQueryScope -SearchApplication "Search Service Application" -Name FullTextIndexTitleScope -Description "My Simple Scope" -DisplayInAdminUI 1 -ExtendedSearchFilter ‘fulltextindex=title’

Now with these two scopes all these queries will return the same results:

Query1: title:vista

Query2: scope:titlevistascope

Query3: vista scope:fulltextindextitlescope

Advantages using scope filters

· A scope filter can be any FQL statement, which opens up for very powerful query expressions like xrank, sorting etc.

· After the initial compilation of the scope no additional compilation is necessary if changing the filter syntax. However there is an internal cache that needs to update with the new filter string – the update interval of the cache is 5 minutes (not ready immediately as I originally wrote).

· When using different fulltext indexes you can get the same recall for a query, but different rankings depending on which fulltext index you use.

· Use of scopes in search center is the same for FAST Search and SharePoint Search; both products support global scopes, site scopes and contextual scopes. Adding a dropdown and search tab backed by scopes is same for both products.

Limitations with scope filters

There are some limitations on how you can use search scopes with a FAST Search for SharePoint back-end.

You do not get stemming on filters

Unfortunately, stemming is not applied on terms used in a scope filter. This means that if you issue a query to test the effect of a scope filter, make sure stemming is turned off to see how the filter behaves.

You cannot OR scopes filters together

With FAST Search multiple scopes are AND’ed together regardless of operator used, this means that the query scope:myscope OR scope:myotherscope is rewritten to the FQL filter and(scope:myscope, scope:myotherscope) Where the scopes are replaced with their respective FQL terms of course

No UI support

As mentioned earlier FAST Search scopes with no rules are reported as empty and scope rules used both for FAST Search and SharePoint Search will report item counts matching the SharePoint index, not the FAST Search index.