SharePoint 2013 - How to crawl IIS Virtual Directories created under SharePoint Web application

  • Article

We have recently come across a scenario where one of our customers is unable to crawl the contents of a virtual directory that's created under the Sharepoint web application . This blog post will talk about how to configure Sharepoint Search to crawl files of a virtual directory.

Lets consider the below as example:

SharePoint Web application URL: https://sp2103:4141/

Virtual Directory URL: https://sp2103:4141/test

By default, when you configure Sharepoint search , the Sharepoint site https://sp2103:4141/ will be crawled using the "Local Sharepoint Sites" Content source. To crawl the virtual directory , if a new content source is created with start address as https://sp2103:4141/Test/ , it may not crawl the contents of the Virtual directory , but trigger another Sharepoint crawl on the start address https://sp2013:4141/.

 

Lets see how we can achieve this - The first and foremost , you need some content within the Virtual directory. So lets create a few sample HTML /ASPX files

 

 

 

 

 

 

 

 

 

 

 

Then, follow the 3 steps mentioned below

1) Create a new content source.

Start Address: https://sp2013:4141/test

Content Source Type- Web site

Only crawl the first page of each start address.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2) Create Crawl rule as below

Path: https://sp2013:4141/test/*

Crawl SharePoint content as http pages

The crawls to https://sp2013:4141/test/* would be website (non-Sharepoint) crawls.

To discover other pages we need links embedded in the index page “default.aspx”.

 

 

 

 

 

 

 

 

 

 

 

 

 

3) Do full crawl on the new content source.

Now only the items under the virtual directory is crawled.

 

 

 

 

 

 

 

WARNING:  Its highly recommended NOT to modify the settings of IIS website created by Sharepoint. A proper back up of the Site / files along with a good documentation of your IIS settings will be useful in case disaster recovery scenario arises.

 

POST BY : Ajay Xavier [MSFT]