Crawl logs showing Site URL crawled twice successfully. Search result showing duplicate results

Umr 521 Reputation points
2021-03-28T18:34:11.747+00:00

Hello ,

when user search for the site name , it shows up twice in the search results. Both the results are for the same SharePoint site home page.
When checked in the crawl logs , i see 2 successful crawled results for the same URL with 2 different ID . If i remove one of them using "Remove item from the index" search results shows one item in the search results but at the next crawl it shows 2 items in the crawl logs and 2 in search results again.
enabling "trim duplicate search results" also does not help. My question is why is it crawling same URL twice in crawl logs with different ID , same URL and same content source as shown in the image below? Its happening on the home page of the sub sites.

Any advice where to further look for this ?
Thanks in advance.

82182-image.png

SharePoint Server
SharePoint Server
A family of Microsoft on-premises document management and storage systems.
2,214 questions
SharePoint Server Management
SharePoint Server Management
SharePoint Server: A family of Microsoft on-premises document management and storage systems.Management: The act or process of organizing, handling, directing or controlling something.
2,798 questions
0 comments

3 answers

Sort by: Most helpful
Most helpful Newest Oldest
  1. Echo Du_MSFT 17,116 Reputation points
    2021-03-29T07:13:03.063+00:00

    Hello @Umr ,

    If Item ID is different, it means that these are two different items. Although the two items are the same and point to the same page, they will be treated as two items in the crawling.

    82311-crawl1.png

    If you want to exclude duplicate search results, please do the following steps:

    1.Go to the results.aspx and click "Edit page"

    2.Select Search Results web part and click on "Edit Web Part"

    82273-search1.png

    3.On the Search Results web part, click "Change query" button

    82200-search2.png

    4.On the Build Your Query page, go to the "SETTINGS" page and select "Remove duplicates" checkbox.

    82289-search3.png

    5.Apply it and Save page

    82269-search4.png

    Thanks,
    Echo Du

    ===================

    If an Answer is helpful, please click "Accept Answer" and upvote it.
    Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.

    0 comments
  2. Umr 521 Reputation points
    2021-03-29T14:31:20.843+00:00

    Hi Echo,

    Thanks for testing and sharing he screenshots. In my case like mentioned earlier "Remove Duplicate" is checked. it is checked by default btw, but I am still seeing duplicate results. the only time I don't see duplicate result when hard remove from index from crawl logs.

    I also tried using api with "trimduplicates=true" and also used search query tool that allow you to see the additional properties of the results. In each case I am seeing link for the same page twice.

    Any other tips, please share.
    Thanks.

  3. Umr 521 Reputation points
    2021-03-30T20:51:11.357+00:00

    Yeah, resetting index might fix lots of issues but in this case thousands of sites home page is dependent on index as they are build on using search web parts. So cant take all of them down for hours. cuz recrawl in this environment can easily take 48 hours to build again.
    Thanks for your suggestions thou. Any other advice will be helpful.