Crawling Rules in SharePoint 2010

Some times when you make search results page, you may notice that there are some pages in the results that link to internal pages for lists and libraries, site settings page, and so on. And you don’t want these pages to appear for end users.

I have seen posts on the internet where people were in need to hide these links, they were trying to find out ways to change the XSLT for the search results web part. But the question is, why to retrieve these pages in the first place?

You can always make a crawl rule, that allows you not to crawl specific page URLs and in this way, these pages won’t be rendered in the search results web part.

You can reach the crawl rules by opening central administration >> Application Management >> Manage service applications >> Choose Search Service Application >> below crawling category click  Crawl Rules.

Here you can add rules that can exclude links from being crawled. Click New Crawl Rule, Type something like this: *://*/allitems.aspx and click OK. This will exclude all pages that have allitems in their URLs. Doesn’t matter if they were http or https because we wrote *

Hope this would help understanding the use of crawling rules and how would they be used in real situations.