SDPS - SharePoint 2013 Search Guidance.docx

Software Assurance Planning Services

SharePoint 2013 Search Guidance

SharePoint 2013 Search Guidance Table of Contents Introduction................................................................................................................................................... ............................................................................................................................................................................................ ......................................... 1 Search Overview ............................................................................................................................................................................ ............................................................................................................................................................................ 1 Common Search Scenarios ......................................................................... ................................................................................................................................................... ............................................................................... ..... 1 What's New in SharePoint S harePoint 2013 Search? ................................................................... .................................................................................................................................... ................................................................. 2 Search Center enhancements (Search Hub) .................................................................................................................. ....................................................................................................................... ..... 2 Relevance enhancements............................................................................ ...................................................................................................................................................... ............................................................................... ..... 2 Changes in crawling ........................................................................................................................................ ..................................................................................................................................................................... ............................. 3 Discovering structure and entities in unstructured content ........................................................................ ......................................................................................... ................. 3 Health Monitoring M onitoring Reports......................................................................... Reports................................................................................................................................................... ............................................................................... ..... 4 New search architecture ........................................................................................................ ............................................................................................................................................................. ..................................................... 4 Multi-tenant hosting ....................................................................................................................................... .................................................................................................................................................................... ............................. 5 What's in SharePoint 2013 Search? ............................................................................................................................. .............................................................................................................................................. ................. 5 Capabilities for Information Workers ................................................................................................................... .................................................................................................................................... ................. 5 Capabilities for IT Professionals .............................................................................. ............................................................................................................................................... ................................................................. 7 Terminology ........................................................................................................................................................................................... ........................................................................................................................................................................................... 8 Planning for Search ............................................................................................................................................................ ........................................................................................................................................................................... ............... 11 Understanding the End User ...................................................................... ................................................................................................................................................... ............................................................................. 12 Understanding the Corpus ...................................................................................................................................................... ...................................................................................................................................................... 13 Planning Your Information Architecture ............................................................................................................................ ............................................................................................................................ 19 Planning Your Search Application ........................................................................................................................................ ........................................................................................................................................ 20 Search Engagement Guidance ..................................................................................... ...................................................................................................................................................... ................................................................. 21 Types of Engagement: The Search Maturity Model ...................................................................................................... ...................................................................................................... 21 Search Engagement Success Indicators .......................................................................................................................... ............................................................................................................................. ... 23 Mitigating Risk ....................................................................................................................................... .............................................................................................................................................................................. ....................................... 23

Software Assurance | SDPS - SharePoint 2013 Search Guidance.docx

i

SharePoint 2013 Search Guidance Table of Contents Introduction................................................................................................................................................... ............................................................................................................................................................................................ ......................................... 1 Search Overview ............................................................................................................................................................................ ............................................................................................................................................................................ 1 Common Search Scenarios ......................................................................... ................................................................................................................................................... ............................................................................... ..... 1 What's New in SharePoint S harePoint 2013 Search? ................................................................... .................................................................................................................................... ................................................................. 2 Search Center enhancements (Search Hub) .................................................................................................................. ....................................................................................................................... ..... 2 Relevance enhancements............................................................................ ...................................................................................................................................................... ............................................................................... ..... 2 Changes in crawling ........................................................................................................................................ ..................................................................................................................................................................... ............................. 3 Discovering structure and entities in unstructured content ........................................................................ ......................................................................................... ................. 3 Health Monitoring M onitoring Reports......................................................................... Reports................................................................................................................................................... ............................................................................... ..... 4 New search architecture ........................................................................................................ ............................................................................................................................................................. ..................................................... 4 Multi-tenant hosting ....................................................................................................................................... .................................................................................................................................................................... ............................. 5 What's in SharePoint 2013 Search? ............................................................................................................................. .............................................................................................................................................. ................. 5 Capabilities for Information Workers ................................................................................................................... .................................................................................................................................... ................. 5 Capabilities for IT Professionals .............................................................................. ............................................................................................................................................... ................................................................. 7 Terminology ........................................................................................................................................................................................... ........................................................................................................................................................................................... 8 Planning for Search ............................................................................................................................................................ ........................................................................................................................................................................... ............... 11 Understanding the End User ...................................................................... ................................................................................................................................................... ............................................................................. 12 Understanding the Corpus ...................................................................................................................................................... ...................................................................................................................................................... 13 Planning Your Information Architecture ............................................................................................................................ ............................................................................................................................ 19 Planning Your Search Application ........................................................................................................................................ ........................................................................................................................................ 20 Search Engagement Guidance ..................................................................................... ...................................................................................................................................................... ................................................................. 21 Types of Engagement: The Search Maturity Model ...................................................................................................... ...................................................................................................... 21 Search Engagement Success Indicators .......................................................................................................................... ............................................................................................................................. ... 23 Mitigating Risk ....................................................................................................................................... .............................................................................................................................................................................. ....................................... 23


i

SharePoint 2013 Search Guidance Topology Planning ...................................................................... ................................................................................................................................................. ...................................................................................................... ........................... 23 Architectural Components .................................................................................................................................................... ....................................................................................................................................................... ... 23 Scalability, Availability, Resiliency ......................................................................................................................................... ......................................................................................................................................... 24 Logical Architecture Components ........................................................................................................................................ ........................................................................................................................................ 25 Physical Architecture / Server Roles ...................................................................... ..................................................................................................................................... ............................................................... 29 Example Topology, Indicating Virtualization ............................................................... .................................................................................................................... ..................................................... 30 Capacity Planning ................................................................................................................................. ........................................................................................................................................................................ ....................................... 31 Planning Objectives ........................................................................................................................ ........................................................................................................................................................................... ................................................... 33 Search Migration .................................................................................................................................................................... ................................................................................................................................................................................ ............ 34 Planning Migration ..................................................................................................................................................................... ..................................................................................................................................................................... 35 Migration Paths ............................................................................................................................................................. ............................................................................................................................................................................ ............... 36 Feature Migration Recommendations ................................................................................................................................ ................................................................................................................................ 36 Migration Steps ............................................................................................................................................................. ............................................................................................................................................................................ ............... 38


ii

SharePoint 2013 Search Guidance Introduction This document should be considered a component that should be included in any SharePoint Deployment Planning Services (SDPS) engagement where there is a need to provide users the ability to discover content using search. This guide is intended to supplement the core platform guidance, and in some cases, may contain recommendations that supersede information that you find in that document. SDPS is a planning offering, and in many cases, should be considered an accelerator for either an actual deployment or a launching point for deeper planning. For a Microsoft funded SDPS engagement is concerned, you rarely will have sufficient time to fully document a sophisticated solution. As such we will describe a minimum set of topics that you should cover with your customers, even if that coverage is in some cases superficial. The objective is to: Understand the overall importance of search to your customer. 



Document what you can about the content and its characterization.



Document what you can about the user communities to be served, and their search requirements.



Ultimately, incorporate what you learn back into the logical and physical architecture diagrams you will be providing back to your customer.

Search Overview Search is perhaps one of the most important aspects of any SharePoint 2013 deployment, which allows users to quickly discover content that is relevant to some need they have. In some instances, a user may know many characteristics of the content they are looking for— perhaps they’ve seen a specific document before but have forgotten where that document is stored. In other situations, a user may not know content specifics beyond a single keyword. Even in the most well designed and intuitive information architectures, search effectively allows users to spontaneously create a taxonomy of their own that facilitates both the navigation and discovery of relevant information in their SharePoint 2013 deployment. This document includes search planning resources for SharePoint 2013. The following Microsoft TechNet resource centers and blogs are relevant to content in this guide:  

Enterprise Search Resource Center Enterprise Search Team Blog

Common Search Scenarios While this is no means exhaustive, here are some reasons why a customer may be interested in search: 







To support the specific needs of a single SharePoint Web application. Only content managed by that web application is indexed and made available for queries. For small deployments, this is a typical scenario. To support the needs of a larger SharePoint 2013 deployment that may include multiple Web applications or tenants. Search can be configured to either isolate the visibility of content to users of a particular web application or tenant, or it could be configured to support cross application queries. As a dedicated search deployment to support the needs of an enterprise. For instance, content managed by disparate SharePoint 2013 deployments, file shares, and other content repositories could be aggregated into a single enterprise index, thereby enabling users to discover content without having to know where that content is stored. As a search ―service farm‖ that might support some non-SharePoint 2013 application. For example, a customer may have a public facing Web site built in ASP. A SharePoint search service farm could be used to index the content on that Web site and provide a surface through which the Web site can broker queries.


1

SharePoint 2013 Search Guidance What's New in SharePoint 2013 Search? You can use this section to gain a better understanding of what is new in enterprise search for SharePoint 2013.

Search Center enhancements (Search Hub) User interface Improved navigation helps people to discover the search box quickly, perform a search, and find results that are easily scoped to a site or higher levels. Search results are logically grouped and provide query suggestions based on past results. The hover panel allows people to see additional data about an item in the search results without having to click the item.

Result types and display templates Administrators can easily customize search results to provide rich results to end users. For example, administrators can customize results so that end users can see a content preview of a document, a picture of the author, top links and a preview of a web site, an d other items to help users take action on search results. To accomplish this, administrators can create display templates to customize the appearance of result types in search res ults. Then administrators can manage when result types should be used from the Manage Result Types page in Central Administration.

New capabilities for site administrators Site administrators can create and maintain Search Centers, and can customize the styling and settings that affect search res ults.

Relevance enhancements Relevance is determined by how well a search result, suggestion or recommendation satisfies the information need and the intent of the person issuing the query. SharePoint 2013 provides relevance improvements in freshness, and in linguistics and document parsi ng. There are also relevance improvements in the following areas:    

New ranking models Analysis of content and user interaction Query rules Result sources

New ranking models SharePoint 2013 provides new ranking models for people search, intranet sites and internet sites. A ranking model determines which items are displayed in the search results (recall), and the order in which they are displayed (ranking).

Analysis of content and user interaction The relevance of search results is influenced in part by how content is connected, how often an item appears in search results, and which search results people click. The new analytics component in SharePoint 2013 tracks and analyzes this information and uses it to continuously improve the search r elevance. You can also use analytics information to tailor the user experience by add ing web parts that display recommendations and popular items, or to display "deep links". Deep links are shown in the search results as direct links to sub-sections of a main page that are frequently visited. The search service also uses analytics to compute data for search usage reports, so that administrators can get the reports in a timely manner, even on large deployments.

Query rules Without any custom code, you can create query rules to help direct search results to the intent of your users. In a query rule, you specify conditions that will cause the rule to fire. For example, a condition might be that a word in a user's query matches a term in a SharePoint term set, or that a word in a query commonly appears in queries typed on a particular site on yo ur intranet. When a query meets conditions specified in a query rule, the rule specifies actions to improve the relevance of the associated search results. A query rule can specify the following types of actions:


2

SharePoint 2013 Search Guidance 





Add Promoted Results (formerly called Best Bets) that appear above ranked results. For example, for the query "sick leave", a query rule could specify a particular Promoted Result, such as a link to a site that has a statement of company policy regarding time off work. Add one or more groups of results, called result blocks. A result block contains a small subset of results that are related to a query in a particular way, and like normal results, you can promote a result block or rank it with other search results. For example, for a query that con tains ―Fabrikam sales report‖, a query rule might use a taxonomy dictionary to recognize ―Fabrikam‖ as a customer, and then display a result block with pertinent results about Fabrikam from your customer relationship management (CRM) system. Change ranked results, such as by modifying their relevance. For example, for a query that contains ―download toolbox‖, a query rule could recognize the action term ―download‖ and boost results from a particular download site on your intranet.

Result sources In SharePoint Server 2010, federated locations and scopes were both ways to delimit sets of search results. In SharePoint 2013, result sources replace federated locations and scopes. You create and use a result source to specify a location to get search results from, and to specify a protocol for getting those results. In comparison, in SharePoint Server 2010, you specified a location and a protocol by creating a federated location. In SharePoint Server 2010, you could specify the protocol as local SharePoint index, FAST Search Server 2010 for SharePoint index, or OpenSearch. In contrast, for protocol in SharePoint 2013 (which is called the Source Type), you can specify local SharePoint index, remote SharePoint index, OpenSearch, or Microsoft Exchange Server index. If you specify remote SharePoint index as the Source Type, you do not have to supply any custom code to handle authentication as you did in SharePoint Server 2010. With a result source, you can also restrict queries to a subset of content. The pre-defined result sources in SharePoint 2013 show how you can do this by using query transformations. For example, the pre-defined "Local Video Results" result source uses a query transformation to return only video results from the local SharePoint index. In SharePoint Server 2010, this kind of restricted query was called a search scope. You can use a result source in several ways. For example, in a Web part you can display results from a result source, as compared to displaying results from a federated location for the same purpose in a previous release. As another example, you can configure a result block to contain results from a result source. In SharePoint Server 2010, only a Search service application administrator was able to manage and configure federated lo cations. In contrast, in SharePoint 2013, site collection administrators, site owners, and site desig ners can also create and configure result sources to meet their specific requirements, rather than having to rely on Search service application administrators.

Changes in crawling Continuous crawl In SharePoint 2013, you can configure crawl schedules so that crawls are performed continuously.

Host distribution rules removed In SharePoint Server 2010, Search application administrators could remove items from the index through Central Administration. In SharePoint 2013, you can remove items from the index only by using the crawl logs.

Discovering structure and entities in unstructured content You can configure the crawler to look for "entities" in unstructured content, such as in the body text or the title of a document. These entities can be words or phrases, such as product names. To specify which entities to look for in the content, you can create and deploy your own dictionaries. For locations, you can use the pre-populated location extraction dictionary that SharePoint 2013 provides. You can store these entities in your index as separate managed properties, and later use those properties for example in search refiners to help users filter their search results. To improve search relevance, the document parsing functionality in the content processing component analyzes the structure of documents in addition to the contents. Document parsers both extract useful metadata and remove redundant information. For example, parsers extract headings and subheadings from Word documents, as well as titles, dates and authors from within slides in


3

SharePoint 2013 Search Guidance PowerPoint presentations. For HTML content, redundant generic information such as menus, headers and footers are classified as such and removed from document summaries in the search results.

Health Monitoring Reports In SharePoint 2013, you access health monitoring reports (previously called Search Administration Reports) on the left navigation pane under Crawling or Queries & Results for individual Search service applications. Enhancements to the reports make them easier to interpret. Query health monitoring reports include:      

Query Latency Report Query Latency Report for Index Engine Query Latency Report for SharePoint Default IMS Flow Query Latency Report for Federation Query Latency Report for Local SharePoint Search Results Query Latency Report for People Search Results

Crawl health monitoring reports include:    

Crawl Summary Report Crawl Rate Report Crawl Processing Per Document Report Crawl Queue Load Report

New search architecture Search uses a new, component based architecture that provides maximum flexibility in defining the topology to support search requirements for performance, availability and fault-tolerance. Content Sources HTTP File shares SharePoint User Profiles

Content Crawl Component

Exchange

Content Processing Component

Front-end

Query Index Component

Query Processing Component

Client Application

Lotus Notes Documentum

Custom

Search Administration Component

Crawl Database

Analytics Processing Component

Link Database

Analytics Reporting Database

Event Store

Search Administration Database

The crawl component crawls content sources to collect crawled properties and metadata from crawled items. It sends this information to the content processing component. The content processing component transforms the crawled items so that they can be included in the search index. The component also maps crawled properties to managed properties. In addition, the content processing component interacts with the analytics processing component. The analytics processing component analyzes the crawled items and how users interact with their search results. The information is used to improve the search relevance, and to create search reports and recommendations. The index component receives the processed items from the content processing component and writes them to the search index. The component also handles incoming queries, retrieves information from the search index and sends the results back to the query processing component.


4

SharePoint 2013 Search Guidance The query processing component analyzes incoming queries to help optimize precision, recall (which items are returned in the results) and ranking (the order of those items). The query is then sent to the index component which returns a set of search results for the query. The results can then be further processed before they are presented to the user as the search results for his/her query. The search administration component runs the required search processes and adds and initializes new instances of search components. The crawl database contains detailed tracking and historical information about crawled items such as documents and links. The database holds information such as the last crawl time, the last crawl ID and the type of update during the last crawl (add, update, delete). The link database stores the information extracted by the content processing component and click -through information. The analytics reporting database stores the results of search and usage analysis, such as the number of times an item has been viewed. The search administration database stores settings for the Search service application, such as the topology, crawl rules, query rules and the mappings between crawled and managed properties.

Multi-tenant hosting SharePoint 2013, the search system supports multi-tenant hosting.

What's in SharePoint 2013 Search? You can use this section to gain a better understanding of the key features in enterprise search for SharePoint 2013.

Capabilities for Information Workers SharePoint 2013 provides new capabilities for formulating and submitting queries, and for working wi th search results.

Query Capabilities SharePoint 2013 enables end users to create and run more effective search queries. It also enables users to issue search queries from the desktop in Windows 7. The new query capabilities are: 









Boolean query syntax for free-text queries and for property queries SharePoint 2013 supports use of the Boolean operators AND, OR, and NOT in search queries. For example, a user can execute a query such as the following: (―SharePoint Search‖ OR ―Live Search‖) AND (title:‖Keyword Syntax‖ OR title:‖Query Syntax‖) Prefix matching for search keywords and document properties Search queries can use the * character as a wildcard at the end of a text string. For example, the search query "comp*" would find documents that contain "computer" or "component" or "competency". Similarly the query "author:Ad*" would find documents created by "Adam" or "Administrator". Therefore, the query "comp* author:ad*" would find documents that contain "component" and that were created by "Adam", as well as finding documents that contain "computer" and that were created by "Administrator". Suggestions while typing search queries As a user types keywords in the Search box, the Search Center provides suggestions to help complete the query. These suggestions are based on past queries from other users. Suggestions after users run queries Search center also provides suggestions after a query has been run. These suggestions are also based on past queries from other users, and are distinct from the 'did you mean' feature. Connectors for enterprise search in Windows 7 From an Enterprise Search Center, users can easily create a connector for their SharePoint searches in Windows 7. By typing


5

SharePoint 2013 Search Guidance search queries into the Windows 7 search box users can find relevant documents from SharePoint and take advantage of Windows features such as file preview and drag-and-drop for d ocuments returned in those search results.

Search Results Capabilities SharePoint 2013 provides many improvements for getting and viewing search results. The new search results capabilities are: 





Results display The search results page includes a refinement panel, which provides a summary of search results and enables users to browse and understand the results quickly. For example, for a particular search query the summary in the refinement panel might show that there are many Web pages in the search results and many documents by a particular author. A summary might also indicate that there are mostly Microsoft Word® and Microsoft Excel® documents in the top set of results. The refinement panel also enables users to filter results—for example by kind of content (document, spreadsheet, presentation, Web page, and so on), content location (such as SharePoint 2013 sites), content author, or date last modified. A user can also filter by category based on managed properties and enterprise content management (ECM) taxonomy nodes that an administrator configures. View in Browser The View in Browser capability allows users to view most Microsoft Office documents in the browser by using Office Web Applications. Office Web Applications is the online companion to Word, Excel, Microsoft PowerPoint® and Microsoft OneNote®, and it enables information workers to access documents from anywhere. Users can view, share, and work collaboratively on documents by using personal computers, mobile phones, and Web browsers. Office Web Applications is available to users through Windows Live. It is also available to business customers with Microsoft Office 2013 volume licensing agreements and document management solutions based on SharePoint 2013. People search People search enables users to find other people in the organization not only by name, but also by many other categories, such as department, job title, projects, expertise, and location. People search improvements include: Relevance in people search results o Results relevance for people search is improved, especially for searches on names and expertise. Self search o The effectiveness of people search increases as users add data to their profiles. When a user performs a search, the search system recognizes this as a ―self search‖ and displays related metadata. The metadata can include information such as the number of times the My Site profile was viewed and the terms that other people typed that returned the user’s name. This can encourage users to add information to their profile pages to help other users when they search. As users update their My Site profiles, other users can find them more easily in subsequent searches. This increases productivity by helping to connect people who have common business interests and responsibilities. Phonetic name matching and nickname matching o Users can search for a person in the organization without knowing the exact spelling of their name. For example, the search query ―John Steal‖ could yield ―John Steele‖ in the search results; results for the search query ―Jeff‖ include names that contain ―Geoff.‖ In addition, nickname matching makes it possible for a search query for ―Bill‖ to yield results that include ―William.‖ NOTE: Phonetic matching applies to the following languages supported by SharePoint 2013: English Spanish French German Italian Korean Portuguese (Brazil) Russian 
















6

SharePoint 2013 Search Guidance 

Enhancements for relevance of search results SharePoint 2013 provides improvements to increase the relevance and usefulness of search results, such as the following: Ranking based on click-through history o If a document in a search result set is frequently clicked by users, this indicates that information workers find the document useful. The document is therefore promoted in the ranking of search results. o Relevance based on extracted metadata Document metadata is indexed along with document content. However, information workers do not always update metadata correctly. For example, they often re-purpose documents that were created by other people, and may not update the author property. Therefore, the original author's name remains in the property sheet, and is consequently indexed. However, the search system can sometimes determine the author from a phrase in the document. For example, the search system could infer the author from a phrase in the document such as "By John Doe". In this case, SharePoint 2013 includes the original author, but also maintains a shadow value of "John Doe". Both values are then treated equally when a user searches for documents by specific authors.

Capabilities for IT Professionals SharePoint 2013 includes new ways for administrators to help provide the most benefits for end users who are searching for information. IT professionals can take advantage of the following new and improved features: 









Improved administrative interface SharePoint 2013 includes search administration pages that centralize the location for performing administrative tasks. With SharePoint 2013, administrators have an interface that provides the following advantages: o A single starting point for all farm-wide administration tasks, including search administration. The most common search tasks are highlighted. A central location where farm administrators and search administrators can monitor server status and activity. o Farm Configuration Wizard After the Installation Wizard finishes, the Farm Configuration Wizard runs automatically. The Farm Configuration Wizard helps simplify deployment of small farms. It provides the option to automate much of the initial configuration process with default settings. For example, when you use the Farm Configuration Wizard to deploy the first application server in a farm, the wizard automatically creates a fully functional search system on that server, including the following: A Search Center from which users can issue queries (if the person installing the product selected this option in the o Farm Installation Wizard). A fully functional search topology that can support an index of up to 10 million crawled documents. o The ability to crawl SharePoint 2013 sites in the server farm immediately after the Farm Configuration Wizard o finishes running. Search service administration independent of other shared services In SharePoint 2013, you can create and manage Search service applications independently of one another and independently of other service applications. This is because of the new, more granular, architecture of SharePoint 2013. Support for automating administrative tasks You can automate many search administration tasks by using Windows PowerShell™ 2.0 scripts. For example, you can use Windows PowerShell 2.0 scripts to manage content sources and search system topology. Windows PowerShell support is new for SharePoint 2013. Performance, capacity, and reliability SharePoint 2013 provides many new ways to configure and optimize a search solution for better performance, capacity, and reliability, as follows: Scalability for increased crawling capability o With SharePoint 2013, you can scale the number of crawl components by adding additional servers to your farm and configuring them as crawlers. This enables you to do the following: Increase crawl frequency and volume, which helps the search system to provide more comprehensive and up-to-date results. 


7

SharePoint 2013 Search Guidance Increase performance by distributing the crawl load. Provide redundancy if a particular server fails. Scalability for increased throughput and reduced latency o You can increase the number of query components to do the following: Increase query throughput—that is, increase the number of queries that the search system can handle at a time. Reduce query latency—that is, reduce the amount of time it takes to retrieve search results. One of the general aims of enterprise search with SharePoint 2013 is to implement sub-second query latencies for all searches. You can achieve this by adding multiple index components to your farm, taking advantage of the modular index architecture of SharePoint 2013. Provide failover capability for a ll search components. Topology management during normal operations You can tune the existing search topology during regular farm operations while search functionality remains available to users. For example, during usual operations, you can deploy additional search components to accommodate changing conditions. Operations management SharePoint 2013 provides capabilities for monitoring farm op erations and customizing reports for enterprise search. Specifically, administrators can review status information and topology information in the search administration pages of the Central Administration Web site. They can also review crawl logs, as well as health reports, and can use Systems Center Operations Manager to monitor and troubleshoot the search system. Health and performance monitoring Health and performance monitoring features enable an administrator to monitor search operations in the farm. This can be especially helpful for monitoring crawl status and query performance. SharePoint 2013 includes a health analysis tool that you can use to check for potential configuration, performance, and usage problems automatically. Search administrators can configure specific health reporting jobs to do the following: Run on a predefined schedule. o o Alert an administrator when problems are found. Formulate reports that can be used for performance monitoring, capacity planning, and troubleshooting. o Search Analytics Reports SharePoint 2013 provides reports that help you to analyze search system operations and tune the search system to provide the best results for search queries. For example, reports can include i nformation about what terms are used most frequently in queries or how many queries are issued during certain time periods. Information about peak query times can help you decide about server farm topology and about best times to crawl. Searches of diverse content by crawling SharePoint 2013 can search content in repositories other than SharePoint sites by crawling or federating. For example, the search system can crawl content in repositories such as file shares, Exchange public folders, and Lotus Notes using connectors included with SharePoint 2013. Additional connectors for crawling databases and third-party application data are created easily by using the Business Connectivity Services connector framework. Support for creating c onnectors using SharePoint® Designer 2013 or Microsoft Visual Studio® 2011 s treamlines the development process. Searches of diverse content by federation SharePoint 2013 search results can include content from other search engines. For example, an administrator might federate search results from www.bing.com or from a geographically distributed internal location. 





















Terminology It is important that you have a solid understanding of the terms and definitions used throughout this document. Note that some key terminology has changed since SharePoint 2010.


8

SharePoint 2013 Search Guidance Term Best Bet

Definition Best Bets are URLs to documents that are associated with one or more keywords. Typically these documents or sites are ones that you expect users will want to see at the top of the search results list. Best Bets are returned by queries that include the associated keywords, regardless of whether the URL has been indexed. Site collection administrator can create keywords and associate Best Bets with them. Connector Connectors are components that communicate with specific types of system, and are used by the crawler to connect to and retrieve content to be indexed. Connectors communicate with the systems being indexed by using appropriate protocols. For example, the connector used to index shared folders communicate by using the FILE:// protocol, whereas connectors used to index Web sites use the HTTP:// or HTTPS:// protocols. Content Source Content sources are definitions of systems that will be crawled and indexed. For example, administrators can create content sources to represent shared network folders, SharePoint 2013 sites, other Web sites, Exchange public folders, third-party applications, databases, and so on. Crawl Rule Crawl rules specify how crawlers retrieve content to be indexed from content sources. For example, a crawl rule might specify that specific file types are to be excluded from a crawl, or might specify that a specific user account is to be used to crawl a given range of URLs. Crawl Schedule Crawl schedules specify the frequency and dates/times for crawling content sources. Administrators create crawl schedules so that they do not have to start all crawl processes manually. Crawled Property Crawled properties represent the metadata for content that is indexed. Typically, crawled properties include column data for SharePoint list i tems, document properties for Microsoft Office or other binary file types, and HTML metadata in Web pages. Administrators map crawled properties to managed properties, in order to provide useful search experiences. See Managed Properties for more details. Crawler The crawler is the component that uses connectors to retrieve content from content sources. Crawler Impact Rule A crawler impact rule governs the load that the crawler places on source systems when it crawls the content in those source systems. For example, one crawler impact rule might specify that a specific content source that is not used heavily by information workers should be crawled by requesting 64 documents simultaneously, while another crawler impact rule might specify less aggressive crawl characteristics for systems that are constantly in use by information workers. Federation Federation is the concept of retrieving search results from multiple search providers, based on a single query performed by an information worker. For example, your organization might include federation with Bing.com, so that results are returned by SharePoint 2013 and Bing.com for a given query. IFilter IFilters are used by connectors to read the content in specific file types. For example, the Word IFilter is used to read Word documents, while a PDF IFilter is used to read PDF files. Index (or ―search index‖) The searchable index contained in physical files spread across one or more index partitions (each of which contains one or more index replicas) which is used to satisfy user queries. The index is file-system based, and NOT stored in a relational database. Index Component Index Components manage the content included in the index. Each ―index component‖ handles one ―index replica‖.


9

SharePoint 2013 Search Guidance Index Partition

Index Replica

Managed Property

OpenSearch Partitioned Index

Properties Database

Query Object Model

Query Processing Component Query Rules

Result Source Ranking

Relevance

Search Center

Search Document Search Item

A logical portion of the index that corresponds to a ―column‖ in FAST Search For SharePoint 2010. Given N index partitions, each index partition contains 1/Nth of the index. Creating multiple index partitions is how to increase overall index content capacity. Furthermore, each index partition can contain multiple copies of that data, called ―index replicas‖. A logical portion of the index representing 1 copy of 1/Nth of the index content that corresponds to a ―row‖ in FAST Search For SharePoint 2010. Each index partition can have multiple copies of its content by assigning multiple index replicas to that partition – which is good for availability and query capacity. Administrators create managed properties by mapping them to one or more crawled property. For example, an administrator might create a managed property named Client that maps to various crawled properties called Customer, Client, and Cust from different content sources. Managed properties can then be used across enterprise search solutions, such as in defining search scopes and in applying query filters. OpenSearch is an industry standard that enables compliant search engines to be used in federated scenarios. See Federation for more details. SharePoint 2013 includes a new concept that enables administrators to spread the load for queries across multiple query servers. This is achieved by creating subsets of an index, and propagating individual subsets to different query servers. The subsets are known as partitions. At query time, the query object model contacts each query server that can satisfy the search so that all results to be returned to the user are included. Managed properties and security descriptors for search results are not stored in the physical index files. Instead, they are stored in an efficient database that is propagated to query servers. Query servers typically satisfy a query by retrieving information from both the index file and the properties database. The query object model is responsible for accepting inputs from search user interfaces, and for issuing appropriate queries to query servers. The search Web Parts provided by SharePoint 2013 use the query object model to run queries. Developers can also create custom user interfaces and solutions that run queries by using the query object model. Query processing components pass queries from the s earch application to the index component, executing any applicable query rules, and returning search results to satisfy user queries. This is a new concept in SharePoint 2013. There is no direct corollary in SharePoint 2010. A query rule defines processing that occurs on the query when the user executes a search. Corresponds to a ―federated search source‖ in SharePoint 2010. See ―Federation‖. Ranking defines the sort order in which results are returned from queries. Typically, results are sorted in order of descending relevance, so that the most relevant documents are presented near the top of the results page. However, information workers might choose to apply a different sort order, such as by date modified. Relevance describes how well a given search satisfies a user’s information needs. Relevance includes which documents are returned in the results ( document recall) and the order of those documents in the results (ranking). Search Center is a site based on the Search Center site template, and provides a focused user interface that enables information workers to run queries and work with search results. See Search Item A search item represents a document, list item, file, Web page, Exchange public folder post, or database row that has been indexed. Search items are sometimes referred to as search documents, but the key point is that these items are returned by s earch queries.


10

SharePoint 2013 Search Guidance Search Component

Corresponds to any of a number of processes or services running search functionality. The search components are:      

Search database

(See the logical topology diagrams elsewhere in this document to see the interrelationship between the various search components and search databases.) The search databases contain status and configuration information, not the actual search index itself. (see ―index‖ for more information) There are four distinct search databases:    

Search ―Hub‖ Stemming

Stop Word

Synonym

Visual Best Bet Word Breaker

Search Administration Component Crawl Component Content Processing Component Analytics Processing Component Query Component Index Component

Search Administration Database Crawl Database Link Database Analytics Reporting Database

See ―Search Center‖, above. Words in each language can have multiple forms, but essentially mean the same thing. For example, the verb 'To Write' includes forms such as writing, wrote, write, and writes. Similarly, nouns normally include singular and plural versions, such as book and books. The stemming feature in enterprise search can increase recall of relevant documents by mapping one form of a word to its variants. Stop words (sometimes known as noise words) are those words which are ―very common‖ in the corpus. Querying for them may affect performance by resulting in too many hits, so a cap can be placed on the hits returned to resource consumption. Note that these terms are indexed, unlike some previous search mechanisms. Synonyms are words that mean the same thing as other words. For example, you might consider laptop and notebook to mean the same thing. Administrators can create synonyms for keywords that information workers are likely to search for in their organization. Additionally, synonyms that can be used to improve recall of relevant documents are stored in thesaurus files. An enhanced ―Best Bet‖ mechanism in FAST Search For SharePoint 2010 that displays rich HTML content obtained from a URL (vs. normal ―Best Bet‖ display) Streams or words are retrieved from content sources, and those streams are broken down into discrete words for indexing. Word breakers are the components that break down streams into individual words. Streams to be indexed are normally broken down by identifying spaces, punctuation marks, and the particular r ules of each language. Also, when a user enters multiple words into a search box, that query is broken into discrete terms by a word breaker.

Planning for Search Search planning centers around understanding your end user requirements, and understanding the content that is to be searched.


11

SharePoint 2013 Search Guidance Understanding the End User The end user is ultimately the most important factor that needs to be considered when you deploy any application. Your search solution is no different and you will not only need to consider both their wants and needs, but also who these individuals a re and their relationship to the system. Common questions are: Question Impact Where are users in relation to the Particularly in global or Internet-facing deployments, the user base may system? be connecting from a variety of different locations, each subject to the unique characteristics of the network that connects them to the system. In some instances, you may need to factor in performance expectations, as describe in the next s ection. However, search also can benefit remote users as it might allow them to find content more quickly, even when they know exactly what they need, with less interaction with the system. For instance, a user might be l ooking for a document that using site navigation will take 10 clicks to get to. If, because of poor network connectivity to the system, it takes 10 seconds for each page request to be fulfilled, the total time to destination is significantly greater than if the user can enter a succinct search term on the home page and within ten seconds have a result set that includes a direct link to the desired content. From a planning perspective, make sure that you provide sufficient support for users getting relevant results. What are their performance Commonly, end user performance expectations relate to the amount of expectations of the system? time it takes from the execution of a query to the time that the system presents them with results. This may be covered in a service level agreement or it may be subjective, based on experience with other search systems. Factors that influence the perception of the overall speed of the system can include such things as adequate capacity planning, the relative location of the system to the end user, and even what additional operations the system may have to do before it can return a result set. When planning a search deployment, you should work to quantify the performance expectations and attempt to honor those expectations during your capacity planning. In addition, there may be factors that you cannot manage within the environment itself that can be mitigated through alternate approaches. For instance, a remote user may have a poor network connection to a corporate deployment and would benefit from interacting with a regional system. This regional system could potentially federate results from the corporate deployment or could itself index all or some of the same content indexed in the head office environment. Performance expectations also relate to the number of users capable of interacting with the system simultaneously. The average and peak ―queries per second‖ or ―QPS‖ that the search implementation will support is a primary KPI. Are end users already familiar with If users are familiar with either of the search technologies supported by search in SharePoint 2010 or FAST SharePoint 2010, then they may be very familiar with how queries are Search For SharePoint 2010? accepted and qualified (for instance, using Scopes). If they have been using earlier versions of SharePoint, some improvements will be obvious in SharePoint 2013. However, while some things may look the same, there are substantial improvements in what a user can enter in the search box, such as support for wildcards and Boolean operators. From a planning perspective, it is important to stress end user training so that end users are aware of the new capabilities.


12

SharePoint 2013 Search Guidance Is there a search paradigm that users may already be familiar with?

One possible motivator for this deployment may be to replace an existing system. Regardless of whether SharePoint 2013 improves on every capability offered by the previous system or not, their reception of SharePoint 2013 may be biased by the absence of one or more features that they were accustomed to having. For example, a g overnment agency may have had an advanced search form that enabled users to quickly characterize the type of content they were looking for using checkboxes. Users at a new deployment may only be familiar with public facing search engines such as Bing or Google. An even more subtle example is a situation where users expect two search terms to be processed with a specific join term (―dog house‖ would be processed as ―dog and house‖) or even treated as an explicit phrase. From a planning perspective, end user training may reduce the learning curve and also deflect any initial negative perceptions about such things as relevance. For more sophisticated search interfaces that training cannot accommodate for, the rich SharePoint 2013 search object model coupled with a much more extensible set of out of box search Web Parts, may be leveraged to support these needs. Are end users going to be issuing While language packs are discussed in the core platform guidance, you queries use language other than the must consider the impact that this has on such search related default system language? configurations, as well. For example, localization may need to be considered for noise words, synonyms, best bets, and word breakers. Are there any expectations that one set End users may be confused, distracted, or annoyed by having their of content will be kept separate from queries return results across a diverse set of content. For example, a another set of content? user may only want to s ee Finance documents and not Human Resources results when executing a query. Depending on the underlying requirement, this may necessitate planning for separate content sources, search scopes, search applications, or even separate search farms. Are there any expectations around This would typically be captured in a Service Level Agreement or SLA. search availability? Basically, you want to understand what the true importance of search is to an organization as it relates to this deployment. For example, in a public facing Internet site, search may be the primary vector by which end users get to the content they need to. Whereas in a small deployment, a search outage might be more tolerable. If there is a greater the need for availability, you should build great levels of redundancy into your design.

Understanding the Corpus The corpus is the entire volume of content that the customer wishes to have their deployment crawl and make available for query fulfillment. When you discuss this with the customer, it is generally a good idea to capture the information you are collecting in a diagram. This diagram should initially include a placeholder for the farm design and also capture who will be interacting with the system, ideally segmenting this pool of users into separate objects, where each object represents a group of users with a common set of needs or characteristics. As you gain a better grasp of the content exposed through search, you may need to consolidate or break up the users into different groups.


13

SharePoint 2013 Search Guidance Question Where and what are the repositories that the solution should be indexing?

Impact Generally, the answer you get will map to SharePoint Content Sources, but at this point you are most concerned with the various systems that SharePoint 2013 will need to interact with in order to gain access to that content. This has broad impact on your search planning, as it may indicate a need for specific content connectors to communicate with those repositories. The repository types supported out-of-box by SharePoint 2013 are SharePoint Products and Technologies sites, Web sites, Microsoft Exchange public folders, and file shares. Databases may also be crawled using Business Connectivity Services (BCS), however, it will necessitate the design or development of these connectors. Microsoft also provides a connector for Lotus notes. Some third party companies may provide connectors for other repositories, and the rich API set provided with SharePoint 2013 permits the development of custom connectors. For each repository, how is content Some content repositories require SharePoint 2013 to first authenticate secured from both an authentication against that repository before gaining access to the content that it perspective as well as an authorization manages. This may necessitate having a special privileged account one? solely for this purpose. Where content within a repository has or can have unique access requirements, the crawl account must have sufficient permission to read that content in order for it to be included in the index. Dependent on the type of repository SharePoint 2013 connects to, the repository may be unable to provide any authorization restrictions back with the content. For example, a customer may want a particular non-SharePoint 2013 Web site crawled. Although the site requires users authenticate against the system and SharePoint 2013 can honor this requirement, there is no security i nformation returned when SharePoint 2013 crawls that content. Consequently, all users of the SharePoint 2013 environment are able to see results that pertain to that secure system regardless of whether they themselves have access to that system. There are a number of strategies that you can pursue to prevent this from happening, if the customer believes it is an issue. One tactic is to attempt to index Web content using the BCS. This has a dependency on the site being database driven, the data within the database being receptive to describing using the BCS, and the database being visible to the crawler. An additional layer of security can be applied to the BCS, either directly aligning to the authorization scheme associated with the web application or through the application of broader access rules around the BCS. In other instances, while the content connector may be able to return authorization information along with the content, it may necessitate mapping credentials from the foreign system back into a format appropriate to SharePoint 2013. This is the case with the Lotus Notes connector. Finally, SharePoint 2013 still supports the concept of custom security trimmers which, at query time, can determine (typically by leveraging the user’s SharePoint 2013 credentials against the repository) whether a result should be included or not.


14

SharePoint 2013 Search Guidance What is the volume of content stored in the repository?

The answer you get should be normalized in units of search items. For SharePoint 2013 repositories, this count will include user profiles, list items, documents, and pages. For a database, this is the number of rows that are to be indexed. For Web sites, this is the number of unique Web pages. Be aware that a customer may have a Web site with a parameter driven page the rendering of which is driven by a given query string parameter. If, by altering that parameter, 4000 unique pages are delivered, the search items for this site would be 4000, not just one physical page. From a planning perspective, this volume will impact the ability of SharePoint 2013 to crawl that content. By using this example again, a full crawl will require the crawler to make 4000 HTTP requests against the web content source. Other content connectors, such as the BCS, may be able to crawl multiple items in a single request, but more commonly, the more search items there are, the more outbound requests by the crawler. This is one of the primary impacts on crawl performance and can be countered by the introduction of multiple crawlers targeting the same content source. Is the content repository ready for you In many cases ownership of s econdary repositories may be different to crawl it? from those managing the system you are designing. It is important to get approval from the owner of the secondary repository so that you understand how your crawl activity may impact that system, particularly if that repository is not sized to service these types of requests. If this type of load is of concern, defining less impactful crawl schedules or even throttling the crawling of particular content repositories may be part of the answer. For those instances where ownership is one and the same, particularly for deployments where search is indexing content managed by the same SharePoint 2013 instance that hosts it, you will still need to be cognizant of the impact that crawling has on other activities on the system. Again crawl schedules might be part of the solution, while another option may be to setup a dedicated Web FrontEnd (WFE) server that is not included in the end user r otation, so that only the crawler uses it when indexing the deployment. Crawl performance is described by the ―documents per second‖ or ―DPS‖ that the search implementation will su pport. DPS is a KPI for the overall capacity of the search infrastructure. And as described above, that includes the performance of the data source repositories. How frequently does the content in this Most repositories are never entirely static; new content is added, repository change? existing content updated, and old content retired. This information is one of the inputs that you may use to determine the crawl schedule associated with a particular repository. If the change frequency within a repository varies dependent on the datasets in the repository, you may want to consider a strategy for dividing the repository into multiple content sources to enable you to set different crawl schedules, thus maximizing the freshness of the overall index. Are there any predictors for growth? The number of searchable items in the repository today is important, but it is just as important to understand what is expected to be in that repository a year or more from now. A customer that adds 100,000 SKUs to a product database each month will impact how you plan for now for future capacity.


15

SharePoint 2013 Search Guidance What level of parity should the index A full crawl always requests everything managed by the system. This have as it relates to the content in the volume of requests impacts crawler performance and the sys tem being repository? indexed incurs load during a crawl. You also need to consider the amount of time that it takes for a full crawl to complete; for some content repositories, this may be measured in days. Incremental crawls target only content that has experienced a change, but the crawler is dependent on the system being crawled to be able to provide this information. If it cannot (as is the case with Web sites), an incremental crawl is, effectively, treated as a full crawl. It is largely an ongoing effort to develop a crawl schedule that both attends to end user expectations on content freshness, takes into consideration the frequency of change within that repository, while also being sensitive to the resource demands placed on both the system crawling as well as the system being crawled. You might benefit from content source segmentation, allowing volatile areas of content to be indexed more frequently than others. Choosing off peak crawl sc hedules, scaling crawlers to enable parallel indexing, and relying more on incremental crawls are possible alternatives. Does the repository include content This could be directories, certain types of files, or even certain named that should not be crawled? files. SharePoint 2013 offers improved support for defining crawl rules using regular expressions. It is possible to define patterns for content not being included in the index.


16

SharePoint 2013 Search Guidance What types of content are stored in the For this you are primarily concerned with how the data structures that repository? each of the search items in the repository are contained. For databases, this is likely a row of text data. For a web site, it might be a web page or a hyperlinked document. For SharePoint 2013, this could be user profiles, a rich assortment of document types, as well as a list item. While it is rarely possible to be exact regarding this, you should strive to approximate how the volume of content captured earlier is distributed across various data structures. Knowledge of the types will assist you in several ways: 









While SharePoint 2013 includes out of box a number of IFilters, others may be required to truly gain access to content stored within those documents. The amount of processing time associated with crawling one file type versus another may be significant, even if we are talking milliseconds. Ultimately, we are concerned about the aggregate impact. Different file types typically have different index densities. By this we mean that there may be more actual content that will make its way into the index for a small Microsoft Word file than in a large JPEG file. Some file types expose metadata along with their content. For example, a Microsoft Word file may have properties on it that supplement the content contained within the actual file – keywords, customer name, and subtitle are a few examples. A web page may expose HTML META tags that are exposed by an IFilter as properties. Dependent on how you choose to respond to these properties, they may end up in the property store, increasing its overall size. SharePoint 2013 may need additional configuration to crawl specific file extensions, even if an existing IFilter could crawl the content.

Does content in one repository need to There may be regulatory or security motivators for doing this. As with be kept separate from another? the similar topic in the end user section, there can be a number of solutions to this problem. If the motivation is to prevent users from seeing content that they should not be able to see, assuming the content is already secured and the crawler can consume the permissions on that content, isolation by way of security trimming should already occur. It is also possible to logically isolate content by creating multiple content sources that target different segments of a repository, or even redefining the default search scope or defining new scopes that at query time, to restrict the query to only a portion of the corpus. If the customer demands the more physical isolation, separate crawl and query applications (along with their required da tabases) could be setup to ensure that content is truly managed separately. If the demands are even more extreme, it may necessitate production of a separate farm to honor this isolation req uirement.


17

SharePoint 2013 Search Guidance For each type of content within a This is often an estimate, but may influence some of your repository, what is the average size of recommendations on capacity planning. For files, every crawl operation that content? will demand moving the file across the network to the crawler. For database search items, it is rows of data that move across the network. You must consider network saturation, but you will also need to remember that larger files could result in more of an index signature than smaller files. Larger files that have more data require more processing time to extract that data. Is content within the repository While some content connectors will not do much of this, others can do augmented by additional metadata? it greatly. For example, the file share connector will only surface such things as where the file r esides and security information around the file. With the SharePoint 2013 content connector, however, a document may be heavily decorated with system metadata, but also be supplemented with data corresponding to custom columns in a document library or even as a result of the document association with a given content type. This metadata, depending on configuration, can be saved in the property store. The more properties on a particular piece of content, the larger its signature within the property store. This is heavily dependent on a customer implementation, but you must consider this when sizing the property database. More information on Planning for Search can be found on TechNet (http://technet.microsoft.com/en-us/library/cc263400.aspx).


18

SharePoint 2013 Search Guidance Planning Your Information Architecture During the Planning Phase, you will need to work with your customer to define the information architecture of the index. Search information architecture is tightly tied to the specific application usage. As such , only high level planning is requ ired during the Envisioning. Detailed information architecture is best left to the Planning phase, where detailed information about search application user requirements are defined. Search information architecture includes the definition and use of configurable features that support search applications:   

crawled properties (i.e. meta-data from data sources) managed properties index schema

Crawled Properties These are the attributes that are discovered and indexed at crawl time, including attributes from content source systems, such as the last modified date for files in file shares, and the column data for items in SharePoint lists and libraries. They also include embedded property values from the property sheets of specific file types, such as Microsoft Office documents.

Managed Properties Managed Properties. These represent a virtual mapping between one or more crawled properties a nd the values for each item that are stored in the search index.

Index Schema The Index Schema is the structure of the data stored in the index. It is defined by the list of managed properties that are available for use in the search index. New in SharePoint 2013: 

The ability to include crawled property values in the index without assigning them to managed properties.



The ability to modify – within limits – the index schema at a tenant or site collection level.


19

SharePoint 2013 Search Guidance Planning Your Search Application When planning your search application, you start with the user experience, but always with the Information Architecture in mind. The Index Schema (described above) *is* your Search Information Architecture, and is the pivot point for planning your Search App lication.

The Five Questions Of Search A search application can be thought of (at the highest level) as a black box that spits out search results. The questions you need to ask in creating it are: 1. 2. 3. 4. 5.

What do you search for? How do you filter search results? How do you sort search results? How do you group (i.e. drill-down or refine) search results? What do you return in a search result?

The answer to each of these five questions is, in the end, a list of Managed Properties that defines your Index Schema. These managed properties are what you map your crawled properties to from your data sources. Answering ―The 5 Questions‖ well improves overall user experience!

Search Requirements Drive Information Architecture The relationship between the Search Application, the Index Schema, The 5 Questions, and the Data Sources to be crawled looks like this:


20

SharePoint 2013 Search Guidance Search Engagement Guidance Types of Engagement: The Search Maturity Model There are four primary search use cases that form the Search Maturity Model. Understanding which fits your customer’s situation helps you to better understand your customer’s business related to search , as well as provide a roadmap for the future. In general, business class search solutions can be categorized into four primary cumulative use cases: Basic Search : based on the out-of-the-box user search experience without customization Enterprise Search: adds the complexity of additional data sources and customizations Unified Access Platform: adds integration of enriched data sources combined with governance provided to support search driven applications Search-Driven Application: specifically designed search-based line of business and mission-critical solutions

  



As a customer’s volume of searchable content may vary in size from thousands to millions of documents, these use cases help you identify opportunities to introduce or extend the use of search within your customer’s portfolio, as well as be catalyst for conversations about search with your customer.

Basic Search Basic Search relies on ―out-of-the-box‖ functionality provided by SharePoint 2013 and can be delivered in a short timeframe, with minimal customizations and at a minimal cost. A Basic Search solution delivers query results in relevant order by consolidating information (e.g. structured and unstructured data) from one or more sources into a single searchable index up to 10 million documents. Leveraging a standardized user interface (UI), a Basic Search solution also supports multiple document formats and languages. If desired, a Basic Search solution can also include the following search capabilities: 





  

Content can be augmented prior to indexing by adding metadata or by extracting concepts or select text within content. This information can then be used by users as additional search criteria or to refine search results. Users are provided with advance query capabilities (Boolean searching such as ―Cat AND Dog‖ as well as wildcard searching) that returns results in order of relevance or sortable by customer selected criteria. Search results are supplemented with Refiners, such as Document Type, Author, Company, which enable users to quickly drill down into results. Query Suggestions provide a list of suggestions that appear as queries are typed. Document thumbnails for Word and PowerPoint documents give users a quick view at the first page of documents. PowerPoint documents can be viewed in their entirety directly in search results, preventing users from having to open multiple documents.

Enterprise Search An Enterprise Search solution, by definition, integrates large volumes of data from a wide variety of heterogeneous data repositories often dissimilar information sources. They also typically leverage employ the most more advanced ―Out of the Box‖ capabilities SharePoint 2013 – and frequently extends them – in order to address the a customer’s unique enterprise-wide business requirements. For example, some of these advanced capabilities may include real-time content refinement (data mining), extracting named entities and key concepts (entity extraction), customized dictionaries (linguistics), data-driven navigation (refiners), and extending the search to incorporate results from legacy applications (federation), among others. Features like relevance tuning, entity extraction, data for example, by categorizing the document based on important terms that it contains; enrichment, advanced linguistics, spelling checks, word stemming and anti-phrasing, multi-language support, and query federation. (Note: If your customer is not speaking in these terms, or others like them, they are probably better suited to a Basic Search solution at this time, which will give them the solid footing they need to grow into the more robust Enterprise Search solution over time.) Users of Business Enterprise Search solutions go to provide a single, coherent search application interface to find the best, most relevant documents for a topic of interest from available s that they are searching for in many data repositories or collections. In many cases, the interface is customized to deliver a very specific user search and results navigation experience, which may include a variety of search


21

SharePoint 2013 Search Guidance aids that enables the users to better structure their search, more specifically target desired c ontent, and internally refine it to yield apply enterprise business rules. This user application is typically a web page that is presented to all users within the enterprise, which is made up of predefined, yet highly configurable SharePoint Web Parts that can be extended via APIs. Users are able to query and drill down on search results across data repositories, so the days of having to login and query multiple back end systems are gone. Just as Business Enterprise Search solutions provide a coherent search interface, so do they provide a coherent, standardized view of the various data and content sources. No business, or even organization, is ―standard‖. Each has its own unique requirements. SharePoint 2013 really shines when implementing those requirements in an Enterprise Search solution. Just as the presentation layer can be extended via APIs, the data processing layer can also be extended via APIs. This is a very powerful capability that allows organizations to fine tune search for their business by standardizing and enriching the information collections. For example, using those APIs, it would be possible to: • • • •

Provide a common name for entities with different names in different systems, such as. For example, where when one system says contains ―Human Resources‖ and the other says contains ―HR‖. Recognizing and extracting pattern-based entities, such as phone numbers or email addresses. Combining information about a single document topic or document from multiple systems sources as it the content is being indexed. Recognizing and redacting content, such as social security numbers.

By implementing Enterprise Search, customers benefit by centralizing their users’ point of access for finding documents. This increases end user efficiency, reduces the time users waste consume by individually querying separate repositories, and increases the likelihood of IP re-use within the user community. As the customer’s experience with their Enterprise Search solution grows, they will find that the centralized searc h experience can become the foundation upon on which knowledge management and corporate taxonomies can be built – which begins to move the customer beyond Enterprise Search to the next level in the Enterprise Search Maturity Model, Unified Access Platforms

Unified Access Platform (UAP) Unified Access Platforms (UAPs) are considered to be an Enterprise Search platform’s next evolutionary step. While similar to Enterprise Search in the characteristic of access to multiple data sources, UAPs introduce the concept of Search as a Service. With Search as a Service, a single search engine services multiple applications that consume the content, ranging from simple departmental search pages to Search Driven Applications (SDAs). Due to its purpose to support multiple interacting applications, UAPs are often more complex in data enrichment and normalization, index an increased number of data sources, and have more stringent knowledge management and corporate taxonomy governance models to ensure data consistency. By implementing a Search as a Service model companies benefit by reducing or eliminating the need for multiple search engine deployments, consolidated development and operations staff, increased control of data normalization and standardization, gain the ability to restrict data access and provide customized relevancy models to all consuming applications.

Search-Driven Applications (SDAs) Search Driven Applications, or ―SDAs‖, are solutions built using search as the foundational, enabling technology. Typically, they to deliver a unified work environment that supports the customer’s specific task or workflow, and may appear in the form of a Tier- 1 or mission-space application. Like the other search use cases listed above, the underlying use of search technology enables the SDA to integrate large volumes of data from multiple sources across the enterprise, both structured and unstructured, without the need for expensive data migration i nto a single line of business application. However, unlike the other search use cases, SDAs are primarily follow an application development format, rather than a technology infrastructure deployment format. Adding search technology to the solution team’s integration and application development toolbox provides a powerful, and unique, set of capabilities in addressing the business needs of the customer. The more you are able to engage with the customer in addressing the business requirements driving the need for the SDA, the more value you will bring to that customer. An SDA’s def ining characteristic is that the underlying search engine is hidden under a compelling user interface. Interaction design and workflow eliminate the need for users to jump between applications. Users may not even be aware that the application they are using to do their job is driven primarily by search. Shielding the user from the complexity of operating separate applications reduces the user’s cognitive load, enabling them to focus on getting their work done.


22

SharePoint 2013 Search Guidance As a Tier-1 or mission-space application, an SDA integrates domain-specific knowledge into the user experience in terms d irectly applicable to that the business. This reduces the user’s learning curve and enables users to be more efficient in achieving their business objectives by including, for example, industry taxonomies and vocabularies, internal processes, task workflows, industry business rules, specialized collections of information, and decision heuristics typical of the field.

Search Engagement Success Indicators Some search engagements begin with un-realistic expectations on the part of the customer. This may be due to a lack of understanding of what it takes to implement search successfully. Experience has shown that some of the key indicators of success for search engagements include: Search requirements are deeply understood Proper provisioning of people, infrastructure, organizational influence & commitment, etc. The customer has a central search architect o o Expert users are actively involved in the definition and validation of the implementation Administration teams are closely involved o Search is a core part of all aspects of enterprise architecture planning: content management o knowledge management o application development o o operations etc. o

• •

•

Mitigating Risk The three keys to mitigating risk in a search engagement are universal to all engagement risk management: 1. 2. 3.

communication communication communication

Primary recommendations include: • • • • • •

Address issues/risks/questions as soon as possible with project team Require short turnaround times to provide data and make decisions Document and track non-verbalized or un-documented expectations Make sure to put risks on the table during the requirement discussions if it truly is an expectation and success factor Focus on the real critical drivers and make sure they are addressed during engagement For expectations deemed out of scope, realign expectations or create change request

Topology Planning Architectural Components There are four key architectural components that need to be understood prior to pursuing any topology design work:

Crawler The crawl component invokes connectors that are capable of communicating with content sources. Because SharePoint 2013 can crawl different types of content sources (such as SharePoint sites, other Web sites, file shares, Lotus Notes databases, and data exposed by Business Connectivity Services), a specific connector is used to communicate with each type of source.


23

SharePoint 2013 Search Guidance Content Processing Once the crawler has captured a document from the source, it passes it, and its meta-data, to the Content Processing component, which converts it to text (if necessary), and performs other processing on the document to prepare it for indexing - such as language detection, meta-data detection and extraction, entity extraction, document weighting based on user analytics, etc. New in SharePoint 2013 is the idea of ―continuous crawling‖ for SharePoint data sources. This is where a particular subset of SharePoint data (a site or list) is incrementally crawled extremely frequently.

Indexing The index component performs two main functions: managing documents in the index (received from the content processing component) and responding to queries (from the query component). The index components are tuned for to to allow new documents to be discoverable to query components with very short latency. There is no longer any need to wait for a full crawl before documents are searchable.

Query Processing The query component is responsible for returning results from the index in response to a query received via the query object model. The query component is also responsible for processing query rules prior to submitting the query to the index component.

User Interface and Query Object Model As mentioned above, searches are formed and issued to query components by the query object model. This is typically in response to a user performing a search from the user interface in a SharePoint site, but it may also be in response to a search from a cust om solution (either hosted in or out of SharePoint 2013). The search might have been issued by custom code, such as from a workflow, or from a custom navigation component. In any case, the query object model parses the search terms, and issues the query to a query server in the SharePoint farm. The results of the query are returned from the query server to the query object model, and the object model provides those results to the user interface components (or other components that may have issued the query). Query rules can be applied to all queries, whether coming from the search user interface, or a custom search application. However, only the search user interface in SharePoint supports the use of result types in r endering those results.

Analytic Processing Analytic processing combines information gleaned during the document crawl, as well as usage information captured as users interact with the search application. These analytics are used to influence the relevance of documents based on user activity.

Scalability, Availability, Resiliency Search in SharePoint 2013 provides scalability and availability in an entirely di fferent manner than either SharePoint 2010 Search or FAST Search For SharePoint 2010. Each component to be able to run multiples in a single implementation, spread over a configurable number of servers. You can duplicate components multiple times, and isolate the servers to provide availability despite hardware failures. This level of flexibility means that you can scale your SharePoint Search farm to a degree not seen before. The enterprise search features in SharePoint 2013 are intended to provide sub-second query latencies for all queries, regardless of the size of your farm. You can achieve this by implementing a scaled-out architecture. SharePoint 2013 enables you to scale out every logical component in your search architecture. The search components fulfill high-availability requirements by supporting multiple, distributed instances. The database components fulfill high-availability requirements by supporting mirroring and clustering.


24

SharePoint 2013 Search Guidance Logical Architecture Components Search in SharePoint 2013is re-architected with new components to facilitate greater redundancy within a single farm and to allow scalability in multiple directions. The search architecture consists of components and databases that work cohesively to perform the search operation. All components reside on application servers and all databases reside on database servers. An overall diagram showing the relationship of the various components is shown here: Content Sources HTTP File shares SharePoint User Profiles

Content Crawl Component

Exchange


Front-end

Query Index Component


Client Application

Lotus Notes Documentum

Custom


Crawl Database


Link Database


Event Store


Crawl and content processing architecture The crawl and content processing architecture includes the crawl component, crawl database and content processing component. Both components can be scaled out based on crawl volume and performance requirements.

The crawl component uses connectors to traverse each content source, according to crawl rules that an administrator can define. For example, the crawler uses the file connector to connect to file shares by using the FILE:// protocol, and then traverses the folder structure in that content source to retrieve file content and metadata. Similarly, the crawler uses the Web connector to connect to external Web sites by using the HTTP:// or HTTPS:// protocols, and then traverses the Web pages in that content source by following hyperlinks to retrieve Web page content and metadata. The crawl component is responsible for crawling content sources. It delivers crawled items – both the actual content as well as their associated metadata – to the content processing component. The crawl component invokes connectors or protocol handlers that interact with content sources to retrieve data. Multiple crawl components can be deployed to crawl simultaneously. The crawl component uses one or more crawl databases to temporarily store information about crawled items and to track crawl history. Crawls content based on what is specified in the crawl databases. Add crawl components to address capacity requirements and to increase crawling performance. Typically add 1 crawl component per 10 million items.

The crawl database contains detailed tracking and historical information about crawled items. This database holds information such as the last crawl time, the last crawl ID and the type of update during the last crawl. Crawls of a single data source can be spread across multiple crawl databases to improve crawling performance. There is no longer a need to assign a single data source host to a s ingle crawl database. Stores the crawl history Manages crawl operations Each crawl database can have one or more crawlers associated with it.  

  


25

SharePoint 2013 Search Guidance As a rule of thumb, a typical implementation will add 1 crawl database per 10 million indexed items.

The content processing component is placed between the crawl component and the index component. It processes crawled items and feeds these items to the index component. The content processing component transforms crawled items into artifacts that can be included in the search index by carrying out operations such as document parsing and property mapping. Both the content processing component and the query processing component perform linguistics processing. Examples of linguistics processing during content processing are language d etection and entity extraction. The content processing component writes information about links and URLs to the link database. In turn, the analytics processing component writes information related to the relevance of these links and URLs to the s earch index via the content processing component.

Analytics Architecture The analytics architecture consists of the analytics processing component, analytics reporting database and link database.

The analytics processing component analyzes crawled items (search analytics) and how users interact with search results (usage analytics). It uses the information to improve search relevance, and to create search reports, recommendations, and deep links. This component extracts the following: 



Search analytics information such as links, anchor text, information related to people, metadata, etc. from items that it receives via the content processing component and stores the information in the link database. Usage analytics information such as the number of times an item is viewed, from the front-end via the event store.

The analytics processing component analyzes both types of information. The results are then returned to the content processing component (using a partial update) to be included in the search i ndex. Results from usage analytics is also stored in the analytics reporting database.

The link database stores information extracted by the content processing component. In addition, it stores information about search clicks; the number of times people click on a search result from the search result page. This information is stored unprocessed, the analytics processing component performs the analysis. Stores the information extracted by the content processing component and also stores click-through information.

The analytics reporting database stores the results of search usage analysis.

Index and Query Architecture The index and query architecture includes the index component, index partition, and query processing component, all of w hich can be scaled out based on content volume, query volume, and performance requirements.

An index partition is a logical portion of the entire search index. The search index is the aggregation of all index partitions. Each index partition holds one primary index replica and zero or more secondary index replicas. Each index replica is an index component. When you provision an index component you associate it with an index partition. The index replicas within an index partition can be distributed across multiple application servers. The index is the aggregation of all index partitions. Each index replica within an index partition contains the exact same information. The search index can be scaled in two directions:

1

To translate from the language of FAST Search For SharePoint 2010, an “index partition” can be thought of as a “column”, and a n “index replica” can be thought of as a “row”.


26

SharePoint 2013 Search Guidance 1.

2.

Index replicas can be added within index partitions according to query load or fault tolerance needs. Each index partition has one primary index replica and zero or more secondary index replicas. Within an index partition, each index replica contains the same information. For example, in a farm with one index partition that contains one primary index replica and two secondary index replicas, each index replica serves one-third of the total queries. Index partitions can be added to handle increased content volume. For example, in a farm with three index partitions, each index partition contains one-third of the entire search index. 2

As a rule of thumb, an index replica should be sized at no more than 10 million documents. This example shows an index configured with 3 index partitions and 3 index replicas. Index partition 1

Index partition 2

Index partition 3

Primary index replica



Secondary index replica






The index component receives processed items from the content processing component and writes those items to an inde x file. The index component receives queries from the query processing component and provides results sets in return. There are two types of index components: primary index replica and secondary index replica. Each index component (i.e. index replica) generally runs on its own server.

The query processing component is placed between the search front-end and the index component. The query processing component analyzes and processes search queries and results. Both the query processing component and the content processing component perform linguistics processing. Examples of linguistics processing during query processing are word breaking and stemming. When the query processing component receives a query from the search front-end, it analyzes and processes the query to attempt to optimize precision, recall and relevancy. The processed query is then submitted to the index component. The index component returns a result set based on the processed query back to the query processing component, which in turn processes that result set before sending it back to the search front-end.

Search Administration Search administration is composed of the search administration component, and its corresponding d atabase.

The search administration component is responsible for running a number of system processes that are essential to search. This component carries out provisioning, which is to add and initialize additional instances of the other search components. Monitors incoming user actions and updates the search administration database. Only one search administration component can be active per Search s ervice application. Additional search administration components can be added for redundancy.

2

Note that the FAST Search For SharePoint 2010 “High density” index configuration has no direct corollary in SharePoint 2013. (The SharePoint 2013 strategy is to scale out, not scale up…)


27

SharePoint 2013 Search Guidance The search administration database stores search configuration data, such as the topology, crawl rules, and the mappings between crawled and managed properties. Stores search configuration data. Only one search administration database is allowed per Search Service Application.


28

SharePoint 2013 Search Guidance Physical Architecture / Server Roles Search in SharePoint 2013 has three different server roles, defined by which search components are configured to run on them:   

Application Server With Index Application Server With Search Components Database Server

Application Server, With Index The application server (with index) contains an index component. It can contain other search components as well, but it is the presence of the index component that makes it this kind of server. At least one index partition must be configured per farm. If only one index partition is configured, it holds the entire search index. Otherwise, it holds 1/Nth of the index, if there are N index partitions configured by the administrator. Each index partition has at least one or more index replicas, for redundancy and increased query throughput. Each index replica is an index component. The query processing component routes incoming queries to index replicas. Add one index partition for every 10 million items in the search index. Note that the distribution of ―index components‖ across search application servers (with index) is more flexible than with FAST Search For SharePoint 2010. Though you will probably want to for performance reasons, you are no longer limited to a single ―index component‖ per search application server. Note that the target scaling model is to scale-out, i.e. have many 10M document index components, each running in its own server, which are likely to be VMs.

Application Server, With Search Components The application (with search components) server holds all the search components if only one server is configured. Otherwise, this server holds the various components that are configured by the administrator. Note that at least one of each type of search component must be configured per farm, but that multiples should be configured on separate servers to provide redundancy. The (non-index) search components to be configured on one or more of this type of server are:     

Search Administration Component Crawl Component Content Processing Component Analytics Processing Component Query Component

Database Server The database server hosts the four search-related databases:    

Crawl Database Link Database Search Administration Database Analytics Reporting Database

It can host other SharePoint 2013 databases. As with any other database, it can be mirrored or clustered. To increase performance and capacity, consider adding disks to the database server or adding database servers (depending on the bottleneck).


29

SharePoint 2013 Search Guidance Example Topology, Indicating Virtualization Here is an example of a small SharePoint 2013 search topology that can contain up to 10M items. This farm is intended to provide the full functionality of SharePoint 2013 search. The three search administration components provide redundancy. (A redundant setup for search administration requires more than one search administration component to be available at all times.)

Application Server

Qu

Cr

Ad

In

Co

Qu

Ad

Legend Ad


Ad


An


An


Co


Cr

Crawl Component

Cr

Crawl Database

Li

Link Database

In

Index Component

Qu


Co

An

An

Application Server

Database Server

Cr

In

Qu

Application Server

Ad

Li

Ad

An

One Virtual Machine

One Application Server

What is not indicated in this diagram is the fact that the two index components constitute primary and secondary index replicas in the same index partition. Also not indicated is any database mirroring or clustering to provide database availability.


30

SharePoint 2013 Search Guidance Capacity Planning Capacity planning for search has 3 axes: •

Content capacity (the number of documents in the index)

•

Query capacity (the number of simultaneous queries the index can support)

•

Crawling/Indexing capacity (the rate at which documents can be ingested into the index)

Content Capacity Unlike SharePoint 2010 Enterprise Search, and like FAST Search For SharePoint 2010, search in SharePoint 2013 does NOT put the content of the index into a relational database. The index itself is stored directly in the file system for each ―Application Server, With

Index‖. Databases *are* used to s tore administrative, link, and analytical data – as well as detailed tracking and historical information about crawled items. But not the crawled items or content. As a rule of thumb, (and going on the premise that disk is relatively inexpensive) the ―scale-out‖ strategy for index components means that content capacity can be estimated based on the number of documents in the index. Since 1TB of disk is recommended for each index component, which will support up to 10M documents, simply divide the total required documents by 10M, and multiply by the number of index replicas per index partition to determine overall storage requirements. See TechNet (http://technet.microsoft.com/en-us/library/jj219628.aspx) for more details on server requirements, disk sizing and database requirements for each index component.

Most customers will estimate their content by citing the size of the database supporting SharePoint. This is not quite sufficient. What is needed is a document, or item, count. Therefore, you must help your customers to determine their total item count before you estimate the architectural requirements outlined in this document. The first pass estimate is to take the database size and divide by an estimated average document size. This does not take into account things like versioning. For those who care to spend the time, the right database queries (against the document table, etc.) will provide a more accurate estimate of the documents to be indexed – keeping in mind that only the most recent version of each document is indexed. A more typical and manageable approach is to estimate corpus size rather than measuring it. The steps for estimating corpus size are as follows: 1.

Categorize the different content forms, such as files, Web pages, lists items, and database items.

2.

Multiply the average size of each content form by the number of items in each form, to obtain si ze estimates of each content form.

3.

Add together all of the size estimates.

Customers must also estimate corpus growth characteristics, based on past growth patterns, and gather expected growth characteristics from analysts and systems management staff in the organization.

Although the main governing factor that affects index size is the size of the corpus, the relationship is not a simple one. Most productivity search engagements do not analyze the content characteristics. Most simply estimate the number of doc uments, and then adjust capacity demands based on discoveries made during the Envisioning, Planning and sometimes, even Build Phase of the engagement. An example of this might be where the team discovers that documents are larger, or smaller, than expected. Frequently, customers will discover that they have more or less documents than they originally expected.


31

SharePoint 2013 Search Guidance However, if you decide to pursue it, the following characteristics of the content in the corpus can affect the relationship between overall corpus size and index size: File Formats. These affect the ratio between total file sizes and index size. File compression can also affect this ratio. For



example, the compressed nature of the a more recent Office Word format results in a smaller file size than if the equivalent content is stored in an Office Word 2003 file. Content Density. This is the ratio of textual content in files to embedded objects. For example, a PowerPoint presentation with



15 slides of images will have less density than a PowerPoint pre sentation with 15 slides of text. The former may have a larger file size, but the latter will have a larger index footprint. Content Uniqueness. This represents the uniqueness of the content that is being indexed. SharePoint 2013 tokenizes indexed



words for efficient storage and lookup; the less unique the words that are being indexed, the lower the ratio between the corpus size and the index size. This factor applies both to uniqueness of words within files, and to uniqueness of content between files: o

Uniqueness within files. If a 10 MB file contains technical content about SharePoint 2013, it is likely to have many occurrences of the words such as SharePoint, search, Microsoft, enterprise, document, file, server, index, and q uery. Because of the tokenizing of these common words, the space required to index the file will be smaller than that required to index 10 MB of a novel that has a rich and varied vocabulary.

o

Uniqueness of content. The size of a index from indexing a corpus that consists of many unique documents about various subjects is larger than a corpus that consists of many copies of similar documents. For example, if an organization stores a copy of terms and conditions in each project site within a site collection, the terms and conditions are likely to be very similar for each project, with perhaps only minor variations on a project-by-project basis. The words within these documents are tokenized by the indexer and result in a smaller index than if each file had relatively unique content.



Diminishing Uniqueness. Because all vocabularies are essentially limited, there is a relationship between total corpus s ize and the ratio of that size to the index space requirements. This is simply a statistical phenomenon: 10 terabytes of data usually contain less unique content as a proportion of the corpus size than 1 terabyte of data. To illustrate this point further, as a corpus grows, it tends to include more and more occurrences of words that have already been used elsewhere in the corpus, until at some point the corpus contains every word in the organization’s vocabulary and further additions to the corpus does not introduce

new words.

Another factor that affects the ratio between corpus size and index size is the versioning strategy in the farm. 

SharePoint Versioning and Indexing. The SharePoint 2013 indexer only i ndexes one version of each item, so it is not possible to index all of the versions of files in a document library, or all of the versions of items in a list.



Versioned Corpus and Index Ratios. If a corpus is characterized by many versions of items in SharePoint lists or libraries, t he ratio of the entire corpus (including all item versions) to the size of the index file is higher than if versioning in SharePoint lists and libraries is disabled. You should draw your customer’s attention to this if the corpus size measurement is based on

content database sizes. 

Content Access Accounts and Versioning. The content access account affects the versioned content that is being indexed (although it does not affect the ratio between corpus size and index space requirements). SharePoint technologies can maintain multiple versions of a page or document and present specific versions to different users based on their roles. For example, if a user checks out and modifies a published page, and then saves it but does not checked it back in, the next time that she requests the page, she is presented with the saved version. Anyone else who requests the page is presented with the latest published version. Then, if the user makes further changes and checks the page back in and submits it for approval, the next time she requests the page, she is presented with the edited version that is waiting for approval. And any person who is in the approver’s role is also presented with that version. However, all other readers are presented with the

latest published version. In the same way, when the indexer requests a page or file for indexing purposes, SharePoint


32

SharePoint 2013 Search Guidance technologies presents the version of the item that is appropriate for the account that is being used to perform the crawl. Although there is no fixed rule for selecting content access accounts, it is important to specify an appropriate account for the crawl. In general, if only approved, published content is indexed, a reader’s account should be used to crawl SharePoint

content. However, for unpublished content, perhaps for a volatile authoring environment, then an editor account, approver account, or another administrative account would be appropriate.

Query Capacity Query capacity is defined by the number of simultaneous queries that the index can support, as indicated in ―queries per second‖ or ―QPS‖. Most intranet/productivity search implementations tend to have fairly low user QPS - <5QPS on average. However, with SharePoint 2013, multiple queries may be executed against the back end index for a single user query. (This is dependent upon things like the query rules that are implemented, etc.) This raises the QPS required for the underlying search engine, and increases the likelihood that additional index replicas will be required for each index partition – which is the mechanism for scaling-out query capacity. For planning purposes, you want to determine the average and peak user QPS required of the system, and balance that against the expected query latency (the time for the search to return) that is acceptable. In other words, if it’s OK that searches take longer at peak times, then you can get away with less server hardware. But if you need to handle the peak load without increasing your query l atency, then you are likely to need more server hardware. If historical query data is available, that is the most accurate predictor of future usage. It is not un-common for basic intranet, productivity search applications – even for some large organizations whose names you would recognize – to be measured in ―queries per day‖ or ―queries per hour‖. But you do need to document the req uirement.

Crawling/Indexing Capacity Crawling/Indexing capacity is defined by the number of documents per second that the search engine can ingest, as indicated in:  

―documents per second‖ or ―DPS‖. document latency (i.e. how long until a changed or new document is searchable)

This DPS rate defines the time it will take you to perform the initial, bulk crawl of the corpus. Tuning indexing performance for DPS is as much art as science, and requires knowledge of:   

 

The source repository performance characteristics. (Is it bound on CPU, disk, memory or network access?) Physical infrastructure (networking, servers, RAM, disk I/O, CPU…) The index topology (How many search components? And of which type? Which servers are they on? Which VM Hosts are the VMs on?) The CPU, disk I/O, RAM and network capacity usage of each type of search component The corpus itself (are the documents text based or binary formats, etc.)

Tuning for document latency requires analyzing the needs of the various crawl schedules, targeting the most valuable (& presumably volatile) information for more frequent incremental crawls. SharePoint 2013 search is an entirely new architecture, that supports being able to search for documents immediately after indexing, without having to wait for a full crawl to occur. It also supports ―continuous crawling‖ which lets you focus on high value, highly volatile SharePoint data.

Planning Objectives Using all the information that you have been able to collect thus far, you should be nearly prepared to propose a search architecture that includes the following:    

The number of servers and server roles required to support search The number & type of search components required to support performance, isolation, and redundancy requirements The number of databases and their association with the search components The number and distribution of index partitions and index replicas across those servers


33

SharePoint 2013 Search Guidance  

The distribution of search components across servers The distribution of servers across virtual machine hosts

An excellent, actionable set of guidance can be found here: http://technet.microsoft.com/en-us/library/cc263400.aspx You should also have sufficient information to describe what could be considered a starting point for content source planning including: 

 



A listing of all content repositories that this deployment will need to interact with. Note that in this instance, you should have already divided this repository into what should be approximations to SharePoint content sources A characterization of the type, volume, and size of content within each content source Any special considerations related to that content source – such as whether a BCS or other custom connector will need to be developed or procured A characterization of the frequency in which content changes within that content source and a description of how your customer needs to account for those changes with crawl schedules.

You should also have sufficient information to describe the basic search application interface (including the information architecture…) to be supported by the search solution, including:    

A wire-frame diagram of the search user interface A listing of the managed properties required by that search user interface A listing of the crawled properties by data source, Any unique crawled-property-to-managed-property data mapping requirements

You should also have sufficient information to describe the user communities to be supported by the search solution, including:   

A description of each community The general search requirements for that user community Any unique search requirements for that user community

You should also have sufficient information to identify any specialized search requirements that call for customization of the search solution, including:    

Custom data source integration Custom query processing Custom document content processing Custom user interface functionality

Search Migration SharePoint 2013 provides limited out-of-the-box automated assistance for search migration, but does provide a PowerShell cmdlet interface that makes search migration a good entrée for services engagements. Due to the changes in the various underlying search implementations between SharePoint 2010 and SharePoint 2013, customers are able to migrate some configuration information - s uch as content sources, crawl rules, start addresses, server name mapping, and federated location – from SharePoint 2010 to SharePoint 2013 search using a database attach strategy. Some other, but not all, FAST Search for SharePoint 2010 feature configurations are able to be migrated to SharePoint 2013, using 3 custom PowerShell scripts. Considering the major architectural changes to search i n SharePoint 2013, there is no direct mapping for either the search topology or the index itself. You cannot use the database attach approach to upgrade any other search databases, such as crawl databases or property databases. (These databases are re-created when you perform a full crawl in the new farm.) Also, the upgrade process does not preserve or upgrade logical components of the SharePoint Server 2010 farm topology. After you perform the upgrade, you must manually re-create a topology as appropriate to meet the requirements of the organization.

3

Information on migration scripts to address the “other” feature configurations is pending, as of the time of this writing.


34

SharePoint 2013 Search Guidance Note that for customers migrating from SharePoint 2010 Search, this configuration migration will appear fairly complete. However, for customers using FAST Search For SharePoint, their ―migration‖ will result in degraded search functionality that results in SharePoint 2010 search functionality – omitting much of the ―newer‖ functionality of FAST Search For SharePoint 2010.

Planning Migration Planning for migration engagement adds the following steps to an overall implementation plan: 1. 2. 3. 4.

Analyse the existing use of the feature Document (in detail) the configured items for each migrate-able feature Document the user functionality implemented by each configured items for those f eatures that are not able to be migrated. Determine which features to migrate and which to re -implement from scratch in SharePoint 2013

For those features that you wish to migrate, follow the steps in the Appendix.


35

SharePoint 2013 Search Guidance Migration Paths The SharePoint 2013 migration process is not automated, and requires a combination of PowerShell scripting and manual processes. All ―migration‖ provided is to SharePoint 2013 search mode that we are calling ―SP 2010‖ mode, shown in the diagram below, prior to custom delivery of full SharePoint 2013 search functionality. The four main strategies are shown in the diagram below: 1. 2. 3. 4.

Simply re-implement search on SharePoint 2013 Implement search on SharePoint 2013, and publish the service to SharePoint 2010 for a gradual migration strategy. Migrate from SharePoint 2010 search. Migrate as much as possible from FAST Search For SharePoint 2010

The supported feature migration paths are indicated in the table below: ―From‖ ―To‖ SharePoint 2013 ―To‖ SharePoint 2013 (SharePoint 2010 mode) (SharePoint 2013 mode) SharePoint 2010 Search Yes In-direct SharePoint 2010 Search + Yes, with some loss of functionalityIn-direct FAST Search For SharePoint Requires custom scripting Requires custom scripting Other SharePoint versions No No Other versions of FAST SearchNo No All other migration paths are out of scope, including anything involving earlier versions of either SharePoint Search or FAST Search, and should be treated as ―new‖ search implementations for SharePoint 2013.

Feature Migration Recommendations This section provides recommendations on migration of FAST Search For SharePoint 2010 feature configurations to SharePoint 2013. They break down into these main categories:


36

SharePoint 2013 Search Guidance    

Data Source Integration Information Architecture Content Processing Application Integration

Data Source Integration Features Recommendation for feature

Feature

Script-ability proven?

migration

Content sources

PowerShell scripting

Yes

Crawl rules


Yes

Crawl schedules


Yes

BCS connector and its associated

(TBD - May require re-integration of data

configuration

source.)

Custom & 3rd party connectors and their associated configuration

(out of scope)

No No

Information Architecture Features Feature Custom crawled properties & custom crawled property categories Custom managed properties created in addition to the default set Custom mappings between crawled properties and managed properties

Recommendation for feature migration



Yes


Yes


Yes

Deep settings can be configured manually Deep settings of the managed properties

as soon as the managed properties have

No

been migrated Managed properties for people search

(TBD)

No

Content Processing Features Script-ability

Feature


Inclusion and exclusion dictionaries (entity

Should be recreated manually once running in full SharePoint

extraction, spellcheck)

2013 mode (likely PowerShell scripting)

proven?

Should be recreated manually once running in full SharePoint

Optional processing dictionaries

2013 mode (likely PowerShell scripting)

Pipeline extensibility

(TBD)

No No No

Application Integration Features Feature



Scopes


Partially scriptable

Federated locations


Yes

File types


Yes

Best Bets


Yes

Visual Best Bets are converted to Best Bets in Visual Best Bets

SharePoint 2010 , then you must “enhance” them as

Partial

Query Rules once in full SharePoint 2013 mode


37

SharePoint 2013 Search Guidance Synonyms tied to Best Bets and Visual Best Bets


Synonyms used for query

Synonyms can be enhanced manually as Query Rules

expansion

once running in full SharePoint 2013 mode

Yes No

Promotions/Demotions are not supported in “ SharePoint

2010” mode. (The general scripting idea is to export Promotions/Demotions

from FAST Search For SharePoint 2010 , and “manually”

No

add them into SharePoint 2013 as Query Rules, skipping “SharePoint 2010” mode)

Rank tuning Search results page

Consult TechNet for rank tuning guidance Consult TechNet documentation for guidance on new methods for front end development

No No

Migration Steps The basic steps in the migration process are: 1.

2.

3.

4.

If using FAST Search For SharePoint 2010 search a. Backup databases from the SQL Server 2008 database instance supporting FAST Search For SharePoint 2010 using documented backup procedures: i. Content SSA ii. Admin DB iii. Query SSA b. Restore the databases to a SQL Server 2012 database instance (or SQL Server 2008 R2 for SharePoint 2013) c. Merge the Content SSA, Query SSA and Admin SSA databases into a single Content SSA database i. This requires custom Powershell scripting d. Run the ―migration tool‖ against that Content SSA to create an intermediary search configuration database in ―SharePoint 2010 mode‖ i. This is a PowerShell cmdlet… ―restore Search SSA‖ provided by the SharePoint 2013 search product team ii. This also creates the Search SSA in SharePoint 2010 mode… e. Create crawled properties and & crawled property categories, and managed properties i. This requires a custom Powershell script If using SharePoint 2010 search a. Backup the Content SSA database out of the SQL Server 2008 database instance supporting FAST Search For SharePoint 2010 using documented backup procedures b. Restore the Content SSA database to a SQL Server 2012 database instance (or SQL Server 2008 R2 for SharePoint 2013) At this point, the SharePoint 2010 mode Search SSA has only legacy SharePoint search functionality. a. It does not contain ―advanced‖ (read: FAST Search For SharePoint 2010) functionality, so we (FAST) LOSE functionality. b. The search center DOES operate, but only with SP Search functionality (i.e. without FAST Search For SharePoint 2010 functionality) c. Synonyms are only one-way, and no longer affect the query. They only trigger Best Bets d. Visual Best Bets turn into ―regular‖ Best Bets e. Promotion & Demotion of documents does not function To convert from ―SharePoint 2010‖ mode to ―SharePoint 2013‖ mode, you have to run another Powershell cmdlet to convert the SharePoint Site a. Convert each SharePoint Site (CONTENT SITE!) from SharePoint 2010 mode to SharePoint 2013 mode i. This actually changes the underlying DB tables to support full SharePoint 2013 fun()…


38

SDPS - SharePoint 2013 Search Guidance.docx

Recommend Documents