FAST ESP

TM

The worlds most intelligent, secure, high-performance search platform

FAST ESP

FAST ESP is the worlds leading, most advanced search platform, delivering secure, relevant answers and insight to accelerate decisions, improve corporate performance, and increase prots.

The volume of data in a large organization doubles every year. Hundreds of disparate silos and the resulting information clutter can threaten the organizations ability to respond quickly to market and environment changes. FAST ESP liberates the organization and its external and internal users from information silos by delivering a secure, integrated point of access to all structured, unstructured, and rich media data. End users can solve business problems condently and intuitively by searching through any corpus of content inside and outside the organization and conducting on-the-y analysis and discovery. With FAST ESP organizations can make better decisions, improve operations, promote their assets, realize greater prots, and expect truly differentiating search solutions. Features FAST ESP is a complete enterprise search platform that offers key, game-changing capabilities to its customers and their users: Contextual InsightTM Contextual navigation Deep analysis Maximum transparency Extreme performance Congurability Contextual InsightTM Contextual Insight is the next generation for search relevancy, drastically improving precision without sacricing recall. Imagine the query, Which companies work with the iPod? Conventional search engines will return links to documents that include the terms company and iPod. Contextual Insight returns a list of names of companies that work with the iPod and for

each the sentence that identied the relationship. It answers the question. It is that simple. Contextual Insight is based on a core search innovation, introducing a semantic index that recognizes and retains the inherent structure in a document and scopes the search to the structural element level. Conventionally, the scope is the document, database record, or web page, but these are too ambiguously large. With Contextual Insight the scope can be a sentence or paragraph or, in general, any XML or structural element. Further, Contextual Insight annotates the scope with entity meta-data by extracting all the entities it can nd to provide the answers. Entities include such things as peoples names, phone numbers, geographic locations, and in our example, company names. FAST ESP supports about 40 entities and the list is growing. In fact there is a standard, open interface for dening your own. There is also the use of environmental context to modify the results. Geospatial location is familiar, especially with the mobile community, but context can be derived from other sources: patterns in the query log (e.g. music queries lead to ring tone requests), customer proles, click streams, search histories, to name a few. The natural language processing needs to be contextually aware as well. FAST ESP has added contextual support for the who, what, where, when, and why scenarios, and disambiguating the results automatically to provide direct answers even if the search made no explicit reference to them. Finally, wrapping all this together is FAST ESPs unique, adaptable relevancy model. FAST ESP provides a fully transparent and multi-dimensional relevance model that measures and ranks the result documents from several angles. The

Contextual InsightTM annotates the scope with entity metadata by extracting all the entities it can nd to provide the answers.

FAST ranking model is based on the following individually tunable key metrics: Freshness How fresh is the document compared to the time of the query? How well does the query match superior contexts (title, URL)? Is the document considered an authority for this query? How well does the content of this document match the query overall? What is the quality of the document? How important is it? How far away is the result from a geographic reference location?

Completeness

Deep analysis FAST ESP is the rst to integrate quantitative and qualitative analyses in one environment and design its architecture to permit the integration of external analytical mining engines. This new synergy makes the knowledge workforce more productive. For qualitative analysis, FAST ESP supports a series of advanced linguistic capabilities: Spell Checking Detects and corrects miss-spelled words and phrases Matches regular and irregular grammatical inections Detects names and phrases and either automatically rewrites the query or provides a search tip Enables approximate matching based on phonetic similarities Removes words that do not contribute to the querys meaning

Authority Statistics

Lemmatization

Quality

Phrase Detection

Geo-spatial

Phonetic Search Contextual navigation Effective exploration of content requires both extreme precision and means to produce data-driven Table-Of-Content views. Previously, successful navigation has been limited to global metadata. Contextual Navigation unlocks the semantic meaning of contextual metadata. Facts, relationships, and answers contained inside documents can be effectively visualized in the discovery interaction. FAST ESP supports 3 navigation types: Taxonomic navigation uses concepts and categories from a taxonomy as navigators. Documents are auto-categorized under these categories. Faceted browsing exposes database table elds as navigators. The navigators are the possible values as dened by the results of the previous search. Entity discovery extracts textual entities from the results of the previous search. Unlike taxonomies and facets, entity extraction draws its navigators directly from the results and is contextually aware.

Anti-Phrasing

Natural Language Recognizes constructs such as who, what, where, when, and why Thesaurus A custom synonym dictionary to dene your specic business terms

Quantitative analysis, or data mining, deals exclusively with numbers, specically their distribution and statistical trends. FAST ESP supports simple calculations like average and total, but the real logic is left to those who do it well, analytical engines and mathematical libraries such as SPSS and SAS. Our pipeline architecture makes it easy to embed external analytical engines such as these.

FAST ESP

Maximum transparency FAST ESP is a vendor agnostic, globally aware search platform. It supports a wide range of platforms, including Windows, Red Hat Enterprise Linux, SUSE Linux Enterprise, and Solaris. FAST ESP supports 77 distinct languages (and multiple encodings for each) for search and sorting, and advanced linguistics for 29 of them. It supports Java, .NET, C++, and Web Services for its programmatic interfaces. And it has an extensive connector library for getting intelligent access to source content: FAST Web Crawler Public and secure Websites File Traverser Local le system directories (over 370 different le types) Native XML indexing, custom mapping of XML formats to elded search FAST offers a wide range of connectors including to: MS Sharepoint 2003 MS Sharepoint 2007 MS CMS MS Exchange StarTeam

Extreme performance FAST ESP is quite simply the most efciently scalable search platform today. Here are some statistics in current installations: Largest dataset Largest no. docs Highest query rate Highest compression Fastest update rate Most analytics 40 petabytes 10 billion documents 2,000 queries/second 250M documents/ machine 500 updates/second 200 parameters on real-time trading data
Effective exploration of content requires both extreme precision and means to produce data-driven Table-OfContent views.

XML Content

Connectors

FAST ESP is designed to run in a low-cost, commodity hardware grid-like environment. If query rates run high, the system distributes the queries across several rows of search nodes. If the index gets large, the system partitions the index into several columns of search nodes. Fault-tolerance is an important requirement for systems that run continuously and where avoiding downtime is of critical importance. FAST ESP provides granular fault-tolerance for feeding, indexing, and search, for example by deploying multiple rows of search engines. Each row includes an identical copy of the given index partition. In case of a node failure, the faulttolerance protocol ensures that the recovered node does not accept any new queries before synchronizing its index with another node within the same columns. Congurability FAST ESP supports tailoring at both product and programmatic levels. At the product level, you can create collections and search proles. A collection is a group of documents based on content source, type or authorization level. The collections may also be used to narrow the scope of a search or enable multiple search applications within one FAST ESP installation. Think of it as a logical index inside the larger corpus.

Oracle JDBC Lotus Notes Documentum Meridio CaliberRM

FAST partners also provide a number of additional connectors including: Kapow Vignette FileNet SAP EP BEA Weblogic OpenText LiveLink InterwovenTeamSite Communique SAP PLM IBM WebShere

FAST ESP has an image matching component for recognizing and matching image patterns. Audio Miner and Video Miner provide near real-time multi-lingual audio to text conversion, timbre recognition, and SMPTE time-code alignment to jump to any specic frame in a video. Also included is an intuitive management tool that congures streams and auto-generates thumbnails.

Content refinement
Index Query server

Content Analysis
Query processing

Collections

XML

Content processing Result processing Alert server

Content API

FAST ESP enables any type of information to be accessed through two core engines: the FAST Search Engine and the FAST Alerting Engine.

Search profiles

Result analysis Application API Search administration & management Administration Studio Search Business Center FAST Home

Figure 1 Search Engine architecture

A search prole enables a single installation to support multiple views of its indexed content to accommodate multiple purposes or user roles. Even though search applications may share common content in the index, they can have their own pre-dened common search settings and attributes. One Search Prole represents one specic usage of FAST ESP. FAST ESP is a true platform in that it can be extended at different points while it is running. The software is built with modularity in mind with well-dened frameworks and APIs for customer integration (see Figure 1): Document processing pipeline for adding custom pre-index content renement stages Query processing pipeline for adding custom query processing stages Results processing pipeline for adding custom results processing stages Connector API for developing your own source connector or push application Search API for developing your own custom front-end connection to FAST ESP Application API for building custom application modules or redirecting FAST ESPs administration to another program

As is true for any serious enterprise search environment, FAST ESP supports a rigorous security model, protecting your IT department and securing your content. FAST ESP applies security measures to protect your servers, the data hosted on the servers, and the ability to access content on your content repositories. Security controls for the search application domain ensure that end-users are permitted to see only the search results they are entitled to. Collection level security denes different authorization levels per collection. End users have access to the set of collections mapping to their authentication level. Document level security veries the ACL for each document is in the search result. Users will only have access to documents that correspond to their authentication. FAST ESP recognizes the security protocols for its connectors and standard models such as Microsofts Active Directory. Of course, the ability to tune the system at all is in part a determination of how easy it is to use the product. The Search Business Center (SBC) represents a major step towards holistic search management. The SBC workow supports both

Query API

FAST ESP

action-oriented and functionally-oriented entry points. You choose. In the SBC, non-technical users can dene search proles from scratch or from copies of existing search proles. They can preview their changes before deploying to the production system. They get access only to the search proles, content collections, and functions matching their access rights. The SBC also monitors the relevancy of the system, based on out-of-the box or user dened tests, and hence provides direct feedback as basis for further optimization of the installed search proles. Architecture FAST ESP is built on a distributed easy-todeploy-and-maintain, web-based architecture. FAST ESP enables any type of information to be accessed through two core engines: the FAST Search Engine and the FAST Alerting Engine (see Figure 1). Data is either pushed to the system using the ESP Content APIs or retrieved from external sources using the predened and/or custom content connectors. Incoming content is fed through a document processing subsystem with a number of processors that perform tasks such as document conversion, automatic information extraction and classication, document summary generation, and CJK language tokenization. A similar, exible query and result processing subsystem applies natural language query mining, spell-checking, and other relevance enhancements to the queries. The search API on the query side enables easy integration with customer applications. Both these processing subsystems are open frameworks, designed to be modied and customized by content owners and rapidly integrated with external data sources. The search engine performs core indexing, matching and relevancy tasks based on proven, high-capacity, linearly scalable search technology. The engine can be distributed across multiple servers for optimal query and data volume performance.

The alerting engine is based on proven, highcapacity real-time lter technology. It takes multiple sets of lter conditions, referred to as alert queries, and matches them to an incoming stream of fresh data in the form of events such as news articles, stock quotes or other documents processed by FAST ESP. Alerts or ltered content streams are provided in real-time and converted to appropriate applications or end-user devices via XML. Search Management and Administration in FAST ESP are run via a set of GUI based tools. FAST Home is a personalized portal for all users working with FAST ESP. FAST Administration Studio gives access to the low level conguration and monitoring options of ESP. The SBC is designed for non-technical users to manage the search experience and setup and tuning of different applications realized by an ESP installation. These tools are built by using the application layer in ESP which is exposed in an Application API and can also be used to develop custom applications based on FAST ESP. Conclusion The description on these pages is merely an introduction into the wonders that are FAST ESP. There is so much more to explore, and we invite you to do so at your earliest convenience.
FAST ESP is a true platform in that it can be extended at different points while it is running.

FAST, A Microsoft Subsidiary www.fastsearch.com info@fastsearch.com Regional Headquarters The Americas +1 781 304 2400 Europe, Middle East & Africa (EMEA) +47 23 01 12 00 Japan +81 3 5511 4343 Asia Pacic +612 9929 7725

2008 FAST, A Microsoft Subsidiary. All rights reserved. FAST, FAST ESP, and all other related logos and product names are either registered trademarks or trademarks of FAST, A Microsoft Subsidiary in Norway, the United States and/or other countries. All other company, product, and service names are the property of their respective holders and may be registered trademarks or trademarks in the United States and/or other countries.

PLA.001.D.01.011706