Getting Started with Endeca Latitude

Getting Started with Endeca Latitude 2 Student Guide

(Version 2.2.1)

D75432GC10 Edition 1.0 February 2012 D75730

Copyright © 2000, 2011, 2011, Oracle and/or its affiliates. All rights rights reserved. Disclaimer This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way. Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization of Oracle. The information contained in this document is subject to change without notice. If you find any problems in the document, please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not warranted to be error-free. This training manual may include references to materials, offerings, or products that were previously offered by Endeca Technologies, Inc. Certain materials, offerings, services, or products may no longer be offered or provided. Oracle and its affiliates cannot be held responsible for any such references should they appear in the text provided. Restricted Rights Notice If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United States Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract. Trademark Notice Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Copyright © 2000, 2011, 2011, Oracle and/or its affiliates. All rights rights reserved. Disclaimer This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way. Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display, perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization of Oracle. The information contained in this document is subject to change without notice. If you find any problems in the document, please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not warranted to be error-free. This training manual may include references to materials, offerings, or products that were previously offered by Endeca Technologies, Inc. Certain materials, offerings, services, or products may no longer be offered or provided. Oracle and its affiliates cannot be held responsible for any such references should they appear in the text provided. Restricted Rights Notice If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United States Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS The U.S. Government’s rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract. Trademark Notice Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Module 1 – Latitude 2 Core Concepts___________________________ ____________________________________ _________ 1 Lesson 1 – Introduction Introduction to Latitude Latitude ___________________________ ___________________________________ ________ 1.1 Lesson 2 – Data Ingest __________________________ _____________________________________________ ___________________ 1.2 Lesson 3 – Capabilities Overview_________________________ _____________________________________ ____________ 1.3 Lesson 4 – Latitude Studio Overview ______________________________ __________________________________ ____ 1.4 Lesson 5 - Data Exploration in a Latitude Application ______________ _______ _____________ ______ 1.5 Module 2 – Basic Data Design ______________________ _________________________________________ ___________________ 2 Lesson 1 – Project Basics with Latitude _______________________________ 2.1 Lesson 2 – Working with Latitude Latitude Studio ____________________________ ______________________________ __ 2.2 Lesson 3 – Data Modeling and Joining Data ___________________________ ____________________________ _ 2.3 Module 3 – Search and Navigation Configuration Enhancements _____________ _________ ____ 3 Lesson 1 – Introduction Introduction to Search _______________________ ____________________________________ _____________ 3.1 Lesson 2 – Enhancing Navigation Navigation ______________________ ____________________________________ ______________ 3.2 Module 4 – Advanced Topics____________________ ___________________________________________ _______________________ 4 Lesson 1 – Data Ingest and Modeling Overview ______________ _______ _______________ ___________ ___ 4.1 Lesson 2 – Integrating with Other ETL Tooling __________________________ 4.2 Lesson 3 – Security ___________________________ _______________________________________________ ____________________ 4.3 4.3 Lesson 4 – Text Enrichment in Latitude Latitude ________________ ________ ________________ _______________ _______ 4.4 Lesson 5 – Latitude 2 Deployment Features ____________________________ 4.5 Lesson 6 – Next Steps for Enablement Enablement ________________________________ ________________________________ 4.6 Module 5 – Analytics ________________________ _________________________________________________ _________________________ 5 Lesson 1 – Introduction to Endeca Analytics ____________________________ 5.1 Lesson 2 – Analytics Workshop ____________________________ ______________________________________ __________ 5.2

Module 1 – Latitude 2 Core Concepts

Student Guide Getting Started with Latitude 2

Lesson Time: ~45 ~45 min. min. Lab Time: ~30 min.

Lesson 1 – Introduction to Latitude

Copyright©2011 Endeca Technologies, Inc. All rights reserved. Proprietary and confidential.

1.1.1





1.1.2





1.1.3



Endeca provides agile information management management software software that guides people to better decisions. decisions. We We do this with two products: Latitude for agile BI and InFront fo r agile commerce. One improves daily decisions for employees, the other for consumers. Both products are based on the MDEX E ngine, a hybrid search-analytical database that empowers IT to cope with the increasing diversity and chang e inherent in delivering information info rmation in today's enterprises. More More on the MDEX later… We provide the new kind of information management companies need for better decision-making: decision-making: agility to handle diverse and changing cha nging data, to answer diverse and changing chang ing questions asked by people with no training in the underlying technology. Endeca Company Facts:

• • •

Based Based in in Cambr Cambridg idge, e, MA wit with h global global sal sales/ es/ser servi vice ces s offic offices es $100 $100M M+ annu annual al reven evenue ue 600+ cu customers



1.1.4



Endeca takes takes a unique approach to agile BI based on three principles. principles. No data left behind :

We bring together structured structured data and unstructured unstructured content, whether inside inside the company or out. Our core technological technological innovation, innovation, the the MDEX engine, enables you to easily combine combine all data needed for true insight. Latitude also also offer off er analytic analytic apps to help you solve problems previously previously unthinkable. Additionally Additionally,, Latitude is the only BI technology that unifies structured content & social media for re today’s toda y’s decisions. Guidance in daily decisions:

We make software people actually actually want to use – software that guides guides them to better decisions than they would reach with any alternative, alterna tive, through the industry’s only patented Guided Navigation. Navigation. Latitude offers user interfaces that that tell people about the data before they interact with it, extending discovery discovery for every user. Agile delivery :

Using agile delivery principles, principles, Endeca has developed the Slipstream iterative iterative release methodology. This unique methodology enables IT to be both more responsive responsive to the business and more cost-effective. Because the engine at the heart of our solution is so different from traditional BI technologies, technologie s, IT IT can quickly load disparate disp arate data, get a look at it in the application, and then collaborate with the business busine ss to refine it. This helps IT discover the true requiremen ts of the project more quickly, and in turn leads lea ds to faster user adoption.



1.1.5



Latitude is powered by the patented pate nted Endeca MDEX engine, the industry’s only solution that is a cross between a search engine and an analytical database. As a result of the internal data architecture, each record has its own set of attributes based on the original source data as well as any metadata added by Endeca during data integration. This data architecture simplifies modeling requirements and ensures new content can easily be ad ded in the future. Furthermore, the work of creating conformed dimensions and facts, which is typically done when setting up multiple multiple data marts, is not required here. As an added benefit, the the engine is able to handle your unstructured content in the same flexible manne r. The MDEX engine operates ope rates as a memory leveraged model, optimizing the use of main memory, caching, and storage resulting in faster faste r responses and flexibility to deal dea l with large volumes of data. data . Additionally, you can load your source data as is – meaning you practice “load, expose, refine” instead of “extract, transform, and load”.



1.1.6



There are several advantages to being able to combine data as-is. Diverse Data:

Here we can see records from four very different source systems combined together in the MDEX Engine. In this example, all the sources are considered ‘structured’. We are not conforming the data to fit a target schema; we are simply stacking records in the ir native format on top of each other. This enables you to get data into the hands of the end-users rapidly, without having to wait for a ‘perfect’ consolidated data model. Similarly, as additiona l source systems or attributes are indexed over time, Endeca can easily accommodat e the new or changed data without needing any target schema redesign. Unique Attributes:

Every source system will contain its own unique set of attributes. The MDEX Engine summarizes and allows navigation on all of these unique attributes without compromising their original structure so that it fits into a neat box. Jagged Data:

The MDEX Engine allows information to stay naturally ‘jagged’. Every record may have its own set of unique attributes; even if the records are from the same source system. Accommodating jagged data eliminates issues caused by data sparseness. For example, the MDEX Engine negates the need to create expensive ‘dummy’ fields tha t are not relevant to the record on which they are attached (e.g., flavor=’NO FLAVOR’ for a pair of shoe s).



1.1.7



Global Attributes :

Information sources will also contain common or ‘globa l’ attributes across many records and source systems (e.g. common part numbers, employees, regions an d territories, suppliers, etc). Whenever a record contains an attribute that is shared with anoth er record or system a relationship is identified. Records in the MDEX Engine are related whenever a common attribute exists. This means that relationships exist at the attribute level – not at the table or record level - which is the finest granularity possible. End-users interacting with the Endeca application can use any starting point to enter the data, even a basic search term. The user is not forced down a pre-defined analysis path as with traditional BI ‘drill down’ technologies (OLAP, ROLAP, etc.), and as a result they constantly discover new insights in the data as they interact with it. Not on ly can they drill down, they can drill anywhere and pivot across any type of information in any source system wherever a relationship exists across common attributes in the data. This offers far more flexibility than relationships defined in advance in a specific data model.



1.1.8



It is not enough that IT be convinced Endeca is a great compliment to the BI ecosystem. The ultimate test lies with the business user and how successfully they are utilizing the solution to make effective and actionable business decisions every day. Endeca applications help users not only find what they need, but analyze what they found, so they can better understand their options. The Endeca user experience brings together: Search – Easy access to all related data with state-of-the-art search. Navigable Unstructured Content – The MDEX engine makes documents and web-content part of the searchable decision data – not just reference information. Guided Navigation – Our patented Guided Navigation drives users to ask precise questions, enabling them to better express what they’re really looking for. Guided Navigation arms users with the ability to navigate your unstructured content – encouraging them to discover unanticipated, and oftentimes better answers. Interactive Visualizations – Users can point and click on visualizations to learn more – without the need to formulate a query or traverse a p repared hierarchy of data. All in all, Endeca’s consumer ease-of-use combined with all of the data users need, enables them to reach high confidence answers; answers that can only be derived by bringing together otherwise siloed information into a single pane of glass.



1.1.9



No matter how good any technology may be, it has to be in the hands of your users in order to solve the business problem and create value. That’s why we have focused a great deal of our attention on fast deployments and time-to-value. Our Iterative Release Methodology, Slipstream, is designed to put production applications – not conference room pilots – in the hands of your users in as little as 1012 weeks. Slipstream is based on the premise of delivering an operating application every 2 to 3 weeks. In order to do that we have spent a great deal of effort building out our technology and delivery processes to speed the delivery of configured applications for your users. Specifically, the manner in which we handle data in our MDEX engine simplifies the process of loading data and supports the incremental addition of new or changed data sources. Similarly, because it is impossible to define all requirements upfront we developed the Latitude Studio to speed the initial configuration and deployment as well as support a collaborative improvement process between users and IT. Endeca’s Iterative Release Methodology allows development to start before requirements are finalized, getting critical information into the hands of the busine ss early. As requirements are realized throughout the project lifecycle the application can be quickly changed to accommodate unforeseen benefits



1.1.10



MDEX Engine:

At the heart of Endeca Latitude is MDEX Engine™ technology, the industry’s only hybrid search-analytical database designed to bring together all of the information required for making critical business decisions, regardless of its original type, format, or source. This allows users to explore data and content in an unconstrained and impromptu manner, rapidly addressing new questions as they arise. Bringing together diverse datasets without complex modeling and integration, the engine conducts search, navigation, and analytics. With support for ITmanaged data governance and data security as well as ind ustry standards such as XQuery, XML, and SOA, the MDEX Engine is designed to fit easily into existing enterprise architectures. The Latitude Information Integration Suite:

Endeca Latitude supports an agile approach to deployment that reduces the data modeling and integration inherent in traditional software deployments. It provides powerful integration capabilities through the Latitude Data Integrator (LDI), the Endeca Content Acquisition System (CAS), and various custom MDEX connectors. It provides an extensible framework for ingest of a broad range of unstructured content, and through support for best-of-breed data adapters and ETL tools. These tools enable rapid integration of both structured data and unstructured content and the flexibility to easily add new data sources.



1.1.11



Latitude Studio:

The user interface configuration part of Endeca Latitud e provides a library of components that deliver advanced search capabilities, Endeca’s patented Guided Navigation® experience, rich visualizations, and analytics for the rapid configuration of Latitude applications. This enables IT organizations to respond to changing business needs by re-designing reports and dashboards in minutes, incorporating diverse data sources in days, and deploying entirely new applications in just a few weeks. How Latitude Works:

1. Data is loaded into the Latitude Information Integration Suite; LDI, or another connector, then loads data into the MDEX Engine 2. Records received from the LDI are indexed in the MDEX Engine, which then receives and processes query requests from Latitude Studio 3. Latitude Studio application contains components, which you use to query the MDEX Engine and manipulate the query results.



1.1.12

12





1.1.13





1.1.14



Features such as refinement hierarchy, dynamic refinement ranking, grouping, precedence rules all help you organize your information. Navigation is contextual to the characteristics of the search results. The goal of navigatio n is to provide refinements to offer guidance to your next steps. Because Endeca enables all of the attributes in your data set to be use for refinement, Endeca offers way in which you can organize your attribu tes and attribute values for a better user experience.



1.1.15





1.1.16





1.1.17





1.1.18





1.1.19





1.1.20



Definition:

Part of the Latitude Information Integration Suite, Endeca’s Content Acquisition System combines a robust and extensible framework for retrieving and manip ulating semi-structured content with a broad library of data adapters and text enrichment modules. Features:

•

Extensible framework for data source connectivity and data manipulation

•

64-bit, multi-threaded platform available on Windows and Linux

•

Out-of-the-box connectivity to Documentum, FileNet, Lotus Notes, Sharepoint, Interwoven, and OpenText

•

Crawlers for file systems and the web; includes data store for capturing crawl output

•

Text and metadata extraction for 500 file formats

Benefits:

•

Easily integrate data from a variety of unstructured data repositories

•

Best-of-breed partner technologies available for text enrichment

•

Low cost introduction of new data sources and manipulators through extensibility SDK



1.1.21



The Latitude 2.1 Information Integration Suite features a new ETL tool, the La titude Data Integrator. Definition: Part of the Latitude Information Integration Suite, the Latitude Data Integrator (LDI) is a

high-performance data integration platform that lets you extract source records from a variety of source types (from flat files to databases) and send that data to the Data Ingest W eb Service. The Data Ingest Web Service in turn loads the records into the MDEX Engine. Features: Designer with powerful graphical interface, Connectors to leading enterprise applications, A

comprehensive enrichment library, Integrations with industry-leading ETL tools for data ingest, A direct connection to the MDEX Engine, Best-of-breed crawlers for file systems and web sites Benefits: The right tool for the data at hand, Easily unify multiple data sources, The Designer

enables quick and easy way to graphically lay out the most complex graphs using drag an d drop functionality For more information: Endeca has produced the Latitude Data Integrator Guide , available for

download on EDeN. For further reference, an extensive set of Clover documentatio n is available at http://www.cloveretl.com/resources. From that Web site, you can access the Quickstart Guide, the Designer User's Guide, and the Server Reference Manual. Other resources, such as a users forum, can also be accessed from this Web page. Lesson 1 – Introduction to Latitude


1.1.22



Definition:

Other ETL connectors will become available as Latitude evolves. Features:

•

Provides custom transformations to build the records

•

Provides custom targets that reassemble single-assignment records into complete Endeca records and loads the complete records into an MDEX Engine

•

Provides a custom connection object that specifies the host, port, and other configuration information of the server running the MDEX Engine



1.1.23



Definition:

The core data management solution of Latitude is the MDEX Engine technology, a hybrid search-analytical database that is highly-scalable and column-oriented. The MDEX Engine provides support for guided exploration, search, and analysis on any kind of information; structured, semi-structured or unstructured, from inside the firm or f rom external sources. Features:

•

Unification of structured, semi-structured and unstructured data through "No data left behind" approach

•

Uses advanced data structures and algorithms that allow real-time responses to client requests

Benefits

•

Enablement of users to explore data and content in an unconstrained and impromptu manner to rapidly address new questions

•

Frees IT from the burdens of traditional data modeling

•

Support of the broad exploration and analysis needs of business users



1.1.24



Definition:

Latitude Studio is a highly-interactive, component-based environment for building enterpriseclass analytic applications powered by MDEX Engine technology. It is built on robust webbased infrastructure technologies that enable building analytic applications delivered through the use of popular Web browsers. Features:

•

Components-based analytics and visualization product

•

Assortment of templates and themes for flexible styling

•

Robust library of components (e.g., Data Explorer, Guided Navigation, Charts)

•

Granular role-based authorizations

•

Multi-language support

Benefits:

•

Support of decision-making through quick access to data

•

Enables rapid time-to-value through minimal configuration effort

•

Support of agile development methodology

•

Seamless upgrades of applications as new components and features are released



1.1.25





1.1.26





1.1.27





1.1.28





1.1.29





1.1.30





1.1.31





1.1.32





1.1.33



Lesson Time: ~25 min.

Lesson 2 – Data Ingest


1.2.1





1.2.2





1.2.3



The Latitude Information Integration Suite features a comprehensive set of data integration options. Included in the Suite are the Latitude Data Integrator , an out-of-the-box ETL product; the Content Acquisition System; and connectors for industry-standard ETL products. The Endeca Latitude connectors are used to load data into the MDEX Engine. These connectors enable you to extract and load both structured and unstructured data sources into the MDEX. In addition, open Web service APIs for data and configuration integration make it possible to integrate more varied information into the MDEX Engine.



1.2.4



Data integration is a continual challenge f or enterprises. To address these concerns, information access applications need to be based on technology that both provides out-of-the-box connectors to common enterprise systems and supports connecting to novel systems through simple extension points. Endeca's Content Acquisition System, consists of a mixture of standards-based connectivity, packaged adapters, and an extensible framework for creating and configuring adapters to new systems. Key benefits of the Endeca Content Acquisition System include: •

Consolidated view of information from diverse sources to support analysis

•

Low-risk, rapid deployments using pre-packaged functionality

•

Support for new data sources as business requirements change

•

Low maintenance overhead

•

Information security



1.2.5



The Latitude Data Integrator is a high-performance data integration platform that enables you extract source records from a variety of source types (from flat files to databases) and load the records into the MDEX Engine. From a high-level, the LDI consists of the Designer, Endeca-developed connectors and (optionally) a Server.



1.2.6



As previously mentioned, Latitude connectors are part of the palette, or available design elements, in the LDI Designer . The Designer is where you create the graphs for loading your data. A graph is a pipeline of components that processes the data. The simplest graph has one Reader component to read in the source data and one of the Latitude connectors to write (send) the data to the MDEX Engine. An edge, the line that connects component ports, carries data between components. More complex graphs will use additional components, such as the Transformer and the Joiner components. The Designer offers a powerful graphical interface, providing an easy way to visually lay out even the most complex graphs. The Designer is also simple to use--you are able to drag and drop the components from the Palette and then configure them by clicking on the component icon. The Designer tool is covered in detail later in this course.



1.2.7



The Endeca-developed Latitude connectors have been specifically engineered to work with the MDEX Engine's Data Ingest Web Service and Bulk Load Interface. These connectors are incorporated into the Designer Palette, so that you can easily drag them into your graphs. These connectors can be found in the Palette under Endeca, and include: •

Add KVP (Key Value Pairs)

•

Add Managed Values (Hierarchy definition)

•

Add/Update Records

•

Bulk Add/Replace Records

•

Delete Data

•

Export Config

•

Import Config

•

Reset MDEX

•

Transaction RunGraph

•

ReadRecordStore (only available through Endeca Consulting)

Each of these connectors is discussed in detail later in this course (with exception of ReadRecordStore).



1.2.8



Because the Server is not a mandatory component for loading data into the MDEX Engine, it is not documented in detail in this course. For information on the setup and use of the Latitude Data Integrator Server, see the Data Integrator Server pdf.



1.2.9





1.2.10



The Latitude Web services provide interfaces to the MDEX Engine. The Data Ingest Web service provides an interface to ETL tools and a means of loading, updating, and deleting records. The Configuration Web service allows you to update the schema and configuration. These two Web services are public, enabling you to program custom connectors against them. There are also two private Web services: ‘Private’ Web services include: the Conversation Web service, which provides the primary means of querying data in the MDEX Engine; and the Administration Web service, which is used by IT engineers and administrators to integrate the MDEX Engine server and its reporting with third-party IT tools.



1.2.11



Latitude provides an open data ingest API, the Data Ingest Web service, to allow you t o use whichever ETL tool is most appropriate for the job. For structured data, you can use your ETL tool of choice, or use our out-of-box Latitude Data Integrator. For unstructured data, use CAS and combine it with your structured data in ETL.



1.2.12



Delimited: • Is the most frequently used format because it is easy to create and maintain • First row identifies field names • Each subsequent row represents a record • Can use any field and row delimiters XML: • •

Requires any non-Endeca XML data to be transformed with XSLT Is useful if the source data is already in an XML format

Fixed Width: • Is much less frequently used because it is difficult to maintain and can cause values to be truncated • Requires you to specify the field names, starting position, and field widths within the ITL configuration



1.2.13





1.2.14



The Content Acquisition System supports the f ollowing content management systems: • Documentum • eRoom • FileNet P8 • FileNet Document and Image Services • Hummingbird DM • JSR-170 compliant repositories • OpenText LiveLink • Lotus Notes • Microsoft Sharepoint 2003 & 2007 • Interwoven TeamSite Document conversion is provided by a licensed version of Oracle OutsideIn which can convert over 500 file formats. For data sources that are not supported by CAS, the Content Adapter Development Kit (CADK) provides the capability to create custom adapters.



1.2.15



The data model in the MDEX Engine consists of records and attributes. •

Records are the fundamental units of data.

•

Attributes are the fundamental units of the schema.

For each attribute, a record may be assigned one or more attribute values. An attribute is the basic unit of a record schema. Attributes describe records in the MDEX Engine. For a data record, an attribute provides information about that record. For example, for a list of books, the Author attribute contains the author of the book. For an attribute schema record, an attribute provides configuration information. For example, each attribute is itself represented by an attribute schema record. Each attribute in that attribute schema record controls an aspect of the attribute, such as the format of the attribute values and whether the attribute can be used for searches. The term attribute collectively ref ers to both standard attributes and managed attributes.



1.2.16



The term attribute collectively ref ers to both standard attributes and managed attributes. Standard and managed attributes are both described by system records. The system records that describe standard attributes are known as Property Description Records (PDRs), where the system records that describe managed attributes are known as Property Description Records (PDRs) and Dimension Description Records (DDRs). These will be discussed in detail later in the course. Each standard attribute, includes the following attributes: • Name and type • Display name • Configuration parameters (whether an attribute is searchable) • Navigability settings (whether to show record counts for available refinements, whether to enable multi-select, and how to sort ref inements) A managed attribute has the same name as the associated standard attribute, but it is used to enable the creation of hierarchical standard attribute values. Each managed attribute includes the following schema attributes: • Name • Enable refinements (True or False) • Enable value search (True or False) • Enable record search (True or False) Lesson 2 – Data Ingest


1.2.17





1.2.18





1.2.19





1.2.20





1.2.21





1.2.22



Each record is a collection of attribute/value pairs, with no segmentation into tables and no overarching schema. Each record is self-describing, representing its own, possibly unique schema that accommodates both multi-valued and unstructured fields.



1.2.23



•

String: represents character strings

•

Numeric – supports representation of three numeric data types: •

Integer – represents a 32-bit signed integer

•

Long – represents a 64-bit signed integer

•

Double – represents a floating point value, or a values that can be specified in a decimalpoint format (such as 20.0) or in a scientific notation format using "e" or "E"

•

Geocode – represents latitude and longitude pairs, used for geospatial filtering and sorting

•

Boolean – used for tracking true/false conditions; 1 = true and 0= false

•

dateTime – represents the year, month, day, hour, minute, and seconds of a time point, with the optional specification of fractional seconds. You can specify a dateTime value as either a universal (UTC) date time or as a local time plus a UTC time zone offset. Note that specifying just a local time is not supported.

•

Time – represents an instant of time that recurs every day; the hour and minutes of an instance of time, with the optional specification of fractional seconds. A time zone is not allowed as part of the time representation.

•

Duration – represents a duration of the days, hours, and minutes of an instance of time. A time zone is not allowed as part of the time representation.



1.2.24





1.2.25





1.2.26





1.2.27





1.2.28





1.2.29




Lesson 3 – Capabilities Overview


1.3.1





1.3.2





1.3.3





1.3.4



To configure settings related to Record Search, you edit the recsearch_config index configuration document and send it to the MDEX Engine through the Latitude Data Integrator. This document is covered in detail later in the course.



1.3.5



The search interface is a named group of at least on e attribute or a set of attributes against which a search term(s) is applied. Search Interfaces utilize match modes, which allow you to specify whether you want to match on all, some, or at least one search terms entered by the user. There is also a Boolean match mode that allows you to be more explicit with your search (and, or, not, etc.), as well as numerous other settings that can be applied to record sea rch results and the type-ahead feature in the Search component. Search Interfaces and Match Modes are discussed in detail later in this course.



1.3.6



The thesaurus feature allows the system to return matches for related concepts to words or phrases contained in user queries. This is covered in detail later in the course.



1.3.7



The stemming feature broadens search results to include word roots and word de rivations. Stemming is enabled in the MDEX Engine by default and is available only for English. Stemming is intended to allow words with a common root form (such as the singular a nd plural forms of nouns) to be considered interchangeable in search operations.



1.3.8



Spelling correction is a closely-related feature to stemming and thesaurus functionality, because spelling auto-correction provides an additional mechanism for computing alternate versions of the user query.



1.3.9



The snippeting feature provides the ability to return an excerpt from a record — called a snippet — to an application user who performs a record search query. A snippet contains the search terms that the user provided along with a portion of the term’s surrounding content to provide context. A front-end application in Latitud e Studio displays these snippets on the record list page of a query’s results. With the added context, users can more quickly choose the individual records they are interested in. You enable snippeting on individual members (fields) in a search interf ace that typically have many lines of content. You can also enable snippeting on a per-query basis.



1.3.10



‘Stacked’, as it pertains to relevancy ranking, means: •

The first module applied has the greatest impact on result order •

Each record receives a score

•

Records are displayed in descending score order

•

If the first module results in the same score for multiple records, a second module can be applied to break ties

•

Each subsequent module is treated as a tie-breaker to the module preceding it

You can create and control relevance ranking for both record search and value search at a systemdefault level. You can also apply record search relevance ranking as you are creating a search interface, or afterwards.



1.3.11



In this example, the value search performs a wildcard attribute search returning all valu es containing the string entered so far, ‘pro’. Thesaurus entries are NOT applied when a user selects an attribute or attribute value that is of fered via the type-ahead functionality because they are making an attribute refinement. In this sense, it functions in a similar manner as Guided Navigation, assisting the user in their exploration of the data. By default, all string attributes are value searches.



1.3.12





1.3.13



The Range Filter allows business analysts using Latitude Studio to create a set of filters, where each filter is a specific attribute. End users can then use these filters to ref ine the displayed data to only include records that have attribute values within a specified range.



1.3.14



Geographic Filters enable users to filter within a geographic area. The map component allows a user to filter geographically by clicking and dragging to select the desired region.



1.3.15





1.3.16



Features such as refinement hierarchy, dynamic refinement ranking, grouping, precedence rules all help you organize your information. Navigation is contextual to the characteristics of the search results. The goal of navigatio n is to provide refinements to offer guidance to your next steps. Because Endeca enables all of the attributes in your data set to be use for refinement, Endeca offers way in which you can organize your attribu tes and attribute values for a better user experience.



1.3.17



It is recommended that Latitude applications be initially config ured with all attributes as Multi-OR. As your users work with the data, you can refine this. The other configuration for attributes is Mulit-AND.



1.3.18



Standard Attribute:

• Only a single refinement can be chosen from a standard attribute • Once a refinement has been chosen, that attribute is no longer included in the response object Multi-OR Attribute: • Multiple refinements can be chosen from a Multi-OR attribute • As more refinements are chosen, the result set grows (OR filter) • Once a refinement has been chosen, that attribute remains in the response object • The refinements that are returned now indicate valid next steps given every other specified filter excluding selections within this attribute Multi-AND Attribute: • Multiple refinements can be chosen from a Multi-AND attribute. Only attributes with multiple assignments per record are logical for Multi-AND. • As more refinements are chosen, the result set narrows (AND filter) • Once an refinement has been chosen, that attribute remains in the response object. • The refinements that are returned now indicate records with additional assignments exist.



1.3.19



The sorting functionality allows the user to define the order of Endeca records returned with each query. Record sorting only affects the order of records. It does not aff ect the ordering of attributes or attribute values that are returned for query refinemen t. Note that all attributes are automatically enabled for record sorting when they are created. Therefore, no attribute configuration is required for sorting.



1.3.20



Dynamic Refinement Ranking prevents the page size problems that large lists of refinements can create.



1.3.21





1.3.22





1.3.23





1.3.24



When specifying a Dimension Value as the Source for a Precedence Rule, the Target Dimension will become visible only after the specified DimVal of the S ource Dimension has been refined on. In the above example, the Resolution dimension will only become visible when you select Digital Cameras from the list of Categories. Whereas the Processing Speed dimension will only become visible when you select Computers from the list of Categories. Alternative Use Case:

Another great use case for this type of scenario is when you want to use different "price range" dimensions for different categories of products: You might setup a different set of price range dimension values for phones ($75-$200, $201-$300, $301-$400) then you would use for televisions ($75-$499, $500-$749, $750- $1000, $1000-$2499, $2500). You could setup a precedence rule to show the "Phone Price Ranges" dimension when the user browses on "Phones" as the category and setup a 2 nd precedence rule to show the "tv Price Ranges" dimension when the user browses on "Televisions".



1.3.25





1.3.26



Latitude Studio includes out of the box support for analytics, including integrated support for analytic queries within any standard or custom component. This capability ena bles you to directly leverage high performance MDEX query capabilities.



1.3.27





1.3.28



The Endeca MDEX Engine provides a platform for constructing applications that enable data consumers to quickly and efficiently locate information of interest in large dat abases, using the Guided Navigation interface. Endeca Analytics builds on the core ca pabilities of the Endeca MDEX Engine to enable applications that examine aggregate information such as trends, statistics, analytical visualizations, comparisons, and so on, all within the Guided Navigation interface.



1.3.29



Analytics Statements can include: •

Unary operators - e.g., absolute value, square root, ceiling/floor,…

•

Aggregate operators - average, total, min, max, standard deviation,…

•

Multi-dimension aggregation - e.g., group by region and then by gender

•

Having and Where Clauses - e.g., restrict results by pre/post-aggregate values

•

Arithmetic and Logical Expressions - e.g., A times B minus (X plus Y)

•

Sorting and Paging - e.g., top ten results by metric in descending order



1.3.30





1.3.31




Lesson 4 – Latitude Studio Overview


1.4.1





1.4.2



Latitude Studio is an extensible, component-based application with a focus on light-weight architecture, standard component interfaces, and configurability. Latitude Studio is generally used f or internal-facing applications that exist behind the firewall. However, it is also appropriate for lowthroughput external-facing applications; for example, extranets. Latitude Studio was created to provide more granular layout and configuration control for a broader audience of users, enabling them to control their own experience. Driven by increasing customer demands and competition, every organization needs to rapidly address new challenges of performance and profitability at the very heart of their operations. To help enterprises address these challenges by leveraging existing IT investments, Endeca has developed an innovative solution that provides the ability to rapidly access the most important enterprise information. This search-based approach connects all necessary information in a user-friendly environment, that provides ease of discovery for key decision-making tasks. This solution is Latitude Studio. Latitude Studio enables rapid configuration of search applications that offer the highly interactive Guided Navigation® user experience across a full range of structured and unstructured enterprise data. Search applications leveraging Latitude Studio use Latitude technology to deliver the clearest visibility into information, driving significant cost savings and increased revenue for our customers.



1.4.3



•

Intuitive interface for application developers and administrators

•

SOA (Service Oriented Architecture) framework for application integration

•

Granular, role-based authorizations

•

Enables creation of personal user pages

•

Multi-language support

•

Extensible standards-based system for creating components



1.4.4





1.4.5



Latitude components are a core set of standard components included with Latitude Studio that enable out-of-the-box configuration of the initial search application and empower you to be more productive quicker. Components are a planned concept to enable partners and 3 rd party Independent Software Vendors (ISV) develop and customize components. Latitude Components: • Are JSPs that include all logic for − Retrieving/manipulating MDEX data − Rendering results • Provide specific features for specific market-oriented applications − For example in manufacturing, guided navigation, analytics result grid for Spend Analysis − Promotes high value functionality • Are included with the source license − Customers can extend component features and customize, if needed • Are self-contained and will contain all of their own custom business/event/data retrieval/rendering logic



1.4.6





1.4.7



The development team chooses the components in your Latitude Studio implementation and power users typically configuring their layout and certain functionality, such as analytics. Though end users cannot configure the components directly, it is important that they are familiar with their functionality.



1.4.8



Latitude studio offers several charts, tables, and metrics-based components that are driven by analytics. Business Analysts can include Endeca analytics statements in analytics-enabled components to create powerful data visualizations for end users .



1.4.9



Security Themes Authentication - User credentials validated against external systems •

Supports Active Directory / LDAP authentication

•

Supports Multi-Domain Security (“Single Sign-On”)

Authorization: User and entitlement data maintained in and imported from source systems •

Single point of control for operational simplicity

•

Enables rapid update of users and entitlements

•

Complex role expressions (AND, OR, NOT) enable dynamic query-time entitlement filters

System: Endeca builds on the security of the underlying OS and server •

Set up separate user account for Endeca processes and data

•

Isolate Endeca servers from other network segments

•

Mutually-authenticated SSL to secure network traffic (if desired)

Endeca indices constitute a complete copy of the corpus: •

Files on disk not encrypted

•

Rely on OS/network layers to protect sensitive data from unauthorized user access



1.4.10





1.4.11

Module 1 – Latitude Core Concepts

Student Guide

Getting Started with Latitude 2


Lesson 5 – Data Exploration in a Latitude Application


1.5.1



Student Guide



1.5.2


Student Guide


The sample application you will build in this course is the same application used in the Latitude Getting Started Experience. This application uses data f rom a fictitious bicycle manufacturer named Adventure Works, and serves as the reference data set for Endeca Latitude. Included in the data set are financial and transactional sales data, which support numerous business intelligence scenarios. The Latitude Sampl e application is designed to enable you to quickly learn how to load data into Latitude, and then to explore the main features of the front-end application powered by Latitude.



1.5.3


Student Guide


Latitude applications have two primary user types – business analysts and end users. A business analyst is typically responsible for configuring Latitude Studio components and creating the pages or dashboards that end users access. End users of Latitude applications are generally experts in their field of business, but not technology specialists. Typically, end users rely on business analysts to create Latitude applications that enable them to use discov ery techniques to answer open-ended questions, and research and vi ew their existing data in new and powerful ways. Because the Latitude Sample Application is shipped with the Latitude product as a teaching tool, this application is designed to address a broad user audience. Toward that end, all components in this application are displayed in business analyst mode with Preferences available. Note that in a liv e environment, end users are not able to access component preferences for editing.



1.5.4



Student Guide



1.5.5


Student Guide


Notice the position of the Guided Navigati on component, the tabs f or different pages in the application (Welcome, Sales Overview, Products), and the Search box. The components on the left update are used to reveal information in the components on the right, such as when a record or value search is performed. Note how search and nav igation choices are tracked in the Breadcrumbs component. Ideally, most applications will hav e the Search Box, Breadcrumbs and Guided Navigation down the lef t side of the page.



1.5.6



Student Guide



1.5.7



Student Guide



1.5.8



Student Guide



1.5.9



Student Guide



1.5.10



Student Guide



1.5.11

Module 2 – Basic Data Design


Presentation: ~ 40 minutes Lab: ~50 minutes

Lesson 1– Project Basics with Latitude


2.1.1





2.1.2





2.1.3



The Latitude Information Integration System is composed of the Latitude Data Integrator (LDI), the Content Acquisition System (CAS), and the Connector for Informatica PowerCenter. Additional third-party ETL connectors may be added in the future. CAS includes the Endeca Web Crawler and the Endeca CAS Server, as well as a rich set of packaged adapters. These components allow you to create and manage a set of file system, Content Management System (CMS), and Web crawls. Packaged adapters reach the most common systems, including JDBC and ODBC. The Content Adapter Development Kit (CADK) allows developers to write custom adapters and Java manipulators. Data and configuration are ingested using the Latitude Data Integrator. Examples of configuration information are: •

Which fields to enable for text search

•

Which options will be available for browsing and refining

•

Precedence rules

The MDEX engine, the core of the Latitude solution, loads the index into memory at start-up. Once the MDEX engine is online, it is ready to handle queries from Endeca-powered Web applications. Latitude Studio is the tool used to create and configure an Endeca f ront-end application. A Latitude Studio application is configured to query the MDEX Engine via the Conversation Web Service. This capability is built into the various Latitude Studio components (e.g., search box, breadcrumbs, guided navigation, results list). Lesson 1– Project Basics with Latitude


2.1.4



Data is ingested into the MDEX via the Data Ingest Web Service or the Bulk Load Interface. The Latitude Data Integrator comes with Latitude Connectors which interact with these services. Configuration is ingested into the MDEX via the Configuration Web Service. This is done using LDI components that that use the Configuration Web Service to communicate with the MDEX Engine. The Configuration Web Service is used to configure the record schema and MDEX Engine features. A Latitude Studio application is configured to query the MDEX Engine via the Conversation Web Service. This capability is built into the various Latitude Studio components (e.g., search box, breadcrumbs, guided navigation, results list). The Administration Web Service is used by IT engineers and administrators to integrate the MDEX Engine server and its reporting with third-party IT tools (e.g., monitoring and backups). This service interacts with the MDEX Engine server outside of the MDEX Engine data layer, while the other Endeca Web services interact with the indexes within the MDEX Engine data layer.



2.1.5





2.1.6



The Latitude Data Integrator is based off CloverETL Designer technology. The Designer and the Server products rely on the Engine library to assemble and execute graphs. All products build on the Java platform.



2.1.7





2.1.8





2.1.9





2.1.10





2.1.11





2.1.12





2.1.13





2.1.14





2.1.15





2.1.16





2.1.17





2.1.18



When starting the Latitude Data Integrator, it will first ask which workspace to use. Within the workspaces, projects are defined and used to manage work across the whole Eclipse platform. It is important to define the project structure carefully, because other plugins may want to access your project (e.g. version control), and changes to project descriptors may aff ect Designer or Eclipse behavior.



2.1.19



It is usual to add other sub folders which may be project specific For example, when testing it is not unusual to add an mdex sub folder where the dgraph log files would then appear in the project. Config-in is also usually created to store configuration input files.



2.1.20





2.1.21





2.1.22





2.1.23



Navigator pane Lists projects; Each project contains folders such as data-in (input data files) and graph (pipeline of components that process the data) Graph Editor pane Where you create a graph/pipeline and configure its components Palette pane List of components available for use in building a graph: •

Readers

•

Transformers

•

Joiners

•

Writers

•

Latitude components •

Add/Update Records

•

Bulk Add/Replace Records

•

Add KVPs

•

Add Managed Values

•

Delete Data

Palette pane also provides functionality to connect components and define metadata. Lesson 1– Project Basics with Latitude


2.1.24



Outline pane Components used in graph, metadata defined in graph, connections defined (DB, JMS, QuickBase) Tab pane Report windows for Properties, Console (reports output of running a graph), Problems, CloverETL Regex Tester, CloverETL – Graph tracking, CloverETL – Log



2.1.25



For information on how to complete t hese steps, see the Latitude Data Integrator Designer Guide.



2.1.26



Delimited. This is the type of records in which every two adjacent fields are separated from each other by a delimiter and the whole record is terminated by record delimiter as well. Fixed. This is the type of records in which every f ield has some specified length (size). It is counted in numbers of characters. Mixed. This is the type of records in which fields can be separated from each other by a delimiter and also have some specified length (size). The size is counted in number of characters. This record type is the mixture of the two cases above. Each individual field may have different properties. Some fields may only have a delimiter. Others may have specified size. The rest of them may have both delimiter and size.



2.1.27





2.1.28





2.1.29





2.1.30





2.1.31





2.1.32





2.1.33





2.1.34





2.1.35





2.1.36





2.1.37





2.1.38



mkmdex command default values --bulk-_load_port = http port + 1 --threads = 2 Note: The Latitude Sample Application has graphs with built-in scripts for deletion and re-creation of the MDEX and running of a dgraph. Investigate the InitMDEXWindows or InitMDEXLinux graphs for detailed information. These scripts can be modified for your purposes. They are intended for testing; usually you will develop separate scripts for production use.



2.1.39



If you do not have the path to the MDEX installation/bin on your path you need to specif y it in your command to point to where the mkmdex command exists (e.g. c:\Endeca\MDEX\7.1.0\bin) Note: The bulk load port defaults to the HTTP port + 1 (e.g. 7776 7777); the default number of threads is two.



2.1.40



Note: This process is subject to modification. This is the process implemented by the Latitude Sample Application. Your application may apply different logic e.g. You may do the above process occasionally (which would be equivalent to a complete Baseline Update for earlier versions of the MDEX engine). However you may run incremental processes for revising data and metadata in between (roughly equivalent to a Partial Update in earlier releases).



2.1.41





2.1.42



Presentation: ~25 minutes Lab: ~30 minutes

Lesson 2 – Working with Latitude Studio


2.2.1





2.2.2





2.2.3





2.2.4



Before attempting to access Latitude Studio, ensure that you have started the application server instance with Latitude Studio loaded on it. The email address and password provided on this slide are the defaults. For information on how to change the credentials, see the Latitude Studio User’s Guide . Note: Latitude Studio 2.x is compatible with Microsoft Internet Explorer 8.0( with compatibility mode disabled), Mozilla Firefox 3.6 and Firefox 6.



2.2.5



The Dock provides access to various commands and menus in the portal. Some of these will be covered in greater detail later in this lesson. Note: Only business analysts can see these menu items, unless end user permissions have been configured otherwise.



2.2.6



Themes are hot deployable plugins which can completely transform the look and feel of the pages in your Latitude Studio application. Most organizations have their own standards for look and feel, which they tend to use across all Web sites and Web applications in their infrastructure. Themes make it possible for a site designer to customize a company’s Latitude Studio application and align its look and feel with those of existing applications. The Latitude Studio includes one theme out of the box; however, there are several theme plugins available on Liferay's web site. Choosing a Liferay theme that closely resembles what you want to do and then customizing it is much easier than building a theme from scratch. For more information on theme development, see the Liferay Developer's Guide. To configure your Latitude Studio application’s theme, go to the Manage Pages page (accessible through the Dock menu) and click the Look and Feel tab.



2.2.7





2.2.8



To add a page to the portal: 1. Click Add Page. 2. Type the name for the new page in the yellow textbox. 3. Click Save or press Enter . Note: Only business analysts can add pages, unless end user permissions have been configured otherwise.



2.2.9



A Layout Template is a portal tool that provides pre-determined page arrangements for component placement on a page. Each template divides the body of a portal page in various ways, providing a container of sorts, into which you drag, drop and arrange your components. The Latitude Studio ships with 10 layout templates from which you can choose; however, if you have a complex page layout, you may wish to create a custom layout template of your own. To access the Layout Template tool, click Layout Templates in the Dock drop-down menu.



2.2.10



The Add Component menu enables you to add components (and other features/tools) to the Latitude Studio portal. The Add Component menu provides access to Content Management features, the Latitude standard components, Layout Control components, and Tools. To access the Add Component list, choose Add Component from the Dock drop-down menu. To add components to your portal page, drag a component onto the main page layout or click Add.



2.2.11



You will create this page in the lab that accompanies this lesson.



2.2.12





2.2.13





2.2.14





2.2.15



The Guided Navigation ® component provides classic "left hand" navigation for end users. In this component, attributes are grouped, and only a single managed attribute’s refinements are available to the end user at any given time via fly out menu. The Guided Navigation component can enable end users to expand and/or close multiple attribute groups; view values for a single attribute at a time via fly out menu; and utilize the Submit Selections control for multi-OR/multi-AND attributes (i.e., selections not submitted on checkbox select).



2.2.16



On the Preferences page for the Guided Navigation component, power users can configure component behavior and manage attribute groups. For information on how to conf igure this component, see the Latitude Studio User’s Guide.



2.2.17



The Breadcrumbs component summarizes any Guided Navigation selections, keyword searches, or range filters that have been specified. Business analysts need only select a data source to configure the Breadcrumbs component for use.



2.2.18



The Search Box component allows end users to type in a keyword for searches in Latitude Studio. It also offers type-ahead functionality, which is designed to provide end users with suggestions for their search as they type. Provided suggestions are based on available attributes in the data source selected for that component, and they are clickable; in other words, end users can choose to click a suggestion in the type-ahead list to view the results. End users may also choose to select the Search within Results checkbox, which will limit their search to the results that are currently displayed.



2.2.19



On the Preferences page for the Search Box component, power users can configure search interfaces and type-ahead behavior. For information on how to conf igure this component, see the Latitude Studio User’s Guide.



2.2.20



The Results List component displays a list of records reflecting the current refinement state in a list format similar to web search results.



2.2.21



On the Preferences page for the Results List component, business analysts can configure display configuration and sorting options as well as design the Results Template (pictured). For information on how to conf igure this component, see the Latitude Studio User’s Guide.



2.2.22





2.2.23





2.2.24





2.2.25



To access the Control Panel: 1. Point to the Dock in the upper-right corner of the page. A drop-down menu displays. 2. Choose Control Panel from the drop-down menu.



2.2.26



The Data Sources page allows business analysts to view configured data sources and test their connection to them. End users have the ability to reload updated configurations. Data sources colored red are not active (have no MDEX engine currently running on that port)



2.2.27



You can use the Data Source Bindings area to configure which data source your components use.



2.2.28



The Attribute Settings page allows the power user to select a data source; to create, edit, and delete attribute groups; and to change display names for any attribute in a selected data source.



2.2.29





2.2.30



Timings Presentation: ~55 minutes Lab: ~40 minutes

Lesson 3 – Data Modeling and Joining Data


2.3.1





2.3.2





2.3.3





2.3.4





2.3.5





2.3.6





2.3.7





2.3.8





2.3.9





2.3.10





2.3.11





2.3.12





2.3.13





2.3.14





2.3.15





2.3.16





2.3.17





2.3.18





2.3.19



* KPI = Key Performance Indicator is an industry jargon term for a type of measure of performance.



2.3.20



* KPI = Key Performance Indicator is an industry jargon term for a type of measure of performance.



2.3.21





2.3.22





2.3.23





2.3.24





2.3.25





2.3.26



When you are writing your analytics statements be sure you are rolling up (Group By) the correct grain so that your calculations are accurate.



2.3.27





2.3.28





2.3.29





2.3.30





2.3.31





2.3.32





2.3.33





2.3.34





2.3.35



A hash join reads slaves into memory, then for each row of each subsequent source, does an in-memory lookup to find a matching row in the first source. This is works when the slaves are small enough to fit into memory, and doesn’t require anything to be sorted.



2.3.36





2.3.37



An inner join ensures that all shared records are processed. With this type of join only the records that have all properties from all sources will be processed. In the above example notice that the record with ID=B and ID=E both get discarded because they do not contain all values.



2.3.38





2.3.39



A left join uses the first data source to determine which records get processed. In the example above all records from Data Source 2 that have a corresponding record in Data Source 1 are merged and processed. Since the record with ID=E in Data Source 2 does not have a corresponding record in Data Source 1 it is not processed. that a join in LDI does result in null columns – it is only when the data is loaded into the MDEX that missing attributes will occur where data is null. Note



2.3.40





2.3.41



An outer join ensures that all records from both data sources get processed. After the join, some records may not have values for all properties. This occurs when a record in one data source does not have a corresponding record in the other data source.



2.3.42



1. Instantiate a reader component to read in FactSales data 2. Instantiate a reader component to read in DimDate data 3. Connect FactSales reader component’s output to the master input port of the ExtHashJoin component 4. Connect the DimDate reader component’s output to the slave input port of the ExtHashJoin component 5. Double-click the ExtHashJoin component. 6. Select Inner join from the drop-down menu 7. Click in the value field for Join key and then click the ellipsis 8. Drag and drop the appropriate slave field to the Slave Key field column 9. Click OK 10. Click in the value field for Transform and then click the ellipsis 11. Left-click all rows in the Field column 12. Click Auto-rules 13. Verify Transformations are defined 14. Click OK



2.3.43



A merge join requires all data sources to be sorted, and finds matching rows by just reading the next row from each source and seeing if the keys match, and keeping on reading rows from each source it finds a non-matching row. This has the advantage of working even if all the data sources are too big to fit into memory, but requires them to be sorted.



2.3.44





2.3.45





2.3.46



Although not strictly speaking a Joiner component as it is found on the Transformers component tab, the DataIntersection can be considered to be a type of join.



2.3.47





2.3.48





2.3.49





2.3.50





2.3.51





2.3.52





2.3.53



Star schema = Generic representation of a dimensional model in a relational database In order for the bulk ingest LDI component to submit records into the MDEX engine it requires a unique identifier, a record specifier, for each record. For this project the record specifier is defined as a transform in the FactSales_SalesTransaction data. It is defined to be FactSales_RecordSpec and is the concatenation of FactSales_SalesOrderNumberFactSales_SalesOrderLineNumber.



2.3.54





2.3.55

Module 3 – Search and Navigation Configuration Enhancements


Timing Presentation: ~75 minutes Lab: ~90 minutes

Lesson 1 – Introduction to Search


3.1.1





3.1.2





3.1.3



Note: Capital characters are mapped to lower case equivalents because Endeca search indexing is

always case-folded.



3.1.4



Search Key: An attribute or search interface to search across for a match Search Match Mode: Criteria for a record to be successf ully matched, based on the number of terms Search Terms: One or more search terms entered by the end user



3.1.5



Alternative ingest method: 1. Read in XML directly using a UniversalDataReader LDI component 2. Ingest the XML directly into the MDEX using the Add/Update Records LDI component(see LDI Guide documentation for details) OR 3. Ingest the PDR into the MDEX using the WebServiceClient LDI component



3.1.6





3.1.7





3.1.8



Notice that value search is enabled by default and record search is disabled by default. You may want to disable value search for specific managed attributes to av oid unnecessary MDEX performance degradation. The Type property is set to mdex:string by default but also supports the f ollowing data types: mdex:boolean; mdex:dateTime; mdex:double; mdex:int; mdex:long; mdex:duration; and mdex:geocode. The Selectability property can also be set to Multi-AND or Multi-OR. The Sortability property has one other option: Lexical. PDR Property Names Key = mdex-property_Key Display Name = mdex-property_DisplayName Type = mdex-property_Type Value Search Enabled = mdex-property_IsPropertyValueSearchable Single Assign = mdex-property_IsSingleAssign Record Search Enabled = mdex-property_IsTextSearchable Is Unique = mdex-property_IsUnique Wildcard Record Search Enabled = mdex-property_TextSearchAllowsWildcards Selectabilty = system-navigation_Select Show Record Counts = system-navigation_Select Sortability = system-navigation_Sorting

For more information, see the Data Ingest Guide. Lesson 1 – Introduction to Search


3.1.9



DDR Property Names Key = mdex-dimension_Key Enabled for display in Guided Navigation component = mdex-dimension_EnableRefinements Enabled for hierarchical value search = mdex-dimension_IsDimensionSearchHierarchical Enabled forhierarchical record search = mdex-dimension_IsRecordSearchHierarchical

For more information, see the Data Ingest Guide.



3.1.10



Example on the slide is taken from the Getting Started Experience (GSE) sample application and can be used as a starting point. Alternative ingest method: 1. Read in XML directly using a UniversalDataReader LDI component 2. Ingest the XML directly into the MDEX using the Add/Update Records LDI component



3.1.11



The developer needs to define the transformation of metadata based on the input metadata. This is shown in the slide using CTL (Clover Transformation Language).



3.1.12



Required inputs to configure: WSDL URL (e.g., http://localhost:5555/ws/config?wsdl Request Structure: The Edit Request structure dialog reads in an xml string that is generated by the Create Search Interface (Denormalizer) component.



3.1.13





3.1.14



Note that the Build Requests Denormalizer is introduced into the graph to batch web service requests based upon a generated singleAggregateKey value calculated if required in the Reformat (Make Attributes Searchable) component.



3.1.15





3.1.16





3.1.17



* Note that in Latitude 2.2 that Precedence Rules are now submitted by submission of a record type, not an Index Configuration Document.



3.1.18





3.1.19





3.1.20



A search interface can be configured with one or more search interface members. RelRank strategies must be loaded before Search Interfaces via LDI graphs for both record searches and value searches. The technique is similar.



3.1.21



Alternative ingest method: 1. Read in XML directly using a UniversalDataReader LDI component 2. Ingest the XML directly into the MDEX using the WebServiceClienLDI component(see the LDI MDEX Component Guide for details)



3.1.22



Note: for searchable attribute definition, only the AttributeName field is used. Both fields are used when defining search Interfaces

In this case you are reading in only a list of 1 attribute, SurveyResponse. Required input: Define the full pathname (File URL) for the file to be read (e.g. SearchAttributes.txt

found in the data-in directory in the LDI project).



3.1.23



This denormalizer creates a element for each attribute listed in the input file provided in the SearchableAttributes.txt file read in from the UniversalDataReader component. Required inputs to configure: Key, denormalize transformation



3.1.24



Required inputs to configure: WSDL URL (e.g., http://localhost:5555/ws/config?wsdl Request Structure: The Edit Request structure dialog defines a search interface named “all” that

contains all the members defined by the XML generated by the Create Default Search Interface (Denormalize) component. Sample of a resulting recsearch_config XML document generated by a similar LDI graph (Item in bold would be the $xmlString contents): SurveyResponse



3.1.25





3.1.26





3.1.27



Note: Cross-field matches only apply to record searches with multiple search terms specified .



3.1.28





3.1.29





3.1.30



By default, Endeca search operations match user query terms to entire words in the indexed text.



3.1.31



Note: If only a subset of search members of a search interf ace are enabled for wildcard search,

wildcard search only works for those members as configured. Wildcard Search Performance Impact:

Enabling wildcard search generally increases the time and disk space required for indexing. To optimize performance, the MDEX Engine performs wildcard indexing for words that are shorter than 1024 characters. Refer to the Latitude Developer’s Guide chapter “Using Wildcard Search” for more details.



3.1.32





3.1.33





3.1.34



Stemming files are XML files built from electronic dictionaries, and each language is stored in a separate file. Performance Impact:

Stemming and thesaurus equivalences generally add little or no time to data processing and indexing,and introduce little space overhead (beyond the space required to store the raw string forms of the equivalences). In terms of online processing, both features will expand the set of results f or typical user queries.While this generally slows search performance (search operations require an amount of time tha t grows linearly with the number of results), typically these additiona l results are a required part of the application behavior and cannot be avoided. The overhead involved in matching the user query to thesaurus and stemming forms is generally low, but could slow performance in cases where a large th esaurus (tens of thousands of entries) is asked to process long search queries (dozens of terms). Typical applications exhibit ne ither extremely large thesauri nor very long user search queries. Because matching for stemming entries is performed on a single-word basis, the cost for stemmingoriented query expansion does not grow with the size of the stemming database or with the length of the query.



3.1.35



With Stemming disabled, a search for baby only returns the 212 results for baby. A search for babies only returns the 37 results for babies. With Stemming enabled, a search for either baby or babies returns all 249 results for baby & babies .



3.1.36





3.1.37



The Endeca Thesaurus requires you to explicitly enumerate all thesaurus entries for each specific MDEX configuration. The Thesaurus also works with an XML file that can be externally generated or modified. Slang and acronyms are both great reasons to use Thesaurus entries on your site. One-way XML construct : California Sonoma Napa

Two-way XML construct: United States USA



3.1.38



In this example, since a one-way thesaurus entry has been created from California to Sonoma & Napa, when a user searches for California they will get all 255 results for all three terms: California, Sonoma & Napa. Yet, if they search for either Sonoma or Napa (the other side of the one-way entry), they get the same individual results they would get without a th esaurus entry.



3.1.39



In this example, since a two-way thesaurus entry has been create d for United States and USA, when a user searches for United States they will get all 131 results for both United States & USA. The same is true if the user searches for USA.



3.1.40





3.1.41



Be aware of how the expansion can potentially impact the results sets & possibly perfo rmance; for example, the end result for an initial search on “cab” returns matches for cab, cabs, cabernet, and cabernets.



3.1.42





3.1.43





3.1.44



Stop words are not removed phrases enclosed in quotes. All words between quotes are treated as a single search term.



3.1.45





3.1.46



Spelli Spelling ng Correcti Correction on :

The ability of the MDEX Engine Engin e to return expected search results even when the user misspells one or more of the query terms. Endeca provides two spelling spelling correction features: Auto-Correct (for record & value search), DidYou-Mean (for record search only). The MDEX Engine auto-computes auto -computes alternate spellings for user query terms returning zero results.



3.1.47



Did-You-Mean:

The ability of the MDEX engine eng ine to provide alternative suggestions suggestion s for searched terms resulting in fewer matches than the threshold. It is is always enabled. The threshold is 30. The number of suggestions is 1. It is currently not configurable in the MDEX. MDEX.



3.1.48





3.1.49



A snippet contains the search terms the user provides along with a portion of the surrounding content to provide context.It context.It is used on individual individual members of a search interface that contain many lines of content (e.g., Descripti Description, on, Abstract, Abstract, etc.) It It allows the user to more quickly choose individual individual records of interest. Snippeting is enabled in the recsearch_config XML configuration document. . For details refer to the Developer’s Guide and the LDI MDEX Components Guide. Match Modes: MatchAll, MatchPartial*, MatchAny, MatchAllAny, MatchAllPartial*, MatchPartialMax*,

MatchBoolean The value of “some" for partial match modes is defined in the recsearch_config recsearch_config index XML configuration document (as part of the search interface definition). For details refe r to the Data Ingest Guide. The standard alphanumeric characters for English are a-z, A-Z, and 0-9 . Non-alphanumeric characters considered to be punctuation are: !@#&()_[{]}:;’,?/* and are treated as whitespace by default. Non-alphanumeric Non-alphanumeric characters characters considered considered to be symbols are: `~$^+=<> `~$^+=<> and are treated as whitespace whitespace by default. For details refer to the Latitude Developer’s Developer’s Guide.



3.1.50





3.1.51



Timing Presentation: ~45 minutes Lab: ~75 minutes

Lesson 2 – Enhancing Navigation


3.2.1





3.2.2





3.2.3



A second method to control displaying some attributes (managed attributes ONLY) is is by disabling the mdex-dimension_EnableRefinements property in in the DDR. This disables the refinement computation in in the MDEX engine entirely.



3.2.4



Enabling refinement counts is set on by def ault for both standard and managed attributes. This is defined in the PDR ( system-navigation_ShowRecordCounts). For relatively simple applications this can be very u seful. However, for applications with complex data, refinement counts may

not refer to anything obvious to the user. This is because the refinement counts are computed in terms of the physical Endeca record data model which the application often never shows this to the end users (e.g., via record lists, charts, etc.). Instead, analytics are used to roll up the physical records into more meaningful entities.



3.2.5



Refinements for any attribute, managed or standard, can be sorted in the guided navigation component in two ways: by record count (aka “dynamic ranking”) or lexically. You can configure this programmatically at data ingest time by setting the system-navigation_Sorting property on the PDR for each attribute, or configure it manually after the fact in the Latitude Studio Attribute Settings component.



3.2.6



Dynamic Refinement Ranking allows control of the maximum number of managed attribute values that get returned from a navigation query. It ranks managed attribute values according to the number of records associated with a refinement or alphabetically. Dynamic Refinement Ranking is very useful for managed attributes with many managed attribute values and can be configured on a attribute by attribute basis. With the application displayed above, notice that Michael G Blythe is first in the refinement list for Male employees while Jillian Carson is first in the list for Female employees.



3.2.7



You can limit the number of refinements that appear for an attribute. If an attribute, managed or standard, has a lot of values, you often want the Guided Navigation component to show only the first few attribute values and then display a “more” link. This is true regardless of how the refinements are sorted, or whether you’re showing refinement counts in parentheses after the refinement text. You configure this in the preferences menu of the Guided Navigation component in Latitude Studio.



3.2.8





3.2.9





3.2.10





3.2.11



When specifying an attribute (standard or managed) as the Source for a Precedence Rule, the Target attribute will become visible af ter any attribute value of the Source attribute has been refined on. In this example, once you select any country, the State/Province attribute will become visible. Also, once you select any State/Province, the City attribute will become visible.



3.2.12





3.2.13



When specifying an attribute value as the Source for a Precedence Rule, the Target attribute will become visible only after the specified attribute value of the Source attribute has been refined on. In the above example, the Resolution attribute will only become visible when you select Digital Cameras from the list of Categories. Whereas the Processing Speed attribute will only become visible when you select Computers from the list of Categories.



3.2.14





3.2.15





3.2.16





3.2.17



In this case you are reading in only a list of 2 precedence rules DimGeography_StateProviceCode > DimGeography_City DimGeography_StateProviceName > DimGeography_City Required input: Define the full pathname (File URL) for the file to be read (e.g. PrecedenceRules.csv found in the data-in directory in the LDI project). NOTE: any file format that you can read in and convert to the required Precedence Rule record format is acceptable



3.2.18



In this case you are reading in only a list of 2 precedence rules DimGeography_StateProviceCode > DimGeography_City DimGeography_StateProviceName > DimGeography_City Required input: Define the full pathname (File URL) for the file to be read (e.g. PrecedenceRules.csv found in the data-in directory in the LDI project).



3.2.19



This Reformat creates the record structure for the precedence rule for each trigger -> target data listed in file provided (PrecedenceRules.csv) and read in from the UniversalDataReader component. Required inputs to configure: transform (written in CTL)



3.2.20



This Denormalize component creates the list of precedence rule records. Required inputs to configure: transform (written in CTL)



3.2.21



Required inputs to configure: WSDL URL (e.g., http://localhost:5555/ws/config?wsdl Request Structure: The Edit Request structure dialog defines a precedence rule that contains all the record list generated by the Reformat component.



3.2.22





3.2.23





3.2.24





3.2.25





3.2.26





3.2.27





3.2.28



1. Under the Attribute Groups list, in the Key for a new group text box, type the name of the group (must be NCName compliant). 2. In the Display name for a new group field, type the display name (does NOT need to be NCName compliant). 3. Click Add. 4. In the All Attributes, list check the attributes you want to add to the group. 5. From the Add selected attributes to group drop-down list select the group. 6. Click Add.



3.2.29



Default order of attributes are configured in Latitude Studio’s Attribute Settings component. They are set on a per group basis. For new Latitude Studio components that use groups (e.g., Guided Navigation, Record Details, Compare) this default display order will be used. The Power User can configure a different display order for individual components using the preferences option for that component.



3.2.30



If you delete an attribute from a group, it is restored as a member of the default Other group. When you attempt to delete an attribute group you will be prompted to confirm the deletion because this action cannot be undone. However, you can recreate the attribute group and reselect attributes to be added to it.



3.2.31





3.2.32





3.2.33





3.2.34





3.2.35





3.2.36





3.2.37



This data structure format is required by the Add Managed Values Latitude component. Required metadata definition is to be found in the Getting Started project. You can use your own custom import formats if you use the Web Service component and generate the required XML request structure through your own custom logic. This may be required if you wish to add custom properties to each hierarchical value (not currently supported by Latitude Studio standard components).



3.2.38





3.2.39



Note that this approach will only work if all expected data values for the managed attribute are represented in the taxonomy structure read by the “Read Taxonomy” component.



3.2.40





3.2.41



This example creates a Restaurant_Cuisine hierarchical managed attribute of the form: American Traditional Regional Asian Chinese Thai European Italian Italian South Italian North French Others ...

The source data record is in the format: Note: In the diagram, the Normali ze Cuisines component splits the Cuisine field below into separate Cuisines. This operation is not required if your data has single values for Cuisine stored in the fi eld. Restaurant_Name Fred's Diner Gastropod Herges

Cuisine American:Regional:French Italian: Italian North German



3.2.42





3.2.43

Module 4 – Advanced Topics



Lesson 1 – Data Ingest and Modeling Overview


4.1.1





4.1.2





4.1.3



In the first example, “separation”, we have chosen to have a unique record f or each color variation of vest. Perhaps there is a unique order amount or inventory total for each of these items. In our data source, our data may already be organized this way. Or we may split it apart into individual records when we extract the data or during the ETL process. In the second example, “consolidation”, we have chosen to have each record represent a type of product, in this case, a style of sweater. Each record has multiple color Properties associated with it. In our data source, the data may already be organized this way. Or we may combine multiple rows into a single record during the extraction process or while running the data through the ETL tool.



4.1.4



One of the drawbacks to the single record design, is a loss of precision among attributes. In the example above, there is no correlation between the Color attribute and the AmountOrdered attribute.



4.1.5



When choosing to separate data into distinct records, one drawback is t he potential for large result sets of what is largely the same thing. In t his case, our record list contains many similar items, which vary based on color and size. This can be overwhelming for end users.



4.1.6



Endeca record design always means incorporation of elements of both of these techniques where data is combined.



4.1.7





4.1.8





4.1.11



Your application may also have diff erent types of records. For instance, a car database may contain car records, accessory records, dealer records, and user review records. We refer to these different types as being heterogeneous. In the above example, a financial services index contains 3 types of records: Holdings, News, and Research. Each of these records has different sets of standard and managed attributes. For instance, the News record has a managed attribute called NewsPublisher, but neither the Holdings nor the Research record types have this managed attribute. There are a few options for how data is modeled in your index under these conditions to enable association among records of different types. The next few slides discuss options available to you to enable relational type queries.



4.1.12





4.1.13





4.1.14



All record types may have common attributes, e.g. AuthorName, but each of them certainly has distinct attributes, e.g. AuthorAge and PublishYear. By default, there is no association among the different record types that enables you to determine a distinct attribute of a record based on a shared field with another record of a different type. For example, you cannot query to return books written by 40 year-old authors. Nor can you return authors who published books in 2006.



4.1.15



One option is to denormalize the data down to a single record type. In the example above, we have attached Author secondary data (like Age) to each book record. In this way, each book record will have all related author information associated with it. The merging of data would occur either before or during the ETL process. Performance is also improved because the association is made during

indexing.



4.1.16



There are two primary drawbacks to denormalization; the first is loss of precision. By merging multiple author records, we lose the delineation between each author and his/her associated age. The second drawback is an increased memory footprint. The book records must now contain redundant author data for every book that is written by the author. Because this example is very simplistic, its impact may not be apparent. However, consider an index that contains parts, suppliers, and orders. If we denormalize all data into part records, each part must contain every related supplier, and for each supplier, every related order. You can begin to see how thiis method can produce an exponentially large index.



4.1.17





4.1.18





4.1.19





4.1.20





4.1.21





4.1.22





4.1.23





4.1.24



With the date record type (sometimes called the calendar record type), you can add a record to the MDEX for each day (going back 3 years for example) and merely attribute that record type with some of the date dimensions required. Then in analytics statements we can do things like show total sales by month for the last 90 days: DEFINE “Last90Days” AS SELECT “p_Date” AS “Date” WHERE “IsLast90Days”=’Y’ AND “RecordType” = ‘Date’; RETURN “90DayTrendReport” AS SELECT SUM(“p_Sales_Trans_Amt”) AS “TotSales” WHERE “RecordType” = “Sales” AND [“p_Date”] IN “Last90Days” GROUP BY “Region” This allows us to have “trending” reports that update dynamically day after day so long as we keep the calendar record type up-to-date. This is much easier that updating attribution on millions of sales transactions. Lesson 1 – Data Ingest and Modeling Overview


4.1.25





4.1.26



Language can be identified using setLanguageId method (Java API) or navigationcommand.LanguageId instance property (RAD.NET API).



4.1.27



Note that where blank fields are shown in the table that this property does not exist on the record.

Lesson 1 – Data Lesson Data Ingest Ingest and Modeling Overview


4.1.28





4.1.29





4.1.30





4.1.31





4.1.32





4.1.33





4.1.34





4.1.35





4.1.36





4.1.37

Student Guide Module 4 - Advanced Topics



Lesson 2 – Integrating with Other ETL Tooling


4.2.1





4.2.2





4.2.3



As covered in previous lessons, the MDEX Engine comes packaged with the Latitude Data Integrator. The Latitude Data Integrator is a high-performance data integration platform that extracts source records from a variety of source types and sends the data to the MDEX Engine via the Data Ingest Web Service (DIWS) or the Bulk Load Interface. The Latitude Data Integrator communicates with the DIWS and the Bulk Load Interface using a series of Endeca-developed Latitude connectors which are accessible through the Latitude Data Integrator. The MDEX Engine can also connect with other ETL clients via custom-developed DIWS connectors. Customers who use ETL clients other than the LDI can develop their own connectors using stubs generated from the Data Ingest WSDL and calls from the ETL client’s SDK. For more information on the data ingest process, see both the Latitude Data Integrator Guide and the Latitude Data Ingest Guide .



4.2.4



The Data Ingest Web Service (DIWS) loads data into a running MDEX Engine and can also update existing records. The DIWS therefore allows you to use a data integration platform to load data i nto an Endeca application. Endeca is packaged with one such platform—the Latitude Data Integrator. Customers also have the option of creating custom connectors for other ETL cli ents which run against the DIWS. The data-related tasks listed in the slide are performed via two DIW S operations: • ingestRecords : adds, modifies, and deletes records (including removing managed value assignments) • ingestDimensionValues: adds and updates managed values The DIWS is able to modify a record multiple times in a single transaction (any combination of creating, adding assignments, deleting assignments, and deleting records). For more information on the Data Ingest and Configuration Web Services, see the Developer’s Guide.



4.2.5





4.2.6



The Bulk Add/Replace Records connector adds or replaces records via the MDEX Engine’s Bulk Load Interface. It does NOT use the Data Ingest W eb Service. This connector loads data source records only; it cannot load PDRs, DDRs, managed attribute values, the GCR, nor the MDEX Engine index configuration files. When employing this connector, existing records in the MDEX Engine are added, not updated. Therefore, the key-value pair list of the incoming record will completely replace the key-value pair list of the existing record. The Bulk Add/Replace Records connector is intended to be used with bulk data when delayed update visibility and compromised concurrent query performance are acceptable. Some of the use cases for this connector are: •Full index initial load of records, with no loaded schema. •Full index initial load of records, with your record schema already loaded. •Adding more records to the MDEX Engine any time after the initial loading of records. •Replacing existing records in the MDEX Engine any time after the initial loading of records. Note that at this time, the Bulk Load Interface is a private, proprietary interface, and custom connectors cannot be built against it. In order to employ the bulk ingest process, you must use the Latitude Data Integrator. For more information on this connector, see the Endeca Latitude Data Integrator Guide, specifically the chapter entitled “Latitude Connector Reference.” Lesson 2 – Integrating with Other ETL Tooling


4.2.7



The four Endeca-developed Latitude connectors are used as follows: •Add/Update Records connector : Adds new records to a running MDEX Engine. The records can be added to an empty MDEX Engine (a full index initial load) or to one that already has records. This connector can also be used to load the Latitude application’s record schema via the PDRs and DDRs. Endeca standard attributes that do not already exist are created automatically. •Add KVPs connector : Updates existing records by adding new key-value pair (KVP) assignments to those records. This connector can also create new records for the keyvalue pairs. •Add Managed Values connector : Adds a taxonomy (managed values) to a running MDEX Engine. If the managed values belong to a managed value that is currently not in the MDEX Engine, the managed attribute is created automatically. •Delete Data connector : Removes KVP assignments from records and/or deletes entire records from the MDEX Engine. •Reset MDEX: deletes all records (including schema records) and configuration, and provisions the MDEX Engine.



4.2.8



The Export Config and Import Config connectors are intended to be used in the following cases: • Both of these connectors support cases where, after loading the default configuration, you change portions of it in Latitude Studio, such as attribute groups, or attribute group names. From this point on, you may want to keep using this changed configuration, even if you run subsequent data updates. The connectors allow you to do this. • The Export Config should also be used as part of the graph in which you run a baseline update for loading data (although, it is not intended to be used with the initial baseline update). In a typical scenario of a repeatable baseline update, you create a graph in which you start a transaction using the Transaction RunGraph component, export all configuration and schema using Export Config, run the Reset MDEX to remove all records and provision a new MDEX Engine, import the previously saved configuration and schema with Import Config, and then reload the records. At this point, the transaction can close and the node on which the baseline update was run can resume answering queries.



4.2.9



In order to create a custom connector you must generate client stubs from the DIWS and Configuration Web Service and MDEX Engine W SDL. For information on this process, see the Endeca Latitude Data Ingest Guide and Endeca Latitude Developer’s Guide for more details.



4.2.10





4.2.11

Student Guide Module 4 – Advanced Topics



Lesson 3 - Security


4.3.1


Lesson 3 - Security



4.3.2


Lesson 3 - Security



4.3.3



Stateless Same data structures as Guided Navigation

Lesson 3 - Security


4.3.4


Lesson 3 - Security



4.3.5


Lesson 3 - Security



4.3.6


Lesson 3 - Security



4.3.7



Supported authentication solutions: LDAP, Central Authentication Service (CAS), NTLM, OpenID, Atlassian Crowd, OpenSSO, SiteMinder A list of common Active Directory LDAP bind errors 525 - user not found 52e - invalid credentials 530 - not permitted to logon at this time 532 - password expired 533 - account disabled 701 - account expired 773 - user must reset password 775 - account locked For more details: http://www.nordicedge.se/support/kb/questions/179/NSD1103+Common+Active+Directory+LD AP+bind+errors

Lesson 3 - Security


4.3.8


Lesson 3 - Security



4.3.9


Lesson 3 - Security



4.3.10


Lesson 3 - Security



4.3.11


Lesson 3 - Security



4.3.12


Lesson 3 - Security



4.3.13



In lab you will see how Joe Manager’s name is renamed Hawkeye Pierce after LDAP integration is setup. The new name was imported from LDAP.

Lesson 3 - Security


4.3.14



You can make a custom theme that displays a different greeting. It’s configured in dock.vm as $user_greeting. You can either modify dock.vm directory or change the setting of $user_greeting in your init_custom.vm to #set ($user_greeting = “Welcome “ + $user_first_name + “ “ + $user_last_name + “!”).

Lesson 3 - Security


4.3.15


Lesson 3 - Security



4.3.16


Lesson 3 - Security



4.3.17


Lesson 3 - Security



4.3.18

Student Guide Module 4 – Ad Advanced Topics

Lesson Les son 3 - Security Security



4.3.19





4.3.20



1. From the Dock, Dock, in the drop-do drop-down wn menu, menu, choose choose Control Panel 2. In the Latitude Studio section of the Control Panel navigation panel, select Framework Settings

3. Change ange the df.mdexStateManager or df.mdexSecurityManager property to the full name of your class (e.g., df.mdexStateManager = com.endeca.portal.extensions.YourStateManagerClass) 4. Click Update Settings 5. Resta Restart rt the the Latit Latitude ude Studi Studio o



4.3.21





4.3.22





4.3.23





4.3.24

Student Guide Module 4 – Advanced Advanced Topics Topics



Lesson 4 – Text Enrichment in Latitude


4.4.1





4.4.2





4.4.3





4.4.4





4.4.5





4.4.6





4.4.7





4.4.8





4.4.9





4.4.10



Content Enrichment is the process you set up to get your raw information (from a PDF, for example) into the form you need for your application to be effective. This process transforms the information found in your file or document into Endeca record format – key-value pairs that contain data and metadata.



4.4.11





4.4.12





4.4.13





4.4.14




Lesson 5 – Latitude 2 Deployment Features


4.5.1





4.5.2





4.5.3



The Getting Started Application for Latitude 2.2 is already enabled for transactions. Example graphs are also included showing how to execute Begin, Commit and Rollback transactions as separate operations. Baseline.grf uses the Transaction RunGraph component to wrap all called graphs with transactions. Documentation: DevGuide.pdf: About the Transaction Web Service LDIMDEXcomponents.pdf: Transaction RunGraph component



4.5.4





4.5.5



Note that when testing individual graphs and not using the Transaction RunGraph component, you will need to Begin, Commit and Roll back transactions through web service calls. The BeginTransaction.grf, CommitTransaction.grf, and RollbackTransaction.grf graphs provided in the Getting Started application are designed for this usage. Note that if a transaction cannot be closed that you will need to shut down the MDEX engine with a call to (wget is in the MDEX bin folder): wget http://:/admin?op=exit e.g. http://localhost:5555/admin?op=exit



4.5.6





4.5.7





4.5.8



This component needs an associated Universal Data Writer to write the output file. It is recommended that the export graph is run by the Transaction RunGraph component in a separate graph to do this in a transaction to prevent any likelihood of changes during the operation. Note: the default buffer sizes in LDI are usually not large enough to support the generation of the output file using this component. Details of changing the buffer sizes can be found under Troubleshooting in the LDIMDEXComponents.pdf. It requires extraction of a defaultProperties text file from DataIntegrator\plugins\com.cloveretl.gui_3.1.0\lib\lib\cloveretl.engine.jar and then placing it in your LDI project co-located with the workspace.prm file. Edit the file to change settings as required, and then refer to the file in the Run Configuration for the graph under arguments using –config defaultProperties



4.5.9



See the LDIMDEXComponents.pdf for detailed instructions.



4.5.10





4.5.11





4.5.12



All updates and changes are directed at the Leader Node. The MDEX database file system is shared by all nodes. Followers only use the updated database when notified by the Cluster Coordinator. All operations on the Leader node must be placed in transactions; transactions cannot be nested. The cluster must entirely run on W indows or Linux. The leader node returns an HTTP 403 if a transaction is in progress which allows the load balancer to direct requests to other nodes.



4.5.13





4.5.14




Lesson 6 – Next Steps for Enablement


4.6.1





4.6.2





4.6.3



For more information on Latitude 2, navigate to http://eden.endeca.com. Click Encyclopedia, then Latitude 2 Enablement Wiki.



4.6.4





4.6.5





4.6.6





4.6.7





4.6.8





4.6.9





4.6.10





4.6.11





4.6.12

Student Guide Module 5 –Analytics


Timings Presentation: ~60 minutes Labs: ~90 minutes

Lesson 1 – Introduction to Endeca Analytics


5.1.1





5.1.2





5.1.3





5.1.4



The Endeca MDEX Engine provides a platf orm for constructing applications that enable data consumers to quickly and efficiently locate information of interest in large databases, using the Guided Navigation interface. Endeca Analytics builds on the core capabilities of the Endeca MDEX Engine to enable applications that examine aggregate information such as trends, statistics, analytical visualizations, comparisons, and so on, all within the Guided Navigation interface.



5.1.5


Module 5 –Analytics

Bike Store’s application for analyzing sales •

Business objective: Increase profitability of bike store

•

User: Business Analysts

•

User goal: Improve sales totals and margins

Provides views of the data that the users didn’t even know they wanted until they started exploring the data themselves.



5.1.6





5.1.7





5.1.8





5.1.9





5.1.10





5.1.11



RETURN provides the key for accessing analytics results from the MDEX query result . This is important when more than one statement is submitted with the MDEX query, or multiple analytics statements are nested. You will learn more about this later in the course. Important: Strings with spaces or non-alpha values need to be wrapped in double quotes. Literals should be wrapped in single quotes. Numbers that remain constant can either be wrapped in single quotes or without quotes.



5.1.12



Each key specifies the name for each expression, and defines an attribute to be returned with each Endeca record result. Each expression defines the derived values that are computed for each result record, and computes the Property value to be returned with each ERec object. One important difference with Endeca Analytics and SQL is that with Endeca the AS aliasing is not optional like it is in SQL.



5.1.13



Analytics supports a number of different core functions including: COUNT SUM AVG MEDIAN STDDEV VARIANCE MIN MAX ARB Note that grouping is required for all of these functions.



5.1.14



When multiple groupings are specified, a “bucket” (record) is created f or each unique combination of grouping values. If only GROUP is specified, a single bucket is created. Usage: GROUP BY , … may only be pre-filter standard or managed attribute names; post-filter names (result of an AS) will cause a parser error



5.1.15



Additional Unary functions include: •

ROUND (can take an optional second “numeric” variable as the order of magnitude of the round; 0 is def ault, positive number = round to the right of the decimal, negative number = round to the left of the decimal)

•

SIGN

•

SIN

•

SQRT

•

TAN

•

TO_DURATION (time function)

•

TRUNC (has the same optional second variable like ROUND)

•

POWER (requires a second numeric variable like ROUND)

•

The ARB function returns an arbitrary value from a set of values. In the example above since all the Credit Ratings for a given customer are the same you just want to know the value f or any of them.



5.1.16





5.1.17



Note that ROUND and TRUNC can be both used as unary and binary functions.



5.1.18





5.1.19



The TRUNC function can be used to round a DateTime value down to a coarser granularity. For example, this is useful when performing a GROUP BY on DateTime data based on coarser time ranges (such as GROUP BY “Quarter” given DateTime values). The EXTRACT function extracts a portion of a DateTime, such as the day of the week or month of the year. This is useful in situations where the data must be filtered or grouped by a slice of its timestamps, for example computing the total sales that occurred on any Monday.



5.1.20



In this example, the region with the highest total will be the first record in the Analytics result. Note that multiple sort keys can be specified. Ascending order is the default. Usage: ORDER BY ASC|DESC The must be defined in the return table; this means either a post-filter name (result of an AS) or the GROUP BY name



5.1.21



“YEAR” is reserved because the Latitude Analytic Query language has date-time functions for attributes of Date-Time type. “YEAR” is the f unction to extract the year value from the Date-Time attribute.



5.1.22





5.1.23



Latitude Studio includes five Endeca components that require Endeca analytics query statements to render their results/display. These components include: Chart, Results Table (Analytics), Metrics Bar, Cross Tab and Advanced Visualization. In addition, developers can create custom components that utilize the Endeca analytics query language.



5.1.24



The Chart component enables end users to view data in a variety of different chart types. All charts display with key and/or labeling on the X and Y-axis when applicable. The Chart component allows end users to switch the chart view by selecting drop-down choices; for example, an end user may choose to switch the Metric or Group By that is summarized in t he chart. The Chart component reveals specific numeric data when a cursor is rolled over it and provides a drilldown feature, allowing end users to filter results by the value(s) displayed in the chart. The Chart component can accommodate two types of analytics statements: fixed and parametric.



5.1.25



Using the top portion of the preferences page, power users can configure basic display settings for t he Chart such as title and size, as well as select the style of chart they would like to display. OOTB Corda chart templates include vertical bar, vertical stacked bar, horizontal bar, line, bar, pie, and dual-axis line/bar combo charts. The chart height/width can not exceed the HTML Height and width. 1.

Enter a Chart Title

2.

Select a Style of chart •

3.

Vertical bar, vertical stacked bar, horizontal bar, line, bar, pie, and dual-axis line/bar combo charts

Define the Chart Height and Chart Width •

Or accept defaults



5.1.26



The power user can add a fixed Analytics statement to the Chart component, or configure a parametric Analytics statement that allows end users to see drop-down lists across the top of the chart. The Metric Selector allows you to specify an ORDER BY in the Analytics statement without including it in the display. The Metric Selector does not appear until you have tested and saved the Analytics query. When the Enable Top-N Sort checkbox is selected, the chart's results can be sorted in ascending or descending order. If the Enable end user control checkbox is also selected, the end user can toggle the sort order.



5.1.27



To use a Results Table to display the results of an analytics query: 1. Choose a data source 2. Select the Use analytics statement checkbox 3. Create your statement in the Analytics statement field 4. Click Test analytics query…; if the statement is valid the Save Analytics Configuration button will be enabled



5.1.28



The Cross Tab component provides a table that allows end users to perform comparisons and identify trends across several cross sections of data. A cross tab consists of a specific configuration of rows, columns, and summary cells that together make up the table body. The values displayed in the header rows and columns represent every possible grouping value of the specified data fields. Summary cells exist at the intersection of rows and columns. The value of a given summary cell is a metric corresponding to the intersection of those two groupings.



5.1.29



To configure the Cross-Tab component: 1) Choose a data source 2) Enter an analytics statement; notice that the statement is a fixed statement as parametric analysis for the cross tab component is not currently supported 3) Click Test analytics query, this will populate the configuration with the different metrics, row groups and column groups 4) Configure which metrics you want displayed. 5) Configure the rows and columns of how you want the records shown by dragging to reorder them.



5.1.30





5.1.31





5.1.32





5.1.33



DEFINE is useful for computing intermediate result sets for further analysis by additional statements. For example, DEFINE could help you compute the average customer age from transactional records without improper weighting.



5.1.34



Essentially, FROM allows analytics statements to be nested. Without a FROM component, an analytics statement will always refer to the base record set returned by the Navigation query. Note that you may only have one FROM per query. The default FROM is the current navigation state.



5.1.35



Global WHERE clauses are not frequently used, since Navigation or Record Filters can be used instead. Note these common operators and/comparisons: •

OR

•

AND

•

=

•

>

•

<

•

<=

•

>=

•

IS NULL



5.1.36



HAVING only works on derived attributes (calculated values).



5.1.37



IN can only refer to another Analytics statement that was grouped and whose GROUP BY list matches the Key list indicated within the brackets.



5.1.38



Lookups can only refer to an Analytics statement that was grouped and whose GROUP BY list matches the Key list indicated between the brackets. The example above computes the percentage of sales per “Varietal” per “Region.”



5.1.39



The COALESCE expression allows for user-specified null-handling. You can use COALESCE to evaluate records for multiple values and return the f irst non-null value encountered, in the order specified. The following requirements apply: • You can specify two or more arguments to COALESCE • Arguments that you specify to COALESCE should all be of the same type • The types of arguments that COALESCE takes are: integer, integer64, double and string • You cannot specify managed attributes as arguments to COALESCE. However, you can specify a standard attribute as arguments to COALESCE resulting in a valid query. • The COALESCE expression can only be used in a SELECT clause, and not in other clauses (such as WHERE) In the following example, all records without a specified price are treated as zero in the computation: AVG(COALESCE(price, 0.0))

COALESCE can also be used on its own, f or example: SELECT COALESCE(price, 0.0) WHERE …



5.1.40





5.1.41



By default, any statement that returns results to the application returns all results. In some cases, it is useful to request results in smaller increments for presentation to the user (such as presenting the sales reps ten at a time, with links to page forward and backward). For example, the following query groups the records by SalesRep, returning the 10th through 19th resulting buckets (order in this case is arbitrary but consistent between queries): DEFINE Reps AS GROUP BY SalesRep PAGE(10,10) Paging can also be used in combination with ORDER BY to achieve “top- k” type queries. For example, the following query returns the top 10 sales reps by total sales: DEFINE Reps AS SELECT SUM(Amount) AS Total GROUP BY SalesRep ORDER BY Total DESC PAGE(0,10)



5.1.42



Additional example for the ORDER BY case You can “ORDER BY” with a “GROUP BY” key.



5.1.43



Statements are processed in a specific order.



5.1.44





5.1.45





5.1.46





5.1.47



Popular Group By fields should be pre calculated in the database or during data ingest rather than at runtime.



5.1.48





5.1.49



When parameter based analysis is added to the Chart component end users see drop down menus across the top of the chart that enable various analytic views to be selected/created on the fly. To create parameter based charts, Power users must first add a parametric analytics statement in the Analytics Statement field and then establish the Metrics, X-options and/or Cross tab choices in their respective fields. The contents of the Metrics Options field display for the end user on the Metrics drop down menu and subsequently on the Y-axis of the Chart component; the contents of the X-Axis Options field display for the end users on the Group By drop down menu and subsequently on the X-axis of the Chart component. The Cross tab axis is best thought of as the z axis and gives you another way to look at the data. Use both the XAXIS and CROSSTAB when you want to give the end user options when doing a GROUP BY of two. To configure a parameter based analytic statement: 1. Use an analytic statement and parameterize it with the parametric tokens 2. Define the different fields that should be used for metrics, x-axis and cross tab.



5.1.50





5.1.51





5.1.52



An analytics dashboard is used when: • Information related to a particular organizational function or role needs to be monitored regularly, possibly in real time. • The information sources represent critical metrics that must be understood at a glance. • The information sources are diverse and complex, requiring some degree of summarization or aggregation to be comprehensible. • Deviations from expected or desired values or thresholds are of particular importance and require immediate investigation or action. • The optimal selection of key information and data is specific to a particular individual or role (or may be customized by the end user). • A view of the overall trends for key metrics is required, providing visibility into the relationship between key values and trends. • Users need access to the above functionality without being obliged to build a complex query. • Users need access to the above functionality without being obliged to undergo extensive training or familiarization with the system.



5.1.53





5.1.54



When conveying analytics in Latitude Studio, your users need a concise, high-level overview of a complex set of information, one that usually focuses on key metrics which represent the status of important organizational entities, processes, or activities as a means of determining potential areas that may require investigation or intervention. A well-designed dashboard allows end users to make better decisions by maximizing the visibility of relevant organizational information and data, thus saving time and increasing productivity. It reduces information overload by summarizing key metrics and trends without requiring the user to interact with the application. It groups and aggregates key information sources so that immediate insight may be obtained at a glance, and discovery of patterns and relationships in the data is maximized. It directs users’ attention to the most important areas, and highlights deviations from expected values so that issues requiring further investigation or action can be immediately identified.



5.1.55





5.1.56



Timings Presentation: ~30 minutes Labs: ~4 hours

Lesson 2 – Analytics Workshop


5.2.1





5.2.2



Star schema = Generic representation of a dimensional model in a relational database. In order for the bulk ingest LDI component to submit records into the MDEX engine it requires a unique identifier, a record specifier, for each record. For this project the record specifier is defined as a transform in the FactSales_SalesTransaction data. It is defined to be FactSales_RecordSpec and is the concatenation of FactSales_SalesOrderNumberFactSales_SalesOrderLineNumber.



5.2.3





5.2.4





5.2.5

Getting Started with Endeca Latitude

Recommend Documents