You are on page 1of 14

Reinvent Your Business with Agile BI Infrastructure

Endeca Whitepaper for the ENTERPRISE

Table of Contents

Executive Summary 2

Traditional BI Systems Are Not Designed For Agility 3

Business Speed and Complexity Require New

BI Capabilities 4

Making Existing Infrastructure Go Faster

Is Just Not Enough 5

A New Architecture for BI Agility 6

Endeca Latitude 8

Organizations Experiencing the Benefits of BI Agility 9

Wall Street Financial Services Firm Increases Amount of

Trade Opportunities Per Day

High-Tech Manufacturer Cuts Millions of Dollars in Supplier Costs

CPG Manufacturer Achieves Single View of Channel, Territory

and Brand Performance

National Police Force Provides Officers with User-Friendly

Search Experience to Catch Criminals

Summary 12

About ENDECA 12
Executive summary
Over the past 20 years, the speed of business and the nature of the information
businesses collect, share, and analyze have changed dramatically. During
that same period of time, the approach to business intelligence (BI) has
essentially been fixed and has largely focused on efficiently generating
reports for a few select individuals within the company. While this reporting
is highly valuable and must continue, it is no longer sufficient.
The traditional approach to BI, with its heavy reliance on data modeling
and its appeal to a few, highly sophisticated users, just is not capable of
responding to new business requirements such as the need to make mid-
term course corrections, to push information to a wider audience, and to
consider a wider variety of data types, including both structured data and
unstructured content. Efforts to employ agile development processes and
to replace existing architecture components with faster versions of the
same components simply do not go far enough. To date these solutions
have had little effect on addressing the backlog of BI requests that builds
as demand for modifications and new analyses increase. In addition, just
making the existing infrastructure faster does not address the need to
incorporate new data types or provide the ease of access that the broader
user-base demands.
To adapt to the new business requirements, organizations should augment
their traditional BI reporting infrastructures with technology designed
to handle the new data integration and presentation requirements. By
adding a specialized database optimized for exploration, integration tools
for unstructured content, advanced search, and visualization capabilities,
IT organizations can both provide their business decision makers with
new insight into critical business problems and dramatically reduce the
BI backlog that has plagued them.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 2

The traditional approach to BI, with its heavy reliance on data modeling and its
appeal to a few, highly sophisticated users, just is not capable of responding to new
business requirements.

Traditional BI Systems Are Not Designed For Agility

The traditional approach to business intelligence (BI) has reached its limits. Over the past
20 years we have developed a set of procedures and technologies that allow us to aggregate,
cleanse, prepare, and report on enterprise data. While there have been incremental
improvements in efficiency and speed over that time, the fundamental approach to BI has not
changed much. During that same period, however, businesses and their information needs
have changed dramatically. Enterprises have transformed themselves from rather isolated
entities, often with an inward focus on operational efficiency, to players in geographically
The value of agility lies in dispersed, multi-partied ecosystems that need to have as much understanding of their
the creation of multiple customer’s interests and partners’ operations as they do their own operational efficiency.
scenario casting, a form of The information they collect, transport, and analyze has changed accordingly and, more
intellectual property. This importantly, so has the speed with which they need to act.
level’s sole concern is with
the future tense or “what Today agility is the goal of most organizations regardless of their industry and it is a top
will happen,” driving an goal of business leaders whether they are in sales, marketing, manufacturing, engineering,
anticipatory, opportunistic customer care, or service delivery. Businesses are competing not just on efficiency, but on
type of decision making as their ability to sense market conditions and quickly respond. That is exactly where BI and
one or more scenarios the business needs have diverged.
come into view.
In many organizations, BI systems are used almost exclusively to generate standard monthly
Business Intelligence or quarterly reports. These reports deliver great value – certainly most organizations
and Decision Impact could not survive without them. But traditional BI systems require an expensive and
Gartner, Inc.
May 2010
time-consuming process to identify all the answers the business users ultimately want,
build a data model that captures that information and unify data from disparate sources
into that data model. Often, multiple cycles of this process are required before the needs
of the business users are fully addressed. More than anything, a traditional BI approach is
designed to structure data, ensure consistency across systems, and efficiently handle large
amounts of structured data – goals that seemingly run counter to the need for agility.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 3

Business Speed and Complexity Require New
BI Capabilities
The demands on BI systems are changing significantly. Technology innovations, such as the
Internet, email, video conferencing, and mobile technologies, have accelerated business
communication across the enterprise and between the enterprise and their customers and
partners. With business moving at a faster pace, the amount of business data generated, the
number of data sources, and the variety of data types has also increased. To make informed
decisions, people often must consider a broader set of data from a variety of systems (see
Figure 1). But because this data is so diverse, it defies conventional modeling. In addition,
the new speed of business has caused consumers, partners, and employees alike to expect
instant access to the information they need any time that they need it.

Diverse Information Sources Create Jagged Data

To make informed decisions, people often must consider a broader set of data from a variety of systems. But
because this data is so diverse, it defies conventional modeling

Traditional BI systems are not designed to meet these new demands. In particular,
traditional BI has difficulty supporting three key trends that represent the future of BI and
enterprise information:
> Increased Importance of Unstructured Content Traditional BI processes have almost
entirely ignored unstructured content, such as word processing documents, email
messages, and even text-based fields on structured forms. The reason – the structured
data of relational databases was the low hanging fruit. It was easy to model and the
content from unstructured documents was not. We convinced ourselves that we could

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 4

With business moving at a faster pace, the amount of business data generated, the
number of data sources, and the variety of data types has also increased. To make
informed decisions, people often must consider a broader set of data from a variety
of systems.

live without knowing what information was captured in the text. But the exponential
growth of the Internet has both increased the importance and the volume of unstructured
text. Especially in consumer-oriented businesses, consumer sentiment as shown in
blogs and Web content, including RSS-based news feeds and social media sites such
as Twitter and Facebook, can no longer be ignored. Yet, traditional BI platforms, with
their focus on structured data and heavy emphasis on modeling have had a difficult time
extracting value from that data and combining it with structured data.
If architected and designed
> The Need to Address Emerging Challenges with Midcourse Corrections The
properly, traditional BI applica-
competitive battle is increasingly being waged on how organizations address new
tions provide deep insights into
questions in the moment. Agile operations mean that businesses can make rapid
past and present events, as well
adjustments to new information and new conditions. Adjustments may be slight or
as help identify patterns in past
major and new questions can arise at any time, whether it is mid quarter, mid-month,
sales, marketing, supply chain,
or even intra-day. Typically these involve new data sources that are being introduced
and financial performance.
every day, and existing sources that are undergoing constant change. The heavy
But in this new age of future-
modeling burden of traditional BI complicates the task of unifying data, often removes
facing business analytics, any
its distinctions, and prevents decision makers from having a clear view of the most
BI user without strong advanced
up-to-date information.
analytics capabilities is seriously
deficient and at a competitive > Decision Making and Enterprise Information Extended to More and Less Technical
disadvantage. Traditional BI Users An agile organization is one that can sense trends and changing conditions and
tools effectively answer past- react quickly. That requires all organizations within the enterprise and all levels of
facing questions such as, “What employees, regardless of their technical skill, to be an information receptor and to either
happened?” and “Why did it be able to feed that information back to the central part of the organization or, ideally,
happen?” But they do not offer respond appropriately on their own. Analytic tools associated with traditional BI have
much assistance with proactive generally been designed for sophisticated power users, which has limited the use of
concerns such as, “What will those tools to only a small fraction of the decision-making population in a company.
happen?,” “What may happen
next?,” and “What could possibly
happen if I take action X versus
Making Existing Infrastructure Go Faster
action Y? Is Just Not Enough
Some organizations have responded to the need for greater agility by improving the scale
Business Intelligence (BI) and speed of their data warehouses, databases, and other elements of their BI infrastructure.
Polishes Its Crystal Ball
Forrester Research
Still others have responded by implementing agile software development processes, such as
September 2009 a Scrum process, to help their IT groups respond more quickly to business requests. While
these changes have resulted in BI systems that can process more data than ever before and
software development teams that can release new reports more quickly, they simply do not
go far enough.

Neither improvement, for example, directly improves decision making agility. Analytic tools
remain too difficult to use for a broad audience of less-technical users and unstructured
content is still ignored. So while the speed of the traditional infrastructure has improved,
it has not improved the decision making capability of the organization because we have
not enriched the data that can be considered for decisions and we have not expanded the
number of people that can access and use the data. Increasingly, this makes the IT group
appear as if they are out of touch with the needs of the business.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 5

Analytic tools remain too difficult to use for a broad audience of less-technical users
and unstructured content is still ignored.

Similarly, neither of these improvements provides a good solution for reducing the data
modeling burden, which is the primary bottleneck restricting the throughput of the IT
organization. In fact, according to some experts, even an agile software development
approach does not address this issue because it calls either for ignoring modeling altogether,
which seems unrealistic, or it calls for very rapid modeling, which, if left as a people-driven
process, is easier said than done.

Without a reliable, streamlined data modeling procedure, IT will continue to have difficulty
keeping pace with the requests from business users. Every report and dashboard that is
created spurs additional questions. “Is this normal?” “Why did it change?” Questions
Even the most powerful, that seem simple turn into significant IT projects if the questions have not already been
advanced, and well-architected anticipated and modeled. Because new questions are created faster than they can be
BI applications have one serious modeled, a backlog of BI requests builds. The BI backlog has a number of undesirable side
weakness: They are only as good effects. For decision makers, it can limit their ability to make rapid, mid-course corrections.
as the underlying data model. For IT managers, it can make them look unresponsive to the needs of the business. It also
Whether relational, multidimen- causes unwanted headaches for IT because it drives the business users to develop their own
sional, or hierarchical, data reporting capability outside of IT’s data governance and security models.
models that form the basis for
all BI applications have to be
designed based on all potential A New Architecture for BI Agility
questions that might be asked Since traditional systems and processes cannot be easily modified to address the need
of the BI application in the future. for agility and the new BI requirements, a new agile architecture is needed to extend the
traditional BI infrastructure. This architecture should be designed for ease of use, so that it
Business Intelligence (BI) can help a broad audience, and flexibility, so that it can support a broad set of analytic tools
Polishes Its Crystal Ball
Forrester Research and easily incorporate new, varied, and changing data sources.
September 2009
This new infrastructure should not replace the legacy BI infrastructure. In fact, it is best
to leverage the traditional BI infrastructure for its strength – historical reporting – and
compliment it with the agile architecture. Over time, the agile infrastructure may prove
that it can also handle the historical reporting task as well, but given the enterprise’s
reliance on historical reporting and the value that is still being derived from the traditional
BI infrastructure, a prudent strategy and one that reduces the risks for the IT staff is to
postpone that decision to later.

The agile infrastructure should include the following components:

> A Specialized Database Designed For Exploration The core component of the new
architecture is a database that is specifically optimized for exploration, not reporting.
The database should accommodate any type of information, including structured data
and unstructured content. The resulting combination will be extremely jagged and
sparse, necessitating a database that does not rely on a pre-determined data model,
but instead derives a data model based on the data presented to it. Finally, to support
on-going exploration from users, the database must be able to process and respond
to rapid-fire queries in sub-second intervals and should be able to accommodate
additions, subtractions, and changes in the data sources with minimal rework.
Specialized databases of this type reduce the need for data modeling and support the
broad exploration and analysis needs of the business users.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 6

> High-Level Tools For Integrating Unstructured and Structured Data The system
should have comprehensive ability to turn structured text into searchable data. That
usually comes by way of advanced text processing and analytics capability that includes
Web crawlers for identifying text-based data on the Web and in corporate data stores,
natural language parsers for identifying important words, phrases and concepts
within those documents, and a metadata model for tagging information so that it can
be quickly found and incorporated into an analysis involving either structured data
or additional unstructured content. In addition, the system should have the ability to
present all data types in a consistent and unified interface to allow business users to
consider the impact of all types of data on their analysis.
> Advanced Search The system should have search capabilities that are intuitive to a
non-technical business user and even an occasional user of the system. Advanced search
capabilities in BI today are far from a simple search box to find dimensions. Instead let
the user search both the data model and the data simultaneously. In addition, they have
features such as auto-correct, spell-check, and faceted search that provide context-
sensitive guidance to the user throughout their search experience.
> Multiple Visualization Options While some analyses are best supported by text-based
responses, others are best addressed through a variety of graphical forms. To answer
the broadest array of questions for the broadest number of users, the agile infrastruc-
ture should support many different forms of visualization and should allow the user to
seamlessly navigate between them.
> A Data Governance and Security Model That is Consistent with Existing IT Controls 
The infrastructure should provide the ability to extend existing IT policies, especially
around data governance and data security, to the agile infrastructure without requiring
substantial additional IT overhead. Adherence to IT standards, such as Web services
and SOA, is, therefore, important to simplify maintenance and allow for rapid integration
of distinct data systems.
> Rapid Provisioning Environment To enable rapid creation of search applications,
the infrastructure should provide high-level tools for designing and modifying Web-
based interfaces. These tools should, at a minimum, eliminate the need for scripting
by providing graphical drag and drop interfaces. Ideally they would provide interfaces
that are designed for end users or power users and, therefore, if desired, would allow
IT to offload the interface development or maintenance work to less-technical people
within the organization.

With this architecture, IT is able to quickly set up applications where business users can
fully explore varied data and can answer questions they may not anticipate at the outset
of their exploration. This brings agility to decision making and also reduces the BI backlog
because it reduces the number of BI requests sent to IT.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 7

Endeca Latitude reduces the time, effort, and cost of creating BI applications by
enabling IT to quickly and easily combine data and content of any type, format or source
without complex modeling, integration or app development, even as the data changes.

Endeca Latitude
Endeca solves the new BI requirements around agility with Endeca Latitude – a new
product that combines the simplicity of search with the power of BI.

Endeca Latitude is used to create BI applications that present users with wide-ranging
data sources and data types and provide users with discovery capability – an ability to
ask new questions that lead to better outcomes. Because of their breadth and flexibility, a
single application designed with Endeca Latitude can take the place of literally hundreds
of traditional BI reports. Just as importantly, applications developed with Endeca Latitude
embody the best search technology so business users can productively use them with
essentially zero training.

IT organizations also benefit from Endeca Latitude. Endeca Latitude reduces the time,
In the case of Endeca, based effort, and cost of creating BI applications by enabling IT to quickly and easily combine
on XML architecture, data, data and content of any type, format or source without complex modeling, integration or
metadata, and applications app development, even as the data changes. In addition, Endeca Latitude can enhance a
(reports and dashboards) are traditional data warehouse and reporting infrastructure by offering a cost-effective way to
actually one and the same. deliver applications that answer unanticipated, in-the-moment questions.

Endeca Latitude includes the following major components:

Agile BI Out Of The Box
Forrester Research > MDEX Engine™ The core of Endeca Latitude, the MDEX Engine, is a hybrid search-
April 2010
analytical database that allows users to explore data and content in an unconstrained
and impromptu manner and to rapidly address new questions that inevitably follow
every new insight. In the engine, there is no overarching schema for the data. Instead
every record, every document has its own schema. The MDEX Engine has the ability to
index specific data elements in great detail, including the content within unstructured
documents, and can use that information to automatically identify relationships
between records and documents based on shared attributes. In addition, the MDEX
Engine also supports the rapid inclusion of new data sources or changing data because
those changes update the relationship between data elements, but do not require the
entire model to be reconstructed. These capabilities allow Endeca Latitude to bring
together diverse datasets – even structured data and unstructured content – without
complex modeling and integration. The engine then conducts search, navigation, and
analytics on this jagged data. In addition, the MDEX Engine supports IT-managed data
governance and data security, such as promoting IT policies and support for security at
the application, view, record, and property levels. With support for industry standards
such as XQuery, XML, and SOA, the MDEX Engine is designed to fit easily into existing
enterprise architectures.
> Discovery Framework Provides a library of components and visualization tools
that embody advanced search capabilities, such as Guided Navigation® and multi-
faceted exploration, and enable the rapid configuration of Discovery applications. The
Discovery Framework also provides a set of comprehensive management tools. With
the Discovery Framework, you can respond to the requirements of a changing business
by re-designing reports and dashboards in minutes, incorporating diverse data sources
in days, and deploying entirely new applications in just a few weeks.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 8

> Data Integration A collection of Web content crawlers, and file system crawlers, and
connectors to best of breed ETL tools such as Informatica. These tools enable rapid
integration of both structured data and unstructured content into the MDEX Engine
and allow for rapid data updates to ensure that the MDEX Engine accurately reflects
the source data.

Organizations Experiencing the Benefits of BI Agility

Numerous clients are using Endeca technology to add agility to their operations and decision
making. By combining unstructured and structured content in a single view into their decision-
options, these organizations are discovering new relationships in the data and supporting a
broad range of analyses. Endeca’s customer base represents many different industries each
utilizing the technology to satisfy unique business intelligence challenges, including:
> A large financial services firm that needed to quickly model investment options and
increase the number of trade opportunities per day.
> A high-tech manufacturer searching for a way to combine data from multiple repos-
itories after a large-scale acquisition in order to streamline component selection for
their products.
> A fortune 100 consumer packaged goods seeking to obtain an integrated global view
of sales information to make the most effective product placement and promotion
decisions possible.
> A national police force which required a zero-training user interface for all of their
officers so they could quickly interact with criminal event data to discover unknown
relationships to help identify perpetrators, prevent crime and protect their citizens.

Wall Street Financial Services Firm Increases Amount of Trade

Opportunities Per Day
This financial services organization, specializing in investments, realized it had a
significant barrier to productivity and sales. The dramatic increase in the amount of
information available on a daily basis with regard to market changes and investment
options made it difficult for brokers to keep up with in time to help their clients make
informed investment decisions quickly. In addition, each client holds different investment
interests, so pairing up potential good investment options with the right client demographic
was a cumbersome manual process. Finally, because this process was so difficult, brokers
tended to focus on their larger clients which left many investment opportunities with
smaller clients off the table.

The organization realized there was a need for a new information system that enabled
brokers to quickly see important changes in the market and match those changes up
with the client who, based on their investment profile, would be most interested in that
information. Endeca Latitude was the only technology capable of combining structured
information such as stock tickers with unstructured information such as news reports, and
providing one navigable view of that data.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 9

Today, through a combination of Guided Navigation and visualization options, brokers
are able to quickly identify information that is likely to trigger an investment from specific
customers. By adopting an agile approach to analytics and reporting, this organization has
made hundreds of millions of dollars in incremental sales with smaller clients that were
previously off the table.

High-Tech Manufacturer Cuts Millions of Dollars in Supplier Costs

On the heels of a large-scale acquisition, this high-tech manufacturer’s supply chain
organization was given a mandate to locate one billion dollars in savings for the
organization. The product line they acquired was similar to their own, and they needed to
continue manufacturing the acquired product line, but streamline the set of suppliers they
used. They knew this was possible as the components that made up these machines were
similar in specification, but they did not have a way to combine all of the data across the
organizations to figure out what could be rationalized.

To accomplish the necessary reduction in costs, this manufacture needed an innovative

process to enable its staff to analyze product characteristics at the engineering attribute
level in order to identify opportunities to standardize similar components and optimize
design. Endeca Latitude was able to provide a unified view of data and analytics across a
multitude of data sources that include large scale ERP, engineering and BI systems. With
innovative visualization tools, such as bubble charts and scatter plots, the organization
could identify similar components and appropriate substitutions. This organization has
already been able to achieve millions of dollars in savings through things such as 60%
reduction in amount of steel coils used, to 74% reduction in amount of valves needed, to
77% reduction in number of brand badges needed.

Combining Search and BI Finds Component Supplier Savings

The Endeca Latitude application

showcased above is a generic
example of what an organization
looking to streamline design and
sourcing costs would utilize. It is
showcasing ways for analyzing
total spend by a variety of facets
such as part type, specific
supplier or products.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 10

CPG Manufacturer Achieves Single View of Channel, Territory and
Brand Performance
This leading CPG manufacturer sells many different products and brands globally, which
has resulted in numerous information sources located in a variety of systems. The
organization had not been able to achieve a single view of all product line sales that also
showcased which channel, territory and brand those sales came from. They were also
trying to tap into sources outside of their organization such as consumer blogs to help bring
that information into their decision making process, but had no way to incorporate that
type of content into their analysis. Furthermore, the audience that needed to access this
information was every day business users – marketing, sales and brand managers – who
were not highly technical and required a user-friendly interface for exploring all of this

With Endeca Latitude this organization was able to provide 13,000 users with a zero-
training interface that utilizes the ease of search through Guided Navigation and helpful
visualizations, such as heat maps, for easily analyzing regional sales volumes and promotion
performance. They now have clear visibility into sales performance by brand, product,
distributor, campaign and even dynamic product groupings to analyze sales performance.
This has enhanced understanding of why certain promotional programs are more effective
so that those efforts can be repeated.

National Police Force Provides Officers with User-Friendly Search

Experience to Catch Criminals
This police force had the challenge of anticipating and preventing crime within its jurisdiction
with minimal budget increases. They needed a powerful method for correlating data from
multiple data sources to enable their staff to respond more effectively to incidents, and also
proactively discover emerging crime trends. In addition to their every day crime prevention
needs, there was an international event being hosted in one of their largest cities in the near
term. This made it essential that they find a technology that would be deployable in a short
timeframe to help their force protect the influx of people for that event.

By implementing Endeca Latitude, this customer has a single search interface across
multiple operating systems that includes structured and unstructured content from sources
such as the local newspapers. The application enables proactive discovery of emerging
trends and threats through correlating diverse data to showcase patterns in people, objects,
locations and events. Police officers using this application on a daily basis are able to
discover the important correlations in data that help identify and apprehend perpetrators,
prevent crime, and protect citizens.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 11

A Single Search Interface for Criminal Activity Helps Anticipate Future Incidents

The Endeca Latitude application

shown here is a generic example
of what a defense organization
may use to analyze criminal
activity. It is showcasing ways
for analyzing specific public
transportation locations by
amount of crimes as well as
station usage.

Today most organizations seek to have more agility in their operations and their decision
making. These goals are driving new requirements of the BI infrastructure, such as the
need to expand the number and nature of BI users and the need to easily integrate a greater
volume and variety of enterprise data. Traditional BI has struggled to support the level
of agility required. Simply improving the speed of the IT organization or the existing BI
infrastructure is not enough to address the new requirements. To improve the agility of the
BI infrastructure, organizations must add new technology to their existing BI infrastructure,
such as a specialized database that can lessen the data modeling effort and search tools that
can be used without any training. The agile BI infrastructure promises to reduce the BI
backlog and simultaneously provide users with greater analysis and exploration capability
to improve countless decisions made daily across an enterprise.

About Endeca
Endeca is a search and business intelligence software company that improves daily
decisions for employees and purchase decisions for customers. Endeca’s unique innovation,
the hybrid search-analytical database, enables IT organizations to deliver solutions with
consumer ease-of-use on diverse and changing information. Endeca solutions drive
hundreds of millions of dollars in reduced costs and increased revenues in the world’s most
demanding decision-making environments in organizations like Boeing, Cox Newspapers,
the U.S. Defense Intelligence Agency, Ford Motor Company, Hyatt, IBM, the Library of
Congress, Texas Instruments, and

Headquartered in Cambridge, Mass., Endeca has operations in North America, Europe,

and Asia.

ENDECA TECHNOLOGIES  |  Reinvent Your Business with Agile BI Infrastructure 12

© 2010 Endeca Technologies, Inc. All rights reserved. Endeca and Guided Navigation are registered trademarks, and MDEX Engine and Guided Summarization are trademarks of Endeca Technologies, Inc. All other product
and service names mentioned herein are or may be registered trademarks or trademarks of their respective companies or organizations.

ENDECA 101 Main Street, Cambridge, MA 02142 USA | T 617.674.6000 | F 617.674.6001 | |
Follow us on Twitter: | Read our blog: