DATA MANAGEMENT & WAREHOUSING DAVID M WALKER

CYPRUS 11

AGILE BUSINESS INTELLIGENCE

AGILE MANIFESTO
We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

CYPRUS 11

Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more. http://www.agilemanifesto.org/
14-Oct-2011 2011 Data Management & Warehousing

Utah, Feb 2001

2

TWELVE PRINCIPLES OF AGILE SOFTWARE

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Business people and developers must work together daily throughout the project. Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Working software is the primary measure of progress.

CYPRUS 11

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. Continuous attention to technical excellence and good design enhances agility. Simplicity--the art of maximizing the amount of work not done--is essential. The best architectures, requirements, and designs emerge from selforganizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

14-Oct-2011

2011 Data Management & Warehousing

LARGE ORGANIZATIONS FORGET

CYPRUS 11

Most companies start with an innovative and entrepreneurial people and processes that get things done they are inherently agile As organizations grow they put structures in place that standardize the organization but these limit creativity, increase timescales and reduce risk at the cost of reducing benefits and reward
14-Oct-2011 2011 Data Management & Warehousing 4

THE PROCESS IMPROVEMENT PARADOX

CYPRUS 11

The paradox () is that process improvement is good, but process improvement programs aren't, or at least they often aren't. Organizations become more and more averse to risk as they "mature. An organization under the gun to demonstrate increased CMM level is not going to go looking for real challenge. This () entices the play-it-safe behavior of low-risk, and therefore low-benefit, projects.
Peopleware; Tom De Marco & Timothy Lister; 1987
14-Oct-2011 2011 Data Management & Warehousing 5

SUCCESSFUL LARGE ORGANIZATIONS RECOGNIZE AND REACT

SkunkWorks

CYPRUS 11

Lockheed Martin Advance Development Projects A group within an organization given a high degree of autonomy and unhampered by bureaucracy, tasked with working on advanced projects. Responsible for U-2 & Blackbird Spy Planes, Stealth Fighter & Bomber Founded in June 1943 specifically to overcome the hurdles of government and large corporations working together SkunkWorks has 14 rules and practices
Very Similar to the 12 Agile Principles

Ben Rich & Leo Janos; SkunkWorks; 1996

14-Oct-2011

2011 Data Management & Warehousing

OTHER EXAMPLES

CYPRUS 11

In the late 80s/early 90s Telcos led the way in Agile BI development
Vodafone: Mast Placement Project Swisscom: Geo-located SMS Marketing

But they have been overtaken by the search and social networks websites that rely on rapidly creating and consuming BI data to survive
Examples: Facebook, Google & LinkedIn These companies are also creating and using the next generation of BI tools and BI engines
14-Oct-2011 2011 Data Management & Warehousing 7

AGILE BI IS NOW A 2-SPEED HIGHWAY

Truly Agile BI Organizations

CYPRUS 11

Small, strong, highly skilled teams with a strong, trusting, focused user relationship Project team having delegated authority and responsibility for delivery with long term funding The ability to operate outside standard corporate procedures (e.g. procurement) Regular delivery of incremental improvement

Those that describe themselves as wanting to be more agile

Willing to adopt some of the methods but unable to break free from the corporate chains
14-Oct-2011 2011 Data Management & Warehousing 8

CYPRUS 11

TECHNIQUES

BECOMING MORE AGILE

14-Oct-2011

2011 Data Management & Warehousing

HOW TO BE MORE AGILE

14-Oct-2011

CYPRUS 11

Team Structures Wiki-fy Everything Use A Dedicated Platform Build Literal Staging Areas Throw out your ETL Tool? More Dynamic Reporting Tools Develop reports with your users Embrace (not so) New Technologies
2011 Data Management & Warehousing 10

TEAM STRUCTURES

CYPRUS 11

Agile teams are generally smaller but with broader and deeper skill sets
These resources are more expensive individually but cheaper collectively Smaller teams significantly reduce the management and communication overhead Close communications and broad skills are more likely to generate innovative solutions
14-Oct-2011 2011 Data Management & Warehousing 11

WIKI-FY EVERYTHING
Projects need a consistent, persistent, versioned knowledge store Use a Wiki and train your business users how to use it Optimizes the documentation set collected in a single structured way Abandon office products and SharePoint for critical documents Dont store your requirements in e-mails, IM chats and documents
14-Oct-2011 2011 Data Management & Warehousing

CYPRUS 11

12

USE A DEDICATED PLATFORM

CYPRUS 11

Marketplace has a choice of BI specific, low maintenance, low TCO appliances

IBM Netezza Teradata 14 (AsterData) Sybase IQ

And some high power workhorses

Oracle Exadata Teradata

There are plenty of emerging technologies

Curt Monash: http://www.dbms2.com/
14-Oct-2011 2011 Data Management & Warehousing 13

BUILD LITERAL STAGING AREAS

CYPRUS 11

These are 1:1 copies of your source systems (hence the Literal) Do this whilst others are collecting requirements, building data models, etc. Do some test reporting off the LSA with the users it starts the user engagement, helps evolve the business requirements, and develops communication They will be an essential source for the data warehouse as it evolves and remove load from the operational systems
14-Oct-2011 2011 Data Management & Warehousing 14

THROW OUT YOUR ETL TOOL ?

Value add features are under-utilized Developers and DBAs use them as glorified scheduling tools

CYPRUS 11

ETL tools require product specific expertise

Good Source Code Control and Management Scripting can compensate for much of the lost benefits Significantly reduces the project cost and increases the available skilled resources InsureTheBox/Netezza; YapiKredi/SybaseIQ; NonDisclosure/Greenplum have major data warehouses with no ETL product
14-Oct-2011 2011 Data Management & Warehousing 15

MORE DYNAMIC REPORTING TOOLS

Reports on your hand held device
Qlikview, RoamBI, PushBI, Tableau

CYPRUS 11

Use Dynamic Web Deployed Tools

Panopticon, Tableau

RSS Feeds
Usable by any RSS Reader on any platform

Requires dedicated, reactive reporting experts that work enthusiastically with the business users and are willing to go the extra mile
14-Oct-2011 2011 Data Management & Warehousing 16

DEVELOP REPORTS WITH YOUR USERS

CYPRUS 11

De La Rue / Tableau Reporting First 50 reports deployed over 28 days Initially deployed to users from the LSA and then migrated to the data marts Near real-time (fifteen minutes maximum delay) data

Averages
Day 5: 17 changes / 5 users Day 10: 1 change / 42 users Day 15: 0 changes / 50 users
14-Oct-2011 2011 Data Management & Warehousing 17

EMBRACE (NOT SO) NEW TECHNOLOGY MAPREDUCE/HADOOP/NOSQL

CYPRUS 11

Software frameworks that supports data-intensive distributed applications and enables them to work with thousands of nodes and petabytes of data. Hadoop is a filesystem (HDFS) and distributed programming framework (MapReduce) whilst a NoSQL database consists of key-value pairs, no joins, data is sharded and replicated, no single point of failure. Notable BI Use (but not the only one): Rapid loading and processing of volume data to create and validate data sets for onward inclusion in the data warehouse whilst doing large scale, near real-time dirty analysis Notable Users: Facebook, Twitter, LinkedIn, Yahoo!, eBay, Amazon, Google Technologies in use since at least 2004

14-Oct-2011

2011 Data Management & Warehousing

18

EMBRACE (NOT SO) NEW TECHNOLOGY COMPLEX EVENT PROCESSING (CEP)

CYPRUS 11

Complex event processing (CEP) consists of processing many events happening across all the layers of an organization, identifying the most meaningful events within the event cloud, analyzing their impact, and taking subsequent action in real time. Already used for fraud and network management but has applications in customer service and market to fine tune customer interactions in real time Notable BI Use: real-time information alongside existing BI rather than trying to create an entire real-time solution Technologies in use since at least 1999
14-Oct-2011 2011 Data Management & Warehousing 19

BUT REMEMBER
Agile is all about approach and people:
Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan Your management need to be willing to create: A group within an organization given a high degree of autonomy and unhampered by bureaucracy, tasked with working on advanced projects.

CYPRUS 11

14-Oct-2011

2011 Data Management & Warehousing

20

REFERENCE MATERIAL
Websites
Agile Manifesto
12 Principles of Agile

CYPRUS 11

SkunkWorks
14 Rules & Practices

Hadoop and NoSQL Myth-busting

Books
Peopleware: Productive Projects & Teams SkunkWorks: A Personal Memoir Mythical Man Month: Essays on Software Engineering
14-Oct-2011 2011 Data Management & Warehousing 21

THANK YOU

CYPRUS 11

AGILE BUSINESS INTELLIGENCE