Inside the Oracle BI Server (10g

)
Mark Rittman, Director, Rittman Mead ODTUG Kaleidoscope 2010, Washington D.C.
T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com

Introductions
• Oracle BI&W Architecture and Development Specialist • Co-Founder of Rittman Mead Consulting ‣ Oracle BI&W Project Delivery Specialists • 10+ years with Discoverer, OWB etc • Oracle ACE Director, ACE of the Year 2005 • Writer for OTN and Oracle Magazine • Longest-running Oracle blog ‣ http://www.rittmanmead.com/blog • Ex-Chair of UKOUG BIRT SIG • ODTUG Director • Editor of Oracle Scene • Co-Chair of ODTUG BI&DW SIG • Speaker at IOUG and BIWA events • Ex-member of the Blue Man Group *
* Not technically true T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com

Inside the Oracle BI Server
• In this session, we will look in-depth at the Oracle BI EE Oracle BI Server • What components are in the BI Server • What is the Navigator, and how does it turn incoming logical requests into optimized physical SQL and MDX • What happens when it processes a query • What happens when the BI Server joins resultsets in memory • And what influences SQL generation, join strategy and BI Server resource usage

T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com

Why Is This Topic Important?
• Now this could be a pretty boring topic • But its one of the most important parts of the BI EE technology stack • It provides all of these key features ‣ Query Federation ‣ Heterogeneous Connectivity ‣ The Semantic Model ‣ Aggregate Navigation ‣ Time Travel (sort of) • So what really happens when Answers fires off a query, and the BI Server goes and gets the data...?

T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com

The Oracle BI EE 10g Server Architecture

T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.com

rittmanmead.The Oracle BI Server 10g Architecture • Logical SQL interface via ODBC and JDBC • Intelligent Request Generator.com W: www.com . made up of ‣ Logical request generator ‣ Navigator ‣ Optimized Query Rewrites ‣ Execution Engine ‣ Cache Services • Other services include ‣ Load balancing (in conjunction with cluster controller) ‣ Query governance ‣ Session Management ‣ System / Perf Monitoring (JMX etc) T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.

rittmanmead.Intelligent Request Generation and the BI Server Navigator • The Navigator is the key component that parses incoming requests and generates the relevant physical SQL and MDX queries • Invoked if cache cannot be used • Handles the following translations ‣ Function push-down ‣ Multi-pass logic (time series etc) ‣ Aggregate navigation ‣ Fragmentation navigation • Responsible for generating optimized SQL and MDX for the mapped data sources • Uses rules and definitions present in the RPD T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com .com W: www.

the BI Server will functionally compensate by calculation. joining data sources in memory Parse Incoming Logical SQL Generate Optimized Physical SQL(s) Receiving Incoming Resultsets Apply any required functions and joins Return data to user T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.rittmanmead.com .The BI Server Functional Compensation Model • The BI Server tries to push functions and joins down to the underlying databases • Where not possible.com W: www.

com .com W: www.rittmanmead.How Does The BI Server Process Queries? • • • • • • So what happens when the BI Server processes an incoming query? What are the steps involved in translating logical requests to physical queries? What does the Navigator do? What rules and statistics does it use? How can we influence the physical queries that it generates? How can we tell what is going on? T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.

rittmanmead. against Semantic Model • Passed to Intelligent Request Generation function T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.com .Step 1 : Request Arrives at Intelligent Request Generator • Query arrives from Answers or other ODBC tool.

rittmanmead.com W: www.com .Step 2 : Request Passed to Logical Request Generation Engine • Request passed to Logical Request Generation engine. request requires multiple columns and a derivation (calculation) T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead. responsible for generating the one or more logical SQL required to satisfy the query • In this case.

rittmanmead.com W: www.com . nested formulas etc T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead. Cache Services is checked for the result • Avoids calls to physical databases for frequently-requested data • Intelligent checking of cache including derivation. projection.Step 3 : Check Cache if Enabled • If caching is enabled.

com .Rules for a Cache Hit • Caching is enabled (CACHE=Y in the NQSConfig. being either the same or more aggregated than the cached query • No further aggregation (for example. the cached copy has to use this attribute as well • Aggregation levels have to be compatible.INI file). RANK) can be used in the incoming query • Compatible ORDER BY clause T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead. • WHERE clause is semantically the same or a logical subset of a cached statement. • All of the columns have either to be present or derivable from cached copies • It has equivalent join conditions • If DISTINCT is used.

Step 4 : Request Passed to the Navigator • Navigator is responsible for navigating the semantic model and generating the execution plan.com W: www.com .rittmanmead. and most optimal physical SQLs. for the request • Contains multiple functions that are used dependent on the nature of the request T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.

rittmanmead.com .Step 5 : Multi-Pass / Sub-Request Logic • • • • Analyzes request and works out the complexity of the query Determines whether multi-pass logic is required (time-series etc) Works out whether one request depends on the results of another (sub-request) Navigator uses this function to determine the most efficient way to satisfy the request T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.

rittmanmead.com . the Fragment Optimization function will transform base-level logical SQL into logical SQL fragments • Also automatically resolves fan-trap issues T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.Step 6 : Fragment Optimization • When a measure is fragmented.com W: www.

Step 7 : Aggregate Navigation • Aggregate Navigation checks aggregates mapped in RPD for relevance • Execution Plan will be written to use aggregates if they return less rows than the default logical table sources T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.com .rittmanmead.

Step 8 : Optimized Query Rewrite.com . and pass to Execution Engine • Optimized Query Rewrite checks database capabilities for function shipping and ability to join subqueries • Generates optimal physical SQL or MDX for relevant data sources • Passed to Execution Engine for transfer to physical databases T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.

Step 9 : Copy Back to Cache Services. results are copied back to Cache Services T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.rittmanmead.com W: www.com . if Enabled • Results are returned to calling application (Answers etc) • If caching is enabled.

com W: www.INI matrix T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.rittmanmead. based on DBFunctions.com .Key Navigator Functionality • • • • • • BI Server will pick one or more LTSs at run-time based on query context Generates an execution plan Will try and generate as few physical SQL as possible Will try to push down functions if possible Will try to push down joins if possible Will write the optimal SQL or MDX for the data source.

com .com W: www. and the physical SQL / MDX generated for the request T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead. step-by-step row counts.rittmanmead.Level 5 Query Logging and Logical Execution Plans • The execution plan generated by the Navigator can be viewed when Level 5 logging is enabled • Shows the execution plan.

rittmanmead.com W: www. you don’t notice this because they are pushed down to the DB • For more complex queries. the BI Server may need to perform the joins instead T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.BI Server Joins • (Almost) Every request requires joins • For simple queries.com . the BI Server may need to get involved ‣ Where more than one physical database provides data for the request ‣ Where multi-pass logic is used ‣ Where measures are fragmented ‣ Where more than one fact table is used in the request • Basic principle is that the BI Server tries to push down all joins • Depending on the database type and version and nature of request.

com W: www. fragmentation. federation etc) T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.rittmanmead.com .The BI Server Source-to-Target Metadata Model • Facts and dimensions are mapped to physical sources Logical Table Sources • LTS are chosen at run time based on RPD definitions (level mapping.

So What Does The BI Server Do.com W: www. When It Joins Data In Memory? • So just what goes on when the BI Server joins data in memory? • How can we tell when the BI Server has performed the join? • Why does the BI Server sometimes do the join. and sometimes get the database to do it instead? T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.rittmanmead.com .

joined by a logical join • Both logical tables have single LTS.com . which both point to tables in the same physical database • Physical tables are joined using a physical foreign key Single Database LTS Mapping Products Logical Dimension Sales Logical Fact LTS Mapping Products Physical Table Sales Physical Table T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.rittmanmead.com W: www.BI Server Join Type 1 : Fact + Dimension from Single Database • Single logical fact and dimension table.

PR O D_SUB CATE G O R Y_DESC] [ fo r da t abase 3023:2820.PR O D_ID [ fo r da tabase 0 :0 ] GroupBy: [PR O D U CTS.44 ] PR O D U CTS.QUANTITY_S OLD by [ PRO D U CTS.rittmanmead.44 ] OrderBy: c1 asc [ fo r database 3023:2820.44 ] .PR O D_SUBCATE G O RY_DES C] ) as c2 GB [ fo r database 3023:2820.PR O D_SUB CATE G O R Y_DESC as c1 GB [ fo r da tabase 3023:2820.PR O D_ID = T2911. sum(T2911.44] Chi ld Nodes (RqJoinSpec) : <<2136>> [ for database 3023:2820:orc l3 .44] PR O D U CTS T2874 SALES T2911 Deta i lF i l te r : PRO D U C TS.PR O D_ID = SALES. T2911 w here ( T2874.Single Source Fact and Dimension Execution Plan & SQL • BI Server generates execution plan with single logical SQL including join clause RqLis t <<2105>> [ fo r database 3023:2820:orc l3 .com .com W: www.PR O D_ID ) group by T2874.PR O D_SUB CATE G O R Y as c1 .PR O D_SUBCATE G O RYorder by c1 SALES T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.44] • Single physical SQL with join pushed down to database se lec t T2874. sum(SALES.QUANTITY_S OLD) as c2from PROD U CTS T2874.

BI Server Join Type 2 : Fact + Dimension from Two Databases • Single logical fact and dimension table.com .rittmanmead.com W: www. which point to different databases • Physical tables are joined using a physical foreign key Database #1 LTS Mapping Products Physical Table Database #2 Products Logical Dimension Sales Logical Fact LTS Mapping Sales Physical Table T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead. joined by a logical join • Both logical tables have single LTS.

QUANTITY_S OLD by [ SALES.44 ] PR O D U CTS T2502 OrderBy: c2 asc [ fo r database 3023:2500.com .44 ] .rittmanmead.44] Chi ld Nodes (RqJo inSpec) : <<2504>> [ fo r database 3023:132:orc l .PRO D_ID as c2 [ fo r da tabase 3023:132.44] OrderBy: c2 asc [ fo r database 3023:132.44 ] .0 ] ) as D1 OrderBy: c1 asc [ fo r database 0 : 0.PRO D_ID] [ for da tabase 3023:132.c2 = D902.44] ) as D902 OrderBy: c2.PRO D_ID] ) as c3 [ fo r da tabase 3023:132. actual j oin vectors: [ 0 ] = [ 1 ] ( RqLis t <<2517>> [ fo r database 3023:2500:orc l2 .44 ] PR O D U CTS.44 ] SALES. RightOuterJoin and FullOuterJoin ( RqLis t <<2495>> [ fo r database 3023:132:orc l .com W: www.44] Chi ld Nodes (RqJo inSpec) : <<2523>> [ fo r database 3023:2500:orc l2 .PR O D_ID as c2 [ fo r da tabase 3023:2500. sum(SALES. PR O D U CTS.PR O D_SUB CATE G O R Y_DESC as c1 GB [ fo r da tabase 3023:2500.44 ] SALES T211 GroupBy: [SALES.Multiple Source Fact and Dimension Execution Plan • Excerpt of plan (source rqList cropped) • Two logical SQL blocks with InnerJoin between fact and dimension tables ‣ Can also be LeftOuterJoin.0 ] T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.44 ] ) as D901 InnerJoin <<2492>> O n D901. c3 [ fo r database 0 : 0.c2.

PR O D ID as c2.Q U A N TIT Y S OL D) as c3 f ro m Send ing query to da tabase na med orc l2 ( id : <<6074>>) : se lec t T2502.Send ing query to database named orc l( i d : <<6057>>) :se lec t T211.P R O D_ID as c2 f rom PRO D U C T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.PR O D_SUB CATE G O R Y as c1 .rittmanmead. T2502.com W: www.com . su m(T211.Multiple Source Fact and Dimension Physical SQL • Two physical SQL queries sent to the two databases • BI Server joins the two sources in-memory using a sort-merge join algorithm .

com W: www.BI Server Join Type 3 : Fact + Dimension Using Driving Table • • • • Single logical fact and dimension table.com . joined by a logical join Two tables typically LTS mapped to different physical databases One of the logical tables defined as a driving table in the BMM logical join Driving table uses Parameterized Nested Loop Join (PNLJ) to “probe” other table Database #1 Products Logical Dimension LTS Mapping Database #2 Products Physical Table PNLJ Probe Lookup Sales Logical Fact LTS Mapping Sales Physical Table T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.rittmanmead.

federated queries will cause both tables to be loaded into the BI Server memory. and then joined (as per previous example) • Can be expensive if one or more of the tables are large • An alternative is to designate one of the tables as the “driving table” ‣ Needs to be small.rittmanmead. typically <1000 rows • This table is then queried and filtered first.com .Driving Table Considerations • By default.com W: www. then it’s remaining rows are then applied as a filter to the second table • Driving table is defined in the Logical Join dialog T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.

Start reading rows from the driving table Submit a parameterized query request to non-driving table Bind values from the drive-table rows into the parameterized query and fetch results Repeat until all rows from drive table are processed PR O D U CTS. :PARA T2502. :PA R A M 2.PRO D_ID as c2 .com W: www.PR O D SUB se lec t T2502. :PA R A M 15.com .PR O D_ID in ( :PA R A M 1.Parameterized Nested Loop Join Algorithm & Physical SQL 1. :PARAM14.QUA NTITY_S OLD) as c3f rom SALES T211where (T211. 3. sum(T211. se lec t T211. :PARA M 19.PR O D_SUB CATE G O R Y as c1 . :PARA M10. :PAR A M 16. :PA R A M 3. :PARA M5. :PARA M7. :PARA M 13. RqLis t QUANTITY_S OLD: [DAggr (SALES.:PARA M18. :PARAM20) ) g roup by T211.rittmanmead. :PA R A M 11. 2.:PARA M12. :PARA M6. :PA R A M 17.P R O D_ID order by c2 T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.QUANTITY S O LD by [ PRO D U CTS. :PARA M4.PRO D_SUBCATE G O RY as c1 GB. 4. :PARA M9.PR O D_ID as c2 f rom PRO D U CTS T2502order by c2 M8.

46 ] OrderBy : c2 asc [ fo r da t abase 3023:132.com W: www.PRO D_SUBCATE G O R Y as c1 GB [ fo r da tabase 3023:2500.44] .P R O D _ID as c2 [ fo r da tabase 3023:132.P R O D _ID = ?1 or SAL E S.PR O D_ID = ?13 or SALES.c2 .PR O D_ID = ?20 [ for database 0:0] GroupBy: [SALES.PR O D_ID = ?17 or SALES.PRO D_ID as c2 [ fo r da tabase 3023:2500.46] .com .PR O D_ID = ?9 or SALE S . su m(S AL E S.PRO D_ID = ?4 or SALE S.PRO D_ID] ) as c3 [ fo r da tabase 3023:132.PR O D_ID = ?7 or SA L E S.46 ] ) as D 902 T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.PR O D_ID = ?10 or SALE S.PR O D_ID = ?6 or SALE S.Q U A N TITY_S O L D by [ SALES.46 ] SALES T211 Deta i lF i l ter : SAL E S. PROD U CTS.c2 = D901.46 ] Ch i ld Nodes (RqJo inSpec) : <<13676>> [ f or da tabase 3023:132:orc l .PR O D_ID = ?14 or SA L E S.44 ] ) as D901 InnerJoin ( l ef t dr ive) <<13689>> On D902.PRO D_ID] [ fo r database 3023:132.PR O D_ID = ?5 or SALE S .PRO D_ID = ?12 or SALES.44 ] Ch i ld Nodes (RqJo inSpec) : <<13658>> [ f or database 3023:2500:orc l 2.P R O D _ID = ?15 or SAL E S. ac t ua ljo in vec t ors :[ 1 ]= [ 0] ( RqL is t <<13673>> [ for da tabase 3023:132:orc l .rittmanmead.PR O D_ID = ?8 or SALE S.PR O D _ID = ?16 or SALES.44 ] PR O D U CTS T2502 OrderBy: c2 asc [ fo r da tabase 3023:2500.44 ] PR O D U CTS.P R O D _ID = ?2 or SAL E S.PR O D_ID = ?11 or S A L E S.PR O D_ID = ?19 or SALES.PR O D_ID = ?18 or SALES.P R O D _ID = ?3 or S A L E S.46 ] SAL E S.Fact and Dimension Driving Table Execution Plan • Excerpt of plan • Note InnerJoin (left drive) and parameterized DetailFilter in plan ( RqL is t <<13655>> [ for da tabase 3023:2500:orc l2 .

rittmanmead.com W: www.BI Server Join Type 4 : Two Facts Using a Single Database • Query requires data from two fact tables.com . to avoid “fan trap” • Join is pushed down to the database if possible Single Database LTS Mapping Costs Logical Fact Products Physical Table Costs Physical Table Products Logical Dimension LTS Mapping Sales Logical Fact LTS Mapping Sales Physical Table T : +44 (0) 8446 697 995 E : enquiries@rittmanmead. sharing a common dimension • BI Server creates two logical queries and “stitches” results together.

PRO D_SUBCATE G O RY_DESC as c2 GB [ f or da tabase 3023:2820.46 ] Ch i l d Nodes (RqJo inSpec): <<3544>> [ fo r da tabase 3023:2820:orc l3 .46 ] ) as D1 Ful lOuterSt i tchJoin <<3565>> On D1.PR O D_SUBCATE G O R Y_DESC] ) as c1 GB [ f or da tabase 3023:2820.PRO D_SUBCATE G O RY_DESC as c2 GB [ f or da tabase 3023:2820.c2 ( RqL is t <<3511>> [ fo r da t abase 3023:2820:orc l 3.PR O D_ID = SALES. PR O D U CTS.46 ] Ch i l d Nodes (RqJo inSpec): <<3507>> [ fo r da tabase 3023:2820:orc l3 .46 ] s um(SALES.PR O D_SUBCATE G O R Y_DESC] [ fo r da tabase 3023:2820.c2 = D2.PRO D_ID [ fo r database 0:0 ] GroupBy: [ PRO D U CTS.46] PR O D U C T S T2874 C O STS T2830 Deta i lF i l te r : CO STS. with a FullOuterStitchJoin between them • Capabilities of physical database determine whether join is pushed down to database ( RqL is t <<3474>> [ fo r da t abase 3023:2820:orc l 3.com W: www.46 ] .PR O D_SUB CATE G O R Y_DESC] ) as c1 G B [ fo r da t abase 3023:2820.46 ] ) as D2 T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.UNIT_C O ST by [ PRO D U CTS.rittmanmead.PRO D_ID = PR O D U CTS.PR O D_SUBCATE G O R Y_DESC] [ fo r da tabase 3023:2820.com . PR O D U CTS.Facts with Conformed Dimensions Execution Plan • Two result sets are generated.AM O U NT_S OLD by [PR O D U CTS.46 ] .46 ] s um(COSTS.46] PR O D U C T S T2874 SALES T2911 Deta i lF i l te r : PRO D U CTS.PRO D_ID [ fo r da tabase 0 :0 ] GroupBy: [ PRO D U CTS.

c1 as c2.UNIT_C O ST) as c1 .c2 is not nu l lthen S A WIT H0.Facts with Conformed Dimensions Single Physical SQL • Where database supports subquery factoring (Oracle 10gR2+) a single physical SQL will be generated with a full outer join between all shared dimension columns WITH SA WITH0 AS (se lec t sum(T2830.PR O D_SUBCATE G O RY_DESC as c2 f ro m PRO D U CTS T2874.PR O D_SUBCATE G O RY_DESC as c2 f ro m PRO D U CTS T2874.com W: www.PR O D_SUBCATE G O RY_DES C) se lec td is t i nct SA WITH0. case when SA WITH0.c2 order by c3 T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.PR O D_SUBCATE G O RY_DES C) .c2 when SA WITH1. COSTS T2830 where ( T2830.rittmanmead. SALES T2911 where ( T2874.PR O D _ID = T2911.com .PR O D _ID = T2874.AM O U NT_S OLD) as c1.c2 = SA WITH1. T2874. SA WITH1 AS (se lec t sum(T2911. SA WITH 1.PR O D_ID ) group by T2874.PR O D_ID ) group by T2874.c1 as c1. T2874.c2 is no t nu l lthen SA WITH1.c2 end as c3 f ro m SA WITH0 fu l l outer jo in SAWITH1 On SA WITH0.

A M O U N T_ S O L D) as c1 .Send ing query to database named orc l3 ( id : <<3055>>) : se lec t su m(T2911.com .PR O D _ID = T2911.UNIT_C O S T) as c1 . SALES T2911 where ( T2874.Facts with Conformed Dimensions Multiple Physical SQL • Otherwise. T2874.PR O D_SUBCATE G O RY_DESC as c2 f ro m PRO D U CTS T2874.PR O D_ID ) group by T2874.PR O D _ID = T2874.com W: www. T2874. COSTS T2830 where ( T2830.PR O D_SUBCATE G O RY_DES C order by c2 +++Ad min i s t ra to r :2a0000:2a0005: -2010/02/28 15 :05:31 . multiple physical SQL queries will be generated • BI Server will perform in-memory sort-merge stitch full outer join on shared dim columns .PR O D_ID ) group by T2874.PR O D_SUBCATE G O RY_DESC as c2 f ro m PRO D U CTS T2874.Send ing query to database named orc l3 ( id : <<3018>>) : se lec t su m(T2830.PR O D_SUBCATE G O RY_DES C order by c2 T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.rittmanmead.

Stitch Joins and Federated Queries • Stitch joins are used whenever multiple fact tables are used in queries • Also used when federated fact table sources are used.ProductCategory. and aggregate sources • Basic principle is to join facts using FullOuterStitchJoin across all dimension attribute columns used in the query .com W: www.rittmanmead. T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.ProductCategory =Nul lsEqual D902. . . actual j oin vectors: [ 0 1 ] = [ 0 1 ] ( . . .ProductCategory =Nul lsEqual D902.ProductCategory and D901. ) as D901 Ful lOuterSt i tchJoin <<7800>> On D901.com .

if not have the BI Server do it • Logical execution plan and resulting physical SQLs visible in the Level 5 log file T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com .com W: www. push down stitch join to database using full outer join if possible.rittmanmead.Rules for BI Server Join Strategy • • • • • Try to generate as few physical SQL / MDX queries as possible Push as many joins (and functions) down to the database as possible Where required. iterate through join using nested loop algorithm Generate one logical SQL query per fact table access. join fact and dimensions in memory using inner and outer sort-merge joins If driving table is specified.

rittmanmead. one for Northern Region Known as fragmentation BI Server has to UNION the resultsets together T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.com . one for real-time One for Southern Region.BI Server Join Variations 1 : Fragmentation (Partitioned Measures) • • • • • RPD has a measure mapped through two or more LTS to different physical tables One physical source for historic data.

PR O D_ID = SALES_UPTO_2003.44] RqL is t <<7668>> [ f or da tabase 3023:6594:orc l7 . SALES_BEY O N D_2003. BI Server will attempt to push down to database if possible (if not.AM O U NT_SOLD as c3 [ fo r da tabase 3023:6594.PRO D_SUBC ATE G O R Y _DESC as c2 [ fo r da tabase 3023:6594.44 ] PROD U CTS. two physical SQLs will need to be retrieved and UNION done in memory) ( RqL is t <<7613>> [ f or da tabase 3023:6594:orc l7 .rittmanmead.44 ] .44 ] PROD U CTS. SALES_UPT O_2003.44] Ch i ld Nodes (RqJoi nSpec) : <<7617>> [ fo r da tabase 3023:6594:orc l7 .PR O D_ID = SALES_BEYO N D_2003.44 ] Ch i ld Nodes (RqJoi nSpec) : <<7672>> [ fo r da tabase 3023:6594:orc l7 .44 ] PROD U CTS T6596 SALES T6629 Deta i lF i l te r : PRO D U CTS.44 ] .PR O D_ID [ for da tabase 0:0 ] RqUnion Al l <<7690>> [ for database 3023:6594:orcl7 .44 ] PROD U CTS T6596 SALES T6637 Deta i lF i l te r : PRO D U CTS.com .AM O U NT_S OLD as c3 [ fo r database 3023:6594.Fragmentation (Partitioned Measures) Logical Execution Plan • Execution Plan will contain two logical SQL blocks (excerpt shown below) • RqUnion All union’s the two resultsets.PRO D_SUBC ATE G O R Y _DESC as c2 [ fo r da tabase 3023:6594.PROD_ID [ fo r da tabase 0 :0 ] ) as D3 T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.com W: www.

com .com W: www.rittmanmead.BI Server Join Variations 2 : Time-Series Queries • • • • Generated when time-series functions are used in a request Requires time dimension to be defined in RPD Currently limited to AGO and TODATE functions Generates two or more logical SQL queries which need to be joined together to return timeseries results T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.

.44 ] D901 Ful lOuterSt i tchJoin <<4892>> On D901. c2 = 'May-2007 '[ fo r da tabase 0 :0 ] GroupBy: [ D3.c1.44 ] . D2.c2 . . .OR DID and O R D E R S. c2 as c1 [ fo r database 3023:4210.com . Deta i lF i l te r :I T E M S. D2.44 ] . will either be shipped as one SQL (using subqueries) or as two or more SQLs (with results full outer stitch joined in memory) T : +44 (0) 8446 697 995 E : enquiries@rittmanmead.c2 and D3. .c1 as c2 G B [ fo r da tabase 3023:4210.com W: www.c1 = D 4. actual j oin vectors: [ 0 ] = [ 0 ] ( RqLis t <<4963>> [ fo r da tabase 3023:4210:orc l4 .c3 ][ fo r da tabase 3023:4210.44 ] D2.44] ) as D1 OrderBy: c1 asc [ fo r database 3023:4210.O R DID = O R D E R S . D3.c1 and D3.c3 as c3 [ fo r da tabase 3023:4210.44 ] .Time Series Logical Execution Plan • • • • Execution plan will show multiple nested logical SQL blocks More complex set of nested blocks will be doing the time offset Time offset block will need to be FullOuterStitchJoined to the present-day resultset Excerpt shown below .c1 =Nul lsEqual D902.rittmanmead.O R D E R D A T E = D 4. ) as • Depending on DB capabilities.

com .Memory and Disk Usage During BI Server Joins • • • • So what happens when the BI Server performs an in-memory join? The BI Server requires memory to join the data sets using a sort-merge algorithm Initial memory usage is determed by size/complexity of the RPD Memory usage then rises with # of concurrent sessions.rittmanmead. use of BI Server functional compensation is largely I/O intensive rather than memory intensive • Extent of paging driven by VIRTUAL_TABLE_PAGE_SIZE parameter ‣ May benefit from increase on Unix systems ‣ Docs do not imply much benefit though T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead. aggregations • Typically paged to file at {ORACLEBIDATA}/ tmp • Because of paging. in-memory joins.com W: www.

com W: www.So What Influences Execution Plans and Physical Query(ies)? • Logical Execution Plan largely determined by the LTS mappings in the RPD • BI Server Navigator generates execution plan based on context of the request and the LTS configuration in the BMM layer • All can be determined by looking at the RPD ‣ No equivalent to the Oracle database CBO • Physical SQL then determined by execution plan and DBFeatures.com .rittmanmead.INI T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.

com .Are There The Equivalent of Object Statistics in the RPD? • There are some limited statistics in the RPD that influence the query plan ‣ Row counts on physical tables are ignored • Number of Elements at this Level value can be used by the Navigator to determine which of several aggregates to use for a request • Needs to be manually populated (rough figure is OK) T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com W: www.rittmanmead.

(Customer Dim) Districts ‣ Sales Fact Agg2 -> (Time Dim)Year. • Also assume I have a couple of aggregate tables : ‣ Sales Fact Agg1 -> (Time Dim)Year. Days (730) ‣ Products Dim-> Type (10). SKUs (1000) ‣ Customers Dim-> Regions(20). • This can be satisfied by either Agg1 or Agg2. T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead. Districts(500). (Product Dim) Sub-Type • Consider this Query Example : • Select Year. Month (24).com .com W: www.rittmanmead. but BI Server has no way of knowing which is the best option unless the number of elements are entered.Dimension Level Element Counts In Use • Consider the following dimension ‣ Time Dim -> Year (2). Sum(Revenue) from Sales . Sub-Type (20). Customers (100000) • Sales Fact joins to these dimensions at the lowest level.

What Does the Logical Execution Plan Tell Us? • It tells us how the BI Server Navigator turned the logical request into physical SQL and MDX queries • But there’s nothing that you couldn’t infer from the RPD • There are no parameters etc that you can use to alter join strategies etc ‣ This is driven by the repository design only • Extra information provided by Level 7 logging is interesting ‣ Times logged by each entry/step ‣ Tells us how long the BI Server takes to do each join • Can be used to fine-tune the VIRTUAL_TABLE_POOL_SIZE parameter T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.com .com W: www.rittmanmead.

com W: www.com . and how to read the execution plan • Understand what a FullOuterStitchJoin is.So What Does This All Tell Us? • • • • Co-locate data as much as possible. and how to avoid unneccesary ones • Understand the role of LTSs and when to add or remove them • There is some diagnostic data available for the BI Server.rittmanmead. mimimize functional compensation Understand what happens with BI Server joins. but most of what you need is contained in the RPD design T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead. minimize in-memory joins Make sure the database version and capabilities is set correctly for each source Maximize function push-down.

com/2010/03/01/ ins ide-the-oracl e -b i -server -par t 2-howis-a -queryprocessed/ ‣ ht tp : / /ww w.r i t tmanmead.rittmanmead. com/2010/02/25/ ins ide-the-oracl e -b i -server -par t 1the-b i server arch i tec t ure / ‣ ht tp : / /ww w.com .com ‣ Email : mark. and on OBIEE in general.Thank You for Listening • More details on this topic.rittman@rittmanmead.com W: www.r i t tmanmead. com/2010/03/03/ ins ide-the-oracl e -b i -server -par t 3-b i -server in -memoryj o ins / • Thank you for your time • Any questions : ‣ Web : http://www.com ‣ Twitter : @markrittman T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.rittmanmead.r i t tmanmead. can be found on our blog ‣ ht tp : / /ww w.

com W: www.Inside the Oracle BI Server (10g) Mark Rittman. Director. Rittman Mead ODTUG Kaleidoscope 2010.C. T : +44 (0) 8446 697 995 or (888) 631 1410 (USA) E : enquiries@rittmanmead.rittmanmead. Washington D.com .