Professional Documents
Culture Documents
Large-Scale Testing of
the SAP NetWeaver
BI Accelerator
on an IBM Platform
s t Databases
r ge
La
d ’s
o rl
W
e
th
in
ts
lis
cia
Spe
SPONSORED
RESEARCH
PROGRAM
Large-Scale Testing of
the SAP NetWeaver
BI Accelerator
on an IBM Platform
RICK BURNS
March 2008
Table of Contents
1 The Challenge Facing SAP NetWeaver BI Accelerator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 4
Methodology
WinterCorp was retained by SAP and IBM to provide an independent assessment of the large scale tests of SAP
NetWeaver BI Accelerator on IBM infrastructures conducted as part of Project Jupiter., The assessment is based
on extensive review of Project Jupiter test plans, on-site observation of some Project Jupiter testing, and detailed
examination of Project Jupiter test results.
WinterCorp, SAP and IBM jointly agreed to publish the results of this assessment. As part of its agreement with SAP
and IBM, WinterCorp retains complete editorial control of the content and presentation of the published assessment.
SAP and IBM had the opportunity to comment prior to publication, but all findings, analysis, and conclusions are the
responsibility of Winter Corporation.
1
See: The SAP NetWeaver BI Accelerator: Transforming Business Intelligence, by Rick Burns & Robert Dorin,
WinterCorp, September 2006. This research report is available on the SAP and WinterCorp web sites.
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 5
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 6
The sections below describe the details of the test system configuration, the data model and queries
employed for the test, and test protocol.
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 7
Data was partitioned as it was loaded into the BI Accelerator. The number of partitions at each scale
point is shown in table 3. Note that despite the mismatch at the larger scale points between the degree
of data partitioning and the degree of parallelism, data is distributed uniformly across the nodes of the
BI accelerator to insure balanced workload across the entire parallel environment.
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 8
5 TB 27 27
15 TB 81 40
25 TB 135 40
Queries executed against these data models involved thousands of variations on 14 complex reporting
templates. All queries accessed data via MultiProviders. They involved a fact table joined to large numbers
of dimensions with numerous predicates and multiple aggregation fields. Query predicates included
widely varying time periods, and location, product, and customer ranges that required processing
large volumes of data. Depending on query selectivity, individual reports frequently touched more
than one underlying InfoCube, averaging 1.6 InfoCubes accessed per report. Since the data volume
in each period grows as the test scale increases, the work performed by each query also grows fairly
linearly as the test scale increases.
3. 3 Test Proto co ls
HP’s LoadRunner, a widely used, commercial test driver software package, controlled test execution.
Tests were initiated from the query driver servers. Query concurrency was varied from a single test
stream to 800 concurrent query streams. Query streams cycled continuously through the thousands of
query instances with a two-second inter-query delay. For multi-user tests, each query stream executed
the sequence of queries starting from a different offset to insure that concurrent in-flight queries were
processing different data. Each test involved data from all InfoCubes.
WinterCorp reviewed and analyzed all test results and monitored test execution for the last week of
the test period.
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 9
5 TB 27 5 6:40:30
15 TB 40 10 11:07:07
25 TB 40 10 15:04:29
The difference in load rates of roughly two times, between the 5 TB scale and the larger scale points,
appears to be due in large part to the different levels of concurrency used during the respective loads.
Using five concurrent load streams, rather than 10, while keeping the BI Server load parameters the same
at all three scales, resulted in slower load rates
Figure 3: BI Accelerator Load Rates
at the 5 TB scale compared to the two larger
scale points.
On the other hand, the BI Accelerator load
process did not demonstrate linear scalability.
Given that the scale of the BI Accelerator
system grew in proportion to the growth in
the data volume, the question arises of why
the elapsed time to load the BI Accelerator
increases at all as the data volume increases.
The answer appears to be that the work
performed by the BI Server during BI
Accelerator load has a significant impact on
the scalability of the load process. For all
Project Jupiter tests, the NetWeaver BI Server
was a 64 processor IBM P595 SMP server. As
the BI Server approached saturation, it limited
load throughput, despite the growth of the
Figure 4: BI Accelerator Load: Processor Utilization
BI Accelerator system (Figure 4). Reducing
the role that the BI Server plays in the load
process may be instrumental in improving
future BI Accelerator load scalability.
Nonetheless, load rates above 1 TB per hour
are more than 50 times greater than rates
observed in earlier load testing on a much
smaller, 4 processor BI Server system, and are
likely to be more than sufficient for periodic
updates to very large BI Accelerator databases
for the next several years.
4. 2 Single Us e r Tests
Project Jupiter tested basic query scalability of
the NetWeaver BI Accelerator through single
user test of each or the 14 query templates as
the system scaled between 5 TB and 25 TB. At
each scale point, all variations of each query
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 10
template were executed consecutively through a single query stream, and the query response time was
measured. The average response time for each query was calculated (Figure 5).
Analysis of the single user tests demonstrated very good responsiveness across all scales, with average
response times of less than 1 second overall, and below 2 seconds for nearly all of the queries, even
Figure 5: Average Response Time by Query at the largest scale. Note that, on average,
each query processed millions of InfoCube
records, ranging from an average of 4.7
million records at the 5 TB scale, to 25.6
million at the 25 TB scale point. This shows
the power of the parallel BI Accelerator
engine.
Figure 5 also shows response time increasing
as the scale increased. At first look, this
appears to indicate less than linear scalability
across the three scale points tested. This
appearance is deceptive however, as the
volume of data processed by each query
grew more than linearly as the test scale
increased. When adjusted for this super-linear growth in the volume of data processed by each query,
the ability of the SAP NetWeaver BI Accelerator to scale linearly across the tested scale points becomes
clear (Figure 6). In Figure 6, linear growth in data volume and response time would be shown by a relative
growth rate of 1.0. As the chart shows, data
Figure 6: Query Scalability
volume grew at a faster than linear rate as the
test scale increased, while the response time
growth rate stayed relatively flat.
4. 3 Multi - Us e r Tests
In order to test the scalability of the SAP
NetWeaver BI Accelerator under heavy load, a
series of multiuser tests were executed at each
of the three test scale points. The number of
concurrent query streams was varied between
100 and 800, with each stream running the
query sequence from a different start point,
so that each in-flight query was processing
different data. (At the 5 TB scale, test were only
executed through 400 concurrent streams.) A
short inter-query delay of two seconds was
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 11
used for all multiuser tests, simulating tests of Figure 7: Multiuser Throughput
thousands of concurrent users. To assess multiuser
scalability, both query throughput, measured as
queries per hour, and query response time were
measured.
At each tested scale, multiuser query throughput
peaked at more than 100,000 queries per hour at
200 concurrent query streams and then plateaued
at higher levels of concurrency (Figure 7). Beyond
200 concurrent query streams the NetWeaver
BI system reached saturation on processor
resources, first on the BI Server, and then on the
BI Accelerator at a somewhat later point (Figure
8). For this reason, analysis of BI Accelerator
scalability relied on the 200 concurrent query
streams test points exclusively.
Overall, the Project Jupiter tests demonstrated
linear scalability, for both query throughput and response time, through the 25 TB test scale (Table 6),
achieving in excess of 100,000 requests per hour and a flat response time of roughly 4 seconds. This is
especially significant since, similarly to the single user test results, the average volume of data processed
per query rose faster than linearly.
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 12
5 TB 71.20 3.93
15 TB 66.65 5.82
25 TB 59.89 5.61
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
Large-Scale Testing of the SAP NetWeaver BI Accelerator on an IBM Platform 13
• Testing at all scales demonstrated consistently high throughput at a rate of more than 100,000 user
reports per hour;
• Multi-user tests also showed the ability to support an active user population numbering in the
thousands through high concurrency tests that achieved a concurrency of 100 to 800 query streams,
with optimal results at 200 concurrent query streams;
• In all tests, the system demonstrated effective parallelization of query activity across up to 135 nodes,
using up to 10 IBM blade chassis.
These results demonstrate the scalability of the BI Accelerator running on an IBM infrastructure
to efficiently manage large databases up to 25 TB, and to effectively exploit very large parallel BI
Accelerator environments.
5 Conclusion
To meet the widespread demand for better business intelligence, SAP NetWeaver BI customers need to
scale their NetWeaver BI systems to unprecedented levels. This will entail rapid growth in data volume,
user concurrency, and diversity and complexity of the workload. The workload dimension is especially
dynamic, as the demand for data exploration and analysis increasing tilts the workload mix away from
standard reporting and toward unpredictable, ad hoc query processing.
The project Jupiter test program, by testing query and load performance of SAP NetWeaver BI Accelerator
on IBM systems at scales up to 25 TB of user data, with hundreds of concurrent query streams against
a variable workload, demonstrates an ability to meet the need to for more users to process much more
data in increasingly unpredictable ways, quickly and efficiently. These test results also provide strong
evidence of the capability of the BI Accelerator hosted on IBM systems to efficiently exploit hardware
resources well beyond a single blade chassis to deliver the class of scalable performance required to
meet these demanding business challenges for
©2008 Winter Corporation, Waltham, MA. All rights reserved. Duplication only as authorized in writing by Winter Corporation.
A leading center of expertise in very large databases,
WinterCorp provides services in
consulting, research, architecture and engineering.
Our focus is databases near, at and beyond the frontier of database scalability.
visit us at www.wintercorp.com