Professional Documents
Culture Documents
Stress Testing Web Intelligence and Crystal Reports
Stress Testing Web Intelligence and Crystal Reports
Applies to:
SAP BusinessObjects XI 3.1. For more information, visit the Business Objects homepage.
Summary
This paper discusses stress testing an SAP BusinessObjects Enterprise system to compare the efficiency of
Web Intelligence and Crystal Reports. For both Web Intelligence and Crystal Reports it looks at how
document size and number of users can affect the impact on system resources. The results demonstrate that
Crystal Reports is a better fit for deployments where there are a large number of us ers who are viewi ng and
refres hing pre-built reports.
Author:
Al Gulland
Company: gulland
Created on: 8 November 2010
Author Bio
Al Gulland is a senior BI consultant for IBM Global Services and has been active in Business Intelligence for
over 15 years, specializing in SAP BusinessObjects and data warehouse solutions. Al maintains a blog at
www.gulland.com and has posted many articles on SAP BusinessObjects.
Table of Contents
Introduction............................................................................................................................................. 3
Test Environment .................................................................................................................................... 3
Test Documents Used .......................................................................................................................... 3
Stress Test Approach .............................................................................................................................. 4
Performance Monitoring ....................................................................................................................... 4
Test Results............................................................................................................................................ 5
Conclusion.............................................................................................................................................. 5
Appendix 1: Server Configuration ............................................................................................................. 6
Web Intelligence Server Configuration ................................................................................................... 6
Crystal Reports Server Configuration .................................................................................................... 7
Properties for Crystal Reports Processing Server ................................................................................................................... 7
Introduction
The overall objecti ve of this exercise was to compare Web Intelligence to Crystal Reports under identical
stress test conditions. Here the stress test was to have a set number of users viewing and refreshing a
document while at the same time measure the memory and CP U consumption on the SAP BusinessObjects
Enterprise (BOE ) server. The test was executed for both a Web Intelligence document and a Crystal Report
of the same size. In both cases the number of users was increased and measures of memory and CP U
usage were taken. The Boe st stress testing application was used to execut e the stress tests and to analyze
the results.
This article reviews the testing environment, how the stress tests were executed, discussion of the test
results and finally conclusions.
Test Environment
The same BOE server was used for both the Web Intelligenc e and Crystal Reports stress tests. As the tests
were a comparison of the two report formats it wasn't necessary to have a high end capacity server and so a
virtual machine was used to host the server. The environment specification is as follows.
OS
CPU
RAM
2 GB
Virtualisation Technology
VM Player 3.1
XI 3.1 sp3
Database
Boest was installed on a Windows 7 desktop that had access to the BOE Server. The same desktop was
host to the virtual machine running the BOE server.
Test Documents Used
The test documents used for both Web Intelligence and Crystal Reports were created to be as much as
possible of the same size and complexity in order to ensure a valid comparison test. In this case the same
SQL statement was used in both the Web Intelligence document and Crystal Report. This ensured that not
only the database ex ecution of the SQL statement would be the same but that the number of rows in the
report would be the same. In addition the display and formatting of the data in the reports was the same for
both document types which was a simple table of dat a.
In both cases a small and a medium sized document was created where the difference in the document size
is purely based on the number of rows returned by the SQL statement.
The SQL used by all reports is,
SELECT
ID, Store, Product, Purchase_Date, Quantity, On_Sale
FROM
SalesDataMartDemo.dbo.TBL_LARGE2
WHERE
ID <= 10000
ORDER BY
Store, Product, Purchase_Date
The only difference between small and medium reports is the value used in the condition. For small reports
the condition is ID <= 10000 and for medium it is ID <= 100000.
The following reports are used in these tests.
Name
Type
Number of Rows
PerfTestWebi_Small
Web Intelligence
224
9,991
PerfTestWebi_Medium
Web Intelligence
1,921
99,991
PerfTestCrystal_Small
Crystal Report
168
9,991
PerfTestCrystal_Medium
Crystal Report
1,214
99,991
For comparison this is same amount of dat a saved as CSV where the file was generat ed from a SQL Server
export.
Name
Type
Size (kb)
Number of rows
Small
CSV
522
9,991
Medium
CSV
4,266
99,991
Test Results
The results were analyzed by first taking an average across the 5 repeated tests. This average was then
plotted against the number of users to produce a chart of memory usage by number of users and also a
chart of CPU us age by number of users (see appendix). The chart of number of users versus memory was a
good fit for a straight line and so in eac h case linear regression analysis was used to determine the best fit
for a straight line through these points. This straight line can then be used to determine a relation between
number of users and memory which can then be extrapolated to find a maximum load for the system.
The table below summarizes the results. The initial memory is the intersect of the straight line wit h the
vertical axis and is equivalent to the amount of RAM consumed prior to start of test. The maximum memory
is the maximum memory reached during the test and this was always when the full load of 50 users was
reached. The next column displays the gradient of the straight line and this is the inc rease in RAM per user.
The final column is the maximum percent age CPU reached during the test, note this is out of 200% as the
system was configured with two CPUs.
Test
Initial Memory
(m b)
Maximum
Memory (mb)
63.6
168.7
2.17
190.31
44.2
502.5
8.31
195.02
257.5
292.5
0.78
66.87
255.5
291.9
0.81
78.75
From this it can be seen that although Web Intelligenc e starts from a lower bas e the increase in memory per
user is much greater than Crystal Reports and for the medium doc about 0.5GB of RAM is reached for 50
users. For Crystal Reports 50 users viewing a report of equivalent size consumes 291MB of RAM.
It should also be noted that Web Intelligenc e places more demand on CPU usage indicating more
processing required to render the displayed report.
Conclusion
The above analysis clearly shows that there is, as you would expect, a direct correlation between increasing
users and an increase in memory for Web Int elligence and Crystal Report types. However although Crystal
Reports has a higher initial memory the rate of increase in memory is much less than that of Web
Intelligence such that for a large number of users Crystal Reports uses less memory than Web Intelligence.
CPU usage is also different and by inspection of the graphs it can be seen that Crystal Reports is less CPU
intensive than Web Intelligence.
From this it can be concluded that Crystal Reports is a better solution for deployments with a high number of
users who need to view and refresh prebuilt reports and that Web Intelligenc e is more efficient for a low
number of users.
Does this mean then that Web Intelligence should never be used and that Crystal Reports is best used in all
scenarios ? No. These are two different products that meet two similar but different requirements. Web
Intelligence is designed for a business user to create new reports and perform some basic data analysis.
Crystal Reports on the other hand has been designed for large volume report generation and distribution.
This exercise was designed to compare Crystal Reports to Web Intelligence in a tightly controlled
environment. The above values are not expected in a real world scenario and the results should not be
interpreted as a realistic measure of demand in a production system.
Report size has a significant impact on system demand and in a production environment it is expected that
there will be a wide range of reports of different sizes and complexity. Furthermore, user demand will not be
regular and there will be peak times when users are accessing the system.
These factors and more will affect the maximum load that a system can comfortably handle and a more
detailed capacity planning exercise should be undertaken when designing a deployment.
Setting
120
50
4370
1800
100
100
20
Off
Images Directory:
Maximum Document Cache Size (KB):
1000000
On
50
Maximum Connections:
100
50000
On
Off
70.00%
1500
Off
On
1000
100
1000
1200
Setting
IdleJobTimeout
1000
On
On
20
Number Users
Average CPU
Average Memory
14.08
88.44
74.34
10
24.28
158.12
86.30
15
30.51
171.25
96.21
20
35.20
183.75
106.89
25
39.36
180.31
114.55
30
43.24
188.12
130.41
35
45.78
185.00
137.69
40
48.47
188.43
155.01
45
51.28
190.31
163.68
50
52.51
189.06
168.73
Linear regression analysis of number of users versus average memory gives the following metrics,
Gradient:
2.174
Intersect:
63.598
Correlation Coefficient:
0.997
Graph of memory consumption vs. number of users for the small Web Intelligence document
Graph of average CPU (lower line) and maximum CPU vs. number of users for the small Web
Intelligence document
Average CPU
Average Memory
48.96
183.74
101.54
10
77.52
190.62
138.43
15
90.11
191.24
163.31
20
98.9
190.96
207.18
25
104.56
193.43
248.38
30
110.52
192.49
273.9
35
115.17
195.02
325.73
40
114.25
193.75
355.32
45
115.7
192.81
410.15
50
118.1
194.68
502.53
8.306
Intersect:
44.236
Correlation Coefficient:
0.989
Graph of memory consumption vs. number of users for the medium Web Intelligence document
Graph of average CPU (lower line) and maximum CPU vs. number of users for the medium Web
Intelligence document
Average CPU
Average Memory
3.16
17.19
257.43
10
4.04
25
264.48
15
5.71
35.94
270.04
20
6.66
46.56
274.37
25
8.33
56.25
279.28
30
8.88
64.37
283.61
35
11.24
66.87
286.48
40
8.41
55.62
289.37
45
10.88
67.81
291.71
50
10.94
63.13
292.53
0.780
Intersect:
257.469
Correlation Coefficient:
0.980
Graph of memory consumption vs. number of users for the small Crystal Report
Graph of average CPU (lower line) and maximum CPU vs. number of users for the small Crystal
Report
Average CPU
Average Memory
2.67
14.37
255.24
10
2.98
19.69
263.06
15
4.56
28.75
268.57
20
6.13
34.06
273.48
25
8.56
57.19
277.41
30
8.48
52.18
283.42
35
9.55
55.31
286.22
40
10.51
60.94
289.05
45
10.28
71.56
290.8
50
12.81
78.75
291.94
0.814
Intersect:
255.547
Correlation Coefficient:
0.978
Graph of memory consumption vs. number of users for the small Crystal Report
Graph of average CPU (lower line) and maximum CPU vs. number of users for the small Crystal
Report
Related Content
Boe st
Boest is a Java application built specifically for stress testing the execution of Crystal Reports and Web
Intelligence documents in an SAP BusinessObjects Enterprise deployment. Further details of the application
and download can be found at Sourceforge.net.
Linear Regression Analysi s
The linear regression analysis was calculated using Web intelligence using the method discussed in the
article Linear Regression Analysis and Web Intelligence.
Other Related Topics
BusinessObjects Enterprise XI Release 2 Sizing Recommendations Guide. Although this was written for XIr2
much of the content remains valid for XI3.
Selecting the Right Reporting Tool A comparison of Web Intelligence and Crystal Reports
Business Objects Community: Crystal Reports
Business Objects Community: Webi
For more information, visit the Business Objects homepage