Performance Testing: SAP Volume Testing Facts

Author : Anshuman Das

Email: anshudas@in.ibm.com
Address: IB !lobal Ser"ices# India
Subraman$a Arcade#
%o. &'# Bannerghata ain (oad
Bangalore )*+ +',
1
Abstract
SAP is the most successful product for enterprise resource planning (ERP), touching nearly every facet of
business like accounting, logistics, R!, "R, data #arehouse, etc$ %n many companies, transactions of
thousands of users must be processed concurrently by SAP and the underlying database system$ "ence, it is
truly a mission&critical technology that re'uires solid performance and high availability to deliver the level
of service the business re'uires$ (ailure of this critical system introduces tremendous business risk, both to
the successful e)ecution of internal processes and to the service level agreements as established$
*he purpose of this paper is to give an overvie# of various facts and to large e)tent the best practices
follo#ed during SAP volume and stress testing$ *his paper describes best methods of creating automation
scripts and e)periences from real time e)ecution of various SAP applications involving modules like Sales
+ ,istribution, !aterial !anagement, "uman Resource !anagement, R!, -usiness .arehouse$ *he
paper also sho#s ho# an SAP system can be monitored during volume test$ *his paper is structured in a
format to give an overvie# of the activities to be done before the test, during the test and after the test$
/oadRunner 0$1 from "P&!ercury is the automation tool used to create automation scripts and mimic end
user #orkload$
1
Table of -ontents
1. Test Engagement
*he entry criteria for volume and stress in any SAP system are due to follo#ing reasons$
%ncreasing the number of users or adding ne# users
A ma2or soft#are upgrade like database or operating system migration
A ma2or hard#are upgrade like implementing -%A infrastructure for -.
%mplementing a ne# module or ne# functionality like accessing through SAP Portal instead of
SAP 34%
*he common concern #hich arises in minds of pro2ect team before any SAP application go live can be
classified as follo#s$
"o# does the database react #ith e)isting or desired concurrent SAP users5
.ill the implemented SAP application meet the end user performance re'uirements #ith
respect to response time and throughput5
%s this SAP implemented solution scalable5
.ill there be any dumps related to R( connections and local printing hanging issues during
production operation5
an -%A infrastructure improve the performance of -%. 'ueries5
an specified number of R! users create desired volume of tickets per hour5
%s this implemented solution capable of handling estimated production level transaction
volumes #ithin the available time frame5
.as the hard#are si6ing done accurately5
Are Application Servers and ,atabase Servers configured accurately to #ithstand desired
number of processes or re'uests5
.hich component can fail for #hat time5
.hich component of the system limits the number of concurrent users5
%n every volume test the engagement model begins in the follo#ing manner (refer (igure 1)$
Re'uest from Pro2ect *eam7 *he re'uest can be made in various #ays like Service Re'uest
(SR), by email or phone call and sometimes pro2ect team member #alking in #ith re'uest for
testing$
%nitiate 8ick off meeting7 *he testing team should initiate kickoff meeting to discuss and
understand re'uirements$ ,uring this meeting the scope and ob2ectives of testing should be
defined$
E)plain the strategy of testing to pro2ect team as per the re'uirement$
reate Service Re'uest #ith a uni'ue id$
reate the test plan$
Assign resources as per availability in pool and re'uired skill set$
9
(igure 17 :olume *esting Engagement !odel;(rame#ork
2. Effort Estimation
*o estimate an effort for volume testing is a key task$ *he effort can be measured based on 9 phases$
Phase %7 Script ,esign
(igure 17 Script ,esign
According to (igure 1 the script comple)ity is measured by the scripting team$ Effort involved is
estimated based on this scripting$ As a best practice, the scripting effort is designed using a formula
keeping in mind the comple)ity level$
onsidering a medium comple) script a script analyst can cover 1 script in 1 days$ "ence the formula
is
# of Scripts * 2 = # of Man Days
<ote7 *his formula also considers script analyst skill in /oadRunner scripting$
Phase %%7 *est Environment ,esign
Re'uest from Pro2ect *eam in
the form of SR ;email ; Phone
all ; .alk in
%s re'uest
converted
into SR5
*esting team initiate 8ick =ff
meeting to understand testing
re'uirements$ Scope and
=b2ectives of testing defined$
Strategy e)plained to Pro2ect
team as per the re'uirements$
>es
<o
reate SR
*est !anager does the testing
effort estimation and prepares the
high level test plan$ Sends to
pro2ect team against the SR$
<o
Resources allotted as
per capacity plan$ %f
the resources are not
available in the testing
pool then a re'uest for
the same is raised$
Resource availability
*A* 9&? #eeks$
S*=P
>ES
S*AR
*est !gr;/ead as per the test
plan doc identifies the
business critical transactions
and sends it across to
scripting team$
Scripting team identifies the
protocol and comple)ity of
the script$
*est !gr;/ead sends the
delivery date of scripts to
pro2ect team and e)ecution
team as per the input from
scripting team$
S*=P
*est !gr; /ead based on test
plan doc sends the test
environment details to
e)ecution team and pro2ect
team$
E)ecution team prepares the test
environment #ith help from %S
Provider$ *he %n2ectors and SAP
34% client installed
*est !gr; /ead sends
confirmation mail to
pro2ect team and
e)ecution team$
?
S*AR
Pro2ect
team and
*est !gr
discuss the
plan and
agree
S*AR
S*=P
(igure 97 *est Environment ,esign
*est environment setup is 2oint venture bet#een e)ecution team and %S Provider$ -ased on
re'uirements captured in test plan document the environment is prepared$ *his involves setting up7
SAP test environment7 %t includes identifying and creating a landscape similar to production
environment consisting of Application Servers and entral %nstance #ith ,atabase$
SAP test data7 /oad copy of production data in test environment$ ,ata copying involves fe#
steps to be follo#ed in order to be sure that the data copied is of e'ual si6e in the test
environment$ %f #e copy data to different environment, say production to @A, then it is
normal backup from production and then restore in @A$ And if #e #ant to copy data to
different platform (say .indo#s to 4<%A) or different ,- #e need to use Br9loadB$ *his is a
tool from SAP to import and e)port table during installation$ %n certain cases the data used
should be confidential$ As a result scrambled data is prepared using a third party tool called
,S! (,ata Sync !anager from EP%&4SE)$ *his tool is used to scramble and depersonali6e
data$
/oadRunner %nfrastructure7 *he number of in2ectors to simulate or generate the load is
prepared based on number of concurrent users captured in test plan document$ As a best
practice in case of SAP R;9 CD virtual users can be simulated from a single in2ector #ith
configuration of C11 !- RA! and ?D 3- hard disk$ .ith similar configuration as a best
practice in case of SAP -%. 1D virtual user can be simulated from a single in2ector$ *o get
better and consistent connectivity bet#een /oadRunner controller and in2ectors, it is advisable
to have same version of /oadRunner package and fi) pack installed in controller and in2ector$
Enable scripting at client7 *he scripting option should be enabled in SAP 34% client$ %f the
scripting option is disabled the /oadRunner scripts can be neither created nor e)ecuted$ %n
order to enable scripting at client level go to Run regedit$ Set the follo#ing parameter
value to 1$
o E"8E>F/=A/F!A"%<EGS=(*.AREGSAPGSAP34% (rontGSAP (rontend
ServerGSecurityH
4serScripting I 1
Enable scripting at server7 *he scripting option can be enabled at server level by e)ecuting
transaction Jr611K$ heck for parameter Jsapgui;userFscriptingK and set it to L*rueM$

Phase %%%7 E)ecution Phase
C
!ake the necessary
parameter changes$
>es
(igure ?7 E)ecution Phase
,uring e)ecution phase a specific e)ecution plan is created considering all the activities involved
during e)ecution$ *his master e)ecution plan is captured in e)cel sheet #ith follo#ing #orksheets7
Pro2ect Activities7 *his #ork sheet consists of details tasks or activities to be accomplished before
and during e)ecution$
Script Plan7 *his #ork sheet consists of all the scripts as per defined business critical scenarios$
Scenarios7 *his #ork sheet consists of various types of e)ecution scenarios to be e)ecuted during
the volume testing$ *hese scenarios are captured initially in test plan document$
%n2ectors7 *his #ork sheet consists of all the in2ector details #ith %P address or machine name and
location #here they are installed$ *his list also consist of spare or backup in2ectors
Systems and !onitoring7 *his #ork sheet consists of detailed information of Servers #hich #ill
be monitored and parameters #hich #ill be considered for monitoring$
Stakeholders7 *his #orksheet is replica of contacts info section in test plan document$ *his
information helps to contact right person during e)ecution if any problem arises$
*est /og7 *his #orksheet is updated by the e)ecution team as a best practice to maintain a log of
activities done during e)ecution$ (or e)ample during e)ecution if any bottleneck is found in
application server then e)tra application server is added and again tested$ *his kind of details
should be captured in test log #hich during final reporting helps to prepare a good report$
Actions %ssues7 Any action items to be taken care by stakeholders or issues encountered during
any phase of volume testing are registered in this #orksheet$
/essons /earnt7 *he captured data from lessons learnt are documented in SAP :olume *esting
-est Practices for future reference$
3. SAP .oad(unner Scri/ting
SAP scripting begins #ith creating fe# utility scripts like users creation, resetting pass#ord and finally
scripts for identified business critical scenarios$ %n order to create users, initially a template user should
be defined #ith help of functional team or consultant$
aution7 %n order to create users the user id should have SAPF<E. access$ *he users should be
created as a replica of template user defined by functional team$ *he template user is defined based on
the business critical scenarios$
aution7 <o test user should be created #ith SAPFA// access$ A user #ith SAPFA// security access
goes through least security check during login$
E)ecution *eam e)ecutes
the test$
Encounter
-ottleneck
s
>es
-ottleneck details
sent to Pro2ect team$
Any
further
e)ecuti
ons
*est !anager;/ead
sends the
consolidated report to
pro2ect team$
,ocument the
necessary
parameter
changes
re'uired$
<o
<o
N
S*=P
S*AR
4. 0hat 1 ho2 to record3
/oadRunner scripting is done in recording environment called as :ugen$ :ugen is a component of
/oadRunner$ SAP scripting is done #ith a specific protocol called JSAP34%K or JSAP&.ebK(refer
(igure C) $ As a best practice the login and logout are recorded in JvuserFinitK and JvuserFendK
sections respectively$ Rest other SAP transactions and other navigations are recorded in user defined
Actions #hich are reusable and can be iterated$
(igure C7 /oadRunner protocol menu
*he script correlation should be taken care to avoid duplication of dynamic value$ Automatic
correlation #ill help to make the script more dynamic to deal in a better #ay against SessionF%,Ms,
database primary keys and almost all security done over "**P in case of SAP&.eb$
Parameteri6ation is another factor #hich should be handled carefully$
5. SAP Scri/ting -hallenges
/oadRunner scripting challenges #hile using SAP34% or SAP&.eb protocol arises due to many
factors$ *his may either due to functionality or due to custom developments$
-efore starting recording using :ugen it is al#ays advisable to change the recording options$
%n :ugen go to Recording =ptions 3eneral apture Screen Shots <one$ *his #ill make the
script much faster to record$
/essons leant7
.esson .earnt &: .hile performing a Purchase =rder (P=) transaction it becomes necessary
sometime to capture P= number to pass onto subse'uent transaction$ !anually add the
sapguiFstatusFbarFgetFparam to retrieve the parameter from the status bar$ %t is necessary to be careful
to pick up the right P= number
.esson .earnt '7 %f you are going to record in multiple sessions you can use the Record at the ursor
option & #hich plays the script until that point, and then you can record from that point O useful for
SAP #here there may be several long&#inded transactions as pre&re'uisites for that transaction$
.esson .earnt 4: =ne action should be created per SAP Screen$
<aming convention7
AnnnF*ransaction<ameFshortdesc
.here
) I uni'ue script code (every script in the load test has a different code)
nnn I se'uential number
P
(igure N7 /oadRunner Actions
.esson .earnt 5:
(or every Action
use lrFstartFtransaction(LactionFnameM)
and lrFendFtransaction(LactionFnameM)
Put all think time outside of this transaction$ *his #ill allo# us to track the actual transaction response time
during the test$
.esson .earnt ): Recovery from errors
*he script can be instructed to e)it the iteration and start the ne)t iteration on error$ .e need to include
some code at the top of the first action to select the #indo# and enter the neutral transaction nsDDD$
*he instructions in the script #ill override the runtime settings of continue on error (#hich donMt make
sense in a long script #ith dependencies as #e kno# that nearly all the subse'uent steps #ill fail)$
Include the follo2ing in the first lines of the first action:
;; start of fi) continue on error
lrFcontinueFonFerror(9)Q
sapguiFselectFactiveFconnection(BconEDHB)Q
sapguiFselectFactiveFsession(BsesEDHB)Q
sapguiFselectFactiveF#indo#(B#ndEDHB)Q
sapguiFsetFokFcode(B;nsDDDB,
-E3%<F=P*%=<A/,
BAdditional%nfoIsapgui?D0B,
E<,F=P*%=<A/)Q
;; end of fi)
Include the follo2ing in the last lines of the last action:
lrFcontinueFonFerror(D)Q
6. SAP Ser"ers onitoring
SAP servers can be monitored either through /oadRunner or manually e)ecuting defined transactions
in SAP$SAP monitoring is one of the most important task in SAP :olume *esting$ As a best practice a
caution should be taken not monitor too many parameters using /oadRunner controller$ *oo many
parameters generate huge amount of data and log files$ As a result at the end of e)ecution #hile
/oadRunner is collating data from all in2ectors and servers there are high chance of controller crash or
incomplete collation of results$
Parameters to be monitored
Application Server7
R ,isk *ime
R Processor *ime
(ile ,ata =perations;Sec
0
%nterrupts;Sec (!emory)
Page (aults;Sec (!emory)
Pages;Sec (!emory)
Processor @ueue /ength
S@/ ,atabase Server
R ,isk *ime
R Processor *ime
Pages;Sec (!emory)
Processor @ueue /ength
=racle ,atabase Server7 %mport tnsnames$ora file in controller to connect to =racle database$
,-.R transaction table #rites (:SS>SS*A* 1)
physical reads (:SS>SS*A* 1)
physical reads direct (:SS>SS*A* 1)
physical #rites (:SS>SS*A* 1)
physical #rites direct (:SS>SS*A* 1)
4<%A Server7 %f central instance is in 4<%A$
Average load (4ni) 8ernel Statistics)
P4 4tili6ation (4ni) 8ernel Statistics)
,isk *raffic (4ni) 8ernel Statistics)
Paging rate (4ni) 8ernel Statistics)
onitoring SAP transactions
*o monitor user distribution in application servers and central instance e)ecute
transaction JA/D0K$
*o vie# runtime errors during volume testing e)ecute transaction JS!11K +
JS*11K$ (igure P and 0 sho#s e)ecution of S!11 transactions$

(igure P7 S!11 (igure 07 S!11 ,etailed
*o monitor all R( e)ecute transaction JS!NNK$
*o monitor dialog and report user steps e)ecuted after volume testing e)ecute
transaction JS*D9<K$ *his transaction is a crucial transaction as this helps to
calculate the volume of transactions done in the #hole testing$ (igure T sho#s
clearly the number of dialog user steps and report user steps e)ecuted during
volume testing$
T
(igure T7 S*D9< to calculate dialog and report steps$
%n the above figure PA1D, PA9D, PA?D, P=19 and PPD1 are dialog user steps$
.hereas SFA"RFN1D1N11T and SFA"RFN1D1N?T? are report user steps$
7. -ase Stud$
-ased on above best practices here is a case study based on volume testing conducted for a ARA
ogani6ation$
6rgani7ation E8/erience
ARA is a #orld famous manufacturing company$ *hey are the oldest and #ell kno#n company in their
niche of manufacturing and production$ *hey employ more than 0C,DDD people #orld#ide #ith CD
factories in different locations and product sold in 10D countries$ *hey are about to implement a
complete SAP "R solution$ *he ob2ectives of this implementation are as follo#s
%mprove the "R service level to ARA and its employees
Reduce the overall business comple)ity and operating costs
(ocus on "R value&added activities and their support to the business
%mplement a better payroll system
%mplement a better reporting tool using -%.

Sam/le SAP 9( S$stem for A(A
(igure 1D7 Sample SAP "R system
1
IB:s Pro/osal
Test 6b;ecti"es
As per the best practices the goal of the test #as identified and #as documented in test plan document
during kickoff meeting$ *he goal of the tests #ere
*o run 9DD "R users in central instance and application servers$
Ensure running a load on the backend$
Ensure #ith load on backend the desired number of transactions are e)ecuted in
specified time$
Ensure R! application can support creating CDD tickets;hour #ith 1DD concurrent
users$
Ensure there is a drastic difference bet#een e)ecuting -%. 'ueries #ith and #ithout
-%A infrastructure$

Sco/ing of test

*he follo#ing SAP interfaces #ere identified from the above landscape to conduct volume testing
Standard SAP foreground reports OSAP 34%
ustom ,evelopment and @ueries O SAP 34%
-ackend transactions O SAP 34%
e&learning O Content player
R! O *elephony and .eb interface
Pay stub display
*ime entry and display ; leave re'uest
-%. (.eb)

Sco/ing of Scenarios
.ith reference to above interfaces the business critical scenarios #ere identified$
A group of comple), medium and simple scenarios #ere selected$ *he scenarios #ere
PA1D & ,ialog4ser
PA9D & ,ialog4ser
PA?D & ,ialog4ser
P=19 & ,ialog4ser
PPD1 & ,ialog4ser
SFA"RFN1D1N11T O /ong Reports
SFA"RFN1D1N?T? O /ong Report
SFP"TF?NDDD119 O /ong Reports
>-P!=!DDF(4<*,A*A8E> O Short Reports
>-P!=!D1F"=4<*F O Short Reports
>-P!=!D1F"EA,<*A* O Short Reports
>-P!PADDFE!PF/%S* O Short Reports
>-P!PAD1FSER:FA<<%: O Short Reports
U-!A=!DDF"EA,<* O Short Reports
U-!AP>=F"R14FS%PREFD11 O Short Reports
<sers Distribution
*ransactional 4sers 7 1?9
/ong Running Reports 7 P
Short Running Report 7 CD
1
E8ecution (esults
*here #as a peak in response time #hen users logged in$
,ialog or transactional users sho#ed good response times #ithout the report users$
Report users #ere loaded after dialog users$
-oth report and dialog users e)perienced longer response time$
P4 reached TTR in central instance #hen report users #ere loaded (refer (igure 11)$
*he transaction fre'uency decreased #hen report users logged in (refer (igure 11)$
%t #as concluded that report 'ueries should be t#eaked before #e conduct ne)t phase of
testing$
(igure 117 entral %nstance .indo#s Resources
(igure 117 *ransactional (re'uency
Sco/ing -( Scenarios
*#o scenarios #ere identified
Search #ith employee code and create a ticket
Search #ith employee name and create a ticket
PD concurrent users over /A< and 9D users over .A<$
E8ecution (esults
1
Scenario <sers Tic=ets
> hour
Test
duration
> mins
SAP ste/s -P<
Test & ?+ .A%
4+ 0A%
)4) &++ &5+++ @normal:
Test ' ?+ .A%
4+ 0A%
),5 )+ A*++ @normal:
*able 17 R! Results
,uring this test a specific parameter #as tuned to increase the R( calls #hich increased the
tickets creation;hour$
Parameter <ame 7 @rdisp;tmFma)FnoM value can be increased based on desired results$
Sco/ing BI0 Scenarios

*he ob2ective of this test #as to compare the response time of different -%. 'ueries #ith and
#ithout -%A for ?D concurrent users$ *he haracteristics of the test scenario #ere as follo#s7
C 'uery scripts #ere created that accessed different cubes in order to test different types
of 'ueries$ !ost 'ueries had 1 drilldo#ns
?D users #ere ramped up #ith C users at a time O the C users ran for 1D minutes before the
ne)t #ere added$ Each of the C users e)ecuted one of the C 'ueries
*ests #ere made using "**P and not "**PS$
aching7 =/AP aching on the server #as off
aching7 .orkstation caching #as disabled for the /oadRunner scripts
E8ecution (esults
%n order to be sure that the /oadRunner response times #ere representative of the relative end&user
response time #e conducted a Lsingle user testM #here #e compared7
B0 B0 2ith BiAccelerator
@*P test @*P test
!anual test !anual *est
/R test /R test
*able 17 -. e)ecution strategy
*est Results O Single 4ser *est
Summar$ Table for 9( Dashboard Buer$
-. -. #ith -iA
@*P 9T$Ps ?$0s
!anual Appro)7 9Cs Appro) 1&1 s
/R 9N$Ps 1$Cs
*able 97 -. single user results
*he results sho# that7
@*P adds a fe# seconds to its time calculation
*he results are largely in line bet#een the different methods
E)ecution time for the scripts remained more or less constant across iterations
*he results validated the test approach for using /oadRunner #ith no cache for e)ecuting the tests$
1
*est Results 7 -. #ith;#ithout -%A comparison

*he test scenario #as e)ecuted in an identical manner #ith and #ithout the -% Accelerator$
*he results sho#ed a significant decrease in response times #ith the -% Accelerator s#itched on$
8ey results are7
Summar$ (esults com/aring B0 2ith and 2ithout BiA
-. -. #ith -iA
Response times @ueries #ere ) times faster
P4 Appro)7 9CR Average ?1R
Response time
variation #ith
number of users
Response time #as erratic O in
general #as C&T times slo#er
#ith ?D users compared to C
users <o significant change in response time
*ransactions per
second
Ramp up correlating #ith
number of users O peaked at ?$C
tps
Ramp&up correlating #ith number of users$
Peaked at 11 tps and remained consistent$
*otal number of
transactions
?&P times more transactions created at peak
compared to #ithout -iA (depending on 'uery)
*able ?7 -. omparative Results
*he follo#ing chart sho#s the acceleration effect per script as the number of users increased$ .e have
selected only the -. actions (not the logins and log offs) for comparison$
(or some actions the effect of the accelerator #as in the range of ?) to 19) faster$ (or others, the
acceleration effect #as 1CD) O C?P) faster$
Acceleration Effect of Bi Accelerator
0
100
200
300
400
500
600
5 10 15 20 25 30 34 35 40
Number of Users
A
c
c
e
l
e
r
a
t
i
o
n

E
f
f
e
c
t
Number of Vusers
Dashbrd_020_Variable_Transaction
DB07_020_Variable_Transaction
DB07_030_Drilldon1_Transaction
DB07_040_Drilldon2_Transaction
!"02_020_Variable_Transaction
!"02_030_Drilldon1_Transaction
!"02_040_Drilldon2_Transaction
!"02_050_Drilldon3_Transaction
##04_020_Variable_Transaction
##04_030_Drilldon1_Transaction
##04_040_Drilldon2_Transaction
##05_020_Variable_Transaction
##05_030_Drilldon1_Transaction
##05_040_Drilldon2_Transaction
(igure 197 Acceleration of -%A
1
8. Summar$
%n this paper #e gave an overvie# of testing various SAP systems #hich are robust and comple)$
After over vie#ing best practices implemented in SAP volume testing, #e took up a case study to
describe ho# #e implemented those practices to find bottlenecks and various tuning options in
transaction processing and 'uery processing$ .e also briefly surveyed the monitoring part of SAP #ith
/oadRunner and through e)ecuting SAP transactions$
9. (eference aterials
222.sdn.sa/.com
###$support$mercury$com
10. Author:s Biogra/h$
Anshuman ,as is #orking #ith %-! since 1DD?$ "e has over N years of e)perience in %* and *esting$
urrently, Anshuman is #orking as a SAP Performance *esting consultant$ "e is involved in
consulting and creating performance testing strategies to ensure application performance enhancement$
"e has vast e)perience in domains like container and terminal management, logistics and ERP$ "e is
S*E certified from @A%$
11. A//endi8
SR 7 Service Re'uest$ Every pro2ect re'uest in testing is tracked through a uni'ue id$
*A*7 *otal *urn Around *ime
%S 7 %nformation Services
@A 7 *his is used in conte)t #ith environment #hich is a test or @uality Assurance environment$
P= 7 Purchase =rder$
*PS7 *ransactions Per Second
-.;-%.7 -usiness %ntelligence .arehouse
-%A7 -usiness %ntelligence Accelerator is a separate analysis platform, including both soft#are and
hard#are, #hich is net#ork attached to the SAP <et.eaver -% server via gigabit Ethernet link$
1