D83171GC10 Ag

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY.
COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
D85093
Edition 1.0
January 2014
D83171GC10
Activity Guide
Workshop
Performance Tuning
Oracle WebLogic Server 12c:
Oracle University and In Motion Servicios S.A. use only

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY. COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED
Copyright 2014, Oracle and/or its affiliates. All rights reserved.
Disclaimer
This document contains proprietary information and is protected by copyright and other intellectual property laws. You may copy and
print this document solely for your own use in an Oracle training course. The document may not be modified or altered in any way.
Except where your use constitutes "fair use" under copyright law, you may not use, share, download, upload, copy, print, display,
perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express authorization
of Oracle.
The information contained in this document is subject to change without notice. If you find any problems in the document, please
report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
warranted to be error-free.
Restricted Rights Notice

If this documentation is delivered to the United States Government or anyone using the documentation on behalf of the United
States Government, the following notice is applicable:
U.S. GOVERNMENT RIGHTS

The U.S. Governments rights to use, modify, reproduce, release, perform, display, or disclose these training materials are restricted
by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract.
Trademark Notice
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective
owners.
Authors
Elio Bonazzi, Mark Lindros
Technical Contributors and Reviewers

TJ Palazzolo, Bill Bell, Mathew Slingsby, Serge Moiseev, Kevin Tate, Takyiu Liu, Jose Alvarez,
Ahmed Gaid, Colin Stevenson, David Anderson, Mark Scarton, Mike Williams, Diganta
Choudhury, Eric Gross
This book was published using: Oracle Tutor

Table of Contents
Practices for Lesson 1: Course Introduction ........................................................................................... 1-1

Practices for Lesson 1............................................................................................................................. 1-2
Practices for Lesson 2: Performance Tuning - Overview ........................................................................ 2-1

Practices for Lesson 2: Overview............................................................................................................. 2-2
Practice 2-1: Setting Up the Course Practice Environment........................................................................ 2-3
Practice Solution ..................................................................................................................................... 2-7
Practice 2-2: Exploring the Course Applications ....................................................................................... 2-8

Practice 2-3: Executing a Benchmark ...................................................................................................... 2-19
Practice 2-4: Stress Testing an Application and Identifying Bottlenecks..................................................... 2-30
Practices for Lesson 3: Tuning Operating System Resources ................................................................ 3-1

Practice 3-1: Monitoring Linux CPU Utilization.......................................................................................... 3-3
Practice 3-2: Monitoring Linux Disk I/O .................................................................................................... 3-11
Practice 3-3: Monitoring Linux Virtual Memory and Network ..................................................................... 3-16
Practice 3-4: Analyzing and Tuning Linux Kernel Parameters ................................................................... 3-23
Practices for Lesson 4: Tuning HotSpot JVM .......................................................................................... 4-1

Practice 4-1: Tuning JVM Garbage Collection .......................................................................................... 4-3
Practice 4-2: Using Command-Line Tools ................................................................................................ 4-12
Practice 4-3: Using Java VisualVM .......................................................................................................... 4-23
Practice 4-4: Using VisualGC, jconsole, and GCHisto............................................................................... 4-31
Practice 4-5: Using Mission Control ......................................................................................................... 4-40
Practice 4-6: Using Flight Recorder ......................................................................................................... 4-49
Practices for Lesson 5: Monitor and Tune Server Performance .............................................................. 5-1
Practice 5-1: Configuring Server Stuck Thread Behavior .......................................................................... 5-3
Practice 5-2: Configuring Automatic Restarting of a Failed Server ............................................................ 5-7
Practice 5-3: Configuring WebLogic Overload Protection .......................................................................... 5-13
Practice 5-4: Configuring Log Filters for the WebLogic Server .................................................................. 5-19
Oracle WebLogic Server 12c: Performance Tuning Workshop Table of Contents

iii
Practices for Lesson 6: Monitor and Tune Cluster Performance............................................................. 6-1

Practice 6-1: Configuring Multicast and Unicast Clustering ....................................................................... 6-3
Practice 6-2: Tuning Session Persistence ................................................................................................ 6-11
Practices for Lesson 7: Tuning JDBC Performance ................................................................................ 7-1

Practice 7-1: Tuning JDBC Connection Pools .......................................................................................... 7-4

Practice 7-2: Testing Performance of JDBC Statement Caching ............................................................... 7-9
Practice 7-3: Testing Performance of "Pinned-To-Thread". ....................................................................... 7-12
Practice 7-4: Configuring Batch Updates .................................................................................................. 7-16
Practice 7-5: Tuning an Oracle Database for WebLogic............................................................................ 7-22
Practices for Lesson 8: Configuring Work Managers .............................................................................. 8-1

Practice 8-1: Creating and Using Work Managers .................................................................................... 8-3
Practice 8-2: Configuring Work Manager Constraints ............................................................................... 8-10
Practices for Lesson 9: Tuning J2EE Applications.................................................................................. 9-1

Practice 9-1: Tuning JSP......................................................................................................................... 9-3
Practice 9-2: Tuning Stateless EJBs ........................................................................................................ 9-10
Appendix A: Convenience Cheat Sheet ................................................................................................... 10-1

Using Convenience Scripts and Aliases in this Course ............................................................................. 10-2
Oracle WebLogic Server 12c: Performance Tuning Workshop Table of Contents

iv
Practices for Lesson 1

Chapter 1 - Page 1
Chapter 1

Course Introduction
Practices for Lesson 1:

Practices Overview
There are no practices for this lesson.

Chapter 1 - Page 2
Practices for Lesson 2: Overview

Chapter 2 - Page 1
Chapter 2
Overview

Performance Tuning -

Practices Overview
In the practices for this lesson, you set up the course lab environment and learn how to perform
basic application load testing tasks.
IMPORTANT NOTE!! This is a practice environment, with load generation tools running on the
same machines being performance tested. Each machine is also a virtual machine (VM) running
on the same single physical machine. This class will also get deployed across a big variety of
different environments around the world. As a result, you should focus mainly on the
methodology and use of the tools in this course rather than the actual results that can be

skewed by all these factors. Your performance metric results will almost certainly not match
those shown in the course, or even match those of other students in the same class.
The practices in this course were developed using a 64-bit machine with two physical CPUs and
8 GB of RAM, hosting two VM instances that represent host01 and host02. Each VM is
configured with two CPUs and 3.5 GB of RAM.

Chapter 2 - Page 2
Practice 2-1: Setting Up the Course Practice Environment
Overview
This practice shows you how to set up the initial course environment used for all the practices of
the course. You also verify that the WebLogic domain used for the course is functioning
properly. The following image depicts the architecture of the domain used for this practice.

Dependencies
All practices depend on the completion of this practice.
Tasks
1. Connect to the host01 and host02 machines.
a. Connect via VNC to host01:
On the main machine's desktop, double-click the VNC Viewer - host01 icon to start
VNC for host01.
Enter oracle as the password.

Chapter 2 - Page 3
b. Connect via VNC to host02:

On the main machine's desktop, double-click the VNC Viewer - host02 icon to start
VNC for host02.

Enter oracle as the password.
2. Set up the initial course environment.
Note: This step is required for all practices and solutions in this course to function properly.
a. Open a terminal window on each machine by clicking the terminal icon in the launch
panel located at the top of the screen.
b. Set the title of each terminal window by selecting Terminal > Set Title and entering
MAIN as the title. This makes it easier to distinguish the purpose of each window.
c. Run the following script on each machine to set the environment for this course. You
will not have to run this script again.
$ . /practices/tune/bin/setenv.sh
Note: This script writes this same command to your /home/oracle/.bashrc file so
the environment is automatically set for you from now on.
3. Set up the practice environment.
Note: When running any of the scripts in this course that start WebLogic servers, you can
generally ignore the following errors displayed in the server console windows:
Message Description
java.rmi.RemoteException: This is intentional because the scripts
[Deployer:149145]Unable to contact start the administration server first. Then
"serverX". Deployment is deferred until before starting the managed servers, it
"serverX" becomes available. performs WLST and deployment tasks
that are automatically realized when the
managed servers start. This eliminates
the need to restart servers for changes
that may not be dynamic.
<Sep 11, 2013 6:37:23 PM UTC> <Error> This is a harmless exception that is
<Cluster> <BEA-000109> <An error occurred occurring as part of the Auction
while sending multicast message: application. It will not affect the
java.io.NotSerializableException: functionality of the application or the
com.sun.jersey.server.impl.cdi.CDIExtension course.
a. In the MAIN terminal window, navigate to the domain folder on host01 and host02:
$ cd /u01/domains/tune/wlsadmin
Note: This is the WebLogic domain used for the practices in this course.

Chapter 2 - Page 4
b. Run the following command on host01 to start the domains AdminServer. The
startAdmin.sh script is located in this course's bin folder, which is in your PATH so
it can be executed from any folder location on host01.
$ startAdmin.sh
Note: The scripts used in this course automatically set the terminal window title to
AdminServer, server1, and server2 for each WebLogic server instance that runs within
that window.
c. Wait for the administration server to start.
d. Run the following command on host01 to start the domains server1 managed server.
The startServer1.sh script is located in this course's bin folder, which is in your

PATH so it can be executed from any folder location on host01.
$ startServer1.sh
e. Run the following command on host01 to start the domains server2 managed server.
The startServer2.sh script is located in this course's bin folder, which is in your
PATH so it can be executed from any folder location on host01.
$ startServer2.sh
Note: This script uses ssh to open a new terminal window on the host02 machine,
and starts server2 on host02.
f. Wait for both managed servers to start.
g. Copy the WebLogic proxy plug-in configuration for this course to the Oracle HTTP
Server (OHS) config folder on host01:
$ cp /practices/tune/bin/mod_wl_ohs.conf
/u01/app/fmw2/instances/webtier_1/config/OHS/ohs1
h. Start OHS by executing the following command in the MAIN terminal window on
host01. OHS is already configured for this course:
$ cd /u01/app/fmw2/instances/webtier_1/bin
$ ./opmnctl start
$ ./opmnctl startproc ias-component=ohs1
i. Ensure that OHS has started properly:
$ ./opmnctl status
Processes in Instance: webtier_1
---------------+--------------------+---------+---------
ias-component | process-type | pid | status
---------------+--------------------+---------+---------
ohs1 | OHS | 27615 | Alive
4. Verify the domain's configuration.
Note: All browser steps should be run from either host01 or host02 for this course.
a. Launch a web browser and log in to the Administration Console:
http://host01:7001/console
b. Log in by using weblogic as the username and Welcome1 as the password.
c. In the Domain Structure panel, expand Environment and select Servers.

Chapter 2 - Page 5
d. Verify that all three servers, AdminServer, server1, and server2, are in the RUNNING
state.
5. In the MAIN terminal window on host01, run the cleanup script (optional).
Note: This course uses the cleanup.sh script to restore the domain to its original state. It

is called by all setup and solution scripts in this course.
$ cleanup.sh
The cleanup.sh script performs the following tasks:
Ensures that it is executed by the oracle user
Ensures that it is executed on the host01 machine
Executes the stopDomain.sh script. This script gracefully shuts down the domain
and ensures that all java processes are closed on both machines.
Closes any terminal windows that are named AdminServer, server1, and server2.
Restores the original wlsadmin domain on both machines.

Chapter 2 - Page 6
Practice Solution
There is no solution for this practice. All practices depend on this lesson, most notably step 2 to
set the initial environment.
Practice Solution
Chapter 2 - Page 7
Practice 2-2: Exploring the Course Applications
Overview
This practice provides you with a very high-level view of the applications used in this course.
The Auction Application:

The Auction application consists of a web application that is built using the following Java EE
features. Most of the practices in this course use the Auction application.
Component Description

Java Server Pages (JSPs) The presentation tier of the application is implemented using
JSP pages that represent the V, or view, of a standard MVC
application design pattern. MVC stands for Model (data), View
(presentation), Controller (logic).
Servlets The business logic tier of the application is partially
implemented using Servlets that represent the C, or controller,
of a standard MVC application design pattern.
CDI (Context Dependency The data tier of the application is partially implemented using
Injection) Managed Beans CDI managed beans that represent the M, or model, of a
standard MVC application design pattern. The CDI objects in
this application act as services that are called by servlets to
perform data operations against the database. The CDI beans
are persistence-aware, but not persistence-capable. They
manage the JPA persistence-capable classes of the
application.
JPA (Java Persistence API) The data tier of the application is partially implemented using
JPA entities that represent the M, or model, of a standard MVC
application design pattern. JPA classes provide object to
relational mapping capabilities and their data is persisted to the
database. The JPA layer abstracts database calls from the
application, enabling developers to focus on their object model.
Application deployment package:

/practices/tune/apps/SimpleAuctionWebAppDb.war

Chapter 2 - Page 8
Auction Use Cases:

Populating the initial database:

Chapter 2 - Page 9
Viewing the auction list:

Chapter 2 - Page 10
Viewing an auction:
The Create Auction and Bid features of the application are not used for this course.

Chapter 2 - Page 11
The Shopping Cart Application:

The Shopping Cart application consists of a web application that is built using the following Java
EE features. This application is used to show session replication performance in this course
because it keeps a users cart in an HTTP Session.
Java Server Pages (JSPs) The presentation tier of the application is implemented using
JSP pages that represent the V, or view, of a standard MVC
application design pattern. MVC stands for Model (data), View
(presentation), Controller (logic).

Servlets The business logic tier of the application is partially
implemented using Servlets that represent the C, or controller,
of a standard MVC application design pattern.
HTTPSession The users session state is stored in a standard HTTPSession
object.
Application deployment package: /practices/tune/apps/ShoppingCart.war
Shopping Cart Use Cases:

Viewing the Shopping Cart:

Chapter 2 - Page 12

Chapter 2 - Page 13
Going Shopping:

Chapter 2 - Page 14

The StressTestOS Application:
The StressTestOS application was built to illustrate the concepts covered by Lesson 3, Tuning
Operating System Resources.
StressTestOS.jar The StressTestOS application exposes a set of methods
through the JMX interface. The methods allow interactive users
to stress test the CPU, the I/O subsystem and the network.
SocketServer.jar Implements a socket server, which responds to requests on
port 9090, generating large volumes of random data. The goal
is to stress test the network. SocketServer is run on the host02
node. The client application connects to the socket server from
host01. The generated high volume of data is sent across the
network card simulating a high level of traffic between host01
and host02.
jconsole The JDK standard utility jconsole is used as a basic GUI
interface to interact with the various methods implemented as
JMX beans.

Chapter 2 - Page 15
The jconsole GUI allows interaction with StressTestOS. Users can start and stop the process
that cause stress on the system:

The StuckThreads Application:
The StuckThreads applications only purpose is to provoke stuck threads within the WebLogic
Server (Practice 5-2.)
index.jsp The applications entry point. It allows interactive users to
select how many threads should be spawned, and how long
each thread should run for.
StuckServlet The business logic tier of the application. The servlet invokes
EJB methods asynchronously, provoking a prolonged CPU
stress on the WebLogic Server.
StuckThreadExecutor Asynchronous stateless EJB which simulates multi-threaded
behavior

Chapter 2 - Page 16

The StuckThreads application allows users to create stuck threads in WebLogic.
Feedback page after StuckThreads has been called.
The Memory Exhaustion Application:

The Memory Exhaustion application is used in practice 5-3 to simulate running close to
maximum available memory, in order to force WebLogic to change the state of a server to
Overloaded.
index.jsp The applications entry point. It allows interactive users
to select how many threads taking up memory should
be spawned, and how long each thread should run for.
MemoryServlet The business logic tier of the application. The servlet
invokes EJB methods asynchronously, provoking
prolonged memory exhaustion on the WebLogic Server.
MemoryExhaustionExecutor Asynchronous stateless EJB, which simulates multi-
threaded behavior

Chapter 2 - Page 17
The Memory Exhaustion application allows users to simulate running out of memory in a

controlled manner.
Feedback page after Memory Exhaustion has been called.
The TestBatchUpdates Application:

The TestBatchUpdates application is used during practice 7-4 to compare the difference in
performance between single inserts with a commit after each insert, versus batch updates with
one commit after all inserts.
index.jsp The applications entry point. It allows interactive users to
select how many insert statements should be submitted, and
what type of insertion (single or batch) should be executed.
BatchUpdatesServlet The business logic tier of the application. The servlet invokes
EJB methods causing the insertion of randomized strings into
the database.

Chapter 2 - Page 18
BatchUpdatesExecutor Stateless EJB, which interacts with the Oracle database

The TestBatchUpdates application allows users to submit either single insert statements or the
same number of statements, but in batch. The initial screen shows the source code used for
each of the two methods used to submit insert statements.

Chapter 2 - Page 19
Practice Solution
There is no solution for this practice. All practices are based on this information, but no practices
depend on this practice.
Practice Solution
Chapter 2 - Page 20
Practice 2-3: Executing a Benchmark
Overview
This practice shows you how to set up Grinder to place a load on an application. You load the
Auction application to get some metrics to establish the baseline performance characteristics of
the application when running in a steady state. You will record performance metrics including
CPU utilization, application throughput, response time, hard disk utilization, memory utilization,
and network utilization. These performance metrics will be used in subsequent practices for
comparison with other performance numbers to see if performance improves or declines. The
primary purpose of this practice is to give you hands-on experience with running the tools and
analyzing the resources used in this course.

Note that the entire environment is running on a single machine. Host01 and host02 are VMs
running on the same machine, and the application and load testing software are also running on
the same physical hardware. In normal performance testing scenarios, WebLogic servers would
run on different physical hardware, as would the load testing software, to avoid skewing the
actual performance of the application itself. In this practice environment, you have to accept that
everything is running on the same physical hardware, and keep in mind that the comparisons
should still be valid because every test involves the same topology.
The image above depicts the architecture of the domain used for this practice:
There are two machines, host01 and host02.
Each machine is hosting a WebLogic managed server, server1 on host01 and server2
on host02.
Both servers form a cluster and have the Auction application deployed.

Chapter 2 - Page 21
The Auction application uses the Oracle database running on host02.

Oracle HTTP Server (OHS) is running as the load balancer for the managed servers.
The Grinder agent is running on host01, and is configured to load the application using
the OHS load balancer.
When the application is running under load, various tools are used to monitor the
performance of the machines, application, and JVMs. Note that you are not using JVM-
based tools such as VisualVM, Mission Control, and Flight Recorder yet because these
topics have not been covered in this course yet.
About This Course's Approach

Most practices in this workshop course provide you with high level tasks that you must
accomplish in order to complete the practice. In general, the practice provides you with the
information that you need to perform the task on your own. Sometimes, there is more than one
way to approach a task, so this course gives you some flexibility to perform the task in your own
way, while using the guidelines presented by the practices. A workshop course tends to not use
direct, step-by-step, detailed instructions that give you every single aspect of a task to perform.
This approach requires more free-thinking on your part, but also can provide a better learning
experience.
There are some exceptions to this rule for certain tasks that teach you how to use a particular
tool. After you are shown how to use the tool, it is up to you to apply that knowledge across the
practices of this course.
Dependencies
This practice depends on practice 2-1 for setting up the course environment.
Tasks
1. Connect to the host01 and host02 machines by using VNC.
a. Perform the following commands on host01 to set up the environment for this practice:
$ cd /practices/tune/practice02-03
$ ./setup.sh
This script does the following:
Restores the domain to its original state
Starts the wlsadmin AdminServer
Deploys the SimpleAuctionWebAppDb application to the cluster
Starts the server1 managed server on host01
b. Wait for all servers to log that they are in the RUNNING state before continuing.
3. Configure the Grinder proxy.
The Grinder proxy runs on the machine and intercepts web browser requests made with the
browser. It uses this to create load scripts by capturing and recording these requests to
form the infrastructure used to load test your application.
a. Launch Firefox on host01.

Chapter 2 - Page 22
b. Select Edit > Preferences to load up Firefox preferences.

c. Click the Advanced icon.
d. Click the Network tab.
e. Click the Settings button.
f. Copy and paste the current settings into a text file, so you can restore them later.
g. Select Manual proxy configuration and edit the following values:
Field Value
HTTP Proxy localhost

Port 8001
No Proxy For <empty>
Click OK.
h. Click Close and restart Firefox. You will receive an error because the proxy server is
not running yet. Important! Firefox is configured to automatically open the WebLogic
Administration Console page. If for some reason you must restart your TCPProxy
process, you want to make sure that you do not record the WebLogic administration
page as part of your testing. Doing so would mean you are not really performing a
clean test of the application because another application would be part of the test. If
you prefer, you can use Firefox settings to disable the start page to avoid this possible
issue.
i. Open a new terminal window. You can do this in several ways:
Click the terminal window icon on the launch bar.
Double-click the terminal window icon on the desktop.
Execute the following command from within your current terminal window:
$ prompt.sh
Note: The most efficient method is to use the prompt.sh script because it will open
the new terminal window with a current working directory that matches the one in the
current terminal window. The other methods require navigating to the practice folder
again. Remember to enter commands in the new terminal window after executing the
prompt.sh script.
j. In the new terminal window, run the setenv.sh script in the practice folder. This sets
the environment for The Grinder.
$ . ./setenv.sh
k. Execute the following command to start the Grinder proxy process:
$ java net.grinder.TCPProxy -console -http > auction-
application.py
l. Confirm that the proxy started successfully:
17:05:38.211 [main] INFO : Engine initialised, listening on port
8001
m. The TCPProxy Console window will display (because you specified the -console
option):

Chapter 2 - Page 23
4. Record a test case.

Note: The Auction database is already created and populated for your convenience. If for
some reason it is not created, execute the
/practices/tune/bin/backup/BuildAuctionDatabase.sh script to create and
populate the database. This script works whether it is run from host01 or host02.
a. Browse to http://host01:7777/SimpleAuctionWebAppDb using Firefox.

b. Use the application to simulate a real user using the application. View the auction list
and look at individual auctions to look at its details. If you try to create an auction and
are not able to login or find an image for your auction, it may still be a valid user test
case for use with the Grinder tool. The same is true for trying to bid on existing
auctions. Do not worry if the functions do not work as much as making sure that you
are just using the application as a normal user would try to use it. If you make a
mistake or something goes wrong while you are trying to do this step, restart the
TCPProxy process and start step 4 over.
Note: The functionality of this application is limited and using all aspects of this
particular application are not required for completing this course.
c. In the TCPProxy Console, click the Stop button to stop recording your test case.
d. Inspect the generated script, auction-application.py, just to see the Jython code
that the tool has put together. This script represents a set of test cases based on the
recorded actions you just made using the web browser. Grinder uses this script to load
test the application. You can reuse this test scenario for the entire course for practices
that use the same application, and compare performance characteristics of the
application in each practice against its baseline performance in this practice.
e. Restore Firefox's original proxy settings so web browsing works the way it usually
works because you are no longer recording your activity.
5. Start the Grinder console and agent.
a. Execute the following command to start the Grinder console. Use the same terminal
window you used to run TCPProxy or a window that has the Grinder environment set
properly.
$ java net.grinder.Console
b. Verify that the Grinder console window appears:

Chapter 2 - Page 24

c. Locate the Ignore 0 samples field in the console and set its value to 10 to give
WebLogic time to tune its own performance based on the simulated load. This is
equivalent of letting WebLogic warm up a little.
d. Open a new terminal window (remember that using prompt.sh from the MAIN
terminal window is most efficient).
e. Name the new window Grinder Agent.
f. Open the grinder.properties file found within the current practice folder. This file
is provided for you for convenience. It contains the settings used by Grinder to indicate
how it should load the application, and specifies which script file to run. Take a few
moments and view the settings in the file. Keep the file open for future editing.
g. Within your new Grinder Agent terminal window, execute the following command to
start the Grinder agent process. Note that if you used prompt.sh to open the new
window, your environment is still properly set. Otherwise, execute the setenv.sh
script in the practice folder to set the environment for Grinder before continuing.
$ java net.grinder.Grinder
h. Confirm that your agent started successfully and that it discovered your console:
2013-09-10 09:49:56,602 INFO agent: The Grinder 3.11
2013-09-10 09:49:56,734 INFO agent: connected to console at
localhost/127.0.0.1:6372
2013-09-10 09:49:56,735 INFO agent: waiting for console signal
6. Run the test case.
Good Practice Tips: Review the table below for ways to make using Grinder easier.
Topic Description / Tip
Options Grinder always leaves turning sampling on and off,

Chapter 2 - Page 25
Topic Description / Tip

and resetting the console when resetting worker
threads, up to you as a manual task by default. This
results in a lot of extra (or forgotten) clicking when
using Grinder in earnest. Select File > Options >
Miscellaneous and check the Reset Console with
Worker Processes check box so Grinder will do this
automatically for you.
Collecting Samples When a script has completed loading your application,
Grinder continues to collect samples from its agents.
If there is no load, then the samples come in with no

statistics of importance and you will see your mean
TPS figure diminish with each sample. Remember to
do the following when using Grinder to do your load
tests:
Use Start collecting statistics just before clicking
Start the worker processes.
Use Stop collecting statistics right after a load

test has stopped producing figures, and before
clicking Reset the worker processes prior to

running another load test.
These steps synchronize statistic collection with the
actual load test to ensure numbers are more
accurate.
Dialog check boxes Whenever there is a dialog window with a check box
that states something similar to Dont ask me this
again, check it to avoid the extra clicking for future
tests. This will save you some frustration when
running multiple tests using Grinder.
a. On host01 and host02, open a new terminal window and execute the top command to
view performance metrics. This will let you know the CPU usage measurements for
each host.
$ top
b. In each terminal window where you ran the top command, press z to turn color mode
on, and then press 1 to cause top to display each CPU separately.
c. Take note of how the system looks prior to running any tests. CPU utilization should be
relatively low on each host. Take into consideration what software is running on each
machine:
Host01 is running Grinder, the AdminServer, server1, and whatever windows or
applications you are using.
Host02 is running server2 and the database. Resource use depends a lot on how
much work the database is doing, but in general you should see less CPU activity
on host02.

Chapter 2 - Page 26
d. Here is a brief explanation of the columns for CPU utilization for the top command.
Use this as a reference when you are monitoring CPU utilization on each machine.
Column Description
Cpu0..n Represents each CPU instance on the machine
%us Percentage of time spent executing user processes that were not
prioritized, or niced
%sy Percentage of time spent executing the kernel and its processes
%ni Percentage of time spent executing user processes that were
prioritized, or niced

%id Percentage of the time the CPU was idle (doing nothing)
%wa Percentage of the time the CPU spent waiting for I/O to complete
%hi Percentage of the time the CPU spent servicing hardware
interrupts
%si Percentage of the time the CPU spent servicing software
interrupts
%st Percentage of the CPU time that was stolen by the hypervisor
from this virtual machine for other tasks
e. Return to the Grinder console and click the Results tab.
f. On the main toolbar, if it is not already selected, click the Start collecting statistics
button.
g. On the main toolbar, click the Start the worker processes button. Click OK if you are
prompted to run the agents default property file.
h. As the agent runs, the left pane displays the number of collected samples along with
the response time and throughput statistics of the samples.
i. As the agent runs, use the Linux System Monitor to monitor CPU utilization.
j. In the Results tab of the Grinder console, locate the Successful Tests column.
k. Return to the Grinder Agent terminal window and verify that the test script finished:
2013-09-10 14:27:09,771 INFO host01.example.com-0: finished
2013-09-10 14:27:10,261 INFO agent: finished, waiting for
console signal
l. Locate the Total section in the lower left area of the Grinder console. This contains the
performance numbers for the Grinders load testing.

Chapter 2 - Page 27

Note: Your numbers will be different.
Chapter 2 - Page 28
m. You can also review the total lines in Grinders table output:

7. Record performance metrics.
a. The following table describes which tools are used to capture each type of
performance characteristic of the system. Use these tools to capture performance
metrics for your application tests. You will learn more about these tools in the lesson
titled Tuning Operating System Resources.
Tool Description
top Provides a real-time view of your running machine. It displays the most
CPU-intensive processes and shows important system resource utilization
in general, including CPU, memory, and virtual memory.
vmstat Provides information about processes, memory, paging, blocking I/O, traps,
and CPU
ps Displays the running processes on the machine. It also provides some other
important information, including thread details
free Displays the total amount of free and used physical and swap memory on
the machine
iostat Provides CPU load, and I/O statistics for devices such as disks and network
file systems
sar Used to collect, report, and save information about the running system,

Chapter 2 - Page 29
Tool Description
including network activity
mpstat Displays activities for every processor on the machine
pmap Displays the memory map of a running process
netstat Displays network connections, routing tables, and related statistics
ss Displays network socket statistics
Grinder Grinder displays reports that provide average response and throughput
metrics for its load tests. You can also use the Grinder console to see charts
for performance characteristics of your tests.

b. While the application is running, monitor the following system resources using the tools
described in the table above. Write down the performance numbers you see in a form
similar to the following. You may also want to take note of the number of successful
tests and number of errors.
Use a separate sheet of paper for each host so you can compare numbers easily.
CPU Disk Network Used Free Mean Mean # #
I/O I/O Memory Memory Response TPS Successful Errors
time Tests
c. On the main toolbar, click the Reset the worker processes button. When prompted
to reset the console statistics, click Yes.
d. Run the tests more than once to give yourself a chance to try different tools to capture
these metrics in different ways. Record the results of each test, and compare each of
them to get an idea of the deviations between each test run and exactly how the
application it performing. When working on each test, make sure that there is sufficient
time left to complete all the practices in the lesson.
8. Increase the load and record performance metrics again.
Now you will change the load that Grinder places on your application to push the system a
little harder. You are trying to determine a reliable steady state load that the current
environment and configuration can manage without problems.
a. Modify the grinder.properties file so that it looks like the following:
grinder.processes=2
grinder.threads=50
grinder.runs=2
grinder.logDirectory=logs
grinder.numberOfOldLogs=2
grinder.script=auction-application.py
b. Record the same metrics again and ask yourself the following questions:
How do the numbers differ from the previous run?
What factor does having the Grinder application running on the host01 machine
have?
Was the application response time or throughput better or worse?

Chapter 2 - Page 30
Were there any errors?

Where would you check for errors?
Did there seem to be any system resource limitations that were reached? How you
could you tell?
Did the application appear to have any performance limitations? How could you tell,
or what symptoms did you notice?
Click the Graphs tab in the Grinder console. What information is available on this
tab?
9. Leave the environment running for the next practice.
The next practice uses Grinder and the same environment so it is best to leave it running.

Chapter 2 - Page 31
Practice Solution
Perform the following tasks if you did not complete this practice and want to use the finished
solution. Any practices that use Grinder depend on students knowing how to perform these
tasks.
Solution Tasks
1. Open a new terminal window on host01.
2. Change the current directory to the current practice folder.
3. Execute the solution script for this practice:

$ ./solution.sh
This script performs the following:
Sets the practice to its starting point by calling the setup.sh script, which in turn
calls the cleanup.sh script to reset the domain and general environment
Starts all the servers of the domain
Ensures that the Firefox proxy settings are set properly
Copies solution files to the practice folder
Opens two terminal windows with the Grinder environment set, Grinder Console and
Grinder Agent, for you to execute practice commands
4. Wait for all servers on host01 and host02 to fully start.
5. Continue starting at step number 5, ignoring the instruction to reuse the TCPProxy terminal
window.
Practice Solution
Chapter 2 - Page 32
Practice 2-4: Stress Testing an Application and Identifying

Bottlenecks
Overview
Now that you have set up and run the Grinder load testing tool against the Auction application,
you are ready to start putting more load on the application to see how far it can be pushed. This
practice builds on the previous practice, and has you modify the test case settings to increase
the load on the application, and has you use various tools to measure the performance
characteristics of the application, machine, and JVM. You will push the application until a
system resource that the application depends upon reaches capacity. Then you will use

monitoring tools to identify the bottlenecks of the application.

This practice provides only guidelines for how to accomplish the objectives of the lesson. You
take lessons learned from the slides or previous practices and apply them for this practice on
your own.
Dependencies
This practice depends on:
Practice 2-1 for setting up the course environment.
Practice 2-3 for configuring Grinder and establishing initial baseline performance
numbers of the Auction application in this environment.
Tasks
1. Ensure that the environment from practice 2-3 is still running.
a. This practice reuses the practice 2-3 environment. If the environment is not running,
then perform the solution steps for practice 2-3, and then return to this practice.
2. Increase the load placed on the system and the application using Grinder.
a. Increase the settings of the grinder.properties file to increase the load on your
application.
b. While the application is running, monitor the following system resources using the
same tools described in the table from practice 2-3. Write down the performance
numbers you see in a form similar to the following. You may also want to take note of
the number of successful tests and number of errors.
Use a separate sheet of paper for each host so you can compare numbers easily.
CPU Disk Network Used Free Mean Mean # #
I/O I/O Memory Memory Response TPS Successful Errors
time Tests
c. Run the tests more than once to give yourself a chance to try different tools to capture
these metrics in different ways. Record the results of each test, and compare each of
them to get an idea of the deviations between each test run and exactly how the
application it performing. Just keep your mind on how long each test takes to run
because you have other practices to perform.
Practice 2-4: Stress Testing an Application and Identifying Bottlenecks

Chapter 2 - Page 33
3. Based on your results, try to answer the following questions:

a. How do the numbers differ from the load tests you did in practice 2-3?
b. Was the application response time better or worse? Why?
c. Were there any errors? If so, what do you think caused them? How could you tell there
was an error?
d. Did any of the WebLogic servers crash? If so, why do you think they crashed?
e. Did you see any system resource limitations during your load testing? How could you
tell, or what symptoms did you notice? What do you think caused it?
f. Did you determine what the primary bottlenecks were while running your load tests?

What were they? Why do you believe these are your bottlenecks?
g. Based on your load testing overall, what do you think are the baseline performance
metrics for the auction application in this environment? Keep these figures for future
practices for comparison. You should also record the Grinder configuration that you
think hits the system hard without getting any errors in case you need it later.
h. What was the workload for each WebLogic managed server? Hint: Use the WebLogic
administration console to view each server. Use the servers Monitoring tab and its
sub-tabs to see its threads, memory, subsystem health, network channel statistics,
JDBC connections, and so on.
4. Load test again at a steady state and identify application bottlenecks.
a. Configure Grinder to run a load against the application that allows the application to
function properly without errors.
b. Run Grinder to load your application again, and use the tools to analyze different
aspects of the application. See if certain components are taking longer than others to
narrow down areas of the application that can be improved to increase performance.
c. It is all right if you do not have all of the answers now. This is just the first lesson, and
you have not been shown all the tools that enable you to see inside WebLogic, the
JVM, and your application yet.
Course Developer Corner:

There are no real right or wrong answers for these questions in a workshop. Every student will
have a different experience, based on the factors of their machines.
You need to keep in mind that when you are getting your results, you must be familiar with how
one system resource can affect other system resources. Most students should experience an
over-utilized CPU for both host01 and host02. However, some students may think that there is
an I/O problem because they may see broken pipes or socket problems in the WebLogic server
console windows. The main reason that there were networking or other processing problems is
mainly because the CPU was completely pinned and the system had nothing left to schedule to
process networking and other tasks, so they fail. If the issue was truly an over-saturated
network or other networking issue, then the CPU on the system would have been partially idle,
with plenty of processing capability available.
If you properly hit the system hard enough, which would work with Grinder configured to use 6
processes, 50 threads, and 2 runs, then you should have encountered errors due to CPU
limitations. Your tests at some point should actually fail themselves because Grinder is running
on the same system that is being loaded. You should see exceptions listed in the Grinder Agent
terminal window. The tests may fail entirely at some point, whereby you might expect more

Chapter 2 - Page 34
results returned than actually are. This can occur because the Grinder Agent was not able to
contact the Grinder Console to report its statistics.
In my load testing, none of the WebLogic servers crashed. I am not too surprised because
WebLogic has been designed to handle heavy loads and failures gracefully. Although we have
not configured any overload protection features at this point, WebLogic managed to outperform
the CPU and the system.
After you get a suitable load that pushes the system without over-saturating it, you can run the
system load at that level, and then you can analyze the application and system to see where
performance of the current system shows room for improvement. For example, the system is
running and providing 100 TPS, but analysis of the application may show that requests are
taking a long time to return from a particular application component, or the database, and so on.

That is how you begin to identify the bottlenecks of your system. After you identify such a
bottleneck, you can implement ways to improve the bottleneck to see if you can squeeze more
performance out of the system with the same load. Better yet, get even more performance by
being able to run a higher load because the bottleneck has been eliminated.
The primary focuses of these practices are to get you used to running load tests against your
application, using tools to capture performance metrics, analyzing the results, and determining
the current capabilities of your system.

Chapter 2 - Page 35
Practice Solution
There is no solution for this practice. No practices depend on this practice.
Practice Solution
Chapter 2 - Page 36
Practice Solution
Chapter 2 - Page 37

Tuning Operating System
Resources
Chapter 3

Chapter 3 - Page 1
Practices Overview
In these practices, you will selectively stress test the major components of a server computer
(CPU, Disk/IO, and network) using a provided stress test application; you will use the tools
provided by Linux to monitor the activity induced on the system by the stress test tool.
The table below shows what Linux/UNIX tool you will be using, its parameters (if applicable),
and what practice/step you encounter the commands/tools:
Linux/UNIX tool Practice Step

Gnome System Monitor 3-1 6
gkrellm 3-1 7
sar -u 1 3 3-1 16
ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10 3-1 17, 20
iostat -c -d 2 3-1 18
mpstat p ALL 3-1 19
iotop 3-2 4
iostat -x 1 3-2 5
iostat -d 2 3-2 8
netstat -i 3-3 3
sar -n DEV 3-3 5
vnstat -l -i eth0 3-3 6
nethogs 3-3 7
vmstat 2 10 3-3 8

Chapter 3 - Page 2
Practice 3-1: Monitoring Linux CPU Utilization
Overview
In this practice, you use the StressTestOS application, located in
/practices/tune/practice03-01, to generate some load on the CPU of the host01
server and, while the CPU is under stress, you use various utilities (mpstat, Gnome System
Monitor, gkerrlm and sar to monitor and assess the CPU load produced by StressTestOS.
The StressTestOS Java application will be run on the host01 host. A companion application,
called SocketServer, must be run on host02. The socket server application running on
host02 is accessed by StressTestOS when network activity must be simulated.

This practice provides detailed instructions for how to accomplish the objectives of the lesson.
You take lessons learned from these practices and apply them to subsequent practices on your
own.
Dependencies
Practice 2-1 for setting up the course environment
Tasks
2. Open three terminal windows on host01.
3. In a terminal window on host01, navigate to /practices/tune/practice03-01 and
launch the StressTestOS application.
$ java jar StressTestOS.jar
StressTestOS started
4. Open a terminal window on host02.
5. In a terminal window on host02, navigate to /practices/tune/practice03-01 and
launch the SocketServer application.
$ java jar SocketServer.jar
Waiting for clients to connect
6. In an available terminal window on host01, launch the Gnome System Monitor application:
$ gnome-system-monitor &
Note the ampersand (&) character at the end of the line. It forces the shell to launch the
application and return the prompt so that a user can submit more commands.

Chapter 3 - Page 3
7. The Gnome System Monitor starts:

8. In the same window where you launched sysmon, enter the command which launches
gkrellm:
$ gkrellm &

Chapter 3 - Page 4
9. The gkrellm application starts:

10. In the third terminal window on host01, launch jconsole:
$ jconsole
11. Under "Local Processes" select StressTestOS.jar and click "Connect." The initial
connection attempts fails, because an insecure (that is, non-TLS) connection is detected.
Click "Insecure" to proceed using an insecure connection:
12. jconsole starts. Click the "MBeans" tab, and then click the StressTestOSAgent
MBean. Click the "Controlbn" node under StressTestOSAgent. Finally, click the
"Operations" node in the StressTestOSAgent MBean tree. The right pane in the

Chapter 3 - Page 5
jconsole window displays a page with several controls which can be used to invoke
methods provided by StressTestOS:

13. The three methods StressCPU(), StressDisk() and StressNetwork() receive one
parameter (p1). In all cases, the parameter is the number of threads carrying out the
stressing task. Enter "3" as a parameter for the StressCPU method. Click the StressCPU
button to start stressing the CPU using three threads:

Chapter 3 - Page 6
14. jconsole responds with a little feedback window, stating that the method was successfully
invoked:

15. Observe the two windows hosting the sysmon and gkrellm applications, which you
previously launched. They should both show intense CPU activity:
16. Select the second terminal window you created in step 2, where the last command entered
was the launching of gkrellm. Experiment with the sar, ps, iostat, mpstat and
vmstat commands:
$ sar -u 1 3
Linux 2.6.39-400.109.3.el6uek.x86_64 (host01.example.com) 09/20/2013 _x86_64_
(2 CPU)
12:32:01 AM CPU %user %nice %system %iowait %steal %idle

12:32:02 AM all 50.26 0.00 2.05 0.00 0.00 47.69
12:32:03 AM all 50.77 0.00 1.54 0.00 0.00 47.69
12:32:04 AM all 47.92 0.00 2.08 0.00 0.00 50.00
Average: all 49.66 0.00 1.89 0.00 0.00 48.45
[oracle@host01 ~]$
The "sar u 1 3" command displays real-time CPU usage every 1 second for 3 times. In your
environment, expect to see slightly different numbers; however, with three active threads
stressing the two CPUs, you should see an idle time close to 50%

Chapter 3 - Page 7
17. The command "ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10"

displays the ten most expensive, in terms of CPU cycles, processes running on the system:
$ ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
%CPU PID USER COMMAND
55.4 6905 oracle java -jar StressTestOS.jar
14.6 4759 oracle /usr/NX/bin/nxagent -D -options
6.6 5216 oracle gnome-system-monitor
5.0 7474 root ps -eo pcpu,pid,user,args
3.6 4677 nx sshd: nx@notty
3.0 7138 root gkrellm

3.0 2540 oracle ora_vktm_oggcoh
2.5 5455 oracle jconsole
1.0 4775 nx /usr/NX/bin/nxssh -B -E
Note that StressTestOS, in the example above, takes more than 55% of CPU time. In
your environment, that percentage could vary, but it should be in the same range.
18. The iostat command, run with the "-c" option, displays CPU usage statistics. The "-d 2"
options force iostat to re-compute the statistics every two seconds:
$ iostat -c -d 2
avg-cpu: %user %nice %system %iowait %steal %idle

88.38 0.00 0.76 0.00 0.00 10.86
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn

sda 1.00 0.00 32.00 0 64
dm-0 1.00 0.00 32.00 0 64
dm-1 0.00 0.00 0.00 0 0
dm-2 0.00 0.00 0.00 0 0

90.66 0.00 0.76 0.00 0.00 8.59

sda 2.50 0.00 64.00 0 128
dm-0 8.00 0.00 64.00 0 128
dm-1 0.00 0.00 0.00 0 0
dm-2 0.00 0.00 0.00 0 0
Once again, your environment will show similar, but not identical, statistics. With three
threads running CPU stress tasks, iostat shows you that the CPU of the system is only
idle approximately 10% of the time, while 88% of the CPU cycles are taken by the system
to satisfy the CPU stressing tasks. Less than 1% of CPU time is spent in system (or
"kernel") mode, which shows a very healthy system. It is the system time which should be
carefully monitored, values exceeding 20~30% indicate that the system is spending too
long in kernel mode.
19. The command mpstat shows the multi-processor CPU statistics. When invoked with the
"-P ALL 2" options, it displays statistics for all CPUs, every two seconds:

Chapter 3 - Page 8
$ mpstat -P ALL
04:13:00 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
04:13:02 AM all 86.87 0.00 0.76 0.00 0.00 0.00 0.00 0.00 12.37
04:13:02 AM 0 86.87 0.00 1.01 0.00 0.00 0.00 0.00 0.00 12.12
04:13:02 AM 1 87.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 12.63
04:13:04 AM all 88.61 0.00 0.51 0.00 0.00 0.00 0.00 0.00 10.89
04:13:04 AM 0 88.32 0.00 0.51 0.00 0.00 0.00 0.00 0.00 11.17
04:13:04 AM 1 87.94 0.00 1.01 0.00 0.00 0.00 0.00 0.00 11.06
04:13:06 AM all 89.90 0.00 1.26 0.00 0.00 0.00 0.00 0.00 8.84

04:13:06 AM 0 88.89 0.00 1.52 0.00 0.00 0.00 0.00 0.00 9.60
04:13:06 AM 1 90.95 0.00 1.01 0.00 0.00 0.00 0.00 0.00 8.04
20. Stop the StressTestCPU() method. Select the jconsole window and click the
"StopCPUStress" button. Keep an eye on sysmon and gkrellm, and after a few
seconds re-run the CPU statistics commands you run previously (sar, ps, mpstat,
and so on.) Compare the differences in their output, after the CPU stress tasks have been
stopped. Example: after having stopped the CPU stress tasks, the system used to develop
this course showed that the StressTestOS application, which previously was taking more
than 55% of CPU, dropped to 5.7%:
$ ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10
%CPU PID USER COMMAND
15.0 4759 oracle /usr/NX/bin/nxagent -D -options
6.8 5216 oracle gnome-system-monitor
5.7 7804 oracle java -jar StressTestOS.jar
3.7 4677 nx sshd: nx@notty
3.1 7138 root gkrellm
3.0 2540 oracle ora_vktm_oggcoh
2.5 5455 oracle jconsole
1.0 4775 nx /usr/NX/bin/nxssh -B -E
0.5 4926 oracle /usr/lib/vmware-tools/sbin64/vmtoolsd -n
As usual, your environment will show some differences, but the crucial point is that after
stopping the CPU stress tasks, the StressTestOS application is not anymore at the top of
the list of most expensive applications running on the system.
21. Feel free to experiment with StressTestOS and jconsole. For example, increase the
number of threads from three to five or ten, click on StressCPU and observe how the
system reacts to the heavier stress tasks. DO NOT exceed twenty threads; otherwise, the
system could become unresponsive and difficult to control, short of an abrupt shutdown.
22. When you finish experimenting with CPU statistics, leave jconsole open but remember to
click the StopCPUStress button. jconsole will be used in the next practice.

Chapter 3 - Page 9
Practice Solution
There is no solution for this practice. However, you can apply the lessons learned in this
practice to subsequent practices in this course.
Practice Solution
Chapter 3 - Page 10
Practice 3-2: Monitoring Linux Disk I/O
Overview
In this practice, you use the StressTestOS application, set up during practice 3-1, to cause
Disk I/O activity on the system and you use the various Linux provided commands to assess
and analyze the Disk I/O statistics as reported by the system under stress.

own.

Dependencies
Practice 3-1 for setting up the environment for this practice.
Tasks
1. Enter 5 as a parameter for the StressDisk() method, which will instruct StressTestOS
to spawn five threads which will generate heavy disk I/O activity.

Chapter 3 - Page 11
3. Within the new terminal window, use the su command to log in as the super user (root).
The password for the root user is oracle:
$ su
Password: ******
#
Note: You must be logged in as root to run the iotop command
4. In the window where you are logged in as root, invoke the iotop command:

The output from the iotop command shows the five Disk I/O stress threads writing more
than 1,600 kilobytes per second each.
Leave the window where the terminal session is connected to the host01 system as "root"
open, as it will be re-used later on.
5. Select the terminal window connected to the system as "oracle," the same you used in
the previous practice to enter the various sar, iostat, etc. commands. Use the iostat
command to display the I/O statistics. The "-x 1" option specifies that you want to refresh
the output every second:

Chapter 3 - Page 12
[oracle@host01 ~]$ iostat -x 1

16.67 0.00 5.56 3.03 0.00 74.75
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm
%util
xvda 0.00 2788.00 0.00 160.00 0.00 11064.00 69.15 13.73 29.76 0.65
10.40
xvdd 0.00 0.00 0.00 10.00 0.00 80.00 8.00 0.01 0.50 0.10
0.10

19.19 0.00 3.03 19.19 0.00 58.59

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm
%util
xvda 0.00 4732.00 0.00 640.00 0.00 55496.00 86.71 90.93 156.09 1.10
70.10
xvdd 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00
6. The "-d" option for the iostat command causes the display of only I/O statistics:
/home/oracle>iostat -d 2
Linux 2.6.32-279.19.1.el6.x86_64 (host01.example.com) 09/20/2013 _x86_64_ (2 CPU)

xvda 0.44 0.48 5.39 2094974 23387992
xvdd 0.11 0.25 2.23 1084523 9699141

xvda 176.00 4.00 13184.00 8 26368
xvdd 0.00 0.00 0.00 0 0

xvda 301.00 4.00 24400.00 8 48800
xvdd 144.50 1328.00 0.00 2656 0

Chapter 3 - Page 13
7. The gkrellm application, which you left running in the background, visually show a heavy
disk activity in its "Disk" portion, towards the bottom:

8. Feel free to experiment stopping the StressDisk application and restarting it with a larger
number of threads (do not exceed twenty threads; otherwise the system could become
unresponsive and difficult to control, short of an abrupt shutdown. After the StressDisk()
method has been called with a higher number of threads, re-run the statistics using iotop
and iostat to assess if the larger number of threads made a difference.
9. IMPORTANT: The StressDisk application produces very large files in the /tmp directory.
You should periodically delete those files by using the command: "rm f /tmp/*.txt"
10. When you finish experimenting with disk I/O statistics, leave jconsole open but remember
to click the StopDiskStress button. jconsole will be used in the next practice. After you
stop the disk stress application, remember to erase the files stored in the /tmp directory
(execute "rm f /tmp/*.txt")

Chapter 3 - Page 14
Practice Solution
Practice Solution
Chapter 3 - Page 15
Practice 3-3: Monitoring Linux Virtual Memory and Network
Overview
In this practice, you use the StressTestOS application, set up during practice 3-1, to cause
network activity on the system and you use the various Linux provided commands to assess
and analyze the virtual memory and I/O statistics as reported by the system under stress.

own.

Dependencies
Practice 3-1 for setting up the environment for this practice.
Tasks
1. Make sure the socket server application is running on host02. You should have launched
the SocketServer application during practice 3-1, step 5. Select the VNC host02 window
and verify that the SocketServer application is still running in a terminal window:
2. Enter 10 as a parameter for the StressNetwork() method, which will instruct

StressTestOS to spawn ten threads, which will generate heavy network activity.

Chapter 3 - Page 16

Click the "StressNetwork" button.
3. Keep an eye on sysmon and gkrellm, which you left running in the background. Select
the terminal window connected to the system as "oracle," the same you used in the
previous practices to enter the various commands such as sar, iostat, and so on. Use
the command "netstat -i" repeatedly, every four or five seconds, to see the increasing
numbers of kilobytes sent and received:
/home/oracle>netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 2773 0 0 0 2413 0 0 0 BMRU
lo 16436 0 1257 0 0 0 1257 0 0 0 LRU
eth0 1500 0 59374 0 0 0 59022 0 0 0 BMRU
lo 16436 0 1726 0 0 0 1726 0 0 0 LRU
eth0 1500 0 93960 0 0 0 93604 0 0 0 BMRU
lo 16436 0 1846 0 0 0 1846 0 0 0 LRU
/home/oracle>

Chapter 3 - Page 17
4. The Gnome System Monitor application, which you left running in the background, should
show and increasing network activity, in the lower part of the window, under "Network
History:"

5. For a more detailed analysis of network activity, you can use the sar command, together
with the "-n DEV" option:
$ sar -n DEV
Linux 2.6.39-200.24.1.el6uek.x86_64 (host01.example.com) 09/22/2013 _x86_64_ (2 CPU)
10:50:01 PM IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
11:00:01 PM lo 35.57 35.57 10.73 10.73 0.00 0.00 0.00
11:00:01 PM eth0 4470.36 4472.29 3627.79 321.72 0.00 0.00 0.00
11:10:01 PM lo 30.67 30.67 8.58 8.58 0.00 0.00 0.00
11:10:01 PM eth0 18.29 17.87 11.22 4.81 0.00 0.00 0.00
11:20:01 PM lo 10.67 10.67 4.12 4.12 0.00 0.00 0.00
11:20:01 PM eth0 0.28 0.46 0.03 0.04 0.00 0.00 0.00
11:30:01 PM lo 10.65 10.65 4.12 4.12 0.00 0.00 0.00
11:30:01 PM eth0 0.31 0.46 0.04 0.04 0.00 0.00 0.00
11:40:01 PM lo 24.55 24.55 7.42 7.42 0.00 0.00 0.00
11:40:01 PM eth0 13.56 11.95 8.52 3.36 0.00 0.00 0.00
11:50:01 PM lo 21.62 21.62 6.22 6.22 0.00 0.00 0.00
11:50:01 PM eth0 1416.89 1416.84 1149.16 101.72 0.00 0.00 0.00
Average: lo 22.29 22.29 6.86 6.86 0.00 0.00 0.00
Average: eth0 987.50 987.53 800.18 72.01 0.00 0.00 0.00

Chapter 3 - Page 18
The table below explains the output produced by sar when invoked with the "-n DEV"
option:
sar column Meaning

IFACE Stands for the nic card interface name
rxpck/s Total packets received per second
txpck/s Transmitted packets per second
rxkB/s Total number of kilobytes received per second

txkB/s Total number of kilobytes transmitted per second
rxcmp/s Compressed packets received
txcmp/s Compressed packets transmitted
rxmcst/s Packets multicasted per second.
6. Another useful command available in Linux is vnstat, a console-based network traffic

monitor. It can be used without root privileges and when invoked with the live option (-l) it
displays the current transfer rate for the requested network interface in real time, until
interrupted. After interruption, vnstat prints a statistical summary:
$ vnstat -l -i eth0
Monitoring eth0... (press CTRL-C to stop)
rx: 102.29 Mbit/s 15623 p/s tx: 8.85 Mbit/s 15624 p/s
eth0 / traffic statistics
rx | tx
--------------------------------------+------------------
bytes 149.51 MiB | 12.99 MiB
--------------------------------------+------------------
max 105.58 Mbit/s | 9.17 Mbit/s
average 102.07 Mbit/s | 8.87 Mbit/s
min 92.84 Mbit/s | 8.12 Mbit/s
--------------------------------------+------------------
packets 188000 | 188001
--------------------------------------+------------------
max 16201 p/s | 16201 p/s
average 15666 p/s | 15666 p/s
min 14356 p/s | 14357 p/s
--------------------------------------+------------------
time 12 seconds

Chapter 3 - Page 19
7. For administrators/troubleshooters who have root privileges on the system under

observation, a useful tool is nethogs, a net top tool grouping bandwidth per process.
Select the window running a terminal session connected to the system as "root." (You
should have already such a window, created during practice 3-2, step 3.) Type "nethogs"
and enter. The output runs continuously, until interrupted via a CTRL-C. You can see the
network traffic by user and application running under that user:

After a while (approximately 30 seconds), type CTRL-C to stop nethogs.
8. Select the terminal window connected to the system as "oracle," the same you used in
the previous practices to enter the vnstat command. This time you want to assess the
memory usage (including virtual memory) by the system host01. The command to use is
vmstat, which can be forced to refresh its display at predefined intervals (in seconds) for a
predetermined number of times. For example, vmstat invoked with "2 10" will print its
output every two seconds, repeating the cycle ten times:
$ vmstat 2 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 0 1946724 51100 1193516 0 0 19 28 338 458 1 2 96 0 0
6 0 0 1943732 51100 1196412 0 0 0 0 19918 24532 2 18 79 0 0
1 0 0 1941484 51100 1199400 0 0 0 0 20038 24839 3 18 79 0 0
3 0 0 1938244 51108 1202312 0 0 0 20 20207 24604 2 17 81 0 0
1 0 0 1933212 51108 1205100 0 0 0 0 19057 23210 2 17 80 0 0
3 0 0 1930484 51116 1207972 0 0 0 18 19926 24219 2 18 79 0 0
2 0 0 1930416 51116 1210824 0 0 0 0 19126 23508 2 18 80 0 0
2 0 0 1924904 51116 1213616 0 0 0 0 19170 23510 2 18 80 0 0
2 0 0 1922068 51124 1216416 0 0 0 24 19419 23618 2 18 80 0 0
2 0 0 1917968 51124 1219212 0 0 0 0 19416 23711 2 17 80 0 0
9. Feel free to experiment with StressTestOS stop the current execution, increase the
number of threads (with moderation) and re-launch the network stressing application.
Re-run the various monitoring commands and check if the increased number of threads
determine a visible increase in system load.

Chapter 3 - Page 20
10. IMPORTANT: The StressNetwork application produces very large log files in the /tmp
directory. You should periodically delete those files using the command: "rm f
/tmp/*.log.*" to prevent the system disk from becoming full.
11. When you finish the practice, you can stop the StressTestOS application. You can either
request an application shutdown by clicking the "Request Shutdown" button in jconsole,
or you can type CTRL-C in the window that is running StressTestOS. Remember also to
stop the SocketServer application running on host02. Select the VNC host02 window and
type CTRL-C in the window which is running SocketServer.
12. Leave both VNC windows open and do not close any of the terminal sessions opened so
far inside either host01 or host02. They will be re-used in the next practice.

Chapter 3 - Page 21
Practice Solution
Practice Solution
Chapter 3 - Page 22
Practice 3-4: Analyzing and Tuning Linux Kernel Parameters
Overview
In this practice, you change several Linux kernel parameters which mainly affect the way the
system handles a large number of connections. You will modify four kernel parameters making
those changes permanent (the parameter changes will survive reboots.)

own.

Dependencies
Practices 3-1 through 3-3 for setting up the environment for this practice.
Tasks
1. Select the VNC host01 window. Select the terminal session connected to the system as
root (it is recognizable because the prompt ends with a hash sign (#) instead of a dollar
sign.) Print to screen the current value for the kernel parameters:
1) tcp_keepalive_time
2) tcp_keepalive_intvl
3) tcp_keepalive_probes
4) tcp_max_syn_backlog
The values are stored in the directory /proc/sys/net/ipv4 located in the virtual file
system which stores the runtime system information. To print their values, you use the
cat command:
# cat /proc/sys/net/ipv4/tcp_keepalive_time
7200
# cat /proc/sys/net/ipv4/tcp_keepalive_intvl
75
# cat /proc/sys/net/ipv4/tcp_keepalive_probes
9
# cat /proc/sys/net/ipv4/tcp_max_syn_backlog
2048
2. This means that the keepalive routines wait for two hours (7200 secs) before sending the
first keepalive probe, and then resend it every 75 seconds. If no ACK response is
received for nine consecutive times, the connection is marked as broken. In addition, the
half-open connection queue length is set to 2048, which is not enough. The defaults values
are set too high for keepalive and too low for max_sync_backlog, so you want to
change them to something more appropriate. You decide to configure the host so that
keepalive starts after ten minutes of channel inactivity, and then send probes in intervals
of one minute. Because of the high instability of your network trunk and the low value of the
interval, suppose you also want to increase the number of probes to 20. You also want to

Chapter 3 - Page 23
increase the half-open connection queue to 4096. Edit (using either gedit or vi) the file
/etc/sysctl.conf and append the following lines at the bottom of the file:
net.ipv4.tcp_keepalive_time=600
net.ipv4.tcp_keepalive_intvl=60
net.ipv4.tcp_keepalive_probes=20
net.ipv4.tcp_max_syn_backlog=4096
3. Save the /etc/sysctl.conf file and leave the editor. In order to force the system to
load the new kernel parameters, you must run the sysctl p command as logged in as
"root." In the same window where you edited the /etc/sysctl.conf file, run the
sysctl command. The output from sysctl p is a list of all parameters loaded:

# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
Many lines omitted for brevity
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 20
net.ipv4.tcp_max_syn_backlog = 4096
[root@host01 ~]#
The kernel parameters stored in /etc/sysctl.conf are reapplied every time the system
boots.
4. You must change the parameters also on the host02 node. Open a new terminal window
on host02.
5. In the new terminal window, enter the su command to log in as root:
$ su
Password: ******
#
6. Edit using either gedit or vi the /etc/sysctl.conf file and append the parameter
values at the bottom of it:
net.ipv4.tcp_keepalive_time=600
net.ipv4.tcp_keepalive_intvl=60
net.ipv4.tcp_keepalive_probes=20
net.ipv4.tcp_max_syn_backlog=4096

Chapter 3 - Page 24
7. Save the file and leave the editor. Run the sysctl command. The output from
sysctl p is a list of all parameters loaded:
[root@host01 ~]# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1

net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 20
net.ipv4.tcp_max_syn_backlog = 4096
[root@host01 ~]#
8. Close all terminal windows you opened on the host02 host. Select the VNC host01 window
and close all open terminal windows.

The primary focus of these practices is to get you acquainted with the system troubleshooting
activity, which inevitably will be incurred while supporting WebLogic Server installations. You
must be familiar with several Linux/UNIX tools and remember how and when to use them
(including what the main command qualifiers and options are) to be able to capture performance
metrics, analyze the results, and determine the best course of action when issues occur.
While the focus of this workshop is the WebLogic Server, a decision has been made to stress
test the various subsystems (CPU, memory, I/O, and so on) of the Linux host not from a running
WebLogic Server domain. The WebLogic Server has several inherent self-defense mechanisms
to try and prevent overloading and memory saturation (as you will appreciate in practice 5) and
try to induce performance stress through an application running inside the WebLogic Server in
the context of a lab experiment, while not impossible, is difficult and requires modifying many
WebLogic parameters to force WebLogic out of its default, conservative (in terms of resources)
behavior.

Chapter 3 - Page 25
Practice Solution
Practice Solution
Chapter 3 - Page 26

Chapter 4 - Page 1
Chapter 4

Tuning HotSpot JVM

Practices Overview
In the practices for this lesson, you monitor and tune parameters associated with the Java
HotSpot JVM. First, you tune Java garbage collection, load test an application, and monitor
garbage collection and application performance to perceive differences with different collectors.
Next, you use Java command-line tools to monitor and manage JVM processes. Then, you use
various graphical tools to monitor and manage JVM processes.

Chapter 4 - Page 2
Practice 4-1: Tuning JVM Garbage Collection
Overview
In this practice, you experiment with configuring different garbage collectors for an application,
loading the application, and monitoring the performance results for comparison.
The following image depicts the architecture of the domain used for this practice.

your own.
Dependencies
Reading GC Log Output Hints

Use these hints to help you read and analyze GC log data. Extrapolate these hints to all forms
of collectors used in these practices.

Chapter 4 - Page 3
Default Minor GC on WebLogic

This screenshot shows a minor garbage collection log, which includes the following:
A timestamp of when collection occurred.
A minor collection is indicated by the term GC in the output.
The collection occurred in the young generation as indicated by the term
PSYoungGen.
The collection consisted of:
A parallel collection using multiple threads, as indicated by the P in PSYoungGen.
An initial 20052KB of combined live objects before collection
An ending 2123KB of combined live objects after collection
The total size of the young heap is 23552KB.
The total size of the young and old heap combined before collection was 70676KB.
The total size of the young and old heap combined after collection was 53579KB.
The total committed size of the young and old heap combined is 84992KB.
The collection took 0.0140650 seconds, or 14 milliseconds.
The collection used approximately 0.02 seconds of CPU time.

Chapter 4 - Page 4
Default Full GC on WebLogic

This screenshot shows a full garbage collection log, which includes the following:
A timestamp of when collection occurred.
A minor collection is indicated by the term Full GC in the output.
The collection occurred in the young generation as indicated by the term
PSYoungGen.
The collection occurred in the old generation as indicated by the term ParOldGen.
The collection occurred in the permanent generation as indicated by the term
PSPermGen.
The collection consisted of:
A parallel collection of the young generation using multiple threads, as indicated by
the P in PSYoungGen.
A parallel collection of the old generation using multiple threads, as indicated by the
Par in ParOldGen.
A parallel collection of the permanent generation using multiple threads, as
indicated by the P in PSPermGen.
The collection took 0.5523550 seconds, or 552 milliseconds.
The collection used approximately 1.37 seconds of CPU time.
Tasks
$ ./setup.sh

Chapter 4 - Page 5

3. Explore the existing default memory and garbage collection settings for WebLogic managed
servers.
a. Navigate to the domains bin folder on host01 and open the domains environment
script.
$ cd /u01/domains/tune/wlsadmin/bin
$ gedit setDomainEnv.sh &
b. Explore the file and answer the following questions:

Question Answer (include why)
What is the default
minimum heap size?
What is the default
maximum heap size?
What is the default
permanent generation size?
What is the default
maximum permanent
generation size?
What are the default
garbage collector settings?
Include information for all
generations.
Do the servers start up in
client or server JVM mode?
What is the This is a trick question because if you are familiar with
setUserOverrides.sh WebLogic at all, then you have never heard of this file
script? before WebLogic 12.1.2. This is a new file you can create
in your DOMAIN_HOME folder that is called by the
setDomainEnv.sh script to allow you to override and
add settings to start your domain instead of modifying the
other WebLogic scripts directly. So keep this in mind
when you are changing settings for this practice!
Hint: If you are having trouble figuring out the setting in the script file, you can start the
domain and review each managed servers console output for its start up command-
line arguments and memory settings, and work your way backwards in the script file to
see where the values are set. Some of the questions you may not be able to answer
by only looking at the script. You will need to change some start up settings to get the
answers. See the setUserOverrides.sh script for possible settings.
4. Configure GC monitoring.
a. Shut down server1 so you can modify some settings.
b. Create the setUserOverrides.sh file in your DOMAIN_HOME/bin folder. Be sure to
place any environment variable settings in this file. There is a starter file in the practice

Chapter 4 - Page 6
resources folder if you prefer to not create the file yourself. Remember that you must
create this file for each host if you want to monitor both managed servers because the
domain folder is not a shared file system. However, for this practice, using the file for
server2 is not recommended because it will take too long. There is a lot of work to do
for this course and the goals are accomplished by focusing on one server. You will
modify this same file, directly in the DOMAIN_HOME/bin folder for the remaining
practices in this lesson.
c. Experiment by setting the different settings in the table below and starting the servers
with the new settings. Remember to start the servers using the startAdmin.sh,
startServer1.sh, and startServer2.sh scripts. Try answering the questions in
step 3b again.

Setting Setting
-XX:+PrintCommandLineFlags -XX:+PrintGC
-XX:+PrintGCDetails -XX:+PrintGCTimeStamps
verbose:gc
d. If you do not have Grinder running, then start the Console and Agent now. For
convenience, a script has been created to make it easy to start Grinder by using a
single command. Execute the following steps to run Grinder:
Open a new terminal window.
Execute the following commands to run Grinder:
$ runGrinder.sh
The script opens two terminal windows and automatically starts the Console and the
Agent processes for you. Now all you must do is use the Console to control your
tests.
IMPORTANT! This script causes Grinder to reuse your recorded load test and
grinder.properties settings from practice02-03. You can continue to
use these files when using Grinder for other practices, unless a practice
specifies different instructions. If you do not have these files available, then
run Grinder from within the practice02-03/solution folder and use the
files in that location instead.
e. Keep the garbage collection monitoring settings in place and run your Grinder load
again. Make sure that both managed servers are running for your tests. Record the
performance numbers for this test to ensure that garbage collection monitoring has not
impacted your baseline numbers. Keep these performance metrics available for this
practice to compare with other load test metrics.
f. Find the garbage collection output in the server console window and examine the
garbage collection metrics that were recorded during your load test. Use those metrics
to answer the following questions:
Pause Time: Does the
collector introduce pauses
that stop the world?
Memory Footprint: What is
the total memory allocated

Chapter 4 - Page 7

versus the total memory
available to the application?
Parallel: Does the garbage
collector make use of
multiple CPUs?
CPU Usage: What is the
percentage of CPU spent in
garbage collection?
Concurrency: Is there a

contention for a CPU
between application threads
and the garbage collector?
If so, can this be overcome
by use of a system with
more CPUs?
g. What you should have noticed is that WebLogic and the application run well using the
default memory settings. Garbage collection occurs very infrequently, and should only
be minor collections. This is denoted by the log entries that start with GC. Full
garbage collections start with Full GC and are much slower.
h. Because the application is running well with the default memory settings, you must
lower the heap size of the server to force a performance issue with garbage collection.
Try setting the heap size and maximum heap size to 90 MB and run Grinder again to
see how the application performs. Remember that you must also set the maximum
permanent generation size when making custom memory settings in your
setUserOverrides.sh script. Answer the following questions:
Does the application
perform faster or slower
than before? If so, why do
you think that is?
Do you notice anything
different about the
applications garbage
collection process? Is it
using more CPU? Is it
affecting overall application
performance?
How far apart are garbage
collections?
Minor GCs?
Full GCs?
Note: You may notice that when you press Ctrl + C to restart a server, the JVM
displays more garbage collection information, much like the following example when
the serial collector was used:
Heap

Chapter 4 - Page 8
def new generation total 15360K, used 12654K

[0x00000000ece00000, 0x00000000edea0000, 0x00000000edea0000)
eden space 13696K, 92% used [0x00000000ece00000,
0x00000000eda5b880, 0x00000000edb60000)
from space 1664K, 0% used [0x00000000edd00000,
0x00000000edd00000, 0x00000000edea0000)
to space 1664K, 0% used [0x00000000edb60000,
0x00000000edb60000, 0x00000000edd00000)
tenured generation total 34176K, used 34176K
[0x00000000edea0000, 0x00000000f0000000, 0x00000000f0000000)
the space 34176K, 100% used [0x00000000edea0000,

0x00000000f0000000, 0x00000000f0000000, 0x00000000f0000000)
compacting perm gen total 101120K, used 101045K
[0x00000000f0000000, 0x00000000f62c0000, 0x0000000100000000)
the space 101120K, 99% used [0x00000000f0000000,
0x00000000f62ad4d8, 0x00000000f62ad600, 0x00000000f62c0000)
No shared spaces configured.
i. Set the servers heap size to 50 MB and run the test again. Answer the questions again
for this heap size.
j. Change the garbage collection algorithm for server1 and server2 to two of the settings
in the following table (one at a time), restart the managed servers for each setting, run
your Grinder load tests again for each setting, record the performance results, and
compare them to the metrics from the last baseline test. Try to answer all of the
questions in 4f for each run. Review the GC Algorithms slide in this lessons
presentation for details about each setting. There is not enough class time to try to run
them all.
Setting Setting
-XX:+UseSerialGC -XX:+UseParNewGC
(use with the concurrent collector)
-XX:+UseParallelOldGC -XX:+UseConMarkSweepGC
(use with the ParNew collector)
-XX:+UseG1GC
5. Shut down the environment.
a. When you are done with this practice, execute the following command to shut the
environment down.
$ cleanup.sh
b. If any dialog windows appear asking if you want to close a terminal window, click OK to
close the window.
c. You should notice that the AdminServer, server1, and server2 terminal windows and
the servers that were running in them are all shut down now. The Grinder Console and
Agent applications should still be running.

You should notice that when you change to a heap size of 50 MB that the garbage collector
struggles to keep enough free memory available for the JVM to run. You should also notice
more frequent full garbage collections that heavily use the CPU. In some cases, all the tests will

Chapter 4 - Page 9
complete, and in other cases you may have to stop the test because the JVM is failing to keep
up with the load.
I was impressed with the G1 collector when run in a 50 MB heap. Not only was it able to finish
all the tests, but it also managed some decent performance numbers even in such a
constrained environment. I never even set any threshold parameters for it to follow. Did you
notice similar results?
My baseline test for a default 256-512 MB heap and parallel throughput collector settings were:
11.2 ms average response time, 72.5 average / 281 peak TPS
My 50 MB G1 collector test run produced the following numbers, which were very on par with
much less memory:
19.0 ms average response time, 75 average / 265 peak TPS

The primary focus of this practice is to get you used to setting and analyzing JVM garbage
collection performance of a WebLogic Server by using JVM command-line options. The goal in
tuning the garbage collector and Java heap spaces is to minimize the occurrence of full GC
events (and minimize the frequency of GC events in general) while maintaining acceptable
pause times. For finer details and explanations of each aspect of garbage collection output, see
the D82344GC10 Java Performance Tuning with Mission Control and Flight Recorder course.

Chapter 4 - Page 10
Practice Solution
There is no solution for this practice. No practices depend on this practice. However, the
setUserOverrides.sh script is introduced in this lesson. This script is used in other
practices, but is supplied by only this practice for this lesson. Other lessons provide their own
script file by default.

Chapter 4 - Page 11
Practice 4-2: Using Command-Line Tools
Overview
In this practice, you use Java command-line tools to obtain and analyze performance-related
metrics of running WebLogic Server JVM processes. These tools may be used freely throughout
the rest of the practices in this course if they seem appropriate and suitable to you. The
following image depicts the architecture of the domain used for this practice.

your own.
Dependencies
The setUserOverrides.sh script in
/practices/tune/practice04-01/resources. You will need to copy this file
and use it again any time you run the setup.sh script for a practice.
IMPORTANT! You can now leave Grinder running and reuse it for practices that use the
Auction application, or run it separately for each practice when needed the same way you did
for practice 4-1.

Chapter 4 - Page 12
Tasks
$ ./setup.sh
Restores the domain to its original state. This includes removing the
setUserOverrides.sh script. If you need this script again in the future, copy it
from practice 4-1 to the domains bin folder again, open it in the gedit editor and

leave it open to save again whenever it is needed.
3. Use jps to analyze JVM processes.
a. Read this note on jps: The main purpose of jps is to find the process ID of a running
JVM. It is often referred to as the lvmid, or local virtual machine identifier, because
Java is a platform-independent language. Essentially, this is Javas way of not tying
itself to an operating system construct, such as a process ID (PID), in case there is
another mechanism for identifying JVMs outside of PIDs. However, because operating
systems of today all use PIDs, the lvmid correlates to the PID of the running process
that is executing the JVM.
b. Review the following jps usage description.
jps [-q] [-mlvVJ] [<hostname>[:<port>]]
Option Description
-q Suppresses all output except for the lvmids of JVM processes. This
option cannot be used with the other options.
-m Displays the arguments passed to the main method of the running JVM
-l Displays the full package name of the main class or full path of the
applications JAR file
-v Displays the arguments passed to the JVM
-V Displays the arguments passed to the JVM using a flags file
-J Used to pass options to the underlying JVM run with jps
hostname Specifies the hostname of the remote host where jstatd is running so
you can use jps to retrieve lvmids of JVMs remotely on that host
port Specifies the port of the remote host where jstatd is running. This port
should match the port used to run jstatd.
Example usage with WebLogic on the same machine:

Chapter 4 - Page 13
$ jps -lv | grep server1

Example usage with WebLogic on a remote machine (after jstatd is running on the
remote machine):
$ jps -lv host02:4001 | grep server2
c. Review the jstatd usage description below.
jstatd [-nr][-p port][-n rminame][-Joption]
Option Description
-nr Tells jstatd to not create an RMI registry if one is not already
running

-p port The default RMI registry port for jstatd is 1099. Use this to specify
a different port. Your remote clients must also specify this port.
-n rminame The default name for jstatd in the RMI registry is
JStatRemoteHost. Use this to specify a different name for this
particular jstatd instance.
-Joption Used to pass options to the underlying JVM run with jstatd
Policy file The jstatd tool does not authenticate remote clients, but does
enforce security to dissuade attackers from accessing JVM
processes. You can use jstatds use of RMISecurityPolicy to
run jstatd for this practice. You do this by creating a policy file,
such as this example:
jstatd.all.policy file:
grant codebase "file:${java.home}/../lib/tools.jar" {
permission java.security.AllPermission;
};
Example usage (You will need to create the jstatd.all.policy file as described in
the table above):
$ jstatd -J-Djava.security.policy=jstatd.all.policy -p 4001
d. Reflect on why you would use jps and jstatd. Could you obtain the same
information using operating system commands? If so, what would the differences be?
Example OS remote method using SSH:
$ ssh host02 "ps -ef | grep -v grep | grep server2"
e. Experiment with these commands to perform the following tasks:
Find the PID of each JVM running on host01.
Find the PID of each JVM running on host02, but do it from host01.
Find the PID of server1 by using jps and grep.
Find the PID of server2 by using jps, jstatd, and grep.
f. Experiment with these commands on your own for 5-10 minutes.
4. Use jcmd to manage JVM processes.
a. Read this note on jcmd: The jcmd tool is the JRockit jrcmd tool ported for use with
the HotSpot JVM. It is used to send diagnostic commands to the JVM. This is very
useful for obtaining runtime information about your WebLogic Server instances.
b. Review the following jcmd usage description.

Chapter 4 - Page 14
jcmd [options]
jcmd [pid | main-class] PerfCounter.print
jcmd [pid | main-class] command [arguments]
jcmd [pid | main-class] -f file
Option Description
-l Displays the list of running Java processes including their PID, main
class, command-line arguments
-h -help Displays a help message

pid Specifies a particular PID (that represents a running JVM) to which
jcmd will send commands
main-class Specifies a particular main Java class (running in a JVM) to which
jcmd will send commands. If multiple Java processes use the same
main class, then commands are sent to all matching processes.
command The jcmd command to send to the JVM. Arguments included depend
[arguments] on the particular command being used.
-f file Performs commands read from the specified file
Here are some examples of using jcmd (for reference, 19719 refers to a WebLogic
Server instances PID):
Example Description
jcmd Displays the PID and class names for
running JVM processes
jcmd 19719 help Displays the available jcmd commands
for the process with the specified PID
jcmd 19719 help VM.flags Displays detailed jcmd help for an
available command
jcmd 19719 Thread.print Displays a thread dump of a JVM
jcmd 19719 GC.class_histogram Displays a list of classes ordered by
those that are using the most memory
jcmd 19719 VM.command_line Displays the JVM command line. Use
this to display the command line used to
start WebLogic Server.
jcmd 19719 VM.system_properties Displays the system properties that are
set for a JVM. Use this to display
system properties set for WebLogic
Server.
jcmd 19719 VM.version Displays the version of the JVM running
in the specified process. Use this to
know what JVM version is running your
WebLogic Server instances.

Chapter 4 - Page 15
Example Description
jcmd 19719 PerfCounter.print Displays a bunch of performance
counters for the JVM. Use this to get a
dump of numbers you can use to
analyze WebLogic / JVM performance.
c. Experiment with jcmd to perform the following tasks:

Find the PID of each JVM running on host01.
Find the PID of server1 by using jcmd and grep.

Display the available help commands for the JVM running server1.
Display detailed help for one of the available commands for the JVM running
server1.
Print a thread dump of the JVM running server1.
Print a class histogram for the JVM running server1. Which classes seem to be
using the most memory? Why?
Print the command-line arguments for the JVM running server1. Why is this useful?
What does it help you avoid?
Print the system properties for the JVM running server1. What parameters do you
find useful in the display? Why?
Print the JVM version of the JVM running server1. Why is it useful to know what
version of the JVM is running WebLogic?
Execute the PerfCounter command to view performance data related to the JVM
running server1. What type of information does the tool provide?
d. Experiment with these commands on your own for 5-10 minutes.
5. Use jinfo to analyze JVM processes.
a. Read this note on jinfo: The jinfo tool is used to display configuration information
for a Java process, core file, or a remote debug server. This is very useful for obtaining
runtime information about your WebLogic Server instances.
b. Review the following jinfo usage description.
jinfo [option] pid
jinfo [option] executable core
jinfo [option] [server-id@]remote-hostname-or-IP
Option Description
-flag name Displays the name and value of the specified
command-line argument
-flag [+|-]name Enables or disables the specified boolean
command-line argument
-flag name=value Sets the specified command-line argument to the
specified value
-flags Displays command-line arguments passed to the
JVM

Chapter 4 - Page 16
Option Description
-sysprops Displays Java system properties for the JVM
-h -help Displays a help message
Here are some examples of using jinfo (for reference, 19719 refers to a WebLogic
Example Description
jinfo -help Displays the jinfo usage statement

jinfo 19719 Displays the command-line arguments
and system properties that are set for a
JVM. Use this to display system
properties set for WebLogic Server.
jinfo -flags 19719 Displays the JVM command line. Use
this to display the command line used to
start WebLogic Server.
jinfo -sysprops 19719 Displays the system properties that are
set for a JVM. Use this to display
system properties set for WebLogic
Server.
c. Experiment with jinfo to perform the following tasks:

Display the jinfo help message.
Print the command-line arguments for the JVM running server1. Why is this useful?
What does it help you avoid?
Print the system properties for the JVM running server1. What parameters do you
find useful in the display? Why?
d. Experiment with these commands on your own for 5 minutes.
6. Use jstat to view JVM performance statistics.
a. Read this note on jstat: The jstat tool is used to display performance statistics for
an instrumented HotSpot JVM process. This is very useful for obtaining runtime
information about your WebLogic Server instances.
b. Review the following jstat usage description.
jstat -help | -options
jstat option [-t] [-h lines] vmid [interval [count]]
The first example uses a single general command-line option, either -help or -
options. When a general option is used, no other options are valid. General options
are described in the following table:
Option Description
-help
-options Displays a list of options available with the jstat

Chapter 4 - Page 17
Option Description
command. The description of these options is listed
in the next table.
Output options and other arguments are described in the following table:
Option Description
-class Displays class loader statistics
-compiler Displays JIT compiler statistics

-gc Displays garbage collector statistics
-gccapacity Displays size statistics for the generations of the
heap
-gccause Displays garbage collector statistics, plus the cause
of the last and current GC events
-gcnew Displays young generation statistics
-gcnewcapacity Displays size statistics for the young generation
-gcold Displays old and permanent generation statistics
-gcoldcapacity Displays size statistics for the old generation
-gcpermcapacity Displays size statistics for the permanent generation
-gcutil Displays garbage collection statistics
-printcompilation Displays compilation method statistics
-t Includes a timestamp column in the output
-h n Displays a column header every n number of rows
-Joption Passes Java options to the java application
launcher
vmid Specifies the vmid of the JVM process to use. The
format for vmid is:
[protocol:][//]
lvmid[@hostname][:port][/servername]
See the documentation for the description of each
aspect of the URL.
interval Time interval in between samples
count The number of samples to display
Several of jstats command options produce displays with headings that can be
cryptic at first glance. Execute the following command to view the documentation for
jstat to see descriptions of the columns for each option:
$ firefox resources/jstat.html &

Chapter 4 - Page 18
Find the examples of using jstat in the documentation and review them. The jstat
tool is very useful for analyzing Java heap and garbage collection statistics. The table
below shows more examples of using jstat:
Example Description
jstat -gcutil -h 10 19719 2000 Displays a list of garbage collection
statistics, including the size and
performance of each space in the heap.
Prints a header every 10 lines, and prints
a new row every 2000 ms.
jstat -gcutil -h 10 Displays the same statistics as the

7429@host02:4001 2000 previous command above, but is reporting
statistics for a remote JVM running on
host02. Note: For reference, 7429 refers
to the PID for server2 obtained by using
jps and jstatd.
jstat -gcnew 19719 2000 10 Displays statistics for the young
generation, and prints a new row every
2000 ms for a count of 10.
jstat -gcnewcapacity 19719 2000 Displays capacity-related statistics for the
10 young generation, and prints a new row
every 2000 ms for a count of 10.
jstat -gccause -t -h 25 19719 Displays garbage collection statistics,
2000 including the reason for GC if any. Prints
a header every 25 rows, and time in
between rows is 2000 ms. Includes a
timestamp in the output.
c. Experiment with jstat to perform the following tasks. You can optionally run Grinder
to cause the numbers to have more meaning:
Display the jstat help message.
Display the list of jstat options.
Display garbage collection and heap statistics for the JVM running server1.
Display garbage collection and heap statistics for the JVM running server2 from the
host01 machine. Hint: Requires using jstatd, which may still be running from an
earlier step in this practice.
Include timestamp information in your output.
d. Experiment with these commands on your own for 5 minutes.
7. Use jstack to view and analyze JVM thread stack traces.
a. Read this note on jstack: The jstack tool is used to display stack traces of Java
threads for a Java process, core file, or a remote debug server. This is very useful for
obtaining runtime information about your WebLogic Server instances. Thread dumps
are displayed in the console window where the command is executed, rather than the
servers logs or console window. You should redirect the output of jstack to different
files for comparisons. This is because a thread dump is only a single snap shot in time
of the state of a server. Thread dumps have to be analyzed over a series of thread

Chapter 4 - Page 19
dumps to track any trends in how threads are executing or having issues. From a
performance perspective, jstack provides only information pertaining to threads that
are stuck, long-running, locked, or busy. They do not provide performance numbers or
metrics to measure performance. It is up to you to analyze what the threads are doing
and to determine if they are stuck. And from an administrative perspective, you would
not be expected to understand the methods listed in stack traces because they are
written by developers. You would be focused mainly on trends of thread deadlocks or
overtly active threads. Our application is functioning well at this point, so you should
not detect any locked threads or performance issues at this point. When you are
working with stuck thread behavior in this course, you will have a chance to use
jstack again.

b. Review the following jstack usage description.
jstack [option] pid
jstack [option] executable core
jstack [option] [server-id@]remote-hostname-or-IP
Option Description
-F Force a thread dump when jstack [-l] pid
does not respond.
-l Display a long listing that includes information about
locks, such as a list of owned
java.util.concurrent ownable subscribers.
-m Display a mixed mode listing of Java and native
C/C++ frames in the thread dump.
-h -help Display a help message.
Here are some examples of using jstack (for reference, 19719 refers to a WebLogic
Example Description
jstack -l 19719 Displays a thread dump of Java threads
running in the JVM running WebLogic
server1. Prints a detailed long listing.
jstack -m 19719 Displays a thread dump of Java threads
running in the JVM running WebLogic
server1. This prints native methods as
well as Java methods.
c. Experiment with these commands on your own for 10 minutes.


What you should have noticed with this practice is that there are several different Java
command-line commands available to you. Some of these commands provide overlapping
features, such as getting the process information for a running JVM, or the arguments passed to

Chapter 4 - Page 20
the JVM main class. You may naturally have a better feel for one or more tools than other tools
initially. The most commonly used tool is gcutil, because it supplies good details of the Java
heap space.
During this practice, you were asked what the difference was between using jps to get JVM
information remotely versus using operating system commands. The first thing that you should
have noticed is that you have to use SSH to execute commands on the other host. SSH is much
harder to set up than jstatd and the command to get information is easier. Of course if you
have access to the remote server you could just run the local jps command instead. The jps
command only requires that Java is installed on both machines, and that the jstatd process is
running.
The primary focus of this practice is to get you used to obtaining and analyzing JVM

performance-related metrics of a running WebLogic Server using JVM command-line options.

Chapter 4 - Page 21
Practice Solution
There is no solution for this practice. Other practices in this lesson depend on the experience
gained in this practice. Additionally, the experience gained in this practice can be applied in
subsequent lessons.

Chapter 4 - Page 22
Practice 4-3: Using Java VisualVM
Overview
In this practice, you use Java VisualVM to monitor your running WebLogic Server instances
graphically. The following image depicts the architecture of the domain used for this practice.

your own.
Dependencies
Tasks
1. Connect to the host01 and host02 machines using VNC.
2. Set up the practice.

Chapter 4 - Page 23
a. Perform the following commands on host01 to set up the environment for this practice.
If your servers are still running from the previous practice, then you should skip running
the setup.sh script.
$ ./setup.sh

3. Start Java VisualVM.
a. The script to start VisualVM is located in /u01/app/jdk/bin. This directory is
already in your path, so you can start VisualVM from any location. Execute the script to
start the tool:
$ jvisualvm
b. Verify that the GUI appears.

Chapter 4 - Page 24
4. Connect to local WebLogic managed server with VisualVM.

a. In the left-hand panel, double-click one of the WebLogic JVMs in the list. The managed
server is likely the last entry because it started later. This opens the JVM tab.
b. In the Overview tab, verify it is server1 by reviewing the JVM Arguments subtab and
checking that -Dweblogic.Name=server1.
c. Review the descriptions of the JVM tabs in the following table:
Tab Description
Overview Provides an application snapshot where you can see the PID, host, main
class, main class arguments, JVM information, Java version, Java Home,
and JVM flags. There are additional subtabs that let you view JVM

arguments and system properties.
Monitor Provides a graphical multiple chart view of the JVM process, including
CPU and GC utilization, memory use, class count, and thread count. This
page also displays the PID, uptime, and some buttons for performing a
GC and a heap dump. Hovering the mouse over a chart causes a popup
window to appear with more detailed information.
Threads Provides a list of all JVM threads and allows you to view threads in
timeline, table, or detailed mode. Each thread is displayed and is color
coordinated to distinguish between running, sleeping, waiting, and
monitoring threads. You can select some threads and right-click to get
more thread details. You can choose to see all threads, some threads, or
just one thread.
Sampler Provides an interface for analyzing runtime behavior of your Java
applications. You set CPU and memory settings here and get calculated
metrics of the following, which is analyzed from collected stack trace data
over several intervals:
CPU: Displays which Java methods or threads are using the
most CPU.
Memory: Displays which Java methods or threads are using the
most memory.
Profiler Provides an interface for analyzing runtime behavior of your Java
applications. You set CPU and memory settings here and get real time
metrics of the following, which is captured in real time as methods are
executed:
CPU: Displays which Java methods or threads are using the
most CPU.
Memory: Displays which Java methods or threads are using the
most memory.
Note: The Sampler and Profiler tabs seem similar but work very differently. Because
Java VisualVM is a generic tool for all Java processes, it is not necessarily the right
tool for every job, such as profiling huge application code bases, such as WebLogic.
The Sampler tab is faster, gathers information from thread dumps at regular time
intervals. The Profiler tab instruments the byte code of the JVM, and in the case of an
application server like WebLogic, this instrumentation can take a long time, and
oftentimes can simply just not work when profiling is complete. It is not recommended
to use the profiling tool for WebLogic. The Sampler is faster but less accurate. In any

Chapter 4 - Page 25
case, understanding which methods or threads are using the most memory or CPU is
also something that interests a developer more because they are the ones who have
to go back to the code and try to improve the results.
5. Use VisualVM to answer the following questions. You may run Grinder to load the
application for some of the questions.
a. What is the operating system architecture?
b. Is the server running in production mode?
c. Find the graphic charts for CPU, heap, and thread monitoring. What are the
characteristics of these metrics? You should see a gradual increase in memory,
followed by occasional declines as garbage collections occur. If the heap appears to be
too jagged, meaning that there are frequent memory spikes and drops, then memory

may not be sized properly or there could be some other problem.
The following image shows a server that may be doing too much garbage collection.
Then again, it could be ok. It depends on how much work the server is doing. Is the
CPU heavily used for garbage collections? In reality, this is normal, at rest garbage
collection. It only looks jagged because it encompasses a time period of about an hour.
Here is an example of another chart showing the heap and garbage collection in a
shorter time period:
6. Use VisualVM to connect to server2 on host01.

a. Configure VisualVM to connect to jstatd, running on host02, so that it can remotely
monitor the server2 managed WebLogic Server JVM.
b. Open server2 so it has its own monitoring tab.
c. Verify that it is the JVM for server2 by checking its server name property.
d. Notice that server1 and server2 each have their own tabs.
e. You can drag the server2 (or any tab) off the main VisualVM window so it creates its
own window. This makes it so you can view both servers side-by-side.

Chapter 4 - Page 26
f. Alternatively, you can dock the tab in the main window so both tabs are visible at the
same time for side-by-side viewing as well.
g. Run Grinder and watch the monitoring tabs for each server to see how each behaves.
Alternatively, you can stop remotely monitoring server2 and start another instance of
VisualVM locally on host02 so you can see CPU metrics.
While the application is loaded by Grinder, you can see the CPU reacting to the
workload. The orange line represents the CPU utilization, while the blue line represents
the CPU utilized to perform garbage collection.

Here is an example of heavier garbage collection activity while the application is under
load. Notice how the heap size is growing toward the maximum 512 MB setting.
7. Use VisualVM to monitor server1 threads.

a. Use the Threads tab to view the active threads in the server1 JVM. The legend on the
lower right-hand side shows the color scheme used to denote which threads are
running, sleeping, waiting, or monitoring.

Chapter 4 - Page 27

b. You can right-click a thread in the list and select Thread Details to view more
information about that particular thread. The following image shows a thread that spent
91.1% of its lifetime in a Wait state, and 8.8% in a Running state.
8. Experiment with the Sampler tab to view some metrics.

9. Avoid using the Profiler tab.
10. Use VisualVM under load.
a. Set the heap size to 50 MB again in the setUserOverrides.sh file and restart
server1.
b. Run Grinder to load the application.
c. Use VisualVM to monitor CPU, heap, and threads.

Chapter 4 - Page 28
d. What do you notice about CPU activity in this scenario?

e. How does GC CPU utilization compare to overall CPU utilization?
f. How often does garbage collection appear to be happening?
g. What are the basic memory and garbage collection activities in this scenario?
h. Are there more or fewer threads running now?
11. Experiment with VisualVM on your own for 5 minutes.

You saw how the server functioned when you set the heap size to a very small size by using
Java command-line tools and options. This practice lets you see the same thing graphically
using VisualVM. The primary focus of this practice is to get you used to using VisualVM
graphical tools to capture performance metrics, analyzing the results, and determining the
current capabilities of your system.

Chapter 4 - Page 29
Practice Solution
subsequent lessons.

Chapter 4 - Page 30
Practice 4-4: Using VisualGC, jconsole, and GCHisto
Overview
This practice continues your journey into exploring some of the graphical tools available to you
for gaining visibility into your JVM processes that are running WebLogic Server. You will install
VisualGC as a plug-in into VisualVM, and use it to graphically visualize how garbage collection
is really taking place in the application. Next, you run jconsole and connect it to your running
WebLogic server instances to view some of the same metrics, but also to browse the live
MBean tree of your WebLogic server. Then you install and run GCHisto to view garbage
collection history of your running WebLogic Server. The following image depicts the architecture
of the domain and tools used for this practice.

your own.
Dependencies
Practice 4-3 for using VisualVM.

Chapter 4 - Page 31

Tasks

$ ./setup.sh
3. Wait for all servers to log that they are in the RUNNING state before continuing.
4. Use VisualGC to monitor garbage collection.
a. First, you have to install the VisualGC plug-in for VisualVM. The plug-in file is located in
the /install/java folder, and the file is named
com-sun-tools-visualvm-modules-visualgc.nbm. Within Visual VM, select
Tools > Plugins.
b. The Plugins dialog window appears. Click the Downloaded tab and click Add Plugins.

Chapter 4 - Page 32
c. Browse to the file location and click OK to install the plug-in.

d. Click Install and complete the installation wizard to install the VisualGC plug-in.
e. Click Close to return to VisualVM.
f. Close any tabs you may have open for monitoring WebLogic JVMs. They must be
reopened in order for the VisualGC tab to appear.
g. After you reconnect to the server1 JVM, click the VisualGC tab that now appears on
the end.

Chapter 4 - Page 33
h. Now you should be able to visually see your Java heaps and how they are being used.
You can change the refresh rate, but it refreshes automatically at an acceptable rate.

i. Watch the display until you see a garbage collection take place. It usually take a
minute or two. You will know when it takes place because Eden space gets cleared to
0 bytes, and you will see objects move from one Survivor space to the other. Note the
numbers of the heap sizes for every space. They provide the minimum, maximum, and
used space sizes. Note what the maximum sizes of each heap space are on paper.
They should be similar to the sizes shown in the screenshot above.
j. Run Grinder again with server1s memory settings set to 90MB again so you can see
garbage collection activity using VisualGC. You will have to reconnect to the JVM
again to monitor it. Maximizing the VisualVM window gives the best viewing results.
k. Now note the heap sizes for every space again and compare them to your previous
sizes. How are they different? How does this affect the performance of your
application?
l. Note the number of collections reported for each space. How many collections were
there for each? How much time did it take collectively to perform those collections?
m. Did your application show any errors? What can you ascertain from this?
n. You should notice that although the application is running fine that the garbage
collector is working very hard to keep the application running. Just as you saw with

Chapter 4 - Page 34
command-line tools, this visual representation provides you with an instant and intuitive
way of knowing how garbage collection is working. You get more detailed information
from the command-line, but get fast results with a visual tool.
o. Close all VisualVM instances.
5. Use jconsole to monitor and analyze your JVMs running WebLogic Server.
a. Verify that your domain is running and that server1 is running with a heap size of
50 MB.
b. The jconsole tool is included with the JVM. It is located in the Java bin directory,
which is in your path. Execute the following command to start jconsole:
$ jconsole

c. The New Connection window appears. This window displays local JVMs that
jconsole has discovered, and optionally allows you to connect to a remote JVM.
d. Select the server1 JVM and click Connect.

e. You should get an SSL failure because you have not configured an SSL connection for
jconsole. Click Insecure to continue on a non-SSL connection.
f. The interface for jconsole is very similar to other consoles, such as VisualVM. There
is a tab (on the bottom) for the JVM process, and there are multiple tabs for each
functional area, including overview, memory, threads, classes, VM summary, and a
new one called MBeans.
g. Click the Memory tab to see the heap in more detail. Notice that just like VisualGC,
there is a graphical representation of the Java heap. You can place the cursor over a

Chapter 4 - Page 35
space to see which space the graphics represent. You can click each space to change
the overall chart to display the graph for that space.

h. Click through each tab and experiment with each. The capabilities are very similar to
what you have already used with other tools so there are no detailed instructions for
this part.
i. Click the MBeans tab to display the list of MBeans for this JVM.
j. You should see a list of MBeans similar to what is shown in the following image. The
MBeans that start with com.bea are WebLogic Server runtime MBeans that contain
the runtime information of server1. You should notice within the com.bea node that
there are two server1 nodes. The first one represents WebLogic configuration MBeans
that contain the configuration associated with server1. This data is persisted in the
domains config.xml file. The second one represents the servers runtime
properties. This is where metrics associated with the running server are maintained.
When the server JVM is gone, this information is also gone unless it was persisted in
some way.
k. Browse to server1 (try each one to find the correct attributes) > AuctionImageServlet >
SimpleAuctionWebAppDb > ServletRuntime > server1_/SimpleAuctionWebAppDb >
Attributes. You should see a list of attributes for the MBean.
l. Run Grinder to load the application. Let it run for a few moments to generate some
data.
m. Click Refresh on the page in jconsole to update attribute data values. You should
see some real numbers in your display. If you keep refreshing the screen, you can see
the numbers change as the server tries to work through the load that is generated by
Grinder. This includes execution average time, total execution time, invocation count,
and more. This is one way to view WebLogic Server runtime statistics, but is still a lot
of effort to try to access information that should be more easily available. But still, it
does provide the information you need.
n. Double-click attributes to expand them into a graphical chart. You may need to run
Grinder again to see worthwhile results.
o. Experiment with jconsole on your own for 5 minutes.

Chapter 4 - Page 36
6. Use GCHisto to graphically analyze GC log output.

a. First, you have to locate the GCHisto program. The program is located in the
/install/java/GCHisto folder, and the file is named GChisto.jar.
b. Change your setUserOverrides.sh file again to print GC statistics again. This time,
include the following option to specify a particular file, /tmp/gc.log, for GC statistics
to get written (keep memory settings to 90MB):
-Xloggc:/tmp/gc.log
c. Restart server1 to realize your changes.
d. Verify that the /tmp/gc.log file exists and is getting GC statistics written to it.
e. Execute the following command to start GCHisto:

java -jar /install/java/GCHisto/GCHisto.jar
f. Run Grinder again to load the system and generate some real logs. Let it run for a few
moments before continuing.
g. Configure GCHisto to point to your GC log file by selecting Dynamic HotSpot GC Log
and clicking Add. If you select HotSpot GC Log, then you only get a single historic
snapshot of actual data. The dynamic option allows GCHisto to give you updated
metrics.
h. Browse to your GC log file and select it for GCHisto to use.

i. Experiment with Real Playback and Fast Playback options.
j. Use the GCHisto tabs to view the various charts to review garbage collection history of
server1. GCHisto reads the gc.log file and parses the GC records to create human
readable charts.

Chapter 4 - Page 37

k. Experiment on your own with GCHisto for 5 minutes.
l. Close GCHisto.
7. Reflect for a moment about the tools you have learned so far.
a. Now that you have used command-line tools and graphical tools to monitor and
analyze JVM performance characteristics, what are your thoughts when comparing
each of the tools you used?
b. When would you use each tool? What benefits do you see using command-line tools or
graphical tools? Do you have a preference? Why?
c. Several of the tools provide similar, if not identical, capabilities. Which of these tools do
you think provides you with the best of these capabilities? Why?

Chapter 4 - Page 38
Practice Solution

Chapter 4 - Page 39
Practice 4-5: Using Mission Control
Overview
The primary diagnostic tools that were part of the JRockit JVM implementation are now part of
the HotSpot JVM. The first of these tools is Mission Control. In this practice, you use Mission
Control to monitor and analyze your WebLogic servers. You can compare how this tool works
with the other tools you used in previous practices. The following image depicts the architecture
of the domain and tools used for this practice.

your own.
Dependencies
Java Development Kit version 1.7.0 update 40. Mission Control and Flight Recorder
features are not present before this release.

Chapter 4 - Page 40

Tasks
1. Connect to the host01 and host02 machines by using VNC.

$ ./setup.sh
4. Change your setUserOverrides.sh file again to turn on Mission Control and Flight
Recorder by using the following parameters (memory settings should stay at 90 MB):
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
5. Restart server1 to realize your changes.
6. Start Java Mission Control (JMC).
a. The script to start JMC is located in /u01/app/jdk/bin. This directory is already in
your path, so you can start JMC from any location. Execute the script to start the tool:
$ jmc &

Chapter 4 - Page 41
b. Verify that the GUI appears.

7. Connect to local WebLogic managed server with JMC.
a. In the left-hand panel, right-click one of the WebLogic JVMs in the list and select Start
JMX Console. This opens the JVM tab. Ensure that it is the server1 managed server
JVM using the Server Information tab on the bottom of the screen. You can also find
this information by using the Runtime icon and checking weblogic.Name in the
System Properties section. If it is not the server1 JVM, then close it and try another
JVM.

Chapter 4 - Page 42

b. Review the descriptions of the JVM tabs in the following table:
Tab Description
General Provides a graphical dashboard of gauges and charts that provide heap,
(Overview) CPU, fragmentation, and server information (in a separate tab on the
bottom of the screen). You can use the icons in the upper right hand of
the screen to manage and customize this page.
Overview controls:
Dashboard controls:
Chart controls:
These controls allow you to view more specific data that is focused on
your monitoring requirements. JMS provides a button to easily allow you
to revert to the default settings.
MBeans Displays the MBean tree of the running server. Because you are

Chapter 4 - Page 43
Tab Description
(MBean connected to a managed server, you can only view runtime data. The
Browser) default page displays the java.lang.OperatingSystem MBean,
which displays important performance metrics for the running JVM.
MBeans At the bottom of the MBeans screen, there are two tabs. The Triggers tab
(Triggers) enables you to configure threshold conditions for MBean metrics. You

can also configure actions for JMC to take when these conditions are
met, such as logging a message, dumping a Flight Recording, sending an
email, and starting a Flight Recording. You can then constrain the
condition to take place only during time periods. The tool enables you to
import and export rules, and to perform CRUD operations on your own
rules.
Runtime This tab displays a processor usage chart, JVM statistics, and system
(System) properties by default. You can customize what is shown on this page.
Runtime This tab displays a memory chart, JVM memory statistics, and system
(Memory) memory statistics by default. You can customize what is shown on this
page. Included on this page is a listing of free and used swap memory.
Runtime This tab displays garbage collection statistics for the young and tenured
(Garbage generations. You can customize what is shown on this page. Metrics are
Collection) displayed in a table format. The display includes the number of GC
threads per space by default.
Runtime This tab displays each of the memory heap spaces, the associated usage
(Memory numbers, and graphical bars to show real time memory usage in the JVM
Pools) much like VisualGC. The format is slightly different, but the information is
similar.

Chapter 4 - Page 44
Tab Description
Runtime This tab displays a live thread graph chart, live threads table with CPU,
(Threads) deadlock detection, and allocated bytes details per thread. It also displays
stack traces for selected threads. You can customize what is shown on
this page.

General Tasks
8. Use Mission Control to answer the following questions. You may run Grinder to load the
application for some of these.
a. What is the execution time average for the AuctionImageServlet? Where did you find
this information in JMC?
b. What is the current and maximum JVM CPU usage? Where is this information found?
9. Use Mission Control to perform the following tasks. You may use Grinder to load the
application for some of these.
a. Add a new dial on the General Overview tab that shows the machine CPU utilization.
Hint: The attribute is found in the Operating System node. You may collapse one of the
charts in the display afterwards, and expand it again, which should cause the scroll
bars for the display to work for the entire page instead of just the Dashboard section.
b. Remove the Live Set and Fragmentation dial.
c. Run Grinder and monitor the CPU, JVM CPU, heap, and garbage collection metrics by
using JMC. What are the characteristics of these metrics? Are they the same as when
you monitored the application using VisualVM?
d. Try to create your own dial and graph. Select attributes of your choosing.
10. Reset the display to its default settings.
MBean Tasks
11. Create a chart from an MBean attribute.
a. Open java.lang > MemoryPool > Eden Space > Usage > used.
b. Right-click the value and select Visualize.
c. In the Create Chart dialog, click Add Chart.
Note: You can add attributes to existing charts from this interface.
d. Click OK.

Chapter 4 - Page 45
e. Click on the General Page and review the chart you just created.
12. Export data from an attribute.

a. Return to the Eden Space attribute. Open java.lang -> MemoryPool -> Eden Space.
b. Click any attribute in the display.
c. Press Ctrl + A to highlight the entire table.
d. Right-click the table and select Clipboard Settings.
e. Make sure Copy as CSV is selected.
f. Right-click the table and select Copy.
g. Open a text editor and paste the data into it.
h. Examine the data briefly.
13. View JVM options with the Operations tab.
a. Within the MBeans tab, select an attribute and click the Operations tab to display
available operations for the attribute.
b. Open com.sun.management, choose the HotSpotDiagnostic MBean, and then
choose getVMOption within the Operations tab.
c. Enter the following as parameter 0 (p0): ThreadStackSize
d. Click the Execute button. The value set for that option in the JVM should be displayed.
e. Enter the following: MaxTenuringThreshold
f. Click the Execute button. The value set for that option in the JVM should be displayed.
14. Set JVM options with the Operations tab.
Note: This is an example of setting an attribute value using an MBean operation. There are
many operations available for all different kinds of MBeans. Just remember that you cannot
set, or write, to WebLogic Server runtime MBeans because they are read-only. All changes
to WebLogic Server MBeans must occur through the WebLogic change management
mechanism, where the configuration is locked, changes are made, and then changes are
activated. However, you can freely change JVM values for the JVM running WebLogic to
monitor and tune the JVM.
a. Select setVMOption within the Operations tab.
b. Enter the following values for p0 and p1 to dynamically enable printing GC data for this
JVM: PrintGC, true
c. Click Execute. The PrintGC option is now turned on.
d. Switch to the server1 terminal window where server1 is running. You should now see
GC data in the servers output window.
e. Turn off PrintGC using the values: PrintGC, false.
15. Turn on a notification for this JVM.
a. Select com.sun.management > GarbageCollectionAggregator in the MBean Tree.

Chapter 4 - Page 46
b. Click the Notifications tab.

c. Note that this particular MBean has two notification types.
d. Check the Subscribe box.
e. Wait for 2 or 3 events to occur.
f. Deselect the Subscribe button.
g. Examine the data that was collected.
16. Set Triggers for the JVM.
a. Create a trigger for an attribute by using the MBeans > Triggers tab. Be sure to choose
an attribute that you know will hit a conditional threshold, such as CPU Usage.
b. Configure a condition.

c. Configure an action, but do not configure a constraint. Be sure to choose an action that
you can verify in this environment, such as application alert, console output, or log to
file. Note that if you select console output, it refers to the console where Mission
Control was executed, not the server1 JVM. You may also want to set the Limit Period
to something like 5 seconds so you can see the alerts appear without waiting a full 60
seconds for every notification.
Runtime Tasks
You can run Grinder during your exploration of the Runtime tab to make your results more
interesting.
17. Explore each of the tabs encompassed by the Runtime button on your own for a few
minutes.
18. Enable CPU Profiling, Deadlock Detection, and Allocation in the Threads tab to see JMC
populate all the columns.
Finishing Up

Chapter 4 - Page 47
Practice Solution
subsequent lessons.

Chapter 4 - Page 48
Practice 4-6: Using Flight Recorder
Overview
The primary diagnostic tools that were part of the JRockit JVM implementation are now part of
the HotSpot JVM. The next tools originating from JRockit that is now part of the HotSpot JVM is
the Java Flight Recorder (formerly known as the JRockit Flight Recorder). In this practice, you
use Java Flight Recorder (JFR) to monitor and analyze your WebLogic servers. You can
compare how this tool works with the other tools you used in previous practices. The image
below depicts the architecture of the domain and tools used for this practice.

your own.
Dependencies
Practice 4-5 for using Mission Control
Java Development Kit version 1.7.0 update 40. Mission Control and Flight Recorder
features are not present before this release.

Chapter 4 - Page 49

Tasks

$ ./setup.sh
4. Configure and restart server1 if you ran setup.sh.
a. Change your setUserOverrides.sh file again to turn on Mission Control and Flight
Recorder by using the parameters below (memory settings should stay at 90MB):
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
b. Restart server1 to realize your changes.
5. Start Mission Control and start Flight Recorder.
a. If not started already, start Mission Control.
$ jmc &
b. Find the server1 managed server in the JVM browser.
c. Expand the arrow next to the JVM to show the available options.
d. Right click the Flight Recorder node and select Start Flight Recording. This causes the
Start Flight Recording dialog window to appear. This is where you configure the
settings for how you want Flight Recorder to work.
Note: If you received an error, you may not have enabled the Flight Recorder features.
Review step 4 to ensure you completed all the steps and try again.
e. Name your recording.
f. You have two options for recording:
Option Description
Time fixed recording Records JVM events for the specified period
of time. This is useful for reproducible
scenarios that you know will occur within a

Chapter 4 - Page 50
Option Description
certain time interval.
Continuous recording Records JVM events continuously for the
duration of the JVMs lifecycle. You can
truncate data by setting either a maximum size
of the data file, or a maximum age for events
stored in the data file. This is useful for
constantly monitoring your JVM, and
preserving a certain time period of events
leading up to an issue.

g. Select Timed fixed recording, and set a recording time of 5 min.
h. Flight Recorder comes with two templates by default:
Template Description
Continuous - on server This template is tuned for very low
performance overhead and is used for
continuous production use. It incurs
approximately 2% overhead.
Profiling - on server This template provides a balance of
information that is captured and performance
overhead. It captures more data than the
continuous template, but incurs approximately
2% overhead.
i. Because you are running a time-based recording, select Profiling - on server as the
Event settings value.
j. Click Next. This displays a list of settings for this recording.
k. Review each setting for a moment to learn what they do.
l. Set the Exceptions setting to All exceptions, including errors.
m. Check the Heap Statistics, Class Loading, and Allocation Profiling boxes.
n. Click Next.
o. Navigate to Java Application > Java Exception and ensure that the Enabled check box
is checked.
p. Click Finish.
6. Use application and wait for recording to finish.
a. Run Grinder with server1s heap set to 90MB to load the application with some traffic.
Flight Recorder will record JVM events related to this traffic.
b. Do not wait for Grinder to complete its load test. The five minute recording time should
elapse before Grinder is done. While Grinder is running, you should see a progress
meter in the Mission Control lower right-hand corner. Continue with the next step when
you see the recording appear in Mission Control.
7. Explore the Flight Recording.
a. When the recording process is complete, Mission Control downloads and opens the
recording in its Event tab.

Chapter 4 - Page 51
b. You should see the following tab groups, or icons, along the left-hand side of the
screen. Follow along in your Flight Recorder with each topic. Take a few moments to
experiment with each topic and tab.
General
Sub Tab Description
Overview Provides general high-level information, such
as CPU and heap metrics
JVM Information Provides information about the JVM that was
recorded

System Properties Provides all the system properties that are set
for the JVM that was recorded
Recording Provides a list of recordings and their event
types and settings available to each recording
Memory
Sub Tab Description
Overview Provides high-level memory usage, garbage
collection, and thread allocation statistics
Garbage Collection Provides a graphic and textual snapshot of
garbage collection activity for the JVM that
was recorded. Note that in the Heap graph
that GC pause times are superimposed with
heap use. The Heap and Garbage Collections
graphs also have several tabs you can view.
GC Times Provides high-level statistics about garbage
collections that occurred in the JVM that was
recorded. You can use the values here to see
the longest GC pauses and in which spaces
they occurred.
GC Configuration Provides data for how the heap and
generations are configured and sized.
Allocations Provides information about objects that were
allocated within the JVM, both in thread local
allocation bytes (TLAB) and outside thread
local space. You can use this to see how large
and frequent allocations are in the JVM over
time.
Object Statistics Provides details about objects that are
allocated in the heap during the selected time
period, the top growing objects, and the
percentage of heap they use

Chapter 4 - Page 52
The Code tab group is more for developers that want to see how well their code is
optimized and working within the server JVM.
Code (More for Developers)
Sub Tab Description
Overview Provides high-level information about which
packages and classes are spending the most
time executing in the JVM
Hot Methods Provides information about which methods are
spending the most time executing in the JVM,
and where they were called in the stack.

Developers can use this information to isolate
exactly where their code is taking the most
time, and to see if there are ways to optimize
for better performance.
Call Tree Provides the most common sampled stack
traces for the selected time period, which
developers can use to find performance
bottlenecks.
Exceptions Provides information regarding exceptions
thrown within the JVM. Developers can use
the data in this tab to investigate the classes
that throw the most exceptions, and see if
there are any problems that cause excessive
errors.
Compilations Provides information about code generation
changes that occurred during the recording.
Developers can use the statistics in this page
to see where the JIT compiler is working the
most and see if their code is the most efficient
for what it is doing.
Class Loading Provides information about class loading and
unloading. Developers can use this
information to determine which classloaders
are involved, and which classes that are
loaded and unloaded by each.
Threads
Sub Tab Description
Overview Provides a high-level view of how CPU and
threads are used by the JVM and the system.
Hot Threads Provides a list of threads that spend the most
time executing. You can review the stack trace
for the thread to get more details about what it
was executing at the time.

Chapter 4 - Page 53
Threads
Sub Tab Description
Contention Provides information about threads that
compete for a synchronization lock, blocked
threads, and details about each thread.
Latencies Provides information about when threads are
not executing code. You can see how much
time your threads are waiting, blocked, or
sleeping. Note that if there is no load on the
application that a lot of threads waiting is not

unusual.
Thread Dumps Shows the thread dumps that were captured
during the selected recording period. You can
use this to view the stack traces of JVM
threads over a period of time to see how the
application is functioning.
Lock Instances Provides details about thread locking that
occurred during the recorded time period. You
can see the address and class of the
synchronized object, and the amount of time
the lock was in place.
I/O
Sub Tab Description
Overview Provides information on file and socket reads
and writes. You can use this information to
see how I/O intensive your JVM is, and where
I/O occurs the most.
File Read Provides information on file reads that
occurred during the selected recording time
period. You can view file reads by thread, by
event, over time, and related stack traces.
File Write Provides information on file writes that
period. You can view file writes by thread, by
event, over time, and related stack traces.
Socket Read Provides information on socket reads that
period. You can view socket reads by thread,
by event, over time, and related stack traces.
Socket Write Provides information on socket writes that
period. You can view socket writes by thread,
by event, over time, and related stack traces.

Chapter 4 - Page 54
System
Sub Tab Description
Overview Provides information about the hardware
where the JVM is running
Processes Provides information for all processes in the
system
Environment Variables Provides a list of the environment variables
that were set when the JVM was started. This

is useful for when you want to investigate if
there are issues with an incorrect environment
for your JVM.
Notice that when you click the Events icon that the left-hand pane displays a tree
of event types that you can include in the view of charts on the right-hand pane.
Experiment with each of the settings on the left and see how they change the view on
the right. At this point, you will not see any data associated with WebLogic Server
because that data is captured either through WebLogic Diagnostic Framework (WLDF)
notifications or opening a Flight Recorder recording contained in a WebLogic
diagnostic image.
Events
Sub Tab Description
Overview Provides a detailed breakdown of event data
for a particular time period. You can select a
period of time to analyze, see the producers of
the events, and view details of the different
event types.
Log Provides details for events. You can sort by
column data, Duration for example, to see
which events took the longest to execute.
Graph Provides activity graphically by which thread
performed the work.
Threads Provides performance metrics on a per thread
basis.
Stack Traces Provides stack traces for threads.
Histogram Provides tools for analyzing events grouped
by a certain property, which can also be used
with an operative set of events with matching
properties.
c. Now that you have followed along in Mission Control to explore these topics in this
practice, take a few minutes to explore parts of the interface that interest you before
going to the next step.

Chapter 4 - Page 55
8. Capture and View WebLogic Server Flight Recorder data.

You can make WebLogic take certain actions when a WLDF watch condition is reached, or
if certain server scenarios take place, that will cause WebLogic to create a diagnostic image
of the running server. This provides useful data that you can use to discover causes of
performance problems that occur. You can also manually create a diagnostic image for
analysis as shown here.
a. Login to the WebLogic administration console.
b. Navigate to Diagnostics > Diagnostic Images.
c. Select server1 and click Capture Image to cause WebLogic to dump a diagnostic
image of the server to disk.
d. Specify the location and timeout values on the page and click OK to trigger the image

creation process. This will dump data associated with the default WLDF Low built-in
module that is configured.
e. Within a terminal window, navigate to
/u01/domains/tune/wlsadmin/servers/server1/logs/diagnostic_image
s.
f. Locate the file with the .jfr extension. This is the Flight Recorder recording
associated with your diagnostic image. A copy of it is also contained in the diagnostic
image zip file, called FlightRecorder.jfr.
g. Within Mission Control, select File > Open File and browse to this flight recording file to
open it in the Mission Control tool.
h. Click the Events icon to display the Event Types View in the left-hand pane.
i. Check the WebLogic Server check box to display WebLogic Server events in the right-
hand pane.
j. Use the drag controls in the Overview tab to isolate a section of the recording time. The
Interval, start, and end times change to reflect the selected time frame of the recording.
k. The charts and tables on the page change to reflect the data for the selected time
interval. You can view this data, and then pick and choose what event type data to
include by using the left-hand pane. This enables you to correlate how certain
WebLogic operations relate performance-wise to other metrics, such as CPU Load.
l. Click the Log tab to show more details about events.
m. Click the Duration column to see which events are taking the longest to execute.
n. The Execution Context Identifier (ECID) is an internal representation of related
requests within Oracle products. You can create an operative set within Flight
Recorder to show only the events related to certain data, such as an ECID value.
Right-click an event entry of Event Type Servlet Invocation in the Event Log section
and select Operative Set > Related Events > With ECID=<value> to create an
operative set.
o. Check the Show Only Operative Set check box to display those events by themselves.
This allows you to see where the most time is spent for certain operations. Again, this
may be more developer focused, because it drills down to the code level, but there
may be times that this provides information that is useful to administrators as well.
p. Note the data contained in the Event Attributes section. If one of your operative sets
has a Pool attribute, it should be a database connection pool name. If it is, then you
should also be able to see the SQL statement associated with the request. This can
help with performance problems where database requests are taking too long. You can
find the long running requests in WebLogic, then track it down to the database call, and
uncover which SQL statement is taking a long time to execute. Then database

Chapter 4 - Page 56
administrators (if this is not also you) can investigate those queries and tables to see
where a problem may exist. Perhaps a table wasnt indexed properly? This is one way
to identify the issue and the cause faster.
q. Take a few minutes and experiment with the settings on this page and the other tabs.
9. Experiment with Flight Recorder commands.
a. Review the commands in the following table and use them for the following step.
Command Description
jcmd pid JFR.start [options] Start a Flight Recorder recording from the
command line

jcmd pid JFR.check [options] Check the status of recordings for the
specified process ID
jcmd pid JFR.stop [options] Stop a recording with a specific
identification number. Recording 1 is
stopped by default.
jcmd pid JFR.dump [options] Dump the data collected so far by the
recording with a specific number.
Recording 1 is dumped by default.
The documentation reference page for these diagnostic commands is located in
/practices/tune/practice04-06/resources/app_command_ref.html.
Reference it to experiment with these commands.
Note: You can also configure the JVM to create a recording when the JVM exits. This is
useful troubleshooting issues that cause the server to crash. Additionally, you can configure
conditional triggers in Mission Control that cause a flight recording to get created when the
condition is met. This is useful for capturing data that leads up to an issue, while
simultaneously minimizing the amount of data that gets recorded.
10. Close all Grinder applications and terminal windows.

You may have noticed that the Flight Recorder and Mission Control tools both offer loads of
settings and features. Each tool can probably support a course of their own, and unfortunately,
we are really only able to show the high-level features of the products.
Mission Control provides several of the same features as the other graphical tools. You may
have also noticed that it combines a lot of features within a single program and interface, and
provides more features by default. When you are monitoring your WebLogic environments, the
tools you use will be up to you. There are many tools available to monitor your JVMs. Covering
them all goes beyond the scope of this course.

Chapter 4 - Page 57
Practice Solution
There is no solution for this practice. Although no other practices depend on this lesson, the
experience gained can be applied in subsequent lessons.

Chapter 4 - Page 58

Monitor and Tune Server
Performance
Chapter 5

Chapter 5 - Page 1
Practices Overview
In these practices, you configure the WebLogic Server to handle stuck threads. You then deploy
an application that creates stuck threads, and observe the affected managed server reboot after
the stuck threads thresholds you configured are exceeded.
You also configure WebLogic to react when it transitions to the OVERLOADED state. You
deploy an application that exhausts the servers memory, and observe how WebLogic Server
functions when a managed server is overloaded.
Finally, you configure logging and log filtering, adopting a strategy that allows for accurate

pinpointing of issues without incurring heavy performance penalties in terms of I/O generated
while logging.

Chapter 5 - Page 2
Practice 5-1: Configuring Server Stuck Thread Behavior
Overview
In this practice, you configure WebLogic Server to manage when it encounters stuck threads.
You set the maximum number of threads after which the server is transitioned into the FAILED
state, and the number of seconds a thread must be continuously working before the server
considers it a stuck thread. Additionally, you configure WebLogic to force an immediate
shutdown of the server after its state has been transitioned to a FAILED state.

This practice provides detailed instructions on how to accomplish the objectives of the lesson.

Dependencies
Tasks
$ ./setup.sh
Starts Node Manager on each host
Starts the domain using Node Manager
4. Configure stuck thread handling on server2.
Note: You are modifying the behavior that the WebLogic server will adopt when stuck
threads are detected. You want WebLogic to automatically shut down the server when a
certain threshold of stuck threads is exceeded. By default, the WebLogic server does not
perform any action.
a. Log in to the WebLogic administration console.
b. Navigate to Environment > Servers to display the servers configured for this domain.
c. Click server2.
d. Click the Configuration > Overload tabs to display the configuration settings related to
the WebLogic server behavior under stress.
e. Click Lock & Edit.
f. Set Failure Action to Force Immediate Shutdown of this server. This tells WebLogic to
automatically force this server to shut down when the server enters into a failed state.
g. Set Max Stuck Thread Time to a low value of 6 (seconds). You deploy an application in
the next practice that artificially causes stuck threads and this enables you to observe
the behavior of WebLogic without waiting for too long.

Chapter 5 - Page 3
h. Set Stuck Thread Count to a low value of 4 (threads). In the next practice, you use the
application that causes stuck threads; initially you spawn three stuck threads to
observe the behavior of WebLogic before the thread threshold is exceeded.
Subsequently, you will instruct the application to spawn five stuck threads, causing
server2 to enter into a failed state.

i. Save and activate your changes to realize your configuration.
5. Configure how server2 manages automatic restarts.
Note: The default settings for WebLogic automatic server restarts are set for a practical
real-world scenario. You must configure these settings to promote more activity so you can
witness how WebLogic functions when stuck threads are detected.
a. Click the Health Monitoring tab for server2.
b. Click Lock & Edit.
c. Set Max Restarts Within Interval to 500. This instructs WebLogic to allow this server to
restart 500 times within the default 3600-second time interval. By default, this value is
set to 2 and you could easily miss the server restarting. Then you would have to wait
the full 3600 seconds for WebLogic to perform a restart again.
d. Increase Restart Delay Seconds from 0 to 2. You want to give both WebLogic and
Linux a chance to clean memory and resources (like TCP/IP connections) after the
server has been abruptly shut down after reaching its stuck thread threshold.
e. Ensure Auto Restart is selected so Node Manager will automatically restart this server.
f. Save and activate your changes to realize your configuration.
6. Restart server2 for changes to take effect.
a. Navigate to Environment > Servers > Control.

Chapter 5 - Page 4
b. Select server2 and click Shutdown > Force Shutdown Now.

c. Confirm Yes.
d. Click the icon to cause a continuous refresh of the Summary of Servers page. After
a few seconds, the state for server2 changes to SHUTDOWN.
e. Select server2 again and click Start to instruct WebLogic to request the Node Manager
running on host02 to start server2.
f. Confirm Yes.
g. Wait for the state of server2 to change to RUNNING.
h. This practice configures the server for stuck thread handling. The next practice builds
on this practice to cause a stuck thread condition to occur. Leave the WebLogic
administration console and domain running as it is used in practice 5-2.

Chapter 5 - Page 5
Practice Solution
solution. The subsequent practices of this lesson depend on this practice.
Solution Tasks
$ ./solution.sh

Sets the practice to its starting point by calling the cleanup.sh script to reset the
domain and general environment
Starts all the servers of the domain in the background using Node Manager
Configures overload protection settings for stuck threads and server restart
5. This solution performs all the steps of the practice. You can use the WebLogic
administration console to verify that all servers are in the RUNNING state before moving on
to the next practice.

Chapter 5 - Page 6
Practice 5-2: Configuring Automatic Restarting of a Failed Server
Overview
In this practice, you deploy the StuckThreads application to server2. StuckThreads provides a
very simple input page where you specify how many stuck threads you want to create and for
how long those threads should run. In practice 5-1, you set the stuck thread count threshold to
four. Initially, you will create three long-running stuck threads and you will assess the behavior
shown by WebLogic server when stuck threads are detected. Subsequently, you will ask
StuckThreads to spawn five stuck threads, thus exceeding the stuck thread count threshold and
the failure of server2. After the server has failed, WebLogic will automatically shut down server2
and the Node Manager will restart it.

Dependencies
Practice 5-1 for starting and configuring the domain used for this practice
Tasks
1. Deploy the StuckThreads application.
a. Perform the following commands to navigate to the practice folder and deploy the
StuckThreads application:
$ ./deploy.sh
b. Within the WebLogic administration console, navigate to Deployments and verify that
the application is deployed properly and is ready to service requests.
2. Run the StuckThreads application.
a. Click the StuckThreads link to open its configuration page.
b. Click the Testing tab.

Chapter 5 - Page 7
c. Click the link displayed in the Test Point column that links to the index.jsp page
hosted on host02.

d. The StuckThreads input page displays in a new Firefox tab. Select the StuckThreads
Firefox tab, accept the proposed value for Number of threads to run (3) and increase
the number of Seconds to run to 300:
e. You are instructing the StuckThreads application to spawn three threads, which will run
uninterrupted for five minutes, which is enough time to assess the behavior of the
WebLogic server under that level of stress. Click submit and then click the Firefox tab
corresponding to the WebLogic administration console.
f. Click the Servers link to display the servers running in the domain.
g. Click the icon to trigger a continuous screen refresh.

h. Wait a while and see whether the administration console alerts you to any problems.
Do you see any problems?

Chapter 5 - Page 8
i. Find the System Status pane in the administration console. Does it indicate that there
could be a problem?
j. You should see that WebLogic has changed the state for server2 to Critical. Click the
Critical (1) link in the System Status pane to display the Health Information window,
which shows three stuck threads detected running in server2.

k. After the three threads stop working (approximately five minutes), the server recovers
from its critical state and its health state back to OK.
3. Force an automatic forced shutdown and restart of server2.
a. In the administration console, navigate to server2 > Monitoring > General to display
some metrics for server2.

Chapter 5 - Page 9
b. Write down the activation time for server2. You compare this value with its value after
forcing a restart to verify that Node Manager has in fact restarted the server.

c. Return to the StuckThreads application in Firefox.
d. Click Re-launch to run the application again.
e. Enter the following values and click submit.
Field Value
Number of threads to run 5
Seconds to run 300
f. Return to the WebLogic administration console.
g. Navigate to Environment > Servers to display the servers of the domain.
h. Click the icon to trigger a continuous screen refresh.

i. Watch the console as it refreshes. After a few seconds the state of server2 changes to
STARTING. WebLogic detects that the configured stuck thread threshold is exceeded,
which triggers WebLogics overload protection feature. The action for a failed server is
to shut down the server, so WebLogic forces server2 to shut down. The Node Manager
running on host02 detects that server2 shuts down ungracefully and restarts the server
automatically.

Chapter 5 - Page 10
j. The System Status panel shows one server in critical condition. Click the Critical(1) link
to display the Health Information window, which shows that server2 is not reachable.
This condition is only visible for a few seconds, the time required to restart the server
plus the idle time you set up previously to allow for memory and network cleanup (two
seconds).

k. Return to the servers page and verify that the state of server2 returns to OK.
l. Verify that server2 was indeed restarted by returning to the Servers > Monitoring >
General tab to compare the Activation Time you wrote down previously in step 3b.
4. Feel free to cause server2 to restart several times. Each time try to access different views
of the failed server in the WebLogic administration console. For example, you can see what
information the console displays while the server is restarting when monitoring the servers.
Alternatively, next time you cause the server to fail, you can see what information is
displayed by the console while you are monitoring the deployments.
After you finish experimenting, do not reset the health monitoring parameters to their default
values. Leave the environment as it is and proceed to practice 5-3.

Chapter 5 - Page 11
Practice Solution
There is no solution for this practice.

Chapter 5 - Page 12
Practice 5-3: Configuring WebLogic Overload Protection
Overview
In this practice, you configure the memory parameters accessed by WebLogic when a server
becomes overloaded. You then use an ad-hoc application, SaturateMemory, to induce memory
exhaustion on one server, and monitor the behavior of the WebLogic server when overload
protection is configured.


Dependencies
Practice 5-2 for some parameters to align with the instructions in this practice.
Otherwise, practice 5-2 is not required for completing this practice
Tasks
1. Configure server2 memory overload protection.
a. Within the WebLogic administration console, navigate to Environment > Servers >
server2 > Configuration > Overload to display the overload protection settings for
server2.
c. Change the Failure Action value to Ignore, take no action.
d. Set Free Memory Percent High Threshold to 20.
e. Set Free Memory Percent Low Threshold to 10.
f. Reset the Stuck Thread parameters to their defaults as shown in the following table:
Field Value
Stuck Thread Count 0
Max Stuck Thread Time 600

Chapter 5 - Page 13
g. Save and activate your changes to realize your configuration.

Note: With these settings, you are forcing WebLogic to transition a server to an overloaded
state if the amount of memory available to it falls below 10 percent. The server will clear its
overloaded state if the amount of memory climbs back to at least 20 percent.
2. Restart server2 using the administration console for your changes to take effect.
3. Deploy the SaturateMemory application.
a. Perform the following commands to navigate to the practice folder and deploy the
SaturateMemory application:
$ ./deploy.sh

b. Within the WebLogic administration console, navigate to Deployments and verify that
the application is deployed properly and is ready to service requests.
4. Run the SaturateMemory application.
a. Click the SaturateMemory link to open its configuration page.
b. Click the Testing tab.
c. Click the link displayed in the Test Point column that links to the index.jsp page
hosted on host02.
d. The Memory Exhaustion Application input page displays in a new Firefox tab. Click the
SaturateMemory Firefox tab, accept the proposed value for Number of threads to run
(3) increase the number of Seconds to run to 300.

Chapter 5 - Page 14
e. Click Submit and the application spawns three threads, each allocating a large chunk
of memory. If the amount of memory available in the JVM running server2 is already
sufficiently low and garbage collection does not free up a lot of memory, calling the
SaturateMemory application once is enough to cause an OVERLOADED state for
server2.
f. Sometimes, calling the SaturateMemory application once is not enough to trigger an
OVERLOADED condition. This could happen because after the initial memory allocation
requested by SaturateMemory, garbage collection is performed by the JVM and more
than 10 percent of memory becomes available. In this case, you must click the link
Re-launch SaturateMemory page at the bottom of the page and request another batch
of threads to allocate more memory. Usually, running SaturateMemory twice is

sufficient to trigger an OVERLOADED condition for server2. Rarely would you need to
run SaturateMemory three times to obtain the OVERLOADED state.
g. After the amount of memory shown at the bottom of the Memory Exhaustion page falls
below 10 percent of all the available memory, you can view the servers health state in
the administration console.
h. Verify that the health state for server2 has changed to Overloaded.
i. The System Status pane also reports the overloaded condition:

Chapter 5 - Page 15
5. Click the Overloaded link in the System Status pane to drill down on the server affected by
low memory.
a. The domain Monitoring > Health page is shown, and server2 is displayed as a link.

b. Click the plus (+) sign on the left of the server2 link to expand the health information:

Chapter 5 - Page 16
c. Click the plus (+) sign on the left of server2 again to drill down on server2 components
to get more details about where the problem lies.

d. After approximately five minutes the threads that allocated a significant portion of the
JVM memory stop running. The memory is released and more than twenty percent of
heap memory becomes available to the server. The overloaded state is automatically
cleared and a green tick sign ( ) and OK appear in the Health column for the server2.
6. Feel free to cause server2 to overload several times. Each time try to access different views
of the overloaded server in the WebLogic administration console. Set the Seconds to run
field in the input page for the SaturateMemory application to shorter values (such as 60
seconds) so you can experiment without waiting too long for WebLogic to clear the
overloaded state.
After you finish experimenting, do not reset the health monitoring parameters to their default
values. Leave the environment as it is and proceed to practice 5-4.

Chapter 5 - Page 17
Practice Solution
solution. No subsequent practices of this lesson depend on this practice.
Solution Tasks
2. Change the current directory to the current practice folder
(/practices/tune/practice05-03).
$ ./solution.sh

Starts all the servers of the domain in the background using Node Manager
Configures the server memory thresholds to cause the overload condition
5. Continue with step 3, "Deploy the SaturateMemory application."

Chapter 5 - Page 18
Practice 5-4: Configuring Log Filters for the WebLogic Server
Overview
In this practice, you initially change the default log level to obtain more debugging information.
Subsequently, you define log filters to improve the accuracy of logging, avoiding unnecessary
log verbosity for WebLogic subsystems that you are not interested in tracing.

Dependencies

Tasks
1. Configure server2 logging levels.
a. In the WebLogic administration console, navigate to Environment > Servers > server2
> Logging > General to display the overload protection settings for server2.
c. Scroll to the bottom of the window and click the Advanced link to open advanced
configuration options.
d. Set Minimum severity to log to Trace.
e. Set Severity Level for the Log file to Trace.
f. Set Severity Level for standard out to Debug.
g. Set stdout Stack Trace Depth to -1. This forces all stacktrace frames to be displayed.
h. Save your changes. Do not activate yet.
2. Configure WebLogic debugging for server2, which can potentially cause verbose logging.
a. Click the Debug tab.
b. Expand the weblogic node to display a list of all its debug categories.
c. Expand the transaction node to display its debug categories.
d. Expand the jdbc node, and then select DebugJTAJDBC.
e. Scroll to the bottom of the page and click Enable.

f. Activate your changes to realize your configuration.

Chapter 5 - Page 19
3. Create a log filter.

Note: You want to define a log filter to limit the amount of JDBC information being logged.
a. Within the WebLogic administration console, navigate to wlsadmin > Configuration >
Log Filters to display.
c. Click New to create a new log filter. Name the new filter LogFilter-JDBC and click OK.
d. Click LogFilter-JDBC to open its configuration page.
e. Click Add Expressions to enter new filtering clauses.
f. Set Message Attribute to SEVERITY.
g. Accept the default operator (the equal [=] sign) and set Value to ERROR.

h. Click OK.
i. Save your changes.
Note: The filter you just created will allow only exceptions with a severity level of ERROR or
higher to be logged.
4. Assign LogFilter-JDBC to the logging facility for server2.
a. Navigate to Environment > Servers > server2 > Logging > General to display log
settings for server2.
b. Scroll down to the bottom of the window and click the Advanced link to open advanced
configuration options.
c. Scroll down to the Message destination(s) section and set Filter to LogFilter-JDBC
filter for both Log File and Standard out.
d. Save and activate your changes to realize your configuration.
Note: No server restarts are required after you define and assign log filters, which means
that you can modify logging levels and logging filters dynamically.
What did you achieve through this practice? The ultimate goal is to capture as little log
information as possible, in order not to burden the I/O system of your managed servers. At
the same time, should the need arise, you want to elevate the logging level to accurately
pinpoint issues that may occur in one or more WebLogic Server subsystems. You were
interested in capturing log information about JDBC transactions, so you configured a logger
that logged all stacktrace frames at "trace" level, which is an extremely verbose logger. To
prevent a performance hit due to the high verbosity level of the logger you defined, you
created a log filter that logs only errors and higher severity (CRITICAL, ALERT,
EMERGENCY) exceptions. If you are investigating transaction-related
issues, you could define one more log filter that allows logging to be
captured from a lower severity level (such as WARNING or DEBUG) and
dynamically assign the new log filter to the managed server, thus
increasing the amount of information sent to the log files. As soon as
your investigation is over, you can readily switch back the log filter to
capture only errors and higher severitylevel exceptions.

Chapter 5 - Page 20
Practice Solution

Chapter 5 - Page 21

Chapter 5 - Page 22

Monitor and Tune Cluster
Performance
Chapter 6

Chapter 6 - Page 1
Practices Overview
In the practices for this lesson, you experiment with features that affect performance when using
WebLogic clusters. First, you review the domains default unicast cluster configuration. Next,
you configure the domain to use multicast cluster communication. You monitor both types of
cluster communication by using the WebLogic administration console. Then you configure and
compare the performance of different session persistence types.

Chapter 6 - Page 2
Practice 6-1: Configuring Multicast and Unicast Clustering
Overview
This practice shows you how to use the WebLogic administration console to view unicast and
multicast performance metrics, and how to configure both protocols. The following image
depicts the architecture of the domain used for this practice.
Unicast cluster messaging: Unicast cluster messaging uses the TCP/IP networking protocol
to send heartbeat and JNDI state messages to all server members of a cluster. WebLogic
clustering breaks a cluster up into groups of 10 servers, with each group getting assigned a
group leader. The group leader is responsible for receiving messages from its group members,
and relaying them on to the other group members, as well as other group leaders in the cluster.

TCP/IP is implemented to be a reliable networking transport and provides bidirectional and
retransmission features.

Chapter 6 - Page 3
Multicast cluster messaging: Multicast cluster messaging uses the UDP networking protocol
to send heartbeat and JNDI state messages to all server members of a cluster. All WebLogic
server cluster members send UDP network traffic over a common address and port that all
servers listen to for cluster messages. UDP is a simple networking protocol that does not
provide any guarantee of message delivery, ordering, or duplicate prevention. WebLogic is
responsible for using multicast in a way that makes its use reliable. UDP messages are light and
fast on the network.

your own.
Dependencies
Tasks
$ ./setup.sh

Chapter 6 - Page 4

Copies a new setUserOverrides.sh script to the domain to limit heap sizes for
servers so multiple servers can be started (performed on host01 and host02)
Creates a total of 10 managed servers and adds them to cluster1
Deploys the ShoppingCart application to the cluster with in-memory session
replication configured
Starts the odd numbered managed servers on host01

Starts the even numbered managed servers on host02
b. Wait for all servers to log that they are in the RUNNING state before continuing. This
will take several minutes.
3. Verify that all servers started.
b. Navigate to Environment > Servers and verify that all servers are in the RUNNING
state.
4. Monitor unicast metrics.
a. Because the default cluster messaging protocol is unicast, it is already configured and
ready to explore. Within the console, navigate to Clusters > cluster1 > Monitoring >
Summary to view the statistics for the cluster and all the servers running in the cluster.
b. Review the default table. Answer the following questions:
Which server is the group leader?
Are there any remote groups?
Have any servers dropped out of the cluster? How can you tell?
Hint: One way to cause a server to drop out of the cluster is to press Ctrl + z in its
terminal window to suspend the process. Do not do this to the AdminServer. Wait
for a couple of minutes for the cluster to recognize it is gone. Then enter fg to
cause the process to continue running. Wait a minute again for the cluster to
recognize the server is back.
c. Answers to the questions:
Server1 should be the group leader because it is the server that starts first in the
configuration.
Note that there are no remote groups because unicast clustering causes cluster
members to get divided into groups of 10, with one server assuming the leader role.
Because cluster1 only consists of 10 servers, there is only one group. Total Group
Count should show only one group.
The first question to the Have any servers dropped from the cluster question
should be, Why is this important for performance? The answer to this is that if
servers are dropping out of the cluster, it could be due to performance problems
whereby cluster messages are not reaching their intended targets. When cluster
messages fail to reach their targets and servers fail to respond to these messages,
they can get dropped from the cluster. This does not always point to a performance
issue, but it could. It can be a networking issue or a very oversaturated CPU issue
as well. It could be a sign that one or more group leaders can be overrun with
requests. Admittedly, monitoring these metrics in the administration console is not
ideal. If you were using WLST or an MBean browsing tool, such as Mission Control,

Chapter 6 - Page 5
you could watch the metrics in real time to see how much traffic each server is
handling for cluster messaging.
d. Customize the table to view more metrics. Click the Customize this table link and add
the following columns to the display:
Column Description
Fragments Received The total number of messages received from
the cluster
Fragments Sent The total number of messages sent to the
cluster

Open Sockets Current Count The number of sockets currently open on this
server
Resend Requests The number of messages for state changes
that had to be resent because a server in the
cluster missed the message
Sockets Opened Total Count The total number of sockets opened by this
server
e. Review the table again and answer the following questions:
Which server has the highest open socket count? Why?
Were any cluster messages resent by any servers? If so, does there seem to be any
problem?
f. Answers to the questions:
You should see that the group leader has the highest open socket count. This is
because the group leader has a socket open to every server in its group, as well as
a socket to every other group leader in the cluster. In this case, there are no other
group leaders.
You should see some resent messages for just about every server. This is very
normal. However, if these figures are rising quickly, it could indicate a performance
problem.
5. Explore unicast configuration parameters.
a. Within the WebLogic administration console, navigate to Clusters > cluster1 >
Configuration > Messaging. This page shows the settings for configuring cluster
messaging. As you can see, the Messaging Mode setting is set to unicast. The page
automatically adjusts for when multicast or unicast messaging is selected.
b. The only setting on this page that can affect unicast performance is Unicast Broadcast
Channel. If the machines used by your domain offer network adapters with higher
performance, such as the InfiniBand network adapters on Exalogic platforms, internal
cluster messaging can benefit from separating the traffic onto a separate channel that
is dedicated to cluster messaging.
6. Configure multicast communication.
Note: Although both unicast and multicast cluster messaging are supported, Oracle has
begun to recommend the use of multicast messaging whenever possible. This is because
there are cases where unicast messaging can cause performance problems due to group
leaders being overrun with requests. The documentation is being reviewed to properly
reflect Oracles recommendations.

Chapter 6 - Page 6
a. Remain on the Settings for cluster1 page.

b. Click Lock & Edit to obtain the edit lock for the configuration.
c. Click the Advanced link near the bottom of the page to display advanced settings.
d. Set Messaging Mode to multicast. You should notice that unicast settings become
grayed out, while multicast settings become available for editing. Leave the other
settings at their current values of 239.192.0.0 and 7001.
e. Click Save to save your configuration.
f. Activate your changes.
g. Review the advanced settings for multicast messaging in the following table. Most of
the time you should not have to change these settings because they usually work by

default. However, if you are experiencing certain types of problems, these settings can
help resolve them.
Setting Description and Effect on Performance
Multicast Send Delay Sets the delay in milliseconds to wait before
sending a multicast message to avoid
overflowing the multicast buffer. Setting this
value too low could cause the network buffer
to overflow. You can see that the default value
is pretty low at 3 already. Setting this value too
high could cause slower cluster messaging,
which could potentially be problematic when
there are a lot of messages to send.
Multicast TTL Time to Live indicates the number of network
hops a multicast message can travel. By
default, this is set to 1 to contain messages to
a single subnet. Increasing this value too
much in the wrong environment could cause
performance issues if you are relying on
multicast messages making too many network
hops to reach their destination. If your
multicast messages are timing out, then it
could indicate that TTL requires an increase.
Multicast Buffer Size Sets the multicast socket buffer size in
kilobytes
Idle Periods Until Timeout The number of timeout periods a cluster
member must miss before the cluster
determines the member is absent. If this value
is set too low, then the cluster could
mistakenly remove the server from the cluster
prematurely. If this value is set too high, the
cluster could take longer than acceptable to
detect an absent cluster member.
Enable Data Encryption Causes multicast data (not the headers) to be
encrypted. With any encryption of data, there
is a performance tradeoff. Normally, internal
cluster messages are made between servers
running on internal machines that are behind a

Chapter 6 - Page 7
Setting Description and Effect on Performance

corporate firewall. The need for encrypting
these messages is low, unless there is reason
to suspect that an attacker has some sort of
access inside the company network. This
feature is disabled by default because it may
have a performance penalty.
h. Execute the following scripts to restart the domain on host01.
Note: For multicast clustering to work, set -Djava.net.preferIPv4Stack=true.
This is set in the setUserOverrides.sh script.

$ killServers.sh
$ ssh host02 killServers.sh
$ startAdmin.sh
$ ./startHost1Svrs.sh
$ ./startHost2Svrs.sh
i. Wait for all servers to start again, which will take a few minutes.
state.
8. Monitor multicast metrics.
a. Navigate to Environment > Clusters > cluster1 > Monitoring > Summary to view metrics
for cluster1.
b. You should notice that the metrics are similar to the metrics for unicast messaging.
Review the table and answer the following questions:
Verify that message fragments sent is approximately 10% of messages received.
Why do you think this is the case?
Review the number of resend requests and messages lost. Are the numbers low?
Are they acceptable? Why or why not?
If you see high resend requests and high messages lost values, is there definitely a
cluster messaging performance problem?
c. Answers to the questions:
This is because each server sends data out of 10 servers, and each server receives
the data from all 10 of the other servers, including itself.
As with unicast messaging, it is typical for there to be some resent or lost
messages. The infrastructure is implemented to recover from transient issues like
this. However, if you were to see a lot of resend requests and lost messages, and
the numbers continue to rise, you could be experiencing a performance problem.
Not necessarily. In general, a high number of resend requests and lost messages
could indicate a performance problem. However, if the spike is only occurring on a
single server or is isolated to a particular machine, then it could simply be a problem
with that server or machine. Often, you must analyze performance problems
holistically. Problems exhibited in cluster messaging could be a problem with cluster
messaging performance, but it could also be an issue with CPU utilization. For

Chapter 6 - Page 8
example, if CPU utilization is pinned to 100%, it is very possible that the kernel calls
to execute networking instructions are not even getting scheduled to execute in the
first place. This is a CPU performance problem, not necessarily a cluster messaging
problem.
environment down and return the domain to its original state.
$ cleanup.sh
b. If any dialog boxes appear asking if you want to close a terminal window, click OK to
close the window.
c. You should notice that the AdminServer and all managed server terminal windows and

the servers that were running in them are all shut down now. If any terminal windows
for managed servers remain, close them manually.

The primary focus of this practice is to monitor and configure unicast and multicast cluster
messaging. Although there are utilities provided by WebLogic to test multicast configurations,
such as the MulticastTest and MulticastMonitor tools, they are not really performance-
based tools. That is why they are not covered in this practice. Although we did not create an
application that performs a multitude of JNDI tree updates and have you run Grinder to load the
cluster, you should have a decent grasp on where to look for cluster messaging performance
issues, and how to resolve them. One of the biggest tips is that if you see your unicast group
leaders getting overrun with data, then that could be a good indicator to switch to multicast.
Naturally, there are several issues to using multicast, including if networking equipment handles
multicast packets and if multicast is politically allowed in your production environment. This is
something that must be managed on a case-by-case basis.

Chapter 6 - Page 9
Practice Solution

Chapter 6 - Page 10
Practice 6-2: Tuning Session Persistence
Overview
This practice has you experiment with configuring different HTTP session persistent
mechanisms in WebLogic for a shopping cart application, and monitoring the performance
results for comparison. First, you load the application using in-memory session replication to
persist sessions. Next, you load the application using file-based session persistence. Then you
run JDBC session persistence. You record and compare the performance results of each to
evaluate how well each performs. The following images depict the architecture of the domain
used for this practice.
In-memory session persistence: Session state is replicated in memory between duplicate

copies of the session on server1 and server2.

Chapter 6 - Page 11
File-based session persistence: Session state is persisted in a file store that is configured on
a shared file system that is accessible to server1 and server2.

Chapter 6 - Page 12
JDBC-based session persistence: Session state is persisted in a database that is running on

host02, and is accessible by server1 and server2.

your own.
Dependencies
Tasks
$ ./setup.sh
Deploys the ShoppingCart application to the cluster with in-memory session
replication configured

Chapter 6 - Page 13

3. Start Grinder for the ShoppingCart application.
The way you were running Grinder previously was specific for the Auction application. Now
that you are running it for the ShoppingCart application, you must run it from a new location.
This is to ensure that Grinder uses the proper grinder.properties file, which
references the cart.py script that was recorded using the ShoppingCart application.
a. Within the practice06-02 folder, execute the following command to run Grinder:
$ runGrinder.sh

b. Wait for the Grinder Console and Agent processes to start, and then continue with the
next step.
4. Review in-memory persistence, run Grinder, and record results.
a. Navigate to the WEB-INF folder and open the weblogic.xml file.
$ cd resources/ShoppingCart/WEB-INF
$ gedit weblogic.xml
b. Find the following configuration. This indicates WebLogic to replicate HTTP sessions to
a backup server if the application is deployed to a cluster.
<wls:session-descriptor>
<wls:persistent-store-type>
replicated_if_clustered
</wls:persistent-store-type>
</wls:session-descriptor>
c. Close the file.
d. Run Grinder to load the ShoppingCart application. Watch the results as it runs.
e. While Grinder is running, log in to the WebLogic administration console, select
Deployments > ShoppingCart > Monitoring > Web Applications, and review session
monitoring figures in the table. You can also explore the Sessions tab.
f. Draw a table similar to the following on paper and record the results:
Replication Mean Time Mean TPS Peak TPS Num Tests /
Type (ms) Errors
memory
file
JDBC
5. Configure file persistence, run Grinder, and record results.
a. Navigate back to the practice folder and execute the following script to switch the
weblogic.xml file for the ShoppingCart application. This file configures the
ShoppingCart application to use file session persistence:
$ cd ../../..
$ ./switch.sh -file
Removes any previously persisted session data (file or JDBC)

Chapter 6 - Page 14
Copies the resources/file/weblogic.xml file to the

resources/ShoppingCart/WEB-INF folder
Creates the ./sessions folder to store file-based sessions
Redeploys the ShoppingCart application
b. You can review the weblogic.xml file again if you would like to review the
configuration for file-based session persistence. It is shown here for convenience:
<wls:persistent-store-type>file</wls:persistent-store-type>
<wls:persistent-store-dir>

/practices/tune/practice06-01/sessions
</wls:persistent-store-dir>
c. Do a directory listing within the sessions folder and explore what you find.
d. When the load test completes, record the performance metrics in your comparison
table.
6. Configure JDBC persistence, run Grinder, and record results.
a. Execute the following script to switch the weblogic.xml file for the ShoppingCart
application. This file configures the ShoppingCart application to use JDBC session
persistence:
$ ./switch.sh -jdbc
Removes any previously persisted session data (file or JDBC)
Copies the resources/jdbc/weblogic.xml file to the
resources/ShoppingCart/WEB-INF folder
Creates the wl_servlet_sessions table to store sessions in the database
Redeploys the ShoppingCart application
b. You can review the weblogic.xml file again if you would like to review the
configuration for JDBC-based session persistence. It is shown here for convenience.
Although you are not using the Auction application for this practice, the same data
source is used for connecting to the database. The JNDI name of the data source is
irrelevant.
<wls:persistent-store-type>jdbc</wls:persistent-store-type>
<wls:persistent-data-source-jndi-name>
jdbc/AuctionDB
</wls:persistent-data-source-jndi-name>
c. On host02, execute the following code to verify that sessions are stored in the
database:
$ sqlplus oracle/Welcome1
SQL> select * from wl_servlet_sessions;
d. When the load test completes, record the performance metrics in your comparison
table.

Chapter 6 - Page 15
7. Experiment with the three session persistence configurations and Grinder on your own for
15 minutes. Use monitoring tools to view CPU, network I/O, and disk I/O during your tests.
Based on your results, try to answer the following questions. Use the following table as
reference for some different ways you can monitor system resources:
Command / Tool Description
top CPU and memory usage
sar -P ALL 1
sar -b 1 Monitor network I/O
sar -n DEV

sar -d 1 Monitor disk I/O
sar -q 1 Monitor run queue statistics
sar -w 1 Monitor context switching statistics
sar -S 1 Monitor swap space statistics
sar -r 1 Monitor free and used memory
a. Which session persistence type performed the best? Why?
b. Did you notice any fluctuations in CPU, network I/O, or disk I/O between each test?
Why do you think this is?
8. Explore other cluster-related performance settings.
a. Log in to the WebLogic administration console so you can explore other performance
settings related to WebLogic clustering (most are session related).
b. Server start up time:
Within the console, navigate to wlsadmin > Configuration > General.
Scroll to the bottom of the page and find the Enable on-demand deployment of
internal applications parameter. This setting causes internal applications, such as
the WebLogic administration console, to not be deployed during server start up.
Avoiding the deployment of internal applications speeds up the start up time for your
servers where these applications are deployed. However, the tradeoff is that the first
time the application is used, WebLogic takes the time to deploy the application so
you can use it.
c. Server start up time:
Within the console, navigate to Clusters > cluster1 > Configuration > General.
Click the Advanced node to show advanced settings.
Find the Member Warmup Timeout parameter. This parameter sets a time for
managed servers to wait for other managed servers to synchronize during
initialization. Generally, servers can sync with each other without requiring this wait
period. Setting it to 0 ensures the servers start up as fast as possible.
d. Replication performance:
Within the console, navigate to Clusters > cluster1 > Configuration > Replication.
Click the Advanced node to show advanced settings.

Chapter 6 - Page 16
Review the following table to learn more about these settings:

Setting Description
Cross-cluster Replication Both MAN and WAN clustering provide extra
Type session persistence resiliency, but at the cost of
increased complexity with configuration, and slower
performance. Typically, most customers do not use
MAN or WAN clustering.
Replication Channel Similar to how you can separate unicast cluster
messaging from regular network traffic, you can
separate replication traffic on its own network

channel. This is especially useful for applications
that have more intense replication requirements or
have faster network hardware, such as Exalogic
InfiniBand networking for internal traffic.
Secure Replication Similar to encryption of multicast messages, you
Enabled can encrypt replication messages. Again, this
comes with a potential performance decrease.
Inter-Cluster Comm Link This is the time interval that WebLogic servers in a
Health Check Interval cluster use to check whether a connection to
another server is available. Setting this value to a
lower value increases network traffic, while
decreasing connectivity reaction time.
Enable Replication Allows session replication network calls to timeout.
Timeout This is a good setting to enable to ensure that a
dead connection does not hold on to server
resources indefinitely.
Enable One Way RMI for This enables primary session servers to send
Replication requests to backup session servers using a one
way RMI call optimization. Replication performance
is increased because the primary server can
respond to the client with session data after
receiving the one way RMI call returns. This is
because the underlying network transport protocol
guarantees that session data is successfully
transmitted to the backup server. This avoids the
need for the primary server to wait for an
acknowledgment message from the backup server.
Enable Lazy Session Sessions are serialized and deserialized when they
Deserialization are passed over the network between the primary
and backup servers. The deserialization process is
more expensive than the serialization process.
When using WebLogic on Exalogic, and Exalogic
features are enabled, session data is not
deserialized by default on the backup server, which
avoids extra work and CPU utilization that is only
necessary when a server fails.

Chapter 6 - Page 17

environment down.
$ cleanup.sh
b. If any dialog windows appear asking if you want to close a terminal window, click OK to
close the window.
for managed servers remain, close them manually.
d. The Grinder Console and Agent applications should still be running. Shut them down
and close their terminal windows.

The primary focus of this practice is to get you used to analyzing performance related to the
different WebLogic HTTP session persistence mechanisms. Each mechanism has its own pros
and cons. Generally, in-memory replication provides the best performance in a high-volume
enterprise application. In a small environment like this practice environment, it is difficult to
measure a noticeable difference in performance between each run. This is mainly because the
environment is heavily CPU bound, and the application load test is doing more than just
persisting sessions. Session activity for these tests involves the initial session creation, reading
the session from memory, and writing the session data in the persisted medium whenever there
is an update to session data.
Performance differences occur when sessions are initially created and updated. With in-memory
replication, this is (by default) a synchronous call from the primary server to the backup server
to establish the in-memory backup. This involves serialization and deserialization over the
network. With file persistence, this involves a disk write to a network mounted disk. With JDBC
persistence, this involves a JDBC network call and a disk write to the database table. The
ShoppingCart application load test involves a number of updates to the session as new items
are added to the shopping cart. Even so, the amount of session activity is still difficult to show a
big performance difference in this environment. The biggest difference you likely experienced
was that response time was fastest with memory persistence.

Chapter 6 - Page 18
Practice Solution

Chapter 6 - Page 19

Chapter 6 - Page 20

Chapter 7 - Page 1
Chapter 7

Tuning JDBC Performance

Practices Overview
A web application (SimpleAuctionDbSLSB_EAR), by design performs a lot of database
interaction, seems to be running slow. WebLogic Server creates connections as and when
required. Every JDBC query that is executed is compiled and then executed.
Whenever a new connection is needed, WebLogic Server creates a new connection by
accessing the database. In production environments, this can be avoided by creating all these
connections during server start-up. Furthermore, JDBC queries can be cached to save precious
compile time. In addition, Pinned-To-Thread is an option that can improve performance by
enabling execute threads to keep a pooled database connection even after the application

closes the logical connection. This option can only be used if the back-end database is a single
instance database. In other words, Pinned-To-Thread cannot be used with Oracle RAC
databases. Another JDBC optimization technique that you experiment with is called batch
updates. Finally, you learn how to configure some basic Oracle RDBMS settings to allow for an
optimal interaction with Oracle WebLogic Server 12c.
During the course of these practices (7-1 to 7-3), you will use The Grinder to stress test the
SimpleAuctionDbSLSB_EAR application, modifying JDBC parameters between each test,
and you will record the result of each test in the following table:
Configuration Mean Succes TPS Peak Errors

Test sful TPS
Time Tests
(ms)
Connections in pool: Initial = 5 and Maximum
= 15
Connections in pool: Initial = 15 and
Maximum = 15
Statement caching enabled 15 statements
Statement cache: Type = LRU, Size = 10 and
Pinned-to-Thread, enabled
Statement cache: Type = LRU, Size = 35 and
Pinned-to-Thread, enabled

Chapter 7 - Page 2
The following diagram shows the stress test architecture based on The Grinder:

You will use the Administration Console to modify key performance-related JDBC parameters,
verifying the impact of each change at the end of each test.

Chapter 7 - Page 3
Practice 7-1: Tuning JDBC Connection Pools
Overview
In this practice, you run The Grinder to exercise the SimpleAuctionDbSLSB_EAR application.
The first run shows the impact of setting the JDBC parameters as follows:
Parameter Setting
Initial Capacity 5
Maximum Capacity 10
Minimum Capacity 5

This run represents the baseline against which subsequent runs are compared.
Tasks
a. Perform the following commands in a terminal window on host01 to set up the
environment for this practice:
$ ./setup.sh
Deploys the SimpleAuctionDbSLSB_EAR application to the cluster
3. Verify that the Oracle RDBMS is running. Select the VNC host02 window. Open one
terminal window; in the bash shell, make sure the database is running by verifying that the
pmon process for orcl is running:
$ ps ef | grep pmon
oracle 21491 1 0 Oct13 ? 00:01:10 ora_pmon_orcl
oracle 22816 14868 0 02:27 pts/2 00:00:00 grep pmon
a. Make sure the Oracle listener is running:
$ lsnrctl status
b. If the pmon_orcl process is not running on host02 or if the listener is not running, run
the startDB1.sh script:
$ startDB1.sh
LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 19-OCT-2013 03:05:18
Copyright (c) 1991, 2011, Oracle. All rights reserved.
Starting /u01/app/db11g/product/11.2.0/dbhome_1/bin/tnslsnr: please wait...
TNSLSNR for Linux: Version 11.2.0.3.0 - Production
System parameter file is
/u01/app/db11g/product/11.2.0/dbhome_1/network/admin/listener.ora

Chapter 7 - Page 4
SQL> ORACLE instance started.

Redo Buffers 7999488 bytes
Database mounted.
Database opened.
SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit
Production With the Partitioning, OLAP, Data Mining and Real Application Testing options
4. Select the VNC host01 window and make sure OHS is running.
$ cd /u01/app/fmw2/instances/webtier_1/bin
$ ./opmnctl status
---------------------+--------------------+---------+---------

---------------------+--------------------+---------+---------
a. If OHS is not running, the output from ./opmnctl status is different:
$ ./opmnctl status
opmnctl status: opmn is not running.
b. If OHS is not running, start it:
$ ./opmnctl start
$ ./opmnctl startproc ias-component=ohs1
opmnctl startproc: starting opmn managed processes...
$ ./opmnctl status
--------------------+--------------------+---------+---------
--------------------+--------------------+---------+---------
5. Log in to the WebLogic administration console.
a. Navigate to Environment > Servers and verify that all servers are in the RUNNING
state.
b. In the WebLogic administration console, verify that the SimpleAuctionDbSLSB_EAR
application is deployed and active:
c. Select Services > Data Sources and click the jdbc/AuctionDB link. Select the
Connection Pool tab and ensure the settings for Initial Capacity, Maximum Capacity,
and Minimum Capacity are 5, 10, and 5, respectively.
6. You must now start The Grinder. Open a new terminal and navigate to the practice
directory:
a. Source the setenv.sh script to define some Grinder environment variables and
launch the Grinder console as a background process (end the command line with an
ampersand [&] sign).
$ . ./setenv.sh
$ java net.grinder.Console &
b. The Grinder console starts. In the Ignore 0 samples field in the console, set the value
to 10 to give WebLogic time to tune its own performance based on the simulated load.

Chapter 7 - Page 5
Basically, this setting instructs the Grinder to skip sampling the initial ramp-up time. In
the terminal window where you launched the Grinder console, execute the prompt.sh
command to open a new terminal window. When the new window appears, set its title
to Grinder Agent.
c. Start the Grinder Agent:
$ java net.grinder.Grinder
2013-10-19 18:18:31,834 INFO agent: connected to console at localhost/127.0.0.1:6372
d. In the Grinder Console window, click the Start the worker processes icon to run the
test.

e. When the Grinder run starts, you can assess the status of the tests in the Grinder
Console window.
f. Monitor the Grinder Agent window to help you determine when the run finishes. You
are informed with the message below appearing at the bottom of the Grinder Agent
window:
2013-10-19 18:41:21,665 INFO agent: finished, waiting for
console signal
g. Select the Grinder Console window and click the Stop collecting statistics icon to stop
the collection of statistics.
h. On the right side of the Grinder Console window, click the Results tab and scroll to the
bottom to view statistic totals.
Your results will differ from the example above. Write down the significant statistics and
copy them into the table at the beginning of this practice (Practices Overview).
i. After you have copied the significant statistics to the table in the Practices Overview
section of this document, you must reset the worker processes. Click the Reset the
worker processes icon in the Grinder Console.
7. Modify the JDBC connection pool parameters before launching another Grinder run.
a. In the WebLogic administration console, select Services > Data sources >
jdbc/AuctionDB to display settings for the data source.
b. Click the Connection Pool tab. Scroll down until the parameters Initial Capacity,
Maximum Capacity, and Minimum Capacity are visible. Change their value to 15:
c. Save and activate your changes.

Chapter 7 - Page 6
8. Run Grinder again.

a. Select the Grinder Agent window. The last three lines should be:
2013-10-19 19:13:55,694 INFO agent: received a reset message
The agent console should report that a reset message was received, and that the
agent is ready to receive commands from the Grinder Console.
b. Select The Grinder Console window. Click the Start the worker processes icon to run a
new test.
c. Monitor the Grinder Agent window until you see the message, agent: finished, waiting

for console signal appear, meaning that the run has completed.
d. Click the Stop collecting statistics icon to stop the collection of statistics.
e. In the Grinder Console, click the Results tab, scroll down to the bottom of the window
and write down the totals for the last run. Copy those numbers in the table in the
Practices Overview section of this document.

Chapter 7 - Page 7
Practice Solution
solution. Practice 7-2 depends on this practice because it expects the connection pool
parameters Initial Capacity, Maximum Capacity, and Minimum Capacity to be set to 15.
Solution Tasks
$ ./solution.sh

Configures connection pool parameters for jdbc/AuctionDb

Chapter 7 - Page 8
Practice 7-2: Testing Performance of JDBC Statement Caching
Overview
In this practice, you increase the Statement Cache Size for JDBC connection pools in WebLogic
Server and you assess the impact of this modification using the Grinder.
Dependencies
Practice 7-1 has been completed. The WebLogic domain is running, the Admin Server and the
two managed servers are in a healthy state, the OHS is running and configured as a load
balancer, and the Grinder Console and the Grinder Agent are running.

Tasks
$ ./setup.sh
Deploys the applications to the cluster
Configures the work managers for this practice
2. Log in to the WebLogic administration console.
a. Navigate to Environment > Servers and verify that all servers are in the RUNNING
state.
b. Verify that the SimpleAuctionDbSLSB_EAR application is deployed and active.
c. Select Services > Data Sources > jdbc/AuctionDB > Connection Pool to navigate to
the data sources connection pool settings.
d. Scroll down until the Statement Cache Size parameter becomes visible. Change its
value to 15.
e. Save and activate your changes.
3. After having modified the JDBC Connection Pool parameters, you launch yet another The
Grinder run.
agent is ready to receive commands from The Grinder console.

Chapter 7 - Page 9
b. Select the Grinder Console window. Click the Start the worker processes icon to run a
new test.
c. Monitor the Grinder Agent window until the message, agent: finished, waiting for
console signal appears, meaning that the run has completed.
e. In the Grinder Console, click the Results tab, scroll down to the bottom of the window,
f. After you have copied the significant statistics to the table in the Practices Overview

Chapter 7 - Page 10
Practice Solution
solution. Practice 7-3 depends on this practice because it expects the connection pool
parameters Initial Capacity, Maximum Capacity, and Minimum Capacity to be set to 15 and
Statement Cache Size also to be set to 15.
Solution Tasks

$ ./solution.sh
Deploys applications for the domain

Chapter 7 - Page 11
Practice 7-3: Testing Performance of "Pinned-To-Thread"
Overview
In this practice, you enable the Pinned-To-Thread feature and run the Grinder twice. In the first
run, you enable a Statement Cache Size of 10 statements. In the second run, you increase the
Statement Cache Size to 35 and assess the impact of such a change.
Dependencies
Practices 7-1 and 7-2 have been completed. The WebLogic domain is running, the Admin
Server and the two managed servers are in a healthy state, OHS is running and configured as a
load balancer. The ORCL Oracle database instance is started and running and the listener is

accepting connections. Finally, the Grinder Console and the Grinder Agent are running.
Tasks
$ ./setup.sh
2. Configure Pinned-to-Thread.
a. Within the WebLogic administration console, navigate to Environment > Servers and
verify that all servers are in the RUNNING state.
b. Verify that the SimpleAuctionDbSLSB_EAR application is deployed and active.
c. Select Services > Data Sources > jdbc/AuctionDB > Connection Pool to display
settings for the data source connection pool.
d. Scroll down until the Statement Cache Size parameter becomes visible. Change its
value to 10 and save your changes.
e. Scroll back to the bottom of the Settings for jdbc/AuctionDB window and click the
Advanced link.
f. Select the Pinned-To-Thread check box.
g. Save and activate your changes.

Chapter 7 - Page 12
3. You must restart the two managed servers to realize your configuration changes.
a. Within the WebLogic administration console, select Environment > Servers > Control to
display the controls for managing servers in the domain.
b. Select the check boxes for server1 and server2
c. Click Shutdown and select Force Shutdown Now.
d. Wait for the servers to shut down.

e. Restart the servers.
4. Run the Grinder.
agent is ready to receive commands from The Grinder console.
new test.
c. Monitor the Grinder Agent window until the message, agent: finished, waiting for
console signal appears, meaning that the run has completed.
e. In the Grinder Console, click the Results tab, scroll down to the bottom of the window
5. Using the WebLogic administration console, change the Statement Cache Size parameter
to 35, before running another Grinder test.
a. Select Services > Data Sources > jdbc/AuctionDB > Connection Pool to display
settings for the data sources connection pool.
b. Scroll down until the Statement Cache Size parameter becomes visible. Change its
value to 35.
c. Save and activate your changes.
6. Initiate another Grinder run.
agent is ready to receive commands from the Grinder console.

Chapter 7 - Page 13
new test.
c. Monitor the Grinder Agent window until the agent: finished, waiting for console signal
message appears, meaning that the run has completed.
e. In the Grinder Console, click the Results tab, scroll down to the bottom of the window,
and write down the totals for the last run. Copy those numbers to the table in the

Given the hardware constraints of the lab computers, certain tests could counter-intuitively show
better performance when the opposite was expected. The Grinder runs with the connection pool
capacity set to 15 initial, 15 minimum and 15 maximum should show better results than the
connection pool capacity set to initial 5 and maximum 15. The Grinder runs with the connection
pool set to cache 15 statements should show a significant improvement. Do not expect a
significant difference between Grinder runs when the pool is set to Pinned-To-Thread. To
appreciate that difference, you would need a much more complex application, sending hundreds
of different SQL statements to the back-end engine.

Chapter 7 - Page 14
Practice Solution
solution. No practices depend on this practice.
Solution Tasks
$ ./solution.sh


Chapter 7 - Page 15
Practice 7-4: Configuring Batch Updates
Overview
In this practice, you deploy an application, which allows you to compare the timings of a
sequence of single insert statements against the timings of batch insertions/updates. The
alternative is between accepting the JDBC connection default, which is auto-commit enabled,
and disabling the auto-commit feature, sending a bunch of statements to the database engine
and commit only once, at the end.
This course is aimed primarily at WebLogic Server administrators, rather than developers.
However, the performance gains that can be achieved through this technique are such that the
administrator should be aware of it. Proactive WebLogic administrators should make sure that

developers in their teams use the appropriate means to obtain the maximum performance levels
that the WebLogic Server is capable of.
Assumptions
The WebLogic domain wlsadmin domain is running, the Admin Server and the two managed
servers are in a healthy state. The ORCL Oracle database instance is started and running and
the listener is accepting connections.
Tasks
$ ./setup.sh
3. For this practice, you create a new connection pool.
a. Within the WebLogic administration console, select Services > Data Sources to display
the configuration page for data sources.
b. Click New and select Generic Data Source.
c. Enter jdbc/BatchUpd in both the Name and JNDI Name fields.
d. Accept Oracle as the database type, and click Next.
e. Select Oracle's Driver (Thin) for Application Continuity; Versions: 11.2.0.3 and later,
and click Next.

Chapter 7 - Page 16
f. In the Create a New JDBC Data Source window, deselect Supports Global
Transactions, accept the other default values, and click Next.

g. The next window allows you to enter the database account information. Enter orcl in
the Database Name field, host02.example.com in the Host Name field. Accept the
default port number (1521), enter oracle for the Database User Name and Welcome1
as the password, and click Next.
h. The next window allows you to test the newly created connection. Click the Test
Configuration button.
i. If you omitted essential connection information, or entered a typo and the connection to
the Oracle instance cannot be established, the Administration Console reports
appropriate error diagnostics information in the Messages window. You can go back to
the previous screen and fix the faulty entries. After you obtain the Connection test
succeeded message, click Next.
j. In the Select Targets section of the Create a New JDBC Data Source window, select
server1 and server2, and click Finish.

Chapter 7 - Page 17
k. Active changes. The Administration Console signals that all changes have been
activated and no restart is necessary.
4. Deploy and rename the TestBatchUpdates application.
a. Use the WebLogic administration console to deploy the
/practices/tune/practice07-04/TestBatchUpdates.war application to
server1 and server2. Change the deployment name to BatchUpdates. By default, the
suggested name is TestBatchUpdates.

b. Erase the Test substring and complete the deployment process.
5. Run the BatchUpdates application.
a. Click the BatchUpdates link in the Deployments window. When the Settings for
BatchUpdates window appears, select the Testing tab.
b. Scroll toward the bottom of the window and click the URL that points to the
index.jsp file on server1
http://192.0.2.11:7011/BatchUpdates/index.jsp.
c. Select the second tab that appears in Firefox, named Testing Batch Updates.

Chapter 7 - Page 18
d. The Batch Update Application welcome page is displayed. The page shows two code
snippets that illustrate the different techniques used to submit a high number of insert
statements to the database engine. You can choose the number of insert statements
and the method used to submit them (either Batch Update or Single Update). Try the
Single Update option first.

e. The application displays the number of insert statements submitted to Oracle and how
many milliseconds it took to do so. Your number will obviously be different. Click the
Re-launch BatchUpdates page link at the bottom of the window.
f. This time, select the Batch Update radio button and click submit.
g. The Batch Update statistics should show a definite advantage in using batch updates
over a sequence of single updates.

Chapter 7 - Page 19
h. Feel free to repeat the test several times, varying the number of statements to be
submitted between tests. (Do not exceed 3000 statements to avoid running out of
resources.)

Chapter 7 - Page 20
Practice Solution
Solution Tasks
$ ./solution.sh

Configures a new connection pool named jdbc/BatchUpd
Deploys the TestBatchUpdates application to the cluster
5. You can resume starting at step 5.

Chapter 7 - Page 21
Practice 7-5: Tuning an Oracle Database for WebLogic
Overview
In this practice, you configure some basic Oracle RDBMS parameters, which are usually the
source of problems when a WebLogic server is accessed by a high number of users. Tuning an
Oracle RDBMS requires talent and experience, and is definitely outside the scope of this
course. However, the hints suggested here can be used as the first step when Oracle DBAs are
not readily available.
Tasks
1. Select the VNC host02 window.

a. Open one terminal window by clicking on the Terminal icon.
b. Using sqlplus, connect to the Oracle RDBMS as sysdba. Find out if Automatic
Memory Management (AMM) is enabled. (If the memory_target parameter is
different from zero, AMM is used.) Find out the size of the System Global Area (SGA).
/home/oracle>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Sun Oct 13 00:55:37
2013
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 -
64bit Production
SQL> select value from v$parameter where name = 'memory_target';
VALUE
-------------------------------------------------------------
524288000
SQL> show sga
Total System Global Area 521936896 bytes
Fixed Size 2229944 bytes
Variable Size 385878344 bytes
Database Buffers 125829120 bytes
Redo Buffers 7999488 bytes
c. The number of processes allowed by the Oracle RDBMS instance should be increased
to support a large number of users. The default is 150. You can find out the current
value accessing the V$PARAMETER fixed view.
SQL> select value from v$parameter where name ='processes';
VALUE
-------------------------------------------------------------
150

Chapter 7 - Page 22
2. Double the number of processes.

a. Use the ALTER SYSTEM command.
SQL> ALTER SYSTEM SET PROCESSES=300 SCOPE=SPFILE;
System altered.
SQL>
b. You specified SPFILE (System Parameter FILE) as the scope for the ALTER SYSTEM
command, which means that the parameter will take effect with next database startup.
You now want to configure the OPEN_CURSORS parameter; check its current settings
by selecting its value from V$PARAMETER.

SQL> select value from v$parameter where name ='open_cursors';
VALUE
-------------------------------------------------------------
300
c. Increase the number of open cursors to 600.
SQL> ALTER SYSTEM SET open_cursors=600 SCOPE=SPFILE;
System altered.
SQL>
3. Finding the adequate number of open cursors for your WebLogic environment is a
trial-and-error process. Sometimes the applications running in WebLogic have more
threads querying the database than available cursors in the supporting database. The
classical symptom of such an occurrence is the appearance of the exception:
java.sql.SQLException: - ORA-01000: maximum open cursors
exceeded
To prevent the above exception, increase the number of open cursors available at the
database instance level. Note, however, that a frequent and recurring ORA-01000
exception could be the indication that cursors or connections are not properly closed once
they have been requested. In other words, the applications are not closing ResultSets (in
JDBC) or cursors (in stored procedures on the database). Simply increasing
disproportionately the number of open cursor only delays (without fixing) the inevitable
application failure. Make sure that the developers in your WebLogic team are aware of
cursor leaks. They can be detected by using static code analyzers or appropriate JDBC
logging. Java 7 provides the very handy AutoCloseable interface. You should make sure
developers know about it and use it extensively.
4. Restart the Oracle RDBMS instance and make sure that the parameters have changed
according to the new values.
a. Leave sqlplus. At the OS prompt, invoke the stopDB1.sh shell script:
/home/oracle> stopDB1.sh
b. Use the startDB1.sh shell script to start the ORCL Oracle RDBMS instance again:
/home/oracle> startDB1.sh

Chapter 7 - Page 23
c. Use sqlplus to connect to the Oracle RDBMS as sysdba and verify that the
processes and open_cursors parameters have the new values:
/home/oracle>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Sun Oct 13 02:31:34
2013
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 -
64bit Production

With the Partitioning, OLAP, Data Mining and Real Application
Testing options
SQL> column name format a20
SQL> column value format a20
SQL> select name,value from v$parameter where name in
('processes','open_cursors');
NAME VALUE
-------------------- ------------------------
processes 300
open_cursors 600

Chapter 7 - Page 24
Practice Solution

Chapter 7 - Page 25

Chapter 7 - Page 26

Chapter 8 - Page 1
Chapter 8

Configuring Work Managers

Practices Overview
In the practices for this lesson, you experiment with work managers and how they affect
performance. First, you create and configure work managers for two instances of the Auction
application and run some load tests to see how the applications perform. Next, you deploy the
stuck threads application and experiment with configuring work manager thread constraints.
Then you configure WebLogic to gracefully manage stuck threading scenarios with work
managers.

Chapter 8 - Page 2
Practice 8-1: Creating and Using Work Managers
Overview
Each WebLogic Server instance uses a self-tuning thread pool to process all requests, and all
types of requests have the same level of service by default. Work managers enable
administrators to prioritize different services, applications, and application components by using
request classes and constraints.
The following diagram shows the environment that you will use to experiment with the work
manager feature. You will deploy multiple versions of the Auction application and stress test
both concurrently using the Grinder. You run Grinder tests before and after configuring work
managers.

This image depicts the architecture of the environment for this practice:
1. There are two instances of the database version of the SimpleAuction web application
deployed:
a. SimpleAuctionWebAppDb1: Accessed by context-root:
SimpleAuctionWebAppDb1
b. SimpleAuctionWebAppDb2: Accessed by context-root:
2. You test both versions of the application and verify that it is configured and working
correctly.
3. You configure two work managers to set different processing priorities for each application:
c. SimpleAuctionWebAppDb1: Set to a high priority request class
d. SimpleAuctionWebAppDb2: Set to a low priority request class

Chapter 8 - Page 3
4. You use deployment plans to assign a work manager to each application.

5. You run Grinder to place a load of requests on each application.
6. You review the results.

This practice provides some step by step detailed instructions, and in some cases only
guidelines for how to accomplish the objectives of the lesson. You take lessons learned from the
slides or previous practices and apply them for this practice on your own.
Dependencies

Tasks
$ ./setup.sh
Starts server1 and server2
state.
4. Verify that the two versions of the SimpleAuction application are deployed.
5. Test both SimpleAuctionWebAppDb applications using the current web browser to

ensure that the applications are working as expected:
a. Browse to http://host01:7777/SimpleAuctionWebAppDb1. You should see the
Auction application appear.
b. If you have already created the database, you should be able to click the View Auction
List link and successfully get a list of auctions returned.

Chapter 8 - Page 4
c. Repeat the same using the following URL:

http://host01:7777/SimpleAuctionWebAppDb2
6. Run Grinder to ascertain application performance before configuring work managers.
You are going to run two Grinder instances without using the Grinder console this time.
a. Close the Grinder console and agent if they are running.
b. Launch two terminal windows on host01 and ensure that you are in the current
practice folder.
c. Set the environment in each terminal window:
$ . ./setenv.sh
d. Start a Grinder agent in each terminal window at approximately the same time:

$ java net.grinder.Grinder grinder-db1.properties
e. Open the Grinder log files and find the total test results. Write the results in a table
similar to the following:
Before Work Manager Configuration:
Metric Mean Test Mean TPS Successful Errors
Time (ms) Tests
f. Keep these metrics available for comparison after running the work manager tests.
Also, be sure to keep the terminal windows open to run Grinder again.
7. Create global work managers.
Use the WebLogic administration console to create two work managers. Use the following
table to configure each with the proper settings. Work manager configuration is located in
the console at Environment > Work Managers. Edit each work manager and configure new
request classes for each.
Work Manager Target Request Fair Share
Name Class
HighPriorityWM cluster1 FairShare90 90
LowPriorityWM cluster1 FairShare10 10
8. Redeploy both Auction applications by using a deployment plan to assign a work
manager to each application.
a. Locate the deployment plan files at
/practices/tune/practice08-01/resources. Confirm that each refers to one
of your work managers. For example:
<variable>
<name>SimpleAuctionWeb_DispatchPolicy</name>
<value>HighPriorityWM</value>
</variable>
b. Perform the following commands to redeploy the applications by using the supplied
deployment plans:

Chapter 8 - Page 5
$ ./redeploy.sh
c. Test that each application has deployed successfully using a browser.
Note: This drives some traffic that causes each work manager to do some processing.
d. Return to the console.
e. In the Domain Structure panel, select Deployments.
f. Click the Monitoring > Workload tab.
g. Locate the Work Managers table. Use the Completed Requests column to confirm that
each application is linked to the correct work manager.

Note: You may need to use the Customize this table link to change the number of rows
to display to a number greater than 10 to see the same view as shown in the following.
Keep in mind that your individual request numbers may be different based on your
usage of the applications.

Chapter 8 - Page 6
9. Run Grinder to load the application again.

a. Run the Grinder agent in each terminal window at approximately the same time again:
b. While the client agents are running, return to the console and refresh the page to view
the work load. You should see that HighPriorityWM is getting a larger share of the
workload and LowPriorityWM is getting a lower share of the workload. Due to the
performance of the machine, this may not be totally accurate. The idea is for you to
learn how to tune work managers, even if the numbers do not support the features
exactly.
c. Wait for both agents to finish and exit.

d. Return to the console and once again verify that the requests to each application were
serviced by their dedicated work manager.
Note: Monitoring should be done while the requests are running. When both Grinder
agents are done running, the number of requests should be the same because each

Chapter 8 - Page 7
application was sent the same number of requests. However, while they are
processing WebLogic ensures that the high priority request class processes faster.
e. Open the Grinder log files and find the total test results. Write the results in a table
similar to the following:
After Work Manager Configuration:
Metric Mean Test Mean TPS Successful Errors
Time (ms) Tests

f. Compare these metrics with the metrics of your previous work manager tests. If the
tests results are inconclusive, it is feasible that there are fluctuations in the system
causing different results for every run. In a true QA environment, multiple performance
tests are run and averaged over time to establish proper performance metrics.
environment down and return the domain to its original state.
$ cleanup.sh
close the window.
c. You should notice that the AdminServer and all managed server terminal windows and
for managed servers remain, then close them manually.
d. You can leave the Grinder terminal windows open if you are continuing to the next
practice.

Chapter 8 - Page 8
Practice Solution
Solution Tasks
$ ./solution.sh

Configures work managers for the domain
5. Continue starting at step 4, and skip step 7.

Chapter 8 - Page 9
Practice 8-2: Configuring Work Manager Constraints
Overview
This practice shows you how to configure and monitor work manager thread constraints. You
configure one work manager to use a minimum threads constraint of 10, which causes
WebLogic to guarantee that 10 threads are executing simultaneously for configured
applications. You also configure one work manager to use a maximum threads constraint of 5,
which causes WebLogic to throttle the number of simultaneously executing threads to a
maximum of 5. This shows you two of the possible ways to control how WebLogic dispatches
requests to your applications. The following image depicts the architecture of the domain used
for this practice.

your own.
Dependencies
Tasks

Chapter 8 - Page 10

$ ./setup.sh
Configures the work managers for this practice

3. Verify that all servers started and that SimpleAuctionWebAppDb1 and
SimpleAuctionWebAppDb2 are deployed.
4. Verify that the MaxThreadsWM and MinThreadsWM work managers are created and
targeted to cluster1.
5. Test both applications to ensure that they are working correctly.
6. Create minimum and maximum thread constraints using the following table.
Work Manager Target Thread Thread Data Source
Name Constraint Constraint
Name Count
MinThreadsWM cluster1 MinThreads10 10 N/A
MaxThreadsWM cluster1 MaxThreads5 5 <blank>
7. Redeploy both Auction applications by using a deployment plan to assign a work
manager to each application.
/practices/tune/practice08-02/resources. Confirm that each refers to one
of your work managers.
deployment plans:
$ ./redeploy.sh
8. Test that each application has deployed successfully using a browser.
9. Locate the Work Managers table and verify that each application is linked to the correct
work manager.
10. Run Grinder to load the application.
a. Run the Grinder agent in each terminal window at approximately the same time:
11. While the client agents are running, return to the console and refresh the page to view the
work load. You should see that SimpleAuctionWebAppDb1 is using the minimum threads
constraint of 10, and it should be executing approximately 10 requests. You should also see
that SimpleAuctionWebAppDb2 is using the maximum threads constraint of 5, and it should

Chapter 8 - Page 11
be executing no more than 5 requests. After the Grinder agents stop loading the
application, these metrics will be set to 0, so you have to view the metrics while the
application is loaded by Grinder.
environment down.
$ cleanup.sh
close the window.

for managed servers remain, then close them manually.
d. The Grinder Console and Agent applications should still be running. Shut them down
and close their terminal windows.

The primary focus of this practice is to get you familiar with configuring and tuning work
manager constraints. There is a lot of technical information surrounding work manager
constraints. I could try to discuss the finer details of constraints in this section, but I found an
Oracle blog that does a really good job already. Perform the following commands to view the
blog entry:
$ cd resources/wm_blog
$ firefox Threads_Constraints_in_WM.html
The public URL for this blog in case you want to view it after class is:
https://blogs.oracle.com/WebLogicServer/entry/threads_constraints_in_work_ma

Chapter 8 - Page 12
Practice Solution
Solution Tasks
$ ./solution.sh

Configures work managers for the domain
5. Continue starting at step 3, and skip step 6.

Chapter 8 - Page 13

Chapter 8 - Page 14

Chapter 9 - Page 1
Chapter 9

Tuning J2EE Applications

Practices Overview
In these practices, you experiment with JSP and EJB settings, which can have a significant
impact on performance. Initially, you deploy the SimpleAuctionWebAppDb application without
precompiling its JSP pages and you submit a Grinder run; subsequently, you re-deploy the
same application, this time pre-compiling the JSP pages, and you compare the response time of
the two runs.
Similarly, in practice 9-2, you deploy the SimpleAuctionDbSLSB_EAR application, which
initially runs with the initial-beans-in-free-pool parameter set to 2 and the max-

beans-in-free-pool parameter set to 4. You run The Grinder using this configuration. Then
you redeploy the application, this time increasing initial-beans-in-free-pool to 15 and
max-beans-in-free-pool to 30. After another The Grinder run, you compare the statistics
accumulated over the two runs.

Chapter 9 - Page 2
Practice 9-1: Tuning JSP
Overview
In this practice, you use The Grinder twice, initially to stress test the
SimpleAuctionWebAppDb application, which has been deployed with no precompiled JSP
pages. Subsequently, you run The Grinder again, after having re-deployed the application with
all JSP pages precompiled. You populate the following table at each run, and you compare the
results at the end of the practice.
Configuration Mean Succes TPS Peak Errors

Test sful TPS
Time Tests
(ms)
SimpleAuctionWebAppDb (JSP precompile
= NO)
SimpleAuctionWebAppDb (JSP precompile
= YES)
Dependencies
Tasks
$ ./setup.sh
3. Select the VNC host02 window.
a. Open one terminal window by clicking the Terminal icon.
b. In the bash shell, make sure that the database is running by verifying that the pmon
process for orcl is running:
/home/oracle> ps ef | grep pmon
oracle 21491 1 0 Oct13 ? 00:01:10 ora_pmon_orcl
oracle 22816 14868 0 02:27 pts/2 00:00:00 grep pmon

Chapter 9 - Page 3
c. Make sure that the Oracle listener is running. Enter the lsnrctl command "status:"
/home/oracle> lsnrctl status
d. If the pmon_orcl process is not running on host02 or if the listener is not running,
run the start db script:
/home/oracle>startDB1.sh
e. Select the VNC host01 window and make sure that OHS is running:
/home/oracle> cd /u01/app/fmw2/instances/webtier_1/bin
/u01/app/fmw2/instances/webtier_1/bin>./opmnctl status

---------------------+--------------------+---------+---------
---------------------+--------------------+---------+---------
f. If OHS is not running, the output from ./opmnctl status is different:
opmnctl status: opmn is not running.
g. If OHS is not running, start it:
/u01/app/fmw2/instances/webtier_1/bin>./opmnctl start
/u01/app/fmw2/instances/webtier_1/bin>./opmnctl startproc ias-
component=ohs1
opmnctl startproc: starting opmn managed processes...
--------------------+--------------------+---------+---------
--------------------+--------------------+---------+---------
h. Access the WebLogic administration console and make sure that the application
SimpleAuctionWebAppDb is deployed, active, and with a health state of "OK."
4. You must now start The Grinder. Open a new terminal and change directory to the default
directory for the practice:
/home/oracle> cd /practices/tune/practice09-01
/practices/tune/practice09-01>
a. Source the setenv.sh script to define some Grinder environment variables and
launch the Grinder console as a background process. (End the command line with an
ampersand [&] sign.)
/practices/tune/practice09-01> . ./setenv.sh
/practices/tune/practice09-01> java net.grinder.Console &

Chapter 9 - Page 4
b. When The Grinder console starts, in the "Ignore 0 samples" field in the console,
set the value to 10 to give WebLogic time to tune its own performance based on the
simulated load. Basically, this setting instructs The Grinder to skip sampling the initial
ramp-up time.
c. In the Terminal window, where you launched The Grinder console, enter the
"prompt.sh" command to open an additional Terminal window. When the new
window appears, click the "Terminal"->"Set Title" menu option to name the
window caption "Grinder Agent".
d. Start The Grinder Agent:
/practices/tune/practice09-01> java net.grinder.Grinder

e. In the Grinder Console window, click the "Start the worker processes" icon to
run the test. The Grinder responds with an alert window, stating that the worker
processes will run taking their parameters from the "grinder.properties" file. Click
"OK" to proceed.
f. The Grinder run starts, you can assess the Grinder feedback in the Grinder Console
window:
5. You should monitor the Grinder Agent window every now and then. When the run finishes,
you get the following message, which appears at the bottom of the Grinder Agent window:
2013-10-19 18:41:21,665 INFO agent: finished, waiting for console signal
a. Select The Grinder Console window and click the "Stop collecting statistics"
icon to stop the collection of statistics.

Chapter 9 - Page 5
b. On the right side of the Grinder Console window, click the Results tab.
c. Scroll to the bottom of the "Results" tab.
(Your results will no doubt differ from the preceding example.) Write down the significant
statistics and copy them into the table at the beginning of this practice (Practice 9-1
Overview).
d. After you have copied the significant statistics to the table in the "Practice 9-1
Overview" section of this practice, you must reset the worker processes. Click the
Reset the worker processes" icon in the Grinder Console.

6. You must re-package the SimpleAuctionWebAppDb application, this time asking to have
all *.jsp files precompiled. Invoke the Oracle Enterprise Pack for Eclipse (OEPE) IDE
(which is based on the Eclipse IDE). In your environment, OEPE is located in
/u01/app/oepe:
a. Open one terminal window by clicking the Terminal icon.
b. Change the directory to /u01/app/oepe and launch the eclipse IDE.
/home/oracle> cd /u01/app/oepe
/u01/app/oepe> ./eclipse
c. Eclipse "Kepler" starts. Wait until Eclipse is completely up and running. Select the
"Package Explorer" panel and then click the "SimpleAuctionWebAppDb" package.
Right-click SimpleAuctionWebAppDb and select Export:
d. Select WAR file and click Next.

Chapter 9 - Page 6
e. Accept the default "SimpleAuctionWebAppDb" for "Web project." Specify the

absolute path name: "/practices/tune/practice09-
01/SimpleAuctionWebAppDb.war" in the "Destination" field. Select the "Pre-
compile JSP files" check box. Also, select the "Overwrite existing file" check box and
click Finish.

f. Exit the OEPE IDE. Click "File"->"Exit". When the shell prompt appears, change the
directory to "/practices/tune/practice09-01." Verify that the
"SimpleAuctionWebAppDb.war" file exists, and that its time stamp is very recent
(that is, has been just created).
/u01/app/oepe> cd /practices/tune/practice09-01
/practices/tune/practice09-01> ls l SimpleAuctionWebAppDb.war
-rw-r--r-- 1 oracle oinstall 989176 Oct 27 2013 SimpleAuctionWebAppDb.war
7. You must now undeploy the current SimpleAuctionWebAppDb application and redeploy
the version you just modified, which is packaged with precompiled JSP files.
a. Select the WebLogic Administration Console window. Force stop the
"SimpleAuctionWebAppDb" deployment, and then delete it.
b. After the Administration Console acknowledges the deployment deletion, in "Change
Center," click "Activate Changes" to persist the modifications made.
8. You must redeploy the "SimpleAuctionWebAppDb" application. This time you deploy the
version you modified, which uses precompiled JSPs.
a. In Change Center, click the Lock & Edit" button. Deploy "SimpleAuctionWebAppDb"
selecting the WAR file stored in "/practices/tune/practice09-01."
b. Deploy the application to the entire cluster.
c. Click the Activate Changes button.
d. Start the "SimpleAuctionWebAppDb" deployment servicing all requests.
9. You must now run The Grinder against the newly deployed application.
a. Select the "Grinder Agent" window. The last three lines should be:

Chapter 9 - Page 7
The agent console should report that a reset message was received, and that the agent is
ready to receive commands from The Grinder console.
b. Select "The Grinder Console" window. Click the "Start the worker processes" icon to
run a new test.

Click "OK" to proceed with the test when the "Start Processes" dialog box appears.
Monitor the "Grinder Agent" window, waiting for the message "agent: finished,
waiting for console signal" to appear, which means that the run has completed.
c. Click the "Stop collecting statistics" icon to stop the collection of statistics.
In "The Grinder Console", click the "Results" tab, scroll down to the bottom of the
window and write down the totals for the last run. Copy those numbers to the table in the
"Practice 9-1 Overview" section of this practice.
10. Compare the values in the table (Practice 9-1 Overview).

Chapter 9 - Page 8
Practice Solution
Solution Tasks
$ ./solution.sh

5. Continue starting at step 3.

Chapter 9 - Page 9
Practice 9-2: Tuning Stateless EJBs
Overview
In this practice, you run The Grinder stress testing the SimpleAuctionDbSLSB_EAR
application initially configured to have its EJBs set up with minimal pools. You then redeploy the
same application, this time increasing the pool size for stateless EJBs and you run The Grinder
again, to see if the parameter change made a difference in the performance observed across
the two runs.
Configuration Mean Success TPS Peak Errors

Test ful Tests TPS
Time
(ms)
SimpleAuctionDbSLSB_EAR
initial-beans-in-free-pool = 2
max-beans-in-free-pool = 4
SimpleAuctionDbSLSB_EAR
initial-beans-in-free-pool = 15
max-beans-in-free-pool = 30
Dependencies
Tasks
$ ./setup.sh
state.
c. Navigate to Deployments. Make sure that the SimpleAuctionDbSLSB_EAR
application is deployed, active, and with a health state of "OK."

Chapter 9 - Page 10
d. Expand the SimpleAuctionDbSLSB_EAR deployment and locate the

AuctionService EJB. Click its link:

e. Click the "Monitoring" tab. Make sure that "Pooled Beans Current Count" is set
to 2:
4. Run the Grinder using this configuration.

a. Open a new terminal and change directory to the default directory for the practice.
/home/oracle> cd /practices/tune/practice09-02
/practices/tune/practice09-02>
b. Source the setenv.sh script to define some Grinder environment variables and
launch the Grinder console as a background process (end the command line with an
ampersand [&] sign).
/practices/tune/practice09-02> . ./setenv.sh
/practices/tune/practice09-02> java net.grinder.Console &
c. Open another terminal window and Start The Grinder Agent:
/practices/tune/practice09-01> java net.grinder.Grinder
d. Perform the run and at the end, write down the statistics accumulated during the run.

Chapter 9 - Page 11
5. Redeploy the SimpleAuctionDbSLSB_EAR application by using a deployment plan to

change the initial and max beans in free pool.
/practices/tune/practice09-02/resources. Edit the ejb-plan.xml file and
confirm that initial-beans-in-free-pool and max-beans-in-free-pool
have changed their settings to 15 and 30, respectively.
deployment plans:
$ ./redeploy.sh

c. After the application has been redeployed, use the Administration Console to make
sure that the deployment was successful. Expand the deployment, locate the
AuctionService EJB and click its link. Click the "Monitoring" tab and make sure that
"Pooled Beans Current Count" is now set to 15.
6. Re-run The Grinder using this configuration. At the end of the run, write down the
accumulated statistics and compare them with the previous The Grinder run.

Chapter 9 - Page 12
Practice Solution
Solution Tasks
$ ./solution.sh

Deploys the SimpleAuctionDbSLSB_EAR application to the cluster configured
with minimal initial and max beans in free pool
5. Continue starting at step 3.

Chapter 9 - Page 13

Chapter 9 - Page 14

Appendix A: Convenience
Cheat Sheet
Chapter 10
Using Convenience Scripts and Aliases in this Course

Chapter 10 - Page 1

This is a cheat sheet for using the convenience scripts and aliases that are built in to the
environment. Using these shortcuts makes using the lab environment easier. These aliases are
all defined in the /home/oracle/.bashrc file, and are initialized automatically when you
open a new terminal window after you set the initial environment.
Shortcut Type Description

install Navigation Navigates to the $INSTALLDIR folder:
cd /install
app Navigation Navigates to the $APP folder:

cd /u01/app
apps Navigation Navigates to the $LABHOME/tune/apps folder:
cd /practices/tune/apps
labs Navigation Navigates to the $LABHOME/tune folder:
cd /practices/tune
domains Navigation Navigates to the domains folder:
cd /u01/domains/tune
ohome Navigation Navigates to the $ORACLE_HOME folder:
cd $ORACLE_HOME
fmw Navigation Navigates to the Fusion Middleware installation folder
(MW_HOME):
cd /u01/app/fmw
wls Navigation Navigates to the WebLogic Server product folder:
cd $MW_HOME/wlserver
ide Navigation Navigates to the OEPE/Eclipse folder:
cd $APP/oepe
ohs Navigation Navigates to the OHS/webtier_1 folder:
cd $APP/fmw2/instances/webtier_1
bin Navigation Navigates to the lab bin folder:
cd $LABHOME/tune/bin
wlsd Navigation Navigates to the wlsadmin domain folder:
cd /u01/domains/tune/wlsadmin
23 Navigation Navigates to the practice02-03 folder:
cd $LABHOME/tune/practice02-03

Chapter 10 - Page 2


g Convenience Alias for gedit
wlst Convenience Alias for java weblogic.WLST
cleanup.sh Convenience Kills all java servers related to WebLogic on both hosts, and resets
the initial domain for the course on both hosts.
startohs.sh Convenience Starts the OHS web server from any location
stopohs.sh Convenience Stops the OHS web server from any location
bin/backup/ Convenience Creates and populates the Auction database
BuildAuctionDatabase
.sh

Chapter 10 - Page 3

Chapter 10 - Page 4

D83171GC10 Ag

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

D83171GC10 Ag

Uploaded by

Copyright:

Available Formats

THESE eKIT MATERIALS ARE FOR YOUR USE IN THIS CLASSROOM ONLY.

COPYING eKIT MATERIALS FROM THIS COMPUTER IS STRICTLY PROHIBITED

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Restricted Rights Notice

Oracle University and In Motion Servicios S.A. use only

U.S. GOVERNMENT RIGHTS

Technical Contributors and Reviewers

This book was published using: Oracle Tutor

Practices for Lesson 1: Course Introduction ........................................................................................... 1-1

Practices for Lesson 2: Performance Tuning - Overview ........................................................................ 2-1

Oracle University and In Motion Servicios S.A. use only

Practices for Lesson 3: Tuning Operating System Resources ................................................................ 3-1

Practices for Lesson 4: Tuning HotSpot JVM .......................................................................................... 4-1

Oracle WebLogic Server 12c: Performance Tuning Workshop Table of Contents

Practice Solution ..................................................................................................................................... 5-21

Practices for Lesson 6: Monitor and Tune Cluster Performance............................................................. 6-1

Practices for Lesson 7: Tuning JDBC Performance ................................................................................ 7-1

Oracle University and In Motion Servicios S.A. use only

Practices for Lesson 8: Configuring Work Managers .............................................................................. 8-1

Practices for Lesson 9: Tuning J2EE Applications.................................................................................. 9-1

Appendix A: Convenience Cheat Sheet ................................................................................................... 10-1

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Oracle WebLogic Server 12c: Performance Tuning Workshop Table of Contents

Practices for Lesson 1

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Oracle University and In Motion Servicios S.A. use only

Practices for Lesson 1

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 1

Practices for Lesson 2: Overview

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Oracle University and In Motion Servicios S.A. use only

Practices for Lesson 2: Overview

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practices for Lesson 2: Overview

Practice 2-1: Setting Up the Course Practice Environment

Oracle University and In Motion Servicios S.A. use only

Enter oracle as the password.

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practice 2-1: Setting Up the Course Practice Environment

b. Connect via VNC to host02:

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practice 2-1: Setting Up the Course Practice Environment

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practice 2-1: Setting Up the Course Practice Environment

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practice 2-1: Setting Up the Course Practice Environment

Oracle University and In Motion Servicios S.A. use only

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practice 2-2: Exploring the Course Applications

The Auction Application:

Oracle University and In Motion Servicios S.A. use only

Application deployment package:

Copyright 2014, Oracle and/or its affiliates. All rights reserved.

Practice 2-2: Exploring the Course Applications

Auction Use Cases:

Oracle University and In Motion Servicios S.A. use only