<Insert Picture Here>

Dec 2008 Content Server Performance Tuning

Cordell Melgaard
Principal Technical Support Engineer
Global Customer Service

Content Management
Webinar

Jason Ganske
Principal Technical Support Engineer
Global Customer Service
Welcome Content Management Customers!

Design Benchmarks

Run Benchmarks

Analyze Benchmarks

Adjust the core configuration

Adjust add-on configuration


Five Steps to a Faster Server
General Issues

HELP!! My Content Server is Slow
Searching
Batchloading / Contribution
Page Rendering (HSCP, JSP, ASP)
Publishing

Tuning is the same as any web application
Adjust configuration flags for your specific needs
Optimize Queries
Maximize hardware
Cache! Cache! Cache!
Underlying Problem Areas

All web applications have similar problems
File system: too small or too slow
Bandwidth: too slow to clients or database
Processor: too small or overused
Memory: too little or underused
Database queries are not optimized
Cache: too small, underused, or ignored

Whats the solution?
Theres a reason its called tuning
There is no magic button to fix performance problems


Five Steps to a Faster Server

1. Design Benchmarks

2. Run Benchmarks

3. Analyze Benchmarks

4. Adjust the core configuration

5. Adjust add-on configuration


Gather setup and usage information
Speed is Relative
Need something to compare against
1) Design Benchmarks - Intro
1) Design Benchmarks - Intro

Too few make benchmarks, but its vital to:
Quantify performance
Clean-room vs. real-world
Avoid side-effects
Avoid fixing the wrong problem

Basic benchmarks can be quick and painless

Thorough benchmarks can be slow and painful


1) Design Benchmark Initial Steps

Document and verify setup
Hardware: CPUs, memory, manufacturer
Software: database, web server, CMS version
Network: firewalls, routers, switches, bandwidth
Security: LDAP, Active Directory, SSO

Define where its slow
Searching, Contributing, Viewing

Check the Server Output for errors
Logging even simple errors slows down the system
1) Design Benchmarks Web and
Batch Tests

Design a web script for a typical users session
Navigate to common pages
Site Studio, Publisher, or standard templates
Submit some forms
Search, update, workflow, check in
Recreate a typical workload for a user


1) Design Benchmark - Pitfalls

Make benchmark as real-world as possible
Use the production server, or
Use an exact replica of production server

Isolate benchmark server
Only allow testers to access
Isolation required to quantify performance changes
but isolation does not resemble real-world

Document Document Document
Every test should be well documented to find consistency
Five Steps to a Faster Server

1. Design Benchmarks

2. Run Benchmarks

3. Analyze Benchmarks

4. Adjust the core configuration

5. Adjust add-on configuration

Tools, execution, logging
2) Run Benchmarks General Tools

Free and Simple
Content Server Health Monitor
Content Server System Audit Page
Process Explorer
taskmgr (Win32) or top (Unix)

Complex
Mercury LoadRunner
Java Profilers
Perl / Python scripts
PamIE, SamIE, WWW::Mechanize
2) Run Benchmarks Specific Tools

Database performance analysis tools
HUNDREDS of tools available
Check with your DBA

Apache mod_log_config
Customize logs to record page response time
Prioritize effort on popular pages

Network analysis
Ethereal/Wireshark
trafshow
tracert / traceroute
2) Run Benchmarks Setup and
Execution

Number of users for web benchmark
Concurrent = 1 to 10% of total users
Peak usage = 10 to 80% weekly users

Monitor all processes on all servers
CPU and RAM usage, IO Reads
Content Server: java, mkvdk, htmlexport
Database: oracle, mssql, etc.

2) Run Benchmarks Log Files

Web log files
Refinery, Archiver, and Server logs
Filter Administration for security filter logs

Tracing output
oracle/bin/IdcServerNT.log (Windows)
oracle/etc/log (Unix)
System Audit Info tracing sections:
indexer, pagecreation, requestaudit
socketrequests, searchquery
systemdatabase, userstorage
2) Run Benchmarks Pitfalls

Adjustments
Best-case: turn off logging, dont allow contribution
Bad-case: turn off search cache
Worst-case: run batch and web tests simultaneously

Double-check resource usage
Is a virus scanner running?
Is anybody else accessing the server?
Who else is using CS dependent resources?
File server, database, web server, LDAP/AD/NTLM
Five Steps to a Faster Server

1. Design Benchmarks

2. Run Benchmarks

3. Analyze Benchmarks

4. Adjust the core configuration

5. Adjust add-on configuration



Find possible bottlenecks and retest
3) Analyze Benchmarks

Review benchmarks
Read, Analyze, Read, Analyze
Time required to run batch tests
Requests per second per user for web tests

What is slow?

EXAMPLE goals for business content:
Read-only requests: 20 per second per GHz CPU
Raw Verity queries: 4 per second per GHz CPU
Contribution: 4 per second per GHz CPU
Assume: 100k files and 1GB RAM
3) Analyze Benchmarks Find the
bottleneck

Where is the bottleneck?
Search engine / indexer
Database
Network
Java VM (Content Server, Refinery, add-ons)
File system
Web Security Filter

First benchmarks offer hints
Need verification
Enable more tracing sections, and retest
3) Analyze Benchmarks Search or
Indexer Bottleneck Clues

Template pages are fast
Simple queries take over one second
Navigation pages execute highly complex queries
Analyze requestaudit & searchquery trace
The system has a complex security model
Search cache isnt utilized
System Audit Info: small cache, few hits
Verity collection is not local
Slow network file system?
Database queries not optimized when using DB search

3) Analyze Benchmarks Database
Bottleneck Clues

System Audit Info page
Active and waiting connections
Dropped connection errors
Too many connection errors
Most of request time is spent in database
Database CPU is overworked
SCS and MKVDK underworked

Hint:
Monitor searchquery tracing for timestamps on queries
Watch standard web logs for connection errors


Common Error

An error has occurred. The stack trace below shows more
information. !$gpescm2: !csIndexerAbortedMsg
intradoc.common.ServiceException:
!csIndexerUnableToBuildChanges!csNoConnectionsAvailable,Sys
temDatabase
at intradoc.indexer.IndexerBuildChanges.buildChanges(Unknown
Source)
at intradoc.indexer.IndexerBuildChanges.doWork(Unknown Source)
at intradoc.indexer.Indexer.doIndexing(Unknown Source)
at intradoc.indexer.Indexer.buildIndex(Unknown Source)
at intradoc.server.IndexerMonitor.doIndexing(Unknown Source)
at intradoc.server.IndexerMonitor$4.run(Unknown Source)

3) Analyze Benchmarks Network
Bottleneck Clues

Remote connections (databases) are slow
but remote services arent working very hard
Use trafshow or Ethereal/Wireshark to determine saturation

Hint:
Monitor searchquery tracing for timestamps on queries
Compare with running the query directly on the database


3) Analyze Benchmarks Core Engine
/ Java VM Bottleneck Clues

The Stellent engine runs pretty lean, but
Publishing or page rendering can be slow

It may be a Java VM problem if:
Its a consumption server
It renders a lot of HCSPs or JSPs
It runs a lot of workflows or subscriptions
Java VM uses all the CPU or Memory
Even if searching isnt performed
Multiple CPUs not being fully utilized
Using outdated JVM
3) Analyze Benchmarks File System
Bottleneck Clues

File system is remote, but not high-performance
SAN: Very fast, expensive, long setup
NAS: Slower, cheaper, easy setup
Larger than expected file activity
Monitor I/O reads, and I/O read Bytes
File system is nearing capacity

Hint:
Note: some remote file system problems indicate an
underlying network problem

FAST
RELIABLE
CHEAP

PICK TWO
3) File System Architecture
Performance

Location
Web
Server
File
System
Database
Fastest








Slowest
Local Local Local
Local Local Remote
Local SAN Remote
Local NAS Remote
Local Remote Remote
Remote Local Remote
Remote SAN Remote
Remote NAS Remote
Remote Remote Remote
3) Analyze Benchmarks Web Security
Filter Bottleneck Clues

Users first login takes several seconds
Subsequent page delivery is fast
Many users login as their browsers home page
Pay login cost, but dont navigate further
Unmanaged content is much faster
Use only one thread to test

Hint:
Monitor verbose userstorage trace for timestamps
Note: stack traces are not always errors in this section


3) Analyze Benchmarks - Pitfalls

Java profiler reports misleading

Logging and virus scanners slow systems down

Try to establish real-world benchmarks
No search cache, do contribution and consumption

Watch the search engine carefully
Note the frequency and complexity of queries
Substring queries KILL performance

Five Steps to a Faster Server

1. Design Benchmarks

2. Run Benchmarks

3. Analyze Benchmarks

4. Adjust the core configuration

5. Adjust add-on configuration



Configuration, caching, database, and verity
4) Adjust Core

Easy:
Configuration flags
Java VM tuning

Difficult:
IdocScript caching and tuning
Simplify security model

Harder:
Verity Index Tuning
Database Indexes and tuning (OracleQueryOptimizer)
4) Adjust Core Config Flags

Indexer (Verity)
AdditionalIndexBuildParams, DoAutoMaxMerge,
MaxMergeBaseCount, TimeoutPerOneMegInSec
Search
MaxSearchConnections, CachedResultRowCount
Database
NumConnections
Only increase for testing can have opposite effect
General
UserCacheTimeout, IdcServerSocketQueueDepth

4) Adjust Core Java Tuning

JVM is self-tuning, but benefits from hints
Modify values in intradoc.cfg
Optimize memory and compiler
MetaLink Note : Note 460039.1
JAVA_MAX_HEAP_SIZE=[Size in megabytes]
Multiple CPU systems
Use JVM 1.5.1 or better
Tune GC for multiple CPUs, platform specific
Different tuning options for CS versions

4) Adjust Core IdocScript Tuning

Simplify pages
Some pages needlessly do multiple requests
Reduce number and complexity of queries
Avoid queries that dont cache, such as timestamps

Resolve simple errors
CS Log will constantly log IdocScript errors unnecessarily

Use database search for exact matches
SearchEngineName=DATABASE.METADATA
QueryText=dDocType LIKE ADACCT

4) Adjust Core IdocScript Caching

cacheInclude
Include a dynamichtml fragment from a cache
For fragments that are used multiple times
Global cache, named cache, or session cache

setMaxAge
Uses Cache-Control HTTP header for page caching
For pages that one uses hits frequently

4) Adjust Core Simplify security

Searches attach security clause
The more complex the security, the longer the query
In general, longer queries take longer to run
Worst performance will be from users with mixed security

Monitor searchquery trace in System Audit

User input:
dDocName <matches> `test`

Actual query run:
(dDocName <matches> `test`) <and>
(dSecurityGroup <not> `Secure`)


4) Adjust Core Verity Tuning

Tune and houseclean the Verity collection
MKVDK command-line options
Leverage zone search on common fields
Avoid automatically adding every field to index
Tune the number of parts file
Many parts, fast search; Few parts, fast index
Put commonly searched fields into separate parts
Security group, title, author
Plan ahead!
Adding metadata fields requires an index rebuild
Use dummy or non-indexed fields, or database search
4) Adjust Core Database Tuning

OracleQueryOptimizer
Free component for tuning Oracle DB queries
Add extra indexes to commonly searched columns
Update database statistics as a database grows
Before & after inserting lots of content
Maintains indexes, helps query optimizer
Optimize row-locking algorithm
ORACLE: optimizer_mode = first_rows
Consider custom services or queries
Database-specific queries for better performance
Simplify check-ins when running batches


4) Adjust Core - Pitfalls

Be sure to re-run the benchmarks
Record new performance data
Watch for side-effects!

Test server improvements misleading
Production server will not see exact benefits
unless the benchmark was 100% real-world
Five Steps to a Faster Server
1. Design Benchmarks

2. Run Benchmarks

3. Analyze Benchmarks

4. Adjust the core configuration

5. Adjust add-on configuration


Conversion, publishing, J2EE, and other extras

Batchloader
Dynamic Converter
Site Studio
PDF Converter
Content Tracker
CIS
5) Adjust Add-ons
5) Adjust Add-ons - Batchloader

Temporarily disable other activity
Inbound Refinery / PDF Converter
Auto update indexer cycle Repository Manager
Optimize the search collection (if using Verity)
prior to batch insert.
Analyze your database usage during a
batchload. Goal is to help the database query
optimizer.
Oracle: ANALYZE TABLE COMPUTE STATISTICS
SQL Server: CREATE STATISTICS
DB2: RUNSTATS
5) Adjust Add-ons - Dynamic Converter
Create Template Selection Rules
Predetermines the conversion and layout templates
Upon submission, content is converted and cached.
Cache location:
<stellent_dir>\weblayout\groups\<security_group>\documents\<typ
e>\~export\<content id>~<rev>~<template name>~<layout name>
Dynamic Converter enables on-demand, ad-hoc publishing of native business
documents (255 formats) to HTML, XML or wireless formats.
5) Adjust Add-ons - Site Studio
Analyze page architecture
Turn on tracing and watch the output
Search, Database, Script Errors
Optimize the queries in dynamic lists
Simple
Operators (Matches)
Indexed
Avoid using time and Substring


Site Studio automates Web site creation and contribution methodologies.
Additionally, Site Studio enables business users to contribute content in a user-
friendly WYSIWYG form-based environment with regional-level security and in-
context preview, workflow, and approval.
5) Adjust Add-ons - Site Studio (cont.)

Use front side caching
Reverse proxy server
Use IdocScript caching
Optimize services executed on pages
Consolidate actions. Remove unnecessary work/data.
Client-side vs. server-side page construction
e.g. out-of-the-box fragments use client-side mechanisms to
build navigation.
Static lists vs. dynamic lists
Static Lists XML content
Dynamic Lists Search Queries
5) Adjust Add-ons Inbound Refinery

10gR3 Refinery is another Content Server
Separate the refinery machine from content
server machine
Run multiple refineries and make use of
selective queue processing
Analyze formats being used and adjust rules
Enables the automatic publishing of native business content to web viewable
PDF.
5) Adjust Add-ons - Content Tracker

Content Tracker is performance intensive
Consider changing reduction schedule
Discover what content is inactive
Analyze searches being performed
Helps optimize metadata usage
Oracle Content Tracker monitors content and server access and performs data
extraction to populate a database repository. The data is then viewable through
out of the box reports, as well as standalone reporting tools (e.g. Crystal
Reports).
5) Adjust Add-ons - Content
Integration Suite (CIS)

Cache! Cache! Cache!
Service-level caching exists, but out-of-the-box only
caches DOC_INFO
Cache data that is public information and not tied to
users or security. e.g. Announcements
Caching mechanisms
Page-level HTTP caching
App server specific caching
OSCache toolkit

Oracle Content Integration Suite provides for a reliable and scalable integration
infrastructure for integrations with Enterprise Applications.
Professional Services

Oracle Professional Services is best suited to assist with
performance issues
Oracle Support is not capable of analyzing and tuning of
performance issues
Support can assist with pointing customers in general direction
Support can initiate the request for Consulting Services
Slow does not always equal Broken
May not be a patch or component that can help the issue
May need hardware or architecture solution


Questions?
Cordell Melgaard
Principal Technical Support Engineer
Global Customer Service
cordell.melgaard@oracle.com
Jason Ganske
Principal Technical Support Engineer
Global Customer Service
jason.ganske@oracle.com

Thank you!