<Insert Picture Here>

Content Server Performance Tuning

Welcome Content Management Customers!

Five Steps to a Faster Server

Design Benchmarks
Run Benchmarks Analyze Benchmarks Adjust the core configuration Adjust add-on configuration

General Issues
HELP!! My Content Server is Slow
Searching Batchloading / Contribution Page Rendering (HSCP, JSP, ASP) Publishing

Tuning is the same as any web application

Adjust configuration flags for your specific needs Optimize Queries Maximize hardware Cache! Cache! Cache!

Underlying Problem Areas

All web applications have similar problems
File system: too small or too slow Bandwidth: too slow to clients or database Processor: too small or overused Memory: too little or underused Database queries are not optimized Cache: too small, underused, or ignored

Whats the solution?

Theres a reason its called tuning There is no magic button to fix performance problems

Five Steps to a Faster Server

1. Design Benchmarks
Gather setup and usage information

2. Run Benchmarks 3. Analyze Benchmarks 4. Adjust the core configuration

5. Adjust add-on configuration

1) Design Benchmarks - Intro

Speed is Relative Need something to compare against

1) Design Benchmarks - Intro

Too few make benchmarks, but its vital to:
Quantify performance Clean-room vs. real-world Avoid side-effects Avoid fixing the wrong problem

Basic benchmarks can be quick and painless Thorough benchmarks can be slow and painful

1) Design Benchmark Initial Steps

Document and verify setup
Hardware: CPUs, memory, manufacturer Software: database, web server, CMS version Network: firewalls, routers, switches, bandwidth Security: LDAP, Active Directory, SSO

Define where its slow

Searching, Contributing, Viewing

Check the Server Output for errors

Logging even simple errors slows down the system

1) Design Benchmarks Web and Batch Tests

Design a web script for a typical users session
Navigate to common pages Site Studio, Publisher, or standard templates Submit some forms Search, update, workflow, check in Recreate a typical workload for a user

1) Design Benchmark - Pitfalls

Make benchmark as real-world as possible
Use the production server, or Use an exact replica of production server

Isolate benchmark server

Only allow testers to access Isolation required to quantify performance changes but isolation does not resemble real-world

Document Document Document

Every test should be well documented to find consistency

Five Steps to a Faster Server

1. Design Benchmarks 2. Run Benchmarks
Tools, execution, logging

3. Analyze Benchmarks 4. Adjust the core configuration

5. Adjust add-on configuration

2) Run Benchmarks General Tools

Free and Simple
Content Server Health Monitor Content Server System Audit Page Process Explorer taskmgr (Win32) or top (Unix)

Complex
Mercury LoadRunner Java Profilers Perl / Python scripts PamIE, SamIE, WWW::Mechanize

2) Run Benchmarks Specific Tools

Database performance analysis tools
HUNDREDS of tools available Check with your DBA

Apache mod_log_config
Customize logs to record page response time Prioritize effort on popular pages

Network analysis
Ethereal/Wireshark trafshow tracert / traceroute

2) Run Benchmarks Setup and Execution

Number of users for web benchmark
Concurrent = 1 to 10% of total users Peak usage = 10 to 80% weekly users

Monitor all processes on all servers

CPU and RAM usage, IO Reads Content Server: java, mkvdk, htmlexport Database: oracle, mssql, etc.

2) Run Benchmarks Log Files

Web log files
Refinery, Archiver, and Server logs Filter Administration for security filter logs

Tracing output
oracle/bin/IdcServerNT.log (Windows) oracle/etc/log (Unix) System Audit Info tracing sections: indexer, pagecreation, requestaudit socketrequests, searchquery systemdatabase, userstorage

2) Run Benchmarks Pitfalls

Adjustments
Best-case: turn off logging, dont allow contribution Bad-case: turn off search cache Worst-case: run batch and web tests simultaneously

Double-check resource usage

Is a virus scanner running? Is anybody else accessing the server? Who else is using CS dependent resources? File server, database, web server, LDAP/AD/NTLM

Five Steps to a Faster Server

1. Design Benchmarks 2. Run Benchmarks 3. Analyze Benchmarks
Find possible bottlenecks and retest

4. Adjust the core configuration

5. Adjust add-on configuration

3) Analyze Benchmarks
Review benchmarks
Read, Analyze, Read, Analyze Time required to run batch tests Requests per second per user for web tests

What is slow? EXAMPLE goals for business content:

Read-only requests: 20 per second per GHz CPU Raw Verity queries: 4 per second per GHz CPU Contribution: 4 per second per GHz CPU Assume: 100k files and 1GB RAM

3) Analyze Benchmarks Find the bottleneck

Where is the bottleneck?
Search engine / indexer Database Network Java VM (Content Server, Refinery, add-ons) File system Web Security Filter

First benchmarks offer hints

Need verification Enable more tracing sections, and retest

3) Analyze Benchmarks Search or Indexer Bottleneck Clues

Template pages are fast Simple queries take over one second Navigation pages execute highly complex queries Analyze requestaudit & searchquery trace The system has a complex security model Search cache isnt utilized System Audit Info: small cache, few hits Verity collection is not local Slow network file system? Database queries not optimized when using DB search

3) Analyze Benchmarks Database Bottleneck Clues

System Audit Info page An error and waiting The stack trace Activehas occurred. connections below shows more information. !$gpescm2: !csIndexerAbortedMsg Dropped connection errors intradoc.common.ServiceException: Too many connection errors !csIndexerUnableToBuildChanges!csNoConnectionsAvailable,Sys temDatabase Most of request time is spent in database at intradoc.indexer.IndexerBuildChanges.buildChanges(Unknown Database CPU is overworked Source) SCS and MKVDK underworked at intradoc.indexer.IndexerBuildChanges.doWork(Unknown Source)
Common Error

at intradoc.indexer.Indexer.doIndexing(Unknown Source) at intradoc.indexer.Indexer.buildIndex(Unknown Source) Hint: intradoc.server.IndexerMonitor.doIndexing(Unknown Source) at at intradoc.server.IndexerMonitor$4.run(Unknown queries Monitor searchquery tracing for timestamps onSource)

Watch standard web logs for connection errors

3) Analyze Benchmarks Network Bottleneck Clues

Remote connections (databases) are slow but remote services arent working very hard Use trafshow or Ethereal/Wireshark to determine saturation

Hint:
Monitor searchquery tracing for timestamps on queries Compare with running the query directly on the database

3) Analyze Benchmarks Core Engine / Java VM Bottleneck Clues

The Stellent engine runs pretty lean, but
Publishing or page rendering can be slow

It may be a Java VM problem if:

Its a consumption server It renders a lot of HCSPs or JSPs It runs a lot of workflows or subscriptions Java VM uses all the CPU or Memory Even if searching isnt performed Multiple CPUs not being fully utilized Using outdated JVM

3) Analyze Benchmarks File System Bottleneck Clues

FAST File system is remote, but not high-performance SAN: Very fast, expensive, long setup RELIABLE NAS: Slower, cheaper, easy setup CHEAP Larger than expected file activity Monitor I/O reads, and I/O read Bytes File system is nearing capacity
Hint:

PICK TWO

Note: some remote file system problems indicate an underlying network problem

3) File System Architecture

Performance Web Server Fastest
Local Local Local Local Local Remote Remote Remote

Location File System

Local Local SAN NAS Remote Local SAN NAS Remote

Database
Local Remote Remote Remote Remote Remote Remote Remote Remote

Slowest

Remote

3) Analyze Benchmarks Web Security Filter Bottleneck Clues

Users first login takes several seconds Subsequent page delivery is fast Many users login as their browsers home page Pay login cost, but dont navigate further Unmanaged content is much faster Use only one thread to test

Hint:
Monitor verbose userstorage trace for timestamps Note: stack traces are not always errors in this section

3) Analyze Benchmarks - Pitfalls

Java profiler reports misleading Logging and virus scanners slow systems down Try to establish real-world benchmarks
No search cache, do contribution and consumption

Watch the search engine carefully

Note the frequency and complexity of queries Substring queries KILL performance

Five Steps to a Faster Server

1. Design Benchmarks

2. Run Benchmarks
3. Analyze Benchmarks 4. Adjust the core configuration
Configuration, caching, database, and verity

5. Adjust add-on configuration

4) Adjust Core
Easy:
Configuration flags Java VM tuning

Difficult:
IdocScript caching and tuning Simplify security model

Harder:
Verity Index Tuning Database Indexes and tuning (OracleQueryOptimizer)

4) Adjust Core Config Flags

Indexer (Verity)
AdditionalIndexBuildParams, DoAutoMaxMerge, MaxMergeBaseCount, TimeoutPerOneMegInSec

Search
MaxSearchConnections, CachedResultRowCount

Database
NumConnections Only increase for testing can have opposite effect

General
UserCacheTimeout, IdcServerSocketQueueDepth

4) Adjust Core Java Tuning

JVM is self-tuning, but benefits from hints
Modify values in intradoc.cfg

Optimize memory and compiler

MetaLink Note : Note 460039.1

JAVA_MAX_HEAP_SIZE=[Size in megabytes]

Multiple CPU systems

Use JVM 1.5.1 or better Tune GC for multiple CPUs, platform specific

Different tuning options for CS versions

4) Adjust Core IdocScript Tuning

Simplify pages
Some pages needlessly do multiple requests Reduce number and complexity of queries Avoid queries that dont cache, such as timestamps

Resolve simple errors

CS Log will constantly log IdocScript errors unnecessarily

Use database search for exact matches

SearchEngineName=DATABASE.METADATA QueryText=dDocType LIKE ADACCT

4) Adjust Core IdocScript Caching

cacheInclude
Include a dynamichtml fragment from a cache For fragments that are used multiple times Global cache, named cache, or session cache

setMaxAge
Uses Cache-Control HTTP header for page caching For pages that one uses hits frequently

4) Adjust Core Simplify security

Searches attach security clause
The more complex the security, the longer the query In general, longer queries take longer to run Worst performance will be from users with mixed security

Monitor searchquery trace in System Audit

User input:
dDocName <matches> `test`

Actual query run:

(dDocName <matches> `test`) <and> (dSecurityGroup <not> `Secure`)

4) Adjust Core Verity Tuning

Tune and houseclean the Verity collection
MKVDK command-line options

Leverage zone search on common fields Avoid automatically adding every field to index Tune the number of parts file
Many parts, fast search; Few parts, fast index

Put commonly searched fields into separate parts

Security group, title, author

Plan ahead!
Adding metadata fields requires an index rebuild Use dummy or non-indexed fields, or database search

4) Adjust Core Database Tuning

OracleQueryOptimizer
Free component for tuning Oracle DB queries

Add extra indexes to commonly searched columns Update database statistics as a database grows
Before & after inserting lots of content Maintains indexes, helps query optimizer

Optimize row-locking algorithm

ORACLE: optimizer_mode = first_rows

Consider custom services or queries

Database-specific queries for better performance Simplify check-ins when running batches

4) Adjust Core - Pitfalls

Be sure to re-run the benchmarks

Record new performance data Watch for side-effects!

Test server improvements misleading

Production server will not see exact benefits unless the benchmark was 100% real-world

Five Steps to a Faster Server

1. Design Benchmarks

2. Run Benchmarks
3. Analyze Benchmarks 4. Adjust the core configuration 5. Adjust add-on configuration
Conversion, publishing, J2EE, and other extras

5) Adjust Add-ons
Batchloader Dynamic Converter Site Studio PDF Converter Content Tracker CIS

5) Adjust Add-ons - Batchloader

Temporarily disable other activity
Inbound Refinery / PDF Converter Auto update indexer cycle Repository Manager

Optimize the search collection (if using Verity) prior to batch insert. Analyze your database usage during a batchload. Goal is to help the database query optimizer.
Oracle: ANALYZE TABLE COMPUTE STATISTICS SQL Server: CREATE STATISTICS DB2: RUNSTATS

5) Adjust Add-ons - Dynamic Converter

Dynamic Converter enables on-demand, ad-hoc publishing of native business documents (255 formats) to HTML, XML or wireless formats.

Create Template Selection Rules

Predetermines the conversion and layout templates Upon submission, content is converted and cached. Cache location:
<stellent_dir>\weblayout\groups\<security_group>\documents\<typ e>\~export\<content id>~<rev>~<template name>~<layout name>

5) Adjust Add-ons - Site Studio

Site Studio automates Web site creation and contribution methodologies. Additionally, Site Studio enables business users to contribute content in a userfriendly WYSIWYG form-based environment with regional-level security and incontext preview, workflow, and approval.

Analyze page architecture Turn on tracing and watch the output

Search, Database, Script Errors

Optimize the queries in dynamic lists

Simple Operators (Matches) Indexed Avoid using time and Substring

5) Adjust Add-ons - Site Studio (cont.)

Use front side caching
Reverse proxy server Use IdocScript caching

Optimize services executed on pages

Consolidate actions. Remove unnecessary work/data.

Client-side vs. server-side page construction

e.g. out-of-the-box fragments use client-side mechanisms to build navigation.

Static lists vs. dynamic lists

Static Lists XML content Dynamic Lists Search Queries

5) Adjust Add-ons Inbound Refinery

Enables the automatic publishing of native business content to web viewable PDF.

10gR3 Refinery is another Content Server Separate the refinery machine from content server machine Run multiple refineries and make use of selective queue processing Analyze formats being used and adjust rules

5) Adjust Add-ons - Content Tracker

Oracle Content Tracker monitors content and server access and performs data extraction to populate a database repository. The data is then viewable through out of the box reports, as well as standalone reporting tools (e.g. Crystal Reports).

Content Tracker is performance intensive

Consider changing reduction schedule

Discover what content is inactive Analyze searches being performed

Helps optimize metadata usage

5) Adjust Add-ons - Content Integration Suite (CIS)

Oracle Content Integration Suite provides for a reliable and scalable integration infrastructure for integrations with Enterprise Applications.

Cache! Cache! Cache!

Service-level caching exists, but out-of-the-box only caches DOC_INFO Cache data that is public information and not tied to users or security. e.g. Announcements Caching mechanisms Page-level HTTP caching App server specific caching OSCache toolkit

Professional Services
Oracle Professional Services is best suited to assist with performance issues Oracle Support is not capable of analyzing and tuning of performance issues
Support can assist with pointing customers in general direction Support can initiate the request for Consulting Services

Slow does not always equal Broken

May not be a patch or component that can help the issue May need hardware or architecture solution

Questions?
Visit Oracle.com for more detail.

Thank you!