<Insert Picture Here>

Content Server Performance Tuning

Welcome Content Management Customers!
Five Steps to a Faster Server

• Design Benchmarks
• Run Benchmarks • Analyze Benchmarks • Adjust the core configuration • Adjust add-on configuration

General Issues
• HELP!! My Content Server is Slow
• • • • Searching Batchloading / Contribution Page Rendering (HSCP, JSP, ASP) Publishing

• Tuning is the same as any web application
• • • • Adjust configuration flags for your specific needs Optimize Queries Maximize hardware Cache! Cache! Cache!

Underlying Problem Areas
• All web applications have similar problems
• • • • • • File system: too small or too slow Bandwidth: too slow to clients or database Processor: too small or overused Memory: too little or underused Database queries are not optimized Cache: too small, underused, or ignored

• What‟s the solution?
• There‟s a reason its called „tuning‟… • There is no magic button to fix performance problems

Adjust the core configuration 5. Adjust add-on configuration .Five Steps to a Faster Server 1. Run Benchmarks 3. Analyze Benchmarks 4. Design Benchmarks  Gather setup and usage information 2.

Intro • Speed is Relative • Need something to compare against .1) Design Benchmarks .

1) Design Benchmarks .Intro • Too few make benchmarks. but it‟s vital to: • Quantify performance • Clean-room vs. real-world • Avoid side-effects • Avoid fixing the wrong problem • Basic benchmarks can be quick and painless • Thorough benchmarks can be slow and painful .

memory. Active Directory. Contributing.1) Design Benchmark – Initial Steps • Document and verify setup • • • • Hardware: CPUs. routers. bandwidth Security: LDAP. manufacturer Software: database. web server. CMS version Network: firewalls. SSO • Define where it‟s slow • Searching. Viewing • Check the Server Output for errors • Logging even simple errors slows down the system . switches.

check in • Recreate a typical workload for a user . or standard templates • Submit some forms • Search. update.1) Design Benchmarks – Web and Batch Tests • Design a web script for a typical user‟s session • Navigate to common pages • Site Studio. Publisher. workflow.

1) Design Benchmark . or • Use an exact replica of production server • Isolate benchmark server • Only allow testers to access • Isolation required to quantify performance changes • …but isolation does not resemble real-world • Document Document Document • Every test should be well documented to find consistency .Pitfalls • Make benchmark as real-world as possible • Use the production server.

logging 3. Design Benchmarks 2. execution. Analyze Benchmarks 4. Adjust the core configuration 5. Adjust add-on configuration .Five Steps to a Faster Server 1. Run Benchmarks  Tools.

2) Run Benchmarks – General Tools • Free and Simple • Content Server Health Monitor • Content Server System Audit Page • Process Explorer • taskmgr (Win32) or top (Unix) • Complex • Mercury LoadRunner • Java Profilers • Perl / Python scripts • PamIE. SamIE. WWW::Mechanize .

2) Run Benchmarks – Specific Tools • Database performance analysis tools • HUNDREDS of tools available • Check with your DBA • Apache mod_log_config • Customize logs to record page response time • Prioritize effort on popular pages • Network analysis • Ethereal/Wireshark • trafshow • tracert / traceroute .

etc. mkvdk. .2) Run Benchmarks – Setup and Execution • Number of users for web benchmark • Concurrent = 1 to 10% of total users • Peak usage = 10 to 80% weekly users • Monitor all processes on all servers • CPU and RAM usage. IO Reads • Content Server: java. mssql. htmlexport • Database: oracle.

pagecreation. requestaudit socketrequests. Archiver.log (Windows) • oracle/etc/log (Unix) • System Audit Info tracing sections: indexer. and Server logs • Filter Administration for security filter logs • Tracing output • oracle/bin/IdcServerNT. searchquery systemdatabase.2) Run Benchmarks – Log Files • Web log files • Refinery. userstorage .

database. web server.2) Run Benchmarks – Pitfalls • Adjustments • Best-case: turn off logging. don‟t allow contribution • Bad-case: turn off search cache • Worst-case: run batch and web tests simultaneously • Double-check resource usage • Is a virus scanner running? • Is anybody else accessing the server? • Who else is using CS dependent resources? • File server. LDAP/AD/NTLM .

Adjust the core configuration 5. Run Benchmarks 3. Analyze Benchmarks  Find possible bottlenecks and retest 4. Adjust add-on configuration . Design Benchmarks 2.Five Steps to a Faster Server 1.

Read.3) Analyze Benchmarks • Review benchmarks • Read. Analyze • Time required to run batch tests • Requests per second per user for web tests • What is slow? • EXAMPLE goals for business content: • • • • Read-only requests: 20 per second per GHz CPU Raw Verity queries: 4 per second per GHz CPU Contribution: 4 per second per GHz CPU Assume: 100k files and 1GB RAM . Analyze.

3) Analyze Benchmarks – Find the bottleneck • Where is the bottleneck? • • • • • • Search engine / indexer Database Network Java VM (Content Server. and retest . Refinery. add-ons) File system Web Security Filter • First benchmarks offer hints • Need verification • Enable more tracing sections.

3) Analyze Benchmarks – Search or Indexer Bottleneck Clues • Template pages are fast • Simple queries take over one second • Navigation pages execute highly complex queries • Analyze requestaudit & searchquery trace • The system has a complex security model • Search cache isn‟t utilized • System Audit Info: small cache. few hits • Verity collection is not local • Slow network file system? • Database queries not optimized when using DB search .

server.indexer.ServiceException: • Too many connection errors !csIndexerUnableToBuildChanges!csNoConnectionsAvailable. !$gpescm2: !csIndexerAbortedMsg • Dropped connection errors intradoc.IndexerMonitor.Indexer.doIndexing(Unknown Source) at intradoc.IndexerBuildChanges.run(Unknown queries • Monitor searchquery tracing for timestamps onSource) • Watch standard web logs for connection errors .indexer. connections below shows more information.buildIndex(Unknown Source) Hint: intradoc.server.common.indexer.buildChanges(Unknown • Database CPU is overworked Source) • SCS and MKVDK underworked at intradoc.doIndexing(Unknown Source) at at intradoc.indexer.Indexer.IndexerBuildChanges.Sys temDatabase • Most of request time is spent in database at intradoc.doWork(Unknown Source) Common Error • at intradoc.3) Analyze Benchmarks – Database Bottleneck Clues • System Audit Info page An error and waiting The stack trace • Activehas occurred.IndexerMonitor$4.

3) Analyze Benchmarks – Network Bottleneck Clues • Remote connections (databases) are slow • …but remote services aren‟t working very hard • Use trafshow or Ethereal/Wireshark to determine saturation • Hint: • Monitor searchquery tracing for timestamps on queries • Compare with running the query directly on the database .

3) Analyze Benchmarks – Core Engine / Java VM Bottleneck Clues • The Stellent engine runs pretty lean. but • Publishing or page rendering can be slow • It may be a Java VM problem if: • • • • It‟s a consumption server It renders a lot of HCSPs or JSPs It runs a lot of workflows or subscriptions Java VM uses all the CPU or Memory • Even if searching isn‟t performed • Multiple CPUs not being fully utilized • Using outdated JVM .

easy setup •CHEAP • Larger than expected file activity • Monitor I/O reads.3) Analyze Benchmarks – File System Bottleneck Clues •FAST • File system is remote. expensive. long setup •RELIABLE • NAS: Slower. and I/O read Bytes • File system is nearing capacity • Hint: PICK TWO • Note: some remote file system problems indicate an underlying network problem . cheaper. but not high-performance • SAN: Very fast.

3) File System Architecture Performance Web Server Fastest Local Local Local Local Local Remote Remote Remote Location File System Local Local SAN NAS Remote Local SAN NAS Remote Database Local Remote Remote Remote Remote Remote Remote Remote Remote Slowest Remote .

3) Analyze Benchmarks – Web Security Filter Bottleneck Clues • User‟s first login takes several seconds • Subsequent page delivery is fast • Many users login as their browser‟s home page • Pay login cost. but don‟t navigate further • Unmanaged content is much faster • Use only one thread to test • Hint: • Monitor verbose userstorage trace for timestamps • Note: stack traces are not always errors in this section .

Pitfalls • Java profiler reports misleading • Logging and virus scanners slow systems down • Try to establish real-world benchmarks • No search cache.3) Analyze Benchmarks . do contribution and consumption • Watch the search engine carefully • Note the frequency and complexity of queries • Substring queries KILL performance .

Run Benchmarks 3. Adjust add-on configuration . caching. Design Benchmarks 2. and verity 5. Analyze Benchmarks 4. Adjust the core configuration  Configuration.Five Steps to a Faster Server 1. database.

4) Adjust Core • Easy: • Configuration flags • Java VM tuning • Difficult: • IdocScript caching and tuning • Simplify security model • Harder: • Verity Index Tuning • Database Indexes and tuning (OracleQueryOptimizer) .

TimeoutPerOneMegInSec • Search • MaxSearchConnections. MaxMergeBaseCount. DoAutoMaxMerge. CachedResultRowCount • Database • NumConnections • Only increase for testing can have opposite effect • General • UserCacheTimeout. IdcServerSocketQueueDepth .4) Adjust Core – Config Flags • Indexer (Verity) • AdditionalIndexBuildParams.

4) Adjust Core – Java Tuning • JVM is self-tuning.1 • JAVA_MAX_HEAP_SIZE=[Size in megabytes] • Multiple CPU systems • Use JVM 1.cfg • Optimize memory and compiler • MetaLink Note : Note 460039. platform specific • Different tuning options for CS versions .1 or better • Tune GC for multiple CPUs.5. but benefits from hints • Modify values in intradoc.

METADATA • QueryText=dDocType LIKE ‘ADACCT‘ .4) Adjust Core – IdocScript Tuning • Simplify pages • Some pages needlessly do multiple requests • Reduce number and complexity of queries • Avoid queries that don‟t cache. such as timestamps • Resolve simple errors • CS Log will constantly log IdocScript errors unnecessarily • Use database search for exact matches • SearchEngineName=DATABASE.

or session cache • setMaxAge • Uses Cache-Control HTTP header for page caching • For pages that one uses hits frequently . named cache.4) Adjust Core – IdocScript Caching • cacheInclude • Include a dynamichtml fragment from a cache • For fragments that are used multiple times • Global cache.

the longer the query • In general.4) Adjust Core – Simplify security • Searches attach security clause • The more complex the security. longer queries take longer to run • Worst performance will be from users with mixed security • Monitor searchquery trace in System Audit • User input: • dDocName <matches> `test` • Actual query run: • (dDocName <matches> `test`) <and> (dSecurityGroup <not> `Secure`) .

author • Plan ahead! • Adding metadata fields requires an index rebuild • Use dummy or non-indexed fields.4) Adjust Core – Verity Tuning • Tune and houseclean the Verity collection • MKVDK command-line options • Leverage „zone search‟ on common fields • Avoid automatically adding every field to index • Tune the number of „parts file‟ • Many parts. title. or database search . fast search. Few parts. fast index • Put commonly searched fields into separate parts • Security group.

helps query optimizer • Optimize row-locking algorithm • ORACLE: optimizer_mode = first_rows • Consider custom services or queries • Database-specific queries for better performance • Simplify check-ins when running batches .4) Adjust Core – Database Tuning • OracleQueryOptimizer • Free component for tuning Oracle DB queries • Add extra indexes to commonly searched columns • Update database statistics as a database grows • Before & after inserting lots of content • Maintains indexes.

4) Adjust Core .Pitfalls • Be sure to re-run the benchmarks • Record new performance data • Watch for side-effects! • Test server improvements misleading • Production server will not see exact benefits… • unless the benchmark was 100% real-world .

and other extras . publishing. Adjust the core configuration 5. Analyze Benchmarks 4. J2EE. Adjust add-on configuration  Conversion. Design Benchmarks 2. Run Benchmarks 3.Five Steps to a Faster Server 1.

5) Adjust Add-ons • • • • • • Batchloader Dynamic Converter Site Studio PDF Converter Content Tracker CIS .

• Oracle: ANALYZE TABLE COMPUTE STATISTICS • SQL Server: CREATE STATISTICS • DB2: RUNSTATS . • Analyze your database usage during a batchload. Goal is to help the database query optimizer.Batchloader • Temporarily disable other activity • Inbound Refinery / PDF Converter • Auto update indexer cycle – Repository Manager • Optimize the search collection (if using Verity) prior to batch insert.5) Adjust Add-ons .

• Cache location: <stellent_dir>\weblayout\groups\<security_group>\documents\<typ e>\~export\<content id>~<rev>~<template name>~<layout name> .Dynamic Converter Dynamic Converter enables on-demand. • Create Template Selection Rules • Predetermines the conversion and layout templates • Upon submission.5) Adjust Add-ons . XML or wireless formats. ad-hoc publishing of native business documents (255 formats) to HTML. content is converted and cached.

workflow. Site Studio enables business users to contribute content in a userfriendly WYSIWYG form-based environment with regional-level security and incontext preview.5) Adjust Add-ons . • Analyze page architecture • Turn on tracing and watch the output • Search. Database.Site Studio Site Studio automates Web site creation and contribution methodologies. Additionally. and approval. Script Errors • Optimize the queries in dynamic lists • • • • Simple Operators (Matches) Indexed Avoid using time and Substring .

Remove unnecessary work/data.g. • Client-side vs. out-of-the-box fragments use client-side mechanisms to build navigation.) • Use front side caching • Reverse proxy server • Use IdocScript caching • Optimize services executed on pages • Consolidate actions. • Static lists vs.5) Adjust Add-ons .Site Studio (cont. server-side page construction • e. dynamic lists • Static Lists – XML content • Dynamic Lists – Search Queries .

• 10gR3 Refinery is another Content Server • Separate the refinery machine from content server machine • Run multiple refineries and make use of selective queue processing • Analyze formats being used and adjust rules .5) Adjust Add-ons – Inbound Refinery Enables the automatic publishing of native business content to web viewable PDF.

5) Adjust Add-ons . Crystal Reports).Content Tracker Oracle Content Tracker monitors content and server access and performs data extraction to populate a database repository. • Content Tracker is performance intensive • Consider changing reduction schedule • Discover what content is “inactive” • Analyze searches being performed • Helps optimize metadata usage . as well as standalone reporting tools (e. The data is then viewable through out of the box reports.g.

• Cache! Cache! Cache! • Service-level caching exists.5) Adjust Add-ons .Content Integration Suite (CIS) Oracle Content Integration Suite provides for a reliable and scalable integration infrastructure for integrations with Enterprise Applications.g. Announcements • Caching mechanisms • Page-level HTTP caching • App server specific caching • OSCache toolkit . e. but out-of-the-box only caches DOC_INFO • Cache data that is public information and not tied to users or security.

Professional Services • Oracle Professional Services is best suited to assist with performance issues • Oracle Support is not capable of analyzing and tuning of performance issues • Support can assist with pointing customers in general direction • Support can initiate the request for Consulting Services • Slow does not always equal Broken • May not be a patch or component that can help the issue • May need hardware or architecture solution .

Questions? • Visit Oracle. • Thank you! .com for more detail.

:79.08+9089+ .7.99.:7974:5349+$0.:80 W %02470..9..3/ /$0.:70+ .9.3.08+9089+ W ..6:077:3 W /4.0-01742:8078920/80. 43076:07089.202. /:89470 $25180.:79 904307906:07 W 30307.04307947:3 W 47895071472..6:07 97.80.4250 9080.7.08.9:.:79 W $0..:79 W 4394780.03$8902:/9 W &80735:9 W /4.202.

4/./ W //3209..0. 43.9.33/070-:/ W &80/:2247343 3/00/10/8 47/.7.3/ 30459438 W 0.07.905.893/0 W !:9././/30.-.05.798 1.8080.10/8706:708.35..:947 W !.4224310/8 W .7.8980.:7974:5 990 .943 W '.4224380.0.7.798 W $0.0 43080.3. /:89470 '079%:33 W %:30.79810 W . .9.40.0710/943/0 W %:30903:2-0741 5.798 1.7.9.422.:942.7.0/10/8394805.3/4:80.390'079.

4:238 W &5/.9.-.:8942807.383/008 0586:07459207 W W 592074 4.0.0":07 59207 W 700. /:89470 .190738079349841.1.989.8.8089..9.-.3.06:0708 W //097.9.4792 #459207*24/01789*748 W 438/07.39.3/00894.08 .90/.7.-.9.425430391479:33 7.4224380./.80%:33 W 7. 38037:333-.439039 W .0 W $251.6:0708147-099075071472.80748 W 01470 .0/..-.8.3.9..80 850.08476:0708 W .

 /:89470 !91.9.78 W #0.8 70.1478/0 0110.2.98 W %089807.943807. W .47/305071472. 47/ .020398280./3 W !74/:.9-030198 W :308890-03.073498000.2.7.3.0/.9.8 W 08:709470 7:390-03.072574..

2.0$905894..8 .// 43.8907$07.943 43.2.470..07843 5:-83  .2.78  3.3/4907097.431:7.943  /:89.003.78  /:8990.431:7.78  #:303.07  08303.

.43.07 $ . /:89// 438 W W W W W W .07907 439039%7.2./07 3.07907 $90$9:/4 !43.4.9.

7/8.4.9 W 3-4:3/#01307.-04907. /:89// 438 ./07 W %02547.9..9.

4.7.07 W 59209080.943 1:83'079  574794-.806:07 459207 W 7.0 #0548947.3.07907 W :94:5/.0/:73..9.903/007..9.40..07#%$%%$%$ W #&$%%$ .-.9./ 4. -.80:8.0% !&%$%%$%$ W $"$07.8940590/.!43.04:7/.-.38079 W 3.9.

.98 94% 47700881472.903.90%025.4390398.439039/=70.0790/.90$00.3/.3/.908 W &5438:-28843 .07907 3.943 890039*/70-.0/ W .04.20 .43.0790703.:20398 1472.4:93. /:89// 438 3.43.4:974:5880.3/ .4:99025.9.20=.:2039895 0=05479.43.0-:83088 /4.:79*74:5/4.=9025..07843.2.43./ 4.5:-83413..943#:08 W !70/0907230890..98 W 70.2.-0843 /02.

08 W 3/00/ W .80 $.9080-890. /:89// 438 $90$9:/4 $90$9:/4.943. 0.3/.-.3/$:-8973 .4397-:9432094/4408  //943.80/03.5574.:79.898 W $250 W 507..9.7.:807 1703/$1472 -.080.3/.9:70 W %:734397.:942.7. $90$9:/403.0.75977478 W 5920906:07083/3.2.3.43909570.-08-:83088:807894.0 4714 .05.9478 .904:95:9 W $0.70.3/3 .9..743203997043.4/:83920.3/. . W 3.4397-:90.9.90.4390393.

:90/435.90. /:89// 438 $90$9:/4 .3 W #0.439 W &80174398/0..088..759..747..0780574807..0:330.08 W 4384/.3 W 5920807.9438 #024..07 W &80/4.$.0800.

2.9.8 807.8 /3.43897:.898 W $9.382894 -:/3..9.943 W $9.943 W 0  4:9 41 90 -417.20398:80.7./.":0708 .07 8/05.2...898 . W 039 8/0.898.039 8/020.9.0.439039 W 3.898 $0.

98-03:80/.439039940-.34907439039$07. /:89// 438 3-4:3/#01307 3.7.0883 W 3.3/.301742.072.9.-0 ! W  ##013078.9.0-:83088.5:-83413.30 W #:32:95070130708.0.90907013072.0:8041 800.07 W $05.:942.01472.06:0:0574.3/2..439039 807.-0890..9./:897:08 .

 #054798 W 439039%7.3/.-.9...439039.07 7.0 W 3.7.9.:8.9.9...0 .07.0/:0 W 8.3.889.9./.3/50714728/.90.4.080.9..0785071472.3/807.80. /:89// 438 439039%7.9438.3370/:. 097.08-0350714720/ W 05845920209..4390398 3..039038.0.-0974: 4:94190-47054798 .94394545:.088.07.8903.072439478.0439039%7./..8070548947 %0/..0 W 438/07.43070547939448 0  789.

31472...9:701473907.0.. /:89// 438 439039 3907./08147.3 W $..9438  W .1.943$:90574....3/8...943.9...943$:90 $ 7.897:..9.0.020398 W .-0.:79 0  334:3.3 W 55807...08 * W .04390393907.0%%!.07850..943 317.0 0.985:-.0.09449 .-03907.943893907578055...0 0.3/34990/94 :80784780..0 W $07.30898 -:94:9 41 90 -443 .0/..3828 W !.320.70...

.300/.425430399.3..3/9:3341 5071472.-041.3.7047.943 W $:55479..088:08 W 7.7.5..806:..9:7084:943 .7403 W .088:08 W $:55479.349-0..90.9090706:089147438:93$07.5.08 W $4/408349.3059088:0 W .08 W 7.$07.:894207830307...3.8889954393.3.!74108843.0$:554798349.0!74108843./70..$07.88899 5071472.9.3.088-0898:90/94..7/.47.9.339.

 W %.0 .":089438 W '89 7.34: .421472470/09..

Sign up to vote on this title
UsefulNot useful