BRSPACE Online Reorganizations

Top Ten Pitfalls and Problems

Martin Frauendorfer, SAP Active Global Support martin.frauendorfer@sap.com November 2009

Agenda

1. Overview 2. Pitfalls and Problems

SAP 2009 / Page 2

Overview (1)
Reorganizations in general A reorganization in a database context is the recreation or cleanup of one or several segments. Several types of reorganization exist:
Index reorganization (e.g. via REBUILD or COALESCE), mainly used to reduce index fragmentation Table reorganization with tools like: BRSPACE (online, offline, export / import) Oracle tools (DBMS_REDEFINITION, MOVE, EXP / IMP, EXPDP / IMPDP, ) SAP functionality (ICNV, SE14, ) Tablespace reorganization (reorganization of all tables in the tablespace and recreation of the tablespace)

In this presentation we concentrate on the BRSPACE online reorganization of tables (which also includes tablespace reorganizations).

SAP 2009 / Page 3

Overview (2)
In which situations can a table reorganization be useful? Reduction of table size Cleanup of table fragmentation Change of table storage parameters (e.g. INITRANS) Move of table to a different tablespace Reduction of chained and migrated rows Reduction of number of allocated extents Transition from LONG to LOB columns (Oracle >= 10g) Reduction of hot spots on disk level Activation of Transparent Data Encryption Change of table structure

SAP 2009 / Page 4

Overview (3)
In which situations can a tablespace reorganization be useful? Activation of LMTS Activation of ASSM Cleanup of freespace block corruption Reduction of database size Reduction of backup size Reduction of filesystem size Reduction of fragmentation Reduction of number of datafiles Increase of number of datafiles Transition to new tablespace layout Other change to tablespace layout Optimization of filesystem layout

SAP 2009 / Page 5

Overview (4)
How does a BRSPACE online reorganization work?
Determination of CREATE commands for target table and indexes based on DBMS_METADATA.GET_DDL Creation of target table with naming convention <source_table>#$ Export of source table CBO statistics based on DBMS_STATS.EXPORT_TABLE_STATS Copy of source table data to target table based on DBMS_REDEFINITION.START_REDEF_TABLE. This package will implicitly create a materialized view log MLOG$_<source_table> to track all changes that are performed on the source table while the reorganization is running. Creation of target indexes with naming convention <source_index>#$ Import of source table CBO statistics to target table based on DBMS_STATS.IMPORT_TABLE_STATS Finalizing of the online reorganization using DBMS_REDEFINITION.FINISH_REDEF_TABLE. In this step all changes in the materialized log are applied to the target table and the names of source and target table are exchanged. The source table is dropped. The #$ suffix of the target table indexes is removed.
SAP 2009 / Page 6

Overview (5)
What are the advantages of a BRSPACE Online Reorganization? It uses the advantages of DBMS_REDEFINITION:
The reorganization can be performed in parallel to production operation. This means the table is accessible and records can be inserted / updated / deleted. If anything goes wrong during the reorganization the source table still exists unchanged and no restore / recovery is necessary. Compared to an offline reorganization via export / import all data has to be copied only once and not twice

It provides possibilities for client and server side parallelism in order to speed up the reorganization if sufficient system resources exist and the reorganization should finish within a specific time window. It has already proven to be a reliable reorganization tool at many customers. It performs a lot of useful activities implicitly that would require a lot of manual efforts and impose the risk of errors if DBMS_REDEFINITION is used directly.
SAP 2009 / Page 7

Overview (6)
What are the restrictions of a BRSPACE Online Reorganization? It is only available as of Oracle 9i. It cannot handle tables with LONG or LONG RAW columns. As of 10g it is possible to convert them during an online reorganization on the fly to LOB columns (if SAP >= 6.40 is used). Temporarily twice the space is needed because both source and target table and indexes exist in parallel. If a table has no primary key constraint (or no unique index with NOT NULL columns) a ROWID based online reorganization is necessary that requires overhead (ROWID index, ROWID column). The setup of the materialized view is an overhead that can negatively impact the performance of the reorganization of many very small tables. Structure changes on the source table and its indexes must not be performed during the reorganization (as they would be lost).

SAP 2009 / Page 8

Overview (7)
How is a normal BRSPACE Online Reorganization started? Table reorganization:
brspace -f tbreorg -t <table_name>

Tablespace reorganization:
brspace -f tbreorg -s <source_tsp> -t "*" -n <target_tsp>

Additional useful options exist, e.g.:

-p <degree> BRSPACE parallelism degree -e <degree> PX parallelism degree -l <category> Initial extent size category -i <target_ind_tsp> Definition of target index tablespace (if different from target table tablespace)

For more detailed information see note 646681 and the BRSPACE online documentation.
SAP 2009 / Page 9

Pitfalls and Problems - Overview

Top Pitfalls and Problems

1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

What is a good parameter and redo log configuration? What has to be considered for PSAPTEMP? What about PSAPUNDO? And why is the DEFAULT tablespace sometimes used significantly? Why do tables sometimes grow during reorganization? What about SAP technical settings? Is it useful to convert from LONG to LOB? What is a good way of parallelism? Why should I look for invalid objects? Why does a DROP sometimes take ages?

SAP 2009 / Page 10

Pitfalls and Problems - 1. Configuration

1. Database Configuration
A good online reorganization performance needs a slightly different database configuration than normal system operation. Oracle parameters:
DB_FILE_MULTIBLOCK_READ_COUNT >= 128 Fast full table scan on the source table when creating the target table PGA_AGGREGATE_TARGET as large as possible Fast index creations in PGA with minimum PSAPTEMP accesses If Oracle parallel execution is used: PARALLEL_EXECUTION_MESSAGE_SIZE >= 16384 PARALLEL_MAX_SERVERS >= 2 * BRSPACE parallelism degree * Oracle parallelism degree PARALLEL_ADAPTIVE_MULTI_USER = FALSE Otherwise Oracle may reduce the desired parallelism degree SESSIONS, PROCESSES: Sufficiently large to cope with the increased PARALLEL_MAX_SERVERS value DB_CACHE_SIZE >= 1 GB Large buffer pool doesn't provide significant advantage
SAP 2009 / Page 11

Pitfalls and Problems - 1. Configuration

1. Database Configuration (2)

Redo Log Configuration
Usually all changes of an online reorganization are recorded in the redo logs, so there can be a significant amount of additional redo logs generated With current BRSPACE patches you can use the option "-NBR" to create the target indexes with NOLOGGING No redo log information is created for the indexes Theoretically it is possible to switch off logging for the target table, too. Almost no redo log information created at all Dangerous, because in case of recovery across the reorganization time the table data is lost! Only useful during a dedicated reorganization time window and with a good backup after the reorganization Redo logs should be created large enough to avoid log switches within less than one minute Avoids unnecessary checkpoint and log switch load ARCHIVELOG mode can be disabled in case of a dedicated reorganization time window and a good backup after the reorganization.
SAP 2009 / Page 12

Pitfalls and Problems - 2. PSAPTEMP

2. PSAPTEMP
Problems with PSAPTEMP can result in a long reorganization runtime or terminations. PSAPTEMP recommendations:
Define PSAPTEMP as LMTS/T tablespace (see SAP Note 659946) in order to avoid delays due to space transactions Make sure that PSAPTEMP is assigned as temporary tablespace to the SAP user and to SYS. Make sure that there are no I/O bottlenecks when accessing PSAPTEMP. PSAPTEMP must be large enough to hold the size of the largest reorganized index. If BRSPACE parallelism is used ("-p <degree>" option), PSAPTEMP must be large enough for the <degree> largest indexes that are reorganized in parallel. In case of a sorted reorganization ("-r <index>" option), PSAPTEMP must be large enough to hold the table data. Consider AUTOEXTEND or RESUMABLE to be on the save side

SAP 2009 / Page 13

Pitfalls and Problems - 3. PSAPUNDO

3. PSAPUNDO
PSAPUNDO is only used to a minor extent:
The data transfer into the target table works with APPEND, so no undo is generated. The creation of the indexes also doesn't create undo. Concurrent DML operations logged in the MLOG$ table and finally applied to the target table generate undo.

PSAPUNDO recommendations:
Use Automatic Undo Management (AUM, SAP Note 600141) Usually no increase of PSAPUNDO necessary Larger PSAPUNDO may be required if reorganization is done in parallel to significant DML operations (in general not recommended).

SAP 2009 / Page 14

Pitfalls and Problems - 4. Default Tablespace

4. Default Tablespace
The default tablespace is used in two situations:
The MLOG$ table with the recorded DML operations is created in the default tablespace. If a table has no primary index, a ROWID index (I_SNAP$...) is created in the default tablespace.

Default tablespace recommendations:

Make sure that a default tablespace with sufficient free space or AUTOEXTEND is assigned to both the SAP user and SYS. Use LMTS or make sure that NEXT and MAXEXTENTS is set sufficiently high in order to avoid max extents errors.

SAP 2009 / Page 15

Pitfalls and Problems - 5. Table Growth

5. Table Growth
Normally a reorganization will reduce the table size or at least keep it on the same level. Sometimes the size reduction is not as high as expected or there is even a size increase The most typical reasons are:
The reorganized table has no primary index. In this case a ROWID based reorganization rather than a PK constraint reorganization is done and a ROWID column has to be added to the target table. This column is hidden after the reorganization, but it still allocates space ( table increase up to 30 %) Typically happens with SAP conversion tables (QCM tables) and BW fact tables Should be avoided whenever possible (e.g. cleanup of QCM tables, MOVE based reorganization of BW fact tables) A large INITIAL_EXTENT value is used E.g. because of a former SAPDBA reorg with Compress Extents Use BRSPACE option "-l 2" to set the initial extent to 64K.
SAP 2009 / Page 16

Pitfalls and Problems - 6. Technical Settings

6. Technical Settings
The SAP technical settings contain a data class (TABART) that (TABART) maps a table to a tablespace:
Table DBSTATC DD09L TABART SPROT TAORA / IAORA Tablespace PSAPPROTD

The TABART is used whenever a table or index is newly created (e.g. during a transport of a new table or new index or during a system copy). If an existing table or index is modified, the current tablespace is used regardless of the TABART. Whenever you move tables and indexes to different tablespaces, you should consider the TABART settings. Otherwise segments may be created in wrong tablespaces. BRSPACE is TABART aware.

SAP 2009 / Page 17

Pitfalls and Problems - 6. Technical Settings

6. Technical Settings (2)

Recommendations:
Use the "-l" option to define an appropriate existing or new TABART whenever creating a new tablespace using "brspace -f tscreate": "-l <tabart>": New tabart <tabart> "-l all": All standard TABARTs "-l <source_tsp>": TABART of tablespace <source_tsp> Maintain TABARTs also in the other systems of the system landscape. TABART can point to different tablespaces in different systems. Follow SAP Notes 778784 and 777615 in order to avoid problems during system copies. In BW TABARTs are also maintained in special BW tables like RSDCUBE, RSDODSO, RSTS, RSDS and RSDCHABAS. These entries also have to be adapted. See SAP Note 771191 for more information.

SAP 2009 / Page 18

Pitfalls and Problems - 7. LONG2LOB

7. LONG2LOB Conversion
Tables with LONG and LONG RAW columns can't be reorganized online - unless an implicit LONG2LOB conversion is performed (BRSPACE parameter "-a long2lob") In general it is useful to convert to LOB columns, but there are some restrictions:
Oracle >= 10g, SAP >= 6.40 ORA-00932 errors possible on SAP side (for bugfix see SAP Note 1223716) ORA-00600 errors possible if PX is used (for bugfix see SAP Note 1292525) High water mark enqueues possible with LOBs in ASSM (for bugfix see SAP Note 1166242). Several LOB related DBSL problems in SAP 7.1x (for bugfixes see SAP Notes 1329293, 1269704, 1257979, 1226314 and 1171043) Small performance impact due to LOBs reported (see SAP Note 835552), can usually be neglected

SAP 2009 / Page 19

Pitfalls and Problems - 8. Parallelism

8. Parallelism Settings
Two possibilities of parallelism:
"-p <degree>": BRSPACE parallelism degree, reorganization of multiple different tables in parallel "-e <degree>": Oracle parallel execution (PX) degree, several Oracle PX slaves work on the reorganization of one table in parallel

Reorganization in parallel to production operation

System resources (I/O, CPU) mustn't be exceeded. Time is less critical Use no or a rather small parallelism setting (e.g. "-p 2 - e 2")

Reorganization in a dedicated time window with low or no parallel production operation

Use the system resources (I/O, CPU) as good as possible. Often I/O is the bottleneck and not CPU, but I/O capacities are often hard to determine. Starting point can be e.g. "-p 6 -e 4" for tablespace reorganizations or "-e 8" for individual table reorganization

SAP 2009 / Page 20

Pitfalls and Problems - 9. Invalid Objects

9. Invalid objects
Different kinds of invalid objects can result in errors and terminations during online reorganizations. Therefore the following should be checked in advance:
Make sure that no index or index partition of the tables to be reorganized is in status UNUSABLE. Perform a rebuild if required. Make sure that no invalid triggers exist on the tables to be reorganized (e.g. /BIC/05 BW triggers). Drop or validate these triggers. Make sure that no SYS objects exist in the Oracle DDIC with status = INVALID. Particularly important are packages like DBMS_REDEFINITION, DBMS_METADATA, DBMS_STATS and underlying objects. Make sure that the source tables are consistent (SAP Note 23345). As an example the REBUILD ONLINE bug described in SAP Note 1413928 can result in ORA-01452 (cannot CREATE UNIQUE INDEX; duplicate keys found) errors when creating the unique index on the target table and the whole table reorganization is terminated.

SAP 2009 / Page 21

Pitfalls and Problems - 10. DROP Performance

10. DROP Performance

Usually it is a matter of seconds (or less) to drop a table or an index. But: Under certain circumstances dropping the source table (or the source tablespace) can take longer than the whole reorganization. In real life we have already seen situations where a DROP of a single table took several days! The reason are typically space transactions in dictionary managed tablespaces - every formerly allocated extent needs to be freed individually. Due to an inadequate Oracle index design on the DDIC tables FET$ and UET$ the cleanup has a square complexity (4 times higher runtime for twice the number of extents). Even worse: As space transactions require the unique space transaction (ST) enqueue, other extent operations in the system may be blocked for a long time.

SAP 2009 / Page 22

Pitfalls and Problems - 10. DROP Performance

10. DROP Performance (2)

Recommendations
Check for critical segments in dictionary managed tablespaces with more than 5000 extents. The higher the number of extents, the more critical. More than 100.000 extents is definitly a big problem. If segments are returned, you have several options: Reduce the number of extents in smaller portions in advance (e.g. by copying and truncating the table with REUSE SPACE and doing ALTER TABLE DEALLOCATE UNUSED KEEP <size> with <size> decreasing over time) Reorganize the critical tables individually during certain defined timeframes. Test how long the DROP takes and provide a sufficient time window for the reorganization in production. In extreme situations it can even make sense to recreate the database with LMTS using R3load.

See SAP Note 745639 (section TYPE = ST) for more information.

SAP 2009 / Page 23

Thank you!

SAP 2009 / Page 24

Copyright 2008 SAP AG All rights reserved

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and associated logos displayed are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This document contains only intended strategies, developments, and functionalities of the SAP product and is not intended to be binding upon SAP to any particular course of business, product strategy, and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, or other items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, or non-infringement. SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitation shall not apply in cases of intent or gross negligence. The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in these materials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages

Weitergabe und Vervielfltigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrckliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen knnen ohne vorherige Ankndigung gendert werden. Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte knnen Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind. SAP, R/3, mySAP, mySAP.com, xApps, xApp, SAP NetWeaver, Duet, Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwhnte SAP-Produkte und Services sowie die dazugehrigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Lndern weltweit. Alle anderen in diesem Dokument erwhnten Namen von Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte knnen lnderspezifische Unterschiede aufweisen. Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderen Vereinbarung mit SAP. Dieses Dokument enthlt nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP-Produkts und ist fr SAP nicht bindend, einen bestimmten Geschftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP bernimmt keine Verantwortung fr Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht die Richtigkeit oder Vollstndigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewhr, weder ausdrcklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschlielich, hinsichtlich der Gewhrleistung der Marktgngigkeit und der Eignung fr einen bestimmten Zweck sowie fr die Gewhrleistung der Nichtverletzung geltenden Rechts. SAP bernimmt keine Haftung fr Schden jeglicher Art, einschlielich und ohne Einschrnkung fr direkte, spezielle, indirekte oder Folgeschden im Zusammenhang mit der Verwendung dieser Unterlagen. Diese Einschrnkung gilt nicht bei Vorsatz oder grober Fahrlssigkeit. Die gesetzliche Haftung bei Personenschden oder die Produkthaftung bleibt unberhrt. Die Informationen, auf die Sie mglicherweise ber die in diesem Material enthaltenen Hotlinks zugreifen, unterliegen nicht dem Einfluss von SAP, und SAP untersttzt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewhrleistungen oder Zusagen ber Internetseiten Dritter ab. Alle Rechte vorbehalten.

SAP 2009 / Page 25