DB2 Large Object (LOB) Datatype

Conversion Procedure
Technical Notice

July 2006

FileNet is a registered trademark of FileNet corporation. FileNet Corporation

All other product and brand names are trademarks or 3565 Harbor Boulevard
registered trademarks of their respective companies.
Costa Mesa, California 92626
Due to continuing product development, product
1.800.FILENET (345.3638)
specifications and capabilities are subject to change
without notice. Outside the US, call:

Copyright© 2006 FileNet Corporation. All rights reserved. 1.714.327.3400

www.filenet.com
 DB2 Large Object (LOB) Datatype Conversion Procedure
Contents

Contents
Revision Log ................................................................................................................................................. 3
Overview ....................................................................................................................................................... 4
Required Pre-Conversion Steps ................................................................................................................... 4
Load Procedure............................................................................................................................................. 6
INSERT Procedure ....................................................................................................................................... 7
Export / Import Procedure ............................................................................................................................. 8
Generating Table DDL with LOB Logging................................................................................................... 10
Affected Tables ........................................................................................................................................... 10

FileNet Corporation 2
 DB2 Large Object (LOB) Datatype Conversion Procedure
Revision Log

Revision Log
Date Revision
July 13, 2006 Initial document.

FileNet Corporation 3
 DB2 Large Object (LOB) Datatype Conversion Procedure
Overview

Overview
The CE DB2 database large object (LOB) column datatypes were previously created using the “not
logged” specifier due to an earlier DB2 limitation. Data in unlogged LOB columns are not saved in the
DB2 transaction log, which may result in a loss of LOB data added between backups if the database goes
down for any reason (see IBM's DB2 Data Recovery and High Availability Guide and Reference for
information on turning on transaction logging, and performing a Rollforward Recovery using the
transaction log). The procedure described in this document converts an existing DB2 CE object store
database, which may contain a large amount of data, into a database with all LOB columns “logged”.
This procedure is only necessary for IBM DB2 databases. DB2 8.1 FP9 (aka 8.2 FP2) plus IBM hot fix
patch 14068, or later FileNet-certified DB2 FixPak level is required.
Either before or after this conversion procedure is completed, but before the end-users can use the
system, the FileNet patch, which includes a fix for new LOB column creations to use a logged specifier,
must be applied to the affected Content Engine servers. If the database conversion is performed, but the
required CE patch is not applied, any new "long string" custom property created by a CE administrator
and added to a class results in the new associated column not being created as "logged". In this case,
the conversion procedure would need to be re-run for each new table column created. Therefore, the
FileNet patch that addresses this issue should be applied to the CE servers at the time that this
procedure is performed.
The previous LOB columns were of two types:
BLOB(2G) not logged
DBCLOB(1G) not logged
The BLOB type is used for many CE system columns; the size specifier (e.g. 2G) is just an upper limit.
The DBCLOB type is used for CE system columns as well as for user created “long string” custom
properties.
The new LOB columns will use these datatypes:
BLOB(1G) logged
DBCLOB(512M) logged
There is an upper limit of 1G for BLOB’s and 512M for DBCLOB’s with logging.
In order to accomplish the conversion, any one of the following methods can be used. The Load and
INSERT methods are the fastest. The Load method is the simplest and the method recommended by
FileNet.
• Load
• INSERT
• Export / Import

Required Pre-Conversion Steps

1. Backup the database
2. Inspect LOB column lengths to ensure that they will not be longer than the new lower limit. Fix each
LOB length that exceeds the maximum before proceeding by reconfiguring storage.
There should, quite generally, be no column values that break the new limit, except for the following
situations:
a. For large documents stored in the database (> 1GB), the content table, content column values
should be inspected

FileNet Corporation 4
 DB2 Large Object (LOB) Datatype Conversion Procedure
Required Pre-Conversion Steps

b. For large “long string” custom properties (> 512MB), the column values of the classes associated
table should be inspected
If there is a column value that is too long, the value must be changed. For example if it is a
document, try deleting it and re-add it in compressed form, or break it into parts, or store it in a
file-storage document class.
Here are some queries that can be used to inspect all LOB column value lengths.

For BLOB columns:

select <id>, <name> from <myTable> where length(<lobColumn>) > 1073741823

Alternatively, one can issue the following:

select max(length(<lobColumn>) from <myTable>

The <id>, <name>, <myTable>, <lobColumn> placeholders have to be replaced with the
desired values.

For example, for documents stored in the database:

select element_id, content_size from content where length(content) >
1073741823

For DBCLOB columns, repeat the above but use 536870912 (512M) for the cutoff.

For example, for a “long string” custom property that is called “MyLongStringProperty” that has an
associated column titled “u3f_ MyLongStringProperty”:

select object_id, u2f_documenttitle from docversion where length(u3f_

MyLongStringProperty) > 536870912

The prefixes (e.g. u2f, u3f, etc) can be determined from the database by issuing a “describe
<tablename>” command to list the columns, for example “describe docversion”.
The above example-queries could be performed against all LOB columns for all affected tables
(see the section below “Affected Tables” for a list of tables), although quite generally the main
concern is the content column of the content table (document content is stored there). It should
be known whether “long strings” > 512MB were populated.

3. Inspect the CE object store database’s tablespace user data container size to ensure that the
container is large enough to hold an additional table the size of the largest table in the database, or
hold copies of all tables if the originals will all be kept throughout the procedure (except for the Load
procedure). The conversion procedures typically create a new table within the same database
(although optionally one could use a new database) so there must be enough user space available.
4. Be sure that there is enough available disk space for the transaction logs
5. Take users offline, shutdown any CE applications accessing the object store and stop all CE Services

FileNet Corporation 5
 DB2 Large Object (LOB) Datatype Conversion Procedure
Load Procedure

Load Procedure
1. Before performing this procedure on a CE object store DB2 database, to ensure a successful
conversion, perform the Required Pre-Conversion Steps above.
2. Bring up the DB2 Alter Table dialog (optionally, one could perform this procedure using the
SYSPROC.ALTOBJ procedure via a script):
a. Open the DB2 Control Center (CC) (logging in as a user that has DB Administrator authority
(DBADM))
Note: Issues may exist because of configuration when running the Control Center from the CE
Server to the remote AIX database. In that case it can help to run the Control Center from the AIX
server using an X-Windows interface
b. Navigate to the database tables for the CE object store’s database
c. Right click on the first affected CE schema owner table (see the section below “Affected Tables”
for a list of tables)
d. Select “Alter…”, which brings up the dialog
3. From the “Alter Table” dialog, change each BLOB(2G) not logged datatype to be BLOB(1G), logged.
Change the DBCLOB(1G) not logged columns to DBCLOB(512M), logged:
a. From the Columns tab, select the first LOB column
b. Click on the “Change…” button
c. Set the length to be the maximum for the type (1G for BLOB, 512M for DBCLOB)
d. Check the “Logged” checkbox to enable logging
e. Click OK
f. Repeat the above steps for all LOB columns in the table
g. Click OK on the Alter table dialog to execute the conversion process for the table

Notes:
o According to the DB2 docs:
“Certain changes to the column definition of a existing column may require the table to be
dropped and recreated. DB2 will attempt to transform the existing data into the specified target
data type, and to restore related objects, such as aliases, views, triggers, materialized query
tables, constraints, and SQL stored procedures.”
o For any LOB column values too long (not fixed by the Required Pre-Conversion Steps above),
the behavior of the load is to continue processing the rows, but not insert the affected row, which
is not desirable.
o Note that DB2 recreates the indexes, primary key and statistics for you, although for
completeness you could recompute stats on tables and indexes when this completes.

FileNet Corporation 6
 DB2 Large Object (LOB) Datatype Conversion Procedure
INSERT Procedure

o Here is the process DB2 uses:

4. Repeat above steps for all affected tables

5. Repeat the above steps for each Object Store DB2 Database.

INSERT Procedure
1. Before performing this procedure on a CE object store DB2 database, to ensure a successful
conversion, perform the Required Pre-Conversion Steps above.
2. Generate the new DDL as described in the “Generating Table DDL with LOB Logging” section
below, for each affected table.
3. Rename each of the currently affected tables (e.g. DocVersion Î DocVersion_Orig). Truncate the
table name as necessary to keep the length within DB2 table name length limits.
4. Drop all indexes on the original tables, except for the primary key index.
5. Edit the DDL text file, optionally adding “not logged initially” to each table creation statement. “Not
logged” is only in effect within the transaction (before the commit) during which the table is first
created, or during the activity within a transaction that follows the issuing of an “alter table <table>
activate not logged initially” statement. At all other times the logging is on. It may be preferred to not
include the clause. Be sure that there is enough disk space for the transaction log files, as the insert
can be considerably slower with logging on.
For example:
CREATE TABLE "DB2User"."CONTENT" (
"ELEMENT_ID" VARGRAPHIC(46) NOT NULL,
"EXTENSION" VARGRAPHIC(3),
"CONTENT" BLOB(1G) LOGGED NOT COMPACT,
"CONTENT_SIZE" INTEGER NOT NULL,
"CI_SEQUENCE_NO" INTEGER NOT NULL)
IN "UserTBSP" NOT LOGGED INITIALLY;

6. The DDL text file also has all of the index creation statements. For a faster table INSERT, the non-
primary key and non-clustered indexes for the tables can be created after the INSERT’s complete.
This is optional, but if preferred, remove the index creation statements from the DDL text file. Do not
remove the primary key (alter table statement) and clustered index creation statements. Save the
regular index creation statements in another file for later execution.

FileNet Corporation 7
 DB2 Large Object (LOB) Datatype Conversion Procedure
Export / Import Procedure

7. Execute the DDL text file with the table, primary key and clustered index creation statements, against
the database, to create all of the tables (with LOB logging) and their primary keys and their clustered
indexes (where applicable).
8. For the following steps, the INSERT for each affected table can be done one at a time, committed,
then the original table dropped, repeating the steps for each table. That approach (for the larger
tables) will result in a savings on disk space usage.
Execute the following to INSERT the data from each affected table to the new table as created above:
a. From a db2 command window, run "db2 -t +c" to turn off autocommit and set the semicolon
statement separator.
b. Login to the database as the schema owner.
c. Execute the following from the db2 prompt (for example):
i. alter table docversion activate not logged initially;
ii. insert into docversion select * from docversion_orig;
iii. commit.
d. Drop the original table “drop table docversion_orig”, or save it for data comparison and validation.
e. Repeat the above two steps for each affected table.
Note:
ƒ If there is a column value that surpasses the new size limits (not fixed by the Required Pre-
Conversion Steps above), the INSERT statement fails and no rows are processed.
ƒ If saving the original tables for validation, be sure you have enough container and disk space
9. Compare the new table data to the original table for validation.
10. Drop the original affected tables if they were not dropped after each INSERT above.
11. If the regular indexes were saved in a separate script file, create them now by executing the script. If
keeping the original tables, be sure to drop the indexes on the original table, or rename them, to avoid
index name collision.
12. Run stats on all the new tables and indexes.
13. Repeat the above steps for each Object Store DB2 Database.

Export / Import Procedure

1. Before performing this procedure on a CE object store DB2 database, to ensure a successful
conversion, perform the Required Pre-Conversion Steps above.
2. Export the tables owned by the CE schema owner that have LOB columns (see the section below
“Affected Tables” for a list of tables), for example:
db2 export to d:\db2export\DocVersion of IXF lobs to d:\db2export\lobs lobfile
docverlobs modified by lobsinfile select * from DocVersion
Notes:
o It is important to re-use the same database. Otherwise the sequence object “CISEQUENCE”
would require transfer or re-creation, via the table SYSIBM.SYSSEQUENCES which is where the
sequence object is stored (as a row).
o IXF format must be used. The delimited format can cause errors to occur due to binary data
mixing with the column separator.
o The tables can be exported and then imported individually to save on disk space.

FileNet Corporation 8
 DB2 Large Object (LOB) Datatype Conversion Procedure
Export / Import Procedure

o Export / Import commands can be combined into a script if desired.

o Export / Import can also be done for each table from the DB2 Control Center –

Note: Issues may exist because of configuration when running the Control Center from the CE
Server to the remote AIX database. In that case it can help to run the Control Center from the
AIX server using an X-Windows interface.

From the Export Table dialog:

o Under the target tab:
1. specify IXF format;
2. set the output file and message file for the table.
o Under the columns tab:
1. specify the path for LOB data to be a directory under the output directory;
2. specify the base file name to find LOB data to be a name (e.g. docverlobs).

3. Generate the new DDL as described in the “Generating Table DDL with LOB Logging” section
below, for each table exported above. Be sure to save the entire DDL file including index creation
lines.
4. Drop the table(s) exported above, or rename the original table(s). Performing the export / import one
table at a time allows for data validation comparision between the old and new table. If the original
table(s) are renamed and kept, more container and disk space is necessary, and the original table
indexes should be dropped (or renamed) to avoid name collision when the indexes for the new table
are created in a step later in this procedure.
5. Create the new table(s) with the primary key index and the clustered index, if present, for the table(s)
from the DDL generated and edited as above. The non-primary key / non-clustered indexes for the
tables can be created after the import procedure for a faster import.
6. Import into the new table(s), for example:
IMPORT FROM "D:\DB2export\DocVersion" OF IXF LOBS FROM "D:\DB2export\lobs" MODIFIED
BY LOBSINFILE METHOD P (1, 2, …) MESSAGES "D:\DB2export\msgfile" INSERT INTO
<SchemaUser>.DocVersion (<column1>, <column2>, …);

Notes:
o The import command can be generated and executed from the Control Center, following the
dialog prompts as done above for export.
NOTE: Issues may exist because of configuration when running the Control Center from the CE
Server to the remote AIX database. In that case it can help to run the Control Center from the
AIX server using an X-Windows interface.
o For any column values too long (not fixed by the Required Pre-Conversion Steps), the behavior
of the Import is that it will fail the single row insert (each row is an individual INSERT) and
continue, logging a message in the message file, and summarizing the numbers of successful
and missing rows at the end. This is an undesirable situation when rows are missing; fields too
long should be fixed ahead of time.
o One can specify a COMMITCOUNT to avoid active log overflows. Or use the default of
Automatic commit, which tries to avoid running out of active log space by performing periodic
commits.

FileNet Corporation 9
 DB2 Large Object (LOB) Datatype Conversion Procedure
Generating Table DDL with LOB Logging

7. Check the message log files for any errors and perform any data validation between the original table
and the new table, and then you may drop the original tables.
8. Create the remaining non-primary key and non-clustered indexes for the imported table(s) from the
generated DDL, if they were not already created. If keeping the original table, be sure to drop the
indexes on the original table, or rename them, to table avoid index name collision.
9. Run statistics for all new tables and indexes
10. Repeat the above steps for each Object Store DB2 Database.

Generating Table DDL with LOB Logging

1. From the DB2 command prompt, issue this command. (Alternatively, one can Generate DDL from the
Control Center run on the AIX server.):
db2look –d <databasename> -t ANNOTATION DOCVERSION <etc> -a -e –c
Include all the affected tables. Output should be redirected to a file (append ">ddl.txt" to the above).
2. Modify the DDL in the text file, changing the LOB datatypes as follows:
For each affected table:
a. Change each BLOB(2G) not logged datatype to be BLOB(1G) logged;
b. Change the DBCLOB(1G) not logged columns to DBCLOB(512M) logged.
For example, this table DDL:
CREATE TABLE "DB2User"."CONTENT" (
"ELEMENT_ID" VARGRAPHIC(46) NOT NULL,
"EXTENSION" VARGRAPHIC(3),
"CONTENT" BLOB(2147483647) NOT LOGGED NOT COMPACT,
"CONTENT_SIZE" INTEGER NOT NULL,
"CI_SEQUENCE_NO" INTEGER NOT NULL)
IN "UserTBSP";

Should be changed to this:

CREATE TABLE "DB2User"."CONTENT" (
"ELEMENT_ID" VARGRAPHIC(46) NOT NULL,
"EXTENSION" VARGRAPHIC(3),
"CONTENT" BLOB(1G) LOGGED NOT COMPACT,
"CONTENT_SIZE" INTEGER NOT NULL,
"CI_SEQUENCE_NO" INTEGER NOT NULL)
IN "UserTBSP";

The BLOB datatype defaults to logged if logged is not specified.

Affected Tables
The following tables have, or can have, depending on the creation of user properties, LOB columns,
either BLOB’s or DBCLOB’s and are owned by the DB2 schema owner for the CE object store.
Annotation
DocVersion
Container
Content
FileNet Corporation 10
 DB2 Large Object (LOB) Datatype Conversion Procedure
Affected Tables

Event
Generic
GlobalPropertyDef
Link
ListOfBinary
PropertyDefinition
QueueItem
Relationship
SecurityDesc
SecurityPolicy
Subscription
WorkQueue

All tables can be checked for LOB columns, if desired, for completeness.

FileNet Corporation 11