Can I do Incremental Exports?

Backup and Recovery Tips

Can I do Incremental Exports?

You most certainly can, but you should be aware that Oracle has publicly declared that this facility will disappear from future versions of Oracle (though its still there in 9i). Its not something that will reliably work into the future, therefore. But first, the basics. The way to perform an incremental export is to use the INCTYPE parameter, which accepts three different values: COMPLETE, CUMULATIVE or INCREMENTAL. Incidentally, this parameter can only be supplied when you are doing full database exports (i.e., where FULL=Y), though you can specify FULL=Y without specifying any of these parameters. The rules as to what gets included in each of these types of export can be tricky to put into words, though it can be summarised quite simply. Ill give you the technical details first, and the easy summary at the end!

The Rules of Inclusion

If youre doing incremental or cumulative exports, the export utility has to work out what tables should be included in the latest export, and which should be ignored (because their contents havent changed). How it works out what has changed and what hasnt is instructive. Whenever you specify an INCTYPE parameter, export updates a special table in the SYS schema, called INCEXP. If youve never performed an INCTYPE export before, querying that table is something of a disapointment:
SELECT

FROM SYS.INCEXP;

NO ROWS SELECTED

But if you query it immediately after performing an INCTYPE=COMPLETE export, youll see something that looks like this:
SELECT

FROM SYS.INCEXP;

OWNER# -----5 5 5 66

NAME TYPE# CTIME ITIME EXPID ------------------------- ---------- --------- --------- ----AQ$_INTERNET_AGENTS 2 13/NOV/01 13/NOV/01 1 AQ$_INTERNET_AGENT_PRIVS 2 13/NOV/01 13/NOV/01 1 DEF$_AQCALL 2 13/NOV/01 13/NOV/01 1 EMP 2 13/NOV/01 13/NOV/0 1

Notice in particular the two time columns: CTIME means included in a cumulative export and ITIME means included in an incremental export. Obviously, a COMPLETE export updates both of these columns (since it includes all tables), and you can therefore regard a COMPLETE as being a superset of both incremental and cumulative exports.
Copyright Howard Rogers 2001 15/11/2001 Page 1 of 10

Can I do Incremental Exports?

Backup and Recovery Tips

Let us suppose that I now perform DML on the EMP table, and a day later seek to perform an INCTYPE=CUMULATIVE export. What does the INCEXP table now show?
OWNER# NAME TYPE# CTIME ITIME EXPID ------ ------------------------- ---------- --------- --------- ----66 EMP 2 14/NOV/01 14/NOV/01 3

Again, both columns are updated. Now, another day passes, and I again perform some DML on the table, and perform an INCTYPE=INCREMENTAL export. The INCEXP table now shows this:
OWNER# NAME TYPE# CTIME ITIME EXPID ------ ------------------------- ---------- --------- --------- ----66 EMP 2 14/NOV/01 15/NOV/01 4

Notice this time that only the ITIME column is updated, not the CTIME one. Finally, suppose I now perform a final FULL=Y export, without specifying any INCTYPE (again, a day later, and again after performing DML on the EMP table). What does the table show then?
OWNER# NAME TYPE# CTIME ITIME EXPID ------ ------------------------- ---------- --------- --------- ----66 EMP 2 14/NOV/01 15/NOV/01 4

So, a FULL=Y doesnt update this table at all. Bear these rules in mind, therefore: COMPLETE exports update both time columns. CUMULATIVE exports update both time columns. INCREMENTALs only update the ITIME column. And a basic FULL=Y with no INCTYPE at all updates nothing.

Since the contents of this INCEXP table determine which tables get included in the next export that is run with one of the INCTYPE parameters, these rules are significant. Obviously, a COMPLETE export includes everything. Its therefore functionally equivalent to a FULL=Y, but does update the INCEXP table so performing a new COMPLETE affects what future incrementals and cumulatives will contain, but performing a new FULL=Y doesnt. A CUMULATIVE export causes us to check each tables SCN (timestamp) against the CTIME column of the INCEXP system table. If the tables SCN is greater than the CTIME date, then the table is included in the new export. Since both COMPLETE and CUMULATIVE exports update the CTIME column, tables modified since the last complete or cumulative export are included in a new cumulative export. But since INCREMENTAL exports dont touch the CTIME column, the existence of intervening incremental exports is irrelevant as
Copyright Howard Rogers 2001 15/11/2001 Page 2 of 10

Can I do Incremental Exports?

Backup and Recovery Tips

far as a subsequent cumulative export is concerned: the table gets exported anyway, even if its been included in 20 intervening incremental exports. On the other hand, if youre performing an INCREMENTAL export, we compare each tables SCN with the ITIME column of the INCEXP table and since COMPLETE, CUMULATIVE and INCREMENTAL exports all update that column, then the rule must be that a new incremental export will include any table that has been modified since the time of the last export of any of these kinds. Most importantly, since a FULL=Y export with no INCTYPE parameter specified doesnt touch any part of the INCEXP table, the existence of such exports is totally irrelevant to what gets included in the next export that does specify an INCTYPE. For the purposes of INCTYPE exports, and working out what tables they should include, its as though the FULL exports had never happened.

Summary of the Rules of Inclusion

Trying to put that into very simple English, we deduce the following: FULL=Y includes all objects and does not affect the contents of future exports at all COMPLETE includes all objects and does affect future cumulatives and incrementals CUMULATIVE includes objects modified since the last cumulative or complete INCREMENTAL includes objects modified since the last incremental, cumulative or complete export

Object-level exports
Two important points need to be made here: first, the cumulative and incremental exports include objects that have changed. Not parts of objects. Not just the new rows added since the last export. But the entire table, cluster, index or whatever the object might be. Second, both DML and DDL constitute a change for the purposes of determining whether a particular object should be included in a new export. So you might issue this sort of command, for example: C:\>EXP
SYSTEM/MANAGER FULL=Y INCTYPE=INCREMENTAL ON

EXPORT: RELEASE 9.0.1.1.1 - PRODUCTION

FRI

NOV

16 12:42:49 2001

(followed by a list of all the objects being exported)

. . . . . .

EXPORTING TABLE EXPORTING TABLE EXPORTING TABLE

S_INVENTORY S_ITEM S_LONGTEXT

114 ROWS EXPORTED 62 ROWS EXPORTED 33 ROWS EXPORTED

and so on.
Copyright Howard Rogers 2001 15/11/2001 Page 3 of 10

Can I do Incremental Exports?

Backup and Recovery Tips

If I then perform an update to just a few rows in the S_INVENTORY table, and follow that up immediately with a new incremental backup, Ill see this: SQL> UPDATE S_INVENTORY 4 ROWS UPDATED.
SET AMOUNT_IN_STOCK=0 WHERE PRODUCT_ID=41100;

C:\>EXP SYSTEM/MANAGER FULL=Y INCTYPE=INCREMENTAL [SNIP] . EXPORTING TABLESPACE DEFINITIONS . EXPORTING PROFILES [SNIP] S_INVENTORY . . EXPORTING TABLE

114

ROWS EXPORTED

and youll notice that the same 114 rows as before get included in the new export, not just the 4 that I updated. Thats because export can only ever grab entire objects, so naturally all the rows in a table go along for the ride.

Uses of Incremental and Cumulative Exports

I suppose the inevitable question at this point is: why bother? Why is this functionality useful? Well, cumulative and incremental exports only include a subset of all the possible objects in the database. Objects that havent been subject to DML or DDL are ignored on second and subsequent exports. That means that the export dump files are considerably smaller than they otherwise would be, of course. It also means that the exports themselves take much less time to complete than a full database export would require. However, for precisely that same reason, the use of these types of export poses a number of unique problems. For a start, it means that you can now no longer be certain of what is included within any given dump file. If a User suddenly announces that the EMP table has gone missing, you cant know for sure whether it can be recovered from Mondays, Tuesdays or whatevers export. You could, of course, dash off to query the SYS.INCEXP table and work it out, but that involves some work and some mental agility to convert ITIME and CTIME columns into meaningful results. Its not impossible to do, but its definitely trickier than simply having a single dump file thats guaranteed to be complete. The other problem with this sort of export is that, because each dump file that is produced is not a complete export of the database, you have to keep all the partial files generated available youd need the entire set to re-construct an entire database. To avoid excessive numbers of such files being required, every so often you should perform a COMPLETE export. That way, you only need to retain incremental and cumulative dump files generated since the time of the last COMPLETE one.

Copyright Howard Rogers 2001

15/11/2001

Page 4 of 10

Can I do Incremental Exports?

Backup and Recovery Tips

Individual Table Imports

The real killer with incremental exports, though, is what happens during subsequent imports. Lets keep things simple at this stage (because it only gets worse!). Suppose you have a Sunday complete export, followed by a Monday incremental, and then on Tuesday the EMP table gets accidentally dropped. The temptation is going to be to do what youd probably do with normal system tape backups: restore from the Sunday complete, and then apply the Monday incremental to pick up new changes. But with export/import, that way disaster lies because the Sunday and the Monday dump files both include a complete copy of the EMP table. If you therefore import from Sunday, a fully-populated version of EMP is created. When you come to apply the incremental from Monday, the import will fail, because the object it wants to create (EMP) will already exist. So you might at that point remember to run the second import with the IGNORE=Y parameter at which point, what happens to your data is in the lap of the gods: Suppose you dont have a primary key on the EMP table. Then the second import will duplicate every single row that was already created by the import from Sundays dump file! Now suppose you do have a primary key defined. Then the second import will generate a string of constraint violation messages, followed by the insertion of any rows that were freshly created between the two exports. But what about any deletes that a User performed before the EMP table disappeared on Tuesday. Are those deletes re-performed for us? No: those rows were inserted by performing the first import, and the second import does not contain instructions to delete records, only to insert new ones. So rows that were deleted are back again after performing the imports. What about rows that were updated on Monday? Well, the original values were restored by performing the first import, and the second import didnt touch that row because of the primary key constraint issues so all updates are lost too! I can demonstrate that as follows. On Sunday, the EMP table looked like this: ID ---------1 2 3 4 5 6 7 8 9 10 ENAME MANAGER_ID DEPT_ID SALARY ------------------------- ---------- ---------- ---------BENJAMIN BRITTEN 50 2500 WOLFGANG MOZART 1 41 1450 FELIX MENDELSSOHN 1 31 1400 LUDWIG BEETHOVEN 1 10 1450 GUSTAV MAHLER 1 50 1550 DMITRI SHOSTAKOVICH 2 41 1200 EDWARD ELGAR 2 42 1250 HENRY PURCELL 2 43 1100 AARON COPLAND 2 44 1300 LEONARD BERNSTEIN 2 45 1307

This table does have a primary key, on the ID column. That evening, I perform a complete export:
Copyright Howard Rogers 2001 15/11/2001 Page 5 of 10

Can I do Incremental Exports?

Backup and Recovery Tips

C:\>

EXP SYSTEM/MANAGER INCTYPE=COMPLETE FULL=Y FILE=EXPSUN.DAT

and the export display shows me: . .

EXPORTING TABLE

EMP

10

ROWS EXPORTED

On Monday, I perform the following three bits of DML:

DELETE FROM EMP WHERE ENAME=EDWARD ELGAR; INSERT INTO EMP VALUES (11, SERGEI RACHMANINOV, UPDATE EMP SET SALARY=3000 WHERE ID=1;

3, 41,950);

COMMIT; That evening, I perform an incremental export: C:\>

EXP SYSTEM/MANAGER INCTYPE=INCREMENTAL FULL=Y FILE=EXPMON.DAT

and again the export display shows me: . .

EXPORTING TABLE

EMP

10

ROWS EXPORTED

On Tuesday morning, we have a slight accident:

DROP TABLE EMP;

So I import from the Sunday export: C:\>IMP

SYSTEM/MANAGER TABLES=EMP IGNORE=Y FILE=EXPSUN.DAT FROMUSER=SCOTT ON

IMPORT: RELEASE 9.0.1.1.1 - PRODUCTION

FRI OCT 26 13:27:33 2001

RIGHTS RESERVED.

(C) COPYRIGHT 2001 ORACLE CORPORATION. ALL

CONNECTED TO: ORACLE9I ENTERPRISE EDITION RELEASE 9.0.1.1.1 - PRODUCTION WITH THE PARTITIONING OPTION JSERVER RELEASE 9.0.1.1.1 - PRODUCTION EXPORT FILE CREATED BY EXPORT:V09.00.01 VIA CONVENTIONAL PATH IMPORT DONE IN WE8MSWIN1252 CHARACTER SET AND AL16UTF16 NCHAR CHARACTER SET IMPORT SERVER USES CL8MSWIN1251 CHARACTER SET (POSSIBLE CHARSET CONVERSION) . IMPORTING SCOTT'S OBJECTS INTO SCOTT . . IMPORTING TABLE "EMP" 10 ROWS IMPORTED IMPORT TERMINATED SUCCESSFULLY WITHOUT WARNINGS. Looking good! Now I try to capture the Monday changes by importing from the incremental export:
Copyright Howard Rogers 2001 15/11/2001 Page 6 of 10

Can I do Incremental Exports?

Backup and Recovery Tips

C:\>IMP

SYSTEM/MANAGER TABLES=EMP IGNORE=Y FILE=EXPMON.DAT FROMUSER=SCOTT ON

IMPORT: RELEASE 9.0.1.1.1 - PRODUCTION

FRI OCT 26 13:28:55 2001

RIGHTS RESERVED.

(C) COPYRIGHT 2001 ORACLE CORPORATION. ALL

CONNECTED TO: ORACLE9I ENTERPRISE EDITION RELEASE 9.0.1.1.1 - PRODUCTION WITH THE PARTITIONING OPTION JSERVER RELEASE 9.0.1.1.1 - PRODUCTION EXPORT FILE CREATED BY EXPORT:V09.00.01 VIA CONVENTIONAL PATH IMPORT DONE IN WE8MSWIN1252 CHARACTER SET AND AL16UTF16 NCHAR CHARACTER SET IMPORT SERVER USES CL8MSWIN1251 CHARACTER SET (POSSIBLE CHARSET CONVERSION) . IMPORTING SCOTT'S OBJECTS INTO SCOTT . . IMPORTING TABLE "EMP" IMP-00019: ROW REJECTED DUE TO ORACLE ERROR 1 IMP-00003: ORACLE ERROR 1 ENCOUNTERED ORA-00001: UNIQUE CONSTRAINT (SCOTT.EMP_PK) VIOLATED COLUMN 1 2 COLUMN 2 WOLFGANG MOZART COLUMN 3 1 COLUMN 4 41 COLUMN 5 1450 IMP-00019: ROW REJECTED DUE TO ORACLE ERROR 1 IMP-00003: ORACLE ERROR 1 ENCOUNTERED ORA-00001: UNIQUE CONSTRAINT (SCOTT.EMP_PK) VIOLATED [SNIP
MUCH MORE OF THE SAME]

COLUMN 5 1307 IMP-00019: ROW REJECTED DUE TO ORACLE ERROR 1 IMP-00003: ORACLE ERROR 1 ENCOUNTERED ORA-00001: UNIQUE CONSTRAINT (SCOTT.EMP_PK) COLUMN 1 1 COLUMN 2 BENJAMIN BRITTEN COLUMN 3 COLUMN 4 50 COLUMN 5 3000 1 ROWS IMPORTED IMPORT TERMINATED SUCCESSFULLY WITH WARNINGS. None of which looks quite so good!

VIOLATED

Now when we finally get to look at the contents of the EMP table, we see this:

Copyright Howard Rogers 2001

15/11/2001

Page 7 of 10

Can I do Incremental Exports?

Backup and Recovery Tips

SQL>

SELECT

FROM EMP;

ID ---------2 3 4 5 6 7 8 9 10 1 11 11

ENAME MANAGER_ID DEPT_ID SALARY ---------------------- ---------- ---------- ---------WOLFGANG MOZART 1 41 1450 FELIX MENDELSSOHN 1 31 1400 LUDWIG BEETHOVEN 1 10 1450 GUSTAV MAHLER 1 50 1550 DMITRI SHOSTAKOVICH 2 41 1200 EDWARD ELGAR 2 42 1250 HENRY PURCELL 2 43 1100 AARON COPLAND 2 44 1300 LEONARD BERNSTEIN 2 45 1307 BENJAMIN BRITTEN 50 2500 SERGEI RACHMANINOV 3 1 950

ROWS SELECTED.

Britten is still there with his old salary the update is ignored. Mr. Rachmaninov has been inserted the insert is respected. Mr. Elgar is still sitting there, composing dreadful music deletes are ignored. In short: Its a mess. The reason of course is that the inclusion of complete objects in a dump file, whatever the nature of the export, means that the approach of import from full and apply incrementals is completely wrong. All you need do is import from the last export taken that happens to include the object you want. In this particular example, all we need do is import from the Monday Incremental backup, since that contains all the latest updates, deletes and inserts: C:\>IMP
SYSTEM/MANAGER TABLES=EMP IGNORE=Y FILE=EXPMON.DAT FROMUSER=SCOTT ON

IMPORT: RELEASE 9.0.1.1.1 - PRODUCTION

FRI OCT 26 13:32:44 2001

RIGHTS RESERVED.

(C) COPYRIGHT 2001 ORACLE CORPORATION. ALL

CONNECTED TO: ORACLE9I ENTERPRISE EDITION RELEASE 9.0.1.1.1 - PRODUCTION WITH THE PARTITIONING OPTION JSERVER RELEASE 9.0.1.1.1 - PRODUCTION EXPORT FILE CREATED BY EXPORT:V09.00.01 VIA CONVENTIONAL PATH IMPORT DONE IN WE8MSWIN1252 CHARACTER SET AND AL16UTF16 NCHAR CHARACTER SET IMPORT SERVER USES CL8MSWIN1251 CHARACTER SET (POSSIBLE CHARSET CONVERSION) . IMPORTING SCOTT'S OBJECTS INTO SCOTT . . IMPORTING TABLE "EMP" 10 ROWS IMPORTED IMPORT TERMINATED SUCCESSFULLY WITHOUT WARNINGS.
Copyright Howard Rogers 2001 15/11/2001 Page 8 of 10

Can I do Incremental Exports?

Backup and Recovery Tips

and a check of the EMP table reveals: SQL>

SELECT

FROM EMP;

ID ---------2 3 4 5 6 8 9 10 1 11 10

ENAME MANAGER_ID DEPT_ID SALARY ------------------------- ---------- ---------- ---------WOLFGANG MOZART 1 41 1450 FELIX MENDELSSOHN 1 31 1400 LUDWIG BEETHOVEN 1 10 1450 GUSTAV MAHLER 1 50 1550 DMITRI SHOSTAKOVICH 2 41 1200 HENRY PURCELL 2 43 1100 AARON COPLAND 2 44 1300 LEONARD BERNSTEIN 2 45 1307 BENJAMIN BRITTEN 50 3000 SERGEI RACHMANINOV 3 41 950

ROWS SELECTED.

Mr. Elgar has gone (and good riddance, too). Mr. Britten is being paid what hes worth. And Mr. Rachmaninov makes his expected appearance. All inserts, updates and deletes are therefore being accounted for. In summary, when it comes time to using incremental exports to recover, you start with the latest export and work backwards. As soon as the table is recovered, you can stop you dont need to go any further back. And all this behaviour is as a result of the fact that exports always export complete objects, not individual rows. Apart from remembering to do imports in what is perhaps a non-intuitive way, it also means that you need to be extremely aware of what tables managed to get included in what exports otherwise, youll find yourself running multiple imports against dump files that dont include the relevant table, and wasting time in the process.

Complete Database Imports

Things get rather worse when you try and import a complete database using incremental exports. The order of events becomes critical, because you have to get the data dictionary in the right shape before attempting to recover the actual tables and their data. In fact, Id go so far as to say that its practically impossible to pull off by mere mortals. What you are supposed to do (in this order) is: Import from the most recent incremental or cumulative export, specifying INCTYPE=SYSTEM as one of the import parameters. This gets users, object types, and so on correct, but doesnt import any actual tables.

Copyright Howard Rogers 2001

15/11/2001

Page 9 of 10

Can I do Incremental Exports?

Backup and Recovery Tips

Next, you import from the most recent complete export file, specifying INCTYPE=RESTORE. Then you import all cumulative export files made after the last complete export, again specifying INCTYPE=RESTORE. Then you import all incremental export files made after the last cumulative export, yet again specifying INCTYPE=RESTORE And allegedly that does the deed, with one tiny proviso: the Oracle documentation states unequivocally that you should only perform this sort of restore when the database being imported into has NO user tables whatsoever within it. Thats because the INCTYPE parameter can only be specified when you are performing a full database import, not specific tables or specific schemas. The presence of any tables in the database as you attempt such an import is therefore liable to cause the import process to fail. Note, however, that the recovery process using this technique does actually follow the restore from complete then apply incrementals sequence that you might have assumed to be the normal mode of operation in the first place. That frees you up from worrying about which dump file to use as the basis of a restore, since you simply start with the last complete export, and progress forwards in sequence with all subsequent incrementals. The only slight twist to that is to apply the last incremental first with an INCTYPE=SYSTEM just to get the data dictionary in shape. Even with that in mind, though, it does mean you can perform restores without being intimately familiar with the contents of your various dump files.

Conclusion
Having said all of that, what are the benefits and costs associated with incremental exports? The benefits are easy to state: Your exports take less time, and the dump files are smaller. The costs are significant, too: Importing either requires you to work your way backwards, starting with the latest export, until you happen to restore the right table. Or you have to keep excellent records about which tables got included in which export then you can go straight to the latest export known to contain the table you want. The alternative roll forward technique (start with a complete, and apply all subsequent cumulatives and incrementals) requires a database with no existing tables to work relaibly, which makes it of rather specialised use. It also requires multiple import runs, with great care being required to specify the correct INCTYPE each time. Frankly, the costs are high: imports become extremely fiddly, however you elect to do them. And, for me, that means they will generally outweigh the benefits. So the real answer to the question posed right at the beginning of this paper, Can I do Incremental Exports? is Yes, but you probably shouldnt bother.
Copyright Howard Rogers 2001 15/11/2001 Page 10 of 10