This action might not be possible to undo. Are you sure you want to continue?
The data to bemigrated was real-time and required fairly large data transformation. The basic use of the migration was to keep an older production 9i database in sync with the newer 10g database for several months while applications dependant on this datawere remediated to use the new database. This paper and the accompanying presentation are based on the proof of concept(POC) we performed to prove this functionality.The POC would use Oracle streams, Change Data Capture (CDC) specifically, to capture the data. That data was thenavailable to other 10g databases but not to 9i databases. To enable us to send the data to 9i, we enqueued the captured datainto an AQ queue that was then propagated to a 9i database. Once the data was in 9i, it was a simple process to dequeue thedata and apply it.Below is the short list for setting up Change Data Capture using Oracle Streams. These steps are mostly from the docs with afew tweaks I have added. Following the setup of CDC is a discussion of using AQ to move the data from 10g to 9i. The paper concludes with a detailed discussion of creating your own manual Logical Change Record (LCR). Change Data Capture First the set up: we will use the HR account's Employee table. We'll capture all changes to the Employee table and insert theminto an audit table. I'm not necessarily saying this is the way you should audit your database but it makes a nice example.I'll also add a monitoring piece to capture process. I want to be able to see exactly what is being captured when it is beingcaptured.You will need to have sysdba access to follow along with me. Your database must also be in archivelog mode. The changesare picked up from the redo log.So, away we go!The first step is to create our streams administrator. I will follow the guidelines from the oracle docs exactly for this:Connect as sysdba: sqlplus / as sysdba Create the streams tablespace (change the name and/or location to suit): create tablespace streams_tbs datafile 'c:\temp\stream_tbs.dbf' size 25M reuse autoextend on maxsize unlimited; Create our streams administrator: create user strmadmin identified by strmadmindefault tablespace streams_tbsquota unlimited on streams_tbs; I haven't quite figured out why, but we need to grant our administrator DBA privs. I think this is a bad thing. There is probably a work around where I could do some direct grants instead but I haven't had time to track those down. grant dba to strmadmin; We also want to grant streams admin privs to the user.
this step creates the queue and theunderlying ANYDATA table.department_id NUMBER(4). Unlike AQ.queue_name => 'strmadmin./ This just defines that we want to capture DML and not DDL. END. manager_id NUMBER(6).2).job_id VARCHAR2(10). CREATE TABLE employee_audit(employee_id NUMBER(6). We connect as the streams admin user: conn strmadmin/strmadmin We can create a logging table. You would NOT want to do this in a highvolume production system. We also need to create the employee_audit table.action VARCHAR2(30)).streams_name => 'capture_emp'.employee_audit to strmadmin. . I am doing this to illustrate user defined monitoring and show how you can get inside the capture process.inclusion_rule => true).streams_type => 'capture'.first_name VARCHAR2(20).employees'.user_name VARCHAR2(30).email VARCHAR2(25).commission_pct NUMBER(2.employees to strmadmin.include_dml => true. BEGIN DBMS_STREAMS_ADM. Note that I am adding three columns in this table that do not exist in theemployee table.upd_date DATE.queue_name => 'strmadmin. grant_privileges => true). conn hr/hr Grant all access to the employee table to the streams admin: grant all on hr.streams_queue').BEGIN DBMS_STREAMS_AUTH.2).hire_date DATE. Grant all access to the audit table to the streams admin user: grant all on hr./ The next steps we'll run as the HR user.last_name VARCHAR2(25). CREATE TABLE streams_monitor (date_and_time TIMESTAMP(6) DEFAULT systimestamp.streams_queue'.include_ddl => false. BEGIN DBMS_STREAMS_ADM.SET_UP_QUEUE(queue_table => 'strmadmin.salary NUMBER(8.streams_queue_table'. Here we create the queue.txt_msg CLOB ).GRANT_ADMIN_PRIVILEGE(grantee => 'strmadmin'. phone_number VARCHAR2(20). END. where you have to create a separate table.ADD_TABLE_RULES(table_name => 'hr.
S ET_TABLE_INSTANTIATION_SCN(source_object_name => 'hr. ANYDATA.-Set the command_type in the row LCR to INSERTlcr.GET_EXTRA_ATTRIBUTE('USERNAME') ).END.-.SET_VALUES('new'.GETOBJECT(lcr).attribute_name => 'username'. NULL)./ Tell the capture process that we want to know who made the change: BEGINDBMS_CAPTURE_ADM.SET_OBJECT_NAME('EMPLOYEE_AUDIT').-.instantiation_scn => iscn).DBMS_APPLY_ADM.commit.ADD_COLUMN('new'.I am inserting the XML equivalent of the LCR into the monitoring table.END./ And the fun part! This is where we define our capture procedure. -. old_values).ADD_COLUMN('new'.Get the object command typecommand := lcr.-.SET_VALUES('old'.ConvertDate(SYSDATE)).GET_SYSTEM_CHANGE_NUMBER().BEGIN-.GET_COMMAND_TYPE().-.END.END.Set the old values in the row LCR to the new values in the row LCR lcr. I'm taking this right from the docs but I'm adding a couplesteps.Add an action columnlcr.Get the old values in the row LCR old_values := lcr.rc PLS_INTEGER.Set the new values to the old values for update and deleteIF command IN ('DELETE'.GET_VALUES('old'). Change the source_database_name to match your database. 'UPDATE') THEN-.Set the object_name in the row LCR to EMP_DELlcr.lcr.-.ADD_COLUMN('new'.Add a SYSDATE for upd_datelcr.Add a user columnlcr..command VARCHAR2(30).-.INCLUDE_EXTRA_ATTRIBUTE(capture_name => 'capture_emp'.ConvertVarChar2(command)). CREATE OR REPLACE PROCEDURE emp_dml_handler(in_any IN ANYDATA) ISlcr SYS. DECLAREiscn NUMBER.CONVERT_LCR_TO_XML(in_any) ).EXECUTE(true).Make the changeslcr.BEGINiscn := DBMS_FLASHBACK.SET_COMMAND_TYPE('INSERT'). 'user_name'. 'UPD_DATE'.END IF.Access the LCR rc := in_any./ Create the DML handlers: .employees'.-.old_values SYS./ We also need to tell Oracle where to start our capture.insert into streams_monitor (txt_msg) values (command ||DBMS_STREAMS.Set the old values in the row LCR to NULLlcr.-.include => true). 'ACTION'.source_database_name => 'ORCL'. ANYDATA.LCR$_ROW_LIST.LCR$_ROW_RECORD.
'07-JUN-94'.apply_database_link => NULL. 'JOHN'.employees'.DBMS_APPLY_ADM.operation_name => 'DELETE'.user_procedure => 'strmadmin. The second calls tells streamswhere to put the info.ddl_rule_name => emp_rule_name_ddl). value => 'n').streams_type => 'apply'.'JSMITH@MYCOMPANY.SET_ENQUEUE_DESTINATION(rule_nam e => emp_rule_name_dml. sqlplus hr/hrINSERT INTO hr.employees SET salary=5999 WHERE ./ Turn on the apply process: BEGINDBMS_APPLY_ADM.streams_queue'.COMMIT.object_type => 'TABLE'. NULL. Change the source_database_name to match your database.SET_DML_HANDLER(object_name => 'hr.emp_dml_handler'. so: BEGINDBMS_APPLY_ADM.SET_PARAMETER(apply_name => 'apply_emp'.apply_name => NULL).END. 110).object_type => 'TABLE'.employees'.END.apply_name => NULL).user_procedure => 'strmadmin.UPDATE hr. 777./ We don't want to stop applying changes when there is an error. NULL. This tells streams.SET_DML_HANDLER(object_name => 'hr./ Connect as HR and make some changes to Employees.START_APPLY(apply_name => 'apply_emp').END.source_database => 'ORCL'. 'AC_ACCOUNT'. 'SMITH'.user_procedure => 'strmadmin./BEGINDBMS_APPLY_ADM.BEGINDBMS_STREAMS_ADM.END.include_dml => true./ Create the apply rule.employees VALUES(207./BEGINDBMS_APPLY_ADM.employees'.destination_queue_name => 'strmadmin.apply_name => NULL). yet again. DECLAREemp_rule_name_dml VARCHAR2(30).error_handler => false.include_ddl => false.emp_dml_handler'.employees'.COM'.streams_name => 'apply_emp'.operation_name => 'UPDATE'.SET_DML_HANDLER(object_name => 'hr.emp_rule_name_ddl VARCHAR2(30).streams_queue').ADD_TABLE_RULES(table_name => 'hr.END.END.operation_name => 'INSERT'. that we in fact do want to capture changes.START_CAPTURE(capture_name => 'capture_emp').apply_database_link => NULL.emp_dml_handler'. parameter => 'disable_on_error'. NULL.dml_rule_name => emp_rule_name_dml.object_type => 'TABLE'.error_handler => false.BEGINDBMS_APPLY_ADM.queue_name => 'strmadmin.END.apply_database_link => NULL./ Turn on the capture process: BEGINDBMS_CAPTURE_ADM.error_handler => false.
START_CAPTURE(capture_name => 'capture_emp').employee_auditORDER BY employee_id.COMMIT. parameter => 'disable_on_error'.employees WHERE employee_id=207. it's a little bit more work tocross database instances.COMMIT.UPDATE hr. actionFROM hr. first_name. NULL.DELETE FROM hr.END./ Connect as HR and make some changes to Employees. It takes a few seconds for the data to make it to the logs and then back into the system to be applied. sqlplus hr/hrINSERT INTO hr. 'AC_ACCOUNT'.employees SET salary=5999 WHERE employee_id=206./ Turn on the apply process: BEGINDBMS_APPLY_ADM. It takes a few seconds for the data to make it to the logs and then back into the system to be applied. upd_Date. but it's not that much.END. BEGINDBMS_APPLY_ADM./ Turn on the capture process: BEGINDBMS_CAPTURE_ADM. actionFROM hr. 777.employees VALUES(207.COMMIT. upd_Date.employees WHERE employee_id=207.employee_id=206.employee_auditORDER BY employee_id. NULL.START_APPLY(apply_name => 'apply_emp').COM'. The less code I have to write. 110). 'SMITH'. Of course. 'JOHN'. the less code I have to maintain. value => 'n'). first_name.DELETE FROM hr. Then you can log back into the streams admin account: .END. NULL.COMMIT. last_name. Then you can log back into the streams admin account: sqlplus strmadmin/strmadmin View the XML LCR that we inserted during the capture process: set long 9999set pagesize 0select * from streams_monitor.One of the things that amazes me is how little code is required to accomplish this.'JSMITH@MYCOMPANY. Run this query until yousee data (remembering that it is not instantaneous): SELECT employee_id.SET_PARAMETER(apply_name => 'apply_emp'. That's it! It's really not that much work to capture and apply changes using Oracle 10g. '07-JUN-94'.COMMIT. Run this query until yousee data (remembering that it is not instantaneous): SELECT employee_id. last_name.
You need to create an AQ table.sqlplus strmadmin/strmadmin View the XML LCR that we inserted during the capture process: set long 9999set pagesize 0select * from streams_monitor.One of the things that amazes me is how little code is required to accomplish this.e. it's a little bit more work tocross database instances. Download and install EE.You need two Oracle instances for this.grant dba to strmadmin. the less code I have to maintain. select * from global_name. you can download EE and use the DBCA to run two database instances. It works for me!As I said. AQ 10g to 9i NOTE: Streams is not available with Oracle 10g XE. i. The code below is mostly thesame code that you ran on ORCL.CREATE_QUEUE_TABLE(queue_table => 'lrc_emp_t'. You should already have the CDC codefrom the last article running in ORCL.ORCL must be in archivelog mode to run CDC. use global_name. BEGINDBMS_AQADM. That's what the code below does. SECOND does not need archivelog mode. I used a 10g instance as my source (required) and a 9i database as my target (youcould also use a 10g instance here).dbf' size 25M reuse autoextend on maxsize unlimited.You already created your streams admin user in ORCL so now do the same thing in SECOND.create user strmadmin identified by strmadmindefault tablespace streams_second_tbsquota unlimited on streams_second_tbs. You do not physically need two machines to getthis to work. Having two databases running ona single PC in archivelog mode can really beat up a poor IDE drive. If you are not using the same names for your databases and you are not sure of theexact name of your databases (including domain). I made a few minor changes in case you are running both instances on a single PC: sqlplus / as sysdbacreate tablespace streams_second_tbs datafile 'c:\temp\stream_2_tbs. Of course. Connect as strmadmin.compatible . The less code I have to write. I called my first instance ORCL (how creative!) and I called my second instanceSECOND.anydata'. but it's not that much. AQ queue and then start the queue. That's it! It's really not that much work to capture and apply changes using Oracle 10g.multiple_consumers => TRUE. If you have 1 GB or more of RAM on your PC. ORCL will be my source instance and SECOND will be my target instance.queue_payload_type => 'sys.
Ok.lrc_emp_q'.insert into streams_monitor (txt_msg) values (command ||DBMS_STREAMS.lrc_emp_q@second.END. We are going to modify the EMP_DML_HANDLER (from above) so that weget an enqueue block just above the execute statement: CREATE OR REPLACE PROCEDURE emp_dml_handler(in_any IN ANYDATA) ISlcr SYS.include_ddl => FALSE.Get the object command typecommand := lcr.DBMS_AQADM.GETOBJECT(lcr).destination_queue_name => 'strmadmin.world CONNECT TO strmadminIDENTIFIED BY strmadminUSING 'second.1').queue_table => 'lrc_emp_t').END. Most of the setup for this is exactly the same betweenthe two instances.include_dml => true.old_values SYS.employees'.START_QUEUE (queue_name => 'lrc_emp_q'). You have to have one from ORCL to SECOND but for debugging. create a link: CREATE DATABASE LINK orcl.GET_COMMAND_TYPE()./ You also need to create a database link. now we have running queues in ORCL and SECOND.world CONNECT TO strmadminIDENTIFIED BY strmadminUSING 'orcl.world').ADD_TABLE_PROPAGATION_RULES(table_name => 'hr. Create your link on this side also. BEGINDBMS_STREAMS_ADM.rc PLS_INTEGER.LCR$_ROW_LIST.Access the LCR rc := in_any.streams_name => 'orcl_2_second'.command VARCHAR2(30). Log into ORCL as strmadmin and run the exact same command there.world'. you will create a propagationschedule.CREATE_QUEUE(queue_name => 'lrc_emp_q'.-.LCR$_ROW_RECORD. I like a link in both.I am inserting the XML equivalent of the LCR into the monitoring table.world'.We're almost done with the propagation now.-- .source_database => 'orcl.BEGIN-. So.source_queue_name => 'strmadmin. While you are logged into ORCL.world'.-.CONVERT_LCR_TO_XML(in_any) ).=> '8. CREATE DATABASE LINK second./ This tells the database to take the data in the local lrc_emp_q and send it to the named destination queue. You DO NOT need to run this in SECOND. while you're in SECOND.DBMS_AQADM.
-. recipients DBMS_AQ.-.enqueue_options => enqueue_options.payload => anydata.ADD_COLUMN('new'.BEGIN recipients(1) := sys.message_properties_t.SET_VALUES('new'..ConvertVarChar2(command)).Set the object_name in the row LCR to EMP_DELlcr. We will use that name to dequeue the record in the SECOND instance.message_properties DBMS_AQ.-.employees table and commit them.message_handle RAW(16).FORMAT_ERROR_STACK ).lcr. Then log into strmadmin@second andselect * from the lrc_emp_t table./ The declaration section above created some variables required for an enqueue.Set the old values in the row LCR to the new values in the row LCR lcr.commit.-.SET_COMMAND_TYPE('INSERT').lrc_emp_q@second.We then enqueued our LCR as an ANYDATA datatype.ConvertDate(SYSDATE)).That's it for moving the data from 10g to 9i.That's all it takes.There are not a lot of moving parts so there aren't many things that will go wrong.GET_VALUES('old'). old_values). Propagation is where I have the mosttroubles. You should have as many records there as you inserted.msgid => message_handle). -.SET_VALUES('old'. 'UPD_DATE'. NULL).DBMS_AQ. DECLARE enqueue_options DBMS_AQ.ADD_COLUMN('new'.Make the changeslcr.message_p roperties. . 'ACTION'. Insert some records into the HR.END. 'user_name'. You can query DBA_PROPAGATION to see if you have any propagation errors.GET_EXTRA_ATTRIBUTE('USERNAME') ). ANYDATA.lrc_emp_q'.ADD_COLUMN('new'.'strmadmin. 'UPDATE') THEN-.Set the command_type in the row LCR to INSERTlcr.ENQUEUE( queue_name => 'strmadmin.-.EXECUTE(true).Set the new values to the old values for update and deleteIF command IN ('DELETE'.message_properties => message_properties.SET_OBJECT_NAME('EMPLOYEE_AUDIT').world'.aq$_agent( 'anydata_subscriber'.Add an action columnlcr.Add a user value for the timestamp columnlcr.I put the exception handler there in case there are any problems with our enqueue.END.NULL).-.enqueue_options_t. We created a subscriber (that's the name of theconsumer).aq$_recipient_list_t.convertObject(lcr).recipient_list := recipients.Get the old values in the row LCR old_values := lcr.EXCEPTION WHEN OTHERS THEN insert into streams_monitor (txt_msg)values ('Anydata: ' || DBMS_UTILITY.Set the old values in the row LCR to NULLlcr.Add a SYSDATE value for the timestamp columnlcr. ANYDATA.END IF.
NOTE: Everything below is specifically for a ROW type LCR as opposed to a DDL type LCR. It is very handy if you need to manually create test datathough.You can do a describe to see the methods as well as view the documentation.com/docs/cd/B14117_01/server. I (and a coworker) searched the web and looked through various documentation but was not able to find a concisedescription of how to go about creating an LCR manually. and command type) at the toplevel. First a little definition. I recently had the need to create some test datain LCR format. The LCR has information about what the object is as well as theold and new values. it might look something like: Figure 1 . it is two table collections of an object type embedded within another object type.101/b10727/ap_xmlsc. Normally. If you have manual control of your LCR.Manual Creation of an LCR This part of the paper is not required to move data from 10g to 9i.-----.LCR Structure Excuse the poor diagram.oracle. Beneath that is column and data information in a name/value pair collection.htm&remark=portal+%28Application+development%29The short story is that an LCR stored object level information (database.Anyway.----------------------. You canget the details of that from the documentation in thesupplied PL/SQL Packages and Types documentation. owner. they chose to name it CONSTRUCT. Actually. SQL> desc sys. http://downloadeast.I tend to think of things like this in a relational format.oracle.--------SOURCE_DATABASE_NAME VARCHAR2 IN .102%2 Fb14258%2Ftoc. a constructor has the same name as the object type. you can test specific data issues a lot easier.lcr$_row_record METHOD------STATIC FUNCTION CONSTRUCT RETURNS LCR$_ROW_RECORDArgument Name Type In/Out Default?----------------------------. In this case. The old and new values are exactly the same as :old and :new in a trigger. The concepts would be thesame but the specificcode would change. If I put it in database terms. An LCR format is the format of data that Oracle uses in the redo logs and is used for Oracle Streams(and probably data guard although I am guessing about that).You can see the definition of the LCR format by viewingthe LCR XML Schema. An artiste I am not.http://www.com/pls/db102/to_toc?pathname=appdev.htmAn LCR is an object type. The important thing to note is the constructor. So I decided to write one. object_name.
The important thing to note is the constructor.Just for your info. Normally.--------SOURCE_DATABASE_NAME VARCHAR2 IN COMMAND_TYPE VARCHAR2 INOBJECT_OWNER VARCHAR2 INOBJECT_NAME VARCHAR2 INTAG RAW IN DEFAULTTRANSACTION_ID VARCHAR2 IN DEFAULTSCN NUMBER IN DEFAULTOLD_VALUES LCR$_ROW_LIST IN DEFAULTNEW_VALUES LCR$_ROW_LIST IN DEFAULT Based on that info.You can do a describe to see the methods as well as view the documentation.-----. populating the test LCR is relatively straight-forward.com9 K a l 2 0 0 O e 7 i D d T o U s G c o p e Excuse the poor diagram. An artiste I am not. they chose to name it CONSTRUCT. the type LCR$_ROW_LIST is a collection of LCR$_ROW_UNIT. a constructor has the same name as the object type.----------------------. SQL> desc sys.lcr$_row_record METHOD------STATIC FUNCTION CONSTRUCT RETURNS LCR$_ROW_RECORDArgument Name Type In/Out Default?----------------------------.odtug. In this case.www. .
s et_object_owner(p_table_owner).lcr$_row_record ASv_lcr sys.v_lcr.command_type => p_command.WHEN 'DATE'THENv_lcr.add_column('new'. CREATE OR REPLACE FUNCTION create_lcr( p_table_owner IN all_tables.lcr$_row_record.add_column('old'. 'UPDATE'. .v_database global_name.-.You can override the values in the constructor by calling these methods v_lcr.END IF.convertVarChar2(TO_CHAR(NULL))).You would expand this for all data types -.v_lcr. c1. c1.AnyData.set_object_name(p_table_name).column_name.owner%TYPE.AnyData.global_name%TYPE.AnyData.Loop through the columns and add new and old valuesFOR c1 IN (SELECT column_name. That type has it's ownrules and deserves a blog entry all to itself.Construct the LCR v_lcr := sys.v_lcr.object_name => p_table_name ).Get the database name-.v_lcr.p_command IN VARCHAR2 )RETURN sys. sys.One thing to remember is that the data values that you are dealing with are sys.object_owner => p_table_owner.column_name. 'DELETE')THENRETURN v_lcr.Those types are also documented in thereference guide I mentioned above.verify the command typeIF p_command NOT IN ('INSERT'.table_name%TYPE. sys. c1.lcr$_row_record. c1.v_lcr. You will not want to access those directly though.BEGIN-. data_typeFROM all_tab_columnsWHERE owner = p_table_ownerAND table_name = p_table_nameORDER BY column_id )LOOP-.AnyData data types.construct(source_database_name => v_database. Once you have the LCR you can modifythe values to suit.p_table_name IN all_tables.convertDate(TO_DATE(NULL))).add_column('old'.add_column('new'.set_source_database_name(v_database).column_name.set_command_type(p_command).column_name. You can use the builtinLCR$_ROW_RECORD methods to populate those fields.Here is a function that will create an empty LCR for you automatically for any table.data_typeWHEN 'VARCHAR2'THENv_lcr. sys.Create an anydata based on column data type.I'm going to keep this example fairly simpleCASE c1.This could be parameterized SELECT global_nameINTO v_databaseFROM global_name.-.convertVarChar2(TO_CHAR(NULL))).-.-.
END. sys.set_value('new'.column_name. 'EMPLOYEES'. Old First Name: ' || sys.column_name.-. 'first_name')) ||'. Object Name: EMPLOYEES. .convertDate(TO_DATE(NULL))).lcr$_row_record.BEGIN v_lcr := create_lcr( 'HR'.END LOOP.END. Object Name: ' || v_lcr.get_command_type() ).sys.Display Some ValuesDBMS_OUTPUT.anyData. 'INSERT' ).anyData.v_lcr.get_object_name() ||'.v_lcr. Object Owner: HR.add_column('old'. c1. 'first_name')) ). Object Owner: ' || v_lcr. 'first_name'. sys.PUT_LINE('Database: ' || v_lcr.get_source_database_name() ||'.ittoolbox. sys. c1.-.Thank you for taking the time to read this paper. Command: ' || v_lcr. Old First Name: George And that's all there is to it.AnyData.AnyData.anyData.convertVarchar2('Lewis')).add_column('new'. you might write something like the following: DECLAREv_lcr sys. Command: INSERT New First Name: Lewis.accessVarchar2(v_lcr. To call this function and manipulate it.get_value('old'.com or come by andread my blog.END CASE.DBMS_OUTPUT.get_value('new'. An Expert's Guide to Oracle Technology. sys.PUT_LINE('New First Name: ' || sys.set_value('old'.get_object_owner() ||'.convertVarchar2('George')).accessVarchar2(v_lcr. http://blogs.RETURN v_lcr.Set some valuesv_lcr.anyData.convertNumber(TO_NUMBER(NULL))).AnyData. You can contact me in the future at lewisc@rocketmail. The output from this is: Database: XE.WHEN 'NUMBER'THENv_lcr.convertNumber(TO_NUMBER(NULL))).com/oracle/guide. 'first_name'.
This action might not be possible to undo. Are you sure you want to continue?