This action might not be possible to undo. Are you sure you want to continue?
Lewis R Cunningham AKA LewisC, PricewaterhouseCoopers, LLP Introduction
Recently, I had a project to migrate data from an Oracle 10g EE database to an Oracle 9i EE database. The data to be migrated was real-time and required fairly large data transformation. The basic use of the migration was to keep an older production 9i database in sync with the newer 10g database for several months while applications dependant on this data were remediated to use the new database. This paper and the accompanying presentation are based on the proof of concept (POC) we performed to prove this functionality. The POC would use Oracle streams, Change Data Capture (CDC) specifically, to capture the data. That data was then available to other 10g databases but not to 9i databases. To enable us to send the data to 9i, we enqueued the captured data into an AQ queue that was then propagated to a 9i database. Once the data was in 9i, it was a simple process to dequeue the data and apply it. Below is the short list for setting up Change Data Capture using Oracle Streams. These steps are mostly from the docs with a few tweaks I have added. Following the setup of CDC is a discussion of using AQ to move the data from 10g to 9i. The paper concludes with a detailed discussion of creating your own manual Logical Change Record (LCR).
Change Data Capture
First the set up: we will use the HR account's Employee table. We'll capture all changes to the Employee table and insert them into an audit table. I'm not necessarily saying this is the way you should audit your database but it makes a nice example. I'll also add a monitoring piece to capture process. I want to be able to see exactly what is being captured when it is being captured. You will need to have sysdba access to follow along with me. Your database must also be in archivelog mode. The changes are picked up from the redo log. So, away we go! The first step is to create our streams administrator. I will follow the guidelines from the oracle docs exactly for this: Connect as sysdba:
sqlplus / as sysdba
Create the streams tablespace (change the name and/or location to suit):
create tablespace streams_tbs datafile 'c:\temp\stream_tbs.dbf' size 25M reuse autoextend on maxsize unlimited;
Create our streams administrator:
create user strmadmin identified by strmadmin default tablespace streams_tbs quota unlimited on streams_tbs;
I haven't quite figured out why, but we need to grant our administrator DBA privs. I think this is a bad thing. There is probably a work around where I could do some direct grants instead but I haven't had time to track those down.
grant dba to strmadmin;
We also want to grant streams admin privs to the user.
BEGIN DBMS_STREAMS_AUTH.GRANT_ADMIN_PRIVILEGE( grantee => 'strmadmin',
ODTUG Kaleidoscope 2007
END.odtug. include_ddl => false.ADD_TABLE_RULES( table_name => 'hr. CREATE TABLE employee_audit( employee_id NUMBER(6).streams_queue'). Cunningham END.2). Grant all access to the audit table to the streams admin user: grant all on hr. We connect as the streams admin user: conn strmadmin/strmadmin We can create a logging table.streams_queue_table'. upd_date DATE. Here we create the queue. / This just defines that we want to capture DML and not DDL.Step-By-Step Streams grant_privileges => true). where you have to create a separate table. include_dml => true. department_id NUMBER(4). queue_name => 'strmadmin. conn hr/hr Grant all access to the employee table to the streams admin: grant all on hr. hire_date DATE. queue_name => 'strmadmin. phone_number VARCHAR2(20). BEGIN DBMS_STREAMS_ADM. salary NUMBER(8. I am doing this to illustrate user defined monitoring and show how you can get inside the capture process.com 2 ODTUG Kaleidoscope 2007 . this step creates the queue and the underlying ANYDATA table.SET_UP_QUEUE( queue_table => 'strmadmin. We also need to create the employee_audit table. manager_id NUMBER(6). action VARCHAR2(30)). inclusion_rule => true). job_id VARCHAR2(10). CREATE TABLE streams_monitor ( date_and_time TIMESTAMP(6) DEFAULT systimestamp. commission_pct NUMBER(2. txt_msg CLOB ). streams_type => 'capture'.employees to strmadmin. first_name VARCHAR2(20).employee_audit to strmadmin. email VARCHAR2(25). user_name VARCHAR2(30).streams_queue'. / The next steps we'll run as the HR user. www. last_name VARCHAR2(25).employees'. Unlike AQ.2). streams_name => 'capture_emp'. You would NOT want to do this in a high-volume production system. Note that I am adding three columns in this table that do not exist in the employee table. BEGIN DBMS_STREAMS_ADM.
END IF. old_values SYS. / And the fun part! This is where we define our capture procedure. END.employees'. ANYDATA. rc PLS_INTEGER.Get the old values in the row LCR old_values := lcr. 'ACTION'.ConvertVarChar2(command)).LCR$_ROW_RECORD. DECLARE iscn NUMBER.Get the object command type command := lcr.Add a SYSDATE for upd_date lcr.GET_SYSTEM_CHANGE_NUMBER().INCLUDE_EXTRA_ATTRIBUTE( capture_name => 'capture_emp'. DBMS_APPLY_ADM. instantiation_scn => iscn). source_database_name => 'ORCL'. BEGIN -. CREATE OR REPLACE PROCEDURE emp_dml_handler(in_any IN ANYDATA) IS lcr SYS.SET_COMMAND_TYPE('INSERT').Step-By-Step Streams END. -.GET_VALUES('old'). www. attribute_name => 'username'. / Cunningham Tell the capture process that we want to know who made the change: BEGIN DBMS_CAPTURE_ADM.Set the command_type in the row LCR to INSERT lcr. -.com 3 ODTUG Kaleidoscope 2007 .Set the old values in the row LCR to NULL lcr.Access the LCR rc := in_any.ADD_COLUMN('new'.CONVERT_LCR_TO_XML(in_any) ).Add an action column lcr. -. -. -. Change the source_database_name to match your database.SET_TABLE_INSTANTIATION_SCN( source_object_name => 'hr.SET_VALUES('new'.SET_VALUES('old'. 'UPD_DATE'. 'user_name'. old_values). lcr. -.Set the object_name in the row LCR to EMP_DEL lcr. 'UPDATE') THEN -.GET_COMMAND_TYPE(). -.odtug.LCR$_ROW_LIST.Set the old values in the row LCR to the new values in the row LCR lcr. -.Add a user column lcr.ADD_COLUMN('new'.SET_OBJECT_NAME('EMPLOYEE_AUDIT').GETOBJECT(lcr). NULL). ANYDATA. command VARCHAR2(30). BEGIN iscn := DBMS_FLASHBACK. / We also need to tell Oracle where to start our capture.ConvertDate(SYSDATE)). END.GET_EXTRA_ATTRIBUTE('USERNAME') ).I am inserting the XML equivalent of the LCR into the monitoring table. include => true). -.Set the new values to the old values for update and delete IF command IN ('DELETE'. insert into streams_monitor (txt_msg) values (command || DBMS_STREAMS.ADD_COLUMN('new'. I'm taking this right from the docs but I'm adding a couple steps. -.
user_procedure => 'strmadmin. streams_name => 'apply_emp'.Step-By-Step Streams -. END. The second calls tells streams where to put the info.streams_queue'). dml_rule_name => emp_rule_name_dml.SET_ENQUEUE_DESTINATION( rule_name => emp_rule_name_dml. apply_database_link => NULL. error_handler => false. apply_database_link => NULL. / BEGIN DBMS_APPLY_ADM.emp_dml_handler'. apply_name => NULL). DECLARE emp_rule_name_dml emp_rule_name_ddl VARCHAR2(30).emp_dml_handler'. object_type => 'TABLE'.Make the changes lcr.EXECUTE(true). error_handler => false. / Cunningham Create the DML handlers: BEGIN DBMS_APPLY_ADM. that we in fact do want to capture changes. error_handler => false. include_ddl => false. apply_database_link => NULL. user_procedure => 'strmadmin.com 4 ODTUG Kaleidoscope 2007 . This tells streams.employees'. END.SET_DML_HANDLER( object_name => 'hr. END. destination_queue_name => 'strmadmin. / We don't want to stop applying changes when there is an error. source_database => 'ORCL'.ADD_TABLE_RULES( table_name => 'hr. user_procedure => 'strmadmin. queue_name => 'strmadmin.emp_dml_handler'. VARCHAR2(30).SET_DML_HANDLER( object_name => 'hr. / BEGIN DBMS_APPLY_ADM. BEGIN DBMS_STREAMS_ADM. so: www. operation_name => 'UPDATE'.employees'.streams_queue'. operation_name => 'INSERT'. object_type => 'TABLE'. operation_name => 'DELETE'. include_dml => true. yet again. Change the source_database_name to match your database. apply_name => NULL). DBMS_APPLY_ADM. commit. END. END.employees'. apply_name => NULL).odtug.SET_DML_HANDLER( object_name => 'hr. ddl_rule_name => emp_rule_name_ddl). object_type => 'TABLE'. / Create the apply rule.employees'. streams_type => 'apply'.
Then you can log back into the streams admin account: sqlplus strmadmin/strmadmin View the XML LCR that we inserted during the capture process: set long 9999 set pagesize 0 select * from streams_monitor. 777. END.employee_audit ORDER BY employee_id. END. The less code I have to write. NULL. '07-JUN-94'. but it's not that much. COMMIT. COMMIT. It takes a few seconds for the data to make it to the logs and then back into the system to be applied. NULL. last_name. it's a little bit more work to cross database instances. 'JSMITH@MYCOMPANY.employees WHERE employee_id=207.Step-By-Step Streams Cunningham BEGIN DBMS_APPLY_ADM.com 5 ODTUG Kaleidoscope 2007 . Run this query until you see data (remembering that it is not instantaneous): SELECT employee_id. DELETE FROM hr. END. action FROM hr.employees SET salary=5999 WHERE employee_id=206. COMMIT. UPDATE hr. 'JOHN'. www. / Turn on the apply process: BEGIN DBMS_APPLY_ADM. / Connect as HR and make some changes to Employees.employees VALUES(207. parameter => 'disable_on_error'. 'SMITH'. value => 'n'). / Turn on the capture process: BEGIN DBMS_CAPTURE_ADM. the less code I have to maintain. That's it! It's really not that much work to capture and apply changes using Oracle 10g. first_name.COM'.odtug. Of course. One of the things that amazes me is how little code is required to accomplish this.START_APPLY( apply_name => 'apply_emp').START_CAPTURE( capture_name => 'capture_emp'). 110). upd_Date. sqlplus hr/hr INSERT INTO hr. NULL. 'AC_ACCOUNT'.SET_PARAMETER( apply_name => 'apply_emp'.
Step-By-Step Streams Cunningham AQ 10g to 9i NOTE: Streams is not available with Oracle 10g XE. I used a 10g instance as my source (required) and a 9i database as my target (you could also use a 10g instance here). ORCL must be in archivelog mode to run CDC. I like a link in both.dbf' size 25M reuse autoextend on maxsize unlimited.world CONNECT TO strmadmin IDENTIFIED BY strmadmin USING 'orcl. ORCL will be my source instance and SECOND will be my target instance. grant dba to strmadmin.1'). END.world'. You need two Oracle instances for this. AQ queue and then start the queue. DBMS_AQADM. I made a few minor changes in case you are running both instances on a single PC: sqlplus / as sysdba create tablespace streams_second_tbs datafile 'c:\temp\stream_2_tbs.odtug. The code below is mostly the same code that you ran on ORCL. queue_payload_type => 'sys. create user strmadmin identified by strmadmin default tablespace streams_second_tbs quota unlimited on streams_second_tbs. That's what the code below does. You have to have one from ORCL to SECOND but for debugging. SECOND does not need archivelog mode. compatible => '8.CREATE_QUEUE( queue_name => queue_table => 'lrc_emp_q'. You need to create an AQ table. If you have 1 GB or more of RAM on your PC. It works for me! As I said.START_QUEUE ( queue_name => 'lrc_emp_q'). multiple_consumers => TRUE. 'lrc_emp_t'). You should already have the CDC code from the last article running in ORCL. So.com 6 ODTUG Kaleidoscope 2007 .e. BEGIN DBMS_AQADM. I called my first instance ORCL (how creative!) and I called my second instance SECOND. Log into ORCL as strmadmin and run the exact same command there. while you're in SECOND. Connect as strmadmin. Having two databases running on a single PC in archivelog mode can really beat up a poor IDE drive. you can download EE and use the DBCA to run two database instances. You do not physically need two machines to get this to work. You already created your streams admin user in ORCL so now do the same thing in SECOND. DBMS_AQADM. i. Download and install EE. / You also need to create a database link. create a link: CREATE DATABASE LINK orcl. www. select * from global_name.CREATE_QUEUE_TABLE( queue_table => 'lrc_emp_t'. Most of the setup for this is exactly the same between the two instances. use global_name. If you are not using the same names for your databases and you are not sure of the exact name of your databases (including domain).anydata'.
/ This tells the database to take the data in the local lrc_emp_q and send it to the named destination queue. lcr. BEGIN -. While you are logged into ORCL. now we have running queues in ORCL and SECOND. command VARCHAR2(30). BEGIN DBMS_STREAMS_ADM. We are going to modify the EMP_DML_HANDLER (from above) so that we get an enqueue block just above the execute statement: CREATE OR REPLACE PROCEDURE emp_dml_handler(in_any IN ANYDATA) IS lcr SYS. -.GET_COMMAND_TYPE().SET_VALUES('old'.SET_COMMAND_TYPE('INSERT'). ANYDATA. -. -.enqueue_options_t.ADD_COLUMN('new'.employees'.lrc_emp_q'.SET_OBJECT_NAME('EMPLOYEE_AUDIT'). destination_queue_name => 'strmadmin.Get the object command type command := lcr. END IF.Add an action column lcr.SET_VALUES('new'.world CONNECT TO strmadmin IDENTIFIED BY strmadmin USING 'second.LCR$_ROW_RECORD.world'.world'. old_values SYS. -. 'UPD_DATE'.LCR$_ROW_LIST. -. streams_name => 'orcl_2_second'.Add a user value for the timestamp column lcr.Step-By-Step Streams Create your link on this side also.Access the LCR rc := in_any. old_values). -.Get the old values in the row LCR old_values := lcr.odtug.com 7 ODTUG Kaleidoscope 2007 . 'user_name'. NULL).message_properties_t.CONVERT_LCR_TO_XML(in_any) ). -. DBMS_AQ.Set the old values in the row LCR to NULL lcr. www.world').GETOBJECT(lcr).ADD_COLUMN('new'.ConvertDate(SYSDATE)). source_queue_name => 'strmadmin. -.lrc_emp_q@second.Add a SYSDATE value for the timestamp column lcr. RAW(16). rc PLS_INTEGER.GET_EXTRA_ATTRIBUTE('USERNAME') ). source_database => 'orcl.Set the new values to the old values for update and delete IF command IN ('DELETE'. -. DECLARE enqueue_options message_properties message_handle DBMS_AQ.ConvertVarChar2(command)). Cunningham Ok.I am inserting the XML equivalent of the LCR into the monitoring table. include_ddl => FALSE.Set the object_name in the row LCR to EMP_DEL lcr.GET_VALUES('old'). -. We're almost done with the propagation now. you will create a propagation schedule. END. insert into streams_monitor (txt_msg) values (command || DBMS_STREAMS.ADD_TABLE_PROPAGATION_RULES( table_name => 'hr. 'ACTION'.Set the command_type in the row LCR to INSERT lcr. 'UPDATE') THEN -. ANYDATA. CREATE DATABASE LINK second.ADD_COLUMN('new'. include_dml => true.Set the old values in the row LCR to the new values in the row LCR lcr. You DO NOT need to run this in SECOND.
You can query DBA_PROPAGATION to see if you have any propagation errors.aq$_recipient_list_t. You should have as many records there as you inserted.ENQUEUE( queue_name => enqueue_options => message_properties => payload => msgid => EXCEPTION WHEN OTHERS THEN insert into streams_monitor values ('Anydata: ' || END.EXECUTE(true).world'. Then log into strmadmin@second and select * from the lrc_emp_t table. enqueue_options.lrc_emp_q'.aq$_agent( 'anydata_subscriber'.com 8 ODTUG Kaleidoscope 2007 . That's it for moving the data from 10g to 9i. BEGIN recipients(1) := sys. DBMS_AQ. That's all it takes.lrc_emp_q@second. I put the exception handler there in case there are any problems with our enqueue. message_properties. Insert some records into the HR. We will use that name to dequeue the record in the SECOND instance. / Cunningham 'strmadmin. The declaration section above created some variables required for an enqueue. anydata. There are not a lot of moving parts so there aren't many things that will go wrong.recipient_list := recipients. We then enqueued our LCR as an ANYDATA datatype. 'strmadmin. -.employees table and commit them.convertObject(lcr). END. www. message_properties.Step-By-Step Streams recipients DBMS_AQ.Make the changes lcr. We created a subscriber (that's the name of the consumer). (txt_msg) DBMS_UTILITY.odtug. Propagation is where I have the most troubles. commit. NULL).FORMAT_ERROR_STACK ). message_handle).
So I decided to write one. it might look something like: Figure 1 .lcr$_row_record METHOD -----STATIC FUNCTION CONSTRUCT RETURNS LCR$_ROW_RECORD Argument Name Type In/Out Default? -----------------------------. and command type) at the top level.----------------------. An LCR format is the format of data that Oracle uses in the redo logs and is used for Oracle Streams (and probably data guard although I am guessing about that).com/docs/cd/B14117_01/server. I tend to think of things like this in a relational format. It is very handy if you need to manually create test data though. SQL> desc sys.101/b10727/ap_xmlsc. First a little definition. An artiste I am not. it is two table collections of an object type embedded within another object type.102%2Fb14258%2Ftoc. The important thing to note is the constructor. Actually.htm&remark=portal+%28Application+d evelopment%29 The short story is that an LCR stored object level information (database.LCR Structure Excuse the poor diagram. http://www.com 9 ODTUG Kaleidoscope 2007 . If you have manual control of your LCR. The LCR has information about what the object is as well as the old and new values. object_name. In this case. http://downloadeast. You can do a describe to see the methods as well as view the documentation. You can get the details of that from the documentation in the supplied PL/SQL Packages and Types documentation.-----. Beneath that is column and data information in a name/value pair collection. owner.Step-By-Step Streams Cunningham Manual Creation of an LCR This part of the paper is not required to move data from 10g to 9i.odtug. I recently had the need to create some test data in LCR format.oracle. I (and a coworker) searched the web and looked through various documentation but was not able to find a concise description of how to go about creating an LCR manually. a constructor has the same name as the object type. NOTE: Everything below is specifically for a ROW type LCR as opposed to a DDL type LCR.oracle. The old and new values are exactly the same as :old and :new in a trigger. you can test specific data issues a lot easier. The concepts would be the same but the specific code would change. You can see the definition of the LCR format by viewing the LCR XML Schema.-------SOURCE_DATABASE_NAME VARCHAR2 IN www. they chose to name it CONSTRUCT. If I put it in database terms. Normally.htm An LCR is an object type. Anyway.com/pls/db102/to_toc?pathname=appdev.
odtug.verify the command type IF p_command NOT IN ('INSERT'. That type has it's own rules and deserves a blog entry all to itself.lcr$_row_record AS v_lcr sys. You can use the built-in LCR$_ROW_RECORD methods to populate those fields.set_object_owner(p_table_owner).owner%TYPE. data_type FROM all_tab_columns WHERE owner = p_table_owner AND table_name = p_table_name ORDER BY column_id ) LOOP -.Create an anydata based on column data type -.You would expand this for all data types www. You will not want to access those directly though. Here is a function that will create an empty LCR for you automatically for any table.This could be parameterized SELECT global_name INTO v_database FROM global_name.Step-By-Step Streams COMMAND_TYPE OBJECT_OWNER OBJECT_NAME TAG TRANSACTION_ID SCN OLD_VALUES NEW_VALUES VARCHAR2 VARCHAR2 VARCHAR2 RAW VARCHAR2 NUMBER LCR$_ROW_LIST LCR$_ROW_LIST IN IN IN IN IN IN IN IN Cunningham DEFAULT DEFAULT DEFAULT DEFAULT DEFAULT Based on that info. p_command IN VARCHAR2 ) RETURN sys. populating the test LCR is relatively straight-forward.set_command_type(p_command).Get the database name -.Loop through the columns and add new and old values FOR c1 IN ( SELECT column_name. v_lcr. -. p_table_name IN all_tables. -. object_name => p_table_name ). object_owner => p_table_owner. One thing to remember is that the data values that you are dealing with are sys. v_lcr. CREATE OR REPLACE FUNCTION create_lcr( p_table_owner IN all_tables. -. v_lcr.table_name%TYPE. BEGIN -.global_name%TYPE. Just for your info. 'UPDATE'. v_database global_name.You can override the values in the constructor by calling these methods v_lcr. END IF.lcr$_row_record. command_type => p_command.construct( source_database_name => v_database. Those types are also documented in the reference guide I mentioned above.lcr$_row_record. the type LCR$_ROW_LIST is a collection of LCR$_ROW_UNIT.set_object_name(p_table_name). -.com 10 ODTUG Kaleidoscope 2007 .set_source_database_name(v_database).Construct the LCR v_lcr := sys. Once you have the LCR you can modify the values to suit.AnyData data types. 'DELETE') THEN RETURN v_lcr.
END LOOP.odtug.Set some values v_lcr.get_value('old'.convertVarChar2(TO_CHAR(NULL))). END CASE.com 11 ODTUG Kaleidoscope 2007 .get_object_name() || '. To call this function and manipulate it. -.PUT_LINE( 'Database: ' || v_lcr.data_type WHEN 'VARCHAR2' THEN v_lcr.column_name.AnyData. Object Owner: HR. sys. You can contact me in the future at lewisc@rocketmail. Object Name: EMPLOYEES.accessVarchar2(v_lcr.I'm going to keep this CASE c1. Object Name: ' || v_lcr. sys. sys. example fairly simple Cunningham c1.AnyData. v_lcr. 'EMPLOYEES'. 'INSERT' ).anyData. sys. sys.get_command_type() ).convertNumber(TO_NUMBER(NULL))). DBMS_OUTPUT.add_column('old'.lcr$_row_record.add_column('new'.anyData. c1.add_column('new'. sys. Thank you for taking the time to read this paper. c1.column_name. RETURN v_lcr.AnyData.Display Some Values DBMS_OUTPUT.Step-By-Step Streams -. LewisC www. Command: INSERT New First Name: Lewis. you might write something like the following: DECLARE v_lcr sys.get_object_owner() || '. 'first_name'. Old First Name: George And that's all there is to it. 'first_name')) || '.column_name. c1.convertVarchar2('Lewis')).com or come by and read my blog. c1.accessVarchar2(v_lcr. Object Owner: ' || v_lcr. sys.set_value('old'.PUT_LINE( 'New First Name: ' || sys.column_name.com/oracle/guide.convertNumber(TO_NUMBER(NULL))). Old First Name: ' || sys. WHEN 'NUMBER' THEN v_lcr.add_column('new'. -.AnyData.convertDate(TO_DATE(NULL))). END.convertDate(TO_DATE(NULL))).ittoolbox. Command: ' || v_lcr. The output from this is: Database: XE.AnyData. An Expert's Guide to Oracle Technology. 'first_name')) ).add_column('old'. http://blogs. 'first_name'.AnyData.column_name.convertVarchar2('George')).anyData. v_lcr. v_lcr.get_value('new'. c1.convertVarChar2(TO_CHAR(NULL))). END.column_name.set_value('new'. v_lcr.get_source_database_name() || '.anyData. BEGIN v_lcr := create_lcr( 'HR'. WHEN 'DATE' THEN v_lcr. sys.add_column('old'.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue listening from where you left off, or restart the preview.