4 GW BPMe SOA © WEB 2.0 eer os BPM* SOA ® WEB 2.0 Mark Simpson — Consultancy Director, Griffiths Waite www. griffiths-waite.co.ukur Speaker WEVA Consultancy Director Griffiths Waite A ACE Director impso' Lew) 19 years Oracle development and architecture experience 1 UK Oracle ACE Director for SOA Book Author — SOA and FMW 1% BPEL project 2005, 1* BAM 2006 Regular Speaker - Butler Group (Ovum) Master classes = Gartner, IDC = Oracle Open World x 3yrs = SOA Symposium Oracle SOA Community - Award Winner 2009, 2010 and 2011Dyess inpeiuneeteities |. e|_ Leones 2004 Oracle 9i Advanced Queuing for Business Integration 2005 Delivering the Adaptive Enterprise with BPEL and BAM 2006 Empowering the Business with Oracle Business Rules 2006 BAM Closing the Loop of Process Improvement 2007 SOA Customer Success Case Studies of Process Improvement 2007 Oracle BPA and SOA Best Practices 2008 Migrating Oracle Forms to SOA&ADF 2009 SOA Design Best Practices 2010 BPM and SOA Design Patterns 2010 Key Success Factors for Fusion (Middleware) Projects 2011 Enterprise Architecture and SOA Governance 2011 SOA Case Study for Business Value 2012 EDN and BAM 2012 SOA Case Study — Innovation Award Winner 2013 Understanding SOA Infra Database 2013 Whiteboard Overview of Fusion Middleware 2013 Building a SOA Reference Architecture> Why do we need to understand SOA_Infra DB | > Understanding the Service Engines mi > SOA DB Persistence — BPEL 8 Mediator Human Task — BPM — 0sB EDN > Tuning EM and SOA DB Understanding Purging Project Scenarios — Lessons LearntOracle SOA Suite dehydration store captures the Instance Data from SOA composites Why should we care about SOA_INFRA — Enterprise Manager can become slow to navigate with large volumes — scripts can be quicker — DB Management is essential for smooth running SOA engines, persistence is a large overhead Analysing data in SOA_INFRA helps identify unnoticed issues and helps look for bottlenecks, stuck messages reconcile messages Would you build a Java app without consideration for the Database Layer — Bad Design Decisions can put a large weight on the Database — Helps you understand how SOA Engines work What can the impact be — EM Console Unusable — _sTAtransaction needing to be raised Messages appearing to be stuck — High Engine Faults requiring recovery — slow start up times may be a tell tale signLewy Application composers ‘SOA Composite Editor ‘Opting T] we Oracle Service Bus oo Pj, Web-based consoleDriving Tables .. — COMPOSITE_INSTANCE CUBE_INSTANCE (BPEL) — MEDIATOR_INSTANCE WFTASK (Human Workflow) BRDECISIONINSTANCE (Rules) — BPM_CUBE_PROCESS REFERENCE_INSTANCEXML_DOCUMENT - Stores process input/output msg and large XML variables (inc dlv), linked to composites via INSTANCE_PAYLOAD [Use Java API to get payload - well documented] CUBE_SCOPE — Stores the scope variables and objects AUDIT_TRAIL — Stores information for the EM Console flows in XML AUDIT_DETAILS — Additional Audit Details as defined by the audit levels e.g. assigns) WORK_ITEM — Stores Activities created by BPEL (rollback points), onMessage branches will result in one WORK_ITEM per branch WI_FAULT - Stores recoverable and non-recoverable BPEL faults DLV_MESSAGE — Deferred Processing MEDIATOR_DEFERRED_MESSAGE - stores parallel mediator MEDIATOR_CASE_INSTANCE - one row pet routing rule— BPM_AUDIT_QUERY - Stores BPMN process input/output msg and activity inputs and outputs. Can grow fast - managed by audit levels — BPM_CUBE_AUDIT_INSTANCE - Stores the information on each instance within the BPMN process — BPM_CUBE_ACTIVITY — Static view of all activities in a deployed process AQSEDN_EVENT_QUEUE_TABLE / AQSEDN_OAOO_DELIVERY_TABLE — EDN Messages WLI_QS_REPORT_DATA — blob containing reporting information written from OSB, linked to OSB services via WLI_QS_REPORT_ATTRIBUTE> Composite Distinguished Name (Composite_Dn) - _domain/compositeNamelrevision*label E.g. "default/SOAOrderBooking!1.0*2006-07-09_12-23-10_112 FYI - 2006-07-09_12:23:10_112 is the MDS label assigned to the composite during deployment > Execution Context ID (ECID) — Common through all linked Composites, beware of inadvertently linking 100s of composites (e.g. looping through a file) Links OSB invocations into the flow, Appears in JVM thread dump, useful to pass to 3 party systems > CMPST_ID and CIKEY on Cube_instance — CMPST_ID Links BPEL Process to its containing Composite — _ CIKEY is PK for BPEL Instance and useful in EM BPEL Engine for faster access to flow or for Java Error Recovery Utilities> Sequential Routing - Mediator Mediator_Case_Instance — Mediator_Case_Detail Audit Only > Parallel Routing - Mediator, Mediator_Case_Instance, Mediator_Case_Detail — Mediator_Deferred_Message DLV_Message Multiple Treads, Mutiple Tancactions Deferred Message Processing and Audit> When a Mediator Service Engine is started, it registers itself in the database and gets a container id Used by Mediator to track which nodes are available — Each node gets a (different) container id Note : we have seen problems when nodes get started concurrently with old messages having stale container ids ~ resulting in stuck messages Cconecronoox7ise2erosen1135c0#S65 _21-UN-131153.35.40000000 AM GB-EIRE > MEDIATOR_DEFERRED_MESSAGE_PAYLOAD gets populated on initial thread and moves through the following states — -state 0 ready, 1 locked, 2 done, 3 faulted.> BPEL Process Cube_Scope is a key table Audit_Trail is written seperately They may be written in separate transactions depending on Async audit policy > Dehydration Points Common to see rollbacks to previous point in failure Wait, Receive, Reply, Dehydrate, onMessage, onAlarm, End Transaction=participate extends transaction enyartion on> A-team Blog — https://blogs.oracle.com/ateamsoab2b/entry/list_of_all_states_from List of all states from COMPOSITE_INSTANCE, CUBE_INSTANCE, DLV_MESSAGE tables ‘nimany of my epapemant gel asked eps abel ates one competes n 11gand hoa decehet ‘ten, especaly when we a Youbeshotng Ssies ound purging Nave copied soe aes Nom me ‘COMPOSITE STANCE States Addendum : Composite Instance States 34- Open and Faulted 36- Running with recovery required and unknown state> Find recoverable activities select * from work_item where state = 1 and execution_type != 1 STATE Inactive : Open Active : Open Suspended Open Pending Complete : Open Faulted : Closed Completed : Closed Finalized : Closed Pending Cancel Closed Cancelled : Closed Faulted CemyanVRYNES 10: Closed Aborted Execution Type: 11: Closed Compensated 0; Normal 12: Closed Stale 1: Scheduled 13: Open Pending Recovery> Mediator = Simple Sequential Rule Parallel Deferred Routing (5B etume enn Synchronous Faults Asynchronous Faults > BPEL BPEL Scopes and activities — Mediator wiring to BPEL — BPEL with a wait BPEL Faults> Number of BPEL Instances per day.sql > Number of Cube Scopes per day.sql > Space Monitoring of Dehydration DB.sql > EDN Monitoring.sq| > Faulted Messages of a particular Error Msg.sql > AllFaults.sql > Composite Fault Recovery > Service Engine Recovery> Instance Summary.SQL e T ‘COMPOSITE NAME TV182019 09 0 0 SOA BackOfce AgrwamertCRASUEAD AgreamerteniRcx 107 (1182013 09.09 80 SOA_BackOtce Awami CROSUBAB AgeamerProcess: 107 (5,180 09.980 SOA BackOfce ApeamerCROSuBAB Ageamertrocessc1 87 {5/1803 0 00 10 SOA BackOfce AgoarrtCRASuBAD CeateAgeemert 107 (182013 0 000 SOA BackOtee AzearnetCRUUSUBAG Cretejenmet 107 (1192013 0 00 10 SOA BeckOMce AgeamrtCusioma AgreamertCutome 100, (5182013 0.00 10 SOA BeckOfce AzeomertCusiomaAgeamertCstome 100 (182019 09 0040 SOA GackOfee AgrearrtCastomatigreemertCstome 100 (1192013 0 00 £0 SOA BackOfce Aware DWSUbABC AgeamertDW Proc 105 (182012 0.0 49 SOA BeckOfice AgrsmeEnterpree? Agreamertsterps 101 (5/1803 0 0 10 SOA BackOfce AzesrertSAPSUDAD AgeomertSAPSut 107 (1182013 09 0&0 SOA BackOMcw AzearmmtSAPSWDAD AgerertSAPStt 107 (:19019 09.0 80 SOA BackOfce AzreamrtSemuencelsAgeamertSequne 100, (5:182019 09 00 £0 SOK BackOfce Asam SequencelsApeamertSeqane 100 {5118013 0 00 10 SOA BackOfce Agrarrt SeuencelsAgearertSeqenc 100, {5/1803 0:0 10 SOA BeckOfce CrstomeCANSuBABC CutomerCRNSubr 101 (1182019 09040 SOA BackOfee CrstomerCAMSADABC CstomerCRNSAB 1 {19019 0 000 SOA BackOfce CuatrEaerpreePu Custorefenerpe 104 (5018203 0:00 10 SOA BockORce CustormmepisePy CustomerEerpis 101 {1182013 0 0010 SOA BeckOMce DeaeConactCRVEYO UpdateDealComac 02 (192019 00-0 0 SOA BackOfce DsseConactCAVLADealrContactrace 105 (1182013 09040 SOA BackOfee DsaeContactOVSut DalrConctOWE 1 OK 0 E ‘COMPONENT NAN COMPOS! STATE F (chose ao couPLETED (OPEN #00 RUNING (OPEN 200 RUNHING (CLoseD ano COMPLETED (CLOSED AND COPLETED (LoseD ano COMPLETED (OPEN 200 RUNNING (CLOSED AND COMPLETED (OPEN 280 RUNING Chosen mo coveLeTeo (CLOSED AND COMPLETED NONRECOVERABLE (CLoseD ano COMPLETED (Chosen ano COMPLETED eccccenekeccdocccccccoot ofl | _ 07 M2 fou 7 4 a8 a] ‘8 ¢ 1a 0B inst ee 2 o 1701768 2 4 a4 (MN RU_TIE. SECS one BEB cece secetuwuee> Log Messages — http://prodsoa.mfl.co.u ifra/events/edi — select * from edn_log_enabled; — select count(1) from edn_log_message; — Shows Enqueuing and Dequeing from EDN > AQS_EDN_EVENT_QUEUE_TABLE_S — LEDN_EVENT_QUEVE EDN_JAVA_SUBSCRIBER — 3EDN_EVENT_QUEUE EDN_SQL_SUBSCRIBER> EM queries are heavy on the DB and often produce timeouts/stuck threads. EM is an application to tune for specific usage 1.Educate support users on EM navigation and usage 2. Disable fetching of instance metrics by default 3. Restrict default search display to Ominutes 4. EM authenticates, loads targets and displays page. Loading targets can be cached, e.g. set oracle sysman.emes.ciscoveryvls:FMW_DISCOVERY USE_CACHED_RESULTS-true 5. Increase Perm Size of Admin Server JVM e.g MEM_PERM_SIZE_64BIT="-KX:PermSize=1024m” 6. Decrease the frequency of DMS Application 7. Disable mserver JMX notifications of state changes 8, Ensure users use personal logins> Tune and Monitor Weblogic SOA Data Sources — SOA Data Source , Used for transaction processing SOA Local Tx Data Source , Used for audit trail > SOA DB Maintenance Purging, define a realistic retention period. Do not use for audit purposes, create archive schema if required. Gather Schema Stats and Index Rebuilds > SOA DB Tuning (inc. EM Queries) - see 08 tuning whitepaper, for example. AWR reports for Indexes (e.g. Composite_Sensor_Values) = Specific Tuning advice for large tables (e.g, Hash partitions, Secure LOB etc.). Candidate include AUDIT_COUNTER, CUBE_INSTANCE, CUBE_SCOPE, MEDIATOR_CASE_INSTANCE, XML_DOCUMENT — DB Parameters, such as. > session_cached_cursors (0.¢.50->1000) > sessions and processes (tune in accordance with data sources e.g, 2000) > sga_max size & sga_target (e.g 868 1668) > trace_enabled (e.g. swviteh off if DB performance is an issue)> Loop Purge — Inefficient, but allows state to be ignored. Good for small installations (< 100GB) > Parallel Purge More Efficient but has restrictions > Table Recreation Scripts (TRS) — Backs Up and restores just retained data, introduced in 11.1.1.7 > Partitioned Data — Will greatly enhance the efficiency of purging with large environments (> 500GB) > Truncation — Not supported, but useful in local or development EnvironmentsUse a combination of parallel purge for historic & loop purge for old running instances Purge routines create temp tables first with candidates for purge (e.g. ECID_PURGE, TEMP_CUBE_INSTANCE etc.) Experiment with Batch Sizes and monitor CPU and DB Performance bateh_size => 10000, ‘min/max_ereation date (limit this to periods to better monitor progress) ‘max_runtime => 240, retention_period (how much data to keep, must be >= max_creation_date) DOP=>4, (degree of parallelism) ‘max_count => 1000000 (used to limit the records in the temp ecid_purge table) This will run for 4 hours. increase dop if CPU allows, decrease the batch size if step of creating the temp tables is slow, raise batch size if tracing shows this happens very fast. Purge Routines may have inefficient SQL in your environment - e.g. a SQL Profile needed to be added to help tune.. INSERT INTO reference_instance_purge SELECT ID, ECID FROM REFERENCE_INSTANCE WHERE created time BETWEEN 01-JAN-13 12.0000.000000 AM’ AND 3i-JAN-13 12,00,00.000000 AM" AND ROWNUM <=2233791 and composite_instance_idis nllselect to_char(truncipartition_date,’mm’), YYYY-mm') Month_Year, decode(state, 1/'1-Completed', 2,'2- Running with faults, 3,'3- Completed with faults’, 116,16 - Running with terminated’, 17,17 Completed with terminated’, 19, "19 - Completed with faults and terminated’, 64,'64-7) "state", ccount(distinct ecid) from composite_instance where ( bitand{state,127]=1 OR bitand(state,6) =2 OR bitand{state 15) ORbitand|state,64) =64 OR state between 32 and 63 OR state = 3 OR state =19) group by to_char(trunc|partition_date,'mm’),'YYY¥-mm'), state ‘order by to_char(trunc(partition_date,'mm'),,YYWY-mm), state asc; This SQL will identify Purgaeable Composites, but if the composites have any running mediator or cubes below it the ecids will be remove from purge listGw > Purgeable Instances This SQL will identify Purgaeable Composites, but if the composites have any running mediator or cubes below it the ecids will be remove from purge list RECESS OOS Pao se select to_char(trune(partition_date,'mm'), YYYY-mm') Month_Year, oat 3.eaminadwth a ie decodelstate, 1/1 -Complated’, a * = Running with aul pose att Pa 3,'3- Completed with faults feet -cayiet tne 116,"16- Running with terminated’, een ene “ ca} “a 17,"17- Completed with terminated’, Lea lw-conmervn ions a = 419,'29-- Completed with faults and terminated aes is ge = 64, "64-2, state) "State", count{dstinetecid) Boer fact fe itm from FMWC_SOAINFRA.composite_instance ae os nrnen “3 sroup by to_char(trunc(parttion datemm',Yv¥-mm), state RB s order by to_har(trune{parttion_date mm’, YYV¥-mm, state asc; AS coms harrant se > Total Rows aoc 1 -Conieed aa ewe a-nerguneus i PurgeMonitoring-key tables.sal ee —ernen amt Ll> Deleting smaller chunks or data in a single transaction more efficient than deleting huge data together > Temporary Tables are created each run and is time consuming with large batches, rollback can be resource heavy if purge is aborted. Small and Often can be run in daytime, large parallel should be out of hours ECIDs that are active over purge windows will not be removed Purge Scripts will still leave some dangling reference — see Oracle Support Note Do not run and Purging and heavy DB operations (e.g. RMAN validation ) Don’t forget to purge OSB reporting schema Reporting will leave orphaned records (e.g. XML_PAYLOAD, INSTANCE_PAYLOAD) vVvvvvvyvvvvvyv 5 SOA partitions — 200 + composites 1.1 TB of size, 800m rows 1 million Instances Per Day Limited Usage of OSB Many instances appearing in Engine Recovery (Activity & Invoke) Business Impact —lost messages, backlogs and delays High Application Support usage of Enterprise manager Frequent Stuck threads - restarts 4 month Rectification Project > > Parallel & Loop Purge. 11.1.1.6 purging scripts backported to support ignore_state. EDN purged. EM usage reduced through provision of direct SQL checks and Java Recovery Utility and EM Tuning A-Team Assistance — few hours a week through Rectification Project Mediator Engine Tuning — Parallel Worker threads increased, BPEL invoker increased DB Tuning — Hash Partition Index added, Cube Scope tuned Composites refactored (70% reduction) & redesigned with transactions & threads in mind Stuck Thread Analysis standardisedDescriation| Active Sessione|Percant of Activity {High Watermark Walts 30.78 | 51.96 4 2 "User /0" wait Class 9.79 | 16.550 23 Top SOL Statements 7.52 | 12.715 4 Global Cache Messaging 7.24 | 12.184 5 Butfor Busy -Hot Objects 6.91 | 11.68 1 6 Commits and Rollbacks 5 | 9.122 7 Top Segments by "User /O" and "Cluster" 2.8 | 4.732 8/0 Throughput 2.37 | 4.01 1 9 Unusual "Other" Wait Event 1.69 | 2.863, 10 Global Cache Busy 78 | 1.32 1 ‘ADOM Report> We applied in 11.1.1.4 Patch 12746784 (https://updates.oracle.com/ARULink/PatchDetails/process form?patch_num=12746784): Clean up dangling references from the DLV_MESSAGE table after purge. Patch 13615085 (https://updates.oracle.com/ARULink/PatchDetails/process form?patch_num=13615085): Purge script hanging; resolves a performance issue that essentially hangs the purge procedure. 11.1.1.6 Purge Scripts backported to 11.1.1.4 Check Indexes on Purge Temp Tables — create index temp_cube_instance_idx on temp_cube_instance(cikey); create index dlv_message_cikey_idx1 on dlv_message(cikey); You may need... — create index on DOCUMENT_DLV_MSG_REF.DOCUMENT_ID; create index on REFERENCE_INSTANCE.COMPOSITE_INSTANCE_ID; — create index on DLV_MESSAGE(RECEIVE_DATE,ECID );> Do not Neglect SOA DB maintenance — Use AWR and ADDM for tuning SQL and adding Sl Profiles Perform advice in DB tuning white paper ~ Hash Partition indexes, Table Partitioning etc. — Ensure Purging is considered pre Go Live (on by default in 12c) ~ much harder retrospectively > Consider DB impact in your design — Think about auditing levels (Dev, Prod, Off) — understan implications of Off. Consider the number of composites you will produce (e.g. should logging monitor service be really implemented on as Composite?) — Think about size of payloads, transaction times & thread usage (or DB bottlenecks appear) > Manage Enterprise Manager Usage Personal login credentials are essential — Apply EM Tuning and educate on how to use EM and supporting SQL queriesBee ee 8PM247.COM Pict tel ta en aes