Operations Guide

Operations Guide
October 21, 2011
Copyright 2011, Oracle and/or its affiliates All rights reserved
Oracle Health Insurance Copyright 2011, Oracle and/or its affiliates. All rights reserved. License Restrictions & Warranty Disclaimer The Programs (which include both the software and documentation) contain proprietary information; they are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright, patent, and other intellectual and industrial property laws. Reverse engineering, disassembly, or decompilation of the Programs, except to the extent required to obtain interoperability with other independently created software or as specified by law, is prohibited. The information contained in this document is subject to change without notice. If you find any problems in the documentation, please report them to us in writing. This document is not warranted to be error-free. Except as may be expressly permitted in your license agreement for these Programs, no part of these Programs may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose. Restricted Rights Notice U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the Programs, including documentation and technical data, shall be subject to the licensing restrictions set forth in the applicable Oracle license agreement, and, to the extent applicable, the additional rights set forth in FAR 52.227-19, Commercial Computer Software--Restricted Rights (June 1987). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065. The Programs are not intended for use in any nuclear, aviation, mass transit, medical, or other inherently dangerous applications. It shall be the licensee's responsibility to take all appropriate fail-safe, backup, redundancy and other measures to ensure the safe use of such applications if the Programs are used for such purposes, and we disclaim liability for any damages caused by such use of the Programs. Third Party Web Sites, Content, Products, and Services Disclaimer The Programs may provide links to Web sites and access to content, products, and services from third parties. Oracle is not responsible for the availability of, or any content provided on, third-party Web sites. You bear all risks associated with the use of such content. If you choose to purchase any products or services from a third party, the relationship is directly between you and the third party. Oracle is not responsible for: (a) the quality of third-party products or services; or (b) fulfilling any of the terms of the agreement with the third party, including delivery of products or services and warranty obligations related to purchased products or services. Oracle is not responsible for any loss or damage of any sort that you may incur from dealing with any third party. Trademark Notice Oracle, JD Edwards, and PeopleSoft are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Draft Documentation Notice This documentation is in prerelease status and is intended for demonstration and preliminary use only. It may not be specific to the hardware on which you are using the software. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to this documentation and will not be responsible for any loss, costs, or damages incurred due to the use of this documentation. The information contained in this document is for informational sharing purposes only and should be considered in your capacity as a customer advisory board member or pursuant to your beta trial agreement only. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described in this document remains at the sole discretion of Oracle. This document in any form, software or printed matter, contains proprietary information that is the exclusive property of Oracle. Your access to and use of this confidential material is subject to the terms and conditions of your Oracle Software License and Service Agreement, which has been executed and with which you agree to comply. This document and information contained herein may not be disclosed, copied, reproduced, or distributed to anyone outside Oracle without prior written consent of Oracle. This document is not part of your license agreement nor can it be incorporated into any contractual agreement with Oracle or its subsidiaries or affiliates.
Operations Guide
Table of Contents
1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Target audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Guiding principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 References to artifacts in the OHI Claims distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 6 6 6
2 Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1.1 Backup/restore tips. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1.2 Oracle Maximum Availability Architecture . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2.1 Cloning databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2.2 Active Data Guard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2.3 RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1.1 Database tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1.2 Application tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1.3 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.2 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Schema verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 Setting sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.5 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1.1 How to address security issues? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1.2 Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1.3 Additional topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.5.1.4 Auditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.5.2 OHI Specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.1 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.1.1 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.2 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.2.1 Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.2.2 Statistics types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.2.3 Database parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.2.4 Statistics collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.2.5 Miscellaneous. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.3 I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.3.1 OHI specific guidlines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.3.2 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6.4 Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6.4.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Operations Guide 2
2.6.4.2 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.7 Oracle Text index maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.7.1 Index Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.7.1.1 Dealing with Synchronization errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.7.2 Index Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.8 Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.8.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.9 Fault management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.9.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.9.1.1 Reduce time to resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.9.1.2 Oracle 11g Database Features on Fault Management. . . . . . . . . . . . . . 18 2.10 Do's and dont's . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.10.1 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.10.1.1 Do . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.10.1.2 Do not . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.11 Generate Reporting Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3 Fusion Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.1 Starting the WebLogic Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.1.1 Start the Administration Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.1.1.1 Use the Node Manager to start the Administration Server . . . . . . . . . 21 3.1.1.2 Use the start script to start the Administration Server . . . . . . . . . . . . . 21 3.1.2 Start the Managed Server(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.1.2.1 Use the Administration Console to start the Managed Server(s) . . . . . . 22 3.1.2.2 Use the Node Manager to start the Managed Server(s) . . . . . . . . . . . . 22 3.2 The System's start up routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2.1 Actions performed when the system starts up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2.2 Verify the results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3 Stopping the WebLogic Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3.1 Stop the Administration Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3.1.1 Use the Node Manager to stop the Administration Server . . . . . . . . . 23 3.3.1.2 Use the Administration Console to stop the Administration Server . . . 24 3.3.1.3 Use the stop script to stop the Administration Server. . . . . . . . . . . . . . 24 3.3.2 Stop the Managed Server(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3.2.1 Use the Administration Console to stop the Managed Server(s) . . . . . . 25 3.3.2.2 Use the Node Manager to stop the Managed Server(s) . . . . . . . . . . . . 25
4 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.1 Monitoring Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2 Monitor the Work backlog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.3 Task Processing Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.4 Technical Monitoring and Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.4.1 Accessing Mbeans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.4.2 OHI Claims Event Log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.4.3 Usage scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.4.3.1 Flood control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Operations Guide 3
4.4.3.2 Duplicate Notifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.4.4 OHI Claims Management plugin for EMGC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.4.4.1 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.4.4.2 Installation of the OHI Claims Management Plug-in . . . . . . . . . . . . . . 31 4.4.4.3 Configure metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.4.4.4 Viewing metrics and alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.4.4.5 Installation in a clustered environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.4.5 Monitoring of OHI Claims using profiling tools . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.1 Logging and Auditing Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.1 Types of Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.1.1 Application Log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.1.2 Security Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.2 Logging Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.2.1 Configuration of an appender. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.2.2 Security audit log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.1.2.3 Predefined logging configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.2.4 Performance logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.2.5 Using a custom log4j configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.2.6 Log4j Log Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.1.2.7 Associating a log level with an appender. . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.1.2.8 Making changes to logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
6 Error Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.1 Recover from Application Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.1.1 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.1.2 Recover errored tasks in the Claims Processing Flow . . . . . . . . . . . . . . . . . . . . . . 43 6.1.2.1 Errored Task Listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6.1.3 Recover from external service failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.3.1 Failing to deliver a message for which processing of a claim should not be interrupted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.3.2 Failing to deliver a message for which the Claims Flow should be interrupted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.4 Recover from failures in batch processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.5 Recover messages from the Task Error Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.2 Recover from system failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
7 Copying an Environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
7.1 Clone the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 7.2 Empty the Task Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 7.3 Enable Replication of Setup Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 7.4 Deploy the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 7.5 Validate Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
8 OBIEE Reporting Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

8.1 Setup the OBIEE Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 8.2 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 8.3 Customize the OBIEE Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Operations Guide 4
9 Deeplinking to screens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Operations Guide
1 Introduction
1.1 Purpose
This document is the Operations Guide for Oracle Insurance Claims for Health (OHI Claims). OHI Claims runs on an Oracle platform that consists of an Oracle Database and Oracle Fusion Middleware Application Server. Administration of the environment is the responsibility of the customer. The Operations Guide does not provide instructions for Oracle Database and Application Server administration, it only lists the most significant areas of interest for managing these in order to support OHI Claims. The Operations Guide is not a comprehensive Oracle system administration manual nor does it replace existing documentation or available courses in management of Oracle environments.
1.2 Target audience

The Operations Guide is intended for system and database managers and others responsible for looking after the daily operation of OHI Claims.
1.3 Guiding principle

Guiding principle for the Operations Guide is to use default features of the technology stack as much as possible or unless alternative methods improve OHI Claims operations. Examples of such default features are Automated Storage Management (ASM) and Automatic Optimizer Statistics Collection of the Oracle Database.
1.4 References to artifacts in the OHI Claims distribution

The distribution contains a number of directories that contain all the necessary information and sources to perform installation and upgrade actions. The root directory is the directory where the released files are placed. It can be any location or name of your choosing and will be referenced throughout this document as <OHI_ROOT>.
Operations Guide
2 Database
2.1 Availability
2.1.1 Generic guidelines 2.1.1.1 Backup/restore tips Run the production database in archive log mode. Frequently back up your data The Oracle database supports the creation of on-line backups Refer to the manuals for setting up backup procedures Use RMAN (Recovery Manager) FRA (Flash Recovery Area)
2.1.1.2 Oracle Maximum Availability Architecture Oracle HA technology is designed so that even redundant components can be actively used (e.g.; have you ever checked the air pressure of your spare tire?). To increase availability of the Oracle system, consider using components of Oracle MAA, e.g Real Application Clusters Active Data Guard (read-only access to physical standby database) Rolling upgrades (to new Oracle versions) Hot patching (using opatch, also works for RAC) Scale for growth
For more details see http://www.oracle.com/technetwork/database/features/availability/maa-090890.html 2.1.2 OHI specific guidelines 2.1.2.1 Cloning databases Use RMAN 11g: RMAN duplicate from active database live backup/restore over network in new instance 11gR2: Targetless duplicate possible from recovery catalog Other options (like using backup controlfile) may also be considered
2.1.2.2 Active Data Guard Read-only access to physical standby database Operations Guide Scalable Reader Farms (up to 30 active standby, scale using RAC) Automatic Block Repair (2-way) Query SLA via STANDBY_MAX_DATA_DELAY 7
Ensure that force logging is activated to prevent an unintended NOLOGGING statement from not logging changes OHI Claims is certified for ADG.
2.1.2.3 RAC Use of RAC for OHI applications is certified Oracle ASM (Automatic Storage Management) is required
2.2 Scalability
2.2.1 Generic guidelines 2.2.1.1 Database tier Optional multiple database instances using RAC Optional partitioning Installing large objects ILM Information Lifecycle Management Optional save substantial disk space using compression Oracle 11g Advanced Compression for tables Index compression (no license required, not yet certified for OHI Claims) Exadata boosts throughput Sun Oracle Database Machine and Exadata Storage Server grid-in-a-box; software, servers, storage, networking Smart Flash Cache for hot data
2.2.1.2 Application tier Multiple application servers Exalogic Elastic Cloud
2.2.1.3 Hardware Clustering Virtualization
2.2.2 OHI specific guidelines Use RAC Use Oracle VM to virtualize components of OHI Claims' technology stack Oracle has not certified any of its products on VMware virtualized environments For more information see My Oracle Support Note 249212.1 OHI applications are not yet certified to run on Oracle VM for production deployments
Operations Guide
2.3 Schema verification

Introduction Several components of the application connect to a database, such as the configuration migration utility and the OHI Claims application itself. Each component assumes that is it connected to a database with the correct schema version. Working with a different schema can yield unpredictable behavior. For that reason, at the start of these components, the database schema version is verified. If the schema differs, the start-up procedure will abort. This chapter describes the mechanism that is used to verify that a component is connected to the correct database schema and what to do if this is not the case. Mechanism The verification mechanism is based on two descriptions of the database schema: a digest and a checksum. Digest The digest is a file containing a full textual description of the schemas. The digest contains all the tables, columns, constraints, privileges, indexes, etc. The digest of the expected schema version is packaged with the application and can be found at "util/migration/ohi_digest.txt". Checksum The checksum is a calculated MD5 hash code value of the aforementioned digest. This value is embedded into both the application and the database. These two checksums are compared during application start-up. If they do not match, the start-up is aborted. Processes The schema verification is used in the following processes: Install/upgrade The install utility will generate a digest of the database after it has finished installing/upgrading it. The checksum of this generated digest is compared with the packaged checksum. If these are the same, the installation/upgrade completed successfully and the new checksum value is stored in the database. Startup When the application or the configuration import utility is started, the packaged checksum is compared with the checksum of the connected database. If the checksums do not match, it means the application is connected to a different database and the start-up process is aborted. What to do if verification fails If at some point the schema verification fails and the application will not start, it means the application is connected to a database with an unexpected schema. Oracle can help to diagnose the problem by comparing the digest files. Generate a new digest of the target database and send it to Oracle support. Generating a digest file can be done by opening a command line window at "util/migration" and running the following command: verify_schema. Additionally, the command line options described in the table below can be added. All those arguments except -f are the same as the migration utility, described in "Migration Utility Setup". Furthermore, the same configuration file as migration utility can be used (although only the logging and database settings will be used). Alternatively, to bypass the checksum verficiation (not recommended!) and launch the application regardless of a mismatch, open the file "properties/ohi-claims.properties" and set the property "ohi.startup.verifyschema" to false. Operations Guide 9
Option Short -f Long --file
Argument
Required
Description
digest file path
No
The file path and name of the output digest file that is generated. Uses "digest.txt" if omitted The location of the configuration file. Uses "ohi_digest.cfg" if omitted The name of the target environment, which can be used to specify a section in the environments{ } element of the config file. If omitted, the settings outside the environments{ } element are used. Shows help information Shows the current version of the utility
-c
--cfg
config file path
No
-e
--env
environment name
No
-h -v
--help --version
No No
2.4 Setting sequences

Introduction Database sequences are used to generate a unique technical identifier for each data row. After cloning a database and changing its discriminator value, it is necessary to reset the sequences to their correct configuration. For this purpose the setSequences utility was created. Utility Configuration Navigate to <OHI_ROOT>/util/install, make a copy of the config file "setSequences.cfg.template" and rename it to e.g. "setSequences.cfg". Open this file and modify the settings for your environment(s). The settings are explained in the table below. Settings
Setting log.level Description The level of detail at which log entries are written. Possible values, in order of ascending detail: OFF, SEVERE, WARNING, INFO, CONFIG, FINE, FINER, FINEST, ALL. The connection string of the target database. If you altered the password for the ohi_claims_owner account, enter the new password here. If none is provided, the default password for ohi_claims_owner is used
db_url ohiPasswords
Operations Guide
10
Environments The environments element can be used to define environment specific settings, hence it is not needed to create a separate config file for each environment. The -e or --env command line argument (described below) can be used to switch between groups of settings. Template The configuration file template shown below is located at <OHI_ROOT> /util/install/setSequences.cfg.template:
import java.util.logging.Level // Allowable values: SEVERE | WARNING | INFO | CONFIG | FINE | FINER | FINEST | ALL | OFF log.level = Level.CONFIG // The url of the database connection. // Example 1: db_url = "jdbc:oracle:thin:@//localhost:1521/orcl" // Example 2 (To Connect to RAC Database): db_url = """jdbc:oracle:thin:@(DESCRIPTION= // (LOAD_BALANCE=on) // (ADDRESS=(PROTOCOL=TCP)(HOST=clusnode-1vip)(PORT=1521)) // (ADDRESS=(PROTOCOL=TCP)(HOST=clusnode-2vip)(PORT=1521)) // (ADDRESS=(PROTOCOL=TCP)(HOST=clusnode-3vip)(PORT=1521)) // (ADDRESS=(PROTOCOL=TCP)(HOST=clusnode-4vip)(PORT=1521)) // (CONNECT_DATA=(SERVICE_NAME=ERP)))""" db_url = 'jdbc:oracle:thin:@//<HOST>:<PORT>/<SID or SERVICE>' // For each application a schema owner (ohi_<application>_owner), an application user (ohi_<application>_user) and // an application views owner (ohi_<application>_views_owner) are created. // beside these users, the following users are created: "ohi_nonconfid_views_owner", "ohi_confid_views_owner" // The passwords for these users are defaulted to their username, however, if you want to use different passwords you can enter them here. // For every user that is created that is not listed here, the password will be equal to its username // If the password is set to an empty string, the script will prompt for the password ohiPasswords { // <database_user> = '<user_password>' // <database_user2> = '<user_password2>' } environments { your_environment{ log.level = Level.INFO db_url = 'jdbc:oracle:thin:@//<HOST>:<PORT>/<SID or SERVICE>' ohi_config.passwd = 'ohi_config' // If empty, the script will prompt for the password } another_environment{ log.level = Level.INFO db_url = 'jdbc:oracle:thin:@//<HOST>:<PORT>/<SID or SERVICE>' ohi_config.passwd = 'ohi_config' } }
Operations Guide
11
Running the utility The setSequences utility is available in the <OHI_ROOT>util/install directory. To run the utility, open a command line at this location and execute setSequences.bat (on Windows machines) or setSequences.sh (on Unix machines), followed by the command line arguments described below. The application should be shut down before running the utility. If the application is still up, the utility will not continue.
Command line arguments

Option Short -c Long --cfg config file path No The location of the configuration file. Uses "setSequences.cfg" if omitted The name of the target environment, which can be used to specify a section in the environments{ } element of the config file. If omitted, the settings outside the environments{ } element are used. Shows help information Argument Required Description
-e
--env
environment name
No
-h
--help
No
2.5 Security
2.5.1 Generic guidelines 2.5.1.1 How to address security issues? Hardening (making the database more solid) Assessments (regular checks) Monitoring (without visibility one cannot secure anything) Auditing (examination of the controls) Access control (authentication, authorization) Encryption (mask data)
Note to secure the whole technology stack (also listener, network, etc.) as "a chain is only as strong as its weakest link". 2.5.1.2 Solutions Regularly Check Oracle Security Center1 for Critical Patch Updates2 (security vulnerabilities). Check Oracle Global Product Security Blog3
Operations Guide
12
Use STIG (Database Security Technical Implementation Guide) guidlines & CIS checklists (Center for Information Security) Oracle Data Vault for separation of duties Application using Oracle Label Security (= application using Virtual Private Database) E.g. DA cannot access data Not yet certified for OHI Claims Oracle Audit Vault for reporting, monitoring, detection Application using Data Vault (and other options/features) Secured audit data Not yet certified for OHI Claims Oracle Configuration Management Pack for OEM E.g. for Critical Patch Update Advisory feature, Secure Configuration Scanning Oracle Provisioning and Patch Automation Pack For guided patching Oracle Total Recall (Flashback Data Archive) Enables AS OF queries Long-term storage and auditing of data Certified for OHI Claims Oracle Secure Backup Oracle Advanced Security Option Oracle*Next encryption RMAN encrypted backups
2.5.1.3 Additional topics Use password policy 114g5 defaults6 $ORACLE_HOME/rdbms/admin/utlpwdmg.sql
Uses verify_function_11g verification function as default Consider Security parameters (11g) SEC_CASE_SENSITIVE_LOGON (default TRUE ) SEC_PROTOCOL_ERROR_FURTHER_ACTION (default TRACE) SEC_PROTOCOL_ERROR_TRACE_ACTION (default CONTINUE) SEC_MAX_FAILED_LOGIN_ATTEMPTS (default 10) SEC_RETURN_SERVER_RELEASE_BANNER (default FALSE) Secure default accounts Restrict privileges & roles Encryption E.g. network, tablespace, etc.
2.5.1.4 Auditing 2.5.1.4.1 What to audit? Operations Guide Logon/logoff Object changes Statements (DDL/DML/selects) 13
Security changes Database links Replication Errors
2.5.1.4.2 Auditing requirements Separation of duties Reporting Notification Integrity of audit data Map accountability & credentials Process Automation
2.5.2 OHI Specific guidelines Note password aging for generic OHI accounts Set up maintenance procedure No need to have non-OHI CPUs and PSUs certified by OHI
2.6 Performance
OHI Claims Development is leading in prescribing activities for performance; customer has certain obligations in this area
2.6.1 Parameters 2.6.1.1 OHI specific guidelines Use Automatic Memory Management MEMORY_TARGET To prevent an imbalance in ratios also set following parameters to set minimum sizes: SGA_TARGET DB_CACHE_SIZE SHARED_POOL_SIZE PGA_AGGREGATE_TARGET Note theres sufficient autotuning space Do not use AMM with Linux HugePages; see MOS note 749851.17 GV$SYS_OPTIMIZER_ENV
Keep optimizer parameters8 on default values
2.6.2 Statistics 2.6.2.1 Optimizer The optimizer will use following components to determine the SQL execution plans: Operations Guide 14
1. 2. 3. 4. 5.
Statistics Parameters Space & storage management CPU & I/O characteristics Schema definitions
2.6.2.2 Statistics types 1. 2. 3. 4. System (gathered automatically at instance startup; needs to be gathered during typical workload) Dictionary (gathered automatically during Database Maintenance Window) Fixed objects (should be gathered manually during a typical workload) Objects (table/index; gathered automatically during Database Maintenance Window; also need to be gathered manually for specific events)
2.6.2.3 Database parameters Use the Automatic tuning capabilities of the Oracle Database and the Advisor Framework. 2.6.2.4 Statistics collection Oracle recommends automated statistics collection Default job dbms_stats.gather_database_stats_job_proc in Maintenance Window It is the responsibility of the database administrator to define the maintenance window Use dbms_stats ESTIMATE_PERCENT=AUTO_SAMPLE_SIZE (default) Column statistics and histograms dbms_stats METHOD_OPT = FOR ALL COLUMNS SIZE AUTO Stale statistics; table monitoring facility Tracks INSERT/UPDATE/DELETE; see USER_TAB_MODIFICATIONS GATHER_DATABASE_STATS/GATHER_SCHEMA_STATS will pick up tables modified >10% Use Statistics Preferences (11g) after consulting OHI Note: when schema/database preference is set, it only applies for the current objects (not for new objects); if unwanted, set global preference
2.6.2.5 Miscellaneous Statistics have to be gathered manually after bulk loads of 1. 2. Products, Procedures and Diagnoses Relation or Provider data
Use production statistics on DTA environments (see Oracle Database Performance Tuning Guide).9 2.6.3 I/O 2.6.3.1 OHI specific guidlines Operations Guide Advice: Use Segment Shrink Requires ROW MOVEMENT attribute on tables and indexes 15
Requires Automatic Segment Space Management (= standard for OHI Claims) Use OEM Segment Shrink Advisor10
2.6.3.2 Generic guidelines Spread I/O's over multiple controllers I/O time should not exceed 5-10ms (for a single block read) Calibrate I/O resources (when using Automatic Parallel Execution or Resource Manager Via OEM or dbms_resource_manager.calibrate_io Oracle Database Smart Flash Cache Available on Solaris and Oracle Enterprise Linux Expand database buffer cache E.g. using a solid state disk or I/O card with flash memory Stores unmodified database blocks evicted from SGA buffer cache
If it's unmodified it is written to disk first, then to DSFC Oracle Unbreakable Linux Kernel Linux kernel tuned for Oracle OEL5 update 6 = standard UEK Standard OEL5 = 2.6.18, UEK = 2.6.32
2.6.4 Monitoring 2.6.4.1 Generic guidelines Use Oracle Enterprise Manager Grid Control Prepare: collect performance data AWR (Automatic Workload Repository) Baselines OS utilities (vmstat, iostat, sar, top, topas, prstat, swap) Oracle OS Watcher
Oracle Metalink Note 301137.1, OS Watcher users guide Use autotuning capabilities: ADDM/AWR/AWR SQL/ASH Will be used by OHI Development in case of performance issues Requires STATISTICS_LEVEL=TYPICAL Advice: Use OEM Tuning & Diagnostics Packs Require additional licenses Oracle Diagnostics Pack for Database E.g. Performance Monitoring, events, AWR, ASH & ADDM Oracle Tuning Pack for Database E.g. SQL Tuning Advisor, SQL Access Advisor, SQL Performance Analyzer (11g), SQL Plan Management Auto Evolve (11g), Automatic SQL Tuning (11g), Reorganize objects
2.6.4.2 OHI specific guidelines Note non-OHI software Check alertlog for ORA-006XX/07445 Operations Guide Bugs in optimizing SQL often lead to these errors 16
Always report to Oracle Support Services Report back to OHI Development
2.7 Oracle Text index maintenance

Oracle Text indexes are set up for OHI Relations and Addresses. This chapter specifies operations for maintenance of the Oracle Text indexes. 2.7.1 Index Synchronization The Text indexes are set up to synchronize the index immediately after a commit. The commit does not return until the sync is complete. Note that synchronization is performed as a separate transaction. As a result there may be a period, usually small, when the data is committed but index changes are not. The sync operation has its own transaction context. If this operation fails, the data transaction still commits. Index synchronization errors can be detected using the CTX_USER_INDEX_ERRORS view. 2.7.1.1 Dealing with Synchronization errors If synchronization errors are detected, the index in which the error occurred (column CTX_USER_INDEX_ERRORS.ERR_INDEX_NAME) needs to be rebuild. Rebuilding an index can be done on-line by executing the following command as user OHI_CLAIMS_OWNER:
ALTER INDEX <errored_index_name> REBUILD ONLINE;
For rebuilding all the Text indexes execute the following:

ALTER ALTER ALTER ALTER ALTER INDEX INDEX INDEX INDEX INDEX REL_RELATION_IDX1 REL_RELATION_IDX2 REL_RELATION_IDX3 REL_ADDRESS_IDX1 REL_ADDRESS_IDX2 REBUILD REBUILD REBUILD REBUILD REBUILD ONLINE; ONLINE; ONLINE; ONLINE; ONLINE;
2.7.2 Index Optimization Frequent index synchronization (for each commit) leads to fragmented text indexes. Index fragmentation adversely affects the response times of queries in which text indexes are used. Therefore, Oracle Text indexes must be optimized frequently in order to reduce fragmentation and index size and to ensure optimal query performance. Oracle recommends that the Text indexes are optimized on a weekly basis and that rebuild mode is used to do that. Rebuilding an index is often significantly faster than performing a full optimization, and is more likely to result in smaller indexes, especially if the index is heavily fragmented. Rebuild optimization creates a more compact copy of the $I table, and then switches the original $I table and the copy. Note that the rebuild operation requires enough space to store the copy as well as the original. (If redo logging is enabled, then additional space is required in the redo log as well.) At the end of the rebuild operation, the original $I table is dropped, and the space can be reused. Use the following command to optimize the Text indexes:
begin ctx_ddl.optimize_index('REL_RELATION_IDX1',CTX_DDL.OPTLEVEL_REBUILD); ctx_ddl.optimize_index('REL_RELATION_IDX2',CTX_DDL.OPTLEVEL_REBUILD); ctx_ddl.optimize_index('REL_RELATION_IDX3',CTX_DDL.OPTLEVEL_REBUILD); ctx_ddl.optimize_index('REL_ADDRESS_IDX1',CTX_DDL.OPTLEVEL_REBUILD); ctx_ddl.optimize_index('REL_ADDRESS_IDX2',CTX_DDL.OPTLEVEL_REBUILD); end;
Operations Guide
17
Oracle Text comes with a CTX_REPORT package that can be used to gather Text index data. See the documentation of the Oracle Database for details.
2.8 Storage
2.8.1 Generic guidelines Use ASM Consider Information Lifecycle Management Classify your data and move to appropriate storage tiers Or make read-only or offline Choose a strategy for reorganizing your database Use low cost hardware Consider compressing tables and indexes When copying a database from production E.g. to test and development environments Consider Removing excess data Which may not be removed in production Then compress the rest Data Masking/scrambling Placing database files for non-production environments on cheaper disks
2.9 Fault management

2.9.1 Generic guidelines 2.9.1.1 Reduce time to resolution Prevention by Automatic health checks Diagnostic by Automatic Diagnostic Workflow Solution by Intelligent resolution Delivery by Proactive patching
2.9.1.2 Oracle 11g Database Features on Fault Management ADR (Automatic Diagnostic Repository) Incident Packaging Service SQL Test Case Builder SQL Repair Advisor Data Recovery Advisor (via OEM and RMAN, uses Health Monitor) List, Advice, Repair Proactive checks Operations Guide Health Monitor or RMAN validate database HM: Framework for running diagnostic checks for several database components Parameters to detect corruption 18
DB_ULTRA_SAFE => changes defaults for
DB_BLOCK_CHECKING, DB_BLOCK_CHECKSUM, DB_LOST_WRITE_PROTECT Support Workbench (OEM; database & ASM)
2.10 Do's and dont's

2.10.1 OHI specific guidelines 2.10.1.1 Do Oracle recommends that the Oracle database instance that is used by OHI Claims is used sole ly for the purpose of running the OHI Claims system However: the license for the Oracle database does not restrict it to be used for the purpose of running OHI Claims only In the case that additional database schemas are created in the Oracle database instance, make sure that these are not prefixed with OHI Consider using Database Replay to re-create production database workload in a test environment Use Oracle Resource Manager (when 100% CPU usage) Prioritize sessions (e.g. DBAs/OHI Claims OLTP users/OHI Claims Reportings users) Automated Maintenance Tasks feature relies on the Resource Manager being enabled during the maintenance windows
Resource Manager Plan ensures that Automated Maintenance Tasks consume no more that 25% of CPU capacity Consider using Instance Caging (11g)
2.10.1.2 Do not Make changes to the OHI Claims schemas and objects, e.g. Add tables or views or modify existing tables Add or change indexes (also invisible indexes ) DBA activities without changing the definition (e.g. follow up on advice from Segment Shrink Advisor) are allowed
2.11 Generate Reporting Views

The OHI application has a dedicated database schema for reporting views. Reporting views are a logical presentation on the tables, resolving details such as dynamic field definitions, languages etc. The views should be regenerated after: an upgrade of the application; changing dynamic field or dynamic record definitions; using the Configuration Migration utility.
When running the view generator you might get this message:
Operations Guide
19
ORA-00923: FROM keyword not found where expected
This usually means that you have defined a dynamic field with a reserved Oracle keyword as the name of the field. To resolve this, you must change the name of this field. Queries to detect problematic names are:
select distinct code from ohi_claims_owner.cod_flex_code_systems_tl where upper(code) in (select keyword from v$reserved_words where reserved='Y'); select code from ohi_claims_owner.cod_flex_code_field_usages_b where upper(code) in (select keyword from v$reserved_words where reserved='Y');
v$reserved_words is a public synonym, but still requires a grant to be available. Running the generator The generator is delivered as a so-called runnable jar-file. The way to run it is:
$ java -jar ohiViewsGenerator.jar -h usage: java -jar ohiViewsGenerator.jar <options> -c <configFile> Config file -d <definitionsDir> Directory containing the XML definition file(s) -h,--help Show usage information and quit -n Do not execute the SQL on the database -o <outputFile> The file the generator writes SQL statements to -s Do not regenerate base views
The config file is mandatory. This file can be created from the supplied ohi_create_views.cfg.template. If the -d argument is omitted, no functional views will be generated; if the argument is used, you can provide the directory containing the functional view XML definition files. If you want to run the generator once and apply the result to multiple databases, you have the option to write the SQL-statements to a file by using the -o argument. Note that this will still execute the SQL, unless you use the -n flag.
1. 2. 3. 4. 5. 6. 7. 8. 9. http://www.oracle.com/technetwork/topics/security/index.html http://www.oracle.com/technetwork/topics/security/alerts-086861.html http://blogs.oracle.com/security/ http://download.oracle.com/docs/cd/E11882_01/server.112/e10897/users_secure.htm http://download.oracle.com/docs/cd/E11882_01/server.112/e10897/users_secure.htm http://download.oracle.com/docs/cd/E11882_01/server.112/e10897/users_secure.htm https://support.oracle.com/CSP/main/article?cmd=show&id=749851.1&type=NOT optimizerenv.lst http://download.oracle.com/docs/cd/E11882_01/server.112/e16638/stats.htm
10. http://download.oracle.com/docs/cd/E11882_01/server.112/e10897/storage.htm
Operations Guide
20
3 Fusion Middleware
3.1 Starting the WebLogic Cluster
This chapter describes starting and stopping of the WebLogic Server. The use of a clustered environment is assumed. 3.1.1 Start the Administration Server 3.1.1.1 Use the Node Manager to start the Administration Server This is the preferred approach. WLST scripts that handle starting servers are available in the OHI Claims release bundle <OHI_ROOT>\util\wlst. Perform the following steps: 1. Start the WLST environment:
~/wlserver_10.3/common/bin/wlst.sh
2. Connect to the Node Manager:

nmConnect('<nm-uid>', '<nm-pwd>', '<nm-host>','<nm-port>', '<domain-name> ')
Replace <nm-uid>, <nm-pwd>, <nm-host>, <nm-port> and <domain-name> with correct values. 3. Start the Admininistration Server:
nmStart('<admin-server-name>')
4. Optionally tail the server out and/or log file:

$ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>.out $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>.log
Replace <domain-name> and <admin-server-name> with correct values. 4. Disconnect from the Node Manager:
nmDisconnect()
5. Exit WLST:
exit()
3.1.1.2 Use the start script to start the Administration Server 1. Start the Administration Server
$ nohup ~/user_projects/domains/<domain-name>_domain/startWebLogic.sh > /dev/null 2>&1 &
Replace <domain-name> with a correct value. 2. Optionally tail the server log file: Operations Guide 21
$ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>.log
Replace <domain-name> and <admin-server-name> with correct values. 3.1.2 Start the Managed Server(s) 3.1.2.1 Use the Administration Console to start the Managed Server(s) 1. Access the web-based Administration Console and login using the Administrator credentials. 2. Navigate to the Domain / Control page, select the server instance and select the Start option. 3. Optionally tail the server out and/or log file:
$ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>.out $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>.log
Replace <domain-name> and <managed-server-name> with a correct value. 3.1.2.2 Use the Node Manager to start the Managed Server(s) 1. Start the WLST environment:

Replace <nm-uid>, <nm-pwd>, <nm-host>, <nm-port> and <domain-name> with correct values. 3. Start the Managed Server:
nmStart('<managed-server-name>')
Replace <managed-server-name> with a correct value. 4. Optionally tail the server out and/or log file:
Replace <domain-name> and <managed-server-name> with a correct value. 5. Disconnect from the Node Manager
nmDisconnect()
6. Exit WLST
exit()
Operations Guide
22
3.2 The System's start up routine

3.2.1 Actions performed when the system starts up As part of the start up routine, OHI Claims performs the following actions: Recover in-flight tasks that were not finished when the system was stopped. Note that the system will not restart failed tasks. Recover messages from the error queue in the database. Populate the 'Procedures / Dynamic Logic' cache (page 0): pre-calculate the evaluation of the Dynamic Logic conditions that link Procedures to Benefit Specifications and populate table CLA$PROC_DYN_LOGIC with the results.
3.2.2 Verify the results This paragraph describes how the results of the startup actions can be verified. Verify that tasks that were in a processing state when the system was shut down are restarted; the following query should not return rows:
select , , from , where and and and
task.id taty.name task.process_start_datetime ohi_tasks task ohi_task_types_tl taty task.taty_id taty.language task.status task.process_start_datetime
= = = <
taty.id 'en' 'B' -- Processing to_date(<time_system_stopped>);
Verify that messages that ended up in the error queue in the database are resubmitted for processing; the following query should not return rows:
select * from aq$ohi_task_queue_table where expiration_reason != null;
Verify that dynamic logic evaluation for the 'Procedures / Dynamic Logic' cache executed correctly: were there notifications? Check for errors in the log files.
3.3 Stopping the WebLogic Cluster

3.3.1 Stop the Administration Server 3.3.1.1 Use the Node Manager to stop the Administration Server This is the preferred approach. WLST scripts that handle starting servers are available in the OHI Claims release bundle <OHI_ROOT>\util\wlst. Perform the following steps: 1. Start the WLST environment:
Operations Guide
23

Replace <nm-uid>, <nm-pwd>, <nm-host>, <nm-port> and <domain-name> with correct values. 3. Stop the Administration Server:
nmKill('<admin-server-name>')
Replace <admin-server-name> with a correct value. 4. Optionally tail the server out and/or log file:
Replace <domain-name> and <admin-server-name> with correct values. 5. Disconnect from the Node Manager
nmDisconnect()
6. Exit WLST
exit()
3.3.1.2 Use the Administration Console to stop the Administration Server 1. Access the web-based Administration Console and login using the Administrator credentials. 2. Navigate to the Domain / Control page, select the server instance and select the Force Shutdown Now option. 3. Optionally tail the server out and/or log file:
Replace <domain-name> and <admin-server-name> with correct values. 3.3.1.3 Use the stop script to stop the Administration Server 1. Stop the Administration Server
$ ~/user_projects/domains/<domain-name>_domain/bin/stopWebLogic.sh
Replace <domain-name> with a correct value. 2. Optionally tail the server log file
$ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>.log
Replace <domain-name> and <admin-server-name> with correct values.
Operations Guide
24
3.3.2 Stop the Managed Server(s) 3.3.2.1 Use the Administration Console to stop the Managed Server(s) This is the preferred approach. 1. Access the web-based Administration Console and login using the Administrator credentials. 2. Navigate to the Domain / Control page, select the server instance and select the Force Shutdown Now option. 3. Optionally tail the server out and/or log file:
Replace <domain-name> and <managed-server-name> with a correct value. 3.3.2.2 Use the Node Manager to stop the Managed Server(s) 1. Start the WLST environment:

Replace <nm-uid>, <nm-pwd>, <nm-host>, <nm-port> and <domain-name> with correct values. 3. Stop the Managed Server:
nmKill('<managed-server-name>')
Replace <managed-server-name> with a correct value. 4. Optionally tail the server out and/or log file:
Replace <domain-name> and <managed-server-name> with a correct value. 5. Disconnect from the Node Manager
nmDisconnect()
6. Exit WLST
exit()
Operations Guide
25
4 Monitoring
This chapter covers monitoring of the OHI Claims application. Monitoring should cover the entire technology stack that is used to execute the OHI Claims application: Hardware Operating system Oracle Database Oracle Fusion Middleware and OHI Claims
Oracle recommends to monitor the infrastructure using Oracle Enterprise Manager Grid Control (EMGC). The monitoring approach outlined here focuses on the OHI Claims application. Operational monitoring of the OHI Claims application depends on identified key performance indicators. Examples of these are: Average duration for processing claims. Average time for resolving pend states of a claim (like manual pricing or manual adjudication).
For measurement of such indicators, OHI Claims provides the following mechanisms: Event based integration points that can be used to inform external (reporting) systems about changes in claim states. Database views that be used for creating reports.
The remainder of this chapter deals with monitoring the performance or throughput of OHI Claims. Technical monitoring and management of OHI Claims is documented in a separate chapter.
4.1 Monitoring Throughput

For OHI Claims we define throughput as: the number of claims (or claim lines) that can be automatically processed by the claims process from Canonical Claims In up to the Finalized state in a given time period. Note that this definition depends largely on: The setup of the system. The processing capacity of the system.
Properly sizing the system should ensure that processing capacity is sufficient for handling normal workloads and occasional exceptionally large workloads. The following query determines the number of claims that are Finalized on a given day. It can be executed as OHI_CLAIMS_USER:
Operations Guide
26
select from where and
count(*) cla_claim_status_history status = 'FINALIZED' date_time between trunc(sysdate - 1) and trunc(sysdate)
Of course the timeframe for the measurement could vary. Note however that by itself this is a tricky metric to monitor, i.e. to measure frequently and make meaningful assertions regarding the trend of the measurements. For example, if there is no backlog of work in the queried timeframe, it may appear as if the systems throughput is low (whereas there was simply no work to perform). Hence, the figure by itself is rather meaningless, it should always be judged in the proper context. A better approach is to keep track of the mean time that is required to perform certain tasks. The following query lists the average processing times per task
select taty.name tasktype , round(avg(EXTRACT(DAY FROM(task.process_stop_datetime-task.process_start_datetime) DAY TO SECOND)*86400 + EXTRACT(HOUR FROM(task.process_stop_datetime-task.process_start_datetime) DAY TO SECOND)*3600 + EXTRACT(MINUTE FROM(task.process_stop_datetime-task.process_start_datetime) DAY TO SECOND)*60 + EXTRACT(SECOND FROM(task.process_stop_datetime-task.process_start_datetime) DAY TO SECOND)),0) avg_process_elapsed_time_ss from ohi_tasks task , ohi_task_types_tl taty where task.taty_id = taty.id and taty.language = 'en' and task.process_stop_datetime is not null and task.creation_date between trunc(sysdate - 1) and trunc(sysdate) group by taty.name;
4.2 Monitor the Work backlog

The work backlog in OHI Claims is mainly determined by the number of tasks that the system still has to perform. That number is simply the amount of tasks in the task queue. Use the following query to determine the number of remaining tasks at any given time. It can be executed as OHI_CLAIMS_OWNER:
select count(*) from aq$ohi_task_queue_table where msg_state != 'PROCESSED';
Note that a view is used to query the queue. Do not access the queue table directly, always use the read-only view!
A growing work backlog is either: A good indicator to determine if processing capacity must be added or A signal that the system is not performing as expected
It is acceptable for the work backlog to increase in busy periods. A properly sized system will 'catch up' as soon as the claim volume returns to normal proportions.
Operations Guide
27
Advice: monitor the work backlog frequently and follow the trend. If there is reason for concern, use other metrics for investigating the cause (e.g. is the mean time that is required for processing specific tasks increasing, see previous paragraph). Define a threshold for a specific peak level, typically a level at which meeting service levels is at risk. If the peak level is reached, be prepared to add additional processing capacity by adding servers to the cluster.
4.3 Task Processing Principles

The following principles apply for processing tasks from the work queue: Tasks are processed in order of priority first and time that the tasks were entered into the queue second. Once processing for a claim starts, the priority for subsequent tasks related to the same claim increases. This ensures that claims for which processing started will be finished before processing for newly entered claims starts ('first finish what you started'). The priority for some low-frequency tasks is set to a slightly increased value in order to have acceptable wait time before these are processed. Examples of such tasks are Financial Message Batches and File Import Tasks. Tasks are picked up if and when processing capacity is available.
The following table provides an overview of the (relative) priorities for different tasks. A higher priority means that the task is executed before any tasks that have a lower priority. Any other tasks have a priority of 0.
Task Claims Import Claims Processing Initialization Claims Processing Sanity Checks Claims Processing Enrollment Check Claims Processing Prepricing Claims Processing Pricing Claims Processing Payment Status Check Claims Processing Benefits Claims Processing Adjudication Claims Processing Finalization File Import Financial Message Generation Counters Batch requests Priority 0 1 2 3 4 5 6 7 8 9 1 1 1
Priorities cannot be influenced.
4.4 Technical Monitoring and Management

This chapter deals with technical monitoring and management of the OHI Claims application. For this purpose, OHI Claims is equipped with a technical management interface. It supports the operations like: Operations Guide 28
Event Noticiations, for example in case of failures in evaluating Dynamic Logic scripts. Query status and gathering of statistics like the average run time of Dynamic Logic scripts. Automate technical operations, for example restarting errored tasks.
The management interface provides insight into the technical state of OHI Claims, i.e. it is not meant to monitor the operations of the "Claims Shop". This shows in the management interface; it is implemented using the Java Management Extensions (JMX Managed Beans or MBeans) and not suitable for use by business users. MBeans are accessible through management tools like Oracle Enterprise Manager Grid Control (recommended) or the WebLogic Console, using the WebLogic Diagnostic Framework (WLDF). Technical diagnostics, metrics and notifications can be defined to periodically check the availability of resources and the occurrence of errors. 4.4.1 Accessing Mbeans For accessing OHI Claims MBeans, the following Java Options should be added to the startWebLogic.sh script:
JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.sun.management.jmxremote.port=< free_port_number>" JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.sun.management.jmxremote.authenticate=false" JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.sun.management.jmxremote.ssl=false" JAVA_OPTIONS="${JAVA_OPTIONS} -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WL SMBeanServerBuilder"
Note: do not put this in the setDomainEnv.sh if it is necessary to also use the stopWebLogic.sh script as that will not work due to port conflicts in that case. After starting the WebLogic server with these configuration changes, it is possible to access the OHI Claims MBeans, for example using JConsole, by using the remote URL <machine_name>:< free_port_number>. Security recommendation: set com.sun.management.jmxremote.ssl to true if the environment is configured to use SSL certificates.
4.4.2 OHI Claims Event Log OHI Claims logs events that require manual intervention in its Event Log. The following events are logged: Runtime failures in Dynamic Logic scripts External Web Services not being available Failures to complete a processing task
The events are persistently stored in the system and can be viewed by using the System Event Logs page as is shown in the following graphic:
Operations Guide
29
The event log is designed for interoperability with the OHI Claims EMGC management plugin. 4.4.3 Usage scenario The high-level operation of the eventing mechanism is as follows: OHI Claims logs events in case of certain runtime failures. The OHI Claims EMGC Management Plugin comes with predefined metrics for detecting and handling these events. It creates alerts in case of failures and sends out a notification using the EMGC's notification mechanism. It is up to the customer to select and configure the notification mechanism, e.g. e-mail or SNMP. Upon receiving a notification, the problem(s) should be triaged. The System Events UI page (shown above) provides more detailed information that helps to determine the cause of the error. If the problem is fixed, the event should be marked as Resolved (check box) in the System Events UI page. This will clear the alert in EMGC. For failed tasks the System Events UI page will have a link to the Technical Error page to restart the task immediately. Doing that will automatically flag the event as Resolved.
4.4.3.1 Flood control The metrics that are defined in the OHI Claims EMGC management plugin create an alert as soon as one (and only one) of the predefined events occurs. That will trigger the notification mechanism. Once the alert is raised and as long as the problem is not resolved (i.e. flagged as Resolved in the System Events UI page), no new or additional notifications will be send for similar events. After resolving all the problems of that type the alert is cleared. Only after that new notifications will be send for new events that occur. 4.4.3.2 Duplicate Notifications In some cases, more than one alert may be raised (triggering multiple notifications) as a result of the same problem. For example: processing a claim fails in the Benefits task as a result of a runtime error in evaluating Dynamic Logic. If the alerts for these events were not yet raised, both for the runtime failure in the dynamic logic as well as for the failing task a notification will be send. Operations Guide 30
4.4.4 OHI Claims Management plugin for EMGC 4.4.4.1 Prerequisites EMGC 11gR1 (or better) has to be installed. Agents have to be installed on systems that need to be monitored. Set up notifications in EMGC. The EMGC product documentation describes how that is done. 4.4.4.2 Installation of the OHI Claims Management Plug-in OHI Claims Management Plug-in is bundled in the release bundle (<RELEASE_ZIP> /monitoring/OHIClaimsManagement.jar). The first step is to import this Management Plug-in into EMGC. Perform the following steps in EMGC console: Step 1: From the Enterprise Manager console, click Setup.
Step 2: Click Management Plug-ins from the left navigation bar and click Import.
Step 3: From the Import page, specify the Management Plug-in Archive file ( OHIClaimsManagement.jar) and click List Archive.
Operations Guide
31
Step 4: Select "OHIClaimsManagement" from the list and click OK. Enterprise Manager returns you to the Management Plug-ins page with OHIClaimsManagement plug-in added to the list.
Step 5: After the Management Plug-in has been imported into Enterprise Manager, you can deploy the plug-in to any number of Management Agents. From the Management Plug-in main page, click Deploy icon for the Management Plug-in you want to deploy. The Deploy Management Plug-in: Select Targets page appears, as shown in the following figure.
Step 6: Click Add Agents. The Search and Select: Agents page appears. Choose the desired Management Agent. Use the appropriate search parameters to find the desired Agent. Click Select. Enterprise Manager returns you to the Deploy Management Plug-in: Select Targets page. The selected Agent(s) appears in the deployment list.
Operations Guide
32
Step 7: Click Next to go to the Review page.
Step 8: Click Finish.
4.4.4.3 Configure metrics Now that the Management Plug-in has been deployed to the appropriate Agent(s), you are ready to configure metrics. Step 1: From the Management Plug-in main page, click Agents link.
Operations Guide
33
Step 2: Click on the agent on which you deployed the Management Plug-in.
Step 3: From the Monitored Targets section of the Management Agent home page, choose OHIClaimsManagement from the Add drop-down menu and click Go.
Step 4: Enter the requisite target properties referring the table below.
S.No 1 Field Name Name Field Value Any desired name. For example, OHIClaimsManagementMetrics
Operations Guide
34
Machine name
Machine name or IP address on which WebLogic Server is running WebLogic Server Admin port weblogic (WebLogic Server Admin Username) WebLogic Server Admin Password t3 weblogic.management.mbeanserv ers.runtime WebLogic Leave if blank
3 4 5 6 7 8 9
Admin Port number User Name JVM Admin User Password Communication Protocol Service Name Metric Source Custom lookup provider Class
Step 5: Click on OK button.
4.4.4.4 Viewing metrics and alerts Step 1: From the Agent home page, click on the target instance you added in the previous step in the Monitored Targets list. Enterprise Manager takes you to that target's home page.
Step 2: In the Related Links section, click All Metrics. The All Metrics page appears for the monitored target. An expandable tree list for each metric enables you to drill down to view specific metric parameters, as shown below.
Operations Guide
35
Step 3: Whenever the metric value is greater than the configured critical threshold value (which is 1 by default), a critical alert is raised (which in-turn can trigger an E-Mail notification if you have configured).
The following is a sample critical alert E-Mail Notification.
4.4.4.4.1 Customizing Metric and Policy Settings By default, the configured threshold value is 1 and the metric collection interval is 300 seconds. This means, every 300 seconds EM Management Agent polls for metrics and if the metric value is greater than or equal to 1, it will create an alert. If you would like to change the collection interval or threshold value, perform the following steps: Step 1: In the Related Links section, click Metric and Policy Settings.
Operations Guide
36
Step 2: Click on Edit icon for the metric for which you want to customize the threshold setting. Step 3: Edit the threshold values to suit your needs and click on Continue button.
Step 4: Click on Every 300 Seconds link if you want to customize Collection Schedule.
Step 5: Edit Repeat Every value to suit your needs and click on Continue button.
Step 6: Click OK. 4.4.4.5 Installation in a clustered environment Install the Management Plug-in on one of the nodes in a cluster in order to avoid receiving duplicate notifications.
Operations Guide
37
4.4.5 Monitoring of OHI Claims using profiling tools A word of caution on monitoring Oracle Fusion Middleware, specifically the JVM - using profiling tools like Yourkit. These tools can be very valuable, but can be invasive and impact system performance and stability under certain conditions.
Operations Guide
38
5 Logging
5.1 Logging and Auditing Operations
OHI Claims supports several levels of logging and auditing that can be used to gather and assess information about the runtime state of the system. 5.1.1 Types of Log Files The following types of log files are distinguished: Application log Security log
5.1.1.1 Application Log What's in the application log How to control application log 5.1.1.2 Security Log Definition of Auditing Log What's in the auditing log how to control the auditing log 5.1.2 Logging Configuration 5.1.2.1 Configuration of an appender The following snippet of XML shows an example for the configuration of a log file appender for writing log statements to a file:
<appender name="fileAppender" class="org.apache.log4j.RollingFileAppender"> <param name="File" value="target/ohiApplication.log" /> <param name="Append" value="false" /> <param name="MaxBackupIndex" value="5" /> <param name="MaxFileSize" value="100MB" /> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d{ISO8601} [ %t ] %p %c %m%n" /> </layout> <filter class="com.oracle.healthinsurance.logging.log4j.SecurityLevelFilter"> <param name="inclusive" value="false" /> </filter> </appender>
The following table explains the configuration.

Parameter Appender class Sample Value Explanation
org.apache.log4j.RollingFileAppe Creates log files with rollover. nder Log file rolls over when the size MaxFileSize is reached. 100MB Maximum file size for a log file before it is rolled over.
MaxFileSize
Operations Guide
39
MaxBackupIndex File
5 target/ohiApplication.log
Number of log files that are retained. Path and name of the log files. When roll over occurs, the current file is moved to a file target/ohiApplication.log.X where X is the following number in sequence. Controls the layout and output format according to conversion patterns similar to the C language printf function. See the ConversionPattern Java documentation1 for more details. Inclusion of this filter class and parameter ensures that security message are not written by the corresponding appender. To ensure that security messages are only written to the security log, this filter class and parameter should be defined on all appenders. The specific configuration of the securityAppender is defined below.
org.apache.log4j.PatternLayout and log4j.appender.logfile.layout.Con versionPattern
%d{ISO8601} [ %t ] %p %c %m%n
com.oracle.healthinsurance.loggin false g.log4j.SecurityLevelFilter and inclusive
For more information on log4j configuration see the Apache Log4j website2. Another useful source is the Wiki that describes the Log4jXmlFormat3. 5.1.2.2 Security audit log OHI Claims issues specific, security related log statements. Examples include the following: 2010-06-11 21:01:24,077 [ [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' ] SECURITY com.oracle.healthinsurance.provisioning.service.impl.ProvisioningServiceImpl - User 322 with loginName FN0023P0002 was created. 2010-06-11 21:01:24,398 [ [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)' ] SECURITY com.oracle.healthinsurance.provisioning.service.impl.ProvisioningServiceImpl - User 322 with loginName FN0023P0002 was modified.
For auditing purposes, the setup for the security audit log is likely to be different, e.g. security audit data is retained for a longer period of time. Security related log data can be captured in an appender by specifying filter class "com.oracle.healthinsurance.logging.log4j.SecurityLevelFilter" as is shown in the following example:
<appender name="securityAppender" class="org.apache.log4j.RollingFileAppender"> <param name="File" value="target/ohiSecurity.log" /> <param name="Append" value="false" /> <param name="MaxBackupIndex" value="5" /> <param name="MaxFileSize" value="100MB" /> <layout class="org.apache.log4j.PatternLayout"> <param name="ConversionPattern" value="%d{ISO8601} [ %t ] %p %c %m%n" /> </layout> <filter
Operations Guide
40
class="com.oracle.healthinsurance.logging.log4j.SecurityLevelFilter" /> </appender>
5.1.2.3 Predefined logging configurations OHI Claims comes bundled with a number of predefined log4j configurations: 1. 2. 3. 4. log4j.xml production-log4j.xml development-log4j.xml trace-log4j.xml
By default, log4j.xml is used. To use one of the others, use the -Dohi.log4j.type Java option in the setDomainenv script:
-Dohi.log4j.type=production | development | trace
Note that this is overruled if the flag -Dohi.log4j.config.file is specified (see next section). 5.1.2.4 Performance logging The following configuration enables the logging of timing information for User Interface actions:
<logger name="com.oracle.healthinsurance.sla.filter"> <level value="debug"/> </logger>
For each UI request, the elapsed time needed to process the request is logged like this:
DEBUG com.oracle.healthinsurance.sla.filter.SlaLogger - [UI] Request: userId=42;contextPath=/base; requestURL=http://localhost:7001/base/faces/SearchClaims;page=View Claim CL0025_reprocess_2_15;time(ms)=2152
Note: the entire message is logged on one line but was formatted differently in this guide for readability. The following configuration enables the logging of timing information for Web Services requests:
<logger name="com.oracle.healthinsurance.servletfilter"> <level value="debug"/> </logger>
For each WS request, the elapsed time needed to process the request is logged as follows:
DEBUG com.oracle.healthinsurance.servletfilter.ThreadlocalResolver - [WS] Request: contextPath=/ohi-web-services; requestURL=http://localhost:7001/ohi-web-services/ProviderImportService;t ime(ms)=58
Note: the entire message is logged on one line but was formatted differently in this guide for readability. 5.1.2.5 Using a custom log4j configuration Steps for using a customized log4j configuration: 1. 2. Create a custom log4j.xml configuration file. Use the -Dohi.log4j.config.file Java option in the setDomainenv script to point to the custom configuration file. For example:
Operations Guide
41
-Dohi.log4j.config.file=/fully/qualified/path/to/log4j.xml
5.1.2.6 Log4j Log Levels Possible log levels, in order of no logging to all logging: off fatal error warn info debug trace all
The OHI custom level "security" is between warn and info. 5.1.2.7 Associating a log level with an appender The following configuration example shows how to specify the logging detail level with a specific appender:
<root> <level value="debug"/> <appender-ref ref="fileAppender" /> </root>
It is also possible to customize log levels for specific parts of the system or for specific Java classes. This may be requested for support purposes. Oracle recommends to use a fileAppender for OHI Claims logging only. Do not log to the console also as this does not provide additional data (only the same data in a multiple places) and the additional logging decreases the performance of the system.
5.1.2.8 Making changes to logging Care should be taken in changing logging parameters. For example, more fine-grained logging or logging to multiple channels impacts the performance of the system. Note that any changes made to a log4j.xml configuration file are activated immediately without restarting the system.
1. 2. 3. http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PatternLayout.html http://logging.apache.org/log4j/ http://wiki.apache.org/logging-log4j/Log4jXmlFormat
Operations Guide
42
6 Error Recovery
This chapter describes the capabilities provided by the system to recover from errors. Recoverability in OHI Claims manifests in different places and at different levels of operation. Examples of technical errors are: External web services endpoints are unavailable, e.g. because specific systems are down or as a result of network failures Errors in dynamic logic scripts
Typically, technical errors impact a large number of similar operations and recovery costs could be high (both in terms of system resources that are required for recovery as well as regarding risks to meet service level agreements). Therefore, it is important to detect and resolve any error as soon as possible in order to restart failing or incomplete tasks with a minimum of reprocessing, in as close as possible to the state prior to the incident. Besides recovery from technical errors in individual processes, the system is able to recover from system or node failure.
6.1 Recover from Application Failures

6.1.1 Tasks Any processing in OHI Claims is based on units of work referred to as tasks. Error recovery is centered around tasks. Tasks go through different states: Initial: the task is initialized. This involves execution of basic sanity checks and construction of the task's context. Pending: the task is executed. Completed: execution of the task ended as expected. Errored: an error occurred that prevented the task from completing successfully.
The claims process flow in OHI Claims is built up as a series of distinct tasks. Tasks in the flow allow the following: Fail fast in order to recover as close as possible to the state prior to the incident. Restarting from a particular task prevents reprocessing that would happen if processing of a claim had to be restarted from the beginning. The goal is to resume processing by only repeating the task in which the error occurred.
6.1.2 Recover errored tasks in the Claims Processing Flow In a number of cases, recovery requires manual intervention: after the problem is resolved, processing of errored tasks must be resumed. This is done through the Recovery from Technical Errors screen. The screen was designed with the following principles in mind: Consistency and simplicity: the screen provides a generic recovery facility that applies to any task in the system. Different error-processing on a per-task basis would be harder for operators. Ease of maintenance: when the root cause of the problem is resolved, the system supports (manual) recovery from the error in a simple way for all tasks that failed as a result of it. 43
Operations Guide
Manually restarting a large volume of errored tasks one by one is not a valid option. Therefor, the system supports bulk operations to achieve this.
The screen is used to view tasks that are in a technical error state. Once the root cause of a problem has been identified and resolved, the operator can then trigger the resumption of processing on either an individual claim, or on the whole batch of similar tasks in the error state. The screen has three main features used to: detect tasks in the errored state aid in issue resolution and resubmit them for processing.
6.1.2.1 Errored Task Listing The screen shows a table with the following columns, in this order: Last Updated Date, the date and time when this task was last run Task Name, the name of the task in errored status Type, the entity type of the Target associated with this task, for example a Claim Code, the code of the item associated with this task, or if the entity does not have a code, the ID. Note that no data access restrictions apply on the Claim code in this page. Any user with access to this page should be able to see all Claim codes (and the codes of other entities where data access restrictions might apply in other pages) Actions Details: shows the full stack trace captured when the task errored Process: resubmit the claim to the claims flow
Quick Search and Advanced Search are available on Last Updated Date, Task Name and Type. Advanced Search has between search on Last Updated Date. 6.1.2.1.1 Process All The process all button requests processing on all tasks returned to the screen by a query. Operations Guide 44
6.1.2.1.2 Actions: Details and Process Each row in the errored claim screen has a feature for showing more details about the failed task, or for restarting that task. The exception details are a single text box containing a stack trace, formatted as a readable java stack trace. When a target is re-submitted to the processing flow the status of that task changes from error to pending, and the screen contents are re-queried. In this fashion, a task can not be returned for processing multiple times. 6.1.3 Recover from external service failures For recovery from failures to deliver a message to an external or outbound service the following cases are distinguished: 6.1.3.1 Failing to deliver a message for which processing of a claim should not be interrupted For example: failure to deliver claims events messages. The timing for delivering these types of messages is not critical. The task for delivering the claims event message goes into the errored state (the guiding principle being that the event message needs to be delivered) but processing of the particular claim continues by spawning the next task. The failing task must be re-started from the Recovery from Technical Errors screen. 6.1.3.2 Failing to deliver a message for which the Claims Flow should be interrupted Examples are enrollment request messages: enrollment data is a necessity for adjudication of a claim. The task goes into the errored state and processing of the particular claim is stopped. The failing task must be re-started from the Recovery from Technical Errors screen. 6.1.4 Recover from failures in batch processes Processing the contents of large files may take a long time, potentially several hours. Therefore, OHI Claims file processing is handled in a batch processing mode. The batches keep track of an internal state. A failing batch process can be restarted from its last known commit point. For consistency and ease of recoverability, a task is created for each batch process. This ensures that the Recovery from Technical Errors screen can be used to re-start an unfinished batch process. 6.1.5 Recover messages from the Task Error Queue OHI Claims makes use of a database queue for workload balancing. As soon as unit of work is picked up from this persisted queue, a task is initialized. If task initialization succeeds the task is recoverable, even after a (catastrophic) system failure. in the rare case that dequeuing fails, the message from the database queue is placed in the associated error queue. In order to process these messages, the system provides JMX managed beans to: Detect if there are messages in the error queue Resubmit messages from the error queue to the work queue
6.2 Recover from system failure

Catastrophic system failure Node failure
Operations Guide
45
7 Copying an Environment
There can be several reasons why you would want to copy (or clone) an environment: Copy a setup environment as the initial version of a test environment Copy a setup environment as the initial version of the production environment Copy the production environment to the test environment
This chapter describes the procedure for copying environments. The Oracle software that runs the OHI Back Office application need to be installed first: Follow the instructions for install and configure an OHI database (page 0). Follow the instructions for install and configure Oracle Fusion Middleware (page 0).
7.1 Clone the Database

Use the RMAN backup tool to duplicate the database. To do that, follow the standard Oracle Database Backup and Recovery User's Guide1, especially Chapter 24, Duplicating a Database2.
7.2 Empty the Task Queue

Before the application that uses the cloned database is started, the task queue needs to be emptied by executing the following:
DECLARE po_t dbms_aqadm.aq$_purge_options_t; BEGIN dbms_aqadm.purge_queue_table('OHI_TASK_QUEUE_TABLE', NULL, po_t); END;
Note that performing direct data manipulation on the OHI_TASK_QUEUE_TABLE is not supported.
7.3 Enable Replication of Setup Data

In accordance with the concepts explained in the Installation Guide, paragraph Enabling Replication of Setup Data, it may be a requirement to ensure that unique identifiers that are generated in the cloned environment are unique across all OHI environments (e.g. in case data needs to be transported between environments using the Configuration Migration tool). The following script removes any existing data from the ohi$instances table and inserts a new value for the cloned environment (change the host, port, database sid and description to the required values):
delete from ohi$instances; insert into ohi$instances (ID, HOSTNAME, PORT, SID, DESCRIPTION, DISCRIMINATOR) values (ohi$instance_s1.nextval, '<host setup db>', <port setup db>, '<sid setup db>', '<db description>', <single digit unique discriminator value>);
Operations Guide
46
Now, use the setSequences utility to reset sequences for transportable tables, to conform to the definition in the ohi$instances table. See the Setting sequences (page 10) chapter for instructions.
7.4 Deploy the Application

Follow the instructions in the Installation Guide for installing the OHI Claims application on the application server. Note that properties files and data source configuration need to be specifically set up for the target system.
7.5 Validate Installation

Validate the installation by performing the steps that are documented in the Installation Guide, section Validate Installation.
1. 2. http://download.oracle.com/docs/cd/E11882_01/backup.112/e10642/toc.htm http://download.oracle.com/docs/cd/E11882_01/backup.112/e10642/rcmdupdb.htm#i1008564
Operations Guide
47
8 OBIEE Reporting Views

8.1 Setup the OBIEE Repository
This section describes how to set up the OBIEE environment to make use of the OHI Reporting Views OBIEE repository. Besides the installation of the repository itself, it is necessary to customize the OBIEE repository. This is because the Reporting Views can be changed per installation. This is described in Customize the OBIEE Repository (page 49). Setup In this paragraph a brief instruction is given how to setup the Reporting Views repository (.RPD file) into OBIEE. After this setup and the customization, the reporting views can be accessed with OBIEE. Before the repository can be installed, the OBIEE software must be installed. The installation procedure is described in the following document: http://download.oracle.com/docs/cd/E14571_01/bi.1111/e10539/toc.htm The .RPD file translates the OBIEE analysis to queries to the database. Therefore it is necessary to define the connection details of the database where the views are present. Enter the database SID and the correct user to access the views. This can be done in the connection pool object in the physical layer. After the OBIEE 11g software installation, open the Enterprise Manager of Fusion Middleware Control 11g. Choose the Business Intelligence folder in the domain. Click on 'coreapplication'. An overview of the BI components is shown. Click on the tab 'Deployment'.
The Deployment page is used to load a repository file onto the server. Make the deployment updatable by clicking on the 'Lock and Edit Configuration' button. Under 'Upload BI Server Repository' click on the browse button and choose the 'OHI Reporting Views.rpd' file. Enter the password of the repository (default = oracle, change this to a new password). Click on the 'Apply' button in the top right of the screen. Now restart the OBIEE server and the repository is available in OBIEE.
Operations Guide
48
8.2 Security
Introduction The reporting views make use of the OHI Claims application security settings. This means that users can only see data that they are also allowed to see in the OHI Claims. The context is set before a user is executing a query in OBIEE. Users The users defined in OBIEE must be the same as the application users in the OHI Claims application
8.3 Customize the OBIEE Repository

The concept of the reporting views is that they can be customized per installation. Therefore it is most likely that the default setup in the OBIEE repository does not match the specific customized views. In this chapter, a brief instruction is given how to customize the OBIEE repository to the specific needs. It is recommended that the customizations in the OBIEE repository are done by someone with administration knowledge of OBIEE. Detailed information about administration of an OBIEE repository can be found here: http://download.oracle.com/docs/cd/E14571_01/bi.1111/e10540/toc.htm In the default OBIEE repository only functional views are added (name starting with FUN) Add new view When there are customized own functional views, they can be added to the repository: 1. Import the view in the physical layer Add the view that must be added into the existing 'Datasource Reporting Views' connection group. Views can be imported into the physical layer by using the metadata import functionality (can be found under 'File') Add the new view to the business layer by dragging it into the OHI Reporting Views subject area. Make a connection with the dummy dimension Note: OBIEE requires that a 'fact' table is connected to a 'dimension' table. Because the views are 1:1 added in the business model without dimension modelling, a dummy dimension is created. Create a connection with this dummy dimension. Add the new view to the presentation layer Check in the changes Save the repository
2. 3.
4. 5. 6.
Remove view When a functional view in the OBIEE repository does (not longer) exists in the customized environment, the view can be removed in OBIEE: 1. 2. 3. 4. 5. Delete the presentation object for the view. Delete the business object for the view. Delete the physical definition of the view. Check in the changes. Save the repository.
Operations Guide
49
Change existing view When a column is added, removed or renamed in an view that already exists in the OBIEE repository. The repository needs to be changed: 1. 2. 3. Open the view definition in the physical layer that must be changed Add, delete or rename the column In case of adding or deleting a column, also add or delete the column in the presentation and business layer.
Operations Guide
50
9 Deeplinking to screens
The workflow interface will supply a URL for the user to deeplink to certain UI pages (from outside the application). This section describes the structure of the URL so that it may be used for other deep-linking purposes as well. Claims Pages Example: The url below opens the View Claim page and queries 1 particular Claim (for an authenticated user; otherwise the Login page is presented first): http://host.domain:port/base/faces/SearchClaims?jhsTaskFlowName=ViewClaims& rowKeyValueClaims=17 To create such a url, several parts are concatenated: 1. 2. 3. 4. http://<host:port>/<appl.name>. The part of the URL that is shown before "/faces". /faces/SearchClaims?jhsTaskFlowName=ViewClaims &rowKeyValueClaims=<id>. Ensures that one row is automatically queried when the page is entered. Use the id in the database ID column. For opening a different variation of the claim page, add: &uniqueIdentifier=< edit/mpricing/mbenefits/madjudication/unfinalize>. This will only work correctly if the id of part 4 is of a claim with the right status for that page, otherwise the page states that no rows were found.
Other Pages The url below opens the Persons page and queries 1 particular Person (for an authenticated user; otherwise the Login page is presented first): http://localhost:7001/base/faces/UIShell?jhsTaskFlowName=Persons& jhsSubMenuName=ReferenceDataMenuModel&rowKeyValuePersons=10242 To create such a url, several parts are concatenated: 1. 2. http://<host:port>/<appl.name>. The part of the URL that is shown before "/faces". /faces/UIShell?jhsTaskFlowName=<pageCode>. The page code can be retrieved by going to the right page, opening the "About this page" popup, and looking up the Access Restriction Code. &jhsSubMenuName=<menuName>. The allowable values for menuName are ReferenceDataMenuModel and ConfigurationMenuModel. In theory, any page can be combined with any menu, but of course the menu to which the page belongs should be used. (optional) parameter: &rowKeyValue<pageCode>=<id>. Ensures that one row is automatically queried when the page is entered. Use the same page code as in the previous step, and use the id in the database ID column. If this parameter is not specified, the page will be opened without querying any data.
3.
4.
Operations Guide
51

Operations Guide

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Operations Guide

Uploaded by

Copyright:

Available Formats

Operations Guide

October 21, 2011

Copyright 2011, Oracle and/or its affiliates All rights reserved

8 OBIEE Reporting Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

1.2 Target audience

1.3 Guiding principle

1.4 References to artifacts in the OHI Claims distribution

2.2.1.2 Application tier Multiple application servers Exalogic Elastic Cloud

2.2.1.3 Hardware Clustering Virtualization

2.3 Schema verification

Option Short -f Long --file

digest file path

config file path

2.4 Setting sequences

Command line arguments

2.5.1.3 Additional topics Use password policy 114g5 defaults6 $ORACLE_HOME/rdbms/admin/utlpwdmg.sql

Security changes Database links Replication Errors

Keep optimizer parameters8 on default values

Always report to Oracle Support Services Report back to OHI Development

2.7 Oracle Text index maintenance

For rebuilding all the Text indexes execute the following:

2.9 Fault management

DB_ULTRA_SAFE => changes defaults for

DB_BLOCK_CHECKING, DB_BLOCK_CHECKSUM, DB_LOST_WRITE_PROTECT Support Workbench (OEM; database & ASM)

2.10 Do's and dont's

2.11 Generate Reporting Views

ORA-00923: FROM keyword not found where expected

2. Connect to the Node Manager:

4. Optionally tail the server out and/or log file:

$ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>.log

2. Connect to the Node Manager:

3.2 The System's start up routine

select , , from , where and and and

taty.id 'en' 'B' -- Processing to_date(<time_system_stopped>);

select * from aq$ohi_task_queue_table where expiration_reason != null;

3.3 Stopping the WebLogic Cluster

2. Connect to the Node Manager:

Replace <domain-name> and <admin-server-name> with correct values.

2. Connect to the Node Manager:

4.1 Monitoring Throughput

select from where and

count(*) cla_claim_status_history status = 'FINALIZED' date_time between trunc(sysdate - 1) and trunc(sysdate)

4.2 Monitor the Work backlog

4.3 Task Processing Principles

Priorities cannot be influenced.

4.4 Technical Monitoring and Management

Step 7: Click Next to go to the Review page.

Step 8: Click Finish.

Step 5: Click on OK button.

The following is a sample critical alert E-Mail Notification.

The following table explains the configuration.

org.apache.log4j.PatternLayout and log4j.appender.logfile.layout.Con versionPattern

com.oracle.healthinsurance.loggin false g.log4j.SecurityLevelFilter and inclusive

class="com.oracle.healthinsurance.logging.log4j.SecurityLevelFilter" /> </appender>

6.1 Recover from Application Failures

6.2 Recover from system failure

7.1 Clone the Database

7.2 Empty the Task Queue

7.3 Enable Replication of Setup Data

7.4 Deploy the Application

7.5 Validate Installation

8 OBIEE Reporting Views

8.3 Customize the OBIEE Repository