Operations Guide

October 21, 2011

Copyright © 2011, Oracle and/or its affiliates All rights reserved

Oracle® Health Insurance Copyright © 2011, Oracle and/or its affiliates. All rights reserved. License Restrictions & Warranty Disclaimer The Programs (which include both the software and documentation) contain proprietary information; they are provided under a license agreement containing restrictions on use and disclosure and are also protected by copyright, patent, and other intellectual and industrial property laws. Reverse engineering, disassembly, or decompilation of the Programs, except to the extent required to obtain interoperability with other independently created software or as specified by law, is prohibited. The information contained in this document is subject to change without notice. If you find any problems in the documentation, please report them to us in writing. This document is not warranted to be error-free. Except as may be expressly permitted in your license agreement for these Programs, no part of these Programs may be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose. Restricted Rights Notice U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are "commercial computer software" or "commercial technical data" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the Programs, including documentation and technical data, shall be subject to the licensing restrictions set forth in the applicable Oracle license agreement, and, to the extent applicable, the additional rights set forth in FAR 52.227-19, Commercial Computer Software--Restricted Rights (June 1987). Oracle USA, Inc., 500 Oracle Parkway, Redwood City, CA 94065. The Programs are not intended for use in any nuclear, aviation, mass transit, medical, or other inherently dangerous applications. It shall be the licensee's responsibility to take all appropriate fail-safe, backup, redundancy and other measures to ensure the safe use of such applications if the Programs are used for such purposes, and we disclaim liability for any damages caused by such use of the Programs. Third Party Web Sites, Content, Products, and Services Disclaimer The Programs may provide links to Web sites and access to content, products, and services from third parties. Oracle is not responsible for the availability of, or any content provided on, third-party Web sites. You bear all risks associated with the use of such content. If you choose to purchase any products or services from a third party, the relationship is directly between you and the third party. Oracle is not responsible for: (a) the quality of third-party products or services; or (b) fulfilling any of the terms of the agreement with the third party, including delivery of products or services and warranty obligations related to purchased products or services. Oracle is not responsible for any loss or damage of any sort that you may incur from dealing with any third party. Trademark Notice Oracle, JD Edwards, and PeopleSoft are registered trademarks of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Draft Documentation Notice This documentation is in prerelease status and is intended for demonstration and preliminary use only. It may not be specific to the hardware on which you are using the software. Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to this documentation and will not be responsible for any loss, costs, or damages incurred due to the use of this documentation. The information contained in this document is for informational sharing purposes only and should be considered in your capacity as a customer advisory board member or pursuant to your beta trial agreement only. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described in this document remains at the sole discretion of Oracle. This document in any form, software or printed matter, contains proprietary information that is the exclusive property of Oracle. Your access to and use of this confidential material is subject to the terms and conditions of your Oracle Software License and Service Agreement, which has been executed and with which you agree to comply. This document and information contained herein may not be disclosed, copied, reproduced, or distributed to anyone outside Oracle without prior written consent of Oracle. This document is not part of your license agreement nor can it be incorporated into any contractual agreement with Oracle or its subsidiaries or affiliates.

Operations Guide

1

Table of Contents

1 Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.1 Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Target audience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Guiding principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 References to artifacts in the OHI Claims distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 6 6 6

2 Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.1 Availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1.1 Backup/restore tips. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.1.2 Oracle Maximum Availability Architecture . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2.1 Cloning databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2.2 Active Data Guard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2.1.2.3 RAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2 Scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1.1 Database tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1.2 Application tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.1.3 Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.2.2 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 Schema verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.4 Setting sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.5 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1.1 How to address security issues? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1.2 Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2.5.1.3 Additional topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.5.1.4 Auditing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 2.5.2 OHI Specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.1 Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.1.1 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.2 Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.2.1 Optimizer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 2.6.2.2 Statistics types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.2.3 Database parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.2.4 Statistics collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.2.5 Miscellaneous. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.3 I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.3.1 OHI specific guidlines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6.3.2 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6.4 Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.6.4.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Operations Guide 2

2.6.4.2 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.7 Oracle Text index maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.7.1 Index Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.7.1.1 Dealing with Synchronization errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.7.2 Index Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 2.8 Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.8.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.9 Fault management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.9.1 Generic guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.9.1.1 Reduce time to resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 2.9.1.2 Oracle 11g Database Features on Fault Management. . . . . . . . . . . . . . 18 2.10 Do's and dont's . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.10.1 OHI specific guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.10.1.1 Do . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.10.1.2 Do not . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.11 Generate Reporting Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3 Fusion Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
3.1 Starting the WebLogic Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.1.1 Start the Administration Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3.1.1.1 Use the Node Manager to start the Administration Server . . . . . . . . . 21 3.1.1.2 Use the start script to start the Administration Server . . . . . . . . . . . . . 21 3.1.2 Start the Managed Server(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 3.1.2.1 Use the Administration Console to start the Managed Server(s) . . . . . . 22 3.1.2.2 Use the Node Manager to start the Managed Server(s) . . . . . . . . . . . . 22 3.2 The System's start up routine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2.1 Actions performed when the system starts up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.2.2 Verify the results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3 Stopping the WebLogic Cluster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3.1 Stop the Administration Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3.3.1.1 Use the Node Manager to stop the Administration Server . . . . . . . . . 23 3.3.1.2 Use the Administration Console to stop the Administration Server . . . 24 3.3.1.3 Use the stop script to stop the Administration Server. . . . . . . . . . . . . . 24 3.3.2 Stop the Managed Server(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3.2.1 Use the Administration Console to stop the Managed Server(s) . . . . . . 25 3.3.2.2 Use the Node Manager to stop the Managed Server(s) . . . . . . . . . . . . 25

4 Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.1 Monitoring Throughput . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.2 Monitor the Work backlog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.3 Task Processing Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.4 Technical Monitoring and Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 4.4.1 Accessing Mbeans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.4.2 OHI Claims Event Log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.4.3 Usage scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.4.3.1 Flood control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Operations Guide 3

4.4.3.2 Duplicate Notifications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 4.4.4 OHI Claims Management plugin for EMGC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.4.4.1 Prerequisites . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 4.4.4.2 Installation of the OHI Claims Management Plug-in . . . . . . . . . . . . . . 31 4.4.4.3 Configure metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4.4.4.4 Viewing metrics and alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.4.4.5 Installation in a clustered environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.4.5 Monitoring of OHI Claims using profiling tools . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

5 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.1 Logging and Auditing Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.1 Types of Log Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.1.1 Application Log. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.1.2 Security Log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.2 Logging Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.2.1 Configuration of an appender. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5.1.2.2 Security audit log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 5.1.2.3 Predefined logging configurations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.2.4 Performance logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.2.5 Using a custom log4j configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.1.2.6 Log4j Log Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.1.2.7 Associating a log level with an appender. . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.1.2.8 Making changes to logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

6 Error Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
6.1 Recover from Application Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.1.1 Tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 6.1.2 Recover errored tasks in the Claims Processing Flow . . . . . . . . . . . . . . . . . . . . . . 43 6.1.2.1 Errored Task Listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 6.1.3 Recover from external service failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.3.1 Failing to deliver a message for which processing of a claim should not be interrupted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.3.2 Failing to deliver a message for which the Claims Flow should be interrupted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.4 Recover from failures in batch processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.1.5 Recover messages from the Task Error Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 6.2 Recover from system failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

7 Copying an Environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
7.1 Clone the Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 7.2 Empty the Task Queue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 7.3 Enable Replication of Setup Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 7.4 Deploy the Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 7.5 Validate Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

8 OBIEE Reporting Views . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
8.1 Setup the OBIEE Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 8.2 Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 8.3 Customize the OBIEE Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Operations Guide 4

. . . . .9 Deeplinking to screens . . . . . . . . . 51 Operations Guide 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.1 Purpose This document is the Operations Guide for Oracle Insurance Claims for Health (OHI Claims).3 Guiding principle Guiding principle for the Operations Guide is to use default features of the technology stack as much as possible or unless alternative methods improve OHI Claims operations. The Operations Guide does not provide instructions for Oracle Database and Application Server administration.1 Introduction 1. 1. 1. The Operations Guide is not a comprehensive Oracle system administration manual nor does it replace existing documentation or available courses in management of Oracle environments. Examples of such default features are Automated Storage Management (ASM) and Automatic Optimizer Statistics Collection of the Oracle Database. Administration of the environment is the responsibility of the customer.2 Target audience The Operations Guide is intended for system and database managers and others responsible for looking after the daily operation of OHI Claims. OHI Claims runs on an Oracle platform that consists of an Oracle Database and Oracle Fusion Middleware Application Server. it only lists the most significant areas of interest for managing these in order to support OHI Claims. The root directory is the directory where the released files are placed. Operations Guide 6 . It can be any location or name of your choosing and will be referenced throughout this document as <OHI_ROOT>.4 References to artifacts in the OHI Claims distribution The distribution contains a number of directories that contain all the necessary information and sources to perform installation and upgrade actions.

1.1.g.html 2. Frequently back up your data • The Oracle database supports the creation of on-line backups • Refer to the manuals for setting up backup procedures Use • • RMAN (Recovery Manager) FRA (Flash Recovery Area) • 2.2 OHI specific guidelines 2. scale using RAC) Automatic Block Repair (2-way) Query SLA via STANDBY_MAX_DATA_DELAY 7 .2.1 Availability 2.2 Active Data Guard • Read-only access to physical standby database • • • Operations Guide Scalable Reader Farms (up to 30 active standby.2.1 Generic guidelines 2. consider using components of Oracle MAA..g • • • • • Real Application Clusters Active Data Guard (read-only access to physical standby database) Rolling upgrades (to new Oracle versions) Hot patching (using opatch.1.1.1 Cloning databases • Use RMAN • 11g: RMAN duplicate from active database • live backup/restore over network in new instance • 11gR2: Targetless duplicate possible from recovery catalog Other options (like using backup controlfile) may also be considered • 2. have you ever checked the air pressure of your spare tire?).2 Database 2. e.1 Backup/restore tips • • Run the production database in archive log mode.oracle.2 Oracle Maximum Availability Architecture Oracle HA technology is designed so that even redundant components can be actively used (e.1. also works for RAC) Scale for growth For more details see http://www.1.1. To increase availability of the Oracle system.com/technetwork/database/features/availability/maa-090890.1.

1.2.1.1 OHI applications are not yet certified to run on Oracle VM for production deployments Operations Guide 8 . not yet certified for OHI Claims) Exadata – boosts throughput Sun Oracle Database Machine and Exadata Storage Server • • “grid-in-a-box”.3 Hardware • • Clustering Virtualization 2. software.2. 2.2.1 Database tier • • Optional multiple database instances using RAC Optional partitioning • Installing large objects • ILM Information Lifecycle Management Optional save substantial disk space using compression • Oracle 11g Advanced Compression for tables • Index compression (no license required. servers.2 OHI specific guidelines • • • Use RAC Use Oracle VM to virtualize components of OHI Claims' technology stack Oracle has not certified any of its products on VMware virtualized environments • For more information see My Oracle Support Note 249212. storage.1 Generic guidelines 2.3 RAC • • Use of RAC for OHI applications is certified Oracle ASM (Automatic Storage Management) is required 2.2. networking Smart Flash Cache for hot data • • • 2.• • Ensure that force logging is activated to prevent an unintended NOLOGGING statement from not logging changes OHI Claims is certified for ADG.2 Application tier • • Multiple application servers Exalogic Elastic Cloud 2.1.2.2.2 Scalability 2.1.

constraints. to bypass the checksum verficiation (not recommended!) and launch the application regardless of a mismatch. Digest The digest is a file containing a full textual description of the schemas. at the start of these components. privileges. Startup When the application or the configuration import utility is started. described in "Migration Utility Setup".txt". If these are the same. open the file "properties/ohi-claims. the installation/upgrade completed successfully and the new checksum value is stored in the database. All those arguments except -f are the same as the migration utility. If the checksums do not match. the same configuration file as migration utility can be used (although only the logging and database settings will be used). If the schema differs. the command line options described in the table below can be added. indexes. the database schema version is verified. Additionally. Oracle can help to diagnose the problem by comparing the digest files. What to do if verification fails If at some point the schema verification fails and the application will not start.verifyschema" to false. such as the configuration migration utility and the OHI Claims application itself. If they do not match. The checksum of this generated digest is compared with the packaged checksum. The digest of the expected schema version is packaged with the application and can be found at "util/migration/ohi_digest. Each component assumes that is it connected to a database with the correct schema version. the start-up procedure will abort.3 Schema verification Introduction Several components of the application connect to a database. Generating a digest file can be done by opening a command line window at "util/migration" and running the following command: verify_schema. Checksum The checksum is a calculated MD5 hash code value of the aforementioned digest.2.properties" and set the property "ohi. etc. Processes The schema verification is used in the following processes: Install/upgrade The install utility will generate a digest of the database after it has finished installing/upgrading it. Working with a different schema can yield unpredictable behavior. it means the application is connected to a different database and the start-up process is aborted. Operations Guide 9 . The digest contains all the tables. These two checksums are compared during application start-up. Furthermore. columns. the start-up is aborted. Generate a new digest of the target database and send it to Oracle support.startup. the packaged checksum is compared with the checksum of the connected database. Alternatively. For that reason. This chapter describes the mechanism that is used to verify that a component is connected to the correct database schema and what to do if this is not the case. This value is embedded into both the application and the database. Mechanism The verification mechanism is based on two descriptions of the database schema: a digest and a checksum. it means the application is connected to a database with an unexpected schema.

FINE. If you altered the password for the ohi_claims_owner account.txt" if omitted The location of the configuration file. FINEST. CONFIG.4 Setting sequences Introduction Database sequences are used to generate a unique technical identifier for each data row. Settings Setting log. INFO.cfg. the default password for ohi_claims_owner is used db_url ohiPasswords Operations Guide 10 . Utility Configuration Navigate to <OHI_ROOT>/util/install. in order of ascending detail: OFF. If none is provided. "setSequences. SEVERE.cfg". If omitted. After cloning a database and changing its discriminator value.g. Possible values.level Description The level of detail at which log entries are written. The settings are explained in the table below. which can be used to specify a section in the environments{ } element of the config file.Option Short -f Long --file Argument Required Description digest file path No The file path and name of the output digest file that is generated. FINER. Uses "ohi_digest. Open this file and modify the settings for your environment(s).template" and rename it to e. WARNING.cfg" if omitted The name of the target environment. Uses "digest. it is necessary to reset the sequences to their correct configuration. ALL. The connection string of the target database. enter the new password here. Shows help information Shows the current version of the utility -c --cfg config file path No -e --env environment name No -h -v --help --version No No 2. the settings outside the environments{ } element are used. make a copy of the config file "setSequences. For this purpose the setSequences utility was created.

level = Level. the following users are created: "ohi_nonconfid_views_owner". The -e or --env command line argument (described below) can be used to switch between groups of settings.INFO db_url = 'jdbc:oracle:thin:@//<HOST>:<PORT>/<SID or SERVICE>' ohi_config.util.Level // Allowable values: SEVERE | WARNING | INFO | CONFIG | FINE | FINER | FINEST | ALL | OFF log.level = Level.template: import java.Environments The environments element can be used to define environment specific settings. an application user (ohi_<application>_user) and // an application views owner (ohi_<application>_views_owner) are created. the script will prompt for the password } another_environment{ log. hence it is not needed to create a separate config file for each environment.CONFIG // The url of the database connection.passwd = 'ohi_config' // If empty. // Example 1: db_url = "jdbc:oracle:thin:@//localhost:1521/orcl" // Example 2 (To Connect to RAC Database): db_url = """jdbc:oracle:thin:@(DESCRIPTION= // (LOAD_BALANCE=on) // (ADDRESS=(PROTOCOL=TCP)(HOST=clusnode-1vip)(PORT=1521)) // (ADDRESS=(PROTOCOL=TCP)(HOST=clusnode-2vip)(PORT=1521)) // (ADDRESS=(PROTOCOL=TCP)(HOST=clusnode-3vip)(PORT=1521)) // (ADDRESS=(PROTOCOL=TCP)(HOST=clusnode-4vip)(PORT=1521)) // (CONNECT_DATA=(SERVICE_NAME=ERP)))""" db_url = 'jdbc:oracle:thin:@//<HOST>:<PORT>/<SID or SERVICE>' // For each application a schema owner (ohi_<application>_owner). the script will prompt for the password ohiPasswords { // <database_user> = '<user_password>' // <database_user2> = '<user_password2>' } environments { your_environment{ log. // For every user that is created that is not listed here.INFO db_url = 'jdbc:oracle:thin:@//<HOST>:<PORT>/<SID or SERVICE>' ohi_config.cfg. if you want to use different passwords you can enter them here.passwd = 'ohi_config' } } Operations Guide 11 .logging. however. the password will be equal to its username // If the password is set to an empty string. Template The configuration file template shown below is located at <OHI_ROOT> /util/install/setSequences. // beside these users. "ohi_confid_views_owner" // The passwords for these users are defaulted to their username.level = Level.

If the application is still up.1 Generic guidelines 2.1. followed by the command line arguments described below. etc. the utility will not continue. 2. the settings outside the environments{ } element are used.5.5.sh (on Unix machines).1 How to address security issues? • • • • • • Hardening (making the database more solid) Assessments (regular checks) Monitoring (without visibility one cannot secure anything) Auditing (examination of the controls) Access control (authentication.5.2 Solutions • • Regularly Check Oracle Security Center1 for Critical Patch Updates2 (security vulnerabilities). To run the utility. Check Oracle Global Product Security Blog3 Operations Guide 12 . Command line arguments Option Short -c Long --cfg config file path No The location of the configuration file. Shows help information Argument Required Description -e --env environment name No -h --help No 2. Uses "setSequences.Running the utility The setSequences utility is available in the <OHI_ROOT>util/install directory. If omitted. The application should be shut down before running the utility.1.5 Security 2. open a command line at this location and execute setSequences.bat (on Windows machines) or setSequences.) as "a chain is only as strong as its weakest link". which can be used to specify a section in the environments{ } element of the config file. authorization) Encryption (mask data) Note to secure the whole technology stack (also listener.cfg" if omitted The name of the target environment. network.

1 What to audit? • • • Operations Guide Logon/logoff Object changes Statements (DDL/DML/selects) 13 . etc. monitoring. tablespace.4.1.sql • Uses verify_function_11g verification function as default Consider Security parameters (11g) • SEC_CASE_SENSITIVE_LOGON (default TRUE ) • SEC_PROTOCOL_ERROR_FURTHER_ACTION (default TRACE) • SEC_PROTOCOL_ERROR_TRACE_ACTION (default CONTINUE) • SEC_MAX_FAILED_LOGIN_ATTEMPTS (default 10) • SEC_RETURN_SERVER_RELEASE_BANNER (default FALSE) Secure default accounts Restrict privileges & roles Encryption • E.5. for Critical Patch Update Advisory feature. detection • Application using Data Vault (and other options/features) • Secured audit data • Not yet certified for OHI Claims Oracle Configuration Management Pack for OEM • E.g.3 Additional topics • Use password policy • • • 114g5 defaults6 $ORACLE_HOME/rdbms/admin/utlpwdmg.g.• • Use STIG (Database Security Technical Implementation Guide) guidlines & CIS checklists (Center for Information Security) Oracle Data Vault for separation of duties • Application using Oracle Label Security (= application using Virtual Private Database) • E. DA cannot access data • Not yet certified for OHI Claims Oracle Audit Vault for reporting.1.4 Auditing 2.5. • • • 2. Secure Configuration Scanning Oracle Provisioning and Patch Automation Pack • For guided patching Oracle Total Recall (Flashback Data Archive) • Enables AS OF queries • Long-term storage and auditing of data • Certified for OHI Claims Oracle Secure Backup Oracle Advanced Security Option • • Oracle*Next encryption RMAN encrypted backups • • • • • • 2.1.g. network.5.

17 GV$SYS_OPTIMIZER_ENV • • • • Keep optimizer parameters8 on default values 2.6.5.6.1 Parameters 2.2 Auditing requirements • • • • • • • Separation of duties Reporting Notification Integrity of audit data Map accountability & credentials Process Automation 2.6 Performance OHI Claims Development is leading in prescribing activities for performance.6. see MOS note 749851.1 Optimizer The optimizer will use following components to determine the SQL execution plans: Operations Guide 14 .2 OHI Specific guidelines • • Note password aging for generic OHI accounts • Set up maintenance procedure No need to have non-OHI CPU’s and PSU’s certified by OHI 2.• • • • Security changes Database links Replication Errors 2.6.4.1 OHI specific guidelines • Use Automatic Memory Management • • MEMORY_TARGET To prevent an imbalance in ratio’s also set following parameters to set minimum sizes: • SGA_TARGET • DB_CACHE_SIZE • SHARED_POOL_SIZE • PGA_AGGREGATE_TARGET Note there’s sufficient autotuning space Do not use AMM with Linux HugePages.1.2 Statistics 2. customer has certain obligations in this area 2.1.5.2.

5.1. 2.2. also need to be gathered manually for specific events) 2.6. 4.6. set global preference • • • 2.2.9 2. see USER_TAB_MODIFICATIONS GATHER_DATABASE_STATS/GATHER_SCHEMA_STATS will pick up tables modified >10% Use Statistics Preferences (11g) after consulting OHI • Note: when schema/database preference is set.3 I/O 2. if unwanted.1 OHI specific guidlines • • Operations Guide Advice: Use Segment Shrink Requires ROW MOVEMENT attribute on tables and indexes 15 . it only applies for the current objects (not for new objects).4 Statistics collection • Oracle recommends automated statistics collection • Default job dbms_stats.2 Statistics types 1.3.2.6. 4. Statistics Parameters Space & storage management CPU & I/O characteristics Schema definitions 2. 2. 2.2. table monitoring facility • • • Tracks INSERT/UPDATE/DELETE.6. Procedures and Diagnoses Relation or Provider data Use production statistics on DTA environments (see Oracle Database Performance Tuning Guide).6.5 Miscellaneous Statistics have to be gathered manually after bulk loads of 1.3 Database parameters Use the Automatic tuning capabilities of the Oracle Database and the Advisor Framework. gathered automatically during Database Maintenance Window. 2. 3.gather_database_stats_job_proc in Maintenance Window • It is the responsibility of the database administrator to define the maintenance window Use dbms_stats ESTIMATE_PERCENT=AUTO_SAMPLE_SIZE (default) Column statistics and histograms • dbms_stats METHOD_OPT = FOR ALL COLUMNS SIZE AUTO Stale statistics. Products. System (gathered automatically at instance startup.6. 3. needs to be gathered during typical workload) Dictionary (gathered automatically during Database Maintenance Window) Fixed objects (should be gathered manually during a typical workload) Objects (table/index.

top.g.• • Requires Automatic Segment Space Management (= standard for OHI Claims) Use OEM Segment Shrink Advisor10 2.4. then to DSFC Oracle Unbreakable Linux Kernel • • Linux kernel tuned for Oracle OEL5 update 6 = standard UEK • Standard OEL5 = 2. SQL Performance Analyzer (11g). UEK = 2.6.1.18. topas. OS Watcher users guide Use autotuning capabilities: ADDM/AWR/AWR SQL/ASH • Will be used by OHI Development in case of performance issues • Requires STATISTICS_LEVEL=TYPICAL Advice: Use OEM Tuning & Diagnostics Packs • • • Require additional licenses Oracle Diagnostics Pack for Database • E.2 Generic guidelines • • • • Spread I/O's over multiple controllers I/O time should not exceed 5-10ms (for a single block read) Calibrate I/O resources (when using Automatic Parallel Execution or Resource Manager • Via OEM or dbms_resource_manager. prstat.6.6. iostat.32 2. sar.6.4 Monitoring 2. Performance Monitoring. SQL Tuning Advisor. using a solid state disk or I/O card with flash memory Stores unmodified database blocks evicted from SGA buffer cache • If it's unmodified it is written to disk first. events.6. SQL Plan Management Auto Evolve (11g).6. ASH & ADDM Oracle Tuning Pack for Database • E. AWR.g. swap) Oracle OS Watcher • Oracle Metalink Note 301137. Reorganize objects • 2. Automatic SQL Tuning (11g).2 OHI specific guidelines • • Note non-OHI software Check alertlog for ORA-006XX/07445 • Operations Guide Bugs in optimizing SQL often lead to these errors 16 .g.1 Generic guidelines • • Use Oracle Enterprise Manager Grid Control Prepare: collect performance data • • • • AWR (Automatic Workload Repository) Baselines OS utilities (vmstat.3.calibrate_io Oracle Database Smart Flash Cache • • • • Available on Solaris and Oracle Enterprise Linux Expand database buffer cache • E.4. SQL Access Advisor.

2.CTX_DDL.) At the end of the rebuild operation. and the space can be reused. then additional space is required in the redo log as well. If this operation fails. ctx_ddl.optimize_index('REL_RELATION_IDX2'. Therefore. Oracle recommends that the Text indexes are optimized on a weekly basis and that rebuild mode is used to do that. Rebuilding an index can be done on-line by executing the following command as user OHI_CLAIMS_OWNER: ALTER INDEX <errored_index_name> REBUILD ONLINE. 2. the original $I table is dropped.optimize_index('REL_RELATION_IDX3'. Rebuilding an index is often significantly faster than performing a full optimization. end.OPTLEVEL_REBUILD). Use the following command to optimize the Text indexes: begin ctx_ddl.CTX_DDL. Index synchronization errors can be detected using the CTX_USER_INDEX_ERRORS view.1.7. the data transaction still commits. when the data is committed but index changes are not.OPTLEVEL_REBUILD). usually small. 2. and is more likely to result in smaller indexes. and then switches the original $I table and the copy. Operations Guide 17 . (If redo logging is enabled. ONLINE.CTX_DDL.optimize_index('REL_ADDRESS_IDX1'. The commit does not return until the sync is complete. ctx_ddl. ONLINE.ERR_INDEX_NAME) needs to be rebuild. especially if the index is heavily fragmented. Note that synchronization is performed as a separate transaction.2 Index Optimization Frequent index synchronization (for each commit) leads to fragmented text indexes. Oracle Text indexes must be optimized frequently in order to reduce fragmentation and index size and to ensure optimal query performance. As a result there may be a period. ctx_ddl.7. This chapter specifies operations for maintenance of the Oracle Text indexes. ONLINE.OPTLEVEL_REBUILD).OPTLEVEL_REBUILD). the index in which the error occurred (column CTX_USER_INDEX_ERRORS.CTX_DDL. ONLINE.1 Index Synchronization The Text indexes are set up to synchronize the index immediately after a commit. For rebuilding all the Text indexes execute the following: ALTER ALTER ALTER ALTER ALTER INDEX INDEX INDEX INDEX INDEX REL_RELATION_IDX1 REL_RELATION_IDX2 REL_RELATION_IDX3 REL_ADDRESS_IDX1 REL_ADDRESS_IDX2 REBUILD REBUILD REBUILD REBUILD REBUILD ONLINE. Note that the rebuild operation requires enough space to store the copy as well as the original.optimize_index('REL_ADDRESS_IDX2'.1 Dealing with Synchronization errors If synchronization errors are detected.• • Always report to Oracle Support Services Report back to OHI Development 2.7 Oracle Text index maintenance Oracle Text indexes are set up for OHI Relations and Addresses. Index fragmentation adversely affects the response times of queries in which text indexes are used. Rebuild optimization creates a more compact copy of the $I table.optimize_index('REL_RELATION_IDX1'.7.OPTLEVEL_REBUILD). The sync operation has its own transaction context.CTX_DDL. ctx_ddl.

9. Advice.Oracle Text comes with a CTX_REPORT package that can be used to gather Text index data.1.1.2 Oracle 11g Database Features on Fault Management • • • • • • ADR (Automatic Diagnostic Repository) Incident Packaging Service SQL Test Case Builder SQL Repair Advisor Data Recovery Advisor (via OEM and RMAN. to test and development environments Consider • • • • Removing excess data • Which may not be removed in production Then compress the rest Data Masking/scrambling Placing database files for non-production environments on cheaper disks 2. 2. Repair Proactive checks • • • Operations Guide Health Monitor or RMAN validate database HM: Framework for running diagnostic checks for several database components Parameters to detect corruption 18 .8 Storage 2.9 Fault management 2.1 Generic guidelines • • Use ASM Consider Information Lifecycle Management • • • • • Classify your data and move to appropriate storage tiers • Or make read-only or offline Choose a strategy for reorganizing your database Use low cost hardware Consider compressing tables and indexes When copying a database from production • • E.1 Generic guidelines 2.8.1 Reduce time to resolution • • • • Prevention by Automatic health checks Diagnostic by Automatic Diagnostic Workflow Solution by Intelligent resolution Delivery by Proactive patching 2.9. See the documentation of the Oracle Database for details.9.g. uses Health Monitor) • List.

1 OHI specific guidelines 2.g. When running the view generator you might get this message: Operations Guide 19 . follow up on advice from Segment Shrink Advisor) are allowed • 2.1.10. The views should be regenerated after: • • • an upgrade of the application. resolving details such as dynamic field definitions.10 Do's and dont's 2.g. make sure that these are not prefixed with ‘OHI’ Consider using Database Replay to re-create production database workload in a test environment Use Oracle Resource Manager (when 100% CPU usage) • • Prioritize sessions (e.11 Generate Reporting Views The OHI application has a dedicated database schema for reporting views. • Add tables or views or modify existing tables • Add or change indexes (also invisible indexes …) DBA activities without changing the definition (e. DB_BLOCK_CHECKSUM. languages etc.g. using the Configuration Migration utility. e.1.2 Do not • Make changes to the OHI Claims schema’s and objects.1 Do • Oracle recommends that the Oracle database instance that is used by OHI Claims is used sole ly for the purpose of running the OHI Claims system • However: the license for the Oracle database does not restrict it to be used for the purpose of running OHI Claims only • In the case that additional database schemas are created in the Oracle database instance.10. Reporting views are a logical presentation on the tables. database & ASM) 2.10. changing dynamic field or dynamic record definitions. DBA’s/OHI Claims OLTP users/OHI Claims Reportings users) Automated Maintenance Tasks feature relies on the Resource Manager being enabled during the maintenance windows • • • • Resource Manager Plan ensures that Automated Maintenance Tasks consume no more that 25% of CPU capacity Consider using Instance Caging (11g) 2. DB_LOST_WRITE_PROTECT Support Workbench (OEM.• DB_ULTRA_SAFE => changes defaults for • • DB_BLOCK_CHECKING.

2.com/security/ http://download.oracle.1&type=NOT optimizerenv. but still requires a grant to be available.jar -h usage: java -jar ohiViewsGenerator. unless you use the -n flag. 4.com/docs/cd/E11882_01/server. no functional views will be generated. you must change the name of this field.htm 10.cod_flex_code_field_usages_b where upper(code) in (select keyword from v$reserved_words where reserved='Y').html http://blogs.--help Show usage information and quit -n Do not execute the SQL on the database -o <outputFile> The file the generator writes SQL statements to -s Do not regenerate base views The config file is mandatory.oracle.com/CSP/main/article?cmd=show&id=749851.112/e10897/users_secure. http://download.com/technetwork/topics/security/alerts-086861. If the -d argument is omitted.112/e10897/users_secure.oracle. select code from ohi_claims_owner. 5.template.com/docs/cd/E11882_01/server.112/e10897/users_secure.oracle. 1. Queries to detect problematic names are: select distinct code from ohi_claims_owner.jar <options> -c <configFile> Config file -d <definitionsDir> Directory containing the XML definition file(s) -h. you have the option to write the SQL-statements to a file by using the -o argument.htm Operations Guide 20 .lst http://download.oracle. Note that this will still execute the SQL. To resolve this. If you want to run the generator once and apply the result to multiple databases.oracle. you can provide the directory containing the functional view XML definition files.htm http://download.cfg.112/e10897/storage.cod_flex_code_systems_tl where upper(code) in (select keyword from v$reserved_words where reserved='Y').com/docs/cd/E11882_01/server.com/docs/cd/E11882_01/server.htm https://support.htm http://download. if the argument is used. 7.oracle. 3. The way to run it is: $ java -jar ohiViewsGenerator.oracle.com/technetwork/topics/security/index.com/docs/cd/E11882_01/server. v$reserved_words is a public synonym. http://www. 8.html http://www. 9. This file can be created from the supplied ohi_create_views. 6.112/e16638/stats.oracle. Running the generator The generator is delivered as a so-called runnable jar-file.ORA-00923: FROM keyword not found where expected This usually means that you have defined a dynamic field with a reserved Oracle keyword as the name of the field.

out $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>. 3. Start the Admininistration Server: nmStart('<admin-server-name>') 4. <nm-host>.sh 2.1. '<nm-pwd>'. Perform the following steps: 1. '<nm-host>'. 4. Disconnect from the Node Manager: nmDisconnect() 5.1. <nm-port> and <domain-name> with correct values.2 Use the start script to start the Administration Server 1. WLST scripts that handle starting servers are available in the OHI Claims release bundle <OHI_ROOT>\util\wlst. 2.1 Use the Node Manager to start the Administration Server This is the preferred approach.3 Fusion Middleware 3. Start the Administration Server $ nohup ~/user_projects/domains/<domain-name>_domain/startWebLogic. Optionally tail the server out and/or log file: $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>.1.sh > /dev/null 2>&1 & Replace <domain-name> with a correct value.1.'<nm-port>'. Start the WLST environment: ~/wlserver_10. Optionally tail the server log file: Operations Guide 21 . 3.3/common/bin/wlst. The use of a clustered environment is assumed.log Replace <domain-name> and <admin-server-name> with correct values. Connect to the Node Manager: nmConnect('<nm-uid>'. '<domain-name> ') Replace <nm-uid>. Exit WLST: exit() 3.1. <nm-pwd>.1 Start the Administration Server 3.1 Starting the WebLogic Cluster This chapter describes starting and stopping of the WebLogic Server.

<nm-port> and <domain-name> with correct values.'<nm-port>'. '<nm-pwd>'. select the server instance and select the “Start” option. 3.1. 4. '<nm-host>'. 3.$ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>.log Replace <domain-name> and <admin-server-name> with correct values. Navigate to the “Domain / Control” page.out $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>.2.2 Start the Managed Server(s) 3. Connect to the Node Manager: nmConnect('<nm-uid>'.2 Use the Node Manager to start the Managed Server(s) 1.2.sh 2. Optionally tail the server out and/or log file: $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>. 3. <nm-pwd>.3/common/bin/wlst. Access the web-based Administration Console and login using the Administrator credentials.log Replace <domain-name> and <managed-server-name> with a correct value. <nm-host>. Start the Managed Server: nmStart('<managed-server-name>') Replace <managed-server-name> with a correct value. 5. 2.out $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>.1. Disconnect from the Node Manager nmDisconnect() 6. Start the WLST environment: ~/wlserver_10.1.log Replace <domain-name> and <managed-server-name> with a correct value. 3. Exit WLST exit() Operations Guide 22 . '<domain-name> ') Replace <nm-uid>. Optionally tail the server out and/or log file: $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>.1 Use the Administration Console to start the Managed Server(s) 1.

OHI Claims performs the following actions: • • • Recover in-flight tasks that were not finished when the system was stopped.3.taty_id taty. Note that the system will not restart failed tasks. the following query should not return rows: select .3 Stopping the WebLogic Cluster 3.id 'en' 'B' -.name task.process_start_datetime ohi_tasks task ohi_task_types_tl taty task.2. WLST scripts that handle starting servers are available in the OHI Claims release bundle <OHI_ROOT>\util\wlst.3. . Perform the following steps: 1. 3.2 Verify the results This paragraph describes how the results of the startup actions can be verified. the following query should not return rows: select * from aq$ohi_task_queue_table where expiration_reason != null. • Verify that messages that ended up in the ‘error queue’ in the database are resubmitted for processing.1 Actions performed when the system starts up As part of the start up routine. Populate the 'Procedures / Dynamic Logic' cache (page 0): pre-calculate the evaluation of the Dynamic Logic conditions that link Procedures to Benefit Specifications and populate table CLA$PROC_DYN_LOGIC with the results. where and and and task. • Verify that tasks that were in a processing state when the system was shut down are restarted.status task.2 The System's start up routine 3.2.1.id taty.1 Stop the Administration Server 3. • Verify that dynamic logic evaluation for the 'Procedures / Dynamic Logic' cache executed correctly: were there notifications? Check for errors in the log files. from .3.3/common/bin/wlst.Processing to_date(<time_system_stopped>).sh Operations Guide 23 . 3.process_start_datetime = = = < taty.1 Use the Node Manager to stop the Administration Server This is the preferred approach.language task. Start the WLST environment: ~/wlserver_10. Recover messages from the ‘error queue’ in the database.

3 Use the stop script to stop the Administration Server 1. select the server instance and select the “Force Shutdown Now” option. 3.1. Optionally tail the server out and/or log file: $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>.1. 5.sh Replace <domain-name> with a correct value. Operations Guide 24 .3.2. Disconnect from the Node Manager nmDisconnect() 6.out $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>.'<nm-port>'.3.log Replace <domain-name> and <admin-server-name> with correct values. <nm-port> and <domain-name> with correct values.log Replace <domain-name> and <admin-server-name> with correct values. '<nm-host>'. 4. Exit WLST exit() 3. 3.2 Use the Administration Console to stop the Administration Server 1.log Replace <domain-name> and <admin-server-name> with correct values. Access the web-based Administration Console and login using the Administrator credentials. <nm-host>. Optionally tail the server out and/or log file: $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>. Optionally tail the server log file $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>. '<nm-pwd>'. <nm-pwd>. 3. Stop the Administration Server: nmKill('<admin-server-name>') Replace <admin-server-name> with a correct value. Connect to the Node Manager: nmConnect('<nm-uid>'. Stop the Administration Server $ ~/user_projects/domains/<domain-name>_domain/bin/stopWebLogic. '<domain-name> ') Replace <nm-uid>. Navigate to the “Domain / Control” page. 2.out $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< admin-server-name>/logs/<admin-server-name>. 2.

'<domain-name> ') Replace <nm-uid>.2.log Replace <domain-name> and <managed-server-name> with a correct value. Access the web-based Administration Console and login using the Administrator credentials. <nm-pwd>.log Replace <domain-name> and <managed-server-name> with a correct value.out $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>. select the server instance and select the “Force Shutdown Now” option.3. Optionally tail the server out and/or log file: $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>.2 Use the Node Manager to stop the Managed Server(s) 1. '<nm-pwd>'.3. Navigate to the “Domain / Control” page. 1.'<nm-port>'. 4.2 Stop the Managed Server(s) 3. 5. Stop the Managed Server: nmKill('<managed-server-name>') Replace <managed-server-name> with a correct value. <nm-port> and <domain-name> with correct values.3.3. Optionally tail the server out and/or log file: $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>.3/common/bin/wlst. 3. '<nm-host>'. Disconnect from the Node Manager nmDisconnect() 6.sh 2. Connect to the Node Manager: nmConnect('<nm-uid>'.1 Use the Administration Console to stop the Managed Server(s) This is the preferred approach. 3. 3. 2. <nm-host>. Start the WLST environment: ~/wlserver_10.2. Exit WLST exit() Operations Guide 25 .out $ tail -f ~/user_projects/domains/<domain-name>_domain/servers/< managed-server-name>/logs/<managed-server-name>.

Database views that be used for creating reports.4 Monitoring This chapter covers monitoring of the OHI Claims application. For measurement of such indicators. The remainder of this chapter deals with monitoring the performance or throughput of OHI Claims. Note that this definition depends largely on: • • The setup of the system. The following query determines the number of claims that are Finalized on a given day. Monitoring should cover the entire technology stack that is used to execute the OHI Claims application: • • • • • Hardware Operating system Oracle Database Oracle Fusion Middleware and OHI Claims Oracle recommends to monitor the infrastructure using Oracle Enterprise Manager Grid Control (EMGC). Technical monitoring and management of OHI Claims is documented in a separate chapter.1 Monitoring Throughput For OHI Claims we define throughput as: the number of claims (or claim lines) that can be automatically processed by the claims process from ‘Canonical Claims In‘ up to the Finalized state in a given time period. Average time for resolving pend states of a claim (like manual pricing or manual adjudication). Properly sizing the system should ensure that processing capacity is sufficient for handling normal workloads and occasional exceptionally large workloads. OHI Claims provides the following mechanisms: • • Event based integration points that can be used to inform external (reporting) systems about changes in claim states. It can be executed as OHI_CLAIMS_USER: Operations Guide 26 . The processing capacity of the system. Examples of these are: • • Average duration for processing claims. The monitoring approach outlined here focuses on the OHI Claims application. Operational monitoring of the OHI Claims application depends on identified key performance indicators. 4.

process_stop_datetime-task.process_stop_datetime-task. Use the following query to determine the number of remaining tasks at any given time.1) and trunc(sysdate) Of course the timeframe for the measurement could vary. if there is no backlog of work in the queried timeframe. to measure frequently and make meaningful assertions regarding the trend of the measurements.2 Monitor the Work backlog The work backlog in OHI Claims is mainly determined by the number of tasks that the system still has to perform.name.process_start_datetime) DAY TO SECOND)). i.select from where and count(*) cla_claim_status_history status = 'FINALIZED' date_time between trunc(sysdate .process_stop_datetime-task.e.process_stop_datetime-task. It can be executed as OHI_CLAIMS_OWNER: select count(*) from aq$ohi_task_queue_table where msg_state != 'PROCESSED'. 4. For example.language = 'en' and task.process_start_datetime) DAY TO SECOND)*3600 + EXTRACT(MINUTE FROM(task. Do not access the queue table directly. The following query lists the average processing times per task select taty. ohi_task_types_tl taty where task. Note that a view is used to query the queue.1) and trunc(sysdate) group by taty.id and taty.name tasktype . the figure by itself is rather meaningless. That number is simply the amount of tasks in the task queue.0) avg_process_elapsed_time_ss from ohi_tasks task . round(avg(EXTRACT(DAY FROM(task. Operations Guide 27 .taty_id = taty. it may appear as if the system’s throughput is low (whereas there was simply no work to perform). always use the read-only view! A growing work backlog is either: • • A good indicator to determine if processing capacity must be added or A signal that the system is not performing as expected It is acceptable for the work backlog to increase in busy periods. Note however that by itself this is a tricky metric to monitor.process_start_datetime) DAY TO SECOND)*86400 + EXTRACT(HOUR FROM(task.process_start_datetime) DAY TO SECOND)*60 + EXTRACT(SECOND FROM(task.process_stop_datetime is not null and task. Hence. A better approach is to keep track of the mean time that is required to perform certain tasks.creation_date between trunc(sysdate . it should always be judged in the proper context. A properly sized system will 'catch up' as soon as the claim volume returns to normal proportions.

This ensures that claims for which processing started will be finished before processing for newly entered claims starts ('first finish what you started'). Once processing for a claim starts. If there is reason for concern.Advice: monitor the work backlog frequently and follow the trend. If the peak level is reached. OHI Claims is equipped with a technical management interface. is the mean time that is required for processing specific tasks increasing.g. The priority for some low-frequency tasks is set to a slightly increased value in order to have acceptable wait time before these are processed. the priority for subsequent tasks related to the same claim increases. Define a threshold for a specific peak level. Any other tasks have a priority of 0. Task Claims Import Claims Processing Initialization Claims Processing Sanity Checks Claims Processing Enrollment Check Claims Processing Prepricing Claims Processing Pricing Claims Processing Payment Status Check Claims Processing Benefits Claims Processing Adjudication Claims Processing Finalization File Import Financial Message Generation Counters Batch requests Priority 0 1 2 3 4 5 6 7 8 9 1 1 1 Priorities cannot be influenced. see previous paragraph). • • The following table provides an overview of the (relative) priorities for different tasks. 4. Tasks are picked up if and when processing capacity is available. It supports the operations like: Operations Guide 28 . typically a level at which meeting service levels is at risk. 4. be prepared to add additional processing capacity by adding servers to the cluster. use other metrics for investigating the cause (e. A higher priority means that the task is executed before any tasks that have a lower priority. Examples of such tasks are Financial Message Batches and File Import Tasks.4 Technical Monitoring and Management This chapter deals with technical monitoring and management of the OHI Claims application.3 Task Processing Principles The following principles apply for processing tasks from the work queue: • • Tasks are processed in order of priority first and time that the tasks were entered into the queue second. For this purpose.

initial=weblogic.management.ssl to true if the environment is configured to use SSL certificates. Automate technical operations. for example in case of failures in evaluating Dynamic Logic scripts. the following Java Options should be added to the startWebLogic.ssl=false" JAVA_OPTIONS="${JAVA_OPTIONS} -Djavax. The management interface provides insight into the technical state of OHI Claims. 4.management.e.1 Accessing Mbeans For accessing OHI Claims MBeans. by using the remote URL <machine_name>:< free_port_number>.sh script as that will not work due to port conflicts in that case. MBeans are accessible through management tools like Oracle Enterprise Manager Grid Control (recommended) or the WebLogic Console. After starting the WebLogic server with these configuration changes. for example using JConsole.2 OHI Claims Event Log OHI Claims logs events that require manual intervention in its Event Log.authenticate=false" JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom. Technical diagnostics.jmxremote. it is not meant to monitor the operations of the "Claims Shop".sun. using the WebLogic Diagnostic Framework (WLDF). metrics and notifications can be defined to periodically check the availability of resources and the occurrence of errors. it is implemented using the Java Management Extensions (JMX Managed Beans or MBeans) and not suitable for use by business users.• • • Event Noticiations.port=< free_port_number>" JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.4.jmx.management.management. Query status and gathering of statistics like the average run time of Dynamic Logic scripts.WL SMBeanServerBuilder" Note: do not put this in the setDomainEnv. Security recommendation: set com.sun.jmxremote.mbeanserver.4.builder.sun.sh script: JAVA_OPTIONS="${JAVA_OPTIONS} -Dcom.sun.sh if it is necessary to also use the stopWebLogic.management.jmxremote. it is possible to access the OHI Claims MBeans. for example restarting errored tasks. 4.jmxremote. The following events are logged: • • • Runtime failures in Dynamic Logic scripts External Web Services not being available Failures to complete a processing task The events are persistently stored in the system and can be viewed by using the System Event Logs page as is shown in the following graphic: Operations Guide 29 . This shows in the management interface.management. i.

3 Usage scenario The high-level operation of the eventing mechanism is as follows: • • • OHI Claims logs events in case of certain runtime failures.4. That will trigger the notification mechanism.3.3.1 Flood control The metrics that are defined in the OHI Claims EMGC management plugin create an alert as soon as one (and only one) of the predefined events occurs. It is up to the customer to select and configure the notification mechanism. 4. Upon receiving a notification. the event should be marked as Resolved (check box) in the System Events UI page. flagged as Resolved in the System Events UI page). the problem(s) should be triaged. Operations Guide 30 . If the problem is fixed.The event log is designed for interoperability with the OHI Claims EMGC management plugin. Once the alert is raised and as long as the problem is not resolved (i. more than one alert may be raised (triggering multiple notifications) as a result of the same problem. After resolving all the problems of that type the alert is cleared. The OHI Claims EMGC Management Plugin comes with predefined metrics for detecting and handling these events. If the alerts for these events were not yet raised.g. For example: processing a claim fails in the Benefits task as a result of a runtime error in evaluating Dynamic Logic.e. • • • 4.4. Only after that new notifications will be send for new events that occur. no new or additional notifications will be send for similar events. It creates alerts in case of failures and sends out a notification using the EMGC's notification mechanism.4. e-mail or SNMP. For failed tasks the System Events UI page will have a link to the Technical Error page to restart the task immediately. both for the runtime failure in the dynamic logic as well as for the failing task a notification will be send. Doing that will automatically flag the event as Resolved. e.2 Duplicate Notifications In some cases. 4. The System Events UI page (shown above) provides more detailed information that helps to determine the cause of the error. This will clear the alert in EMGC.

Set up notifications in EMGC. Step 2: Click Management Plug-ins from the left navigation bar and click Import. The EMGC product documentation describes how that is done.4. Step 3: From the Import page.jar).4.4.1 Prerequisites EMGC 11gR1 (or better) has to be installed.2 Installation of the OHI Claims Management Plug-in OHI Claims Management Plug-in is bundled in the release bundle (<RELEASE_ZIP> /monitoring/OHIClaimsManagement.4. Agents have to be installed on systems that need to be monitored.4 OHI Claims Management plugin for EMGC 4.jar) and click List Archive.4. click Setup. 4.4. Perform the following steps in EMGC console: Step 1: From the Enterprise Manager console. Operations Guide 31 . specify the Management Plug-in Archive file ( OHIClaimsManagement. The first step is to import this Management Plug-in into EMGC.

as shown in the following figure. Choose the desired Management Agent. Enterprise Manager returns you to the Management Plug-ins page with OHIClaimsManagement plug-in added to the list. From the Management Plug-in main page. Step 5: After the Management Plug-in has been imported into Enterprise Manager. Click Select. Use the appropriate search parameters to find the desired Agent.Step 4: Select "OHIClaimsManagement" from the list and click OK. The Search and Select: Agents page appears. Enterprise Manager returns you to the Deploy Management Plug-in: Select Targets page. The Deploy Management Plug-in: Select Targets page appears. The selected Agent(s) appears in the deployment list. Operations Guide 32 . Step 6: Click Add Agents. click Deploy icon for the Management Plug-in you want to deploy. you can deploy the plug-in to any number of Management Agents.

4. Step 8: Click Finish. click Agents link.4.Step 7: Click Next to go to the Review page. Operations Guide 33 .3 Configure metrics Now that the Management Plug-in has been deployed to the appropriate Agent(s). you are ready to configure metrics. Step 1: From the Management Plug-in main page. 4.

Step 2: Click on the agent on which you deployed the Management Plug-in. For example. OHIClaimsManagementMetrics Operations Guide 34 . choose OHIClaimsManagement from the Add drop-down menu and click Go. Step 3: From the Monitored Targets section of the Management Agent home page. Step 4: Enter the requisite target properties referring the table below. S.No 1 Field Name Name Field Value Any desired name.

4 Viewing metrics and alerts Step 1: From the Agent home page.management. The All Metrics page appears for the monitored target. 4.mbeanserv ers.2 Machine name Machine name or IP address on which WebLogic Server is running WebLogic Server Admin port weblogic (WebLogic Server Admin Username) WebLogic Server Admin Password t3 weblogic. Step 2: In the Related Links section. as shown below. Operations Guide 35 . click on the target instance you added in the previous step in the Monitored Targets list. An expandable tree list for each metric enables you to drill down to view specific metric parameters. Enterprise Manager takes you to that target's home page.4.runtime WebLogic Leave if blank 3 4 5 6 7 8 9 Admin Port number User Name JVM Admin User Password Communication Protocol Service Name Metric Source Custom lookup provider Class Step 5: Click on OK button. click All Metrics.4.

4. perform the following steps: Step 1: In the Related Links section.4.1 Customizing Metric and Policy Settings By default. click Metric and Policy Settings. it will create an alert.Step 3: Whenever the metric value is greater than the configured critical threshold value (which is 1 by default). 4. Operations Guide 36 .4. This means. The following is a sample critical alert E-Mail Notification. the configured threshold value is 1 and the metric collection interval is 300 seconds. every 300 seconds EM Management Agent polls for metrics and if the metric value is greater than or equal to 1. If you would like to change the collection interval or threshold value. a critical alert is raised (which in-turn can trigger an E-Mail notification if you have configured).

Operations Guide 37 . 4. Step 6: Click OK.5 Installation in a clustered environment Install the Management Plug-in on one of the nodes in a cluster in order to avoid receiving duplicate notifications. Step 5: Edit Repeat Every value to suit your needs and click on Continue button. Step 3: Edit the threshold values to suit your needs and click on Continue button.4.Step 2: Click on Edit icon for the metric for which you want to customize the threshold setting. Step 4: Click on Every 300 Seconds link if you want to customize Collection Schedule.4.

using profiling tools like Yourkit. specifically the JVM . but can be invasive and impact system performance and stability under certain conditions.4. Operations Guide 38 .5 Monitoring of OHI Claims using profiling tools A word of caution on monitoring Oracle Fusion Middleware. These tools can be very valuable.4.

1 Types of Log Files The following types of log files are distinguished: • • Application log Security log 5.1 Logging and Auditing Operations OHI Claims supports several levels of logging and auditing that can be used to gather and assess information about the runtime state of the system. nder Log file rolls over when the size MaxFileSize is reached.1.log4j.log4j.1 Configuration of an appender The following snippet of XML shows an example for the configuration of a log file appender for writing log statements to a file: <appender name="fileAppender" class="org.log4j. MaxFileSize Operations Guide 39 .1.1.2.2 Logging Configuration 5.1.PatternLayout"> <param name="ConversionPattern" value="%d{ISO8601} [ %t ] %p %c %m%n" /> </layout> <filter class="com.apache.RollingFileAppender"> <param name="File" value="target/ohiApplication.1 Application Log What's in the application log How to control application log 5.healthinsurance. 5.apache.1.RollingFileAppe Creates log files with rollover.1.1. 100MB Maximum file size for a log file before it is rolled over.log" /> <param name="Append" value="false" /> <param name="MaxBackupIndex" value="5" /> <param name="MaxFileSize" value="100MB" /> <layout class="org.log4j.apache.2 Security Log Definition of Auditing Log What's in the auditing log how to control the auditing log 5.5 Logging 5.SecurityLevelFilter"> <param name="inclusive" value="false" /> </filter> </appender> The following table explains the configuration.oracle. Parameter Appender class Sample Value Explanation org.logging.

log Number of log files that are retained. 5.ProvisioningServiceImpl . See the ConversionPattern Java documentation1 for more details.healthinsurance.RollingFileAppender"> <param name="File" value="target/ohiSecurity. Path and name of the log files.Default (self-tuning)' ] SECURITY com.oracle.log4j.MaxBackupIndex File 5 target/ohiApplication.2.X where X is the following number in sequence.provisioning.log4j.077 [ [ACTIVE] ExecuteThread: '1' for queue: 'weblogic.service. the current file is moved to a file target/ohiApplication.healthinsurance.User 322 with loginName FN0023P0002 was modified.kernel.layout. When roll over occurs.1.SecurityLevelFilter and inclusive For more information on log4j configuration see the Apache Log4j website2.Default (self-tuning)' ] SECURITY com.ProvisioningServiceImpl .User 322 with loginName FN0023P0002 was created.appender.oracle.oracle. To ensure that security messages are only written to the security log. Security related log data can be captured in an appender by specifying filter class "com. The specific configuration of the securityAppender is defined below.PatternLayout"> <param name="ConversionPattern" value="%d{ISO8601} [ %t ] %p %c %m%n" /> </layout> <filter Operations Guide 40 .provisioning.Con versionPattern %d{ISO8601} [ %t ] %p %c %m%n com.impl.apache.healthinsurance.log4j. Examples include the following: • 2010-06-11 21:01:24.service.kernel.logfile.2 Security audit log OHI Claims issues specific.oracle. • For auditing purposes. security related log statements. Inclusion of this filter class and parameter ensures that security message are not written by the corresponding appender. e. Another useful source is the Wiki that describes the Log4jXmlFormat3.logging.apache.impl.healthinsurance.log" /> <param name="Append" value="false" /> <param name="MaxBackupIndex" value="5" /> <param name="MaxFileSize" value="100MB" /> <layout class="org.398 [ [ACTIVE] ExecuteThread: '1' for queue: 'weblogic. this filter class and parameter should be defined on all appenders. the setup for the security audit log is likely to be different.log.apache.PatternLayout and log4j.log4j. Controls the layout and output format according to conversion patterns similar to the C language printf function.loggin false g.g. security audit data is retained for a longer period of time.SecurityLevelFilter" as is shown in the following example: <appender name="securityAppender" class="org. org. 2010-06-11 21:01:24.log4j.

ThreadlocalResolver .logging. For example: Operations Guide 41 .oracle.3 Predefined logging configurations OHI Claims comes bundled with a number of predefined log4j configurations: 1.healthinsurance.2.time(ms)=2152 Note: the entire message is logged on one line but was formatted differently in this guide for readability.xml trace-log4j.[WS] Request: contextPath=/ohi-web-services.sla.log4j. requestURL=http://localhost:7001/ohi-web-services/ProviderImportService. use the -Dohi.log4j. the elapsed time needed to process the request is logged like this: DEBUG com.t ime(ms)=58 Note: the entire message is logged on one line but was formatted differently in this guide for readability. 4. log4j.oracle.config.oracle.2.[UI] Request: userId=42.type=production | development | trace Note that this is overruled if the flag -Dohi.filter"> <level value="debug"/> </logger> For each UI request.file is specified (see next section). 5.contextPath=/base.type Java option in the setDomainenv script: -Dohi.sla.config. 2.1.2. requestURL=http://localhost:7001/base/faces/SearchClaims.5 Using a custom log4j configuration Steps for using a customized log4j configuration: 1.log4j. log4j.healthinsurance.SlaLogger .servletfilter"> <level value="debug"/> </logger> For each WS request. 5.log4j.log4j.page=View Claim CL0025_reprocess_2_15.xml By default.SecurityLevelFilter" /> </appender> 5.xml production-log4j.servletfilter. To use one of the others.1.xml development-log4j.healthinsurance. the elapsed time needed to process the request is logged as follows: DEBUG com.file Java option in the setDomainenv script to point to the custom configuration file. 3. Use the -Dohi.xml is used.4 Performance logging The following configuration enables the logging of timing information for User Interface actions: <logger name="com.filter.healthinsurance. The following configuration enables the logging of timing information for Web Services requests: <logger name="com. 2.oracle. Create a custom log4j.xml configuration file.healthinsurance.class="com.1.oracle.

1.config. 3.html http://logging. 5.org/logging-log4j/Log4jXmlFormat Operations Guide 42 .file=/fully/qualified/path/to/log4j.1.log4j.1.apache.7 Associating a log level with an appender The following configuration example shows how to specify the logging detail level with a specific appender: <root> <level value="debug"/> <appender-ref ref="fileAppender" /> </root> It is also possible to customize log levels for specific parts of the system or for specific Java classes. http://logging.xml 5. more fine-grained logging or logging to multiple channels impacts the performance of the system.-Dohi. Note that any changes made to a log4j.2/apidocs/org/apache/log4j/PatternLayout. Do not log to the console also as this does not provide additional data (only the same data in a multiple places) and the additional logging decreases the performance of the system.1. This may be requested for support purposes. For example.2.8 Making changes to logging Care should be taken in changing logging parameters. 5.apache. Oracle recommends to use a fileAppender for OHI Claims logging only.xml configuration file are activated immediately without restarting the system. in order of no logging to all logging: • • • • • • • • off fatal error warn info debug trace all The OHI custom level "security" is between warn and info.6 Log4j Log Levels Possible log levels.org/log4j/1.apache.2.2. 2.org/log4j/ http://wiki.

Besides recovery from technical errors in individual processes. 6. 6. Tasks go through different states: • • • • Initial: the task is initialized. Ease of maintenance: when the root cause of the problem is resolved. The goal is to resume processing by only repeating the task in which the error occurred. Completed: execution of the task ended as expected. This involves execution of basic sanity checks and construction of the task's context. Pending: the task is executed. Errored: an error occurred that prevented the task from completing successfully. technical errors impact a large number of similar operations and recovery costs could be high (both in terms of system resources that are required for recovery as well as regarding risks to meet service level agreements). The screen was designed with the following principles in mind: • Consistency and simplicity: the screen provides a generic recovery facility that applies to any task in the system. the system supports (manual) recovery from the error in a simple way for all tasks that failed as a result of it.g.1. recovery requires manual intervention: after the problem is resolved. it is important to detect and resolve any error as soon as possible in order to restart failing or incomplete tasks with a minimum of reprocessing. Restarting from a particular task prevents reprocessing that would happen if processing of a claim had to be restarted from the beginning. processing of errored tasks must be resumed.2 Recover errored tasks in the Claims Processing Flow In a number of cases.1 Recover from Application Failures 6. 43 • Operations Guide . Error recovery is centered around tasks. The claims process flow in OHI Claims is built up as a series of distinct tasks. because specific systems are down or as a result of network failures Errors in dynamic logic scripts Typically. Examples of technical errors are: • • External web services endpoints are unavailable. Therefore.6 Error Recovery This chapter describes the capabilities provided by the system to recover from errors. the system is able to recover from system or node failure. This is done through the Recovery from Technical Errors screen. e.1 Tasks Any processing in OHI Claims is based on units of work referred to as tasks. in as close as possible to the state prior to the incident. Different error-processing on a per-task basis would be harder for operators. Recoverability in OHI Claims manifests in different places and at different levels of operation. Tasks in the flow allow the following: • • Fail fast in order to recover as close as possible to the state prior to the incident.1.

Advanced Search has between search on Last Updated Date. the code of the item associated with this task. the system supports bulk operations to achieve this. Note that no data access restrictions apply on the Claim code in this page. for example a Claim Code. The screen has three main features used to: • • • detect tasks in the errored state aid in issue resolution and resubmit them for processing. 6. The screen is used to view tasks that are in a technical error state.1 Errored Task Listing The screen shows a table with the following columns.1. Any user with access to this page should be able to see all Claim codes (and the codes of other entities where data access restrictions might apply in other pages) Actions • • Details: shows the full stack trace captured when the task errored Process: resubmit the claim to the claims flow • Quick Search and Advanced Search are available on Last Updated Date.1 Process All The process all button requests processing on all tasks returned to the screen by a query. the ID.1. or on the whole batch of similar tasks in the error state.2. the operator can then trigger the resumption of processing on either an individual claim. the entity type of the Target associated with this task. Therefor. Task Name and Type. in this order: • • • • Last Updated Date. Operations Guide 44 . the date and time when this task was last run Task Name.1. 6. or if the entity does not have a code.Manually restarting a large volume of errored tasks one by one is not a valid option.2. Once the root cause of a problem has been identified and resolved. the name of the task in errored status Type.

The batches keep track of an internal state. If task initialization succeeds the task is recoverable. In this fashion. As soon as unit of work is picked up from this persisted queue.2. The task goes into the errored state and processing of the particular claim is stopped. and the screen contents are re-queried. in the rare case that dequeuing fails.1. 6.4 Recover from failures in batch processes Processing the contents of large files may take a long time. When a target is re-submitted to the processing flow the status of that task changes from error to pending. The failing task must be re-started from the Recovery from Technical Errors screen.3.2 Actions: Details and Process Each row in the errored claim screen has a feature for showing more details about the failed task. the system provides JMX managed beans to: • • Detect if there are messages in the error queue Resubmit messages from the error queue to the work queue 6.1. For consistency and ease of recoverability.1. 6. 6. The exception details are a single text box containing a stack trace.1 Failing to deliver a message for which processing of a claim should not be interrupted For example: failure to deliver claims events messages. a task can not be returned for processing multiple times.1.2 Recover from system failure • • Catastrophic system failure Node failure Operations Guide 45 . 6. The failing task must be re-started from the Recovery from Technical Errors screen.1. A failing batch process can be restarted from its last known commit point. potentially several hours. OHI Claims file processing is handled in a batch processing mode. a task is created for each batch process. This ensures that the Recovery from Technical Errors screen can be used to re-start an unfinished batch process.3 Recover from external service failures For recovery from failures to deliver a message to an external or outbound service the following cases are distinguished: 6. formatted as a readable java stack trace. Therefore. the message from the database queue is placed in the associated error queue.5 Recover messages from the Task Error Queue OHI Claims makes use of a database queue for workload balancing. a task is initialized.3.1. The timing for delivering these types of messages is not critical.6. or for restarting that task. The task for delivering the claims event message goes into the errored state (the guiding principle being that the event message needs to be delivered) but processing of the particular claim continues by spawning the next task. even after a (catastrophic) system failure. In order to process these messages.2 Failing to deliver a message for which the Claims Flow should be interrupted Examples are enrollment request messages: enrollment data is a necessity for adjudication of a claim.1.

'<sid setup db>'. NULL.1 Clone the Database Use the RMAN backup tool to duplicate the database. '<host setup db>'.nextval. Duplicating a Database2. PORT. '<db description>'. <single digit unique discriminator value>). The Oracle software that runs the OHI Back Office application need to be installed first: • • Follow the instructions for install and configure an OHI database (page 0). DESCRIPTION. Follow the instructions for install and configure Oracle Fusion Middleware (page 0). To do that. paragraph Enabling Replication of Setup Data. Note that performing direct data manipulation on the OHI_TASK_QUEUE_TABLE is not supported. DISCRIMINATOR) values (ohi$instance_s1. the task queue needs to be emptied by executing the following: DECLARE po_t dbms_aqadm.g. port. database sid and description to the required values): delete from ohi$instances. BEGIN dbms_aqadm. 7. insert into ohi$instances (ID.3 Enable Replication of Setup Data In accordance with the concepts explained in the Installation Guide. po_t).2 Empty the Task Queue Before the application that uses the cloned database is started. Operations Guide 46 . 7. especially Chapter 24. in case data needs to be transported between environments using the Configuration Migration tool). follow the standard Oracle® Database Backup and Recovery User's Guide1. The following script removes any existing data from the ohi$instances table and inserts a new value for the cloned environment (change the host. it may be a requirement to ensure that unique identifiers that are generated in the cloned environment are unique across all OHI environments (e.aq$_purge_options_t. HOSTNAME.purge_queue_table('OHI_TASK_QUEUE_TABLE'. <port setup db>.7 Copying an Environment There can be several reasons why you would want to copy (or clone) an environment: • • • Copy a setup environment as the initial version of a test environment Copy a setup environment as the initial version of the production environment Copy the production environment to the test environment This chapter describes the procedure for copying environments. SID. 7. END.

use the setSequences utility to reset sequences for transportable tables. 2.oracle. See the Setting sequences (page 10) chapter for instructions. 7. section Validate Installation.oracle. to conform to the definition in the ohi$instances table.112/e10642/rcmdupdb.112/e10642/toc.5 Validate Installation Validate the installation by performing the steps that are documented in the Installation Guide. Note that properties files and data source configuration need to be specifically set up for the target system. 7.com/docs/cd/E11882_01/backup. 1.Now.4 Deploy the Application Follow the instructions in the Installation Guide for installing the OHI Claims application on the application server.htm#i1008564 Operations Guide 47 .htm http://download. http://download.com/docs/cd/E11882_01/backup.

Click on 'coreapplication'. Click on the 'Apply' button in the top right of the screen. Now restart the OBIEE server and the repository is available in OBIEE. open the Enterprise Manager of Fusion Middleware Control 11g. This can be done in the connection pool object in the physical layer. An overview of the BI components is shown. Enter the password of the repository (default = oracle. Besides the installation of the repository itself. Enter the database SID and the correct user to access the views. Under 'Upload BI Server Repository' click on the browse button and choose the 'OHI Reporting Views.8 OBIEE Reporting Views 8.com/docs/cd/E14571_01/bi. The installation procedure is described in the following document: http://download. the reporting views can be accessed with OBIEE. change this to a new password). it is necessary to customize the OBIEE repository. This is because the Reporting Views can be changed per installation.oracle. Choose the Business Intelligence folder in the domain. Before the repository can be installed. Make the deployment updatable by clicking on the 'Lock and Edit Configuration' button.1111/e10539/toc. Therefore it is necessary to define the connection details of the database where the views are present. After the OBIEE 11g software installation. Click on the tab 'Deployment'. the OBIEE software must be installed.htm The . After this setup and the customization.1 Setup the OBIEE Repository This section describes how to set up the OBIEE environment to make use of the OHI Reporting Views OBIEE repository. Setup In this paragraph a brief instruction is given how to setup the Reporting Views repository (. Operations Guide 48 . The Deployment page is used to load a repository file onto the server. This is described in Customize the OBIEE Repository (page 49).RPD file translates the OBIEE analysis to queries to the database.RPD file) into OBIEE.rpd' file.

they can be added to the repository: 1.htm In the default OBIEE repository only functional views are added (name starting with FUN) Add new view When there are customized own functional views. Because the views are 1:1 added in the business model without dimension modelling. Create a connection with this dummy dimension.3 Customize the OBIEE Repository The concept of the reporting views is that they can be customized per installation. the view can be removed in OBIEE: 1.8. The context is set before a user is executing a query in OBIEE. Delete the presentation object for the view. In this chapter. Operations Guide 49 . 4. a brief instruction is given how to customize the OBIEE repository to the specific needs.oracle. Save the repository. Therefore it is most likely that the default setup in the OBIEE repository does not match the specific customized views. Detailed information about administration of an OBIEE repository can be found here: http://download. 2.com/docs/cd/E14571_01/bi. Check in the changes. Delete the physical definition of the view. 3. Make a connection with the dummy dimension Note: OBIEE requires that a 'fact' table is connected to a 'dimension' table. Delete the business object for the view. It is recommended that the customizations in the OBIEE repository are done by someone with administration knowledge of OBIEE. Import the view in the physical layer Add the view that must be added into the existing 'Datasource Reporting Views' connection group. This means that users can only see data that they are also allowed to see in the OHI Claims.1111/e10540/toc. Views can be imported into the physical layer by using the metadata import functionality (can be found under 'File') Add the new view to the business layer by dragging it into the OHI Reporting Views subject area. a dummy dimension is created. 4. 5. Users The users defined in OBIEE must be the same as the application users in the OHI Claims application 8. 5. 6.2 Security Introduction The reporting views make use of the OHI Claims application security settings. Remove view When a functional view in the OBIEE repository does (not longer) exists in the customized environment. 3. Add the new view to the presentation layer Check in the changes Save the repository 2.

3.Change existing view When a column is added. The repository needs to be changed: 1. 2. Open the view definition in the physical layer that must be changed Add. also add or delete the column in the presentation and business layer. Operations Guide 50 . delete or rename the column In case of adding or deleting a column. removed or renamed in an view that already exists in the OBIEE repository.

Claims Pages Example: The url below opens the View Claim page and queries 1 particular Claim (for an authenticated user. http://<host:port>/<appl. /faces/UIShell?jhsTaskFlowName=<pageCode>. opening the "About this page" popup. Use the id in the database ID column. This section describes the structure of the URL so that it may be used for other deep-linking purposes as well. For opening a different variation of the claim page. (optional) parameter: &rowKeyValue<pageCode>=<id>. The part of the URL that is shown before "/faces". otherwise the page states that no rows were found. the page will be opened without querying any data. 3. 2. Operations Guide 51 . &jhsSubMenuName=<menuName>.9 Deeplinking to screens The workflow interface will supply a URL for the user to deeplink to certain UI pages (from outside the application). Use the same page code as in the previous step. any page can be combined with any menu. 4. If this parameter is not specified. The page code can be retrieved by going to the right page.domain:port/base/faces/SearchClaims?jhsTaskFlowName=ViewClaims& rowKeyValueClaims=17 To create such a url. 4. add: &uniqueIdentifier=< edit/mpricing/mbenefits/madjudication/unfinalize>. 2. otherwise the Login page is presented first): http://host.name>. several parts are concatenated: 1. In theory. several parts are concatenated: 1. and use the id in the database ID column. 3. /faces/SearchClaims?jhsTaskFlowName=ViewClaims &rowKeyValueClaims=<id>. http://<host:port>/<appl. otherwise the Login page is presented first): http://localhost:7001/base/faces/UIShell?jhsTaskFlowName=Persons& jhsSubMenuName=ReferenceDataMenuModel&rowKeyValuePersons=10242 To create such a url. This will only work correctly if the id of part 4 is of a claim with the right status for that page. The allowable values for menuName are ReferenceDataMenuModel and ConfigurationMenuModel. Ensures that one row is automatically queried when the page is entered.name>. Ensures that one row is automatically queried when the page is entered. and looking up the Access Restriction Code. Other Pages The url below opens the Persons page and queries 1 particular Person (for an authenticated user. but of course the menu to which the page belongs should be used. The part of the URL that is shown before "/faces".