Implementation Study: Dell IT Scales Supply Chain Management with Oracle 10g RAC

By Dave Jaffe and Todd Muirhead Dell Enterprise Technology Center Tiong Tey and Raveendra Avutu Dell Information Technology

Dell | Enterprise Technology Center

dell.com/techcenter

January 2007

Contents
Executive Summary .......................................................................................................3 Introduction ....................................................................................................................4 Dells Supply Chain Management.................................................................................6 The Old Solution Proprietary Unix-based Servers...................................................7 The New Solution Oracle 10g RAC on Dell Power Edge Servers ...........................9 Measured Performance Improvement........................................................................12 Conclusions..................................................................................................................13

Tables Table 1: Supply Chain Management components at Dell ............................................................... 4 Table 2: Global Dell Supply Chain Management Deployments .................................................... 11 Table 3: Performance Gains of Ten Longest Running Database Transactions............................ 12

Figures Figure 1: The Proprietary Unix-based Solution ............................................................................... 7 Figure 2: The Dell-on-Dell Solution ............................................................................................... 10

January 2007

Page 2

Dell Enterprise Technology Center

Section

1
In 2005, Dells Information Technology group was faced with a supply chain management system that had reached its limits on expensive, proprietary servers running the Unix operating system. The group undertook the complete migration of the system to Oracle 10g Real Application Clusters running on low-cost industry standards-based Dell Power Edge servers. By moving the application to Dell servers when it did, Dell IT avoided significant new expenditures in proprietary Unix-based servers, achieved increased server uptime, and provided an easy path for growing the systems as needed by adding additional servers to the cluster. This paper documents in detail the scope of the supply chain management system, and the advantages of migrating it from Unix-based servers to Dell PowerEdge servers.

Executive Summary

January 2007

Page 3

Dell Enterprise Technology Center

Section

2
At Dell, the supply-chain management database systems manage key business functions that support Dells worldwide manufacturing operations. Table 1 shows the major components of Dells efficient inventory management model and fast, direct-to-the customer delivery of computers, accessories, parts and supplies. If these systems are down, Dells factories are down, at the cost of thousands of dollars per minute.

Introduction

Component Configuration Management Procurement Cost Inventory Accounts Payable

Description Manages part numbers and Bills of Material Manages Purchase Orders and communicates Dell production materials requirements to the suppliers Manages key areas of materials cost processes that calculates Bill of Materials cost and cost of good sold for Dell Orders Manages Dell and Hub inventories. Within each facility, it manages inventory stock room locations/bins and on-hand quantity. Manages Dell to suppliers payment processing. It also deals with invoices, receipts, and EDI processing.

Table 1: Supply Chain Management components at Dell When Dell was a smaller company, before the development of powerful, industrystandard servers of the type that Dell manufactures and sells, the Dell Information Technology (IT) unit ran these database applications on large, expensive, proprietary Unix-based servers. However, as the company grew, these systems could not scale with the workload. Instead they were the cause of frequent downtime. Since they were not redundant it was necessary to bring down entire systems to update a single server. And when a server could not handle the required capacity it needed to be replaced by a larger server. In recent years, the increased performance of x86-based industry standard Dell PowerEdge servers has enabled the deployment of such database software as Oracle 10g Real Application Clusters. By sharing a large database across multiple PowerEdge servers, Oracle 10g RAC provides for very cost-effective scaling, high reliability, and the ability to add capacity by adding additional lowcost servers rather than buying larger, more expensive proprietary Unix-based servers. Dell IT has successfully implemented this solution for Dells Americas, Brazil, Europe, and Asia Pacific/Japan regions. By using the same processes for disaster recovery, backup, and monitoring across all Dell operations, Dell IT has achieved a cost-effective and readily supported deployment model.

January 2007

Page 4

Dell Enterprise Technology Center

This article describes in depth how Dell IT made the transition from proprietary Unix-based servers to running mission critical supply chain management applications on Dells own hardware. The problem being solved by these database applications is described in detail in Section 3. The old solution and the Dell-on-Dell solution are described in Section 4 and Section 5. Finally, measured performance improvements are shown in Section 6.

January 2007

Page 5

Dell Enterprise Technology Center

Section

3
The sheer number of transactions and pieces of information that the supply chain management system needs to handle is impressive. Each of the core components of the supply chain management (SCM) system are heavily used and relied on to keep Dells operations running smoothly. Configuration Management: The Configuration Management system manages over 1 million Dell part numbers across approximately 200 product families, and over 2 million Bills of Materials (BOMs) per year. BOMs listing component part numbers are created for manufacturing to build assemblies and sub-assemblies to produce Dells products. Procurement: The Procurement system manages nearly 1.8 million Purchase Order lines per year, from more than 5,000 suppliers worldwide. To streamline the procurement process Dell uses an automated application which includes workflow approvals and vendor communication, and provides for services such as defective part warranty replacement. Cost: The Cost component of the system runs mostly in batch mode to calculate the costs to Dell for all Bills of Materials. These batch jobs run weekly, monthly and quarterly, with each run rolling up total material costs. Inventory: Between all sites there are more than 3 million inventory movements daily from stock rooms to the factory floor. A corresponding 3 million messages are transmitted to various systems for reporting, analysis and factory scheduling. Accounts Payable: Accounts Payable handles approximately 15,000 items per day including payments to Dell suppliers, invoices and receipts. Vendor information includes number, location, negotiated terms and contact information. On top of these order-related transactions, there are several batch process jobs that need to be run to rollup data every week, month or quarter. The longest of these, the end-of-quarter (EOQ) rollup, took 31 hours under the Unix-based solution. In Dell's Americas region the SCM Oracle database application consists of approximately 3,000 database objects (functions, packages, procedures, triggers, tables, and views). The same SCM system is also supported by 6 Dell PowerEdge 2650 application servers, 5 internally developed web-based applications, more than 50 system-to-system integrations, approximately 125 batch jobs, and about 500 user interfaces deployed to support the entire SCM application.

Dells Supply Chain Management

January 2007

Page 6

Dell Enterprise Technology Center

Section

4
The Dell supply chain management solution was built out on Sun E6000-class Unix-based servers (see Figure 1). By 2005 performance was becoming an issue.

The Old Solution Proprietary Unix-based Servers

Figure 1: The Proprietary Unix-based Solution

January 2007

Page 7

Dell Enterprise Technology Center

Many of the batch processes were taking too long. The version of Oracle that was running on the Sun servers was outdated and unsupported, and did not support Real Application Clusters so adding more capacity by horizontally scaling out by adding more servers was not an option. The servers in the old solution shown in Figure 1 are Sun Enterprise E6000 servers running Sun Solaris 8 with Sun Cluster 2.2. The Primary and Secondary servers, each with 16 336MHz processors and 11GB of memory, managed the production database on Oracle 8.0.6. The Disaster Recovery system, with 12 336 MHz processors and 6GB of memory, was the backup. The disk storage was provided by EMC Symmetric arrays. High Availability on the Sun systems was achieved by clustering two servers connected to a shared disk. In this Sun Cluster configuration one server was active while the other server was in a standby or passive state. The database would fail over to the passive node from the active when there was a hardware or OS issue with the active server. The failover would be initiated when the passive node no longer detected a heartbeat signal coming from the active system. To accomplish this the Sun cluster would shut down Oracle on the active node and move the Oracle resources to the passive node. All users were disconnected during this failover and would have to reconnect.

January 2007

Page 8

Dell Enterprise Technology Center

Section

5
With Dells rapid growth and the resulting increase in load on the supply chain management system it became apparent in 2005 that the proprietary Unix-based solution on Sun servers was quickly running out of capacity. A very large investment in even larger proprietary servers would be needed to stay with the Unix-based solution or a much lower cost and more scalable solution could be implemented on Dell PowerEdge servers with Oracle 10g Real Application Clusters (RAC). It was decided that the best approach to provide long term scalability was to switch platforms and implement an industry-standards based platform. It was necessary to act quickly to move to an Oracle10g RAC solution in a short period of time to avoid the need for any further investment in the Unix-based platform. As shown in Figure 2, the Sun systems were replaced with 6-node Oracle 10g RAC clusters running on PowerEdge 6650s, 4-way Xeon based servers. The new environment consists of three 6-node RACs. The Primary system is a 6node RAC environment. The Disaster Recovery (DR) site has two 6-node RACs one using Oracle Dataguard and another using EMCs Symmetrix Remote Data Facility software (SRDF) to provide DR to the primary database. Oracle10g RAC architecture has built-in High Availability features. All RAC nodes/instances share the same physical database. If there is an issue with any of the physical nodes or instance, the users and connections are failed over to the other surviving instances in the RAC cluster. This is a seamless operation and is transparent to the users, meaning that the users do not have to reconnect as they did with the previous solution. Disaster Recovery is achieved through Oracle Dataguard and also using EMC SRDF. A six node RAC was set up using the Dataguard Broker component of Dataguard to ship logs in real time from the Primary system to the DR system (located across town) and apply them. Using Dataguard allows the database to keep in synch and also prevents any physical corruptions from being copied to the DR database (from the Primary). SRDF is used to ship the changed blocks at the storage level from the primary site to the DR site. This is an EMC Symmetrix feature that allows the transfer of changed blocks in real time. This method is fast but doesnt guard against data corruption. The combination of Dataguard and SRDF provides fast and secure replication of data to the DR site. The Oracle RMAN program is used to backup the database and archivelogs from the primary database. Two full (hot) backups per week are sent to tape directly through Legato Networker interface. Archivelogs are backed up hourly to provide the ability to recover the database to a point-in-time, if needed. It is also possible to backup the database from the DR site instead of taxing the primary nodes. Dell IT employs Oracle 10gs Services to load balance within each cluster. In the 6-node cluster, for example, nodes 1-3 handle online users while nodes 4-6 handle batch processing.

The New Solution Oracle 10g RAC on Dell Power Edge Servers

January 2007

Page 9

Dell Enterprise Technology Center

Figure 2: The Dell-on-Dell Solution

Oracle Grid Control was implemented to monitor and manage the RAC environments. Thresholds for various events were set in the Grid control to

January 2007

Page 10

Dell Enterprise Technology Center

create trouble tickets through the problem management system. Grid Control makes the administration of the database very easy and intuitive. Database administrators use Grid Control extensively to perform day to day administration tasks. The old environment could not scale beyond the capacity of one physical server, while in the new architecture additional servers (nodes) can be added to the cluster on demand to meet business requirements and growth. The multi-node RAC architecture provides higher input/output throughput due to the increased number of available interfaces. The old Sun cluster was active/passive which meant the database was only running on a single node at any given time. In case of any failure the database had to be restarted on the passive node which required all users to reconnect. In the new architecture the failover is transparent and the database instances run on all nodes at all times. The failure of any node does not require any users to reconnect because they are automatically and transparently moved to another node in the cluster. As seen in Table 2, this same architecture has been extended to Dell facilities around the world. By standardizing on cost effective Oracle 10g RAC running on Dell PowerEdge servers with Dell/EMC storage, with similar processes for deployment, disaster recovery and backup, the new solution was rolled out around the world in a very short 8 months Region Americas China Brazil Europe Malaysia/Asia Primary 6-node Oracle 10g RAC 3-node Oracle 10g RAC 2-node Oracle 10g RAC 2-node Oracle 10g RAC 2 sets of 2-node Oracle 10g RAC Disaster Recovery Two 6-node Oracle 10g RAC clusters 3-node Oracle 10g RAC 2-node Oracle 10g RAC 2-node Oracle 10g RAC 2 sets of 2-node Oracle 10g RAC

Table 2: Global Dell Supply Chain Management Deployments

January 2007

Page 11

Dell Enterprise Technology Center

Section

6
Detailed measurements were made of database performance with the old system and the new system. The ten longest running end-of-month and end-of quarter transactions are shown in Table 3. The best gain was seen in a material movement extraction step, which dropped from almost 5 hours to 35 minutes, an 88% improvement. The end-ofquarter rollup mentioned in the Introduction dropped from 31 hours to 23 hours.

Measured Performance Improvement

Job Description

Old Time (min) 1860 532 450 360 356 333 289 198 180 157

New Time (min) 1380 340 116 240 137 118 35 192 43 40

Entire end-of-quarter jobs processing Compute cost per order Quarterly BOM calculation Entire end-of-month jobs processing Compute new product material costs Summarize data for all levels of order details Extract data for all material movements transactions Roll up all costs (material, royalty, transportation) Create material management transactions based on inventory transactions Create financial journal entries

Performance Gain 26% 36% 74% 33% 62% 65% 88% 3% 76% 75%

Table 3: Performance Gains of Ten Longest Running Database Transactions

January 2007

Page 12

Dell Enterprise Technology Center

Section

7
Dells Oracle-based supply chain management (SCM) is the heart of each regions operations. Not only are Dells factory operations totally dependent on the health of these systems but Dell internal systems are dependent on this SCM system to provide information and visibility to key business functions. With the implementation on Oracle10g Real Applications Clusters running on industrystandard Dell PowerEdge servers, the SCM database systems have enabled Dell's Direct Model to scale efficiently. This scaling has resulted in lower material costs, near real-time visibility to inventory level, and tighter integration with suppliers. By moving the application to Dell servers when it did, Dell IT avoided significant new expenditures for proprietary Unix-based servers, increased server uptime, and provided an easy path for growing the systems as needed. The move to a Dell-on-Dell solution has been a big win for the company.

Conclusions

THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. Dell and PowerEdge are trademarks of Dell Inc. EMC, Navisphere and Symmetrix are registered trademarks of EMC Corp. Intel and Xeon are registered trademark of Intel Corp. Red Hat is a registered trademark of Red Hat Inc. Linux is a registered trademark of Linus Torvalds. Microsoft and Windows are registered trademarks of Microsoft Corporation. Oracle is a registered trademark of Oracle, Inc. Sun and Solaris are registered trademarks of Sun Microsystems, Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others. Copyright 2007 Dell Inc. All rights reserved. Reproduction in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Information in this document is subject to change without notice.

January 2007

Page 13

Dell Enterprise Technology Center