California State Controllers Office

Preliminary Internal Review of the 21st Century Project

August 2013

Preliminary Internal Review The State Controllers Office (SCO) is responsible for operating the States existing payroll systems. KPMG Peat Marwick audited the States existing payroll systems as part of a 1995 assessment of the SCOs operations. KPMGs audit report concluded the State should replace the payroll systems because they are obsolete. The audit confirmed the systems are decades old, written in an outdated computer language and do not have the flexibility and functionality to meet the needs of todays state government. The 21st Century Project became a focus of the States attention after the issuance of the 1995 KPMG audit report. In 2004, after delays due primarily to budget constraints, the SCO received funding to work with other state stakeholders to purchase and implement a software suite to replace the States existing employment, payroll, timekeeping, leave, and position control functionalities. The SCO and other stakeholders spent considerable time on developing the project procurement process, both because of the fundamental importance of payroll to the States work, and because the State recognized that successful statewide technology implementation changes require thoughtful planning, procurement and contract strategies, coupled with practical business knowledge, technological expertise, implementation methodology and project management. The SCO brought its business expertise and knowledge associated with successfully operating the States $20 billion annual payroll to the 21st Century Project, including a practical understanding of the complexities of ensuring the States payroll is compliant with federal and state laws and regulations, the terms of state collective bargaining agreements and memorandums of understanding, and related business requirements. After the first system integrator was unable to perform, the SCO engaged in an extensive re-procurement process and ultimately selected SAP to be the system integrator. The objective in selecting a new system integrator was to hire the best-qualified candidate to successfully complete the job of configuring SAPs commercial off-the-shelf (COTS) payroll software to ensure that Californias payroll continued to be properly determined. Payroll is properly determined when it is consistent with federal and state laws and regulations, the terms of state collective bargaining agreements and memorandums of understanding, and related business requirements. Unfortunately, over time, SAP proved itself unable to meet the requirements of the 21st Century Project, and SAPs failure led to its termination. As the system integrator and per contract requirements, SAP was responsible for planning, implementing, and executing a successful data conversion strategy. Their strategy did not include a data conversion tool that was sufficient to perform data validation or to convert the volume and complexity of data required. Without that tool, SAP was unable to perform its responsibilities and the project was delayed. After a cure notice was served upon SAP on August 19, 2011, SAP took steps to secure an effective tool. After resolving the data conversion issue, a new plan was developed for the project with a small pilot of only 1,400 employees scheduled for June 2012. After go-live significant errors surfaced during the first payroll

August 12, 2013

cycle and persisted through each of the subsequent eight monthly payroll cycles. SAP was never able to stabilize the system even for this small set of employees. SAPs termination resulted from its delivery of an unstable payroll system plagued with errors. SAP could not explain or fix many of the errors, and was unresponsive to the States request that it focus on identifying the root causes of these errors and then create a corrective action plan to demonstrate how SAP would address them so the State could have confidence that the 21st Century Project should go forward. SAPs lack of responsiveness to the States significant concerns lead the SCO to issue a cure notice demanding that SAP address the payroll systems defects. SAP responded to the cure notice with an unequivocal denial that the payroll system had any issues. SAP then went on to allege that if there are issues with the payroll system, they are all the fault of the State. As a consequence, and taking into account contractual remedy limitations and other considerations, the SCO terminated SAPs contract in February 2013. SCO employees being paid through the SAP-built payroll system were rolled back to being paid on the States legacy payroll systems. What Should Happen Next? The SCO is focused on completing the following tasks: Complete payroll reconciliation (also known as Do the Math) to ensure the SCOs 1,400 employees and its business partners are made whole; Support the States legal pursuit of full contractual restitution from SAP for its failed work; and Reduce MyCalPays maintenance costs.

In addition, the SCO believes the State should move forward with an independent assessment(s) to answer the following questions: What project work to date is salvageable and could be leveraged as part of a go-forward California plan? Can the SAP system meet the States requirements? Can the State simplify and modernize its payroll practices? What options and alternatives exist in terms of completing the replacement of the payroll system, along with a cost-benefit analysis for each? What lessons can be learned from the project to improve the States information technology project management practices? What is the States best path forward to achieve its original goal of modernizing its human resources and payroll systems?

SCO is working with the Department of Finance and Department of Technology to ensure the independent assessment is completed. To start this process, below the SCO provides its preliminary internal review of the project and what can be learned from it.

August 12, 2013

Preliminary Review of What Went Right 1) The project followed state guidelines, best practices and industry standards related to project management. In managing MyCalPAYS, the project adhered to the general framework and processes contained in the Californias Statewide Information Management Manual (SIMM) guidelines and Project Management Methodology (PMM). The PMM is consistent with the Project Management Body of Knowledge (PMBOK) standard issued by the Project Management Institute. The 21st Century Project Management Plan consists of nine components: Risk, Issue, Schedule, Change Control, Document, Status Reporting, Quality, Requirements Traceability Management plans and the projects Governance plan. Each of these components was managed pursuant to the PMM. In addition, the States Project Management Office (PMO) maintained internal Deliverable, Contract and Cost management plans for overseeing the contract work. 2) The project established strong formal governance and four levels of oversight. Consistent with the States best practices, the project had formal governance and oversight in place. Project governance included the PMO, Project Leadership Team, Operational Steering Committee, Executive Steering Committee, and Project Sponsor in the decision-making process. The project is governed by an Executive Steering Committee, chaired by the State Controllers Office (SCO) and composed of voting members from the Business Transportation and Housing Agency (BTH), California Environmental Protection Agency (CalEPA), California Natural Resources Agency, State and Consumer Services Agency (SCSA), Labor and Workforce Development Agency, Department of Human Resources (CalHR), Department of Finance (DOF), California Health and Human Services Agency (CHHS), and Department of General Services (DGS) and non-voting members from the California Technology Agency and SCO project leadership. The Legislative Analysts Office (LAO) also provided periodic oversight reporting to the Legislature. In addition, general project oversight was provided by the Independent Validation and Verification (IV&V) and the Independent Project Oversight Reporting (IPOR) processes, both with contractors and consultants embedded in the project. These leadership, decision-making, and oversight bodies and organizations were kept abreast of project status, issues and risks on a regular basis through defined governance, meetings, and status reporting. 3) The project identified critical system defects in Pilot 1 and stopped deployment, thereby preventing additional impact to employees.

August 12, 2013

The two-pilot and three-wave deployment strategy was designed to limit the risks of any state payroll issues to as small a group of employees as possible by ensuring the project did not progress unless the current pilot was complete and stable. This mitigation strategy prevented harm to additional state employees. It limited the risks of the payroll disruption experienced by many of the 1,400 SCO employees in Pilot 1. Further, SAP was not permitted to start Pilot 2 despite its repeated requests to do so when it could not remedy the issues in the Pilot 1 payroll phase. SAP was ultimately terminated because it could not or would not address the Pilot 1 errors. 4) The project used an FDA-like gate approach to ensure that risk is contained and that key foundational activities are completed before advancing. To manage the risk associated with a statewide implementation, the MyCalPAYS system was designed, configured, developed and tested to help assure scalability by tackling the each design issue early in the project lifecycle. The system would be tested at each stage, and then rolled out to incrementally larger populations in Pilot 1 and Pilot 2 to limit the impact to users and employees should there be significant issues. After stabilizing Pilot 1 and moving to Pilot 2, there was a planned six-month stabilization effort before Wave 3 and deployment to an estimated 51 departments and 78,992 employees. A three-month stabilization period was provided for Wave 3, and then Wave 4 would be implemented to five departments and 79,003 employees. Another three-month stabilization period was provided for Wave 4 before deploying the system to Wave 5, which would comprise the final deployment wave of 118 departments and 61,910 employees. Another three-month stabilization period was provided for Wave 5 and then the system would be accepted, assuming that it met all of the acceptance criteria.
Figure 1

&

MyCalPAYS Project Development and Approval Process , , IVV , ,

Pre Live Research and Development Small Scale Live Pilots Large Scale Deployment State Officially Accepts and Replaces Legacy

Conceptual Vision FSR

Gate

Gate

Gate

Gate

Figure 2

&

FDA Pharmaceutical Development and Process , , IVVApproval , ,

Pre-Clinical Research and Development in Laboratory Clinical Trial With Live Subjects Large Scale Clinical Trial FDA Approval and Manufacturing Begins

Drug Conceptual Vision

Gate

Gate

Gate

Gate

August 12, 2013

Figure 1 is the MyCalPAYS Project Development and Approval Process and Figure 2 is the FDA Pharmaceutical Development and Approval Process. These high-level processes are similar in that they both require initial research and approval, and a structured and defined development process with significant laboratory testing before the product or system can move to a small-scale live pilot. Success along the way by meeting defined acceptance criteria means that the product or system can move to the next phase. The Small Scale Live Pilots (Pilot 1 in the MCP example) serves to validate that all of the laboratory testing is correct and accurate in a limited live environment before introducing it into a larger environment (Pilot 2). This serves to reduce the risk or exposure in a larger population. Both processes apply management disciplines designed to minimize the risk of implementation to a larger population. Failure at any part of the process requires stopping and analyzing what has gone wrong, if it can be fixed, and if so, a plan to move forward. Preliminary Review of What Went Wrong and Lessons Learned When appropriate, this section also offers preliminary lessons learned that can be applied to the States $5 billion information technology portfolio in order to help other projects avoid similar pitfalls and increase the probability of success. 1) The project needed more robust contractual carrots and sticks to manage the vendor. The contract lacked effective tools to manage vendor performance, such as liquidated damages, performance related penalties, or a performance bond. Lesson Learned: These provisions can be useful in focusing a vendors attention on its performance. 2) Public Contract Code section 6611 procurement guidelines were not in place in 2009. The SCO piloted a two-stage procurement using this section as an essential and key component of completing a major procurement in an accelerated time frame (one year). Due to the fact that PCC section 6611 had never been used in an information technology procurement, the Department of General Services (DGS) had no specific guidelines or procedures for conducting a procurement and negotiation under this section. As a result, the State was limited in its ability to thoroughly vet and communicate with the potential bidders for fear of protests accusing favoritism. Lessons Learned: Guidelines could have helped the State achieve a better balance between protecting against protests and claims of favoritism and being able to adequately assess bidder qualifications. In April 2013, DGS adopted guidelines for this code section. 3) The contract payment structure was not designed to align the States and the vendors goals.

August 12, 2013

Because payments were made based on the completion of individual tasks, such as completing a test or crafting a schedule, SAP keyed in on these activities, rather than focusing on the States overall goal of delivering a reliable and accurate payroll system. Lessons Learned: Restructure payments so they are tied to system progress at rationally-defined milestones (e.g., accurately paying 1,400 employees in Pilot 1). This would likely focus the vendor on the States project goals, rather than incremental billing. 4) SAPs testing methodology was inadequate. This factor above all others played very heavily into SAPs termination. System testing was insufficient, as evidenced by the number and types of system defects in Pilot 1. The responsibility for testing rested squarely with SAP, as identified in the Statement of Work, which requires SAP to, among other things, provide the test team, tools, methodology, proper test coverage, and comprehensive test plans for the MyCalPAYS (MCP) solution, including thorough testing of all interfaces. The State trusted SAPs representations about its qualifications and unique integration experience and believed it would provide quality work. Yet, SAPs testing proved substandard. The deficiencies in testing only became apparent when Pilot 1 went live. Payroll comparison testing was also inadequate. At the completion of Payroll Comparison Test 3, SAP assured the SCO that employees pay was correct within the range of $1 and that differences in the test outside of this tolerance were accounted for by known differences that could be addressed. This set the expectation that payroll would be accurate within $1 with the exception of known differences. At the time, the Payroll Comparison Test results were delivered and accepted, SAP assured the SCO that the payroll would be accurate and the SCO relied upon SAP as being uniquely qualified as the developer of the software to perform sufficient testing. These expectations were not met. Lessons Learned: To ensure better checks and balances, require a skilled independent party to design, execute, and validate sufficient testing. The use of a separate team to review the system functionality adds rigor and a formal testing discipline which can lead to improved quality. An independent assessment can ensure the system, people, and processes are ready to move into production as a mandatory gate to moving the system into a production environment. 5) SAP couldnt keep pace with fixing the system defects. Once Pilot 1 went live, far more system errors occurred than were ever anticipated. SAP executives seemed perplexed and could not explain why the errors were not identified in the numerous test cycles. SAP could not keep pace with fixing the system errors, which it was required to do under the contract. SAP could not determine the root causes of the system errors or implement a corrective action plan to address the problems encountered in Pilot 1. The chart in Figure 3 depicts the volume and type of new system defects incurred every month in Pilot 1,

August 12, 2013

plotted against the cumulative number of open (unresolved) defects. Thus, every month a new baseline was established which included the current and the preceding months unresolved defects.
Figure 3

6) SAP could not or would not disclose root causes of Pilot 1 defects. SAP informed the State that a special team of experts deployed to do a root-cause analysis of the problems in Pilot 1 had gathered 200 pages of findings. However, it would not provide those documents to the State. When the State pursued this request, SAP provided an abbreviated document that contained insufficient information to determine the root cause of Pilot 1 errors, or how they could be resolved. Lessons Learned: Without its own escalation team, the State relied on SAP to do the root-cause analysis for Pilot 1 problems. However, SAPs team appeared to be focused on asserting that its software was not at fault and did very little to resolve outstanding issues and get the project back on track. The State could have benefited from commissioning its own analysis instead. 7) SAP failed to provide an operations framework for Pilot 1. SAP failed to deliver its assessment of the SCO support model, provide an operations design framework, provide effective assistance to the State with setting up incident management, provide the knowledge

August 12, 2013

transfer required, and the production support processes required to operate the system. SAP failed to staff production support sufficiently to keep pace with resolving the volume of system defects identified, thus causing additional payroll errors every time payroll was executed.
8) SAPs blueprint validation process proved to be insufficient to design a system to produce

accurate paychecks. The statement of work required SAP to validate the States business requirements and confirm their completeness. In its proposal, SAP confirms that it is responsible for this role: During blueprint validation, SAP consultants work with the various state business owners to define business requirements fully so appropriate designs can be engineered that will provide a foundation to avoid future problems. Because this validation effort proved to be insufficient, SAP came out of this important phase with an incomplete understanding of the States business requirements to design the system. Consequently, SAP apparently assumed that a number of States business requirements would fit within the standard software functionality. Only through later deeper dives into the States requirements and analysis did SAP discover that its standard functionality could not comply with Californias legal and labor agreement requirements. This necessitated customizations and changes to the software. SAP claims that this is a result of attempting to build a system just like the States legacy payroll systems. However, SAP misses the point. The new payroll system must meet applicable legal and contractual compliance requirements. 9) SAP failed to provide the staffing needed to successfully deliver a project of this size and magnitude. Out of the 17 key positions identified in the SAPs procurement proposal, only one key resource remained on the project from the time the bid was let in February 2010 through the suspension in February 2013. The staff that was provided failed to perform required project roles and work activities, and failed to manage their respective teams, pursuant to the contract. SAP's resources should have leveraged the experience SAP represented it had on other large-scale public sector projects, and provided expertise on best practices and key lessons to reduce risk and ensure success. Moreover, SAP failed to consistently maintain the continuity of staffing with the requisite experience required in the SOW. Lessons Learned: The State needs additional means to ensure the vendor provides the level and quality of staffing proposed in its bid. Conclusion The SCO believes SAP failed because it was not committed to the same objectives as the State, despite a uniquely transparent procurement process, jointly-agreed upon contract deliverables, SAPs assurances that it understood the 21st Century Project better than any other potential integrator and SAPs assurances that it had never failed to successfully complete a troubled project. August 12, 2013 8

The State believed and relied on SAPs representations about its qualifications and unique expertise to be successful throughout the pre-production work on the project in areas requiring special COTS software expertise. Ultimately, however, SAP delivered an unstable payroll system that is plagued with material errors and proved to be a non-responsive contractor that could not fix the errors, determine the root cause of the errors, or create a corrective action plan to go forward. The combination of all these factors prevented success and, ultimately, the SCO had no option but to terminate SAP due to its lack of performance.

August 12, 2013