SLI Monthly Report - WA EMR-ICD10 Project - August 2015 - FINAL

WA Department of Social and Health Services
Washington EMR-ICD10 Project

SLI Monthly Report for the WA EMR-ICD10 Project for August 2015
SLI Monthly Report for August 2015

For the
Delivered by
SLI Global Solutions, LLC

th
216 16 Street
Suite 700
Denver, CO 80202
303.575.6881
www.sliglobalsolutions.com
Submitted: Draft September 2, 2015

Final September 14, 2015
SLI-WA-EMR-ICD10 August 2015

Confidential and Proprietary Materials Page 1
Contents
1. Introduction ................................................................................................................ 4
1.1 Overall Assessment for the Period of August 2015 .............................................................. 4
1.1.1 SLI Onsite Visit ............................................................................................................... 5
1.2 Primary Risks ......................................................................................................................... 6
1.2.1 Risk to project schedule and scope due to late milestones (QA-08) ............................ 6
1.2.2 Application is not Production Quality (QA-11).............................................................. 6
1.2.3 Risk of project termination due to inadequate SOC2 audit (QA-12) ............................ 6
1.3 Project Progress Highlights ................................................................................................... 7
1.4 Project Plan Update .............................................................................................................. 7
1.5 Technical and Infrastructure Issues ...................................................................................... 8
1.6 Testing Status and Updates................................................................................................... 8
1.6.1 RTM Updates ................................................................................................................. 9
1.6.2 Cerner Testing Tools.................................................................................................... 10
1.7 Exit Criteria (Go Live Criteria) .............................................................................................. 10
1.8 Project Administration and Controls................................................................................... 10
1.8.1 Project Schedule.......................................................................................................... 11
1.8.2 Communications Plan.................................................................................................. 11
1.8.3 Risk, Decision, and Parking Lot Tracker ....................................................................... 12
1.8.4 Requirements Management ....................................................................................... 12
1.9 Risk and Issues..................................................................................................................... 12
1.9.1 Risks ............................................................................................................................. 12
1.9.2 Issues ........................................................................................................................... 14
1.10 Findings and Recommendations ......................................................................................... 15
2. QA Specific Status ....................................................................................................15
2.1 Schedule Management ....................................................................................................... 15
2.2 Scope Management ............................................................................................................ 15
2.3 Resource Management ....................................................................................................... 15
2.4 Communication Management ............................................................................................ 15
2.5 Issue and Risk Management ............................................................................................... 15

3. QA Activity Report ....................................................................................................15

3.1 Record of Project Meetings and Activities .......................................................................... 15
3.2 Report and Deliverable Review Activities ........................................................................... 16
4. QA Focus Areas for September 2015 ......................................................................17
Appendix A: SLI Findings and Recommendations Log ................................................19
Appendix B: SLI Issues and Risks Log ..........................................................................22
Appendix C: SLI Risk Classification Index .....................................................................24
Should there be any questions regarding this report, please contact Lillian Dorsett, SLI WA
EMR-ICD10 QA Project Manager at ldorsett@sliglobalsolutions.com. Cell phone is (303)
618-2266.

1. INTRODUCTION
The State of Washington Department of Social and Health Services (DSHS) intends to
replace their current manual and automated processes with a Cerner Electronic Medical
Record (EMR) Solution to support the required ICD-10 standards in three Psychiatric
Hospitals operated by Behavioral Health and Service Integration Administration.
SLI Global Solutions (SLI) is contracted to perform QA services for the project and services
were extended through October 31, 2015.
This report covers the month of August 2015. The report is based on interviews with DSHS
staff, meeting attendance, and review of current project artifacts. All dates reported should
be consistent with the latest schedule distributed on August 25, 2015, EI10 Project
Schedule - Integrated Milestones V21.
This monthly report includes:
 An overall assessment for the project
 A summary of SLI Findings and Recommendations
 A summary of SLI Risks and Issues
 QA focus areas for September 2015
 QA activity during August including meetings attended and documents reviewed
Section 1.2 Testing Status and Updates addresses the status of test planning,
development, and execution of the Millennium application with customization specific to WA
DSHS requirements.
Currently, a significant amount of QA focus is on the testing effort and the defect
management processes. Of particular emphasis are issues of the product’s readiness for
testing, delivery of the local customizations, gap resolutions, data migration, and interfaces
support.
Section 1.3 Exit (or Go Live) Criteria addresses the need for detailed exit criteria from
testing and other implementation prerequisites before the application can be approved prior
to entering production.
Appendix A and B contain the SLI Findings and Recommendations Log and the SLI Risk
and Issues Log.
Appendix C contains the SLI Risk Classification Index used to compute the risk score.
1.1 Overall Assessment for the Period of August 2015
The following is an overall assessment of the last period, updates to the project plan,
updates on testing, and finally updates on project administration and controls.
In August 2015, a critical decision was approved by DSHS to move the Go-Live date out
from October 1 to allow additional time for Cerner to improve system functionality and allow
for adequate testing. This decision also included a change to the implementation plan from
a “big-bang” rollout to all locations at once to a phased approach involving individual
implementation efforts for each location. Additional detail regarding the decision to delay
the Go-Live date and rollout strategy appears in Section 1.8.1

DSHS has received the SOC2 audit materials from Cerner. These are submitted to
WaTech for review and approval. SLI has reduced the probability of risk QA-12 with the
receipt of the SOC2 material. However, SLI recognizes that this risk is not fully mitigated
until WaTech approves the SOC2 submission. More detail is available in Section 1.2.2 and
Section 1.9 below.
Revenue Cycle Phase 1A partial code delivery and installation has taken place. A
demonstration of three elements of the functionality installed was conducted and observed
by SLI on August 25, 2015 (see following section). The remaining code is pushed out into
two additional deliveries – 09/10/15 and another on 10/09/15.
SLI QA has repeatedly requested that Cerner provide a list of what was available (or not
available) for the test efforts (for example registration) and how many scripts passed by the
end of the system test period. More details in the Section 1.2 Testing Status and Updates.
SLI has reviewed the Issues Log maintained in the Cerner repository.
The state has requested demos and screen shots of the application to support both testing
script development and the development of the training materials. SLI has recommended
that a complete inventory of all, final as built, screens be developed as soon as possible for
final review. We have formalized this recommendation as Recommendation Test-08.
Concern has been expressed that the inventory of Test Scripts contained in TestTracker is
incomplete, not reflecting Test Scripts that have not yet been developed. This is reportedly
due to defects in TestTracker causing issues with accuracy and maintenance of the scripts.
Once the issues are resolved, SLI recommends that the project management work with
Cerner to establish a comprehensive inventory of all Test Scripts and that these tests be
recorded in TestTracker. Verification that TestTracker contains Test Scripts that are unique
for individual WA implementation sites must be included in this future process.
A project risk was identified regarding the software licensing agreement for the Alpha II
software license during the August 8 management meeting. Negotiations on the content of
the licensing agreement took place, and IV&V was advised that the issue was resolved on
August 20. The project risk has been closed.
1.2 SLI Onsite Visit

During the week of August 24, SLI IV&V attended two days of IT3 Revenue Cycle and
testing meetings at Western State Hospital. These were supposed to include Cerner
presentation of the end-to-end workflow for Active Treatment including scheduling,
documentation, charging and miscellaneous other features. However, the demo was not
full end-to-end as it is reported that the creation of the treatment plan was left out. (Another
end-to-end demonstration has been requested.) This was followed by sessions for Patient
Accounting, Registration and Scheduling and Charge services, a Treatment Mall
Walkthrough, and an Integration Test Script Development session. SLI findings and
recommendations from these meetings are incorporated in Section 1.10.
SLI observes that the inability to readily distinguish System Testing activities and results
from Integration Testing activities has created a great deal of confusion with the DSHS
staff and is undermining confidence in the quality and completeness of the software as
installed. During the August 21 Project Executive Oversight and August 24 DSHS
Executive Leadership (with the Secretary) meetings, the BHSIA CIO clarified that the

project would return to two distinct phases of testing, System and Integration. She also
advises that the next series of test events, scheduled for the week of September 28, is the
first true Integration Testing and that the release associated with this test event will be
referred to as IT-A (removing the numerical indicators, such as IT-3). Additional Integration
Test rounds will take place in October and November.
Revenue Cycle Phase 1A code delivery and installation has taken place. A demonstration
of three elements of the functionality installed was conducted and observed by SLI on
August 25, 2015. The areas presented were Legal Authority, Regional Support Network
(RSN), and Patient Leave. During the demonstration there was significant discussion
between DSHS and Cerner staff on each of the three elements. In each case, DSHS staff
identified areas in which the demonstrated functionality was insufficient to meet DSHS
requirements. It was not clear during these discussions how Cerner proposed to address
these items.
SLI observed that the development of test scripts (and the testing effort in general) is
adversely affected since there are still blocked scripts – scripts that cannot be run or written
because the system functionality has not been delivered.
1.3 Primary Risks
1.3.1 Risk to project schedule and scope due to late milestones (QA-
08)
A growing concern for the last several months has been the number of late milestones. As
of the V21 Integrated Milestones workbook, 36 milestones are identified that are both late
and less than 90% complete. This number represents an increase from the V19 milestone
document. SLI continues to advise that late milestones should be closely monitored by
DSHS and accounted for during discussions intended to establish a new go-live date. The
project team understands the risks between now and implementation and is monitoring
these carefully. However, in our experience with similar projects, it becomes progressively
more difficult to achieve the intended go-live date as the backlog of late tasks and
milestones continues.
The spreadsheet: EI10 Go Live Date Critical Items V20 – latest version was prepared for
and delivered for the 8/11 weekly project management meeting.
1.3.2 Application is not Production Quality (QA-11)
The application is not production quality as defined by meeting the state’s requirements (the
Exit Criteria) and/or contains significant defects. Due to undocumented changes in the
testing process (detail in Section 1.6), and delays in the delivery of critical code that are
adversely impacting the delivery of Test Scripts, SLI believes it is likely the application will
not be satisfactory when delivered to DSHS. However, since the go-live timeframe is
extended by four months, the quality of the product will continue to be a key focus area as
part of the on-going monitoring activity through the end of the QA engagement – October
31, 2015.
1.3.3 Risk of project termination due to inadequate SOC2 audit (QA-12)
Without WaTech approval of the findings of a formal SOC2 audit, the application will not be
certified to retain or manipulate WaTech live patient data. Cerner provided a copy of a
SOC3 audit report. By definition, the SOC3 audit does not contain the level of rigor that is

mandated for a SOC2 report. Further meetings on this subject took place during August
2015. As of the August 18 DSHS / Cerner Management Status Meeting, Cerner delivered
SOC2 audit documentation to DSHS. This documentation is currently under review by
WaTech.
SLI will continue to monitor this risk until WaTech approval of the SOC2 audit has been
received.
1.4 Project Progress Highlights

SLI is unaware of any contracted deliverables that were delivered or approved during
August 2015.
SLI recognizes that special efforts are being made to finalize the design of the Future State
revenue cycle workflows and interfaces, and gain eventual signoff of the design documents.
Since the “correctness” of the test scripts are based on the documented design, it is critical
that these tasks be completed to ensure that the Cerner team can finalize adequate testing
of the implementation.
In the following section, the project plan, technical and infrastructure issues, test status,
risks and issues are discussed, and Exit (or Go Live) Criteria are reviewed.
1.5 Project Plan Update
WA DSHS has determined that it will not be possible for the application to be implemented
on October 1, 2015 due to issues with the application, late deliverables, and loss of
continuity of the project to the documented project plan. Meetings are taking place to
determine a realistic go-live date based upon the current understanding of the status of the
project.
A teleconference was held between IV&V and WA DSHS management regarding status
and issues related to the testing effort on August 5. The consensus on this call is that the
entire concept of System Testing followed by Integration Testing (which has been the
Cerner proposed testing approach since the inception of the project) was abandoned by
Cerner. The project testing activities are unrelated to what has been defined in the project
plans since the inception of the project. However, DSHS is going back to the flow of
System Test followed by Integration Test. More details on this issue are outlined in Section
1.6.
The latest schedule distributed is titled: EI10 Project Schedule - Integrated Milestones V21.
The scope of the clinical project remains unchanged in that schedule, but they are no
longer valid and there is no reason to continue to report them. New dates and schedule will
be reported upon once the information is provided to SLI for review.
The CIO has informed SLI of the new go-live dates - the East side go-live begins on
February 1, 2016 and the West side of the state begins on April 1, 2016.
The scheduled events for September will change significantly after agreement has been
reached on the new project schedule and implementation sequence.
However, events scheduled for September, based upon the EI10 Project Schedule -
Integrated Milestones V21.xlsx include:

Event
Future State System Diagram approved
Revise Requirements Traceability Matrix (RTM) with JIRA Reporting
Begin review of agreed upon metrics & approval criteria
Go/No-go Decision
Build Training Schedule
Event 14b End User Training-Revenue Cycle (Local) but Cerner on-site
Event 14b End User Training-Clinical (Local) but Cerner on-site
DSHS SYSTEM LOCKDOWN
The project is in red status, and will not be delivered according to the original schedule;
Cerner is unable to assign necessary resources to the project due to “conflicts” with the
resource needs of other Cerner projects. SLI has changed Risk #QA-03 into an issue, as
the risk has been realized and there is a documented adverse impact to the project
schedule.
1.6 Technical and Infrastructure Issues

SLI has converted Risk QA-03 (IF the Cerner project staffing is not adequate to meet the
delivery milestones, THEN the project’s fixed schedule may slip.) to an issue, as the risk
has been realized. Although the project is now in red status, and will not be delivered on
schedule, Cerner delayed demonstrations of critical functionality for additional weeks due to
a specific named resource not being available. The resource conflict was discussed during
the August 4 DSHS / Cerner Project Management Status Meeting. It was implied during
the meeting that this resource had been assigned to a higher-priority Cerner project. The
CIO has since reported that Cerner has confirmed this type of resource reassignment
without prior notification and approval will not occur going forward.
All testing continues to be done with fake patient data (as one would expect). No approval
to send real patient data across the network will be given by WaTech prior to approval of
the SOC2 audit. Cerner advised during the August 25 status meeting that the SOC2 audit
has been submitted to DSHS. The CIO confirmed the information has been passed on to
WaTech.
1.7 Testing Status and Updates
System and Integration Testing are behind for a variety of reasons. The team is pursuing
an aggressive list of action items that need to be resolved to get the project on track for the
Go Live target date. Staffing will be a critical issue for freeing people up for testing, training
and Go Live support. The team and project sponsors are working on creative ways to
alleviate chronic staffing issues, including removing overtime restrictions. No additional
information is available to SLI at this time.

A teleconference was held between IV&V and WA DSHS management regarding status
and issues related to the testing effort. The consensus on this call is that the entire concept
of System Testing followed by Integration Testing (which was the Cerner proposed testing
approach since the inception of the project) was redirected by Cerner – Integration Testing
was pushed before System. The following items reflect the status of testing:
 The current approach to testing activities and supporting issues indicate that, until
recently, there was no discrimination between System Test and Integration Test.
However, the DSHS team has taken control – System Test is underway; Integration
Test now planned for end of September.
 There are both defects and a lack of available functionality at the system level that
are preventing the build-out and execution of Integration Test scripts.
 There is no defined System Test exit criteria that must be met before full Integration
Test (end-to-end) testing can begin. DSHS position is to run the System Tests and
prioritize the issues that failed.
 The current plan, as understood by the WA stakeholders is that there is now to be
an end-to-end testing using the following patient models. This testing is for
Integration Test:
o Patient 1 WSH Civil
o Patient 2 ESH Forensic
o Patient 3 WSH Forensic
o Patient 4 ESH Civil
o Patient 5 CSTC Under 9
o Patient 6 CSTC Over 9
 The testing schedule, at best, has been ad-hoc and poorly planned. SLI expects to
see improvement in the planning, sequencing and other details related to testing
activities in the updated project schedule when it is released
As of August 27, 2015, there are 220 open issues, some of which were initially opened in
May of 2015.
1.7.1 RTM Updates
The most recent RTM update was posted on August 5, 2015. There are strikeouts and
other updates to the requirement descripts which are not explained in the document. SLI
understands that Cerner and the state are continuing work on the RTM, and SLI offers the
following recommendations for consideration during the RTM update and review processes:
 The state requires Cerner to update the RTM to include all agreed upon
requirements including the identified gaps and nonfunctional requirements such as
interface testing. These are the set of requirements that then would be traced to the
specific test script(s)
 SLI recommends 100% coverage of all requirements in the RTM are fully tested and
can map to the specific test demonstrating implementation of the requirement.
Further, this metric – the percent of RTM items that are met – is routinely used in

the industry as a discrete tool to determine readiness of the system for production
implementation.
 As Cerner delivers test scripts that verify the implementation of the function that
addresses each specific requirement, the test script id. is entered into the RTM
establishing a linkage between the requirement and the test script identifier.
 Add provisions for associating Change Requests and any other applicable
requirement updates to the RTM. Corrections and updates that materially affect
requirements need to be associated with the Change Request that drove the
update.
The configuration and development of the application is incomplete. Test script
development must be frequently suspended because the system functionality in support of
the business process being tested is unavailable. This is having an adverse impact on both
the ability to create test scripts on schedule and on the quality of the test scripts.
1.7.2 Cerner Testing Tools

In May 2015, SLI made recommendations related to tracking, capturing, and reporting on
metrics on a weekly basis such that trends in the testing effort could be more readily
identified. SLI participated in a teleconference with the BHSIA CIO and the DSHS test lead
on August 5 to discuss all aspects of testing.
During the August18 Project Management Status Meeting, Cerner acknowledged that data
entry into the TestTracker tool has not been taking place reliably, and advised they were
addressing the issue. DSHS Project Manager sent email out to all project team members
on August 26 that all issues should be entered in the TestTracker tool. They have been
complying.
1.8 Exit Criteria (Go Live Criteria)

Following industry standards and best practices, the exit criteria to be considered for Go
Live are included in the Test Plan. SLI recommends a defined process including clearly
documented quantitative and qualitative criteria. SLI has provided a recommended format
and contents for incorporation into a formal Exit Criteria checklist as a baseline to determine
readiness for implementation.
The BHSIA CIO is taking the lead in the definition of the Exit (Go Live) Criteria by working
with the Executive and the Project Oversight boards. Criteria for confirmation that the
products are ready for implementation, using verifiable measurements are anticipated to
improve the probability of a successful implementation effort, and are a necessity for DSHS.
SLI will continue to monitor the DSHS review activities associated with the Exit Criteria
checklist and will develop a series of recommendations for periodic evaluation of progress
being made to address the items appearing on the checklist. CIO reported that she
reviewed close to 30 potential criteria with Executive Governance Board in August. They
are providing feedback to be reviewed in next EGB meeting (9/9).
1.9 Project Administration and Controls

Since the start of the QA contract, SLI has tracked the development of the various project
management processes:
 Schedule Management
 Communications Management
 Risk, Issue, and Decision Management
 Requirements Management
In each case, WA EMR-ICD10 project management working in conjunction with Cerner
project management have developed processes that are industry standard and generally
follow best practices. At this point in the project, these processes are fairly well established
but continue to receive incremental improvements, such as the project schedule.
The following details SLI observations made during August in the implementation of various
project management processes. This section focus is on the process rather than the current
outputs of the process.
1.9.1 Project Schedule

In late July, a critical decision was made by DSHS to move the Go-Live date out from
October 1 to allow additional time for Cerner to improve system functionality and allow for
adequate testing. This decision also included a change to the implementation plan from a
“big-bang” rollout to all locations at once to a phased approach involving individual
implementation efforts for each location. The phased approach would include OFR and the
east side of the state (Eastern State Hospital, Yakima Valley School and Lakeland Village)
implementing as the first wave, with the second wave consisting of the two additional state
hospitals (Child Study and Treatment Center and Western State Hospital). The two
additional RHCs (Rainier and Fircrest Schools) will implement with the second wave.
This decision to delay the implementation has been vetted by the DSHS project team leads,
endorsed by the Executive Governance Board, and approved by the Executive Sponsor,
the BHSIA Assistant Secretary and the DSHS Secretary.
The new Go-Live dates were announced broadly on August 31st. Cerner scheduled a
meeting on 8/06/15 to discuss the Revenue Track/Cycle impact of the delays. The Interim
Assistant Secretary is leading the effort to plan for interim handling of billing between
October 1, 2015 and the new Go-Live.
The latest Cerner schedule was distributed on August 24, EI10 Project Schedule -
Integrated Milestones V21. This schedule does not reflect any proposed changes to the
Go-Live date, as the revised Go-Live date and phased rollout strategy were not formally
announced.
The project plan is discussed during the weekly management coordination calls and
includes discussions of milestones and phase gates. Milestones are used to identify the
coordination points between state, SLI, and Cerner activities.
1.9.2 Communications Plan

The meetings and reports established in the Communications Plan are being implemented.
A project team roster has been updated and distributed.
The CIO reports that efforts are underway to create a more comprehensive
Communications Plan to support the broad awareness and adoption at the facilities –
outside of project team communications.
1.9.3 Risk, Decision, and Parking Lot Tracker
These logs are updated on a routine basis and are maintained in a “current” state.
1.9.4 Requirements Management
Requirements management and the traceability it enables helps to ensure the state that the
functions requested by the state are delivered by the vendor in a quality fashion. The
Testing Track lead is the primary DSHS owner of the RTM in conjunction with the Cerner
project manager.
The process for creating and updating the RTM has been implemented, however updates
to the RTM have not been taking place. This is a deviation from an approved process.
An update on the status of the RTM is provided in Section 1.6.1.
1.10 Risk and Issues

1.10.1 Risks
A risk is an area of concern that may occur and have a negative impact on any one or all of
the primary project constraints or objectives – time, scope, cost, quality, and resources.
Primary risks to the project are articulated in Section 1.2. A risk is usually triggered by an
event, meaning that a set of defined conditions, when present, cause the risk to occur. Risk
Management differs from Issue Management. Risks are events that have not yet occurred
and can be planned for (i.e., mitigation and/or contingency plans).
Primary project risks are presented in detail in Section 1.2. A list of open project risks
appears in Appendix B
For this report, SLI has opened a new risk, QA-10 based upon concerns expressed by the
BHSIA CIO that the project schedule may be compromised due to project-critical state staff
leaving state employment for private sector opportunities. This concern is based upon
recent staff departures.
For this report, SLI has reduced the risk exposure value associated with Primary Risk QA-
12 (Section 1.2.2) as Cerner has provided a SOC2 audit document for WaTech approval.
The risk has not been closed, as there remains the potential for the project to terminate if
the submitted SOC2 is not approved.
The description and status of each identified risk is included in the issues and risks log in
Appendix B.
SLI’s method for assigning risk values is detailed in Appendix C: SLI Risk Classification
Index.

1.10.1.1 Risk associated with an Increasing Count of Missed Project Plan

Milestones (QA-08)
As of the V21 Integrated Milestones workbook, 36 milestones are reported late and are less
than 90% complete. While this number is only slighter larger than the last report (33), it is
still significant and should be closely monitored by DSHS. If the trend of missed milestones
is maintained, the project will not implement on schedule, may not implement with adequate
functionality, or both. Milestone dates will be adjusted to reflect the new Go-Live schedule.
1.10.1.2 Risk associated with Cerner Staffing Issues (QA-03)
Cerner is ultimately responsible for providing adequate staffing to meet their contractual
requirements. The state’s responsibility is to articulate and document the requirements so
that scope and delivery expectations are unambiguously clear.
SLI has converted Risk QA-03 to an issue, as the risk has been realized. Although the
project is now in red status, and will not be delivered on schedule, Cerner delayed
demonstrations of critical functionality for additional weeks due to a specific named
resource not being available. The CIO reports that DSHS had discussions about this with
Cerner and this type of unannounced/unapproved resource change will not occur again.
However, the issue will remain until determined to be mitigated.
1.10.1.3 Risk Associated with Extensive Tailoring of the Cerner Systems per
Facility (QA-06)
SLI has increased the impact of this risk to “High”. During the track sessions, the state and
Cerner have been conducting analysis and documenting processes and workflows at the
three facilities. Although the Cerner products are capable of being tailored to the differences
between the facilities, the quantity of differences between the facilities and the complexity of
these differences are greater than initially anticipated.
This risk will need to be managed more aggressively through a combination of maximizing
commonalities by the identification of a “best practice” facility approach, and standardizing
the other facilities on that approach and simplifying the tailoring that is necessary within the
Cerner product.
SLI will continue to monitor this risk as the current state, future state, and gaps are defined.
1.10.1.4 Risk Associated with Timeliness of Approvals (QA-07)
There are risks associated with any delay in the approval of the workflows and other design
artifacts. Approval delays can cause development delays. If the schedule is fixed then the
testing timeframe is shortened. Conversely, if the testing timeframe were fixed then the
project deliverable would need to be pushed out. Either avenue has risks and costs.
Upon review of the Integrated Milestone V21 workbook, no approval delays are observed.
At this point, the project management is proceeding under the assumptions that approvals
will be forthcoming. However, the need for multiple reviews by the state and related
multiple fixes to the deliverables by Cerner (such as workflows) before deliverables can be
approved affects the approval process.
SLI will continue to monitor this risk as the current state, future state, and gaps are defined.
1.10.1.5 Risk Associated with the Departures of DSHS Personnel (QA-10)

The CIO has reported concern that there is a risk to the project due to the unanticipated
departure of DSHS personnel for private sector employment. SLI has recorded this as a
risk. The CIO’s concern was due to the high desirability of Cerner EHR skills in the
current marketplace, exacerbated by Cerner’s winning the Department of Defense contract.
In addition, the latest Revenue Cycle Business Analyst from OFR left state employment and
is only now being replaced.
1.10.2 Issues
An issue is a condition or event that has already occurred (a project risk that has occurred
by definition, becomes a project issue).
Issues are tracked and included in Appendix B: SLI Issues and Risks Log.
1.10.2.1 Issue Associated with Milestone Completion
SLI has identified an issue related to the growing number of project milestones that are not
being completed as outlined in the project schedule. As of the V21 Integrated Milestones
workbook, 36 milestones are reported late, and are less than 90% complete. This is
significant and should be revisited by DSHS when the new Go-Live dates and updated
project schedule are available.
1.10.2.2 Issue Associated with Data Conversion
At some point, both a representative (if not complete) set of migrated data must be loaded
to the Master Patient Index and the defined interfaces operational by IT2 to fully support
end-to-end Integrated Testing using actual WA DSHS converted data. This cannot happen
without WaTech approval of the SOC2 audit.
1.10.2.3 Issue Associated with Interface Definition Status
The EMR-ICD10 RFP states that the system will need to interface with a number of other
systems although this has yet to be confirmed during the detailed analysis that is scheduled
for the coming months. The Interface Management Plan was delivered and approved in
early June 2015.
The PC cash interface and ACES interface are in test.
For the ACES interface, two custom CCL files have been identified, with the specifications
provided to Cerner. A determination as to final status of both PC Cash and ACES interface
issues was expected on June 10, 2015, but remain ongoing. These interfaces must be
available to support full End-to-End Integration Testing.
Interfaces are scheduled to be ready for integrated testing at the start of IT- A, with the
exception of WSH Lab. Some preliminary testing of the interfaces should be completed
before the start of the integration testing, although testing with real patient data cannot take
place until approval of the SOC2 audit document.
Point to point testing of interfaces should be performed prior to Integration Testing, just as
all processes within Cerner should be tested in System Testing – and then everything
pulled together for Integration Testing.
Activities related to Interfaces are discussed during the weekly status calls.

1.11 Findings and Recommendations

Ten new findings were identified in August 2015.
See Appendix A: SLI Findings and Recommendations Log for the current list and status.
2. QA SPECIFIC STATUS
Status of QA activities as of this reporting period is as follows.
2.1 Schedule Management
As of the V21 Integrated Milestones workbook, 36 project milestones are reported late and
are less than 90% complete.
2.2 Scope Management
There are no QA scope issues or schedule changes.
2.3 Resource Management
There are no QA resource issues or changes for August 2015.
2.4 Communication Management
QA continues to attend the routine and scheduled project executive governance meetings.
2.5 Issue and Risk Management
While the project tracks Issues and Risks, those identified by SLI are tracked in the risk and
issues log as attached in Appendix B.
3. QA ACTIVITY REPORT
3.1 Record of Project Meetings and Activities
SLI staff supported and participated in the following project activities:
Date Meeting Subject Attendees Purpose

DSHS / IV&V Touchpoint Sara Tripp, Kenneth Merrell, Lillian
8/5/2015 Status of testing activities
Meeting on Testing Dorsett, Jon Kanas
DSHS / IV&V Touchpoint
Sara Tripp, Kenneth Merrell, Lillian Logistics for onsite IV&V
8/17/2015 Meeting on IT3 Event
Dorsett, Jon Kanas visit, week of 8/24
onsite
DSHS / IV&V Logistics
Final logistics for onsite
8/19/2015 Meeting on IT3 Event Kenneth Merrell, Jon Kanas
IV&V visit, week of 8/24
onsite
EMR-ICD10 Project
8/21/2015 Executive Oversight DSHS / IV&V Project status
Teleconference

Date Meeting Subject Attendees Purpose

Cerner Active Treatment Presentation on system
8/24/2015 DSHS / IV&V / Cerner
presentation and debrief functionality
Presentation and
Test Script Materials and
8/25/2015 DSHS / IV&V / Cerner materials for test script
Development orientation
development
Patient Accounting Test Test script development
8/25/2015 DSHS / IV&V / Cerner
Script Development workshop
Presentation on system
8/25/2015 Phase 1A Legal Authority DSHS / IV&V / Cerner
functionality
8/25/2015 Phase 1A Patient Leave DSHS / IV&V / Cerner
functionality
8/25/2015 Phase 1A RSN DSHS / IV&V / Cerner
functionality
Discuss management
Weekly Management meeting Cerner, SLI, and State staff
issues.
Weekly Project Management BHSIA CIO and SLI Project Discuss current status
Call Touchpoint Manager and upcoming activities
3.2 Report and Deliverable Review Activities

SLI reviewed the following documents during the reporting period:
Document
150804_PM Status Meeting Notes.docx
Decision_Log_150814.xlsx
DSHS Rev Cycle IT3 Agenda - Week of 08-24-15 V5.pdf
DSHS Rev Cycle IT3 Agenda - Week of 08-24-15 V6.pdf
DSHS Rev Cycle IT3 Agenda - Week of 08-24-15 V9.xlsx
EHR ICD10 Executive Governance Board update July 8 2015.pptx
EHR ICD10 Executive Governance Board update August 12 2015.pptx
EHR ICD10 Project Executive Oversight update August 21 2015.pptx
EI10 Go Live Date Critical Items V20.xlsx
EI10 Interface Tracker 2015-08-05.xlsx
EI10 Project Schedule - Integrated Milestones V20.xlsx
EI10 Project Schedule - Integrated Milestones V21.xlsx
EI10 Solution Tracker 2015-08-05.xlsx

Document
EI10 VC SA Status Tracking 2015-08-05.xlsx
EMR-ICD10 Dashboard 2015-06 SRT.pdf
EMR-ICD10 Dashboard 2015-07 SRT.pdf
Parking_Lot_150814.xlsx
PM Status Meeting Notes 2015-08-11.docx
PM Status Meeting Notes 2015-08-19.pdf
PM Status Meeting Notes 2015-08-25.docx
Risk_Log_150810.xlsx
SLI Monthly Report for the WA EMR-ICD10 Project for July 2015_DRAFT
with comments SRT Aug 4.docx
SLI Monthly Report for the WA EMR-ICD10 Project for July
2015_FINAL.pdf
SOC 3 Report - 2015 (Cerner).pdf
WSH Civil Ginger Template 2015-07-06.xlsx
4. QA FOCUS AREAS FOR SEPTEMBER 2015

Focus areas for the QA team over the next month(s) include:
 Monitor project schedule, WBS, Project Plan for impacts of updated Go-Live date
 Monitoring the DSHS development activities associated with the Exit Criteria
checklists for System Test/Integration Test and Implementation readiness.
 Monitor the feasibility of any changes to the implementation dates or methods
 Monitor state IT support for the project. The state is responsible for integrating the
legacy systems (Vista/Cache) that will support specific hospital functions not
included in the Cerner solution, such as labs. QA will monitor the IT team and their
ability to work out a technical strategy for the integration and working with Cerner to
implement the defined strategy.
 Ongoing monitoring of the testing processes, materials and test results:
o Verification that TestTracker reflects the entire count and status of each Test
Script associated with Integration Testing – this will be done once the
defect(s) within TestTracker are fixed.
o Review and validate test scripts and materials for September 28 scheduled
testing event.
o Validate that updates to the RTM and WA state workflow trackers are being
posted appropriately
o Verify that the weekly test status is sufficient to enable the DSHS staff to
gain insight into the progress of the test effort

 Monitoring the defect management process

o Documenting
o Triage – Core, local customization, interface, or change request
o State input on prioritization and defect resolution scheduling
o Tracking and reporting

APPENDIX A: SLI FINDINGS AND RECOMMENDATIONS LOG

The following table contains all open findings and recommendations from the SLI QA team as of the date of this report:
ID Date Status Finding Recommendations Project Response

SLI recommends that the project
management work with BHSIA to prepare
a cost and resource impact to the state
including additional costs associated with
There is benefit to the
project staffing costs and continued
development of a contingency
PM-03 3/10/2015 Open operation of legacy systems over the 12-
plan for continued operation of
24 month extension. In addition, Cerner
the legacy systems.
should be encouraged to deliver their
impact analysis and strategy for
remediation as soon as possible to provide
space for the state to react.
SLI recommends 100% coverage of all
requirements in the RTM are fully tested
The RTM is not being updated and can map to the specific test
PM-04 4/8/2015 Open as the testing activities are demonstrating implementation of the
taking place requirement. The RTM should be
updated on a regular basis to reflect the
status of testing as applicable.
There is not an accurate count SLI recommends that the project
of test scripts which have yet management work with Cerner to
to be completed. Of specific establish a complete and definitive
PM-05 6/30/2015 Open concern are test scripts which inventory of test scripts necessary for
exercise requirements which Integration Testing, and that the presence
are unique for the individual of these scripts be recorded in the
WA sites TestTracker tool.


The Integrated Milestones
workbook V18 indicates that SLI recommends the completion and
the delivery and approval of submission and approval of the Test Plan The testing plan was
Open the Test Plan (Deliverable 2.7) as soon as possible. The value of the test delivered and
PM-06 6/30/2015
Closed - remains outstanding. Given plan is significantly dimiinished if the approved in late
that testing is underway, this is execution of testing takes place without June.
a critical deliverable to be guidance of a formal test plan
finalized.
Concern has been expressed
SLI recommends that the project
that the inventory of Test
management work with Cerner to
Scripts contained in
establish a comprehensive inventory of all
Test-01 8/3/2015 Open TestTracker is incomplete, not
Test Scripts necessary for Integration
reflecting Test Scripts that have
Testing, and that these tests be recorded
not been developed due to the
in TestTracker
lack of as-built screens.
SLI strongly recommends that the
Executive Committee continue supporting
the efforts to define the Exit Criteria for
Exit criteria for System Test is
Test-02 8/3/2015 Open Go Live to ensure that the state and
ambiguous
Cerner are in agreement as to what
constitutes the minimum criteria for Go
Live.
SLI recommends 100% coverage of all
The test coverage of WA requirements in the RTM are fully tested
Test-03 8/3/2015 Open requirements identified in the and can map to the specific test
RTM is not clearly defined. demonstrating implementation of the
requirement.
SLI recommends that prerequisites be
Test Script workbooks do not
clearly articulated in the body of the test
Test-05 8/25/2015 Open clearly articulate prerequisites
script, either as header information or as
for the test
individual test steps within the test script
Ability to identify issues which SLI recommends defaulting this type of
Test-06 8/25/2015 Open constitute "blockers" needs to issue to "Critical" during recording in the
be fully defined. Issue Log


SLI recommends that a contingency plan
DSHS is concerned that the be developed to ensure that the most
scope of the planned testing critical business processes are subject to
Test-07 8/25/2015 Open effort is too great to be testing, and DSHS establish a minimum
accomplished during the level of System Testing which must be
September testing event. completed before Integration Testing may
proceed
The lack of visibility into the
SLI has recommended that a complete
product screens is adversely
inventory of all, final as built, screens be
Test-08 8/25/2015 Open affecting the ability of DSHS to
developed as soon as possible and
produce Test Scripts and
provided to DSHS for final review
training materials.
There is a risk that testing will SLI recommends a careful evaluation of
be incomplete or inadequate the time allocated for the completion of
Test-09 8/25/2015 Open due to the limited amount of the System and Integration test efforts
time available for completion during the development of an updated
of the testing efforts. project schedule.

APPENDIX B: SLI ISSUES AND RISKS LOG
ID Date Status Type Description Probability Impact
IF the State-based project staff do not remain allocated to the

QA-02 8/11/2014 Monitor Issue project at the current and agreed to percent of time, THEN the
project’s fixed schedule may slip.
IF the Cerner project staffing is not adequate to meet the delivery
QA-03 8/11/2014 Monitor Issue
milestones, THEN the project’s fixed schedule may slip.
IF any significant data conversion is required THEN the project’s
QA-04 8/11/2014 Monitor Issue fixed schedule may slip and/or the quality and effectiveness of
the application may be impacted.
IF there are critical interfaces that are required and these require
QA-05 8/11/2014 Monitor Issue significant development to implement THEN the project’s fixed
schedule may be impacted.
IF extensive facility tailoring is required and more-than-expected
Possible to
QA-06 8/11/2014 Monitor Risk development time is required to implement THEN the project’s High
Occur
fixed schedule may be impacted.
IF workflows are not approved in a timely fashion THEN the
development schedule could be impacted and a fixed schedule, a
Possible to
QA-07 8/11/2014 Monitor Risk longer development schedule would tend to shorten the Medium
Occur
timeframe set aside for testing and impact the entire project
schedule.
If the trend of missed milestones is maintained, the project will
Highly
QA-08 6/30/2015 Monitor Risk not implement on schedule, may not implement with adequate High
Probable
functionality, or both.
Loss of momentum due to departure of DSHS personnel to Possible to
QA-10 8/4/2015 Monitor Risk Medium
private sector Occur

ID Date Status Type Description Probability Impact
There is the risk that the application is not production quality as

defined by meeting the state’s requirements (the Exit Criteria)
and/or contains significant defects. This concern is increasing
with the additional delay in the delivery of Revenue Cycle code. Highly
QA-11 8/4/2015 Monitor Risk High
Finally, the registration and scheduling components are not Probable
scheduled to be delivered until August. Therefore, full end-to-
end testing cannot be completed nor can training content be
developed until all relevant components have been delivered
Risk of project termination due to WATECH disapproval of SOC2 Possible to
QA-12 8/4/2015 Monitor Risk Critical
audit. Occur

APPENDIX C: SLI RISK CLASSIFICATION INDEX

SLI has used the following risk scale to assign a risk value to each identified risk. (As the
project defines their own, scoring procedures these metrics will be adjusted to be consistent
with the project’s scoring method.) Of course, the state has the final say on the scoring of
these risks. Each risk is scored for the likelihood of occurrence and the impact to the project
using the risk probability and risk impact definitions below in Figure 1, Risk Classification
Matrix.
Figure 1: Risk Classification Matrix
After each potential risk is recorded and assigned a probability of occurrence, it is then
assessed for project impact. The probability rating and the impact rating are combined to
provide an impact prioritization of potential risks.


SLI Monthly Report - WA EMR-ICD10 Project - August 2015 - FINAL

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SLI Monthly Report - WA EMR-ICD10 Project - August 2015 - FINAL

Uploaded by

Copyright:

Available Formats

WA Department of Social and Health Services

Washington EMR-ICD10 Project

SLI Monthly Report for August 2015

SLI Global Solutions, LLC

Submitted: Draft September 2, 2015

SLI-WA-EMR-ICD10 August 2015

SLI-WA-EMR-ICD10 August 2015

3. QA Activity Report ....................................................................................................15

SLI-WA-EMR-ICD10 August 2015

SLI-WA-EMR-ICD10 August 2015

1.2 SLI Onsite Visit

SLI-WA-EMR-ICD10 August 2015

SLI-WA-EMR-ICD10 August 2015

1.4 Project Progress Highlights

SLI-WA-EMR-ICD10 August 2015

1.6 Technical and Infrastructure Issues

SLI-WA-EMR-ICD10 August 2015

SLI-WA-EMR-ICD10 August 2015

1.7.2 Cerner Testing Tools

1.8 Exit Criteria (Go Live Criteria)

SLI-WA-EMR-ICD10 August 2015

1.9.1 Project Schedule

1.9.2 Communications Plan

SLI-WA-EMR-ICD10 August 2015

1.10 Risk and Issues

SLI-WA-EMR-ICD10 August 2015

1.10.1.1 Risk associated with an Increasing Count of Missed Project Plan

SLI-WA-EMR-ICD10 August 2015

SLI-WA-EMR-ICD10 August 2015

1.11 Findings and Recommendations

Date Meeting Subject Attendees Purpose

SLI-WA-EMR-ICD10 August 2015

Date Meeting Subject Attendees Purpose

3.2 Report and Deliverable Review Activities

SLI-WA-EMR-ICD10 August 2015

4. QA FOCUS AREAS FOR SEPTEMBER 2015

SLI-WA-EMR-ICD10 August 2015

 Monitoring the defect management process

SLI-WA-EMR-ICD10 August 2015

APPENDIX A: SLI FINDINGS AND RECOMMENDATIONS LOG

ID Date Status Finding Recommendations Project Response

SLI-WA-EMR-ICD10 August 2015

ID Date Status Finding Recommendations Project Response

SLI-WA-EMR-ICD10 August 2015

ID Date Status Finding Recommendations Project Response

SLI-WA-EMR-ICD10 August 2015

APPENDIX B: SLI ISSUES AND RISKS LOG

ID Date Status Type Description Probability Impact

IF the State-based project staff do not remain allocated to the

SLI-WA-EMR-ICD10 August 2015

ID Date Status Type Description Probability Impact

There is the risk that the application is not production quality as

SLI-WA-EMR-ICD10 August 2015

APPENDIX C: SLI RISK CLASSIFICATION INDEX

Figure 1: Risk Classification Matrix

SLI-WA-EMR-ICD10 August 2015

You might also like