PFML Monthly QA Status Report As of Nov 10 2019 v1.1

Washington Employment Security Department (ESD)
Paid Family and Medical Leave

(PFML) Technology
Implementation Project
Monthly Quality Assurance (QA)

Status Report
V1.1
Reporting Period: October 11, 2019 – November 10, 2019

Report Date: November 13, 2019
Contract #: K5642
Engagement #: P0047671
Public Consulting Group, Inc. Washington State Employment Security Department (ESD)
Monthly Status Report Paid Family Medical Leave (PFML) Tech nology Implementation Project
Date: November 13, 2019
Ms. Carla Reyes, PFML Director

Washington State Employment Security Department (ESD)
Paid Family Medical Leave (PFML) Technology Implementation Project
RE: ESD PFML Project Quality Assurance Report – as of November 10, 2019 (Contract #K5642)
Dear Ms. Reyes:

PCG’s assessment of the PFML Project is based on the professional experience and judgment of its
consulting team. This report was prepared independent of project participants and stakeholders, other
than inclusion of responses to QA findings.
Thank you for the opportunity to provide QA services to this project. Please let me know if you have any
questions or comments.
Sincerely,
Dana McLean
Dana McLean
Engagement Manager
Public Consulting Group Technology Consulting
Public Consulting Group, Inc. 1

Document History
Version Date Brief Description of Modifications
1.0 11/13/2019 Initial submission of the Monthly QA Status Report
1.1 11/19/2019 Incorporation of edits
Document Reviewer(s)
Name Title
Linda Kleingartner PFML Project Manager
Carla Reyes PFML Director
Lisa Kissler PFML Technology Manager
Dana Josephs PFML Contracts Manager
Document Receiver(s)
Name Title Signature Date
Carla Reyes PFML Director

TABLE OF CONTENTS
1. EXECUTIVE SUMMARY ............................................................................................................ 1
2. REFERENCE DOCUMENTS ...................................................................................................... 6
3. GLOSSARY ............................................................................................................................... 6
4. SCOPE ...................................................................................................................................... 7
5. APPROACH .............................................................................................................................. 7
6. ASSESSMENT .......................................................................................................................... 8
6.1. Project Integration Management .............................................................................................. 8
6.2. Project Scope Management..................................................................................................... 8
6.3. Project Schedule Management ................................................................................................ 9
6.4. Project Cost Management ..................................................................................................... 10
6.5. Project Quality Management ................................................................................................. 10
6.6. Project Human Resource Management................................................................................. 11
6.7. Project Communications Management .................................................................................. 12
6.8. Project Risk Management ...................................................................................................... 13
6.9. Project Procurement Management ........................................................................................ 13
6.10. Project Stakeholder Management ......................................................................................... 14
6.11. Project Technical Management ............................................................................................. 15
APPENDIX A FINDING STATUS CODE DEFINITIONS ................................................................ 18
APPENDIX B FINDING PRIORITY DEFINITIONS ........................................................................ 18
APPENDIX C MAPPING AGILE TO PMBOK ............................................................................... 21
TABLE OF TABLES
Table 1: Project Trending Status ............................................................................................................... 3
Table 2: Risk Priority Definitions ............................................................................................................. 19
Table 3: Issue Priority Definitions ........................................................................................................... 20
TABLE OF FIGURES
Figure 1: Risk Rating Matrix..................................................................................................................... 19
1. EXECUTIVE SUMMARY
Following is a summary of the monthly Quality Assurance (QA) risk assessment report
conducted by the Public Consulting Group (PCG) QA team for the Washington State
Employment Security Department (ESD) and the Paid Family and Medical Leave (PFML)
Project.
HIGH
Project Status/Summary during the Reporting Period:
PCG maintained the risk level as high during the reporting period ending November 10, 2019.
PFML was able to reach feature complete by the target date of October 28th, 2019, which
has set the project on track to delivering Benefits v1.0 in January 2020 when the program is
mandated to begin accepting and processing benefit applications and claims. PFML has
continued to refine the release schedule to ensure that all key activities can be completed
before the Benefits v1.0 launch.
PFML continues to work to ensure that updates to the schedule and applicable plans (such
as the Release Timeline(s) and the Communications Plan) are complete.
PCG remains concerned around PFML’s ability to execute the complete scope of the project,
for releases after the Benefits v1.0 launch. The project has met a longstanding goal of having
three sprints worth of stories ready placement, and it appears release 1.1 is now fully
populated with feature sets. This is clearly a positive accomplishment.
Below is a high-level summary of the changes made within the report during the current
reporting period:
• Integration Management:
o PCG closed QA05, which was being tracked as a low risk because PFML's
Integrated Project Schedule is being updated on a regular basis, including
accurately tracking and capturing tasks and resource allocations.
• Scope Management:
o PCG reduced QA04 to medium. This scope risk was related to PFML’s ability to
have enough ready stories in the backlog to achieve the required scope. As of the
current reporting period the minimum viable product (MVP) for Benefits v1.0
development is complete.
• Quality Management:
o PCG rewrote the description of QA38. PCG initially thought that QA38 was isolated
to System Operations and Development teams, but it appears that other
workstreams such as the Customer Care Team (CCT) and the Product team are
also included. This medium risk is related to the lack of coordination and
disconnect between workstreams such as: CCT, System Operations, Product and
Development.
• Human Resource Management:
o PCG opened QA42, which is a low risk. As a result of no defined date to migrate
to the Cloud, the Technology workstream is at risk of burnout, which could impact
future development post Benefits v1.0.
• Risk Management:
o PCG opened QA43 as an observation. This observation is related to the internal
audit that PFML is undergoing.

In summary, as of November 10, 2019 PCG is tracking four (4) high risks, seven (7) medium
risks, two (2) low risk as well as two (2) issues (that are archived) and two (2) positive
observations. The details on each of these risks as well as the highlighted findings noted
above can be found in Section 6 of this report and in Table 3 (QA Status Highlights by
Assessment Category) and within the risk log.
Project Accomplishments During the Reporting Period:

The project had several accomplishments during this reporting period including:
1. Achieving Feature Complete with Benefits v1.0 as scheduled.
2. Completed detailed planning for the release train activities (functional regression
testing, integrated performance testing, security testing and deployment).
3. Presented the creative concepts and media campaign to the Executive Steering
Committee and Advisory Committee.
4. Began onboarding additional Customer Care Team (CCT) staff.
5. Established a Production Support Team within System Operations.
Project Trending Status Over the Last Six (6) Months:

The table on the following page includes a six (6) month summary of priority status as defined
by PCG. It also includes the active findings, or the risks/preliminary concerns associated with
each assessment category.

Table 1: Project Trending Status
June July Aug Sept Oct Nov 11/10/2019

2019 2019 2019 2019 2019 2019
Integration (Closed QA05)

Management
Scope QA04 (decreased)

Management
Schedule QA28
Management
Cost Management QA24
Quality QA38
Management
QA18, QA42 (new)
Human Resource
Management
Communications QA41
Management
Risk Management QA43 (new observation)
Procurement -
Management
QA23, QA29, QA34
Stakeholder
Management
QA20, QA22, QA27, QA33
Technical
Management
The following table below includes a description of the QA Status Assessment rankings used by PCG.
Table 2: QA Status Assessment Legend
QA Status Assessment Legend

Green - Poses the least risk Yellow – Poses a Red – Poses
to project success and can moderate risk to significant risk to
generally be considered project success with project success with
near-free of risks to quality, some products or respect to the project
scope, cost and/or processes that quality, scope, cost,
schedule. present risk and/or schedule and
resulting in deficient should be given
quality, scope, cost immediate attention.
or schedule.
To easily track status changes between the current and former reporting periods, the following symbols will be added to the priority
status rating.
Indicates the risk status Indicates the status has been Indicates the status has not
has increased (worsened) decreased (in severity) since changed since the last reporting
since the last reporting the last reporting period. period.
period.
Project Summary by Assessment Area:

The table below includes a summary of each assessment area (i.e., category). Additional
details on each of the findings that are being tracked in these areas can be found in Section
6 of this report.
Table 3: QA Status Highlights by Assessment Category
Assessment Risk Highlights

Category
Nov
2019 11/10/2019
PCG closed QA05, which was a low risk related to the lack of maintenance to the
Integration Integrated Project Schedule. PCG has observed that the Project Schedule is being
Management updated regularly and no longer includes overdue tasks or resource allocations.
Low
Refer to Section 6.1 Project Integration Management for additional information.
PCG Scope Management is reduced to a medium risk assessment area. MVP

Benefits 1.0 development is complete. The remaining technical consideration for the
January launch is the completion of testing and recovery from any test related
discoveries. Story development for release 1.1 appears sufficient, with the project
Scope achieving a goal related to number of stories ready prior to sprint planning. PCG will
Management Medium continue to monitor story development and backlog management in upcoming
releases.
Refer to Section 6.2 Project Scope Management for additional information on one (1)
medium risk (QA04) which PCG is tracking in this area.
PCG continues to track Schedule Management as a high risk. Although PFML has
an updated, baselined Integrated Project Schedule that is being updated on a weekly
Schedule
basis, it remains unclear if PFML will be able to deliver on time given the amount of
Management
High work to be done. Refer to Section 6.3 Project Schedule Management for additional
information on one (1) high risk (QA28) which PCG is tracking in this area.
PCG continues to track the Cost Management assessment area as a low risk. PCG
notes that QA24, which is a low risk around Earned Value Management (EVM)
Cost doesn’t appear to be impactful to the project.
Management
Low Refer to Section 6.4 Project Cost Management for additional information on one (1)
low risk (QA24) which PCG is tracking in this area.
PCG continues to track one (1) risk related to Quality Management. During the
current reporting period, PCG rewrote the description of QA38 to also include
additional workstreams such as Product and the Customer Care Team. This risk is
Quality ensuring that the product is being built with coordination between all workstreams.
Management Med
Refer to Section 6.5 Project Quality Management for additional information on one
(1) medium risk (QA38) which PCG is tracking in this area.

Assessment Risk Highlights

Category
Nov
2019 11/10/2019
PCG maintains the Human Resource Management assessment area as a medium
risk. Although resource utilization is continually monitored by the project, PCG has
concerns about the impact to the Customer Care Team, Policy, Development and
Treasury staff as a result of the work to be done surrounding the Benefits v1.0 launch
and beyond.
Additionally, PCG opened one (1) new risk, QA42 that relates to potential burn out of
Human Resource
the Technology workstream. This low risk is related to the unknown timeframe to
Management
Med migrate to the Cloud, as on-premise development appears to be taking a
considerable amount of time and cost to scale and is requiring significant rework for
each release.
Refer to Section 6.6 Project Human Resource Management for additional information
on one (1) medium risk (QA18) and one (1) low risk (QA42) which PCG is tracking in
this area.
PCG is not tracking any active communications risks or issues and is tracking one (1)
positive observation around PFML’s forethought around meeting cadence,
Low attendance and ability of meeting attendees to digest and incorporate information
Communications discovered during regularly scheduled meetings.
Management
Refer to Section 6.7 Project Communication Management for additional information
on QA41, which PCG is tracking as a positive observation.
PCG opened QA43 as an observation related to the internal audit that is being
conducted by ESD on the PFML project.
Risk
Low Refer to Section 6.8 Project Risk Management for additional information; however,
Management
PCG not currently tracking any risks or concerns in this area as of this reporting
period.
PCG maintains the Procurement Management assessment area as a low risk. At this
point in time, it does not appear that any contract amendments will need to be made
Procurement for the launch of Benefits v1.0.
Management Low
Refer to Section 6.9 Project Procurement Management for additional information on
one (1) issue (QA26) which PCG has archived but continues to monitor in this area.
PCG maintains Stakeholder Management assessment area as a high risk. PFML has
incorporated two user tests to be held during the month of November and continues
to plan to include feedback earlier within the development process. Until that has
Stakeholder been executed there continues to be a risk that the product being built is built with
Management consideration around time constraints rather than usability and stakeholder feedback.
High Refer to Section 6.10 Project Stakeholder Management for additional information on
two (2) medium risks (QA29 and QA34) and one (1) high risk (QA23) which PCG is
tracking in this area.
PCG continues to track the Technical Management assessment area as a high risk.
PCG remains concerned around PFML's ability to complete end to end testing for
future releases but notes that sufficient time appears to have been built into the
High schedule for the Benefits v1.0 launch considering the recent changes to MVP and
scope for the Benefits v1.0.
Technical
Management Refer to Section 6.11 Project Technical Management for additional information on
the two (2) high risks (QA20 & QA33) and two (2) medium risks (QA22 & Q27) which
PCG is tracking in this area.

2. REFERENCE DOCUMENTS
This section provides a list of documents referenced in the development of the Monthly
Status Report:
• Project Management Institute’s (PMI) Project Management Body of Knowledge (PMBOK®)
• PCG Eclipse QATM Methodology and Framework
• PCG Eclipse AgileTM Methodology
• PCG Quality Assurance (QA) Plan
• IEEE Std.730, IEEE Standard for Software Quality Assurance Processes
• ISO/IEC/IEEE 16326, Systems and Software Engineering
• IEEE Std. 1490, Adoption of the PMI Standard A Guide to the PMBOK®
• IEEE Std. 1012, Standard for Software Verification and Validation
• IEEE Std. 1061, Standard for Software Quality Metrics Methodology
• CMMI Capability Maturity Model Integration (CMMI) Standards
• PFML Investment Plan – Signed – 2018 2 21
• PFML Charter V2.0
• PFML Program Charter - Draft 1
• PFML Level 1 Governance Draft
• PFML Level 2 Governance Draft
• PFML Program Charter Milestone Plan
• PFML Budget Reconciliation Process 2018 01 09
• PFML Project Plan – Draft
• PFML Decision Log
• PFML Stakeholder Register
• Volume 1 Technical Proposal Deloitte 103117
• Volume 2 Financial Proposal Deloitte 103117
• WA ESD Implementation SOW FINAL
• WA ESD Microsoft Master Contract Final
• WA ESD Support SOW Final
• WA ESD Admin Account RFP 2017 115 Vol 1 Tech Proposal Final
• WA ESD Admin Account RFP 2017 115 Vol 2 Financial Proposal Final
• WA ESD BAFO Letter DG for Docusign
• PFML Project Management Plan
3. GLOSSARY
The table below lists the acronyms that appear within this Monthly QA Status Report.
Table 4: Acronyms and Definitions
Acronym Definition
AX Accounting Software
BA Business Analyst
BDS Business Design Specialists
CMMI Capability Maturity Model Integration
CRM Customer Relationship Management
ESD Employment Security Department
HR Human Resource

Acronym Definition
IEC International Electrotechnical Commission
IEEE Institute of Electrical and Electronics Engineers
IP Investment Plan
ISO International Organization for Standardization
ITSD Information Technology Services Division
KPIs Key Performance Indicators
MVP Minimum Viable Product
OCIO Office of the Chief Information Officer
OCM Organizational Change Management
PFML Paid Family and Medical Leave
PCG Public Consulting Group
PMBOK® Project Management Book of Knowledge®
PMI Project Management Institute
PMO Project Management Office
PMP Project Management Plan
PO Product Owner
PQMP Project Quality Management Plan
QA Quality Assurance
RFP Request for Proposal
SMART Specific, Measurable, Achievable, Relevant, and Time bound
VSTS Visual Studio Design Services
4. SCOPE
The PCG QA team has provided an independent monthly assessment of the work
products and the planning, management and control processes that directly support the
successful implementation of the PFML Project. As part of this assessment and project
oversight activities, the PCG team will maintain a findings log and work collaboratively
with the PFML project team to develop and implement recommendations for
improvement that support the overall success of the project. The scope of project
activities assessed includes the following areas:
• Project Meetings
• Key Project Management Activities
• Key Project Documents
• Agile Methodology
• Visual Management
• Daily Stand-ups and Agile Ceremonies
5. APPROACH
PCG follows a proven four-step approach to completing each Monthly Status Report.
However, as applicable, this approach is tailored to be consistent with the current
schedule and activities being conducted by the project. PCG’s standard approach is as
follows:

1. Discovery – Review project documentation, work products, deliverables, and

application code as well as interview key project team members to gain a thorough
understanding of the assessment area. It is during this phase that applicable
standards, best practices and lessons learned will be identified as evaluation
criteria.
2. Research and Analysis – Research and analysis will be conducted for specific
aspects of the products or processes being assessed in order to form an opinion of
the validity of the proposed tool selection, design, and/or implementation approach.
Once the initial analysis is completed the assessment results will be documented.
3. Clarification – Seek clarification from key project team members on specific
aspects to ensure agreement and concurrence between the project, vendor and the
PCG QA team.
4. Documentation – Document the results of discovery, research, analysis, and
clarification in a Monthly QA Status Report. These reports will contain key findings,
preliminary concerns and risk assessments. The PCG QA team will provide input
regarding mitigation strategies, as well as any potential impact to the project if risks
are not addressed
6. ASSESSMENT
Following is the detailed analysis and associated findings that resulted while completing
the monthly QA risk assessment for the reporting period September 11, 2019 through
October 10, 2019. Each section includes findings and recommendations that PCG has
identified as critical to the PFML Project’s success. A cumulative list of all QA findings is
maintained in a separate QA Findings and Recommendations Log. This cumulative list
includes updates made to existing, open findings and new findings identified during the
current reporting period.
6.1. PROJECT INTEGRATION MANAGEMENT

PCG is closing QA05 as it relates to Integration Management. This risk was initially
logged during previous reporting periods because the Integrated Schedule
contained items that were past due and not yet complete. PFML's Integrated Project
Schedule is now being updated on a regular basis and PCG has observed during
the recent reporting periods the rigor around tasks, resource allocation and updates
being made on a frequent basis throughout the week, with the schedule being sent
out weekly. Now that the work to be done is accounted for within the project
schedule, PCG feels confident that the PMO is able to accurately forecast if the
work can be done within the given timeframe.
Additionally, PFML scheduled a meeting during the current reporting period that
included PCG and OCIO to review major updates to the schedule. PCG will
continue to recommend that all tasks be monitored and updated, and schedule
analysis be performed to determine impacts to resources and the critical path.
6.2. PROJECT SCOPE MANAGEMENT

PCG reduced the Scope Management assessment area to a medium risk and is
tracking one (1) medium risk (QA04) and one (1) issue (QA25), which has been
archived within the risk log.

PCG reduced QA04 to a medium risk. This risk is around the project’s ability to
maintain enough stories in the backlog that can be completed by the teams to
deliver the required scope. During this reporting period improvement has been
achieved and exhibited in the story creation and grooming for the Benefits v1.0
launch, and development was completed within schedule expectations. Sprint
planning work is in progress for v1.1 through 1.5.
The project team achieved a long-standing goal of having 3 sprints worth of stories
beyond the current sprint allowing for the planning of v1.1 release. This resulted in
the reduction of this risk. Continued ability to have enough stories groomed and
ready through the next few releases will result in further reduction in this risk.
Project Scope Management
# Type Description Priority Recommendation
QA Within the current defined Service PCG recommends that PFML focus on story
Risk
04 Delivery Roadmap, features required for development. PFML should have at least three
each release lack all necessary sprints worth of stories ready in the backlog at all
supporting User Stories and acceptance times.
criteria to achieve feature complete for
the remaining releases. Without these Medium
supporting Stories it may be unclear what
exactly is to be developed within the
given time.
Significance:
The project burnup chart should indicate if the current average velocity development
is on pace to deliver all in-scope items by the defined delivery date. Without all
necessary supporting User Stories and acceptance criteria for each feature in the
upcoming release, features may not be completed on time.
6.3. PROJECT SCHEDULE MANAGEMENT

PCG is tracking one (1) risk related to Project Schedule Management and maintains
QA28 as a high risk. At the end of this reporting period MVP Benefits v1.0
development is completed, and testing is scheduled to begin in support of the January
2020 program launch. Meeting the mandated program launch from a technical
standpoint is currently dependent on successful testing and the ability to promptly
respond to any discoveries made in the process.
During this, and the previous reporting period the PFML’s use of data driven
assessments of capacity and velocity have allowed for increased success in planning
and delivery of functionality. Additionally, development velocity has stabilized,
providing the team with the ability to more accurately anticipate completion dates.
Although this risk was realized during the previous reporting period, PCG notes that
mitigation steps have been taken. The Project has a plan to ensure the Benefits 1.0
launch can occur in January 2020. PCG maintains this as a high risk to ensure that
the scope and milestones are being met and that subsequent releases are planned
for and are achievable.
Continued improvement and consistency in practice will result in further reduction of
this risk.

Project Schedule Management
QA28 Based on the Integrated Project Schedule it PCG recommends the project continues to
Risk is unclear if PFML can deliver on time. update the project schedule. Updates should
continue to be done via daily check ins with
the scrum master with formal updates
High
happening at least weekly.
Ensure that the Project is accurately
capturing within the updated project
schedule, project goals and objectives,
timeline, milestones/deliverables, risks and
dependencies. PCG would want to see
details of the working software that ESD
plans to produce within each Sprint. ESD
should keep in mind that working software
will generally include work across multiple
subsystems.
Significance:
The project continues to face tight timelines and a legislative mandate. Not delivering
on time will impact the project cost, employee morale and confidence from
stakeholders.
6.4. PROJECT COST MANAGEMENT

PCG continues to track one (1) low risk (QA24) because of the minimal adoption to
Earned Value Management (EVM). The project has chosen to accept the risk and is
mitigating cost management through the vendor management process, close budget
monitoring and between the Budget and PFML managers. PCG will work with ESD
on next steps as it relates to EVM only as requested. No change since last reporting
period.
Project Cost Management
QA24 The project is not using Earned Value PCG continues to recommend the expansion of
Risk
Management (EVM) to track Schedule EVM into the technology workstreams using an
Variance and Budget Variance. Low Agile EVM scorecard. This will allow the project
to present a total project performance picture to
stakeholders that may not be familiar with Agile
development methodologies.
(no change since last reporting period)
Significance:
This may make the project appear to be on or under budget when additional costs
are deferred due to schedule and/or deliverable delays.
6.5. PROJECT QUALITY MANAGEMENT

PCG maintains one (1) medium risk related to Project Quality Management. While
PCG first thought that the risk was isolated to System Operations and Development
teams, it is evident that other workstreams such as the Customer Care Team (CCT)
and customer insights, and thus has rewritten this risk.
There appears to be a disconnect between what the developers are building and what
the CCT is needing and/or expecting, with a noted lack of coordination via the Product
team. The newly formed UX (User Experience) Team has been established and
although their role isn’t explicitly intended to ensure coordination between all
workstreams, their efforts are noted. PCG will observe the work efforts of the UX
Team and next steps. Ultimately, PCG recommends that the Product team works to
ensure that all workstreams are aware of the work to be done and efforts to alleviate
this risk. System Operations is continuing to take the opportunity to attend various
standups and is attempting to gain a greater knowledge and understanding of what
is being built. PCG will continue to monitor. PCG believes the desired level of
collaboration and coordination isn’t happening due to the tight timeframes and other
prioritized work of the developers. System Operations continues to explore ways to
engage early with development, or key development teams.
PCG recommends that PFML System Operations works with the Development teams
in critical design and build meetings and deliverable reviews to minimize rework and
potential problems. Additionally, PCG encourages the Product Team to continue
efforts to bridge any gaps between Production, Development, System Operations and
the CCT. PCG encourages PFML to define a triage and support plan and not 'manage
by emergency;' document how to troubleshoot what is being built with inclusion of
PFML System Operations, CCT and all Development Operations teams to lessen
rework and potential production problems. PCG will maintain QA38 as a medium risk
until a product support plan is implemented, along with determining if PFML is
ensuring sprint capacity is being subtracted from available capacity to ensure the
proper support is provided.
Project Quality Management
QA38 There is a risk that the end product is being PCG recommends that PFML System
Risk
built without coordination between all PFML Operations, CCT, Product and Development
workstreams to ensure a quality product is teams are included in critical design and build
Med
being built to support necessary functionality meetings and deliverable reviews to minimize
as well as future growth and flexibility. rework and potential problems. PCG
encourages PFML to define product support
plan and not 'manage by emergency;'
document how to troubleshoot what is being
*Risk re-written as of 11-10-2019 built with inclusion of all applicable
workstreams to lessen rework and potential
production problems.
Significance:
Rework impacts the project schedule/timeline and resource efficiency.
6.6. PROJECT HUMAN RESOURCE MANAGEMENT

PCG is tracking one (1) medium risk (QA18) and one (1) new low risk (QA42) related
to Human Resource Management.
Resource utilization (QA18) remains a medium risk to the project. PCG remains
concerned around Customer Care Team (CCT), Development, Treasury and Policy
staff. The Organization Change Manager continues to work with staff to suggest ways

to ensure utilization is being addressed and burn out is avoided. PFML has continued
the process of onboarding new CCT staff for the anticipated volume of calls during
January following the Benefits v1.0 launch. PCG will continue to monitor resource
utilization, including meeting with key staff resources, the organizational change
manager and agency leadership.
PCG opened one (1) new risk during the current reporting period related to Human
Resource Management. This risk is the result of not having a defined date to migrate
to the Cloud. The Technology workstream is at risk of burnout due to the significant
time needed for development and deployments, which could impact future
development post Benefits v1.0. Additionally, on-premise development requires
increased operational inefficiencies, and additional maintenance and operations
overhead. Ultimately, this risk is related to Human Resources. Currently, PFML has
not clearly communicated when this shift may happen and the Technology
workstreams have expressed concern with burnout. PCG recommends that
Leadership clearly define and articulate when Cloud migration will take place. This
will ensure all staff are clear on the path forward and hopefully reduce the risk of
burnout. PCG will continue to monitor this risk until a formal decision is made and
after we have assessed the potential impact(s) to staff.
Project Human Resource Management
QA18 PCG has observed that project

Risk
PCG encourages ESD to continue to assess

resources are over-utilized. roles and responsibilities and distribution of
work, along with continuing to manage and
Med monitor the schedule to determine the amount
of work that can be done with Customer Care
Team, Policy, Development and Treasury staff.
QA42 As a result of no defined date to migrate

Risk
PCG recommends that Leadership clearly

to the Cloud, the Technology define and articulate when Cloud migration will
workstream is at risk of burnout, which take place. This will ensure all staff are clear on
could impact future development post NEW the path forward and hopefully reduce the risk
Benefits v1.0. of burnout.
Low
Significance:
Resources (Customer Care, Policy and Treasury staff, etc.) may still find themselves
too much work for the time available. Insufficient staff capacity for Human Resource
Management can contribute to the overall schedule challenges the project is
experiencing.
Leadership has made a decision not to migrate to the Cloud before January 2020.
Without a clear path forward there is a risk that significant time and cost to scale
development work as well as an increased downtime for deployments and
maintenance and easier deployments. This ultimately impacts the development
teams' utilization.
6.7. PROJECT COMMUNICATIONS MANAGEMENT

PCG is not tracking any active communications risks or issues. PCG continues to
track one (1) positive observation because PFML continues to identify ways to
appropriately utilize meeting(s) to ensure timely communication and key staff are

able to be present. Please see Section 6.10 (Stakeholder Management) for PCG’s
concerns related to managing Stakeholder messaging. PCG will continue to review
and assess Communications Management as it relates to the PFML Project.
Project Communication Management
PCG recommends the project continue to

Observation
PFML continues to identify ways to

address communication management for the
appropriately utilize meeting(s) to ensure timely
Positive
QA40 life of the project. Ensure that timely top/down

communication and key staff are able to be
communication continues to happen as
present.
N/A decisions are made.
Significance:
Considering who/why/when for meeting participants and output of communications
ensures all PFML staff and contractors are kept informed and properly engaged.
Additionally, it ensures that there is an appropriate cadence and sequence of
meetings, i.e.; having enough time to prepare for meetings given the information
received.
6.8. PROJECT RISK MANAGEMENT
PCG is not currently tracking any risks related to Risk Management. PCG will
continue to validate that there is a documented Risk Management process that
adheres to project management best practices and asses the project risk and issue
management activities. PFML continues to hold regularly scheduled Risk Review
meetings for all of the workstreams and keeps an up to date risk register, including
risk owner, next steps and updates.
PCG opened one new observation related to Risk Management during the current
reporting period. This observation is related to the internal audit that is being
conducted by ESD. PCG has met with the internal auditor and will continue to work
cooperatively with internal audit as requested.
Project Risk Management

Observation
The PFML project is undergoing an internal

QA43 N/A, this is an observation.
audit.
N/A
6.9. PROJECT PROCUREMENT MANAGEMENT

PCG maintains Procurement Management as a low-risk assessment area. Contract
amendments may be needed for the Benefits v1.0 launch. Although this risk has been
transferred to an issue, there is no action required by ESD, and PCG has archived

this issue within the risk log. QA26 should be monitored only for additional contract
amendments (if needed) and will not be reported within the monthly report, unless
needed. QA26 can be referenced within the monthly risk log.
Significance:
The project could incur delays as a result of contract renegotiation(s) and/or
amendments.
6.10. PROJECT STAKEHOLDER MANAGEMENT

PCG maintains the severity of the Stakeholder Management assessment area as a
high risk and is tracking one (1) high risk and two (2) medium risks.
PCG is maintaining QA23 as high risk to ensure that early stakeholder involvement
is considered. Although PFML continues to strive to include feedback as early into
the process as possible into the development cycle, there continues to be a risk that
the product being built is built with time constraint considerations versus stakeholder
feedback and usability. PCG has observed the continued level of effort for
stakeholder engagement and outreach done by ESD to reduce the risk of impact to
the Project. PCG continues to recommend that ESD engage with customers to ensure
their feedback and opinions can also be taken into account, which is currently
scheduled for mid-November via two (2) user tests. PCG will reduce this risk in the
coming reporting periods once PFML has successfully executed these tests and are
able to integrate the feedback into the development process for feature
releases/launches.
PCG is maintaining QA29 as a medium risk related to Stakeholder Management for
the Product Acceptance Testing (PAT) timeline for User Experience testing (UX)
timeline. PFML views PAT as important to the development and the Product
Management Team and continues to try to evaluate the items taken into each sprint
to identify which (if any) are good candidates for PAT. As of the current reporting
period, a user test is scheduled for 11/13-11/19 that focuses on two (2) studies. The
first test being a real scenario with an employee navigating the system and interacting
with the Customer Care Team (CCT). The second test will be focused on portal
usability. The results from this study will be available in early December, with an initial
debrief scheduled for 11/21 or 11/22. Going forward into Benefits development post
January, the Business Design Team (BDS’s) will be taking a more active role in
developing the acceptance criteria as recommended.
PCG maintains QA34 as a medium risk. A stakeholder management plan has been
developed and was presented to the leadership team. The PFML website is being
updated to ensure customers and stakeholders can get the information they need and
that they can walk through the decision of applying for benefits. An earned media
strategy is also under development with program/agency leadership and the
marketing vendor. PCG will continue to monitor this risk as the communication to
stakeholders is executed.
Project Stakeholder Management
QA23 As a result of insufficient stakeholder PCG continues to recommend the project

Risk
involvement early in the development cycle, demo software early and get customer
feedback quickly. The project risks features

there is a risk that the product being built is identified by product owner(s) may result in
incorrect. High user stories that don't reflect user needs.
QA29 The Product Acceptance Testing (PAT) PCG recommends PAT be integrated into
Risk
timeline remains tight and the Agency is every stage of the agile development cycle so
considering a 'just in time' approach with that continuous feedback is received. The
PAT. Customer engagement and Med earlier bugs or issues are discovered, the
involvement may not be enough. cheaper they are to fix.
QA34 Managing external stakeholder expectations PCG recommends that PFML continue to
Risk
as it relates to PFML Project messaging. execute the communications plan taking into
consideration any changes that may impact
Med stakeholder expectations.
Significance:
Insufficient stakeholder feedback could cause rework and schedule delays.
Without adequate customer engagement and involvement, the project may be
creating a product that does not meet the needs or expectations of the end
customer. Insufficient stakeholder feedback could cause rework and schedule
delays.
The number of stakeholders creates a challenge to adequately provide updated
communications to all stakeholders.
6.11. PROJECT TECHNICAL MANAGEMENT

PCG continues to track the Technical Management assessment area as a high risk
and is tracking two (2) high risks and two (2) medium risks.
PCG maintains QA20 as a high risk. This risk is related to the testing timeline for
end to end testing. PCG notes that the project has been refining their approach to
this risk, carefully monitoring scope and schedule as well as other considerations to
ensure sufficient testing is completed within timeframes. PFML is working with test
strategies for releases beyond the Benefits v1.0 launch. PCG will continue to
monitor this risk for reduction as soon as it becomes clear through repeated cycles
that enough time is consistently allotted. Although this risk was realized within a
previous reporting period, PCG would like to see this same rigor built into
subsequent releases and observe the project achieving a stabilization period. This
will ensure a fully functional product is achieved and to ensure that sufficient time is
allotted for Product Acceptance Testing (PAT) and User Experience (UX). As of this
reporting period ending November 10, 2019, the release schedule for Benefits 1.0
launch is complete. Considering the importance of ensuring there is enough time for
end to end testing, and its downstream impact on PAT, PCG continues to assess
this as a high risk.
PCG maintains QA22 as a medium risk. PCG has observed increased rigor around
sprint planning, release planning, change management controls and the use of
tools/artifacts to support the Agile planning process and supports on-going efforts in
this area. Continued efforts in this area will result in reduction or retirement of this
risk.
PCG maintains QA33 as a high risk and will continue to monitor whether time
allotted is sufficient and end to end testing is executed completely as desired. If so,
this would indicate a reduced risk for future releases.

PCG maintains QA27 as a medium risk around EBMS. As part of platform

readiness, when EBMS was deployed in a production environment the risk EBMS
poses to the project was reduced. Although EBMS is being used in production for
premiums, PFML has procured another solution to enhance EBMS (Service Bus)
which has been tested and will be implemented for use with Benefits, which
appears to eliminate the risk to the project. ESD has done another review of the
state of EBMS and the progress of the Service Bus implementation as of late
October. PFML has logged two (2) risks around EBMS relating to the need to
continue to monitor EBMS and its maturity level, which appears to have sufficiently
risen to support production. Regarding EBMS being an unproven batch
management framework, PFML is continuing to monitor. PFML has a plan to revisit
the EBMS risks at end of January. At that point in time ESD will have completed
another reporting quarter and the first month of benefits. PCG will continue to track
QA27 as it relates to EBMS until Benefits has been implemented in January to
ensure the Service Bus solution is working.
Project Technical Management
QA20 The systems integration testing timeline PCG recommends using a refined post
Risk
may pose a risk to a fully functional end- development work approach that includes
to-end product resulting in sub-systems High repeated cycles of sufficient time consistently
that lack full integration. allotted to monitor the systems integration
testing timeline as it applies to Technical
Management.
QA22 There continues to be an observation of PCG continues to recommend planning for
Risk
variance from established norms for integration even at the story level and providing
Agile development which require a demonstrations which include work across all
correction. It should be noted that these teams at the end of each sprint rather than only
are not due to a lack of hard work and one piece of the functionality for a story, i.e.
dedication on the part of the team, but Med they may currently be demonstrating the User
rather the increased demands on their Interface (UI) or Customer Relationship
time and a short-cutting of proper Management (CRM) layer, but not both. PCG
processes. This finding is an opportunity also recommends continued focus on
to course correct and help the teams preparedness for Sprint Planning and having all
work smarter, faster, and more efficiently tasks defined and committed to as the output
for future releases. from sprint planning, and the indicator that the
sprint is ready to start.
Additionally, PCG recommend that PFML
ensure rigor around getting development work
done earlier to ensure there is adequate time to
test and include customer input.
QA27 The Agency has determined that their PCG recognizes that this is an Enterprise level
Risk
batch management framework (EBMS) concern and recommends that PFML continue
is unproven in this product configuration to work with ITSD to ensure that EBMS experts
and lacks maturity for the ESD service are available to troubleshoot any issues for
Med
delivery model. EBMS currently only premiums.
processes one job in a production
environment.

QA33 PFML is not able to perform end-to-end PCG recommends using agile reports such as
Risk
testing since certain elements of the the burnup chart to drive decisions to
software (CRM & AX) are not complete. determine whether to change the scope or
The inability to perform end-to-end schedule.
functional testing poses a risk to the
entire system being built which may PCG recommends continued use of daily check
cause critical functionalities such as ins with the testers, and elements that are
High
communication between subsystems, behind schedule/blockers are reported.
interfaces and databases to behave PCG recommends that the project use a three-
unexpectedly. step design framework for end-to-end testing to
include user functions, conditions and test
cases. User functions should include listing
software features and tracking actions
performed for each feature. Conditions should
be built based on user functions; every user
function should have a set of conditions. Test
cases should be created for every user function
based on user scenarios and every condition
should have a separate test case.
Significance:
The time allotted for the end to end testing is minimal and may negatively impact the
User Acceptance testing timeline. Without completing end-to-end functional testing,
it is uncertain whether the flow of the system is performing as designed from end-to-
end.
Without following Agile processes consistently, it increases the likelihood of not
delivering features on time and of expected quality for desired business value(s).
ESD won't be able to process complex batch jobs using EBMS in all environments,
impacting schedule and acting as a blocker to development teams. Without the use
of an exact replica production environment in the release process users may expose
unknowns usually found via exploratory QA testing done in a staging environment.
Without completing end-to-end functional testing, it is uncertain whether the flow of
the system is performing as designed from end-to-end.

Appendix A Finding Status Code Definitions

A status identifies the current state of a risk or issue. This section provides a description
of each of the status code values that may be assigned.
Table 5: Status Code Values of a Finding
Status Code Description

Value
Open Identifies a risk or issue that has been opened during the reporting period either
through specific deliverable tasks or routine QA project monitoring activities.
Updated Identifies a risk or issue that was opened in an earlier report or deliverable and has
subsequently been updated based on new information. The “Update” column, also
found in the QA Findings and Recommendations Tracking Log, will provide the details
of the update as QA continues to monitor the risk or issue.
Closed Identifies a risk or issue that has been closed during the reporting period. The
“Update” column found in the QA Findings and Recommendations Tracking Log, will
document the reason for closure.
N/A A status of N/A is assigned to a neutral or positive observation. Observations are
reported to document a specific aspect of the project that was reviewed; however, did
not result in a risk or issue.
Appendix B Finding Priority Definitions
A priority is assigned to all risks and issues. Once a priority is assigned to a risk or issue,
it will be reassessed during each subsequent reporting period as part of the routine QA
project monitoring activities and documented in the QA Findings and Recommendations
Tracking Log.
C.1 Risks
A risk is “an uncertain event or condition that, if it occurs, may have a positive or
negative effect on a project’s objectives”. The PCG QA team will identify risks
with negative effects and expand the definition to include both conditions which
may occur and those which may not occur (e.g. lack of a well-defined
requirements traceability process could lead to delivery of an incomplete system,
requiring costly and time-consuming rework).
A key to risk management is understanding the potential risks to the project and
ensuring that these risks and risk mitigation strategies are communicated to key
project stakeholders on an ongoing basis. Risk analysis should begin early
during project planning by determining or identifying the factors that may affect
the project. Risk can impact a project in many ways: project quality, scope, cost,
and schedule. Proper risk identification seeks to determine how the risk may
affect the project and to document the project area(s) impacted by the identified
risk.

Once risks are identified and characterized, both qualitative and quantitative
factors are examined. Figure 1: Risk Rating Matrix
Analysis considers the

project conditions to
determine the probability
of the risk being realized
and the impact to the
project if it is realized.
Overall the risk priority is
determined by
multiplying the probability
rating times the impact
rating. The PCG QA
team will determine the
risk priority using a risk-
rating matrix (see Figure
1) which determines the
priority of each risk
based on an assessment
of probability of
occurrence and the magnitude of impact.
The following table defines the risk priorities that will be used when identifying
risks.
Table 2: Risk Priority Definitions
Risk Priority Definition

A priority of high is assigned if there is a possibility of substantial impact to
High product quality, scope, cost, and/or schedule. A major disruption is likely,
and the consequences would be unacceptable. A different approach is
required. Mitigation strategies should be evaluated and acted upon
immediately.
A priority of medium is assigned if there is a possibility of moderate impact
Medium to product quality, scope, cost, and/or schedule. Some disruption is likely,
and a different approach may be required. Mitigation strategies should be
implemented as soon as feasible.
A priority of low is assigned if there is a possibility of low impact to product
Low quality, scope, cost, and/or schedule. Minimal disruption is possible, and
some monitoring is likely needed to ensure priority does not increase.
Mitigation strategies should be implemented as time permits.
C.2 Issues
An issue is a finding, often previously identified as a risk that documents an event

that has occurred and caused negative impact to the project. Issues,
documented in the QA Findings and Recommendations Tracking Log, identify
the event, its impact to the project, and updates towards resolution.
An issue’s priority is determined by its impact on the project. The following table
defines the issue priorities that QA uses when assigning a priority to an issue.
Table 3: Issue Priority Definitions
Issue Priority Definition

A priority of high is assigned if the issue presents substantial impact to
High product quality, scope, cost, and/or schedule. A catastrophic disruption is
likely, and the consequences would be unacceptable. A different approach
is required. Mitigation strategies should be evaluated and acted upon
immediately.
A priority of medium is assigned if the issue presents a moderate impact to
Medium product quality, scope, cost, and/or schedule. Some disruption is likely, and
a different approach may be required. Mitigation strategies should be
implemented as soon as feasible.
A priority of low is assigned if the issue presents a slight impact to product
Low quality, scope, cost, and/or schedule. Minimal disruption is possible, and
some monitoring is likely needed to ensure priority does not increase.
Mitigation strategies should be implemented as time permits.
C.3 Preliminary Concerns
QA attends meetings, reviews documentation, conducts interviews and performs

independent analysis in order to verify and validate project activities and
progress. Results of the QA analysis are captured as either preliminary concerns,
risks or issues defined above in C.1 and C.2.
A preliminary concern is an item we believe may pose risk to the project, but
more analysis and a better understanding of the subject area is necessary before
classifying the item as a formal risk or issue. Preliminary concerns are
documented in statements which articulate the concern and indicate further
analysis and/or understanding of the matter is required.

Appendix C Mapping Agile to PMBOK






PFML Monthly QA Status Report As of Nov 10 2019 v1.1

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PFML Monthly QA Status Report As of Nov 10 2019 v1.1

Uploaded by

Copyright:

Available Formats

Washington Employment Security Department (ESD)

Paid Family and Medical Leave

Monthly Quality Assurance (QA)

Reporting Period: October 11, 2019 – November 10, 2019

Date: November 13, 2019

Ms. Carla Reyes, PFML Director

Dear Ms. Reyes:

Public Consulting Group, Inc. 1

Public Consulting Group, Inc. 2

2. REFERENCE DOCUMENTS ...................................................................................................... 6

APPENDIX A FINDING STATUS CODE DEFINITIONS ................................................................ 18

APPENDIX B FINDING PRIORITY DEFINITIONS ........................................................................ 18

APPENDIX C MAPPING AGILE TO PMBOK ............................................................................... 21

Public Consulting Group, Inc. 1

Project Accomplishments During the Reporting Period:

Project Trending Status Over the Last Six (6) Months:

Public Consulting Group, Inc. 2

Table 1: Project Trending Status

June July Aug Sept Oct Nov 11/10/2019

Integration (Closed QA05)

Scope QA04 (decreased)

Cost Management QA24

Risk Management QA43 (new observation)

QA Status Assessment Legend

Project Summary by Assessment Area:

Public Consulting Group, Inc. 3

Assessment Risk Highlights

PCG Scope Management is reduced to a medium risk assessment area. MVP

Public Consulting Group, Inc. 4

Assessment Risk Highlights

Public Consulting Group, Inc. 5

Public Consulting Group, Inc. 6

Public Consulting Group, Inc. 7

1. Discovery – Review project documentation, work products, deliverables, and

6.1. PROJECT INTEGRATION MANAGEMENT

6.2. PROJECT SCOPE MANAGEMENT

Public Consulting Group, Inc. 8

Project Scope Management

# Type Description Priority Recommendation

6.3. PROJECT SCHEDULE MANAGEMENT

Public Consulting Group, Inc. 9

Project Schedule Management

# Type Description Priority Recommendation

6.4. PROJECT COST MANAGEMENT

# Type Description Priority Recommendation

6.5. PROJECT QUALITY MANAGEMENT

Project Quality Management

# Type Description Priority Recommendation

6.6. PROJECT HUMAN RESOURCE MANAGEMENT

Public Consulting Group, Inc. 11

Project Human Resource Management

# Type Description Priority Recommendation

QA18 PCG has observed that project

PCG encourages ESD to continue to assess

QA42 As a result of no defined date to migrate

PCG recommends that Leadership clearly

6.7. PROJECT COMMUNICATIONS MANAGEMENT

Public Consulting Group, Inc. 12

Project Communication Management

# Type Description Priority Recommendation

PCG recommends the project continue to

PFML continues to identify ways to

QA40 life of the project. Ensure that timely top/down