Agency Name

Project Name
Deliverable Review Process and Acceptance Criteria

Version Number: (n)

Date: (mm/dd/yyyy)

Document History and Distribution

1. Revision History
Revision # Revision Date Description of Change Author

2. Distribution
Recipient Name Recipient Organization Distribution Method

Deliverable Review Process and Acceptance Criteria

Table of Contents

1 Purpose and Scope..................................................................................................................................1 2 Goals of the Deliverable Review Process..............................................................................................1 3 Glossary of Terms..................................................................................................................................2 4 Meeting Participants, Roles, and Responsibilities.................................................................................3 5 Review Process.......................................................................................................................................3 5.1 Review Process Flow....................................................................................................................3 5.2 Review Process Steps....................................................................................................................5 6 Review Dispositions...............................................................................................................................7 7 Exit Criteria for Reviews........................................................................................................................8 8 Acceptance Process for Project Deliverables.........................................................................................8 9 Acceptance Criteria for Milestones and Deliverables.........................................................................12 10 Approvals..........................................................................................................................................18 Appendix A: Log of Recommended Actions/Changes..........................................................................19 Appendix B: Review Verification Form...............................................................................................20 Appendix C: Request for Acceptance Form..........................................................................................21

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria

1 Purpose and Scope

Acceptance criteria are defined as the list of requirements that must be satisfied prior to the customer accepting delivery of the product. This document defines the acceptance process, the acceptance criteria, and the review/approval required for customer acceptance of the (Agency name) (project name) project deliverables. The purpose of this document is to define a standardized Deliverable Review Process, which will provide a structured method to support the Agency Software Verification and Validation Plan (SVVP) to ensure that appropriate, correct, consistent, and complete deliverables are created for the project. This document describes: Goals of the review process; Definitions; Meeting participants, roles, and responsibilities; Review process; Dispositions for the review meeting; and Review exit criteria. Appendices A and B contain forms that aid the Moderator, Deliverable Owner, Review Team and Quality Assurance (QA) Representative in carrying out Deliverable Review Process activities Appendix A contains a Log of Recommended Actions/Changes. Appendix B contains the Review Verification Form. Appendix C contains the Request for Acceptance Form.

For detailed information about the peer review process, refer to the Guidelines for Agency Peer Reviews located on the IRMC web page in the IRMC Approved Principles, Policies, and Standards section, Project Reporting category.

2 Goals of the Deliverable Review Process

The primary goal of the Deliverable Review Process is to detect and remove deliverable defects as early as possible in the Software Development Life Cycle (SDLC) process. Secondary goals to be attained are: Consistency with IEEE Std 1028-1997, Standard for Software Reviews; Ensure correctness, completeness, consistency, and accuracy of deliverables and for all life cycle activities within the development process;

products

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria Provide reviewers with a common understanding of the review criteria and the deliverable; and Serve as a criterion for the (Agency name) Deliverable Acceptance Process.

3 Glossary of Terms
Following are definitions of commonly used terms in this document: Deliverable: A non-technical or technical component item to be reviewed. Deliverable Owner: The person responsible for the deliverable and the initiation of the review. Document Control: The organization responsible for documentation filing procedures. Item: A thing written down at the Review Process meeting. This consists of recommended action items, point clarifications, and improvement suggestions. Meeting: A forum used to formally present the deliverable during which all participants are able to communicate and interact with each other at the same time. Moderator: The person who leads the meeting and is responsible for the overall review meeting. Peer: A member of the agency staff who is of equal standing with the author of the work document. Peer review: A formal peer meeting held to allow discussion of issues, alternatives, options, and work document approval. Review process: The structured process to review the deliverable; meet to allow discussion of recommended changes, actions, alternatives to improve the quality of the deliverable and reach a consensus on the disposition of the deliverable; record the feedback; and verify completion of the review; and archive the deliverable and review forms. Review criteria: The stated requirements, policies, standards, procedures, methods, and guidelines applicable to the deliverable. Reviewer: A review meeting participant who evaluates the deliverable against an established review criteria. Quality Assurance: (Product) Conformance to technical and business functional requirements to ensure defect-free products (completeness of software or system features and

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria functions, error-free operation). (Process) Verification and validation of project activities and resultant artifacts with respect to established policies, standards, procedures, methods, and guidelines for software development. Quality Assurance (QA) representative: The person responsible for verifying compliance to the established Quality Assurance policies, standards, procedures, methods, and guidelines for the project or the agency. Scribe: A peer review meeting participant who records items identified at the meeting. Software verification and validation (SVV): A disciplined Quality Assurance approach used to assess software products throughout the Software Lifecycle Process. Verification is defined as the process of determining whether products of a given phase of the software development process fulfill the requirements established during the previous phase, and validation is defined as the process of evaluating software at the end of the software development process to ensure compliance to software requirements. Work document: The material to be reviewed.

4 Meeting Participants, Roles, and Responsibilities

The Deliverable Review Process team members include the moderator, the deliverable owner, the reviewers, a scribe, and a QA representative, as defined in the definitions section of this document. The deliverable owner may serve as moderator. The QA representative may also serve as the moderator. A separate scribe may be selected to record all recommended changes or action items discussed at the meeting.

5 Review Process
This section details the review process to initiate, conduct, and finalize reviews. Note that the deliverables may be partitioned into units to allow for incremental reviews. The deliverable reviews are complete after all incremental reviews have been held.

5.1 Review Process Flow

The review process flow is shown below.

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria

(1) Complete the deliverable (DO)

(2) Plan Meeting (DO)

(3) Distribute meeting information & review materials (DO)

(4) Review deliverable, make notes (R); become familiar with deliverable, review criteria, & meeting format (M)

(5) Conduct review meeting (R/M/DO)

(6) Detemine & document review diisposition (R,DO,M)

(7) Approved

(8) Approved with changes

(9) Rework

(7.1) Complete review verification & exit (QA)

(8.1) Complete changes & resolve actions (DO)

(8.2) Verify closure of changes & actions (QA)

DO = Deliverable Owner R = Reviewer M = Moderator QA = QA Representative

(8.3) Complete review verification & exit (QA)

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria

5.2 Review Process Steps

This section details the steps to complete the review process.
Phase Task Prior to Review
1. 2. Complete deliverable subject to review. Plan meeting: Identify reviewers from the Software Verification and Validation Plan (SVVP), Identify review criteria from SVVP, Identify supporting materials, if appropriate (examples, applicable standards, guidelines and practices), Determine meeting format (including meeting agenda, order, communication mode - face-to-face vs. teleconference), Determine meeting date and time, Determine and secure meeting location, Determine and secure any special meeting logistics (overhead, laptops, etc.), Select the moderator Distribute the following information to the reviewers and moderator (and supply a courtesy copy, cc, to the project manager and QA manager) with a lead time commensurate with the size of the deliverable but a minimum of 24 hours: Deliverable, Review criteria, Meeting information (date, time, place), Supporting materials (if appropriate) Review or become familiar with the deliverable, review criteria, and meeting information. Review the deliverable (and supporting documentation, if appropriate) against the review criteria and note recommended changes and actions; and bring notes to review meeting. Become familiar with deliverable, the review criteria, and meeting format.

Owner
Deliverable Owner Deliverable Owner

3.

Deliverable Owner

4.

Reviewers, Moderator

4.1

Reviewers

4.2

Moderator

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria

During the review

5. Conduct the Review Meeting. Reviewers, Deliverable Owner, Moderator Moderator Moderator

5.1 5.2

Introduce meeting goals and expectations Solicit feedback in manner appropriate to deliverable, review criteria, and meeting format. Document suggested changes and actions. Summarize review results. Determine and document review disposition Reach consensus on review disposition: Approved, Approved with changes, Rework Record disposition of the review.

5.3 5.4 6 6.1

Moderator Scribe Reviewers, Deliverable Owner, Moderator Reviewers, Deliverable Owner

6.2

Moderator

After the review, Approved disposition

7 7.1 If Approved disposition Complete review verification and exit QA Representative (QA/Project Manager) QA Representative (QA/Project Manager) QA Representative (QA/Project Manager) QA Representative (QA/Project Manager) QA Representative (QA/Project Manager)

7.1.1

Draft Review Verification Form.

7.1.2

Sign Review Verification Form.

7.1.3

Place deliverable in configuration management

7.1.4

File Log of Recommended Actions/Changes Form and Review Verification Form in the Project Library.

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria

After the review, Approved with Changes disposition

8 8.1 If Approved with Changes disposition Make the recommended changes and/or resolve actions, and return the document to QA representative. Verify closure of change/action. Deliverable Owner

8.2

QA Representative (QA/Project Manager) QA Representative (QA/Project Manager) QA Representative (QA/Project Manager) QA Representative (QA/Project Manager) QA Representative (QA/Project Manager) QA Representative (QA/Project Manager)

8.3

Complete review verification and exit

8.3.1

Draft Review Verification Form.

8.3.2

Sign Verification Form.

8.3.3

Place deliverable in configuration management

8.3.4

File Log of Recommended Actions/Changes Form and Review Verification Form in the Project Library.

After the review, Rework disposition

9 9.1 9.2 If Rework disposition Make the recommended changes and/or resolve actions . Repeat review process. Deliverable Owner Deliverable Owner

6 Review Dispositions
The Review Team recommends one of the following dispositions at the review meeting conclusion to the Deliverable Owner: Approve: the deliverable is approved as is by the review team. Approve with Recommended Changes: if 1) all recommended changes and actions can be easily addressed and 2) the rework and actions are understood by both the reviewers and the Deliverable Owner and both parties agree that no further reviews are needed, the Deliverable Owner will make the changes and resolve the actions, and the QA Representative will verify the closure of these items.

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria Rework: if recommended changes are required that significantly alter the deliverable, the deliverable will enter the rework phase, and the same group of participants will be asked to review the reworked document.

7 Exit Criteria for Reviews

In order to closely manage the process, the exit criteria for the process must be clearly defined. The exit criteria for the Deliverables Review Process includes: Items logged on the Log of Recommended Changes and Actions Form have been addressed and verified as complete. Review Verification Form is completed and signed. The deliverable was placed under configuration management system. Completed Log of Recommended Changes and Actions and Review Verification forms are placed in the Project Library.

8 Acceptance Process for Project Deliverables

The acceptance process for (Project Name) provides a roadmap for incremental acceptance by the customer of the software application and associated project deliverables at the following key milestones. Project Phase Concept Complete Phase Requirements Complete Phase Design Complete Phase Application Ready For Pilot Phase Application Ready For Statewide Rollout Phase Complete

The following project deliverables are subject to acceptance within the context of the above milestones.

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria Milestone Project Phase Concept Complete Deliverables Project Initiation and Implementation Document, Software Project Management Plan Software Requirements Specification Template Software Design Specification Application, Software Test Plan, Software Transition Plan, Training Plan, Users Handbook, Business Continuity Plan Application, Software Test Plan, Software Transition Plan, Training Plan, Users Handbook, Business Continuity Plan Closeout Review, Lessons-learned

Phase Requirements Complete Phase Design Complete Phase Application Ready For Pilot

Phase Application Ready For Statewide Rollout

Phase Complete

The following table defines the sequence of activities that must be performed in support of the acceptance process and who is responsible for those activities. Activity Define acceptance criteria for milestones and deliverables in the current project phase Identify and plan for verification and validation activities necessary to support acceptance criteria for deliverables subject to acceptance Complete project deliverables for milestone Ensure completion of any necessary verification and validation activities for deliverables Individual(s) Responsible QA Manager, Project Manager, IS Sponsor, and Business Sponsor(s) for current phase QA Manager and Project Manager

Project team members responsible for project deliverable QA Manager and Project Manager

7/18/2013 8:04 A7/P7

Deliverable Review Process and Acceptance Criteria Activity Complete the Request for Acceptance form (Appendix A, first three sections). Refer to Section 4, Acceptance Criteria and Approval Required for Document Deliverables, for guidelines on document content and approvers. Forward the Request for Acceptance form attached to the deliverables for the milestone and any outputs from verification and validation activities to the list of approvers for the milestone and its deliverables . Schedule and conduct a meeting for approvers to review the milestone and its deliverables with respect to their acceptance criteria. During acceptance review meeting, sign-off on acceptance and include any desired comments on signature page for milestone/deliverables. (Note: Rejected deliverables are returned to the deliverable owner to rework along with the Request for Acceptance forms and will be reviewed for acceptance again once the necessary changes have been made. Reasons for rejection are documented on the signature form. If sign-off is not obtained within five (5) business days, then the project will proceed as though acceptance were obtained, and an issue will be logged to escalate the official acceptance.) Return Request for Acceptance Signature page to Project Manager Submit completed Request for Acceptance form and signature pages to QA manager for inclusion in the Project Notebook File Request for Acceptance form and signature pages in Project Notebook Place accepted deliverables in the Project Notebook QA Manager, Project Manager, and Business Sponsor(s) for current phase Project Manager Individual(s) Responsible Project Manager

Project Manager

Project Manager

QA Manager, Project Manager, IS Sponsor, and Business Sponsor(s) for current phase

QA Manager QA Manager

7/18/2013 8:04 A7/P7

10

Deliverable Review Process and Acceptance Criteria Activity Ensure the new version of the deliverable is in the Project File repository and is marked as the current version Report overdue Request for Acceptance signature pages as issues on status reports to management Individual(s) Responsible QA Manager

Project Manager

7/18/2013 8:04 A7/P7

11

Deliverable Review Process and Acceptance Criteria

9 Acceptance Criteria for Milestones and Deliverables

The acceptance criteria in the table below define the conditions under which the PROJECT Business Sponsor(s), the PROJECT IS Sponsor, and the Project Manager agree that they will accept completion of the milestones and deliverables subject to these acceptance criteria. Milestone Project Phase Concept Complete Deliverable Project Initiation and Implementation Document Acceptance Criteria Document has been reviewed and approved by: Prioritized scope and high level requirements have been reviewed by the Design Team, which consists of stakeholders from the Business Sponsor organization in addition to key members of the project team. Note: Acceptance of this document is completed by signing off on the documents signature page rather than a Request For Acceptance form. Project Phase Concept Complete Software Project Management Plan Document has been reviewed and approved by: Prioritized scope and high level requirements have been reviewed by the Design Team, which consists of stakeholders from the Business Sponsor organization in addition to key members of the project team. Note: Acceptance of this document is completed by signing off on the documents signature page rather than a Request For Acceptance form. Phase Requirements Complete Software Requirements Specification The Software Requirements Specification describes what capabilities the application should have and includes:

Business process Business requirements Use cases Functional requirements Data requirements Non-functional requirements

The Design Team has reviewed the Specification and

7/18/2013 8:04 A7/P7 12

Deliverable Review Process and Acceptance Criteria Milestone Deliverable Acceptance Criteria its prioritized requirements for correctness, completeness, and consistency with respect to the sponsor organizations business processes and business needs within the scope established in the Project Initiation and Implementation Document. The application development team has reviewed the Specification to ensure its sufficiency for beginning application design and to determine which requirements can be met within the constraints of the current project phase. The software test team has reviewed the Specification to ensure that all requirements are testable. The project manager has reviewed the Specification to ensure that all requirements are traceable to the scope, goals, and objectives established in the Project Initiation and Implementation Document. Phase Design Complete Software Design Specification The Software Design Specification describes how the application should function and be constructed to provide the capabilities defined in the Software Requirements Specification. This document includes:

Screen flows Screen designs Form and letters design Report designs Logical data model Physical data model User roles Security model Data mapping Detailed system interface designs

The Design Team has reviewed the Specification for correctness, completeness, and consistency with respect to the prioritized requirements established in the Software Requirements Specification.

7/18/2013 8:04 A7/P7

13

Deliverable Review Process and Acceptance Criteria Milestone Deliverable Acceptance Criteria The business team has reviewed the specification to ensure that all requirements are fully represented in the design and that the design includes no items that are not part of the established requirements. The application development team has reviewed the Specification to ensure its sufficiency for beginning application development and to validate the feasibility of implementing the design within the constraints of the current project phase. The software test team has reviewed the Specification to ensure that the design is testable. Phase Application Ready For Pilot Software Test Plan The completed application was delivered into the test environment with functionality as specified in the Software Requirements Specification, Supplemental Specification for Nonfunctional Requirements, and Software Design Specification. The completed application passed through system testing with the following results: 1. All test cases completely executed for functional modules with a system test priority ranking between 1 and 3 inclusive 2. Zero severity 1 (critical) defects 3. Zero severity 2 (major) defects in business priority 1 functional modules 4. No more than 2 severity 3 defects (minor) per business priority 1 functional module and no more than 20 severity 3 defects total in business priority one functional modules 5. No more than 1 severity 2 defect per business priority 2 functional module and no more than 5 severity 2 defects total in business priority 2 functional modules 6. No more than 5 severity 3 defects per business priority 2 functional module and no more than 40 severity 3 defects total in business priority 2 functional modules In the above criteria, business priority refers to the business priority assigned to a functional module in

7/18/2013 8:04 A7/P7

14

Deliverable Review Process and Acceptance Criteria Milestone Deliverable Acceptance Criteria the Software Requirements Specification, where priority 1 is business critical, priority 2 is moderately important, and priority 3 is optional. Severity 1, or Critical, defects include issues that result in degraded system performance beyond stated performance criteria, deny access to functionality, and/or corrupt data or display data incorrectly. Severity 2, or Major, defects include issues that prevent a user from correctly completing a task in the system, but can be managed by a workaround, and/or issues that promote data errors or reduce data quality substantially. Severity 3, or Minor, defects include issues that interfere with, but do not prevent a user from performing normal tasks. These are frequently usability or performance issues or defects that cannot be reliably reproduced. These can also be issues that represent an inaccuracy in the application but have no impact on functionality or performance, such as spelling errors, inconsistent fonts, etc. For user acceptance testing, the completed application was delivered in the same condition or better as when it was delivered with the test fixes required to meet the system test acceptance criteria described above. The completed application passed through all acceptance testing scenarios with no severity 1 defects in any functional module and no severity 2 defects in any business priority 1 functional module (priorities and severities as defined above). Acceptance testers agree that the application can move into pilot rollout with all the known nonminor defects or with a limited number of defect fixes that can be completed and tested by the project team in no more than two (2) business days. The accepted application has been moved from the test environment to the pilot environment with all test data removed and is functioning properly (as good as or better than it was during acceptance test).

7/18/2013 8:04 A7/P7

15

Deliverable Review Process and Acceptance Criteria Milestone Phase Application Ready For Pilot Deliverable Users Handbook Acceptance Criteria The Users Handbook describes in detail the procedures for using all of the functionality provided in the application in terms understandable to the typical user. The Software Transition Plan describes the tasks and activities that need to take place to efficiently and effectively move the application from the development or Pilot environment to the production, operations and maintenance environment and to integrate use of into the business processes of the impacted organization(s). The Plan includes deployment schedules, resource estimates, identification of special resources and staffing. The transition plan also defines management controls and reporting procedures, as well as the risks and contingencies. An impact statement outlining the potential impact of the transition to the existing infrastructure, operations, and support staff and to the user community is included. The Plan has been reviewed and approved by the operations, maintenance, and support organization, including Technical Services. Phase Application Ready For Pilot Phase Application Ready For Pilot Training Plan The Training Plan describes what training will provided to users to prepare them to use the application and how this training will be performed. The Business Continuity Plan describes how business functions supported by will be performed in the event that the application is unavailable for a period of time. Any application issues that the Design Team and Pilot users identified as needing to be addressed prior to statewide rollout have been addressed. The application still meets or exceeds the acceptance criteria established for the Phase Application Ready For Pilot milestone.

Phase Application Ready For Pilot

Software Transition Plan

Business Continuity Plan

Phase Application Ready For Statewide Rollout

Application

7/18/2013 8:04 A7/P7

16

Deliverable Review Process and Acceptance Criteria Milestone Phase Application Ready For Statewide Rollout Phase Application Ready For Statewide Rollout Phase Application Ready For Statewide Rollout Phase Application Ready For Statewide Rollout Phase Complete Deliverable Users Handbook Acceptance Criteria The Users Handbook was updated to include any changes necessary to support application changes made after the Pilot. The Software Transition Plan was updated to include any changes necessitated by problems with the Pilot rollout. The Training Plan was updated to include any changes necessitated by problems with the Pilot training. The Business Continuity Plan was updated to include any changes necessitated by problems with the evaluation of business continuity procedures during the Pilot. All project activities defined in the Project Initiation and Implementation Document and any approved change requests have been completed. All users have been trained and provided access to the application as specified in the Software Transition Plan and Training Plan. The Project Closeout Review document includes all project outcomes, costs, and lessons learned for the current project phase.

Software Transition Plan Training Plan

Business Continuity Plan

Closeout Review document

7/18/2013 8:04 A7/P7

17

Deliverable Review Process and Acceptance Criteria

10 Approvals
The signatures below indicate the approval of the (Agency) (Project Name) Deliverable Review Process and Acceptance Criteria process and document.

Signature _____________________________

Date _______________

Signature _____________________________

Date _______________

Signature _____________________________

Date _______________

7/18/2013 8:04 A7/P7

18

Deliverable Review Process and Acceptance Criteria

Appendix A: Log of Recommended Actions/Changes

Deliverable/Product Owner: Moderator: Deliverable/product: Participants: Review Date: Rev/Version:

Assigned To

Actions/Changes

Resolution

Verify Closure

Moderator ________________________________________ Date:__________ Disposition:

7/18/2013 8:04 A7/P7

19

Deliverable Review Process and Acceptance Criteria

Appendix B: Review Verification Form

Deliverable/Product Name and Purpose: Deliverable/Product Owner: Revision/Version Number: Reason for Review:

Verification:

_____________________________________ QA/Project Manager

_______________________ Date

7/18/2013 8:04 A7/P7

20

Deliverable Review Process and Acceptance Criteria

Appendix C: Request for Acceptance Form Request for Acceptance

Section 1 Date: Submitted By: Submitted To: Project: Section 2 Deliverable Description: (Provide a brief description of the milestone and deliverables and any necessary comments.)

Section 3 Signatures: (List individuals whose signatures are required for deliverable in Section 2. Project Manager will input date of signature.) Approval ___________________________ ___________________________ ___________________________ Date ________________ ________________ ________________

7/18/2013 8:04 A7/P7

21

Deliverable Review Process and Acceptance Criteria

Signature Page for Request for Acceptance

Signature Name

Date Title

Rejection Comments: Insert comments explaining rejection of deliverable and the date of rejection: Date: ______________ Comment:

General Comments: Insert any comments and date of comment: Date: ______________ Comment:

7/18/2013 8:04 A7/P7

22