Assessing Quality

Systems

An Overview
Course Objectives
 At the conclusion of this course, participants
will be able to:
 Discuss
 the purpose of Quality Systems Assessments
(QSA)
 the benefits of conducting QSAs
 how QSAs differ from other types of assessments
 Demonstrate a process for planning,
conducting, and reporting the results of QSAs
Quality Systems Assessment
 WHAT
 a systematic, independent, and documented
examination that uses specified criteria to
assess quality systems
 WHY
 EPA Order 5360.1
 48 CFR Part 46 and 40 CFR Parts 30, 31, and
35
Assessment vs. Audit
 Assessment
 the evaluation process used to measure the
performance or effectiveness of a system and
its elements
 Audit
 a systematic and independent examination to
determine whether quality activities comply
with specifications and whether these
specifications are implemented effectively
Types of Assessments
Process

Project/Program

System

Process Audits
- Field and Lab
- I.T.
- Databases
-Software
Project/Program
- Project
-Technical
QSA Perception Check
True or False

 Planning and conducting a QSA requires no expertise

in an organization's activities and how it operates

 QSA results will tell management where to look to

uncover problems (e.g., performance issues, fraud)

 A QSA can serve as a key deterrent to organizational

improprieties
 New organizations don’t need an assessment of the
quality system
 A QSA can indicate the effectiveness of
communicating quality policies in an organization
QSA Benefits
QSA results can be indicators of the:
 Adequacy and implementation of programs
developed to achieve an organization’s
mission
 Quality of products and services
 Compliance with governing specifications
Practice What You Learned
For each scenario, determine:
• What type of assessment is appropriate (system, project/program,
process)?
• Should the assessment be internal or external?
• What benefits may be achieved from the assessment?

 A recent grantee has water sampling scheduled through a QAPP that

the funding organization has just approved. Their QMP had few details,
and it is suspected that they have inadequate processes for overseeing
the work.

 A large organization, with many divisions and several QA staff have

recently amended their QMP to change the way they delegate QAPP
approval.

 An organization had been subject to regular assessments. Each set of

assessment results contains the same findings for insufficient
documentation of the quality system. However, improvement in the
project management QA responsibilities is noted.
Assessment Framework
QSA Results

Reporting

Conducting

Planning
The Assessment Cast
 Authorizing Entity
 Assessee
 Assessment Team Leader
 Assessors
Authorizing Entity
Roles and Responsibilities Actions BEFORE the QSA
 Authorize and provide  Authorize the QSA and
resources for the provides resources
assessment  Approve the
 May set the scope of assessment plan
the assessment
 Review assessment
results and ensure
corrective actions
The Assessee
Roles and Responsibilities Actions BEFORE the QSA
 Acknowledge the QSA  Provide preliminary
communications documentation and
 Work with the assessor scheduling assistance
to accomplish the QSA  Review the assessment
plan
Assessment Team Leader
Roles and Responsibilities Actions BEFORE the QSA
 Select team members  Understand the need,
authority and resources
 Prepare the
for the QSA
assessment plan
 Prepare written and oral
 Complete and submit communication, including
the assessment report assessment plan
 Address comments to  Prepare
the report interview/document
review checklists
The Assessors
Roles and Responsibilities Actions BEFORE the QSA
 Identify and accurately  Understand the purpose
document of the QSA and
 non-compliance assigned areas
 noteworthy practices  Review relevant
 Review and comment documentation
on draft reports to  Assist with assessment
ensure accuracy plan and checklists, etc.
Assessment Planning
"Would you tell me which way I ought to go from here?" asked Alice.
"That depends a good deal on where you want to get," said the Cat.
"I really don't care where" replied Alice.
"Then it doesn't much matter which way you go," said the Cat.

Lewis Carroll, Alice's Adventures in Wonderland (1865), novelist and

poet (1832-1898).
Initial Planning
 The initial planning process involves:
 gaining knowledge and understanding of the
assessee organization
 determining the lines of assessment inquiry
 formalizing assessment objectives
 considering available resources
Initial Planning Steps
 Determine criteria for the assessment
 Determine the assessment scope and
available resources
 Select the assessment team
Initial Planning Steps
Determine Assessment Criteria
Assessment criteria can be set by policy or
directives:
 Quality System Specifications
 Order 5360.1
 ANSI/ASQ E-4
 Assessee specifications/policy
 the assessee’s Quality Management Plan
 assessee reports
 Regulatory requirements
 contracts and assistance agreements
 program specific requirements
Initial Planning Steps
Determine the Assessment Scope
 The scope is set by the:
 directive of the authorizing entity
 allocated resources
 onsite considerations
Initial Planning Steps
Select the Assessment Team
 Select team members that:
 have appropriate skills and knowledge
 are competent and credible
 have no conflict of interest
Documenting the Assessment Process
 The Assessment File contains:
 planning documents
 relevant correspondence
 working papers
 assessment reports
Documenting the Assessment Process
Tracking Tools
 HOW
 Tools for managing document flow and status
from planning, observations and actions
taken, and filing.
 Paper notation
 Electronic data base
 WHY
 Papers flow through many channels
 A need to effectively manage assessment
information
Assessment Communications
 Initial contact with the assessee to begin
scheduling
 Approval of the assessment plan by the
authorizing entity
 Formally notify the assessee of the planned
assessment
 Reporting, tracking corrective actions, close-
out
Case Study – Part I
 Review the Case Study
 Appoint a spokesperson
 Complete the Initial
Planning Worksheet
 Detailed Planning

Whatever failures I have known,

Whatever errors I have committed,
Whatever follies I have witnessed in private and
public life Have been the consequence of
action without thought.
Bernard Baruch, stock broker, advisor to presidents Woodrow Wilson, Harry S. Truman,
Detailed Planning
 Information Review
 Issue Selection
 Identifying Interviewees and Onsite
Documentation
Information Review
 Review the organization’s quality system
 Quality Management Plan
 Applicable regulations for environmental
programs
 Reports from previous assessments
 Quality Assurance Annual Report and Work
Plan/Fiscal reports
 Published products
 Information about personnel (chart, CVs)
Information Review cont’d
 What can the documentation tell you?
 QMP
 Mission/quality policy
 QA support/resources
 Specifications of the quality system- roles (e.g., description of programs
and products)
 Regulations
 QA context for work
 Previous results
 Prior problems
 Annual reports
 Resources, activities, plans
 Fiscal reports
 relative importance
 Published products
 conformance, status
 Staff information
 qualifications, experience
Issue Selection
 Items that help to define/select assessment issues:
 assessment issues issues pre-defined by the
authorizing entity.
 quality system processes affecting costly or important
products
 quality system specifications having a major effect on
environmental data operations
 processes appearing difficult or expensive to
implement
 recently implemented policies and procedures
 processes having been problematic for similar
organizations
For Example
 Roles and responsibilities from the QMP
 The Envirumdata Group, composed of 4
programmers and 2 data entry professionals, is
responsible for ascertaining the quality of all
analytical data entered into the organization’s
database
 Specification of the quality system
 All data (collected by other organizations) in the
database must be assessed prior to use
 Relevance of specification to the program
 Data are used in support of rule making
Resulting Questions/Issues
 What is the process for data quality assessment?
 How is the process documented?
 How are the results documented?
 Do they have adequate expertise to assess data
quality?
Tips for Issue Selection
Look for:
 incomplete and unclear quality system
documentation
 processes that seem too extensive for the
organization’s size and mission
 processes that are not clearly described
either in full or by reference
 end products that lack review and approval
processes
 issues that pass the “So What” test
Where do you draw the line??

RISK
RISK

RISK

RISK
QSA Data Gathering
 Primary QSA data gathering tools
 Interviews
 Interaction with the interviewee provides the team
with direct information about the quality system.
 Document reviews
 Supporting documentation can be consulted and

questions explained and clarified as necessary.

Remember: The right people need to be asked the

right questions.
QSA Data Gathering
Select the Right Documents and the Right People

 Define what documents contain information relevant

to the selected issues. Include information that:
 may be requested prior to the onsite
 will be available during the onsite review

 Identify the appropriate interviewees for the

selected issues
 usually those who dictate the specifications for, create,
contribute to, review, approve or use the
documentation.
QSA Data Gathering
Get the Right Type and Number
Selection considerations:
 size of organization
 primary functions
 personnel factors, including availability,
experience, knowledge of issues selected,
length of time in position, number in each job
category
 document factors, including importance of
work products, size of document pool,
appropriateness to the time period and
process being assessed.
Think About It…
 Your assessment plan instructs you to ask 26
people whether they are familiar with their
organization’s QMP. What assessment finding
could you make?
 You review QA documentation processes and
understand the specifications (what is required,
roles and responsibilities for preparation, review,
approval, and record keeping according to the
QMP). You then examine 26 examples and ask
interviewees about the process they followed.
What assessment finding could you make?
LUNCH!
CASE STUDY PART II
AFTER the PLAN is Done……
 Assessment plan approval
 Confidentiality considerations
 Assessment tools
Review and Approval of the
Assessment Plan
 Who?
 Your management
 Authorizing entity
 Why?
 Provides an opportunity to raise questions
about the approach
 Approval of the plan affirms the authority,
credibility and scope of the assessment
process
 Encourages buy-in and ownership
Confidentiality Considerations
 Dissemination of assessment findings
“generally only to involved parties.”
 Interview notes are discarded when the
report is final
 CBI – important in RCRA, CAA, FIFRA
Assessment Tools
Checklists
 Used to:
 organize interviews and document review
 assure completeness and consistency in
reviews
 promote objectivity among assessors
 Design the checklist to:
 fill information gaps in Quality System
information
 verify Quality System specifications
 address specific observable practices
Assessment Tools
Matrix
 Suited for complex assessments
 Serves as a visual check for the assessment team;
shows where noncompliance may be clustered or
how wide spread a practice may be
Ready, Set………………
Just as there are three R's there are also
three A's of business life. They are:
Ability, Ambition, and Attitude. Ability
establishes what a worker does and will
bring him a paycheck. Ambition
determines how much he does and will
get him a raise. Attitude guarantees
how well he does.
Wilbert Sheer
Conducting the Assessment
 Logistics
 Onsite Activities
 Reporting
Logistics
 BEFORE YOU GET THERE, CONFIRM:
 Building access
 Assessment schedule/issues
 Opening and closing Meetings
 Interview schedules
 Assessment team meeting times
 Availability of meeting rooms/work space
 File review access

 TO GET THERE
 Travel (arrival and departure) itinerary for all team members
 Lodging (phone, address)
 Local city transportation (maps, taxi rates)
The Onsite
Conduct:
 The Opening Meeting
 The Onsite Review
 Documents and Records Review
 Interviews
Opening Meeting
 Typical Format
 Introductions
 Brief description of the scope of the
assessment
 Explain the assessment team’s schedule
 Answer any questions regarding the
assessment
 Limit time to one hour or less
The Onsite Review
 Document and Records Review
 Review documents listed in the assessment
plan
 Focus on documents that verify policies and
processes
 Verify supplemental information (e.g.,
databases, archived information)
 With permission, make copies of relevant
information
Interviews
 Elements of the interview
 Planning
 Opening
 Conducting
 Closing
Planning
 Establish information needs
 What information is needed
 Who needs to be interviewed
 Format assessment questions
Types of Questions
 Open ended
 Prompt the speaker to provide detailed
information
 Close ended
 Questions that offer a set of alternative
answers (e.g., yes or no, true or false, scaled
answers)
Formulating Questions
To find out what they know Ask them to describe
something

To find out how they do Ask them to describe an

something activity

To find out what they think Ask them for an opinion

Tips
 Questions should be:
 as neutral as possible
 asked one at a time
 worded clearly

 Use "why" questions sparingly

Interview Scheduling
 Scheduling considerations
 Schedule in advance
 Plan no more than one hour for each interview
 One on one interviews (preferred)
 Try to avoid the hour right before or after lunch
Opening the Interview
 Sets tone for the interview
 Make introductions
 Keep small talk to a minimum
 Re-state the purpose of the interview and
how you will proceed
Opening the Interview cont’d
 Tell them:
 who will get access to their answers and how
their answers will be analyzed
 the format of the interview
 how long the interview usually takes
 how to get in touch with you after the interview
 that you will take notes during the interview
Note Taking Tips
 Use a notepad
 Record main ideas
 Periodically read back notes to check for
accuracy
 Notes are about PROCESSES, not PEOPLE
Conducting the Interview
 Remain as neutral as possible
 Encourage responses
 Control your behaviors when note taking
 Provide transition between major topics
 Don't lose control of the interview
Closing the Interview
 Briefly summarize your information
 Thank the interviewee for his/her time
 Ask if the interviewee would be willing to
follow up with you if necessary
 PRACTICE WHAT YOU LEARNED
The character of the questions you ask will influence the
appropriateness of the answers.

 You ask the interviewee about his interaction with the QA

Manager and he immediately displays resentment. You ask…..
"So I guess you don’t like him much, huh?"

 You should ask……….

 You ask a database operator ….. "Do you know how to work
this thing?"

 You should ask……………………..

PRACTICE WHAT YOU LEARNED
 One More for Good Measure!
 You ask a field sampler…. "Have you been
following this SOP?“
 You should ask……………………….
Think About It
Based on your draft assessment plan, formulate 2
questions for each of the following categories:

- Are Quality System policies implemented by staff

and management?
- Are the Quality System policies effective?
- Do personnel understand the Quality System
requirements?
Working Papers
 Assessment Working Papers
 Notes from document review
 Copies of documents (with permission!)
 Interview notes
 Cross reference to criteria
 Date each document
Now What???
 Meet with your team each day to review
working papers
 Develop preliminary findings
Finding

An assessment conclusion that identifies a

condition having a significant effect on an item
or activity
Other Assessment Results
 Noteworthy practices
 Positive practices implemented by the
organization
 Observations
 Practices noted during the review. No
corrective action required.
Example: Finding 1
 The QA Coordinators need training in QA
concepts and theories
Example: Revised Finding 1
 The QA Coordinators are unable to perform
the review and approval of QAPPs as
specified in the QMP because they have not
been trained in the process.
Draft Preliminary Findings
 Presented to summarize what you have
observed during the onsite
 Observations should be factual and to the
point
 Findings can change during follow up review
(HINT - You’re not finished!)
Exercise!
Closing Meeting
 Brief management on your preliminary
findings
 Explain your reporting process and timetable
 Remember:
 No Surprises!
 Preliminary findings must not be frivolous
 Keep the meeting to no more than 1 hour
 Try to address immediate concerns but DON’T
 Make promises you can’t keep
 Usurp your management’s authority
The Black Hole Phenomenon
The Report
 The assessment report communicates
assessment findings to management. It
should clearly identify:
 the type of assessment,
 the assessor,
 the assessee,
 what was assessed,
 the findings,
 if requested by the authorizing entity,
recommendations for corrective actions
Effective Reporting
 Clear and concise writing
 Address significant deficiencies in a
constructive manner
 Report the organization’s actions, not the
actions of specific individuals
 Prioritize assessment findings and any need
for corrective actions
Example Report Outline
I. Background and Purpose
II. Assessment Scope
III. Assessment Results and Rationale
References
Appendices
 A. List of Personnel Interviewed
 B. List of Documents and Records Reviewed
(optional)
 C. Corrective Action Plan
Close Out
Close Out
 BEFORE you can close the assessment you
need:
 A response from the assessed organization
with an acceptable corrective action plan (if
necessary) and an indication that the
corrective actions have been completed
 A review of the assessment file to ensure that
it is complete before it is archived.
SUMMARY
 Now you can discuss:
 the purpose and benefits of Quality Systems
Assessments (QSAs)
 how QSAs differ from other types of assessments

 You can demonstrate a process for planning,

conducting, and reporting the results of QSAs

I can give you a six-word formula for success:

"Think things through - then follow through."
Edward Rickenbacker