You are on page 1of 11

Week 7

Course Project

Cynthia Mortenson

Week 7
Course Project

Program Evaluation
Instructor: Dr. Pratt

Cynthia Mortenson
February 27, 2015

Page 1 of 11

Week 7

Course Project

Cynthia Mortenson

Program Analysis
Description of the program. The subject of this program evaluation is the
Neuromodulation Quality Core Curriculum which consists of several training elements, both in
a read and acknowledge format as well as eLearning modules intended to educate and train
our employees to the policies and procedures governing our Quality Management System to
ensure they have the skills needed to perform their responsibilities. Because we work in a
regulated environment, an effective quality systems training program is essential to the success
of our business and the safety of our patients. Both internal and external audits have identified
nonconformances to our policies and procedures which could have a significant impact upon our
business. These nonconformances have resulted in observations and a warning letter from the
FDA. Ineffective training has been identified as the root cause.

Goals/Objectives. The goal of this program evaluation would be to identify issues or gaps
in the current training curricula in order to implement a plan for defining training requirements
that will help functions incorporate the necessary quality assurance and quality control skills
needed to effectively perform their jobs. In order to be as effective as possible, the training
content should be sequentially and efficiently ordered in order to help learners achieve
objectives. The content should be presented in a linear fashion that arranges concepts
according to their superordinate (e.g., corporate mandated training), coordinate (top-level
procedures) and subordinate relationships (e.g., work instructions).

Operations Outcomes. Ultimately, an effective training program that adheres to our Quality
System ensures that we comply with regulations set forth by regulatory agencies such as the
FDA and ISO so that we achieve the highest standard of quality and reliability in our products
and ensures that all regulatory, corporate and business unit responsibilities are met.

The depth

Page 2 of 11

Week 7

Course Project

Cynthia Mortenson

of each employees knowledge and skills contributes to a function or departments ability to

achieve its mission and goals. Thus, all personnel should be trained to a level that advances our
commitment to deliver on our Quality Policy by delivering an effective and efficient training
program. Additionally, by improving the quality of delivery mechanism (e.g., instructor-led
training vs. eLearning) could result in more efficient use of employees time and the companys

Performance History. There are many examples as evidenced by both internal and
external audit findings that there is a training-gap with many of our quality policies and
procedures. Our Quality System is under enormous scrutiny by regulatory agencies and will
continue to be as it is viewed as a gauge for how we conduct our business.

Stakeholders. Primary stakeholders include the Operations Quality Council (OQC), Quality
Systems Board (QSB), Training Review Board (TRB) as well as the VP and Director of Quality.
Fortunately, because of the potential ramifications of not correcting the issues with our Quality
Systems training program, we already have stakeholder engagement and investment. Keeping
them engaged is key, because without their support (based on the results and recommendations
that stem from the evaluation) we wont be able to act on the results or implement the

Potential Ethical Challenges Involved Some of the quality issues identified have
surfaced from customer complaints through our Customer Service and Vigilance Department.
Any data originating from patients is protected by HIPPA and we would need to obtain an
informed consent form from them

Page 3 of 11

Week 7

Course Project

Cynthia Mortenson

Evaluation Model Table







Typically used by government

agencies and consumer
advocates and is typically a
summative evaluation approach
advocating consumer education
and the independent review of

Outputs are valuable

information given to those
who dont have time to
conduct their own research
while advancing
consumers knowledge of
appropriate criteria for
selection of

Dependent upon outside

consumer services which
can potentially increase
product cost. Strict testing
may hinder creativity.

Primarily dependent upon

professional expertise to judge
an institution, program, product
or activity. Very subjective as
an evaluation tool. Examples
include doctoral exams, board
reviews, accreditation,
reappointment/tenure review,

Experts in the field in are

well-versed to make
decisions (SMEs) set
standards and, encourage
improvement through selfstudy.

Personal bias can be a

potential weakness.

This evaluation approach is

designed to determine whether
the objectives of the program
were achieved and if so, to what

A major benefit of this

approach is its simplicity
outcomes (i.e. is the program
doing what the designers
intended for it to do?). Data
derived from this type of
evaluation can influence
whether to maintain,
terminate, or change
course within the program.
Based on the established
goals or objectives and the
defined behavioral terms,
an examination of the
achievement of the stated
goals/objectives can be
shown by a selecting
measurement tool and
collecting performance
data followed by a
comparison of the data
with the behaviorally stated

This approach can sometimes

focus too heavily on program
goals and objectives, , and
not enough on making clear
judgments of a programs
worth (i.e., just because a
programs performance may
meet a specified goal or
objective, that doesnt
necessarily mean the
program is effective. Some
critics argue that goals and
objectives often act like
blinders to an evaluator,
causing him to narrow the
focus of the evaluation,
thereby neglecting important
outcomes not specifically
addressed by those goals and
failing to see a programs
unintended and sometimes
negative side effects
(Worthen, Sanders, &
Fitzpatrick, 1997).

Page 4 of 11

Week 7



Course Project

Cynthia Mortenson

These approaches focus on

evaluations role in providing
information to improve the
quality of decisions made by
stakeholders or organizations.

Among the oldest

approaches to evaluation
and are still frequently
used. Their longevity
speaks to their success.

The focus on decisions

can be viewed as a
drawback in that
stakeholders with less
influence may be
neglected could restrict
valuable information that
evaluators seek.
Evaluators may not be
able to respond to
questions or issues that
may be important because
they dont align with the
concerns and question of
the decision maker who is
the primary stakeholder for
the study.

Encourages multiple data

collection methods, provides
rich, persuasive information,
establishes dialogue with and
empowers quiet, powerless

This type of approach to

evaluation can help
improve program
performance by including
key stakeholders in the
design and decision
making process while also
acknowledging and
disproportionate levels of
power and voice among
stakeholders, It employs
multiple and varied
methods and aims to build
the evaluation capability of

This approach is
sometimes viewed as too
complex for practitioners
and is more appropriate for
theorists. Potential
disadvantages are t hat the
evaluation may be viewed
as less objective because
of stakeholder involvement
and could present
difficulties in addressing
highly technical aspects.
There is also theh potential
for misuse by some
stakeholders to further
their own interests but the
benefits of fully engaging
stakeholders outweigh
these concerns

Involves all stakeholders in the

evaluation and encourages
support for representation of all
participants. . Does not follow
a standard plan and records
multiple rather than single
realities (e.g., focus groups)

Page 5 of 11

Week 7

Course Project

Cynthia Mortenson

Selected Evaluation Model

Upon a thorough review of the evaluation approaches when applied to the
subject for my program evaluation, I selected the objectives-oriented evaluation
approach. With the primary goal of this program evaluation being to identify whether
the previously existing gaps (identified through internal and external audits) within our
Quality Management System (QMS) training program have been addressed with the
implementation of the new Quality Management System eLearning Series.
The training program was designed based upon specific training objectives
outlined by senior management in order to address issues (i.e., FDA findings and
observations and internal audit findings) which could result in devastating
consequences for our business. Because Quality Systems training so important to the
success of our business, adhering to the policies and procedures set forth by regulatory
agencies such as the FDA and ISO, the training was redesigned to ensure regulatory,
corporate, and Neuromodulation requirements are met. The primary goal of the training
program was to implement a plan for defining training requirements that will help
functions incorporate the necessary quality assurance and quality control skills needed
to effectively perform their jobs.
With the end goal of the Objectives-Oriented Evaluation Approach to determine
whether the objectives of the program were achieved and if so, to what extent, this is
the most feasible and practical approach for my program evaluation.
The new training program has recently been implemented and will be rolled-out to all new
employees. The evaluation period will commence six months from the date of implementation
Page 6 of 11

Week 7

Course Project

Cynthia Mortenson

whereby, the program evaluation team will work with primary stakeholders including the
Operations Quality Council (OQC), Quality Systems Board (QSB), Training Review Board (TRB)
as well as the VP and Director of Quality as well as our internal audit department. Due to the
potential ramifications of non-compliance, we already have stakeholder engagement and
investment. In order to determine if the goals of the new training program were achieved , an
examination of both process and product documentation and adherence to (i.e., including
observations) will be conducted to ensure compliance to existing top-level procedures,
procedures, guidelines, and work instructions. From a stakeholder perspective, the ultimate test
will be when the FDA returns for their follow-up audit to ensure that we have implemented the
action plan for addressing the observations and findings outlined during their previous visit.

Evaluative Criteria
The Program Evaluator and Internal Audit Representative will prepare questions
specific to the particular function being evaluated/audited in order to determine whether
the defined policies, procedures or requirements, including regulations for that particular
function have been met. (These policies, procedures and requirements will be used as
a reference tool against which audit evidence is compared. The Program Evaluator and
Audit Representative shall coordinate and ensures availability of auditee participants
during the on-site portion of the audit and will be responsible for assigning auditee
contacts and distributing a schedule to all affected contacts at the Audited Site. They
are also responsible for ensuring attendance of appropriate people to the relevant
meetings that occur at the Evaluation/Audit Site.

Page 7 of 11

Week 7

Course Project

Cynthia Mortenson

The Evaluation Program will cover the scope of the quality system and shall be
established on a risk based approach for each particular function. The focus of the
evaluation will be to identify individual topics and associated regulation and standard
elements with the intent of demonstrating coverage of the training program to
regulations and standards. This will include a determination of the extent of conformity,
evaluation of capability of a system to ensure compliance, evaluation of the
effectiveness of a system in meeting requirements, and identification of potential
improvements. Additionally, the Evaluation Program shall include follow-up on all prior
internal audit observations and findings (both internal and external) to identify any audit
topics needing comprehensive review (such as critical observations identified in
previous audits, repeat nonconformities identified in previous audits, or other
information from the risk assessment(s).
From our stakeholders perspective, the true test of whether or not the new training
program is effective will be whether it adheres to our Quality System by ensuring
ensures that all regulatory, corporate and business unit responsibilities are met.

Page 8 of 11

Week 7

Course Project

Cynthia Mortenson

Reporting Strategy Table





Vice President of

The Program
Evaluator along with
Internal Audit
Representative will
report findings
related to the
success or failure of
the training program
objectives to VP of
Quality during the
next quarterly
Management Review
following completion
of the 1st program
phase/internal audit.

Because recent
audits have identified
issues with our
records. That have
resulted in FDA-483
observations, we are
required to correct
them before our next
inspection. Failure to
do so would result in
a determination that
we are operating in a
state of noncompliance which
could result in
additional actions by
FDA, millions of
dollars spent in
and product recalls
which directly impact
our customers and
Responsible for the
management and
strategic direction
(e.g., initiating and
implementing a plan
to redesign the Core
Curriculum as well as
providing the budget
for it.
The consequences of
poor documentation
practices may include
rejected products,
adverse regulatory
consequences, loss
of valuable time to
investigate nonconformances, and
loss of critical

The Vice President of

Quality for
Neuromodulation is
appointed by executive
management as the
Representative with
responsibility and
authority to ensure the
Quality System is
implemented and

Regulatory action
could result in lay-


Operations Quality
Council and Quality
Systems Board

The Program
Evaluator along with
Internal Audit
Representative will
report findings
related to the
success or failure of
the training program
objectives to the
Operations Quality
Council during their
monthly meetings.
(Staus updates
would be provided to
the team as well as
the evaluation

All Employees

Program Evaluator
would work with the

Ultimately, the VP of
Quality is the
responsible party for
ensuring compliance
and reports into the
Vice President of
Quality at the
Corporate level.

This VP-level
governance forum of
the Neuromodulation
Quality System is
focused on operational
metrics and strategic
The QSB is
responsible for driving
confident compliance
and audit readiness,
representing process
areas with distributed
ownership across
QMS, and managing
future QMS releases.

Page 9 of 11

Week 7

Course Project

Quality Systems
Team to
communicate to all
employees the status
of the program
evaluation. Varied
methods would be
used to communicate
progress and status:
email, Monthly
employee meetings,
Quality website, etc.

offs, potential shutdown of

operations. It would
also affect employees
profit sharing

Cynthia Mortenson

encompasses the
whole Quality System
and includes
establishing and
maintaining an
effective Quality
System that is
compliant to external
regulations as well as
corporate policies.
We all share the
responsibility for
implementing quality in
our work. Because
ensuring patient health
and safety is our
biggest responsibility, it
is incumbent upon
every one of us to
ensure that we are
adhering to and
complying with the
procedures and
policies that are
embedded in our
Quality Management

Values, Standards, and Criteria:

Mission: The Companys Mission is focused on Quality and guides our actions in everything
we do. It represents our core values with respect to our customers, our employees and our
worldwide community. In fact, our Quality Policy stems directly from the Mission and is the
highest requirement that represents who we want to be as a company by setting the standards
for our success and providing a consistent product and a basis for continuous improvement
which is to contribute to human welfare by application of biomedical engineering in the research,
design, manufacture and sale of instruments or appliances the alleviate pain, restore health,
and extend life.
Quality Vision is to be the company most trusted to deliver excellence in products, processes,
services and relationships. Our Quality Policy provides the guidance for how we conduct our

Page 10 of 11

Week 7

Course Project

Cynthia Mortenson

day-to-day work and reflects our commitment to both the Medtronic Mission and Vision.
Quality Policy: We will exceed our customers expectations by striving without reserve for
unsurpassed product quality, reliability, and patient safety through effective, agile and compliant
processes. We are committed to maintain and continually improve the effectiveness of our
quality management systems, comply with all applicable requirements and deliver excellence to
customers through our products, processes, services, and relationships
Criteria: In response to both regulatory and Corporate requirements, our

Neuromodulation Quality System Manual (NQMS) was created and implemented. It is

our top-level procedure which describes the scope and structure of the
Neuromodulation Quality Management System (QMS). It establishes the requirements
for how we perform business processes by defining requirements for what, when,
where, how and by whom. The NQSM which describes the governance, process
structure, document structure, and processes to comply with these requirements. The
Core Curriculum was designed to address the policies and procedures that fall under
the umbrella of the NQMS.
Potential ethical issues: When reporting evaluation results, extreme caution should be
exercised as to not to call out particular departments or functions where non-conformances
have been found when reporting in an open forum (i.e., all employee meetings or in emails sent
out to all employees). Functional managers as well as management would be made aware of
where the issues are occurring but we would not want to subject members of those
teams/functions to public humiliation, if you will, or additional scrutiny by other employees.

Page 11 of 11