CHAPTER 17

Evaluation of Nursing Care with

Communities
Claudia M. Smith and Frances A. Maurer

CHAPTER OUTLINE

Responsibilities in Evaluation of Nursing Care with Communities

Responsibilities of Baccalaureate-Prepared Community/Public Health Nurses
Formative and Summative Evaluations
Community Involvement
Standards for a Good Evaluation
Steps in Evaluation
Questions Answered by Evaluation
Evaluation of Outcome Attainment
Evaluation of Appropriateness
Evaluation of Adequacy
Evaluation of Efficiency
Evaluation of Process
Uniqueness in Evaluation of Nursing Care with Communities
Criteria for Effectiveness
Sources of Evaluation Data
Documenting Evaluations
Analyzing Evaluation Data
Modification of Nursing Care with Communities
Evaluation Methods and Tools
Designs for Evaluation of Effectiveness
Tools for Evaluation of Effectiveness
Efficiency Analysis

Focus Questions
What are the responsibilities of a baccalaureate-prepared community/public health nurse in evaluation of nursing care with
communities?
What are the steps in evaluation?
What questions can be answered by evaluation?
What outcomes are indicators of the effectiveness of nursing interventions with communities?
How does evaluation of nursing care with communities compare with evaluation of care with families and individuals?
How can evaluation of process be used to improve the operation of nursing programs?
How is evaluation used to modify nursing care with communities?
What methods and tools are used in evaluation?
Key Terms
Adequate
Affective learning
Appropriate
Effective
Efficiency
Evaluation
Formative evaluation
Outcome measures
Result
Satisfaction
Summative evaluation
Evaluation of care with communities seeks to determine whether health has improved. Were the desired health goals reached? How
much progress was made toward the goals? What themes, patterns, and results emerged? What side effects were evident? How
have community competence and resilience been enhanced? To what extent are the community changes sustainable? Evaluation
provides information to help community/public health nurses improve the quality of their nursing practice.

Responsibilities in evaluation of nursing care with communities

Evaluation is the process by which a nurse judges the value of nursing care that has been provided. As with any type of nursing care,
the community/public health nurse seeks to determine the degree to which planned goals were achieved and to describe any
unplanned results.
The purpose of the evaluation is to facilitate additional decision making. An evaluation might conclude that what had been done
could not have been done better, that the goals were reached, and that the goals were mutually desirable to the nurse and the
community members. This conclusion would be cause for celebration. As a result of another evaluation, the conclusion might be
that alterations are needed in the plan of care to reach the desired outcomes more effectively; or possibly that, although goals were
reached, the cost in money, time, or other resources was too expensive for the nurse or the community members.
Evaluation is based on several assumptions: first, that nursing actions have results, both intended and unintended; second, that
nurses are accountable for their own actions and care provided; and third, that different sets of actions result in resources being
used differently (i.e., some nursing interventions use more resources than others).
Evaluation involves two parts: measurement and interpretation. Many different schemes or models exist for organizing ideas
about evaluation, which may result in confusion among people who use different terminology for similar concepts.
Basic to the nursing process, however, is the idea of measuring whether planned goals were achieved. Synonyms for this activity
and its result are outcome attainment (Donabedian, 1980), performance evaluation (Suchman, 1967), results of effort, and evaluation of
effectiveness (Deniston & Rosenstock, 1970). The question that the nurse attempts to answer is, “Were the planned goals achieved?”
Another basic idea addresses the quality of the results and the process that contributed to the results. Some terms used to
express this idea are as follows:
• Appropriate—suitable for a particular occasion or use; fitting
• Adequate—able to fill a requirement; sufficient or satisfactory
• Effective—producing an expected result; productive
Each of these terms describes different aspects of measuring quality. The following are some questions that may be asked about
quality. How and why did the interventions work? Were the nurses’ actions ethical? Did the nurses address the most important
goals? Did the nurses involve community members and recipients as participants (American Nurses Association [ANA], 2007)?
Were resources used wisely? How many needs and goals did the plan actually address?

Responsibilities of Baccalaureate-Prepared Community/Public Health Nurses

According to the American Nurses Association (ANA) (2007), community/public health nurses with bachelor’s degrees in nursing
are expected to work with advanced practice public health nurses (masters prepared) and community members in
evaluating responses of the community to nursing interventions.
The responsibilities of community/public health nurses for evaluating nursing care with communities vary, depending on the
size and complexity of the community and whether the community is geopolitical or phenomenological (see Chapter 15).
Baccalaureate-prepared community/public health nurses are expected to work with community members, advanced public
health nurses, and multidisciplinary teams to evaluate nursing care with geopolitical communities. Community/public health
nurses may also work with multidisciplinary teams and nurses who engage in quality assurance and accreditation reviews (ANA,
2007). Baccalaureate-prepared community/public health nurses will be more capable members of evaluating teams if they have been
introduced in their education to ideas and skills in evaluating nursing care with communities.
In some instances, a baccalaureate-prepared community/public health nurse will work with a small phenomenological
community, such as a senior center or school; in this case, the nurse is likely to evaluate his or her own performance with minimal
assistance from a supervisor and peers. Either independently or with help from supervisors, community/public health nurses are
expected to evaluate the effectiveness of intervention programs that involve teaching, direct care, and screening and referral.
Regardless of the type or size of community, the members themselves should, when possible, be involved in planning and
conducting the evaluation (ANA, 2007). The measurement of many health outcomes requires the judgment of the community
members themselves.

Formative and Summative Evaluations

When is nursing care with communities evaluated? Evaluation of the effectiveness of care that takes place after the interventions
have been performed is known as summative evaluation because the nurse is evaluating the sum, the bottom line, the end results.
Summative evaluation involves measurement of community responses to nursing care and interpretation of the degree to which
planned goals were met. Summative evaluation usually consists of measurement of outcomes and goal attainment.
After assessing the members of a senior retirement community, Ridge Center Retirement Community, a community/public nurse
designed a program with the goal of increasing the walking regimen for obese and hypertensive seniors living in Ridge Center. She
developed an intervention plan consisting of a health education program and personal consultation with a sports fitness expert. All
senior participants received a health clearance from their personal physician. A summative evaluation statement for the
intervention was as follows: the target group will increase their walking regimen from an average of 15 minutes three times a week
(data from the assessment) to 30 minutes three times per week.

Summative evaluation may also take place several months or years after nursing care has been provided. This evaluation seeks
to determine whether a long-term impact was made on the health status and the health responses of the community.
Formative evaluation is evaluation that occurs throughout the nursing process but before evaluation of the outcomes of care.
This evaluation occurs during the formation of the nursing care and during the process of its actual delivery. In other words,
formative evaluation considers the day-to-day provision of programs of nursing care. Formative evaluation allows ongoing
modification of nursing practices.
In the intervention program discussed earlier, the program goal remains the same. Several formative evaluation statements for the
interactive portion of the health education program are as follows:
• The discussion portion of the program was longer because of technical difficulties. Originally planned for half an hour, it was 30
minutes longer because the room was locked and printed handouts were delivered late.
• Some senior participants were unable to hear the speaker. Planning did not account for hearing loss in some of the participants. The
room was large, and although the group was small (n = 15), they were initially spread out over the entire room.

In the formative example just mentioned, the nurse can take action to remediate some of the problems identified during the
intervention process. For example, the nurse might take the names of participants and deliver the printed material to them at some
later date. The nurse had the seniors move into a small group in the room and found and used a microphone to help with the
presentation.

Community Involvement
Because the community members are involved in evaluation, at least part of the evaluation must occur in the clients’ community.
Mutuality is an important aspect of evaluation. Because much of the impact of the community/public health nurse is indicated by
self-care and lifestyle changes of community members, a nurse must document and validate outcomes directly with community
members. Additionally, although goals have been achieved, some negative or unexpected results might also have occurred. The
nurse must explore the perceptions of community members to discover and validate the meaning of the experience. Determining
how satisfied community members are with both the outcomes and the nursing interventions is important.
Stakeholders are individuals who have expectations about nursing care but who are not directly involved in its delivery. For
example, there are individuals whose approval was necessary, those who contributed money or supplies, those who volunteered to
assist, and those (such as competitors) for whom the presence of nursing services had an impact. Stakeholders in a community
immunization campaign might be the county health officer, a retail pharmacist who donates syringes, a local pediatrician who is
concerned about financial competition, and parents of persons who were immunized. Community health/public nurses need to
identify the stakeholders and invite them to participate in evaluation.

Standards for a Good Evaluation

Standards for evaluation of nursing care with communities have been formulated by the Quad Council of Public Health Nursing
Organizations and published by the ANA (2007):
• The employing agency is to provide supervision, consultation, and general evaluation plans for the baccalaureate- prepared
community/public health nurse.
• The community members are to participate in the evaluation.
• The nursing care is to be revised based on the evaluation.
• Evaluation is to be documented and disseminated so that the record can strengthen nursing practice and knowledge.
Steps in evaluation
Evaluation is a process that includes several steps: planning, collecting the data, analyzing and interpreting the data, providing
recommendations, reporting the results, and implementing the recommendations (McKenzie et al., 2009). Box 17-1 identifies
evaluation activities in greater detail related to each of the major steps.
Box 17-1

Steps in Evaluation

Plan the Evaluation

1. Review goals and objectives.
2. Meet with stakeholders to identify which evaluation questions should be answered.
3. Develop a budget for evaluation.
4. Determine who will conduct the evaluation.
5. Develop the evaluation design: What will be done?
6. Decide which evaluation instruments will be used to collect information.
7. Analyze how the evaluation questions relate to the goals and objectives.
8. Analyze whether the questions of stakeholders are addressed.
9. Determine when the evaluation will be conducted; develop a timeline.

Collect Evaluation Data

10. Develop specific processes for collecting data through questionnaires, review of records or documents, personal
interviews, telephone interviews, and observation.
11. Determine who will collect the data.
12. Pilot the data-collection instruments.
13. Refine the instruments based on data from the pilot.
14. Identify the sample of persons from whom evaluation data will be collected.
15. Collect the data.

Analyze the Data

16. Determine how the data will be analyzed.
17. Determine who will analyze the data.
18. Analyze the data, generate several interpretations, and make recommendations.

Report the Evaluation

19. Determine who will receive results.
20. Determine who will report the findings.
21. Determine format for the report, including an executive summary.
22. Discuss how the findings will affect the program.
23. Determine which findings will be included in the report.
24. Distribute the report.

Implement the Results

25. Plan how the results will be implemented.
26. Identify who will implement the results.
27. Determine when the results will be implemented; develop a time line.
From McKenzie, J. F., & Smeltzer, J. L. (1997). Planning, implementing, and evaluating health promotion programs: A primer (2nd ed.; pp. 276–277).
Boston: Allyn and Bacon. Copyright 1997 by Allyn and Bacon. Adapted by permission.

Questions answered by evaluation

Evaluation of nursing care with communities involves evaluation of programs of care for populations. Program evaluation includes
evaluation of outcomes (program goals and outcome objectives), as well as evaluation of the structures and processes used to
achieve the outcomes (Ervin, 2002). The ANA considers outcomes, structures, and processes as the primary categories of criteria to be
used to measure the quality of nursing care. Outcomes are the end results; structures are the social and physical resources;
and processes are the “sequence of events and activities” (ANA, 1986, p. 18) used by the nurse during the delivery of care. For
example, evaluation of a health program designed to identify adults with high cholesterol levels would include the following:
• Structure standard: Cholesterol screening will be available to all adults, regardless of whether they can pay for testing.
• Process standard: Cholesterol screening will be performed on all adults who come to the health screening event.
• Outcome standards:
(a) One hundred percent of the adults screened will be given their test results.
 (b) Eighty percent of adults with cholesterol levels above the recommended norm will follow up with a physician’s visit
for evaluation.
Table 17-1 describes the following five categories of questions that can be answered by evaluation: (1) outcome attainment, also
called effectiveness; (2) appropriateness of care; (3) adequacy of care in relation to the scope of the problem; (4) relationship of
resources to results, also called efficiency; and (5) process. This set of questions includes the criteria of outcome, structure, and
process evaluation and adds appropriateness and adequacy. Questions of appropriateness and adequacy evaluate the nursing care
program in relation to the community health needs. Efficiency addresses the relationship of outcomes to structures and processes.
Each of these sets of evaluation questions is discussed in more detail.
Table 17-1
Questions Answered by Evaluation

Variable Questions Examples of Measurement

1. Outcome
attainment
Did change occur? Numbers and rates of children immunized

To what degree was progress made toward the goal? Numbers of cases of cancer found on Papanicolaou smears

What are actual effects on clients? Changes in attitudes regarding people with acquired immunodeficiency
syndrome (AIDS)

What unintended outcomes occurred? Reduction in teenage pregnancy rate

2. Appropriateness

Did the goals fit the need? Plan of care compared with clinical nursing knowledge

Are the goals and plans acceptable to the community? Community preferences

Are the plans likely to achieve the goals? Plan of care is evidence-based

Does the plan duplicate existing efforts?

Variable Questions Examples of Measurement

3. Adequacy

To what degree does the intervention meet the total amount Rate of effectiveness multiplied by number of people exposed to service
of need?

Were some people not served? Outcomes relative to total needs in population

Degree to which need was a priority

4. Efficiency

What resources were used? Relation of effort to outcome

Cost-effectiveness Can a better way be found to attain the same results? Output and input:

What resources were necessary to attain results? Money

Time

Personnel

Client convenience

Benefit-cost analysis Do the benefits justify the use of resources?

5. Process

What did nurses do? When? Where? Number of clinics/or encounters/week or month
Variable Questions Examples of Measurement

How many people were reached? Number of home visits

What were the reasons for the successes or failures? Amount of money spent

What contributed to the results? Education content taught and strategies used

What methods were used? Numbers of people attending screening sessions

Data from Deniston, O., & Rosenstock, I. (1970). Evaluating health programs. Public Health Reports, 85(9), 835–840; Donabedian, A. (1980). The definition of
quality and approaches to its assessment (Vol. 1). Ann Arbor, MI: Health Administration Press; Freeman, R. (1963). Public health nursing practice (3 rd ed.).
Philadelphia: W. B. Saunders; and Suchman, E. (1967). Evaluative research: Principles and practice in public service and social action programs. New York:
Russell Sage Foundation.

Evaluation of Outcome Attainment

Evaluation of outcome attainment, also called effectiveness, addresses the results of nursing interventions. Change toward
predetermined goals, as well as unplanned effects, may have occurred (see Table 17-1). Frequently, large health programs are
evaluated as a total intervention, without distinguishing the effects of nursing interventions from the effects of other health
disciplines and program components. Therefore nursing care may be lumped into a single evaluation for the whole program rather
than being evaluated as a separate intervention. Devising evaluation strategies and criteria for each component of a program is
more useful because evaluators are given a better idea of which strategies are effective and which might need to be revised or
eliminated. Evaluators can then determine nurse-sensitive outcomes. This is also true for multifaceted community/ public health
nursing programs; knowing which nursing intervention is contributing to which outcome is more helpful.
Evaluation of outcome attainment evaluates changes in the population, the health care system within the community, or the
environment. Box 17-2 identifies several variables that can be used as outcome measures of community health. Changes can occur
in the population’s knowledge, behavior and skills, attitudes, emotional well-being, and health status.
Box 17-2

Possible Outcome Measures

1. Knowledge
2. Behaviors, skills
3. Attitudes, commitment to action
4. Emotional well-being
5. Health status (epidemiological measures)
6. Presence of health care system services and components
7. Satisfaction or acceptance regarding the program interventions
8. Presence of policy that allows, mandates, or funds
9. Altered relationship with physical environment

When evaluating the health of a community, more than the outcomes of the population must be considered. Because the
interaction of people in their environment facilitates or hinders health, variables such as the presence of health services, the
satisfaction and acceptance of such programs, the presence of policies, and a harmonious balance with the environment must also
be considered. Each of these variables, which are used as an outcome measure of the health of populations or communities, is
discussed in more detail. Each of these variables can be used as a measure of the effectiveness of specific community/public health
nursing interventions.
Knowledge
A great deal of client teaching and health education is evaluated by measuring the health-related information that the individual,
group, or population has obtained. Although information alone does not result in behavior changes, having information will often
increase the possibility of behavior changes. For example, just because a father knows how to prepare infant formula in the proper
concentration and with adequate asepsis does not ensure that he will actually do so. However, if he does not have that information,
the only way he can prepare the formula would be by trial and error or by chance. Having the information increases the probability
that the formula will be prepared properly.
In response to an increased incidence of syphilis in an urban, Hispanic population, public health nurses and other professionals
provided a 10-week outreach project to more than 2800 individuals through street and business outreach (Endyke-Doran et al.,
2007). The project evaluation measured knowledge changes in the population. Health education addressed knowledge and
prevention of syphilis, and location of testing sites. Before-and-after levels of knowledge were based on interviews with different
individuals. At the beginning of the project, only 4% could identify prevention measures or locations for testing. Near the end of the
project 50% of those encountered had knowledge of prevention and transmission and 64% knew of available testing services.
Additional surveillance data from the local health department showed that the number of Hispanics who sought testing for syphilis
also increased.

When evaluating populations, surveys may be used to determine knowledge about specific health-related topics. These surveys
may be conducted as interviews or through written questionnaires (Polit & Beck, 2010). When working with populations, the
community/public health nurse is interested in the proportion of the population that the teaching reached and the proportion that
retained the information presented. Having information is not sufficient for healthy living; the information must be put to use.

Behaviors and Skills

Integrating health-related behaviors and skills into daily living affects health status—raising children, caring for an older bed-bound
family member, seeking a prostate examination, and preparing nutritious foods require action. These actions are
labeled competent or skilled if they are consistent with existing knowledge and if they are performed in an effective and efficient
manner.
Health behaviors may change as a result of interventions performed by community/public health nurses (see Chapters
18 and 20 for more details regarding health promotion and health teaching).
When evaluating health behaviors of populations, the nurse’s interest is in the proportion of the population who engage in such
behaviors. The usual way to collect information about health behaviors is to ask people what they do. However, people do not
always provide accurate reports because they may have forgotten information or want to look good to the surveyor.
Some data on health behaviors, such as use of a specific health service, can also be collected from client health records and health
care information systems. For example, immunization rates can be determined for populations of preschool children receiving
Medicaid or enrolled in a specific managed care organization by monitoring whether immunizations have been received.
Adherence to drug treatment for latent tuberculosis infection (LTBI) was one of the behavioral outcomes evaluated in a tuberculosis
control and prevention program among a homeless population in Baltimore (Lashley, 2007). Nursing students and faculty partnered
with a faith-based, inner-city mission and the local health department. Interventions included education, tracking persons who
missed appointments, monthly appointment reminders, and incentives. The desired outcome was that 65% of those in treatment for
LTBI would complete a 9-month course of therapy. Although only 33% completed at least 6 months of medication therapy, this far
surpassed the city’s 11% completion rate among the homeless.

Time and money often limit the degree to which behavior change can be measured. Observing the behavior of populations helps
confirm the accuracy of what is reported; however, this process takes much more time and money. Asking people to make a
contract with themselves to make a commitment to specific actions has been shown to increase the likelihood that the actions will be
performed (Sloan & Schommer, 1991). Therefore when measuring actual behavioral changes of populations is not possible,
community/public health nurses can measure the degree to which people commit to specific actions.

Attitudes
Attitudes include opinions and preferences about ideas, people, and things. Persons have attitudes about the concept of health and
the ways in which health may be attained and maintained. Because attitudes predispose the selection of some actions over others,
attitudes are a health-related measure. For example, if a population generally views health as the ability to perform work, people
may take cold medication to allow them to feel well enough to go to work. However, a group may not alter their high-cholesterol
diets because their current diets do not interfere with their immediate ability to work.
Community attitudes also predispose the population to support or work against various policies and services. For example, if
the dominant community attitude toward criminals is that they should be punished and live stark lives, there may be little support
for prison health services. If the predominant community attitude is that health prevention can reduce human suffering and dollars
spent for care of the ill, there may be more support for prison health services.
 Attitudes toward health and health behavior can be changed through planned or spontaneous experiences. Attitudinal change is
also called emotional learning or affective learning. Attitudes of populations can be measured before and after an intervention to
determine whether affective learning has occurred. Changes in attitude may predispose people to change their behaviors. For
example, as more members of a population adopt the attitude that smoking is undesirable, smoking rates decrease. In some
neighborhoods, volunteer or paid members of the community are trained by community/public health nurses to address attitudes
of community members about obtaining health care services such as mammography screening, prenatal care, and treatment for
substance abuse.

Emotional Well-Being and Empowerment

Emotional well-being in a population can be measured by the proportion of members who experience self-esteem and satisfaction
with their lives. Emotional well-being of a community can be measured also by assessing the existing structures and processes to
strengthen human development and connectedness.
A group of nursing students initiated a reminiscence group in which residents of a nursing home were able to reflect on and share
their life experiences. The students’ initial assessment indicated that the residents rarely communicated with each other (even when
in the same room), had few visitors, and reported that they did not “feel at home.” After several weekly meetings, the nursing
students observed that the participants initiated conversation more with each other, and several of the residents reported “feeling at
home.”

Improved quality of life is another outcome related to human well-being.

In an evaluation of a community-based outreach worker program for children with asthma, one of the outcomes was quality of life
of the children’s caregivers (Primomo et al., 2006). Using an existing questionnaire, postintervention phone interviews were
conducted 1 month after services were completed. Compared with before the program, caregivers reported an improvement in their
quality of life, especially because they did not have to change their plans as frequently because of their child’s asthma. They felt less
helpless and frightened, and they got more sleep.

Criteria for emotional well-being of a community also include the degree of acceptance and cohesion among members and
patterns of support, socialization, and decision making. When community members participate in the decision making that leads to
goal achievement, perceptions of self-efficacy are enhanced. Self-efficacy is the belief that an individual can influence his or her
environment and circumstances. Self-efficacy contributes to self-concept and is necessary if community members are to have an
impact on their health.

Health Status
An ultimate measure of the effectiveness of health services and programs is the health status of the population. Community/public
health programs seek to reduce premature deaths, disabilities, and injuries. Health status is measured using epidemiological
statistics about morbidity and mortality (see Chapter 7). Epidemiological statistics that are collected for geopolitical communities do
not distinguish the effects of nursing interventions from the effects of other health disciplines and programs. However,
epidemiological statistics can be used to evaluate changes in health status that result from nursing interventions.
Community/public health nurses in Lincoln-Lancaster County Health Department were concerned about the incidence of low-birth-
weight babies born to high-risk mothers. The intervention plan they developed consisted of an intensive home visitation schedule to
educate and support the high-risk women.
To measure outcomes, a care pathway tool was used to track the 55 clients’ progress during and after pregnancy. The evaluation
outcome revealed that five to nine home visits by a community/public nurse improved health outcomes for mothers and babies.
Mothers had higher hemoglobin levels during pregnancy. No low-birth-weight babies were born to the mothers in the home-visit
group. Program evaluation of outcome measures demonstrated the effectiveness of nursing interventions (Fetrick et al., 2003).