Professional Documents
Culture Documents
S88
Measuring Youth Development Outcomes ❘ S89
development strategies has been heightened in recent skills, and decision making. Caring adult relationships
years by a growing body of literature that supports the included two underlying constructs, staff relationships
impact of positive youth development (whether called and program effectiveness, and basic social skills fac-
developmental assets or protective factors) on the well- tored into constructs measuring self-control, empathy,
being of youth.8,9 and communication. Further details about the develop-
Promoting youth development has become a cen- ment and psychometric properties of this instrument
tral tenet of many youth advocacy organizations in- have been reported elsewhere.10 The final instrument
cluding the Search Institute, National Collaboration for was named READY, the Rochester Evaluation of Asset
Youth, Academy for Educational Development, and Development for Youth, by the youth-serving agency
Public/Private Ventures. Although these and other or- participants in the pilot instrument development. In
ganizations provide valuable resources to community- 2002, READY was made available for routine use by
based organizations in planning and implementing youth-serving organizations that had participated in
youth development programs, evaluating such pro- its development.
grams remains challenging. Few instruments exist to The purpose of this article is to describe the imple-
measure positive youth development outcomes, and mentation and dissemination of the READY tool among
those that do are mainly lengthy, detailed, community- these community programs. We review the lessons
level surveys.1 Community-based programs that may learned during READY dissemination and implemen-
neither have the financial resources nor have the inten- tation by community-based youth-serving agencies in
tion to implement large scale evaluations are in need of Rochester, New York, in 2002–2003, and report on
valid and reliable youth development measures to help youth development outcome combined benchmark
them determine the impact of their program on youth.1 data on the basis of administration of READY by these
Recognizing this need, two major youth-funding or- community-based programs.
ganizations in Monroe County, New York (the United
Way of Greater Rochester and the Rochester-Monroe
County Youth Bureau) convened a project team com-
posed of researchers, funders, and program leaders rep-
● Methods
resenting local youth-serving organizations, to identify
Youth development outcome measure
a valid tool that could help local community-based pro-
grams measure positive youth development outcomes. The READY tool is a self-report 40-item questionnaire
Program leaders identified the need for a brief instru- that assesses key demographics, program participa-
ment that would be easy to use and administer, applica- tion and connectedness, and four common youth de-
ble to a variety of youth-serving programs, and useful velopment outcomes. To measure participation, youth
for the assessment of program impact on the develop- are asked to report their length of involvement in the
ment of youth participants. Finding no existing tools program, the number of days per week they spend in
that met their needs, the project team enlisted help from program activities, and their perceived intensity of in-
researchers at the University of Rochester to develop volvement. Participation intensity is classified as high
such an instrument. if youth report that they “attend and participate in most
To enhance the likelihood of use by the participat- program activities,” medium if they report they “attend
ing community-based programs, the project team be- but don’t always participate in program activities,” and
gan by narrowing a list of 54-candidate youth devel- low if they report that they “don’t attend program reg-
opment outcomes to those that would allow program ularly.” Participant connectedness to program is clas-
leaders to demonstrate their effectiveness to funders, sified as high when youth report feeling connected to
improve the quality of their services, and that were most people, medium when they report feeling con-
able to be affected by their community-based youth nected with some people, and low when they report
program. The four outcomes identified through this that they do not feel connected to people at the pro-
consensus building process were (1) caring adult re- gram. Constructive use of leisure time is measured by
lationships, (2) basic social skills, (3) decision making, asking participants to report the number of days per
and (4) constructive use of leisure time. Survey items week they participate in (1) music, theater, or other arts,
were either written or adapted from various sources (2) sports, (3) clubs or other organizations, (4) religious
to address these four outcomes. The resulting instru- youth groups, (5) religious services, and (6) reading
ment was pilot tested using cognitive interviews with for fun. To assess the remaining three youth develop-
adolescents to establish face validity. In a larger field ment outcomes, we created summary scores for each of
test in 2001–2002 to establish construct reliability, fac- the six underlying constructs (staff relationships, pro-
tor analysis found six principal components measur- gram effectiveness, self-control, empathy, communica-
ing elements of caring adult relationships, basic social tion, and decision making). We developed scores by
S90 ❘ Journal of Public Health Management and Practice
averaging participant responses on Likert-like items BOX 1 ● Agencies participating in combined community
that comprised each construct to yield a simple nor- evaluation of positive developmental outcomes using the
malized score between 0 and 100. READY tool
program section of the Tool Kit. Some organizations grams at one of nine agencies. The average age of par-
also required additional TA to design and implement ticipants was 13 years (SD ±2 years), and 53 percent
their sampling and administration plans. The need for of participants were female. The sample was ethnically
TA, whether related to the computer-based Tool Kit diverse: 53 percent reported that they were Black or
or the overall evaluation process, increased when a African American, 25 percent White, 9 percent mul-
greater amount of time passed between staff attendance tiracial, and 6 percent Hispanic/Latino. More than half
at training sessions and the program’s subsequent ad- (53%) reported that their family had enough for neces-
ministration of the survey. sities and could buy special things, 42 percent reported
that their family had just enough money for necessities,
Program leader feedback and 5 percent reported that their family did not have
enough money to buy necessities.
Nearly all program representatives reported that they
found the tool easy to use. Many of the agencies re- Participation and connectedness to program
ported that their leadership had reviewed their score
reports with various other staff members. A few agen- A third of participants reported they had been involved
cies reported that the data had prompted them to ex- in a program for less than 1 year, 47 percent reported
amine program curricula and activities more closely participating between 1 and 5 years, and 20 percent
to identify whether their programs included compo- reported participating for more than 5 years. Twenty-
nents intentionally designed to impact the measured seven percent of youth reported participating in pro-
outcomes. Some of the larger agencies surveyed youth gram activities 5 to 7 days per week, 34 percent reported
in multiple program activities or at multiple sites, and 2 to 4 days per week, 34 percent reported participating
reported comparing score reports across program units, once a week, and 5 percent reported 0 days a week. The
using the data to identify whether programming dif- majority (71%) reported attending and participating in
ferences might have accounted for any observed score most program activities, 21 percent reported attending
variations. Program leaders voiced uncertainty about but not always participating in activities, and 8 percent
what their scores meant, and with their ability to in- reported not attending program regularly. Most youth
terpret a “good” score versus a “bad” score. However, felt connected to people at their programs: 49 percent re-
the majority acknowledged that data from their first ported connection to most people, 46 percent reported
year would be considered baseline, and that they would feeling connected to some, and 6 percent reported that
need to use the tool over several years to understand they did not feel connected to people at the program.
longitudinal results for their program. Many agencies
were also interested in reviewing community-wide ag- Youth development outcomes
gregated data. Some program leaders also expressed a Most youth reported participating in a leisure time ac-
desire to be able to share results and work collabora- tivity one or more times per week (Table 1). Positive
tively with funders and with similar programs in the youth development construct scores for the combined
community to identify best practices and share strate- community program data and for varying levels of self-
gies to improve program quality. reported participation and connectedness are shown in
Table 2. Participants’ length of involvement in a pro-
Aggregate community-based program outcomes gram was significantly related to each of the three con-
structs measuring basic social skills. Mean scores for
Demographics self-control, empathy, and communication generally in-
We received READY survey data on 1,070 youth at least creased with increased length of involvement in pro-
10 years of age who participated in 15 different pro- gram. Youth who participated at least once a week in
program activities had significantly higher caring adult their READY data. A recent survey of nonprofit orga-
relationship scores (staff relationships and program ef- nizations by the Urban Institute found that outcome
fectiveness) than youth who did not participate weekly. evaluation in general is relatively new for many orga-
Intensity of participation was significantly related to nizations, and that few nonprofits have had training
all six construct scores; youth who reported attending or TA in outcome measurement.11 These authors also
and participating more actively had higher scores than reported that few of the 36 organizations that they sur-
those who reported less-active participation. Partici- veyed used any systematic data collection or sampling
pants who reported increased levels of connectedness strategies.11 Similarly, we found that many of the agen-
also had higher staff relationship scores and higher ba- cies we worked with needed TA in creating their overall
sic social skills scores (self-control, empathy, and com- evaluation plan. For example, larger organizations with
munication). multiple programs required assistance with selecting
samples, in order to be able to administer READY in a
manner that would not require excessive staff resources
● Discussion but that would still give them representative data about
their programs to be used to help improve program ser-
The READY tool appears to be feasible for use by a vices. Although nearly all agencies required some tech-
variety of small and large, local neighborhood-based nical support to use the analysis program, this section
and nationally affiliated youth-serving organizations. of the Tool Kit has since been redesigned and simplified
In its first year of use by community-based programs, as READY, Version 2. The simplified version has been
all participating organizations were able to success- disseminated and is being used in four additional geo-
fully implement and administer the tool using their graphic areas, with several dozen programs. Whether
own staff resources. In addition to initial training, al- this reduces the assistance required for initial learning,
most all organizations were also in need of some on- and whether Rochester area programs are able to use
going TA in planning their evaluation or interpreting READY independently in subsequent years of use are
Measuring Youth Development Outcomes ❘ S93
questions currently under evaluation, and will be ad- may be similarly limited. The United Way of Greater
dressed in future reports. Rochester and the Rochester-Monroe County Youth Bu-
Preliminary data from our development of the reau commissioned a recent independent survey of pro-
READY tool,10 as well as other reports,1,12 have gram directors and direct service staff in agencies that
demonstrated that participation in community-based were using the READY tool. This study confirmed our
programs is indeed associated with better develop- initial findings of the utility of the tool: 8 of 10 program
mental outcomes for youth. The positive effects of directors reported that the READY tool provided help-
program participation are confirmed by these ful information about youth in their programs, and 7
community-wide data. Youth who participated of 10 reported that they used the information collected
more frequently in programs had higher scores on to change how they provide services to youth in their
constructs measuring the outcome of caring adult programs.14
relationships. This suggests that youth need to have Community-based programs represent one link in
at least weekly contact with programs, in order to promoting positive youth development. As noted by
develop meaningful relationships and to feel valued various authors in other articles in this supplement, ex-
and supported. We also found that participants’ per- perts recognize that it takes a coordinated approach of
ceived intensity of participation was associated with families, communities, schools, and community pro-
positive youth development outcomes. Higher scores grams to influence developmental outcomes among
were observed on all six constructs among youth who youth. Another limitation of this study was our in-
perceived that they more regularly and more actively ability to measure and control for other non–program-
participated in program activities than among those related influences, such as family, community, and
who did not. These findings further highlight the need school, on youth development outcomes. Although
to pay attention to multiple dimensions that constitute there is clearly an association between program par-
participation, beyond just attendance checklists, when ticipation and positive youth outcomes, further study
evaluating program effects.13 utilizing additional measures and longitudinal designs
In our initial validation of the READY instrument, is needed to demonstrate independent effects. For some
connectedness to program was related to staff relation- youth, community-based programs provide supports
ships, self-control, empathy, communication, and deci- and services including opportunities to form healthy
sion making.10 In this study, a significant relationship relationships with peers and caring adults and a place
did not exist for the decision-making score. However, in which to develop skills that can help them become
connectedness to program remained significantly as- productive adults. The development and use of valid
sociated with staff relationships, self-control, empathy, indicator measures that can be used by community
and communication. Clearly, this is a crucial element programs to evaluate their influence on the develop-
to improving youth development outcomes. The In- mental outcomes of youth is crucial in helping to im-
stitute of Medicine and the National Research Coun- prove the quality of services and supports that youth
cil’s recommendations for key features of positive de- receive.
velopmental settings for youth include physical and This study has shown that READY is a promising
psychological safety, appropriate structure, supportive tool that could potentially be used by community-
relationships, opportunities to belong, positive social based programs to examine program-attributable de-
norms, support for efficacy and mattering, opportuni- velopmental outcomes for youth and improve program
ties for skill building, and integration of family, school, quality. Unlike program quality assessment tools that
and community efforts.1 Programs with which youth rely on observations made by adults, READY provides
do not feel connected may need to consider whether the programs with feedback from their youth participants.
program context, or individual factors, or both, have Although further work is needed to refine the measure-
resulted in the lack of engagement or connections by ments, examine construct reliability, and establish the
youth. predictive validity of the tool, READY is a promising
Our study has limitations. First, the data we have re- option for measuring positive outcomes, and not sim-
ported are based on organizations that volunteered to ply assessing the absence of negative outcomes,15 in
participate in using the READY tool as a program eval- promoting development of youth.
uation measure. We collected qualitative feedback from
programs on their use of the tool utilizing informal dis-
cussions to identify key themes. These programs may
not be fully representative of youth-serving organiza- REFERENCES
tions in our area, and qualitative feedback may have 1. Eccles J, Gootman JA, eds. Community Programs to Promote
been biased by social desirability. Generalizability to Youth Development. Committee on Community-Level Pro-
other geographic areas and to other types of programs grams for Youth; Board on Children; Youth and Families;
S94 ❘ Journal of Public Health Management and Practice
Division of Behavioral and Social Sciences and Educa- findings on evaluation of positive youth development pro-
tion; National Research Council and Institute of Medicine. grams. Ann AAPSS. 2004;591:98–124.
Washington, DC: National Academies Press; 2002. 9. Resnick MD, Bearman PS, Blum RW, et al. Protecting adoles-
2. Schinke S, Brownstein P, Gardner S. Science-based Pre- cents from harm. Findings from the National Longitudinal
vention Programs and Principles, 2002. Effective Substance Study on Adolescent Health. JAMA. 1997;278(10):823–832.
Abuse and Mental Health Programs for Every Community. 10. Klein JD, Sabaratnam P, Matos M, et al. Development and
Rockville, Md: Center for Substance Abuse Prevention, Sub- factor structure of a brief instrument to assess the impact
stance Abuse and Mental Health Services Administration; of community programs on positive youth development:
2002. the Rochester Evaluation of Asset Development for Youth
3. Kirby D. Emerging Answers: Research Findings on Programs to (READY) tool. J Adolesc Health. 2006;39(2):252–260.
Reduce Teen Pregnancy. Washington, DC: National Campaign 11. Morley E, Vinson E, Hatry HP. Outcome Measurement in Non-
to Prevent Teen Pregnancy; 2001. profit Organizations: Current Practices and Recommendations.
4. Jessor R, Jessor SL. Problem Behavior and Psychosocial Devel- Waldorf, Md: The Urban Institute and Independent Sector;
opment: A Longitudinal Study of Youth. New York: Academic 2001.
Press; 1977. 12. Bartko WT, Eccles JS. Adolescent Participation in Structured and
5. Werner EE, Smith RS. Vulnerable but Invincible: A Longitudinal Unstructured Activities: A Person-oriented Analysis. J Youth Ado-
Study of Resilient Children and Youth. New York: McGraw-Hill; lesc. 2003;32(4):233–241.
1982. 13. Yohalem N, Wilson-Ahlstrom A, Pittman K. Out of School Pol-
6. Pittman KJ. Promoting Youth Development: Strengthening the icy Commentary # 6: Participation During Out-of-School Time:
Role of Youth-Serving and Community Organizations. Report Taking a Closer Look. Washington, DC: The Forum for Youth
prepared for the US Department of Agriculture Extension Investment; 2004.
Services. Washington, DC: Center for Youth Development 14. Fatula J. Youth development providers’ perspectives on
and Policy Research; 1991. the READY survey tool. 2005. Available at: http://www.
7. Carnegie Council on Adolescent Development. A Matter of urmc.rochester.edu/gchas/div/adol/leah/resources.HTM.
Time: Risk and Opportunity in the Nonschool Hours. New York: Accessed June 19, 2006.
Carnegie Corporation of New York; 1992. 15. Roth J, Brooks-Gunn J, Murray L, Foster W. Promoting
8. Catalano RF, Berglund L, Ryan JA, Lonczak HS, Hawkins JD. healthy adolescents: synthesis of youth development pro-
Positive youth development in the United States: research gram evaluations. J Res Adolesc. 1998;8:423–459.