Foster Care Outcomes and Performance Standards

A Report of the Task Force on Foster Care Performance Standards

City of New York Office of the Comptroller Office of Policy Management Alan G. Hevesi Comptroller Barbara Salmanson Senior Policy Adviser

Task Force Chairpersons Barbara Blum National Center for Children in Poverty David Liederman Child Welfare League of America

Fred Wulczyn, Author Chapin Hall Center for Children

May 19, 1997

Barbara Blum is currently Senior Fellow in Child and Family Policy at the National Center for Children in Poverty at Columbia University. From 1986 through 1996, she was president of the Foundation for Child Development where she focused her energies on improving programs and policies affecting children and families at risk. Her other past positions include president of the Manpower Demonstration Research Corporation and Commissioner of the New York State Department of Social Services. She also serves as chair of the Agenda for Children Tomorrow, and is a member of the boards of directors of the March of Dimes and the Academy for Educational Development and a member of the New York State Commission on Child Abuse and Neglect.

David Liederman, since 1984, has been the Executive Director of the Child Welfare League of America, the largest voluntary organization in the U.S. concerned with improving services to abused and neglected children and their families. He has been a strong advocate on behalf of children during his two terms in the State of Massachusetts House of Representatives, as Commissioner of the Massachusetts State Office of Children, as Chief of Staff to Governor Michael Dukakis and as Executive Director for Public Affairs for the Federation of Jewish Philanthropies of New York.

Fred Wulczyn is currently a research fellow at the Chapin Hall Center for Children at the University of Chicago, where he oversees Chapin Halls Managed Care Forum and co-directs the multistate foster care data archive. Prior to joining Chapin Hall in 1997, he was the Director of the New York State Department of Social Services Managed Care Initiative. He is also a member of advisory committees for the National Study of Kinship Foster Care and the National Evaluation of Family Support and Family Preservation Programs.

Alan G. Hevesi
Comptroller

Steven Newman

First Deputy Comptroller

Jeanne Millman
Director

Barbara Salmanson
Project Manager Office of Policy Management !

Fred Wulczyn
Report Author

Steven Budde
Research Associate Chapin Hall Center for Children

Comptroller's Task Force on Foster Care Performance Standards

Co-chairs

Barbara Blum, National Center for Children in Poverty David Liederman, Child Welfare League ofAmerica
Members

Lynette Brown, Council of Family and Child Caring Agencies Steven Cohen, Jewish Board of Family and Children's Services John Courtney, Child Welfare Watch Mario Drummonds, Northern Manhattan Perinatal Partnership Judith Hines, Council on Accreditation of Services for Families and Children Donna Lawrence, Children's Defense Fund Elba Montalvo, Committee for Hispanic Children and Families Sr. Mary Paul, Center for Family Life Elizabeth Schnur, Jewish Child Care Association Janice Shindler, Association of Black Social Workers Michael Solomon, Ncw York State Comptroller's Office Fred Wulczyn, Chapin Hall Center for Children

May 19, 1997

any thanks to the members of the Task Force on Foster Care Performance Standards for their commitment and hard work which culminated in the publication of this report. Their efforts to refocus attention on how the foster care system affects the lives of children and families will improve the condition of foster children and force greater accountability among social service systems serving them and their families. I recognize that this cannot be accomplished overnight. However, this report presents a major first step in that very important process. I express my profound gratitude to both Barbara Blum and David Liederman, co-chairs of the Task Force who led this very important effort. Their support and direction have been invaluable. I would like to give a special thank you to Fred Wulczyn, author of the report, for his hard work, expertise and guidance on these complex issues, and acknowledge the Foundation for Child Development for its financial support of Freds efforts. I also express my apreciation to Commissioner Nicholas Scoppetta and the Administration for Childrens Services for their commitment to improving the quality of foster care services and their encouragement and ongoing support for the work of the Task Force. We look forward to continuing to work closely together to Improve the quality of services for New York Citys children. Finally, I thank Barbara Salmanson for managing this important project through to its completion, and David Neustadt and Mary Leopold for their editorial assistance.

Alan G. Hevesi Comptroller

Highlights

The purpose of a child welfare system is to identify children in dangerous situations and provide the appropriate protections. Oftentimes, this means moving them to temporary foster homes and, as quickly as possible, provide permanent, safe homes. The foster care system should be as supportive as possible to children who are going through an inherently difficult and distressing experience. The safety and well-being of the children must be the paramount goal. As obvious as that sounds, foster care agencies are not currently evaluated on how well they take care of children. Current standards for evaluating agencies focus on process and inputs rather than outcomes. Data is gathered on whether agencies meet timetables for moving children to adoption and filing reports with the Administration for Childrens Service's, but not on the experience of children during the time they are in foster care. As a result, each year hundreds of millions of public dollars are spent in New York City without obtaining a true picture of the impact the foster care system has on the children living within it. Consequently, no accurate evaluation can be made nor can change be implemented to improve the experiences of these vulnerable children. The Comptroller's Task Force on Foster Care Performance Standards is working to refocus the evaluation process by helping to create new performance standards focussed on outcomes and on children and their families. In this report, the Task Force has developed principles that should guide the development of these new child-centered standards. The report mandates accountability based on how the system is succeeding in meeting children's needs in three major areas: 1.System Performance: Safety and Permanence, 2.Child and Family Well-being, and 3.Client Satisfaction. The report and its recomendations identify the outcomes that should be the focus of the work of foster care agencies and the individual indicators which measure progress towards achieving those outcomes. The report focuses attention on child and family well-being indicators which address the physical and emotional state of the child and look at children in the context of their families. For the first time, client satisfaction is being treated as a valuable measure in determining the quality of services provided. Although a family member's level of satisfaction with a service or provider may not be a definitive measure of service quality, it does serve as a valuable resource in understanding and evaluating the effects of services on children and biological and foster parents. i

Foster children are involved with a variety of social service systems. Too often, the buck is passed from one system to the other, and children's needs go unmet. The report recognizes the complexities and interplay between and among the various social service systems and attempts to integrate them to obtain a comprehensive picture of a foster child's experience. This new approach will help the Administration for Children's Services impose accountability on the child welfare system to meet the needs of foster children. As a result of the implementation of such a system, the nature of evaluations of the performance of foster care agency performance will be more qualitative, comprehensive and child-focused. It will give the system greater knowledge about the strengths and weaknesses of program design and the successes and failures of specific programs. Most importantly, it will base performance on what is done for the child. As a result, we hope to achieve greater accountability so the quality and effectiveness of services to children continues to improve, along with casework practice, clinical decision making and planning and resource allocation both externally and internally. In this way, limited flinds can be used in the manner best suited to meet the needs of children and families to assure child safety and family integrity.

ii

Contents
page

Highlights

Part One: Introduction and Overview I. Introduction II. Perspectives on Foster Care Outcomes and Performance Standards III. Overview of Recommendations System Performance: Child Safety and Permanency Child and Family Well-Being Client Satisfaction Part Two: Task Force Recommendations IV. Domain One: System Performance: Child Safety and Permanency Safety and Permanency Issues Length of Stay Recommended Outcome Measures Proposed Tools for Outcome Measurement Phase One: Use Existing Resources Phase Two: Link Information V. Domain Two: Child and Family Well-Being The Challenge of Measuring Well-Being Recommended Outcome Measures Proposed Tools for Outcome Measurement Phase One: Undertake a Study of Child and Family Well-Being Phase Two: Create a Working Group VI. Domain Three: Client Satisfaction The Importance of Client Satisfaction Measures Recommended Outcome Measures Proposed Tools for Outcome Measurement Phase One: Implement Guidelines and Surveys Phase Two: Establish an Advisory Panel

i 3 5 8 10 11 11 13 14 14 14 15 16 17 20 23 23 25 29 29 30 32 32 33 35 35 37

Appendices A. Methodological and Conceptual Issues that Arise in Conducting Research on Child and Family Well-being B. Methodological and Conceptual Issues that Arise in Measuring Client Satisfaction C. Summary of 1996 Focus Group with Parents Regarding Client Satisfaction D. References

Part One Introduction and Overview

I. Introduction
Recent tragedies have highlighted the need to improve the performance of New York Citys child welfare system. A vital part of that process must be the development and implementation of a system of outcomes and performance measures that describes what happens to children and their families when a child enters the foster care system. In short, we need to know what works, what does not, and why. Current measures tend to focus on process rather than results, on inputs rather than outcomes, on how much is spent rather than what is accomplished, on quantity rather than quality. Absent a more coherent approach to fundamental questions regarding how well children are served by the child welfare system, the system improvements long hoped for will remain a remote possibility. The renewed emphasis on child welfare outcomes comes at an important time for other reasons. The mission of the child welfare system - ensuring the safety of children while preserving the integrity of families - is become increasingly difficult to operationalize, let alone fulfill. The federal policies that delineate public responsibility for needy persons, including children, are in the midst of a transformation that will fundamentally alter Americas social safety net. For their part, some states around the country have already enacted time limits for welfare recipients, block grants for child welfare services, and managed care for health and behavioral health care, initiatives that are often accompanied by significant reductions in funding and a devolution of authority in the direction of local government. Enacted with an eye toward making government more responsive to local conditions, the changes stand to have a profound impact on child welfare systems. In short, at a time when the need for child welfare services is as high as it has ever been, the very service delivery system designed to meet that need is under enormous pressure to function more effectively and more efficiently. In a policy and fiscal context that demands the effective and efficient delivery of services, no organization, public or private, can be expected to improve on its performance without clear outcomes and appropriate performance standards. To this end, the analysis of program data, much of which is already routinely gathered, fulfills an essential purpose by providing key insights into what is happening to children and families. Long a staple of effective management within industry, government and not-for-profit agencies are just now learning how to use program data strategically. Within this context, outcome measures specify the results toward which resources must be allocated, while performance standards provide a baseline for comparing accomplishments with expectations. Together, outcome measures and performance standards provide an essential framework for feedback and accountability, making it possible to evaluate critically the decisions made by government agencies and the providers who serve the public.
3

Under Alan G. Hevesi, the New York City Comptrollers Office has made improved performance monitoring of government programs one of its priorities. As part of that effort, the Comptroller convened a Task Force on Foster Care Performance Standards in July 1995 to develop measures based on outcomes achieved for children in care and their families. In completing its work, the Task Force set out to develop both a reasonable set of indicators and appropriate methods that could be used to monitor New York Citys child welfare system. Over a series of meetings held since the late summer of 1995, the Task Force reached agreement on the need for outcome measures and performance standards; the criteria to be used in selecting indicators; and the specific measures and standards that underlie its recommendations. Briefly the recommendations fall into three domains: 1. 2. 3. System performance: child safety and permanency, Child and family well-being, and Client satisfaction.

The complexity of the problems families face and the tension between family preservation and child protection can undermine the singular sense of purpose that leads to clear outcome statements and performance standards. As such, the outcome domains are not discrete. On the contrary, an effective system of outcome measurement and monitoring depends on the ability to draw a comprehensive portrait of how well the system is performing and how well children and families are thriving using together all three outcome domains that were examined by the Task Force. The Report addresses these issues in depth and proposes ways to develop a coherent plan with an eye toward improving upon the state of the art in child welfare. It begins with an overall orientation to outcomes and performance standards, enumerates their uses, and describes the criteria that were used in selecting them. The specific recommendations of the Task Force follow. In addition to specifying outcomes in each of the three domains, it describes the relationship between the outcomes and the mission of the child welfare system, the information burden (cost) associated with collecting information systematically, and other important methodological or resource considerations that govern the use of given sets of indicators. Finally, the Report details long-term recommendations that go beyond the specific outcome measures and performance standards. This second set of recommendations is intended to provide an overarching framework for improving the capacity of New York Citys child welfare system to not only gather outcome data, but also use it effectively.

II.

Perspectives on Foster Care Outcomes and Performance Standards

Although use of the term outcome is increasingly commonplace within child welfare, the term is seldom defined with real precision. The diversity of children and their families, the problems they present, and the settings in which they are served make it difficult to identify a common set of outcomes that are applicable to each and every child served by the foster care system. Consequently, the best place to start is with the core missions or purposes of the child welfare system: safety and permanency. Child safety is the first objective of any child welfare system. The threat to safety is the reason for the systems involvement in the familys life and the primary problem to be solved. Permanency of placement, either through reunification or adoption, is the mandated goal for children in out-of home care and the simplest measure of a systems success. Promoting the welfare of children through stable placements is part of this core objective. These two broad purposes provide the beginning framework for formulating performance standards. Nevertheless, because of the range and complexity of possible outcome indicators, the Task Force had to refine its approach by identifying the purposes of the performance standards, recognizing that the standards themselves will necessarily evolve with experience. Thus, in the view of the Task Force, outcome data and performance standards should at a minimum: 1. 2. 3. 4. 5. 6. Provide more accountability than prevailing standards, so the quality and effectiveness of services to children and families continues to improve; Guide casework practice and clinical decision making; Improve policy analysis; Enhance public agency management, planning, and resource allocation; Enhance contract agency performance, including planning and resource allocation; and Lay the groundwork for a cumulative record of outcomes and performance to which detail can be systematically added over time.

The selection of appropriate outcome measures should be guided by these purposes. The process should also be guided by a pragmatism that recognizes the strengths and weaknesses of the existing system of outcome measurement and takes into account how outcome data are actually used. Since there are so many potential outcome measures, understanding their end uses helps to establish priorities. The Task Force chose these criteria for selecting indicators:

1.

2. 3. 4.

5.

Performance standards should be consistent with existing statutes and regulations as they pertain to utilization review standards, the time it takes to reunify children with parents, and the time it takes adoptions to be finalized. Outcome measures must relate to the two core missions of the child welfare system: safety and permanency. The list of outcomes should reflect the developmental status of the child and the family and their history within the system. Outcomes must be measurable with current information technology. The measures should have a historical baseline. Where no baseline data exist, a long-term commitment to establishing baseline data must be made. Outcomes should be evaluated in light of the ease of tracking the data needed to understand the performance of the child welfare system as a whole and its impact on children and their families. Data requirements that add significantly to the information burden should be integrated slowly unless there is a compelling reason to move fast.

Finally, though it may loom large, foster care is but one influence in the lives of children and families. Properly weighing its effect on human development is a complicated undertaking under even the most controlled conditions. Our expectations for what a system of outcome measurement can accomplish should be driven by a common desire to provide the best possible services to children and families, tempered by an appreciation for the limits of information technology. Consequently, careful attention must be paid to questions that pertain to how outcome data can be best analyzed to yield reliable conclusions about system performance and to whether these conclusions can be generalized to sub-populations within the system. One crucial precaution has to do with the conventional snapshot approach to measuring outcomes. Even if it is used repeatedly over time, the snap shot approach to outcome measurement has only limited applicability because it tends to obscure real changes in the experiences of children and families. Our methods must also be able to distinguish between the appropriateness of the indicator itself and the difficulty of gathering, interpreting, and/or analyzing the data. Careful attention must also be paid to how each measure is going to be used in conjunction with the others to enhance our overall understanding of clinical efficacy and system performance. With these purposes and criteria in mind, we returned to the three broad domains of outcome measures and performance standards: 1. 2. 3. System performance: Child safety and permanency, Child and family well-being, and Client satisfaction.

The discussion of each domain is structured as follows. The section on each domain
6

begins with an overview and a description of the areas covered, continues with an explanation of how the specific indicators within this domain relate to the mission of the child welfare system, and concludes with proposed indicators and measurement tools. In addition to the main body of the report, the Appendices provide more detailed assessments of the methodological difficulties associated with studying child and family well-being and client satisfaction. The Appendices also contain the results of a focus group discussion with parents of children in foster care that was used as an orientation to the broader issues of client satisfaction. Finally, the bibliography at the end of the report offers a list of references that should prove useful to anyone interested in pursuing these topics in greater detail.

III. Overview of Recommendations

The outcome measures selected by the Task Force were culled from a crowded field of potential performance indicators. The indicators that were selected were chosen because they are linked directly to the core mission of child welfare and they have relatively light data collection burdens. Thus, they do not add significant new costs, at least in the short term. For the first domain, system performance: child safety and permanency, the indicators selected are direct measures of safety and permanency. Monitored over time, they provide a basic understanding of how the system is serving the children in its care. Some of the indicators measure child safety as demonstrated by incidence or type of maltreatment. Others measure the systems ability to provide continuity and stability for the children in its care by collecting data on length of stay, number of placements, or timing and frequency of re-entry into the system. The second domain, child and family well-being, presented the greatest challenge to the Task Force. There is no single set of well-being indicators that fits all the children the foster care system serves. Moreover, it is hard to isolate the impact of foster care placement from other factors in a childs or a familys experience. Finally, despite the broad appeal of well-being indicators, the link between well-being and the mission of the foster care system must be carefully established. Well-being indicators should not be left out of a systematic performance review, but neither should they be stretched to include outcomes beyond the mission of the child welfare system. With these considerations in mind, the Task Force developed child and family well-being measures organized around the central mission of foster care: child safety. Thus, the incidence of child maltreatment and measures of the childs health, emotional and behavioral status, educational development, and competencies and achievements can be used to demonstrate child well-being. Other indicators relating to child safety should be collected. These include measures of family safety, such as incidence of domestic violence, and of the familys ability to meet the basic needs of the child. Parental education, level of interaction with the child, drug and alcohol use, emotional health, and the quality of the familys social supports should also be measured. To develop recommendations for the third domain, client satisfaction, the Task Force convened a focus group of parents who had children in foster care either currently or in the past. From these conversations, four areas emerged as particularly relevant to New York Citys foster care system. They are:

1. 2. 3. 4.

overall satisfaction, satisfaction with the workers providing services, satisfaction with specific services, and satisfaction with communication.

Within each area, the Task Force suggests specific indicators to measure client satisfaction. Data Collection and Costs Even the best designed indicators will not provide the desired information if data is too hard to collect. Conventional wisdom holds that the facts needed to monitor the performance of the child welfare system are unavailable. But many jurisdictions, including New York City, collect vast amounts of data about children in foster care including the dates of placement, the agency with custody, the childs legal status, permanency placement goals, and other related information. This report presents a series of steps for gathering the required information, categorized into immediate and long-term phases. The Task Force suggests working with whatever reliable data already exist in the initial phase, then collecting information later on measures for which reliable data is now scarce.

The Recommendations in Brief

System Performance: Child Safety and Permanency Phase 1 Report to the Public: The Administration for Childrens Services should use currently collected data to produce an annual report to the public on the dynamics of New York Citys foster care population. The report would: 1. 2. 3. 4. present the outcome measures the Task Force has recommended for evaluating this domain, cover child safety and permanency outcomes for specific sub-populations of children, such as those who remain in care for long periods of time, summarize statistics on indicators of data quality, such as reporting lag and numbers of corrections required, and describe changes, if any, made to the conclusions from previous reports in response to record corrections.

Monitor Contract Agency Performance: The Task Force recommends using internal indicator reports that track child safety and permanency outcomes by contract agency, as well as the Administrations own direct care caseload. Phase Two The Administration for Childrens Services should add new information to its system and link with existing data resources. This may be accomplished in three ways. 1. 2. 3. by redesigning data systems to include more information about clients, their families, and the services they receive, by using case readings or other survey methodologies to expand the scope of available information, and by linking records with existing sources of computerized administrative data, such as vital statistics records, to form a more comprehensive picture. Notwithstanding the critical issue of confidentiality, record linkage can be an important tool for understanding foster care outcomes.

10

Child and Family Well-Being Phase One Like client satisfaction, this domain has a high information burden. The child welfare system does not currently track well-being indicators. Existing instruments may not be adaptable to this purpose, and baseline data on well-being outcomes is limited. Thus, in Phase One the Task Force proposes that ACS: 1. undertake a highly focused study of child and family well-being, based on a random sample of foster children, that stresses child safety in a permanency planning context, and if possible, model the primary data collection instrument on the safety assessment currently in use.

2.

Phase Two In Phase Two, ACS should establish a working group to develop a long-range plan and a time line for integrating child and family well-being measures into the performance evaluation plan. The working group should consider: 1. 2. 3. instrument development, the role of contract agencies in conducting routine studies of well-being, and the creation of reporting standards to permit comparative analysis of agency performance.

Client Satisfaction Phase One Child welfare agencies currently gather little or no systematic data on client satisfaction. Phase One would make measures of client satisfaction a regular form of feedback within the child welfare system. The Task Force recommends that ACS: 1. 2. develop a set of uniform questions on client satisfaction, and conduct surveys of client satisfaction modeled on the rapid assessment technologies used in the health and behavioral health care sectors and involving small, welltargeted samples of children.

Phase Two The Task Force also recommends long-term collection of information on client satisfaction. ACS
11

could establish an advisory panel to address the cost and technical issues related to measuring client satisfaction across representative populations and related service providers. The panels tasks would include: 3. 4. 5. selecting specific client satisfaction outcome measures, establishing standards for assessment, and developing a plan to include a broad array of clients.

12

Part Two Task Force Recommendations

13

IV. Domain One: System Performance: Child Safety and Permanency

When children are removed from their family home, government officials have a fundamental obligation to provide services that conform with public policy and sound social work practice. With respect to public policy, the child welfare system is governed by generally accepted principles, which in turn lay the groundwork for judging system performance. These principles are firmly rooted in the concepts of child safety and permanency planning. Safety and Permanency Safety refers simply to the fact that children should not be subject to abuse or neglect, as defined in public law, while in the care and custody of those who have legal responsibility for them. This includes the times when a child is in the care and custody of a local social services commissioner and in out-of-home care. Permanency is a somewhat more difficult term to operationalize. Public policy currently includes a strong preference for placement prevention because parent/child separation can be so traumatic. In addition, because foster care is intended to be a temporary solution, placements should be brief unless discharge from foster care would endanger a child. If family reunification does not serve the childs best interests, adoption is considered to be the next best permanency plan. Length of Stay Few statistics are more important than length of stay in creating an overall impression of system performance. However, interpreting this seemingly simple measure is often quite complex. Permanency planning is the overarching framework for foster care outcome measures, but child safety is the reason for intervening in the first place. Measuring length of stay includes tracking childrens movements into, out of, and back into foster care, between foster homes, and between levels of care. Because children thrive best when they maintain stable relationships with their caregivers, repeated, unplanned movements between a parents home and the foster care system have to be

14

viewed as undesirable, all things being equal.1 Yet these movements are often necessary given the need to protect a child. Similarly, because family-like settings are usually the most conducive to child development, restrictive settings should be used sparingly. But some children have needs that cannot be met in a family placement. For such a child, ruling out group care may result in a greater number of moves between foster care settings. Anyone using the length of stay measurement to draw conclusions about the overall performance of the system must consider a number of issues. For example, changes in length of stay should be compared with the incidence of abuse and neglect following discharge, especially when the changes are prompted by specific program initiatives. Recommended Outcome Measures Table 1 lists the indicators of system performance the Task Force recommends. The Task Force selected these indicators to identify core outcome measures that can be assembled from existing data resources. By and large, the indicators are self-explanatory. They are direct measures of safety and permanency that, when monitored over time, provide a basic understanding of what is happening to children and how those events are reflected across the system as a whole.

.The phrase all things being equal comes up time and again in any discussion of outcome measures. The phrase is intended to point to those instances when the analyst must determine the extent to which all things are or are not equal. For example, children who are admitted to foster care this year should be no more likely to move from one foster home to another than the children admitted to foster care last year or the year before (in fact, ideally, they should be less likely). However, an objective analysis of movement histories from existing administrative data sources might reveal an increase in the number of children experiencing multiple movements. Before changes in indicators like movement between foster homes can be judged normatively (i.e., before the information can be used to infer that outcomes for children are getting better or worse), the analyst has to examine the data for competing explanations. In other words, the analyst has to answer the question: Are all other things equal? In many respects, the value of any particular indicator lies not in the indicator per se, but in the availability of other data that permit a detailed search for competing explanations.

15

Table 1. Domain One System Performance Safety and Permanency

Areas of System Performance

Indicators

Issues to consider

Child safety

1. Incidence of maltreatment

1. Applicable to children at home, in care, and after discharge. 2. Indicators should be used in conjunction with other indicators. 3. Patterns should be tracked over time. 4. Measures are relevant at the individual and aggregate levels.

2. Type of maltreatment.

3. Injuries due to maltreatment.

4. Harm by peers in placement or siblings at home.

Continuity and stability of 1. Length of stay. care: permanency

1. Relationship between length of stay, reentry, and subsequent maltreatment is crucial. 2. Patterns must be studied within sub-populations, over time, and with communities. 3. Measures are relevant at the individual and aggregate levels.

2. Number of placements including level of care.

3. Reentry rates and timing.

4. Exit rates by type of discharge.

Proposed Tools for Outcome Measurement In many jurisdictions, it may be true that the information needed to monitor the performance of the child welfare system is not available. However, some jurisdictions, including New York City, collect vast amounts of data about children in foster care, their legal status, permanency planning goals, movement histories, and related information. Even so, misperceptions about the availability of information persist and are confounded by concerns about data quality. This is a fundamental issue, especially when the information is used to make normative judgments about system performance. The Task Forces recommendations for this domain address these concerns. Since New York Citys data holdings are relatively rich, the Task Force opted for Phase

16

One recommendations that would maximize the usefulness of existing data rather than calling for costly, new data collection strategies. Two facts influenced this decision. 1. The New York State Department of Social Services is implementing a large-scale computer project, the Connections Project, that will substantially alter how information is collected and used in child welfare service delivery. There is room for improvement in the way information is presented and subsequently used. The integration of information into decision-making has been hampered by a poor understanding of how foster care tracking data can be used to interpret even simple measures like length of stay.

2.

Since the Task Forces recommended outcome indicators already appear frequently in management reports, the recommendations of the Task Force for Phase One amount to guidelines for reporting data. Phase One: Use Existing Resources The Administration for Childrens Services already collects the data it needs to monitor the Task Forces recommended indicators for safety and permanency. Therefore, the emphasis in Phase One is on maximizing the use of those data without major new investments. Recommendations focus on reporting existing data to create the most accurate possible portrait of what happens to children placed in foster care. While the information burden is relatively low, costs arise from manipulating existing databases, like the Child Care Review Service (CCRS) and the State Central Register (SCR). During Phase One, the Task Force recommends that ACS use existing data to produce an annual report giving information on the specific outcome measurements, using admission cohorts as the foundation for the analysis. The report should include a separate analysis of child safety and permanency related outcomes for specific sub-populations of children. In addition, the report should include summary statistics on indicators of data quality. The Task Force also suggests that in Phase One, data be used to monitor contract agency performance for internal ACS review.

17

Annual Report: The Dynamics of New York Citys Foster Care Population While recognizing that the Administration for Childrens Services already publishes numerous reports describing the experiences of children in foster, the Task Force nevertheless recommends that the Administration produce an annual report describing the dynamics of New York Citys foster care population. The data presented in the report should be based on entry cohorts of children, treating children admitted for the first time separately from children re-entering placement. Research studies based on administrative data have repeatedly demonstrated the shortcomings of point-in-time or snapshot samples of children in foster care. Thus data from these samples often lead to erroneous conclusions about the experiences of children who are placed in foster care. A report that describes the experience of successive cohorts of children should correct for these problems and substantially improve what is already known about the placement experience of children. The report should focus on the indicators and outcome measures recommended by the Task Force: 1. 2. 3. 4. Length of stay, Number of placements, Re-entry rate and timing, and Exit rates by type of discharge.

The report should provide each entry cohorts size, as well as age, racial/ethnic, and gender composition. With respect to length of stay, the report should describe how long cohort members remain in care until their initial discharge, in 90-day increments through 18 months, and in yearly increments thereafter. In addition, the report should provide quartile distributions that describe how much time passes before 25 percent, 50 percent, and 75 percent of each cohort is discharged from placement. Data on the indicators themselves (length of stay, number of placements, etc.) should be reported by type of initial placement, age at first placement, and permanency planning goal. The analysis of movements should describe how the probability of movement from one type of foster care placement is influenced by the prior history of placement. The annual report should include a separate analysis that explores child safety and permanency outcomes for specific sub-populations of children. While the foster care system has broad responsibilities for children and families, certain populations of children are more likely to
18

enter placement, move between foster homes, remain in care for long periods, or re-enter care. Accordingly, the annual report should analyze outcomes for groups of children whose foster care outcomes have historically been poor and children whose placement experiences have recently changed. These sub-populations include: 1. 2. 3. 4. Sibling groups, Children admitted to placement before their first birthday, Children who experience three or more moves, and Children who are discharged after brief placements of 30 or 60 days.

Information for these populations should include, but not be restricted to: 1. 2. Length of stay and type of discharge (particularly for infants); The likelihood of placement into congregate care following several movements between foster homes; and 3. The likelihood of re-entry following brief foster care placements.

Children and families who live in New York Citys impoverished communities face particularly high placement rates. Special analysis should be devoted to understanding foster care outcomes in a community context. In part, the capacity to conduct analysis at the neighborhood level depends on the record-linkage strategies recommended for Phase Two. Still, relatively

simple analyses that group children according to a common zip code are already possible. Zip code analysis would provide a framework for understanding how community variables influence placement outcomes such as length of stay or reentry. This type of analysis addresses the issue of whether all things are equal in a small but important way. Finally, whenever computerized administrative records will be used to monitor programs and client outcomes, the question of data quality has to be addressed. While recognizing the longstanding problems frequently associated with CCRS and other administrative data sources, the Task Force believes that the existing resources provide an essential baseline against which longterm progress must be measured, both in regard to data quality and system performance. In keeping with this view, the Task Force recommends that the annual report include two additional sections 1. One that describe the time it takes case actions to appear on the
19

computerized files (reporting lag), and 2. One that measures how many child records were corrected (as opposed to updated, to reflect new case actions) since the last report. This section will be accompanied by a description of changes, if any, in the conclusions from previous reports related to corrected records. Where the evidence indicates persistent data quality problems, they should be analyzed in the context of current auditing procedures. The Task Force believes that applying existing data to agency management and outcome monitoring is a necessary if not sufficient condition for improving data quality over the long term. Monitoring of ACS and Contract Agency Performance Although it is important to understand how the foster care system is working as a whole, it is equally important to understand how the various components of the system are working. To accomplish this objective, the Task Force recommends that the outcome indicators for this domain be used to construct monitoring reports for individual agencies, using the indicators described above. These reports should then be used to evaluate contract agency performance, to monitor children in the direct care of ACS, and for internal ACS review. These internal reports would serve as the basis for corrective action plans, future contract negotiations with providers, and the exclusion of providers based on performance reviews. With respect to ACS, indicators of system performance such as length of stay and foster care reentry should be used to evaluate program initiatives. For example, efforts to improve efficiency and productivity can be understood in part by changes in how long children remain in care or the time it takes children to pass through service milestones, provided changes in other outcomes are (i.e., foster care reentry) are unaffected. Phase Two: Link Information Like most jurisdictions, the Administration for Childrens Services captures only limited data, so its database reveals only some of the details. As the analysis of outcome data becomes more sophisticated, this limited range of information limits what can be known about foster care outcomes. For example, the CCRS data for New York City does not contain reliable information on why a child was placed in foster care. As a result, any study of foster care outcomes that relies solely on CCRS can not use reason for placement as a way to understand outcomes.
20

Because existing data systems are limited, the Task Force recommends a second phase of work that focuses on linking information to augment and/or integrate existing data resources. There are three general strategies for augmenting the data in administrative data systems. 1. Redesign the systems to incorporate a wider array of data about clients, their

families, and the services they receive. Since the Connections Project is a redesign of existing child tracking systems, this option will not be described in this report. 2. Collect new data, either through case readings or another type of survey

designed to provide more information on a specific subset of the foster care population. In particular, the Task Force recommends that ACS: A. conduct periodic studies of the children placed in foster care. Case readings might be used to develop detailed clinical profiles of the children. These studies could be developed along with the analysis of placement stability that will emerge from Phase One. B. focus, in part, on the sub-populations identified in studies that depend on existing data resources for example, children who stay in care for short periods or children most likely to re-enter care. C. emphasize information on services received (frequency, duration, and intensity), since so little is known about the relationship between outcomes and services received. D. focus, in surveys of satisfaction and well-being such as those described below, on sub-populations of children defined on the basis of placement risk, placement duration, re-entry, age, or other characteristics. 3. Link existing sources of computerized data to form a more comprehensive

picture. For example, vital statistics records can be linked electronically to computerized foster care records. This linkage permits information about a childs status at birth to be used to understand the risk of placement, as well as the influence that factors such as birthweight have on placement duration, movements between foster homes, the likelihood of adoption, or the risk of returning to care. Notwithstanding the critical issue of confidentiality, record linkage represents an important tool in the effort to better understand foster care outcomes. Along these lines, the Task Force recommends that the Administration for Childrens
21

Services develop a data base that links administrative records (CCRS data) with other electronically stored data. The linked database can be maintained continuously (i.e., updated regularly in whole or in part) or developed to meet the needs of specialized studies. The data sources that should be linked include: A. Census data Census tract data is the best way to understand how communities impact child welfare outcomes. Moreover, any attempt to target services by community depends to a large degree on understanding differences in service utilization from one community to the next. B. Vital statistics records Over the past decade, nearly 40 percent of the children entering foster family homes have been infants. Linking birth records to CCRS would dramatically enrich the available data and allow a detailed understanding of placement risks and placement outcomes. Birth records for children admitted to foster care after their first birthday could also yield benefits. C. Other service data CCRS is not the only source of child- or family-level information available to ACS on a routine basis. Facility data, data from completed risk assessments, and data from the State Central Registry can also be integrated with CCRS data to enhance understanding of foster care outcomes. In addition, foster children and their families often receive services from providers outside the child welfare system. Subject to confidentiality regulations, whenever and wherever providers track case histories with computerized tracking systems, those data systems can be used to build a more detailed profile of clients, the services they receive, the problems they face, and the impact services have on their lives.

22

V. Domain Two: Child and Family Well-Being

Of the three domains the Task Force studied, outcomes related to child and family well-being proved to be the most challenging. On the one hand, the broad appeal of child and family well being measures reflects the child welfare system deeply held commitment to helping children and families succeed. On the other hand, first measuring and then interpreting changes in well-being that can be directly attributed to foster care is a substantial undertaking, especially if the results are going to be used in a comprehensive system of public and private agency accountability. The Challenge of Measuring Well-Being In considering these issues, the Task Force identified three points that bear on the use of well being indicators as outcome measures for the child welfare system. 1. There is no one set of well-being indicators that fits all the children in foster care. For any given child, the choice of appropriate indicators involves at least three considerations: A. B. C. 2. the childs age and developmental status, the length of time he or she has been in placement, and the objectives of the specific program providing care.

While foster care placement clearly impacts both the child and the family, isolating its specific impact on development over the life course is difficult. The task must be approached with caution.

3.

Despite the broad appeal of well-being indicators, the link between well-being and the mission of the foster care system must be carefully drawn. The foster care system must pay close attention to child and family well-being indicators in a systematic performance review. Nevertheless, there is a tendency to expand the list of social and developmental outcomes beyond those that are central to the mission of child welfare. The challenge is to construct a reasonable list of well-being indicators that neither trivializes the role foster care plays in child development nor overextends expectations.

23

Bearing these considerations in mind, the Task Force developed the following perspectives on child and family well-being indicators. With regard to the general category of child well-being, safety is the indicator of child well-being most central to the mission of child welfare. Consequently, maltreatment of one form or another serves as the primary indicator of child well being for all children, regardless of the type of program. Child maltreatment outcomes provide essential information for judging the quality of substitute care and for determining whether changes in other outcome measures, such as increases in reunification rates or decreases in length of stay, represent positive changes for children and families. Beyond safety, the scope of the foster care systems responsibility for child and family well-being outcomes is to a degree dependent on how long a child remains in foster care and the therapeutic expectations of a particular program. For example, when children are reunified with parents after a brief stay in a traditional foster family home (e.g., less than six months), the absence of subsequent maltreatment is a more direct measure of success than other measures of child well-being, such as social/emotional development or cognitive skills. Conversely, longer lengths of stay mean that the child welfare system must assume broader responsibility for the social, educational, and developmental well-being of children. Finally, outcomes for children placed in therapeutic placements should include additional measures of well-being or child functioning directly related to specific programmatic objectives, such as independent living. Still, if children and families experience difficulty following reunification but there is no maltreatment or apparent risk of imminent maltreatment, then those problems should perhaps be addressed outside of the traditional child welfare system. This conclusion underscores the importance of relating outcomes to the mission of child welfare, and reminds us that service continuity and improved child and family well-being often depend on integrated service delivery within a community context. In general, then, indicators of child and family well-being other than child maltreatment can be extremely useful in judging system performance particularly when they are examined in relation to the goals of child safety, permanence, and least restrictive placement. 1. Several negative indicators of child and family well-being, such as child behavior problems, domestic violence, and parental drug abuse, are important in this context. That is, from an outcomes perspective, concern about an issue like parental drug abuse stems from the increased likelihood of child maltreatment. 2. The wide array of child and family well-being indicators is most relevant for
24

judging system performance in relation to the length of time a child is in care and the type of care provided, including the quantity and types of therapeutic services. Long-term and intensive programs that address child and family functioning should be expected to affect these outcomes more than short-term and less intensive services. When the permanency goal is independent living, indicators such as vocational aptitudes and life skills become important. quantity of services. 3. Finally, indicators of child safety other than maltreatment are clearly important outcomes in and of themselves. Out-of-home care providers have a fundamental responsibility to address child safety outcomes, such as violence toward self and others, while children and youths are in care. Increases in these outcomes during care or following reunification or adoption must raise questions about the adequacy and effectiveness of services. Recommended Outcome Measures Table 2 presents the specific outcome measures and well-being indicators that the Task Force considered. Child well-being outcomes include indicators of safety and various dimensions of child functioning, such as physical and emotional health, relationships with family members and peers, appropriate behavior, cognitive development, cultural identification, and vocational and independent living skills. Family well-being outcomes include indicators of safety for family members, parental functioning and education, the problems of parents, and family relationships. These outcomes are important to examine both during placement and after reunification, adoption, or independence. During placement, ongoing assessment of child well-being can yield information that is useful for planning treatment and for determining whether change has occurred as a result of therapeutic interventions. After placement, measuring the childs safety and functioning can provide valuable information about the success of planned (or unplanned) permanency arrangements. Still, standards of performance for these programs also should depend on the length, scope, and

25

Table 2. Domain Two Indicators of Child and Family Well-Being

Area of Child and Family Well-Being

Indicators

Issues to Consider

Child safety

1. Child maltreatment including type, timing, frequency, and perpetrator.

1. Indicators of abuse and neglect reflect negative attributes of family life. Positive indicators emphasize the absence of abuse and neglect.

2. Serious harm or fatality caused by maltreatment. 3. Violence toward self and/or others. Family meets basic needs of 1. Health maintenance including children immunizations, well baby visits, and other developmentallly appropriate health care.
2. Nutritous foods. 3. Safe and adequate housing. 2. Housing and income serve as 1. Indicators must reflect what the

family does on behalf of children not simply what the family does not do.

general indicators because of their clse association with child and family well being.
4. Family promotes childs education. 5. Mental health care. 6. Income levels.

Child emotional and behavioral 1. Violence toward self and/others.

well-being

1. Indicators 7, 9, 12 and 13 express positive indications of childs overall adjustment. 2. Expectations and analysis of all the indicators in this domain must take the childs age and developmental status, and service history into account.

2. Stays at home (i.e., child doesnt run

away).

3. Acts in sexually appropriate ways. 4. Does not threaten or verbally abuse others. 5. Is drug and alcohol free. 6. Behavior patterns do not include firesetting, encopresis, eneuresis, etc.

26

7. Maintain control of self. 8. Is free of depressive symptoms. 9. Has positive self-regard grounded in cultural/ethnic identification. 10. Psychiatric hospitalizations are avoided. 11. Police contacts ar avoided - no arrests or re-arrests. 12. School attendance is grade and age appropriate. 13. Amount and quality of involvement with others, especially peer relationships that are positive and age appropriate. Includes relationships with other adults, relatives and family acquaintances.

Childs educational development, 1. School Attendance competencies, and achievements 2. Test scores are developmentally appropriate. 3. Grade point average is appropriate for age and developmental status. 4. Child talents (music, art, or other special skills). 5. Special accomplishments at home, in school, at the house of worship, or at work (i.e., awards, certificates of merit, participation in activities). Youth self-sufficiency/life skills 1. Has positive vocational skills and work attitudes. 2. Has employment experiences (depending on age). 3. Understands personal finances (savings and bill payments). 4. Uses social support as needed including family, community members, health care, and social services for recreational and other forms of support.

27

5. Has earned or is working toward

high school or equivalent diploma. Child Health 1. Child is free from harm caused by maltreatment. 2. Medical history including immunizations, well baby visits, and other developmentally appropriate helath care that is consistent with the childs health status and age. 3. Teens take steps to avoid pregnancy (including abstinence) and sexually transmitted diseases. Family well-being
1. Parents and other household members are drug and alcohol free (as determined through urinalysis if needed). 2. Ability of parents to function as household heads is not impaired for reasons related to their emotional or psychological well-being.

Parenting and parent child interactions

Parents protect children from maltreatment.

1.

2. Parents use appropriate disciplinary methods. 3. Parents exhibit interest in children, are motivated to carry out parental role, and look forward to reunification. 4. Bonding assessment shows positive attachments. 5. Parent(s) enjoy interactions with children, show respect and affection for children and vice versa.

28

Proposed Tools for Outcome Measurement Undertaking a study of child and family well-being costs money. Cost is one of the chief barriers to using well-being indicators in a system of performance monitoring and program evaluation. As in the case of client satisfaction indicators, the child welfare system in New York City and other parts of the country does not systematically process data that describes the well-being of the children or families involved in foster care. The information burden associated with measures of well-being has two specific dimensions. 1. Appropriate measurement tools. Sometimes existing instruments may be adapted, but not before a target population, program goals, and specific developmental issues have been identified. Target populations have different developmental needs, and programs have different objectives. The instruments used to measure outcomes have to reflect this diversity. Reliable baseline data. The cost of developing the basic research capacity to understand the impact of foster care on well-being has to be considered. Developing and maintaining data capacity to track child and family well-being over time will cost more than is now being spent. There will also be a time lag between the day data collection starts and the day when usable descriptive data are available. In addition, there will be a second, more substantial period before analysts are able to use that data to identify how foster care influences child and family development.

2.

Again, since so little baseline data exists, the Task Force proposes a two-phase strategy.

Phase I: Undertake a Study of Child and Family Well-Being The Task Force recommends that the Administration for Childrens Services complete a systematic study of child and family well-being within the next 12 months. The studys primary emphasis should be on child safety. Outcomes should be expanded to include developmentally appropriate indicators of well-being, such as school performance or readiness for independent living, when the childs age or other considerations warrant. Still, the scope of the initial assessment should be limited. Initial risk assessment and follow-up risk assessment data that ACS currently gathers should be factored into the overall design of Phase One. 1. The ACS study should be based on a random sample of foster children stratified by age, placement history (i.e., prior length of placement), placement type (direct care and children placed with voluntary agencies), and permanency planning goal. ACS

29

may wish to add other factors, such as Field Office or Borough. The sample should specifically include children recently admitted into care, with the expectation that their case histories will be followed over time. The sample should be large enough to permit meaningful analysis of well-being. 2. Where appropriate, the primary data collection instrument should be modeled on the safety assessment that is administered as part of risk assessment. Depending on the specific sub-population being reviewed, the initial risk assessment should be supplemented by a follow-up safety assessment. Wherever possible, for purposes of this limited study of child well-being, later collection of safety data should be done by someone other the person who completed the original (or most recent) assessment. Phase II: Create a Working Group The most important outcome questions have to do with the impact of foster care on a childs overall social-emotional development and the well-being of that childs family members. Although foster care is only one influence in the life of a child, it is important to understand how placement in foster care influences development over the life course. This question is never easy to answer. For this reason, the Task Force recommends supplementing systematic studies of child safety with routine studies of well-being among the children New York Citys foster care programs serve. A Working Group directed by the Administration for Childrens Services will develop the focus of these studies. Its issues should include the following: 1. Instrument development:The working group should establish procedures and criteria for developing instruments to measure well-being. Criteria for selection include which populations the sample should include; which outcomes, including safety and other age-appropriate measures, should be targeted; and how frequently the instruments should be administered. 2. Role of the voluntary agency: In addition to establishing the broad parameters for a systematic study of well-being outcomes, the working group should also address the responsibility of contract agencies for studying child well-being outcomes, including safety. 3. Reporting standards: Finally, the working group should develop reporting standards that permit comparative analysis of agency performance for purposes of
30

contracting. Further, the working group should weigh carefully how corrective action should be related to the reporting standards developed.

31

VI. Domain Three: Client Satisfaction

Measures of client satisfaction with services are a valuable source of feedback for policymakers, program administrators, and direct service personnel. The consumers perspective on the efficacy of services (e.g., overall satisfaction and satisfaction with specific services) and important dimensions of service provision (e.g., worker skill and communication issus) may influence service outcomes such as family reunification. Interest in client satisfaction has grown in recent years, as models of child welfare practice (including foster care) have increasingly stressed the importance of including natural parents, extended family members, and foster parents in case planning and treatment. Parallel developments in business that stress quality assurance standards and consumer satisfaction have also spurred interest in this rapidly developing area. The Procurement Policy Board in New York City now requires periodic assessments of client satisfaction as a condition of receiving a contract. The Importance of Client Satisfaction Measures Interest in client satisfaction with services is wholly consistent with the mission of child welfare. Given the focus of the child welfare system on permanency for children, the perspectives of children and youths are a valuable and largely underused resource for understanding and evaluating the effects of services. Second, the ideal of preserving families whenever possible is based on the significance of natural parents involvement in the lives of their children. In assessing the child welfare systems performance, clients (both parents and children) can provide essential information about whether reasonable efforts are made to preserve and reunify families. For example, youths and parents can be asked whether services met their needs and if necessary services were available. Examining their views of various dimensions of the quality of service provision (e.g., cultural sensitivity, good communication with clients and among professionals, client involvement in decision making) provides useful information on the adequacy of service provision. Client satisfaction data can complement system performance outcomes by enriching our understanding of and ability to interpret a variety of data. For example, if clients voice concerns about the availability of aftercare services at a time when length of stay is declining, the findings may be used to strengthen post-discharge services. Alternatively, if length of stay is rising but clients report high levels of satisfaction with reunification services, it may be desirable to increase the availability of services with an eye toward reducing time in care. How client satisfaction data

32

will be linked to other measures of performance depends on the purpose of the analysis. Clearly, the clients voice adds a dimension that has been missing from most discussions about how well the foster care system is working. Recommended Outcome Measures The Task Force recommendations that correspond specifically to client satisfaction can be found in Table II. In reviewing the available literature and talking with clients, four areas within the client satisfaction domain emerged as particularly relevant to the foster care system in New York:2 1. overall satisfaction, 2. satisfaction with the workers providing services, 3. satisfaction with specific services, and 4. satisfaction with communication. Within each of these areas, Table 3 also lists specific indicators that form the basis for conducting focus groups or for constructing instruments that would be used to survey larger groups of clients.

Task Force recommendations in this section are based in part on the results of a focus group conducted with eight parents whose children were or had been in foster care. The results are not intended to be representative of all the individuals who have feedback to give. Nevertheless, the comments and reactions revealed during the focus groups indicate how important client satisfaction is to framing an overall sense of outcomes. The results of the focus group are printed in Appendix X.

33

Table 3. Domain Three Client Satisfaction

Area of Client Satisfaction Overall Satisfaction 1. Indicators Issues to Consider

2.

3. 4. 5. Satisfaction with Workers Who Provide Services 1. 2. 3. 4. 5. 6. 7. 1. 2. 3. 4. 5. 6. 7. 8.

Global ratings of satisfaction Overall satisfaction with specific services (e.g., case management, counseling, substance abuse treatment, probation, education, recreation, respite care, etc.) Overall satisfaction with specific workers Would client recommend program? Would client return to services? Skills Empathy Availability Honesty in communication Respect for clients Level of trust Cultural sensitivity including language spoken Safety of child Visitation opportunities Availability of services
Availability of client advocacy and support groups

1. Information is easy to aggregate across programs, not very specific or refined. 2. Provides general impressions, little specificity about reasons for level of overall satisfaction 3. What is relationship of overall satisfaction to system performance outcomes?

1. Information should be relevant to

various professionals or easily adapted

Satisfaction with Specific Services

2. Should specify which worker 3. May be relevant to overall satisfaction and to system performance outcomes 4. Should include foster parents as well as workers 1. Services usually refers to specific programs, but satisfaction with case management services, a childs placement, and visitation are also important 2. Aspects of service delivery can be practice principles or ideals (e.g., family centeredness: Allen and Petr, 1995) 3. May be related to overall satisfaction and to system performance outcomes

Waiting lists/delays Aspects of service delivery Did services meet needs? Length and amount of services 9. Sorroundings/facility 10. Satisfaction with placement 11. Confidentiality 12. Satisfaction with outcome 13. Cultural sensitivity

34

including language 14. Sensitivity to religious preferences Satisfaction with Involement 1. Informed of events 3. Involved in decision making 4. Time to communicate with lawyers 5. Inter-agency collaboration 6. Adequate opportunity to be heard from in court 1. May be related to overall satisfaction and to system performance outcomes

Communication and 2. Informed of rights

Proposed Tools for Outcome Measurement

Few child welfare agencies today systematically gather data on client satisfaction. Moreover, few client satisfaction surveys have been field-tested with foster children and their parents. The instruments that do exist have been administered by individual programs, but not on a system-wide basis. For all these reasons, the cost of developing assessment tools and conducting regular surveys of client satisfaction-the information burden-will be considerably higher than for other recommended measures. While the instruments used to assess satisfaction can be quite short and easy to administer, data collection, data entry, and analysis across all cases present a secondary, but nevertheless substantial cost. So that measures of client satisfaction become a regular form of feedback within the child welfare system, the Task Force makes the following recommendations.

Phase I: Implement Guidelines and Surveys To meet the short-term need for information from clients, the Task Force recommends two steps that ACS could carry out as Phase One of a comprehensive plan to gather client satisfaction data. 1. Implement guidelines for uniform questions that would be compatible with the requirements of the Procurement Policy Board. 2. Implement a targeted survey of client satisfaction modeled after the rapid assessment technologies used in the health and behavioral health care sectors. Initially, these assessments should involve small, well-targeted samples of children. Rapid assessment technology is an approach to client feedback that emphasizes short, easily answered questionnaires. Data can be quickly entered and analyzed.
34

Development of the rapid assessment technology should be both guided and complemented by feedback from focus groups of children and parents. Feedback from clients should be sought in four basic areas:

1. 2. 3. 4.

Demographic and service information, Overall satisfaction, Child safety, and Progress toward permanency.

Instruments should be administered to a sample of children stratified by age, permanency planning goal (or time in the system), and type of placement. The sample should be large enough to show meaningful differences in satisfaction. If children are too young to participate directly, parents should be asked to complete the assessment. When identifying potential samples, special consideration should be given to clients taking part in innovative programs, so client satisfaction data can be included in the program evaluation. Until a more detailed alternative can be developed, the rapid assessment should be administered to children and their families in panels of roughly the same size. Children who remain in care would remain part of the sample until they are discharged. A sample of children and families should be interviewed at the time of discharge. Where possible, and subject to confidentiality standards, data gathered as part of the assessment should be linked to administrative data from other sources, including data from the Child Care Review Service (CCRS) and the State Central Registry (SCR). This will allow the analysis of client satisfaction to reflect the childs service history and other factors. Phase One should include three additional steps 1. Encourage providers who are already collecting client satisfaction data to use some common questionnaire items. These common questions might measure overall satisfaction and satisfaction with specific services and/or workers; Require both public child welfare agencies and private providers to annually survey small, representative samples of clients by program (e.g., residential, foster care, family reunification) or group (natural parents, youth, children) in order to collect baseline data systemwide; Conduct periodic focus groups in order to hear the views of clients in different programs; to gather input on program decisions, staff training, and program
36

2.

3.

development; and to amplify the findings from other studies of system performance and client well-being. Phase II: Establish an Advisory Panel ACS should establish an Advisory Panel to develop a long-range strategy for gathering client satisfaction input. The Advisory Panel could begin by developing instruments for Phase One, but its primary purpose would be to develop a long-range plan that addresses costs and procedures for measuring client satisfaction across representative populations and their service providers. The Advisory Panel ACS convenes should reach agreement on the following issues: 1. Outcomes related to overall parent and child satisfaction, satisfaction with specific services, satisfaction with workers, and satisfaction with communication. Detailed descriptions of these areas and specific indicators associated with each area are provided in Table 3, above. Recommendationson the continued use of rapid assessment technology and the adaptation of more detailed instruments for in-depth analysis of sub-populations. Standards for assessment that identify the target populations and the frequency of assessment. Plans to bring recipients of services other than foster care into the satisfaction review, including but not limited to recipients of preventive services, adoption services, and child protective services. Provisions for including a broader array of clients, so parents, foster parents, adoptive parents, and provider organizations have a regular means of providing feedback. A plan that describes how the collected information will be used and how it will be disseminated. A statement that defines the responsibilities of public and private agencies for assessing client satisfaction. In particular, the advisory panel should consider whether the tracking of client satisfaction data can be integrated with the Connections Project, and if so, how.

2. 3. 4.

5.

6. 7.

37

VII. Summary
Properly constructed and used, outcomes and outcomes measures can have a dramatic impact on the quality of child welfare services available to New York Citys children. Yet, identifying those outcomes is but a first step in the long process of improving the child welfare system. Along with outcomes, the system itself must maintain a separate but equally important focus on the both the structure and processes that define child welfare service delivery. In the end, it is the combination of process and outcomes that are inextricably linked to highly effective services.

Members of the Comptrollers Task Force readily acknowledge that this report is a work in progress. The integration of outcome data into the delivery of services depends on yet another level of specificity and detail that was beyond the scope of this report. Moreover, the use of outcome data will place new demands on the individuals who work with families each and every day. Their need for new competencies must be acknowledged if outcome data is to be collected reliably and, more importantly, used productively. Finally, no one should under estimate the difficult task of analyzing and interpreting outcome data. If the data are to be used to understand what placement in foster care means for children and their families, policy makers among others must recognize that complex human problems often times command complex answers.

In short, this Report of the Comptrollers Task Force represents an attempt to move the state of the art but one small step forward in the hope that the questions raised and the answers provided will stimulate the important work that remains to be done.

38

Appendix A Methodological and Conceptual Issues that Arise in Conducting Research on Child and Family Well-Being

Methodological and Conceptual Issues that Arise In Conducting Research on Child and Family Well-Being

Like client satisfaction, measures child and family well-being pose important methodological and conceptual hurdles that must be addressed before indicators of well-being can be introduced into a system of routine feedback and performance review. As an orientation to these issues, the following review of the literature is intended to serve as a guide for those interested in understanding how others have used these measures to understand the impact of the child welfare system on child and family well-being.

Connecting System Peiformance Indicators and Child and Family Well-Being. There have been some efforts to examine the relationship between child and family functioning and system performance such as length of stay and foster reentry. The most developed discussion of this issue occurs in evaluations of Wraparound services for behaviorally challenged youth and their families. Researchers recognized the need for socially valid indicators of service effectiveness to complement clinical measures of functioning, which often yield fairly abstract findings that are difficult for people outside of the professional research community to interpret (Bruns et al., 1996). Furthermore, measures of clinical adjustment, while worthwhile, did not address issues of accountability and cost-related concerns typically encountered by public agencies. In response, they developed a set of behavioral indicators that providers, advocates, and community residents thought would be most likely to lead to placement of the youth, resulting in a set of indicators are quite simple, reliable, and easily interpretable. Similarly, some highly regarded substitute care programs, such as the independent living programs at The Children's Village, utilize both traditional clinical assessments of functioning (e.g., locus of control) and concrete indicators of functioning (e.g., work experiences, school success--which can be seen as system performance outcomes for these programs) in examining outcomes.

Data Collection Strategies. Data on child and f amily well-being can be collected from parents, youth, and professionals--usually professionals who are working with children and/or families or who are conducting CPS investigations. Data collection strategies designed to gather quantitative data on child and family well-being include risk and family assessment instruments, child behavior and functioning instruments, and other research tools, such as rapid assessment instruments, goal

attainment scaling, and global assessments of functioning. The following discussion is intended to illustrate some potentially useful strategies and is not a comprehensive list or review of such measurement instruments.

There are risk assessment and family assessment instruments that are specifically designed for use in child welfare. These instruments are intended to be applicable to all child welfare cases and they are of direct relevance to understanding system performance because they assess the factors that influence the likelihood of subsequent maltreatment and whether children can be protected. Assessment instruments are used to guide case planning and decision making and sometimes to set workload requirements (based on the level of risk). These assessments are conducted during both the initial phase of intervention or investigation and, in most approaches to risk assessment, also at one or more subsequent stages in a case.

A wide variety of risk factors are examined by various risk and family assessment instruments, including dimensions of parenting, history of maltreatment, parent-child interactions, individual functioning of parents (e.g., substance abuse, mental illness), family functioning (e.g., domestic violence), family support, and child behavior.

Pecora et al. (1992) describe four approaches to risk assessment. 1) Matrix approaches in which various risk factors, usually identified by clinicians, are rated on a scale that is based on descriptions of levels of severity for each factor. Many states, including New York, Illinois, and Washington, use such matrices, which are the only child and family well-being measurement instruments that are sometimes administered system-wide. These instruments are relatively easy to use and do not represent an additional information burden in terms of data collection. 2) The empirical predictoi;s method attempts to determine the specific factors that are most predictive of subsequent maltreatment (e.g., Baird, 1988; Johnson and L'Espearance, 1984). Such analyses result in a relatively small number of fairly simple and objective factors (e.g., prior involvement with the child welfare system) that predict subsequent maltreatment in 70-83% of cases. The factors used in the matrix approaches have also been examined for their predictive validity in a small number of studies and they yield similar predictive accuracy (e.g., Marks and McDonald, 1989). 3) Family assessment scales (e.g., McCroskey and Nelson, 1989; Magura and Moses, 1986; Magura, Moses, and Jones, 1987) are not technically designed to predict risk, but they are designed to assess parent, child, family, and household functioning in child welfare cases. These instruments use behaviorally anchored scales and provide more refined measurement of functioning than risk matrices approaches, but they are also more time consuming. The Child WellBeing and Family Risk scales (Magura and Moses, 1986; Magura, Moses, and Jones, 1987)) have

been shown to be sensitive to change over time in studies of the effects of family preservation services (e.g., Rodenheiser, Chandy, and Ahmed, 1995). Other important assessment tools designed for use in child welfare include The Childhood Level of Living Scale (Polansky, 1981), which addresses issues of child neglect the Family Assessment Interview Guide (Phillips, 1989). 4) The Child at Risk Field (CARF) (Holder and Corey, 1986) uses open-ended questions and rating scales to examine factors that can influence the risk of future harm, child safety, and child well-being. This is the most comprehensive approach to risk assessment practice and it is the only approach that incorporates risk assessment and plans to protect the child and ameliorate risks into all phases of decision making (including, for example, the decision to reunite children with parents). This approach has a fairly high information burden since it requires a great deal of training and time filling out forms, and there has been relatively little work done on the predictive validity of this model (Pecora et al., 1992).

Despite the relevance and increasing popularity of risk and family assessments in child welfare, Wald and Woolverton (1990) note that child welfare risk assessment instruments are not ready for use as actuarial tools and that they have important methodological and conceptual problems, including: the same instruments and factors may not be applicable to different scenarios (e.g., predictors of reabuse when children are left at home after adjudication may differ from predictors of reabuse when children are reunified) and different types of maltreatment (i.e., different factors may be predictive in abuse vs. neglect cases); instruments don't address or predict the severity of reabuse; risk assessment approaches generally do not consider that factors such as the severity of harm are sometimes more important in initial decision making than factors that predict subsequent harm; predictive models do not take into account the provision of services, which are generally designed to alter risk factors; there are no cut points for making decisions in most models; and there is no quantification of what high or low risk mean. Other issues to be addressed risk assessment approaches include ethnic and gender biases, interactions between factors, and nonlinear effects, and the need to include family strengths (Horesji, 1987; Howing, Wodarski, Gauding, and Kurtz, 1989; Pecora, 1991; Pecora et al., 1992).

Assessment of chitd behavior and functioning. As noted above, child behavior and functioning are of particular importance in evaluating system performance in relation to child safety and placement or permanency decisions. There are a variety of ways of assessing and recording the behaviors and functioning of children and youth over time. Two approaches are briefly described to illustrate the use of a specific behaviorally oriented measurement instrument within broader, well-conceptualized measurement strategies that could provide useful tools and ideas for expanding data collection in

this area.

The Child Behavior Checklist (CBCL) (Achenbach and Edelbrock, 1983) is a 113 item instrument used by many child welfare providers with strong histories of conducting research on child wellbeing in substitute care (e.g., The Children's Village and The Casey Family Program) that is used periodically (e.g., every 4-6 months) to examine change over time. The psychometric properties of the CBCL have been refined and tested extensively and it provides well-sampled norms and clinical cut points that allow the practitioners to determine the severity of a child's problems relative to the general population and clinically defined populations. It measures nine problem areas that cover a wide range of potential problems in functioning and well-being. These areas are grouped within two primary domains, internalizing (schizoid or anxious, depressed, uncommunicative, obsessive compulsive, and somatic complaints) and externalizing (delinquent, aggressive, hyperactive, social withdrawal). Since no one measure of functioning is complete or perfectly "objective," Achenbach and McConaughy (1987) stress the need to obtain data from multiple informants (e.g., parents and teachers) and from multiple types of assessment (e.g., direct observation and standardized instruments) in order to obtain a coherent and relatively complete picture of child functioning. They propose a multiaxial assessment strategy by age of the child that includes the CBCL, teachers reports, cognitive assessment, health assessment, and direct assessment.

Evaluators of wraparound services for behaviorally challenging children and youth in Vermont and Alaska have developed a system for tracking behavior and adjustment that is part of a multiaxial tracking strategy that also examines the restrictiveness of care (Hawkins, Almeida, Fabry, and Reitz, 1992), critical life events, the onset of behavior problems, service provision, educational services, reading level, and the cost of services (Bruns, Froelich, and Burchard, 199). The 24 items included on adjustment indicator checklists were "identified by parents, service providers as most indicative of a child's risk of placement into a more restrictive setting" (Bruns, Froelich, Burchard, Yoe, and Tighe, 1996). Items yielded four empirically based factors: externalizing (items include physical aggression, life threat, property damage, extreme verbal abuse), public externalizing (e.g., police contacts, alcohol and drug use, theft, truancy), abuse-related (e.g., sexual acting out, self-injury), and internalizing (e.g., sad, anxious). Measurement instruments are available for daily, weekly, or quarterly administration. In one study, scales of the negative behaviors and compliance percentage on the Daily Adjustment Indicators Checklist (DMC) were found to correlate significandy with CBCL total problem scores. The DMC was found to have good criterion validity--scales were significantly associated with program costs and level restrictiveness (CBCL total problem scores were not significantly associated with these variables).

Advantages of these checklists and the overall approach include: 1) their explicit link to system performance and accountability issues; 2) socially valid items that are relatively simple, objective, and easily interpretable; 3) flexible administration of instruments; 4) information can be used to guide interventions because it is collected and analyzed frequently, using tracking software that incorporates multiple measures and produces excellent illustrations of case level patterns over time; and 5) items are well-suited for tracking across settings.

Other potentially useful data collection tools include rapid assessment instruments, goal attainment scaling, and global assessments of functioning. Rapid Assessment Instruments (RAI) are relatively short standardized instruments that are designed to measure specific problems frequently over time in order to provide feedback to practitioners and evaluate interventions. There are hundreds of RAIs and they address almost any conceivable aspect of child, adult, and farally functioning (see Fischer and Corcoran, 1994, for comprehensive discussion RAIs in general and for critical information about specific measures). Appropriate use is dependent on applicability of a given instrument to an individual client's problems and the availability of information on the reliability and validity of instruments varies considerably. Hudson (1982) developed a set of RAls for social work interventions (including instruments measuring self-esteem, depression, and family relations) that have shown strong reliability and validity, and that are easy to administer, score, and analyze. Goal attainment scaling provides a tool for standardizing measurement of goal achievement for client-specific goals and for weighting the importance of various goals (Kiresuk and Sherman, 1968; Bloom, Fischer, and Orme, 1994). Global assessment scales (Bloom et al., 1994; Southward, 1996) provide a way for clinicians to determine levels of overall functioning across client problems and diagnoses. Separate instruments are available for children and adults. The reliability and validity of these instruments has not been sufficiently tested.

Measurement across settings over time. It is both important and difficult to measure child and family well-being across settings over time. Any assessment, whether and assessment of the family or the system's performance more generally, should endeavor to understand the child and family problems and strengths that existed before, during and after placement in substitute care.

Yet there are important challenges in examining change across settings, primarily related to the interpretation and measurement of change across placement and permanency transitions. When children move from home to placement or vice-versa, there may be changes in both child and family functioning simply as a result of the transition and unrelated to enduring changes in the underlying reasons for services. For example, children and youth sometimes temporarily conform to behavioral expectations ("honeymoon") when they first enter substitute care and this can

complicate interpretations of baseline data on functioning and comparisons across settings. Strategies for addressing this problem in studying the functioning of children in substitute care include collecting separate baselines for the child's functioning in the community and in substitute care (Nelson, Singer, and Johnson, 1973, 1978) and collecting data over longer periods of time and across placement experiences in order to identify fluctuations. And as noted earlier, some measurement approaches were developed for tracking behaviors across settings (e.g., Bruns et al., 1996).

Measurements of child safety and certain aspects of parental functioning are particularly difficult in family reunification programs when children and parents are not living together. Supervised and unsupervised visits can provide important, but still limited information about interactions and parental behavior. Given the limitations of such data, other data must be used. For example, parents' level and chronicity of ambivalence about the parental role and reunification provide a straightforward and useful indicator that has been found to be associated unsuccessful reunification efforts (Hess and Folaron, 1991).

A critical component of reliable measurement is that procedures for administering an instrument remain constant over time (Bloom, Fischer, and Orme, 1994; Hudson, 1982). It is important to keep in mind that changes in settings over time can disrupt the consistency of measurement procedures (e.g., different raters or contexts).

The importance and difficulty of obtaining follow-up data. Clearly, the safety and well-being of children following cessation of child welfare services is of great importance to the public and to professionals. Researchers in residential treatment have stressed the importance of collecting follow-up information in order to ascertain if child and youth functioning levels are maintained after treatment and to better understand the factors that are associated with reentry into care (e.g., Pecora et al., 1992; Wells, 1991). However, in general, it is difficult and time-consuming to obtain adequate response rates in follow-up surveys, especially with child welfare populations. The only widely available measures are subsequent reports of child maltreatment and the reoccurrence of placement. Administrative data from corrections, mental health, public health, and public assistance are also available and could provide a substantial amount of relevant follow-up data if the data were linked, through existing technologies, to child welfare databases.

Quality of information issues in measuring functioning. The information about children and families that is available to professionals on a given case changes over time. For example, the amount and quality of information available during a CPS investigation is limited by the short time
6

frame and the conflictual context of the information gathering (Budde, 1993). Thus, there is sometimes little information available about risk factors of particular importance in initial investigative decision making such as prior maltreatment, substance abuse, and domestic violence. Conversely, much more is often learned about families during the course of service provision and within the context of therapeutic relationships (Littell Schuerman, Rzepnicki, Howard, & Budde, 1993). This shift in the quatity of information over time can affect the interpretation of findings on system performance outcomes. For example, the provision of family preservation services (FPS) resulted in a slight but statistically significant increase in substantiated reports of maltreatment in Illinois--this illustrates the case finding effects that services can have (Schuerman, Rzepnicki, and Littell, 1994). That is, it is unlikely that FPS actually caused increased maltreatment and quite likely, given frequent contacts with parents in the home, that service provision led to better information about maltreatment. This suggests that, in research terms, the quality of information can be a confounding factor when analyzing outcomes.

Intermediate or instrumental outcomes. A fundamental assumption of most risk or family assessment approaches is that initial assessments can identify factors (usually in the individual functioning or caregiving behaviors of parents) that should be addressed during the course of service provision in order to reduce the likelihood of maltreatment. Positive change over time in levels of severity of risk factors or in ratings of progress on case level goals can be described as improvement in intermediate or instrumental (Nelsen, 1978) Outcomes that are important steps toward, or are facilitative in achieving the primary outcomes of interest (e.g., decreased maltreatment, reunification). Intermediate or instrumental outcomes are of central analytic interest from an outcomes perspectives because they can refine our understanding of the process by which system performance goals are achieved. For example, Fein and Staff (1993) examined the relationship between ratings of progress on case level goals and reunification outcomes. While there was a statistically significant relationship between progress and reunification, there was considerable variability--that is, a some families that were not reunified made progress on goals during the first three months. Thus, progress in specific areas is not necessarily sufficient to influence the decision to reunify.

It should be noted that what is considered an intermediate outcome and what is considered a primary outcome is dependent on the way in which the purposes of child welfare or specific research questions are framed. For example, the occurrence of reunification can be considered an intermediate outcomes in relation to stability of reunification or long term child safety and wellbeing (Usher, Gibbs, and Wildfire, 1995), particularly if one is studying the effects of

reunification on permanency, subsequent maltreatment, or child well-being.

Assessment data and outcomes. Assessment data are often collected with the sole intent of informing decision-making and with no effort to examine change over time using that instrument. Such data can still be useful in providing information about case characteristics and family problems that can enrich analyses of system performance indicators. Phillips (1989) has developed a particularly useful instrument for informing decision-making and collecting data that could be used in subsequent analyses.

Difficulty in finding adequate comparison groups. It is critical to dramatically improve the child welfare system's ability to track and describe outcomes. Efforts to move beyond descriptive research to address questions of program effectiveness require bases for comparison. The strongest comparison group, called a control group, is obtained through randomized assignment procedures used in experimental studies. However, because of ethical concerns surrounding implementing random assignment procedures in situations involving the safety and well-being of children, evaluations in child welfare generally do not use experimental designs to clinical effectiveness. For the most part, weaker forms of comparison group designs are used in which similar cases receive different types of treatment (Pecora et al., 1992). This means that researchers must focus more on learning about how change occurs over time and about the relationships between outcomes and case characteristics and specific services. One potentially strategy that has been employed in research on health care outcomes involves identifying case-mix variables--case factors that are predictive of certain levels or rates of outcomes--that can provide bases for comparisons (Smith, Rost, Fischer, Burnam, & Burns, in press).

Utility of aggregate data for informing practice. While aggregate research findings are often of great value to administrators and policymakers who must address system-wide policies, practitioners often suggest that such findings are of less value in informing practice (Pecora et al., 1992; Whittaker, 1979). A wide variety of research strategies have been developed to try to produce findings that are relevant to researchers, including systematic data collection on case specific outcomes over time in single case designs (e.g., Bloom, Fischer, and Orme, 1994); consideration given to how aggregate findings should be included in case level judgments (e.g., Berlin and Marsh, 1993); ratings of progress on individual goals that can also be standardized across cases; standardized measures (e.g., the DAIC--Bruns et al., 1996; rapid assessment instruments, Fischer and Corcoran, 1994) that can be used frequently in order to provide ongoing feedback to practitioners about client change. In addition, qualitative approaches to research can provide valuable information, illustrating, for example, the impact of permanency decisions (e.g.,
8

Folaron's of the perspectives of parents and children following reunification) and promising approaches to helping children who have experienced repeated discharges from placements (Heineman Pieper and Pieper, 1995). The most important general point to be made here is that producing research findings that will useful to practitioners often requires greater depth of inquiry into individual cases and this can provide a valuable complement to system-wide data collection efforts on outcomes.

Disagreements about the utility and appropriateness of some data collection efforts. It should not be assumed that any typeof systematic data collection would be good if only the information burden could be addressed. There are disagreements among professionals about what data should be collected and how the data should be collected. For example, proponents of the Teaching Family Model utilize a daily point card as part of a cognitive and behavioral approach to intervention in substitute care settings, but others see no benefits and some problems with such approaches. Casey and Berman (1985) found that it was difficult to compare approaches to psychotherapy for children because different approaches focused on different kinds of target problems. Tyson (1995) and Heineman Pieper (1989) have raised concerns about the biases and iatrogenic effects of some data collection strategies (e.g., standardized instruments that clients fill out during counseling sessions, tape recording) and illustrate the use and benefits of naturalistic data collection approaches (such as anamnestic process recordings) that do not interfere with treatment. Thus, m making decisions about data collection, it is important to discuss and examine the benefits and disadvantages of any strategy.

Information burden and proposed responses. Since risk assessment matrices are the only type of data collection that is currenfly administered at the system level, the additional information burden for these instruments lies only in the area of data analysis. Despite important limitations and conceptual problems, these instruments provide an opportunity for system-wide feedback on safety related outcomes that should be further developed to increase the utility and validity of these instruments. Washington and Texas two of the few states that have entered and analyzed risk assessment data on an ongoing basis and the City should make every effort to learn from this research. Efforts should be made to explore the current training and utilization of matrices and to link these data with system performance outcome data.

The information burden in research on child and family well-being is increased by most data collection strategies that provide important benefits, such as using multiple measures, more frequent administration of instruments, gathering follow-up data, and in-depth study of case-level change. Global assessment scales and goal attainment scaling provide potentially useful and viable
9

strategy for standardized measurement across cases system-wide or for specific populations. Strategies such as the risk and family assessment instruments described earlier, the CBCL, the DMC, the Ansell-Casey Life-Skills Assessment, and the Rudson scales should be among those approaches considered for use in longitudinal studies for targeted populations, depending on the goals. Indicators of youth functioning for programs oriented toward independent living should include fundamental outcomes such as those used by the Children's Village (1994) in evaluating the Way Scholarship program: staying in school, completing high school, working, and going to college.

10

Appendix B Methodological and Conceptual Issues that Arise in Measuring Client Satisfaction

Methdological and Conceptual Issues that Arise in Measuring Client Satisfaction

While client satisfaction with services is recognized as an increasingly important source of feedback within the child welfare system, gathering and accurately reflecting the level of client satisfaction is a complicated undertaking. Below, we have reviewed some of the major methodological and conceptual issues that must be addressed as part of a comprehensive effort to make client satisfaction an integral part outcome monitoring.

General concerns about client satisfaction surveys. Godley (1995, citing Nguyen, Attkisson, and Stegner, 1983) notes that "Global consumer satisfaction studies have been criticized from a number of perspectives including (a) the usual finding that service recipients report high levels of satisfaction; (b) the lack of meaningful comparison bases; (c) the lack of a standard scale, and (d) the difficulty in avoiding sampling bias." This represents a good set of issues that need to be kept in mind throughout the process of planning, administering, and analyzing client satisfaction surveys.

Methods of obtaining the views of clients--data collection strategies. Clients' views of services can be obtained through survey instruments and interviews with individuals or groups (i.e., focus groups), and by including clients and client advocacy organizations in ongoing discussions of existing services and reform initiatives. Each of these strategies has benefits and limitations. Because of the emphasis in this report on tracking information over time, the use of survey instruments is highlighted here. Client satisfaction surveys or questionnaires can provide baseline data that can serve as the basis for examining changes over time for individuals and populations. Surveys also can help to identify areas of particular concern to clients. Interviews and focus groups have the advantage of providing more in-depth and specific information about the concerns and suggestions of clients in their own words (rather than in response to questions constructed by researchers), and including clients in ongoing discussions of services and reforms can provide valuable and direct feedback to policymakers about client concerns. Neither of these strategies provide easily standardized data that are useful for ongoing quantitative analyses, but they can be used in combination with other data to provide a more complete picture of outcomes and to aid the interpretation of system performance indicators. Who is the client? Monitoring clien satisfactionfor various provider-consumer relationships can

provide useful information and perspectives on the quality of services. Viewed broadly, clients in the foster care system can include children and youth, natural parents, involved relatives or family friends, foster parents, frontline staff (as consumers of program reforms), agencies that purchase services (e.g., when a public agency contracts with a private agency to provide services or when a private agency subcontracts with another agency), and the local community (where the family resides and, if different, where services are provided). In order to focus on practical steps that can be taken toward including the perspectives of clients, this report focuses on the satisfaction of children, youth, and natural parents with services provided through the foster care system. Children's views have rarely been sought in research endeavors, even in foster care (an example of an exception: Bush & Gordon, 1982; Johnson, Yoken, & Voss, 1995; Wald, Carlsmith, and Leiderman, 1988). It is important to include the perspectives of both parents and children, since they may differ in their levels of satisfaction and in the areas that they deem important (e.g., Stuntzner-Gibson, Koren, and DeChillo, in press; Nelson, 1992). Questionnaires may also need to be revised for use with children and youth. For example, using scales that are familiar to children and youth, such as letter grades and happy or sad faces, can help ensure that they understand how to rate items (e.g., Grimes, 1996). We would recommend that, at a later stage, the perspectives of foster parents, relatives, and staff be selectively included in surveys of satisfaction.

Satisfaction with what? With whom? Client satisfaction can be used to assess specific interventions (e.g., parent education or one day training programs) or for more general or multifaceted interventions (e.g., residential treatment, wraparound services), and they can pertain one worker (e.g., a home visitor or a therapist) or a whole program. Surveys that examine dimensions of specific services provided by specific professionals (e.g., parent education done by a home visitor) can provide refined and relevant information to program administrators but may be of liffle value to other types of child welfare programs. More general surveys that examine overall satisfaction with services or workers are easily applied across programs and clients. Some agencies (e.g., Children's Home and Aid Society of Illinois) address these various weaknesses and strengths by including a small set of questions used in all programs along with items that are specific to the program.

Since global ratings of satisfaction can be easily standardized across settings, they have the potential of providing system wide baseline data and norms for various types of programs and clients. In addition, some dimensions of reliability and validity of such instruments (e.g., the CSQ8) have been examined in mental health and health settings with fairly positive results. One

additional limitation of such measures is that it is very difficult to establish the client's point of reference in making judgments since overall satisfaction can be influenced by the client's experiences with one worker, aspects of the worker's approach, a particular service, or by social desirability factors. When client characteristics and the various indicators described in Table 3 are also examined, quantitative analyses can examine the factors that correlate with overall satisfaction.

Response rates and selection bias. An important concern in administering client satisfaction surveys is that of ensuring an adequate response rate--at least 80% of people who receive the survey should respond in order to have reasonable confidence that findings are representative of the views of the sample being studied. Response rates can also be affected by the length time it takes to complete a survey and the method of administration (by mail or in person). Selection bias problems are likely to be exacerbated when data are collected after discharge, since clients will be harder to find and those who are least satisfied are least likely to stay involved with service providers and to fill out questionnaires. Godley (1995) suggests sampling from the population of current clients during a certain time frame, though she notes that, for clinic based services, there may still be a selection bias problem because those who tend to miss appointrnents will be less likely to respond. Focus groups can be used to supplement surveys when there are low numbers or rates of responses for specific groups of people (e.g., unsuccessful reunification cases or racial groups; Cynthia Parry, American Humane Association, personal communication).

Desirability effects. When responding to a survey, clients may wish to please service providers or specific workers. This can result in a type of measurement error in which ratings of satisfaction are higher than actual satisfaction. This is one possible reason why client satisfaction surveys tend to produce high levels of satisfaction. It is likely that such desirability effects are exacerbated when surveys are administered by program staff (particularly if the staff person works directly with the client) and when surveys are administered during intervention--when some clients may fear negative consequences if they respond negatively.

Appendix C Summary of 1996 Focus Group with Parents Regarding Client Satisfaction

Summary of 1996 Focus Group with Parents Regarding Client Satisfaction

INTRODUCTON Given the importance of client satisfaction in child welfare, it was important to involve parents in helping us thinik about outcomes in foster care. Steve Budde and Joe Rodriguez met with a group of parents to learn their views of the foster care system and of the best ways to go about gathering information from parents on client satisfaction. The group was convened by Mabel Paulino, coordinator of the Child Welfare Organizing Project--a client advocacy organization for parents and children involved in the foster care system based in New York City. This summary provides an example of how focus groups can be used to provide valuable feedback from clients. The group included eight parents whose children are or were previously in the foster care system and one young woman with siblings in foster care. One parent voluntarily placed her child and the other placements were involuntary. Several people previously had drug problems, but they had completed drug programs and their children were returned to them. There were seven women and two men in the group. In the first part of the meeting, parents described their concerns about the system, investigations, private agencies, juvenile court, and workers' responses. Participants then provided feedback on the wording and the importance of a series of questions that could be asked of parents on a client satisfaction survey; they also suggested additional questions to ask parents. Finally, parents discussed what should be done to improve the foster care system, and how they think that the city and the state should learn about parent's opinions on an ongoing basis. BASIC CONCERNS Parents voiced numerous serious concerns about the child welfare system based on their experiences with investigators, private agency workers, foster parents, and juvenile court judges and lawyers. These concerns included: Lack of compassion for natural parents from professionals. The group strongly stated that their experiences with professionals often provide little compassion and support, and that they are not treated or respected as individuals, but instead as case numbers or as labels (e.g., drug addict). Workers focus to much on "criteria and rules" and often have little experience with people (they often have workers who are just out of school). Parents were very negative about going to court and having a lawyer who only talked with them for five minutes, didn't know their situation, and then was supposed to represent them adequately in court. Foster parents are often seen by workers as saviors while natural parents are viewed as "enemies." Workers and foster parents are particularly harsh on parents with drug problems--they don't realize that these parents often care about their children or that they are individuals. Another way that workers sometimes lack compassion and focus on compliance with criteria is by "dangling reunification with children like a carrot" as a reward or a punishment. Some parents feel that ongoing drug testing, after a parent has stayed clean for an extended time, shows that professionals are just waiting and looking for
1

parents to fail rather than to succeed--this is frustrating to parents and may actually lead to more failures. The parent who voluntarily placed her child noted that she was treated in the same disrespectful manner as the other parents. Parents need support when they enter the system and when children are returned. Workers often do not realize what is like for a parent to be thrust into the child welfare and everyone is trying to find out all the dirt on you and you do not know the system--parents need support in these situations. For example, workers often assume that parents should go on their own to drug treatment--they don't understand how hard it is to go to drug treatment, the strength of the parent's denial, or how important and helpful it is for someone to go with you to start treatmenL Similarly, parents emphasized that aftercare is essential--because parents who have kicked drugs and have their children back will have to learn how to live life in new ways. An example of the lack of support during aftercare: a parent who had completed a drug treatment program and had been reunited with her children asked a worker to be available to help her when she got her check so that she wouldn't spend it on drugs--the worker replied, "I'm not your babysitter." Parents expressed great concern about anticipated cuts in funding for aftercare services and that current financial support from the child welfare system (up to $300 per month) is insufficient. The negative effects of placement on their children. Natural parents are concerned about the negative effects on their children due to separation and having to live away from home: "Children often fall apart in foster care." Several people suggested that professionals and foster parents often don't know what is best for their children. They complained about children not taking baths, being allowed to stay out late, and having severe behavior or school problems that were not present when the children lived at home. Sometimes children--whose problems are the result of their experiences in foster care--are taken to psychiatrists, diagnosed ADHD, and put on medication--this angers natural parents. It was also noted that some children are verbally or physically abused in foster care. Lack of information about children provided to parents. There was very strong concern expressed by all participants that natural parents and adult siblings do no receive adequate information. The most pressing complaint involved not knowing where children were placed or who the worker was for weeks or months after a placement was made. They understood that in abuse or domestic violence situations, the child's whereabouts should be kept secret for safety reasons. But the child's safety was not a problem in most instances and they believe that parents should know where children are as soon as possible. They suggest that not knowing your child's whereabouts, even for short periods of time, can cause tremendous strain on natural parents and can make drug problems worse. Parents also often lack information about critical medical and educational issues, even when they retain legal responsibility for such matters. Delays. There was concern about delays that parents experienced, particularly but not solely related to not knowing where their children are. Other delays included court delays (often because workers were absent), delays in responding to parents' inquiries, and delays in getting information caused by workers and lawyers not being available or reachable. Lack of information about rights. Parents suggest strongly that they do not receive adequate
2

information about their rights and their options. For example, natural parents are usually told to keep quiet during court proceedings and do not know that they can have their attorneys ask the judge if they can speak in court. Some parents do not know about their rights to know about their child's treatment or to be involved in decisions. Because they do not know the system and their rights, parents often agree to recommendations by professionals with which they disagree (e.g., waiving their parental rights or pleading guilty). It was noted that parents are often scared to ask about their rights or to complaln about problems because "the worker has your life in his hands." Also, agencies often do not provide information to natural parents about parent advocacy or support groups that will help them pursue their rights--parents say that this is because social workers and agencies often see parent groups as threatening. The rights of adults with siblings in foster care are unclear and often ignored.

Parents do not have adequate input into decisions. Consistent with the above, parents suggested that they do not have adequate input into decisions at any stage of the process. There were several specific areas in which parents emphasized that their views and opinions should be given much more weight included. They felt very strongly that they should have a choice of caseworkers, therapists, and lawyers--and that this should include being able to change to a new person when professionals were not helpful or treated them badly. Parents praised agencies that allowed them to have this kind of input and choice. Some people noted that they had gotten better professionals when such changes occurred. Parents wanted professionals who would treat them as an individual and understand, support, and advocate for them. Another area of inadequate input involved lack of consideration given to the religious, ethnic, and language preferences of the parent when foster home placements were made.

Lack of understanding of drug addiction and what parents with drug problems need. Parents who had previously had drug problems emphasized that workers often do not understand drug addiction. Examples of this from the above discussion include the rigid view of people with addictions, the lack of support, lack of understanding of denial, and insufficient aftercare services.

Financial incentives working against reunification. Some people said that agencies want to keep children in placement because the funding for placement supports their budgets and organizations. Similarly, it was suggested that court appointed lawyers have an incentive to complete cases so that they can keep getting paid. Foster parents also have a fmancial incentive to keep children.

The system sets up foster parents against natural parents. Natural parents suggest that the system views them as enemies and foster parents as saviors. Foster parents buy this, creating tension between these foster and natural parents when they really should be trying to work together to help kids return home.
3

QUESTIONS TO ASK NATURAL PARENTS The group agreed on a number of questions to ask parents on surveys of client satisfaction. Thesequestions were adapted from existing client satisfaction surveys, developed by us, or suggested by parents. Questions that were suggested or modified by parents are designated PS (parents suggested) or PM (parents modified). Parents also stressed that many questions depended on the circumstances of each individual client--some questions may not be relevant to some clients. Consistent with their emphasis on being treated as individuals, parents made excellent suggestions to address such problems: they suggested asking screening questions (questions that establisb that other questions are relevant to an individual) and asking for explanations of what parents meant by an answer. This should not be viewed as a complete list, since other surveys contain relevant and useful questions that were not reviewed with parents because of time constraints. I. Services A. B. C. D. E. F. G. . H. A.

Does your agency provide any services to you? This is a screening question, if the answer is yes, then the other questions can be asked. (PS) Are there any important services that are not available for you or your children? Did you have to wait for a long time to receive any services? Were you offered services before your children were placed in foster care? Have the services you have received been helpful to you in addressing your concerns or solving problems? (Need to relate to specific services) Have the services you received helped you to be a better parent? (How or Why?) Have you been negatively affected by any cuts in services at your agency? (PS) Is your child able to attend the religious services of your choice? (PS) Does your agency have a parent advocacy or parent support group? (PS) 1. If yes, are you involved in the group? (PS) a. If yes, is the group helpful to you? (PS)

II. Parent advocacy

B. C. A. B. C. D. E.

Does your agency have a parents bill of rights? (PS) Do you know your rights? (PS) Where is your child placed? (What type of placement?) (PS) Has your child's placement been a positive experience for him or her? 1. If yes, explain (PS) Are you able to have regular visits with your child? 1. If yes: Is the place where you visit your child close by or far away? (PM) (If relevant) Do you have a good relationship with the foster parent? (PS) How long did it take you to find out where your child was placed? (PS)
4

III. Your childs placement

F. G.

Did you have any choice about where your child was placed?(PS) Were your preferences in the following areas taken into account in the placement that was made (PS): 1. Religion? 2. Etlinicity? 3. Language?

IV. Involvement in decisions (other than placement) A. B. C. D. E. F. G. A. B. C. D. E. F. Have you been given choices of services for you and your child? In general, have you been included in decisions about your children? Do you still have the right to decide about your child's medical care? (PS) Have you been included about decisions about your child's medical care? (PS) Do you still have the right to decide about your child's education? (PS) Have you been included about decisions about your child's education? (PS) Are you able to change workers if you are dissatisfied with your worker? (PS) Were you informed of important meetings or court hearings? Is it easy to get in touch with your worker? Does your worker return phone calls as quickly as possible? Does your worker keep you informed about what is happening in your case? Do different professionals communicate well with each other about your case? Have you had difficulty communicating with workers because of a language barrier? 1. If so, were you then provided with someone who spoke your language? VI. Workers (can relate to more than one worker) A. B. C. D. E. F. G. H. Does your worker treat you respectfully? Is your worker concerned about you as a person? Does your worker understand your opinions? Is your worker well organized and competent? Do you feel comfortable talking with your worker? Can you depend on your worker? Was your worker truthful or "straight" with you? Did your worker make you feel like problems were all your fault?

V. Communication

PARENTS' SUGGESTIONS FOR IMPROVING ThE FOSTER CARE SYSThM

Agencies should encourage parent advocacy groups. Parents suggest these group members can work with current clients and sometimes help in ways that professionals can't because they have experienced what clients experience. For example, some of the participants in the focus group work closely and successfully with clients at St. Christopher's to help them get into drug treatment programs. Agencies should be required to tell parents about advocacy organizations. Parents should receive training on the court system--provided by advocacy groups and Legal Aid rather than by the court system. Parents should be involved in providing training to workers and court personnel about how to communicate and work with natural parents. Parent advocacy groups should be expanded. (Although participants in this focus group noted that it is very hard for parents who are still in the system to look beyond their individual needs and to become involved in such groups.)

PARENTS' VIEWS OF ThE BEST WAYS TO LEARN ABOUT CLIENT SATISFACflON Open meetings: the city is having these now, but parents are often not informed because agencies do not post the information. More outreach and hand outs are required. Surveys that would ask questions such as the ones listed above. Focus groups: do this only if it is going to mean something. Focus groups: do this only if it is going to mean something.

Appendix D References

References
Selected Child Welfare Outcome Frameworks American Human Association & National Association of Public Child Welfare Administrators. (1994). A framework for outcome measures in child welfare services: Executive summary. Prepared for The Second Annual Roundtable on Outcome Measure in Child Welfrzre Services. San Antonio, TX. American Humane Association & National Association of Public Child Welfare Administrators. (1996). Matrices of indicators. Prepared for The Fourth Annual Roundtable on Outcome Measures in Child Welfare Services. San Antonio, TX. Casey Family Program. (1996). Casey family program practice framework and outcome domains. Seattle: Author. Usher, C. L., Gibbs, D. A., & Wildfire, J. B. (1995). A framework for planning, implementing, and evaluating child welfare reforms. Child Welfare, 74, 859-876. Child Welfare System Performance Barth, R. P., Courtney, M., Needell, B., Jonson-Reid, M. (1994). Performance indicators for child welfizre services in California. Berkeley, CA: Child Welfare Research Center, School of Social Welfare, University of California at Berkeley. Berrick, J. D., Barth, R. P., Needell, B. (1994). A comparison of kinship foster homes and foster family homes: Implications for kinship foster care as family preservation. Children and Youth Services Review, 16, 33063. Courtney, M. E. (1995). Reentry to foster care of children returned to their families. Social Service Review, 69 (2), 225-241. Courtney, M. E. (1996). Pathways of older adolescents out of foster care: Implications for independent living services. Courtney, M. E., & Collins, R. C. (1994). New challenges and opportunities in child welfare outcomes and information technologies. Journal of the Child Welfare League ofAmerica, Inc., LXXIII (5), 359-378. Goerge, R. M. (1990). The reunification process in substitute care. Social Service Review, 64, 3, 422-457. Goerge, R. M., Van Voorhis, J., and Lee, B. J. (1994). Illinois's longitudinal and relational child and family research database. Social Science Computer Review 12, 3, 351-365. Goerge, R. M., Wulczyn, F. H., & Harden, A. W. (1994). Foster care dynamics 1983-1992: Cahfornia, Illinois, Michigan, New York, and Texas. A Report from the Multistate Foster Care Data Archive. Chicago: Chapin Hall Center for Children at the University of Chicago.

Hawkins, R. P., Almeida, M. C., Fabry, B., & Reitz, A. C. (1992). Restrictiveness of living environments for troubled children and youths: A simple measure for program evaluation, policy planning, and placement decisions. Hospital and Community Psychiatry, 43, 54-59. Wulczyn, F. H., & Goerge, R. M. (1992). Foster care in New York and Illinois: The challenge or rapid change. Social Service Review, 66, 278-294. Child and Family Well-Being References (not including risk assessment) Achenbach, T. M. & McConaughy, S. H. (1987). Empirically based assessment of child and adolescent psychopathology: practical applications. Newbury Park, CA: Sage Publications. Achenbach, T. M. & Edelbrock, C. (1983). Manual for the Child Behavior Checklist and Revised Child BehavioT Profile. Burlington: University of Vermont, Department of Psychiatry. Bruns, E. J., Froelich, P., Burchard, J. D., Yoe, J. T., & Tighe, T. (1996). Tracking behavioral progress within a children's mental health system: The Vermont community adjustment tracking system (VT-CATS). Bruns, E. J., & Woodworth, K. (1994). User's Manual to the Vermont System for Tracking Client Progress Data Eritry and Graphing Programs for the creation ofMulti-Axial Lift events Timelines and Behavioral Adjustment tracking Graphs: 2nd Edition. Burchard, J., & Bruns, E. (1993). Daily adjustment indicators. A User's Guide to the Weekly Adjustment Indicator Checklist. Burlington, VT: University of Vermont, Department of Psychology. Burchard, J., & Bruns, E. (1993). Local children's service system interagency collaboration checklist A User's Guide to the Weekly Adjustment Indicator Checklist. Burlington, VT: University of Vermont, Department of Psychology. Burchard, J., & Bruns, E. (1993). Monthly adjustment indicator checklist. A User's Guide to the Weekly Adjustment Indicator Checklist. Burlington, VT: University of Vermont, Department of Psychology. Burchard, J., & Bruns, E. (1993). Quarterly adjustment indicator checklist. A User's Guide to the Weekly Adjustment lndicator Checklist. Bufling ton, VT: University of Vermont, Department of Psychology. Burchard, J., & Bruns, F. (1993). Weekly adjustment indicator checklist. A User's Guide to the Weekly Adjustment Indicator Checklist. Burlington, VT: University of Vermont, Department of Psychology. Burchard, J., & Bruns, F. (1993). A User's Guide to the Weekly Adjustment Indicator Checklist Burlington, VT: University of Vermont, Department of Psychology. Burchard J. & Lewis. (1995). Parent 30 day wraparound services checklist and satisfaction rating

scale. Parent Checklist. Burchard, J. D. & Schaefer, M. (1992). Improving accountability in a service delivery system in children's mental health. Clinical Psychology Review, 12, 867-882. Casey Family Program. (1996). Ansell-Casey Life Skills Assessment Manual: Field Test Version 1.0. Casey, R., & Berman, J. (1985). The outcome of psychotherapy with children. Psychological Bulletin, 98, 388-400. Children's Village. (1994, October). The Way Scholarship Program: Work Appreciation for Youth. Corcoran, K., & Fischer, J. (1994). Measures for Clinical Practice: A Sourcebook, Second Edition, Volume 1: Couples, Families, and Children. New York: The Free Press. Endicott, J., Spitzer, R., Heiss, J., & Cohen, J. (1976). Global Assessment Scale: A procedure for measuring overall severity of psychiatric disturbance. Archives of General Psychiatry, 33, 766-771. Folaron, G. (1993). The impact of family reunification: the experiences of four families. (Doctoral dissertation, University of Illinois at Urbana-Champaign, 1992). Dissertation Abstracts International, 29(3), 1099. Hudson, W. (1982). The clinical measurement package: A field manual. Homewood, IL: Dorsey Press. Nelsen, J.C. (1978). Intermediate treatment goals as variables in single-subject research. Social Work Research and Abstracts, 20, 3-10. Kiresuk, T. J. & Sherman, R. F. (1968). Goal attainment scaling: A general method for evaluating comprehensive community mental health programs. Community Mental Health Journal, 4, 443-453. Margura, S., & Moses, B. S. (1986). Outcome measures for child welfare services: Theory and application. Washington, DC: Child Welfare League of America. Margura, S., Moses, B. S., & Jones, M. A. (1987). Assessing risk and measuring change in families: The Family Risk Scales. Washington, DC: Child Welfare League of America. Rodenhiser, R. W., Chandy, J., & Ahmed, K. (1995). Intensive family preservation services: Do they have any impact on family functioning? Family Preservation Journal, Summer, 69-85. Smith, G. R. (1994, November 14). The State of the Science of Mental Health and Substance Abuse Patient Outcomes Assessment. Little Rock, AR: Department of Psychiatry, University of Arkansas for Medical Sciences. Smith, G. R., Rost, K. M., Fischer, F. P., Burnam, M. A., & Burns, B. J. (in press). Assessing the effectiveness of mental health care in routine cljnical practice: critical components, administration, and application of outcomes modules. Litde Rock, AR:

Department of Psychiatry and Behavioral Sciences, University of Arkansas for Medical Sciences. Ware, J. E., & Sherbourne, C. D. (1992). The MOS 36-item short-form health survey (SF-36): 1. Conceptual Framework and Item Selection. Medical Care, 30 (6), 473-

Risk and Family Assessment Baird, C. (1988). Development of risk assessment indices for the Alaska Department of Health and Social Services. In T. Tatara (Ed.), Validation research in CPS risk assessment: Three recent studies. Washington, DC: American Public Welfare Association. Budde, S. (1993). Dilemmas in risk assessment: The example of cocaine exposed infants. In National Roundtable on CPS Risk Assessment: Summary of Highlights (pp.49-74). Washington, DC: American Public Welfare Association. English, D. J., & Pecora, P.J. (1994). Risk assessment as a practice method in child protective services. Journal of the Child Welfare League ofAmerica, Inc., LXXIII (5), 451-474. Holder, W.M., & Corey, M. (1986). Child protective services risk management: A decision making handbook. Charlotte, NC: ACTION for Child Protection. Johnson, W., & L'Espearance, J. (1984). Predicting the recurrence of child abuse. Social Work Research and Abstracts, 20, 21-26. Marks, J., & McDonald, T. (1989). Risk assessment in child protective services.' Predicting recurrence of child maltreatment. Portland, ME: National Child Welfare Resource Center for Management and Administration, University of Southern Maine. Martinez, L. (1989). The family assessment factor worksheet (mimeograph). Springfield: Illinois Department of Children and Family Services. McCroskey, J., & Nelson, J. (1989). Practice-based research in a family-support program: The family connection project example. Child Welftare, 68, 573-587. Phillips, M. (1989). Family Assessment Interview Guide: 1989 Revision. New York: Fordham University Graduate School of Social Service. Pecora, P.J. (1991). Investigating allegations of child maltreatment: The strengths and limitations of current risk assessment systems. Child and Youth Services, 15(2), 73-92. Wald, M. S., & Woolverton, M. (1990). Risk assessment: The emperor's new clothes? Child Welfare, 69, 483-511. Selected General References on Outcomes and Child Welfare Research Bloom, M., Fischer, J., & Orme, J. (1994). Evaluating practice: Guidelines for the accountable professional, 2nd ed. Englewood Cliffs, NJ: Prentice-Hall. Curry, J. F. (1991). Outcome research on residential treatment: Implications and suggested directions. American Journal of Orthopsychiatry, 61(3), 348-357. Daro, D., Abrahams, N., & Casey, K. (1990, September). Parenting Program Evaluation Manual.' Second Edition. Chicago: The National Center on Child Abuse Prevention Research, a Program of the National Committee for Prevention of Child Abuse.

Fein, E., & Staff, I. (1993). Last best chance: Findings from a reunification services program. Journal of the Child Welfare League ofAmerica, Inc., LXXII(l), 2540. Gleeson, J. P., Bonecutter, F. J., & Altshuler, S. (in press). Facilitating permanence in kinship care: The Illinois project. Kinship Care Forum Summary. New York: National Resource Center for Permanency Planning. Hess, P. M., & Folaron, G. (1991). Ambivalences: a challenge to permanency for children. Child Welfare, 70(4), 403-24. Hess, P. McCartt, Folaron, U., & Jefferson, A. Buschmann. (1992). Effectiveness of family reunification services: An innovative evaluative model. National Association of Social Workers, Inc. Horesji, C. R. (1987). Child welfare practice and the Native American family in Montana: A handbook for social workers. Missoula: University of Montana, School of Social Work. Howing, P. T., Wodarski, J. S., Kurtz, P. D., & Gauding, J. M., Jr. (1989). Methodological issues in child maltreatment research. Social Work Research and Abstracts, 25(3), 3-7. Littell, J. H., Schuerman, J. R., Rzepnicki, T. L., Howard, J. & Budde, S. (1993). Shifting objectives in family preservation programs. In E. S. Morton & R. K. Grigsby (Eds.), Advancing Family Preservation Practice. Newbury Park, CA: Sage. Marsh, J. C. & Berlin, S. B. (1993). Informing Practice Decisions. New York: Macmillan Publishing Company. Nelson, K. (1992, March). Assessing interventions: Beyond the "Black Box" in social work research. Paper presented at the Annual Program Meeting of the Council on Social Work Education, Kansas City. Nelson, R. H., Singer, M. J., & Johnsen, L. 0. (1973). Community considerations in evaluation of a children's residential treatment center. Proceedings of the 81 st Annual Convention of the American Psychological Association, 8, 951-952. Nelson, R. H., Singer, M. J., & Johnsen, L. 0. (1978). The application of a residential treatment model. Child Care Quarterly, 7, 164-175. Pecora, P.J., Whittaker, J. K., & Maluccio, A. N. with R. P. Barth & R. D. Plotnick. (1992). The child welfare challenge:Policy, practice, and research. NewYork:Aldine De Gruyter. Pieper, M. Heineman. (1989). The heuristic paradigm: A unifying and comprehensive approach to social work research. Smith College Studies in Social Work, 60, 8-34. Pieper, M. H., & Pieper, W. J. (1995). Treating violent "untreatable" adolescents: Applications of intrapsychic humanism in a state-funded demonstration project. Th K. B. Tyson (Ed.) New foundationsfor sciennflc social and behavioral research: The heuristic paradigm Needham Heights, MA: Atlyn and Bacon. Polansky, N. A., Chalmers, M. A., Buttenweiser, E., & Williams, D. P. (1981). Damaged

parents: An anatomy of child neglect. Chicago: University of Chicago Press. Schuerman, J. R., Rzepnicki, T. L., & Littell, J. H. (1994). Putting Families First: An Experiment in Family Preservation. New York: Aldine De Gruyter. Tatara, T. (Ed.). Validation research in CPS risk assessment Three recent studies. (Occasional Monograph Series of APWA's R&D Department.) Washington, DC: American Public Welfare Association. Wald, M. S., Carlsmith, J. M., & Leiderman, P. H. (1988). Protecting abused and neglected children. Stanford, CA: Stanford University Press. Wells, K. (1991, July). Long-term residential treatment for children: Introduction. American Journal of Orthopsychiatry, 61(3), 324-326. Whittaker, J. K. (1979). Caring for troubled children: Residential treatment in a community con text. San Francisco: Jossey-Bass. Yoe, J. T., Santarcangelo, S., Atkins, M., & Burchard, J. D. (1996). Wraparound care in Vermont: program development, implementation, and evaluation of a statewide system of individualized services. Journal of Child and Family Studies, 5 (1). Outcomes From the Client's Perspective Allen, R. I., & Petr, C. G. (1996). Toward developing standards and measurements for family centered practice in family support programs. In G. H. S. Singer, L. E. Powers, & A. L. Olson (Eds.), Redefining family support: Innovations in public-private partnerships. Baltimore MD: Paul H. Brookes Publishing. Attkisson, C. C., & Greenfield, T. K. (1994). Client satisfaction questionnaire - 8 and service satisfaction scale - 30. In M.E. Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessmenL Hillsdale, NJ: Lawrence Eribaum Associates. Burchard, J., & Bruns, E. (1993). Youth satisfactionlinvolvement and unconditional care questionnaire. A User's Guide to the Weekly Adjustment Indicator ChecklisL Burlington, VT: University of Vermont, Department of Psychology. Bush, M. & Gordon, A. C. (1982). The case for involving children in child welfare decisions. Social Works, 4, 309-14. Carrizosa, S., Poertner, J., & Knutson, L. (Date unclear). Home Visitor Satisfaction Survey. Lawrence, KS: University of Kansas, School of Social Welfare. Coulter, I. D., Hays, R. D., & Danielson, C. D. (1994). The chiropractic satisfaction questionnaire. Gaithersburg, MD: Aspen Publishers, Inc. Godley, S. H. (1995). Plan for statewide parental and youth consumer satisfaction survey. Bloomington, IL: Lighthouse Institute. Grimes, S. (1996). Parent Satisfaction Survey: Residential Program. Park Ridge, IL: The Youth

Campus. Grimes, S. (1996). Parent Satisfaction Survey: Treatment Foster Care. Park Ridge, IL: The Youth Campus. Grimes, S. (1996). Treatment Foster Care: Client Satisfaction Survey. Park Ridge, IL: The Youth Campus. Larsen, D. L., Attkisson, C. C., Hargreaves, W. A., & Nguyen, T. D. (1979). Assessment of clientipatient satisfaction: Development of a general scale. Evaluation and Program Planning, 2, 197-207. Johnson, P., Yoken, C., & Voss, R. (1995). Family foster care placement: The child's perspective. Child Welfare, 74 (5), 959-974. Nguyen, T. D., Attkisson, C. C., & Stegner, B. L. (1983). Assessment of patient satisfaction: Development and refmement of a service evaluation questionnaire. Evaluation and Program Planning, 6, 299-213. Pamperin, B. F., & Bailey, B. (Date unclear). Parent Education Program Evaluation. Menomorrie, WI: Department of Social Science, University of Wisconsin Stot. Southward, K. K. (1995, November 15). Community Treatment Home Program Evaluation Staff Input Survey. Chicago, IL: Children's Home & Aid Society of Illinois. Southward, K. K. (1995, November 15). Community Treatment Home Program Evaluation 1996 Client Satisfaction Survey for Youths. Chicago, IL: Children's Home & Aid Society of Illinois. Southward, K. K. (1995, November 15). Community Treatment Home Program Evaluation 1996 Client Satisfaction Survey for Biological ParentslFamilies. Chicago, IL: Children's Home & Aid Society of illinois. Southward, K. K. (1995, November 15). Design Overview of the Parent Survey for CHASI Child Day Care Programs. Chicago, IL: Children's Home & Aid Society of Illinois.