An Historical Guide to the WIA Performance Measures

Georgia Department of Labor February 2007

February 2007

Please note: This guide was created after the initial enactment of WIA in 2000. With the implementation of Common Measures in 2006, certain elements contained within this document have become obsolete. For the most current information regarding performance that all local areas within Georgia should be following, please refer to the GDOL Common Measures Resources Guide, which was distributed on March 2006, and the USDOL Training and Employment Guidance Letter (TEGL) 17-05. This TEGL may be accessed on the DOL-ETA web site at http://wdr.doleta.gov/directives/attach/TEGL17-05.pdf

1

February 2007

Introduction This guide has been prepared to help local partners involved in planning for WIA to understand the WIA performance measures. Basic information is presented in “Q&A” format in the main body of the guide. More detailed or technical information is contained in appendices that we hope will be useful to those most directly involved in performance measurement. While some important questions regarding common performance measurement remain unanswered at this time, this guide includes what we know now and lays out what you can expect to happen next. We will update and expand the guide in the months to come. Concern has been expressed about placing emphasis on performance measurement at this point in the development of a program that should be customer-driven. We hope that the process of designing local programs will be highly collaborative, focusing on customer needs and the resources available for meeting those needs. Within that process, the performance measures should be seen as parameters within which programs must operate − much like allocation size and applicable laws and regulations. Although we expect that the performance measures are meaningful indicators of how well our workforce investment system is serving its customers, performance on these measures should never become an end in itself. Rather, it should be a product of well-designed and wellimplemented systems, responsive to local needs and both building on and expanding local opportunities.

2

February 2007

Understanding the WIA Performance Measures 1. Why are the WIA performance measures important?

The WIA performance measures are one component in the comprehensive WIA performance accountability system. Other components include the system for identifying eligible training providers, the related “consumer report card,” and shared outcomes and comparable performance measures among partner programs. Together, these components focus attention on: The importance of performance accountability at all levels The interdependence of partner programs The right of the public to valid consumer information and quality service Because outcomes are shared across all programs with a common customer, collaborating programs have a strong interest in one another’s success. At the same time, incentives and sanctions at each level create consequences for performance. Providers may be removed from the eligible provider list for poor performance or attract customers with strong performance. Local areas may be subject to corrective actions for failure to meet their negotiated performance levels, but states may give incentive awards for high levels of performance. States may have their allocations reduced for failure to meet negotiated performance levels, or earn incentives generated from funds withheld from other states. Each level’s success depends on the level below, so each level has an interest in doing what it can to support and promote that success. But most of all, the performance measures are important because they represent the aspirations of our customers − for skill development and acquisition of meaningful credentials, for stable employment over time, for improved earnings, and for positive experiences when they seek our help. 2. What are the performance measures and to whom do they apply?

The WIA performance measures are listed in Table 1, along with the groups of customers to which each measure applies. Here are the rules to remember: • Except for the two customer satisfaction measures, the performance measures apply separately to customers served under three funding streams: adults, youth, and dislocated workers. Within youth services, measures apply separately to older (19 to 21) and younger (14 to 18) youth, based on the age of the customer at registration. Youth age 18 through 21 may be served as youth, adults, or both. Outcomes of customers served under more than one funding stream are counted under each appropriate set of performance measures.

• • •

3

February 2007

Customer satisfaction measures apply across all funding streams to two groups of customers: employers and participants.

Table 1 WIA Performance Measures and Applicability by Customer Group Youth Older Younger Youth Youth (19 – 21) (14 – 18) X X X

Dislocated Performance Measures Adults* Workers Entered Employment Rate X X Employment Retention Rate X X Earnings Change in Six Months X Earnings Replacement Rate X Employment & Credential Rate X X Credential Rate Skill Attainment Rate Diploma or Equivalent Rate Retention Rate Customer Satisfaction Rates: Participants X X Employers * Includes youth age 18 to 21 served as adults. 3.

Employers

X X X X X X X

Within each customer group, are all customers counted in the performance measures?

No, only those customers who are registered under WIA are counted in the performance measures. 4. • • • Which customers must be registered under WIA? Local areas may draw on a variety of resources in providing workforce services, but only customers who receive services funded under the WIA must be WIA-registered. All youth receiving services of any level under the Title I youth funding stream must be registered under WIA. Among adults and youth age 18 to 21 served under the adult or dislocated worker funding streams, WIA registration depends on the level of service received. Only those customers receiving core services with significant staff assistance, intensive services, or training funded by WIA must be WIA-registered.

4

February 2007

5.

What types of services are included in “core services with significant staff assistance, intensive services, and training”?

Table 2, below, lists types of WIA services and provides examples of each. Note that Georgia has not identified any core services that involve “significant staff assistance,” so registration is effectively limited to recipients of WIA-funded intensive services and training.

Table 2 WIA Service Levels and Registration Requirements WIA Service Level Core Information Service Examples General information such as pamphlets or directions Resources such as books or videos Labor market information Consumer Report information (hard copy) Training provider and local Workforce Investment Board program performance reports (hard copy) Information on filing Unemployment Insurance claims Job fair information Orientation to the one-stop Internet browsing Resources on-line, including: Information on job availability, including Job Information System Information on partner agency and local community services Information on support service availability Information on financial aid Self-directed assessment, Georgia Career Information System Labor market information Consumer report information Training provider and local Workforce Investment Board program performance reports Information on filing Unemployment Insurance claims Internet registration for America’s Job Bank, etc. Outreach Group workshops in interviewing, job search, resume writing, financial management, etc. Job fairs WIA Registration Required? No

Core Self-Service

No

Core Staff-Assisted (Minimal Intervention)

No

5

February 2007

Table 2 WIA Service Levels and Registration Requirements WIA Service Level Service Examples Initial assessment Service needs evaluation Information and referral to services Job matching, job referrals, job search assistance, or job development Job clubs Career guidance, Georgia Career Information System Follow-up activities and reassessment for services Filing Unemployment Insurance claims and Worker Profiling Unemployment Insurance Eligibility Review Program Comprehensive, in-depth and diagnostic assessment In-depth interviewing and evaluation Development of individualized employment plan Individual or group career counseling Service coordination or case management Basic workforce readiness, pre-vocational skills Adult basic education, GED preparation Out-of-area job search, relocation assistance Internships, work experience Referrals to training Intensive job development Occupational skills training On-the-job training Programs that combine workplace training with related instruction Training programs operated by the private sector Skill upgrading and retraining Entrepreneurial training Job readiness training Adult education and literacy activities in combination with other training services Customized training WIA Registration Required? No

Core Staff-Assisted (Moderate Assistance)

Intensive Services (Significant Assistance)

Yes, if WIA funded.

Training Services

Yes, if WIA funded.

6

February 2007

6.

Are WIA registrants who are receiving services from a partner program, such as Technical Education, Adult Literacy, or Vocational Rehabilitation, still considered active WIA participants?
Tracking Participation in Partner Programs Suppose a customer enrolls in a technical institute following WIA-funded intensive assessment and career counseling at his local one-stop. The one-stop assigns a case manager to maintain contact with the participant, offering additional services as needed and recording the customer’s activities and progress. When the customer finishes training and goes to work, the case manager records placement data and information on the credential earned and closes the case (hard exit). Subsequent employment and earnings are tracked automatically through UI wage record match. But what if the one-stop doesn’t follow-up with the customer while he was in training? After 90 days with no record of receipt of services, the customer’s case would be closed automatically (soft exit) and, again, subsequent employment and earnings would be tracked. Even if the customer worked part time while he was in school, the effect on the local area’s performance would be negative since postparticipation earnings would be low and the area would not receive credit for the credential when it was earned. The bottom line is that partners need to communicate about their shared customers.

Yes, WIA registrants continue to be considered active WIA participants as long as they are receiving any partner services. However, care must be taken to maintain current records on service participation through case management or tracking systems. (Mandatory and optional partner programs are listed in Appendix A.) 7. • • When are the outcomes of WIA customers counted? Outcomes are counted and reported quarterly. For all measures except the skill attainment rate for younger youth (age 14 to 18) and the employer customer satisfaction measure, measures are based on customers exiting in the quarter being reported. Customers who exit, return, and exit again are counted once in each quarter in which they exit.

USDOL has distinguished between “hard” and “soft” exits. Hard exits occur upon case closure or completion. Soft exits occur when a registrant has not received any WIA or WIA partner funded service for 90 days and is not scheduled to receive any future services. Because soft exits occur automatically, it will be important for local areas to maintain current data on customer services, including partner services.

A customer’s exit date is the last date of WIA funded or partner service. It determines his or her exit quarter. (Notice that soft exits will consistently generate exits in the previous quarter.) Planned gaps in service of more than 90 days due to a delay in the beginning of training or a medical condition that prevents an individual from participating will not trigger a soft exit.

7

February 2007

• 8.

The skill attainment rate is not an exit-based measure, so skill attainments are counted as they occur. WIA mandates the use of Unemployment Insurance wage data. How will that work?

The following WIA measures use wage record data: • • • • • Entered employment rate Employment retention rate Earnings change Earnings replacement rate Employment and credential rate

Nationally, over 90 percent of all workers are covered under Unemployment Insurance (UI). Each quarter, their employers must report the total wages paid to each employee, by Social Security number. The Georgia Department of Labor administers the UI program for our state and will routinely match these wage reports to WIA customer records at the state level. For the WIA performance measures, a customer is considered “employed” in a given quarter if he or she has any reported earnings for that quarter, regardless of the amount earned. However, because extremely high values can be indicators of problems such as duplicate data entry, typographical errors, or use of the same Social Security by more than one person, records with unusually high earnings levels will need to be examined and may be deleted. The WIA measures use five quarters of wage record data: the 2nd and 3rd quarters prior to the quarter of registration (or, in the case of dislocated workers, quarter of dislocation), and the 1st, 2nd, and 3rd quarters following the quarter of exit. There can be up to a three quarter delay between the quarter in which wages are earned and the quarter in which data on those earnings are available. For example, earnings for the 1st post exit quarter for WIA registrants who exit during July through September 2000 may not be available until the July to September 2001 quarter. Complete performance data will not be available for this group until the January through March 2002 quarter. Clearly, UI wage record based measures are not useful for day-to-day program management, and local areas will need to look at other indicators of performance. 9. What about employment that is not covered by Georgia’s Unemployment Insurance program, such as federal employment, self-employment, or employment in another state?

Because, in some states, a significant percentage of the workforce is not covered by Unemployment Insurance, USDOL will permit − but not require − the use of supplemental data sources for reporting employment. USDOL has issued guidelines regarding use of supplemental data (see Appendix B, USDOL Instructions on Reporting Supplemental Data). These include the following:

8

February 2007

Supplemental data may be obtained either by contact with participants or their employers or by record sharing or automated record matches with other employment-related data bases. Such data must be recorded within 30 days after it becomes clear that there is no wage record for an individual. Supplemental data may be used in calculating entered employment, employment retention, and employment and credential rates, but not in calculating earnings change or earnings replacement rate. All data and methods used to supplement wage data must be documented and are subject to audit.

• •

Before deciding whether to collect supplemental data, local areas may wish to consider the following: • • • 10. Georgia’s UI coverage rate is close to the national rate. A federal clearinghouse designed to facilitate interstate wage matching is being developed, and Georgia is participating as a pilot state. Local areas are not required to take non-covered employment into consideration in setting proposed levels of performance. How are the WIA performance measures calculated?

In general, calculation of the WIA performance measures is complex, with subtle variations in the same or similar measures across the five customer groups − adults, older youth, younger youth, dislocated workers, and employers. Operational definitions for the performance measures are contained in Table 3. In looking at this table, pay particular attention to the “Included” column. It’s tricky. The State is developing an automated system that will track performance and calculate the performance measures.

9

February 2007 Table 3 Calculating the WIA Performance Measures Performance Measure Entered Employment Rate Group Adults (18 & Older) Dislocated Workers Older Youth (19 – 21) Included Those who were not employed at registration1 All Those who were not employed at registration1 and who are not enrolled in post-secondary education or advanced training in the 1st quarter after exit Those employed in the 1st quarter after exit Those employed in the 1st quarter after exit and who are not enrolled in post-secondary education or advanced training in the 3rd quarter after exit Those employed in the 1st quarter after exit Numerator Number employed in the 1st quarter after exit Number employed in the 1st quarter after exit Number employed in the 1st quarter after exit Denominator Number who exit during the quarter Number who exit during the quarter Number who exit during the quarter

Employment Retention Rate

Adults (18 & Older), Dislocated Workers Older Youth (19 – 21)

Number employed in the 3rd quarter after exit Number employed in the 3rd quarter after exit

Number who exit during the quarter

Number who exit during the quarter

Earnings Change

Adults (18 & Older)

Older Youth (19 – 21)

Total UI-reported earnings in the 2nd and 3rd quarters after exit minus total UI-reported earnings in the 2nd and 3rd quarters before registration Total UI-reported earnings in the Those employed in the 1st quarter after exit and who are not enrolled 2nd and 3rd quarters after exit in post-secondary education or minus total UI-reported earnings advanced training in the 3rd quarter in the 2nd and 3rd quarters before after exit registration

Number who exit during the quarter

Number who exit during the quarter

10

February 2007 Table 3 Calculating the WIA Performance Measures Performance Measure Earnings Replacement Rate Employment and Credential Rate Credential Rate Group Dislocated Workers Adults (18 & Older), Dislocated Workers Older Youth (19 – 21) Included Those employed in the 1st quarter after exit Those who received training services Numerator Total UI-reported earnings in the 2nd and 3rd quarters after exit Number who were employed in the 1st quarter after exit and received a credential by the end of the 3rd quarter after exit3 Number who were in employment, post-secondary education, or advanced training in the 1st quarter after exit and who received a credential by the end of the 3rd quarter after exit3 Total number of attained basic skills plus number of attained work readiness skills plus attained occupational skills during the reporting period Number who attained a secondary school diploma or equivalent by the end of the 1st quarter after exit Number of youth found in one of the following categories in the 3rd quarter after exit: Post secondary education Advanced training Employment Military service Qualified apprenticeships Denominator Total UI-reported earnings in the 2nd and 3rd quarters before dislocation2 Number who exit during the quarter

All

Number who exit during the quarter

Skill Attainment Rate 4

Younger Youth (14 – 18)

Diploma or Equivalent Attainment Rate Retention Rate

Younger Youth (Age 14 – 18)

All in-school youth and all out-ofschool youth assessed to be in need of basic skills, work readiness skills, or occupational skills Those who register without a secondary school diploma or equivalent and who are not in secondary school at exit Those not in secondary school at exit

Total number of basic skill goals plus number of work readiness goals plus number of occupational skills goals set targeted for completion in the reporting period Number who exit during the quarter

Younger Youth (Age 14 – 18)

Number who exit during quarter

11

February 2007 Table 3 Calculating the WIA Performance Measures Performance Measure Participant Customer Satisfaction Employer Customer Satisfaction Group WIA-registered customers One-stop employer customers Included Participants who have exited from WIA and who are selected for inclusion in the survey sample. Employers who have received a completed service involving personal contact with one-stop staff and who are selected for inclusion in the survey sample. Denominator Numerator Index calculated based on responses to three questions addressing satisfaction, expectations, and service ideal. Index calculated based on responses to three questions addressing satisfaction, expectations, and service ideal.

Notes: 1 Employment status at registration is based on information provided by the registrant. nd rd 2 If there is no date of dislocation, or if the date of dislocation occurs after registration, the 2 and 3 quarters prior to registration are used. 3 Credentials may be obtained while the participant is enrolled in WIA. 4 For additional information on skill goals and skill attainment, see page 13.

12

February 2007

11.

What constitutes a “credential,” for purposes of the credential-based measures?

USDOL has defined “credential” broadly, to include both nationally and state or locally recognized credentials. (See Appendix C, Definitions.) In addition to secondary diplomas, GEDs, and post-secondary diplomas, the definition includes all credentials recognized by state education agencies. Licensure and industry-recognized certificates are also included in the definition. Please note: With the exception of the diploma or equivalent rate for younger youth (age 14 to 18), none of the credential-based measures give credit solely for obtaining a credential. Adults and dislocated workers must also be employed in the 1st quarter after exit, and older youth (age 19 to 21) served under the youth program must also be employed, in post-secondary school, or in advanced training in the 1st quarter after exit. All of the credential-based measures incorporate some lag time between exit and when the credential must be obtained. In the case of younger youth (age 14 to 18) and the diploma or equivalent rate, the credential must be obtained by the end of the 1st quarter after exit. In all of the remaining credential-based measures, the credential must be obtained by the 3rd quarter after exit. In fact, however, credentials will often be earned prior to exit. Data on receipt of credentials will need to be obtained and documented at the local level. 12. The skills attainment measure is very different from the other measures, since it compares goals attained to goals set, rather than specific outcomes to number of individuals exiting from WIA. How will this measure work?

USDOL has established the following rules for measuring skill attainment rate: • • If a customer is deficient in basic literacy skills, he or she must set at least one basic skills goal. In addition, he or she may also set goals for work readiness or occupational skills. Goals must be set for all younger youth (age 14 – 18) who are either in school or out-ofschool and assessed to be in need of basic skills, work readiness skills, or occupational skills. All youth included in this measure must have at least one, and not more than three, skill goals in a year. These goals may involve any combination of the three types of goals. For example, a customer could have two basic skills goals plus one occupational skill goal. Pre-assessment and post-assessment of skill goals are required. Use of standardized test procedures, such as standardized tests or performance-based assessments with standardized scoring methods are encouraged. Where such tests or assessments are not available, assessment techniques must be objective, unbiased and conform to widely accepted, clearly defined criteria, be field tested for utility, consistency, and accuracy, and provide instructions for valid test administration.

• •

13

February 2007

All data and methods used to determine achievement of skill attainment goals must be documented and are subject to audit. Target dates for accomplishing goals may not be more than a year from the date on which they are set, but they may be extended to accommodate periods of time in an inactive or “hold” status. Goal attainment should be recorded in the quarter in which it occurs. Failure to meet a goal should not be recorded until the goal attainment period has expired.

Tests Recommended by USDOL Test of Adult Basic Education (TABE) DOL Workplace Literacy Test (DOL-WLT) Adult Measure of Educational Skills (AMES) Adult Basic Learning Exam (ABLE) Adult Literacy Test (ALT), Simon & Schuster Armed Forces Qualifying Test (AFQT) Basic Occupational Literacy Test (BOLT) California Achievement Test (CAT) Career Ability Placement Survey (CAPS) Comprehensive Adult Student Assessment System (CASAS) General Aptitude Battery (GATB) Iowa Test of Basic Skills (ITBS) Metropolitan Achievement Test (MAT) Reading Job Corps Screening Test (FJCST) Wide Range Achievement Test (WRAT)

Definitions of basic literacy skills deficient, basic skills goal, occupational skills goal, and work readiness skills goal have been established. (See Appendix C, Definitions.) 13. What about the two customer satisfaction measures? How will customer satisfaction be measured?

USDOL requires that we use the copyrighted American Customer Satisfaction Index (ACSI) and scoring methodology to measure the satisfaction of participant and employer customers. This index, which is based on telephone survey data, is widely used in both industry and government. Detailed information on the ACSI and the required data collection process is contained in Appendix D, Customer Satisfaction Measures. 14. Where can I learn more about the WIA performance measures?

The most current and comprehensive federal information on the WIA performance measures is contained in two issuances: USDOL Training and Employment Guidance Letters No. 7-99 and 8-99. These TEGLs and other timely information can be downloaded from www.usworkforce.org and www.icesa.org.

14

February 2007

Negotiated Levels of Performance The levels of performance to which the State and each local area will be accountable will be reached through negotiation − between USDOL and the State and between the State and each local area. Under Section 136(c) of WIA, local levels of performance on each of the performance measures are to be "based on the State adjusted levels of performance,” taking into account “the specific economic, demographic, and other characteristics of the populations to be served in the local area.” For those of us who are accustomed to the JTPA performance standards, in which required levels of performance are generated using regression equations that take into consideration characteristics of the population served and labor market factors, negotiated performance levels will take some getting used to. While the JTPA system meant that local areas did not know until the end of the program year what their final standards were, the standards were “self-correcting.” That is, if the population served turned out to be substantially different from the expected population, required levels of performance changed automatically. WIA’s negotiated performance level approach does not offer that kind of flexibility. However, as we gain experience under WIA, we will have better information on which to base performance levels. 1. Have State adjusted levels of performance been set?

Georgia submitted baseline data and proposed levels of performance to the U. S. Department of Labor in its State Plan. Negotiations have taken place, but our plan has not been approved yet. In addition, because we will not have any information on local program design until initial local plans are submitted, we have advised USDOL that our proposed levels of performance may need to be revised. However, the proposed State levels are listed in Table 4 below. 2. When will negotiation of local performance levels take place?

Although local areas are asked to include planned levels of performance for Year 1 in their initial plans, final negotiation of performance levels will occur when the comprehensive plans are submitted in the fall. Year 1 performance will not be very meaningful in terms of service under the WIA system, since it will be based on the outcomes of JTPA terminees from October 1999 through June 2000 plus WIA registrants meeting the criteria for inclusion in the performance measures during July through September 2000. However, performance in terms of the WIA measures during this transitional period will help all of us become familiar with those measures.

15

February 2007

Table 4 Statewide Performance Levels (Pending USDOL Approval)* Performance Measure Year 1 Year 2 Year 3 Adults Entered Employment Rate Employment Retention Rate Earnings Change Employment & Credential Rate Dislocated Workers Entered Employment Rate Employment Retention Rate Earnings Replacement Rate Employment & Credential Rate Older Youth, 19 - 21 Entered Employment Rate Employment Retention Rate Earnings Change Credential Rate Younger Youth, 14 – 18 Skill Attainment Rate Diploma or Equivalent Attainment Rate Retention Rate

67% 80% $3,425 60%

67.5% 80% $3,435 60%

68% 80% $3,445 60%

74% 89% 94% 60%

74% 89% 94% 60%

74% 89% 94% 60%

68% 81% $2,750 50%

70% 81% $2,750 50%

72% 81% $2,800 50%

72% 55% 58%

72% 55% 60%

72% 55% 62%

Customer Satisfaction Participant 66% 67% 68% Employer 70% 71.5% 73% * Because of the negotiation process, the levels shown are different from the levels that appear in the State Plan and the Local Planning Instructions.

3.

Will local areas be expected to meet the statewide levels of performance?

Ultimately local performance will need to “roll up” into the statewide levels. However, we know that conditions vary substantially across our state. Therefore, local performance levels should reflect local circumstances. 4. What will the State take into consideration in negotiating local performance levels?

Our baseline data have made it clear that the outcomes of those who receive training are likely to be substantially different from the outcomes of customers receiving intensive services only, so service mix for the local area will need to be considered. (See Appendix E, Baseline Performance Data.) In addition, local labor market conditions and characteristics of the population served have historically been related to program outcomes. In the coming months, we will add labor market data to our existing baseline data so that we can study the relationships between the WIA performance measures, customer characteristics, and labor market conditions.

16

February 2007

Appendix A Mandatory and Optional WIA Partner Programs

Mandatory Partners • • • • • • • • • • • • • • Employment Service Adult Education Postsecondary Vocational Education Vocational Rehabilitation Welfare-to-Work Title V of the Older Americans Act Trade Adjustment Assistance NAFTA Transitional Adjustment Assistance Veterans Employment and Training Programs Community Services Block Grants HUD Employment and Training Activities Unemployment Insurance Job Corps U. S. Department of Labor: Indian and Native American Programs; Migrant and Seasonal Farmworker Programs; Youth Opportunity Grants; demonstration, pilot, multiservice, research, and multistate projects; and national emergency grants

Optional Partners • • • • • • School-to-Work Bureau of Apprenticeship and Training Department of Transportation Employment Support Programs TANF USDA Food Stamp Employment and Training Programs Other partners designated by local areas

i

February 2007

Appendix B USDOL Instructions on Reporting Supplemental Data [Proposed reporting instructions issued in the Federal Register on April 3, 2000, included the following language regarding reporting of supplemental data on employment:] While the majority of employment in a state’s workforce will be “covered” in the UI wage records, certain types of employers and employees are excluded by Federal standards or are not covered under a state’s UI law. “Uncovered” employment typically includes Federal employment, postal service, military, railroad, self employment, some agricultural employment, and employment where earnings are primarily based on commission. States have flexibility in methods used to obtain information on participants in “uncovered” employment. Examples include: 1) Case management, follow-up services, and surveys of participants to determine that the participant[s] are employed and written documentation of that employment; or Record sharing and/or automated record matching with other employment and administrative data bases to determine and document employment. These databases include, but are not limited to: • • • • • • Office of Personnel Management (Federal Career Service); United States Postal Service; Railroad Retirement System; State Department of Revenue or Tax (State income tax for self-reported occupations); U. S. Department of Defense; and Government Employment Records (State government, local government, judicial employment, public school employment, etc.).

2)

All data and methods to supplement wage record data must be documented and are subject to audit. Computer records from automated record matching are considered to be a valid written record. A telephone response from the participant must be accompanied by a written document such as a W2 form, pay stub, 1099 form, or other written documentation. Telephone verification of employment with employers is acceptable, but must also be documented. For self-employed individuals, telephone verification with major clients/contracting entities is also acceptable, but must be documented. States must wait two full quarters after the quarter of wage activity before making the decision to use supplemental sources of data. Supplemental data must be recorded within 30 days after the individual was found missing in the wage record.

ii

February 2007

Appendix C Definitions The definitions that follow were issued by USDOL in Training and Employment Guidance Letter No. 7-99. Advanced training – an occupational skills employment/training program, not funded under Title I of WIA, which does not duplicate training received under Title I. Includes only training outside of the One-Stop, WIA and partner, system (i.e. training following exit). Basic literacy skills deficient - the individual computes or solves problems, reads, writes, or speaks English at or below the 8th grade level or is unable to compute or solve problems, read, write, or speak English at a level necessary to function on the job, in the individual’s family, or in society. 1 Basic skills goal - measurable increase in basic educational skills including reading comprehension, math computation, writing, speaking, listening, problem solving, reasoning, and the capacity to use these skills. Credential – nationally recognized degree or certificate or State/locally recognized credential. Credentials include, but are not limited to, a high school diploma, GED or other recognized equivalents, post-secondary degrees/certificates, recognized skill standards, and licensure or industry-recognized certificates. States should include all State Education Agency recognized credentials. 2 Date of dislocation – The last day of employment at the dislocation job. If there is no date of dislocation, date of registration will be used instead. Employed at registration – An individual employed at registration is one who is currently working as a paid employee or who works on his or her own business, profession or farm, or works 15 hours or more as an unpaid worker in an enterprise operated by a member of the family, or is one who was not working, but has a job or business from which he or she was temporarily absent because of illness, bad weather, vacation, labor-management dispute, or personal reasons, whether or not paid by the employer for time-off, and whether or not seeking another job. (definition was adopted from TEGL 14-00, change 1, Attachment E) Employed in quarter after exit quarter – The individual is considered employed if UI wage records for the quarter after exit show earnings greater than zero. UI Wage records will be the primary data source for tracking employment in the quarter after exit. When supplemental data sources are used, individuals should be counted as employed if, in the calendar quarter after exit,
1

States and localities may establish their own definitions of “basic literacy skills deficient.” However, their definitions must include the above language. 2 The USDOL definition includes the following guidance: “In addition, States should work with local Workforce Investment Boards to encourage certificates to recognize successful completion of the training services listed above that are designed to equip individuals to enter or re-enter employment, retain employment, or advance into better employment.” [See definition of “training services.”]

8/4/2000

xix

February 2007

they did any work at all as paid employees (i.e. received at least some earnings), worked in their own business, profession or worked on their own farm. Exit date – the last date of WIA funded or partner service received (except follow-up services). There are two ways to determine exit date: • a participant who has a date of case closure, completion or known exit from WIA-funded or non-WIA funded partner service within the quarter (hard exit); or • a participant who does not receive any WIA-funded or non-WIA funded partner service for 90 days and is not scheduled for future services except follow-up services (soft exit). 3 Participants who have a planned gap in service of greater than 90 days should not be considered as exited if the gap in service is due to a delay before the beginning of training or a health/medical condition that prevents an individual from participating in services. Exit quarter – the quarter in which the last date of service (except follow-up services) takes place. High school diploma equivalent – a GED or high school equivalency diploma recognized by the State. Military service – reporting for active duty. Not employed at registration – An individual who does not meet the definition of employed at registration. Occupational skills goal - primary occupational skills encompass the proficiency to perform actual tasks and technical functions required by certain occupational fields at entry, intermediate, or advanced levels. Secondary occupational skills entail familiarity with and use of set-up procedures, safety measures, work-related terminology, record keeping and paperwork formats, tools, equipment and materials, and breakdown and clean-up routines. Post-secondary education – a program at an accredited degree-granting institution that leads to an academic degree (e.g., AA, AS, BA, BS). Does not include programs offered by degree-granting institutions that do not lead to an academic degree. Qualified apprenticeship – a program appoved and recorded by the ETA/Bureau of Apprenticeship and Training (BAT) or by a recognized State Apprenticeship Agency (State Apprenticeship Council). Approval is by certified registration or other appropriate credential. Training services – include WIA-funded and non-WIA funded partner training services. These services include: • occupational skills training, including training for nontraditional employment • on-the-job training • programs that combine workplace training with related instruction, which may include cooperative education programs • training programs operated by the private sector • skill upgrading and retraining
A soft exit cannot be determined until 90 days has elapsed from the last date of service, but the exit date recorded is the last date of service.
8/4/2000
xx
3

February 2007

• • • •

entrepreneurial training job readiness training adult education and literacy activities in combination with other training customized training conducted with a commitment by an employer or group of employers to employ an individual upon successful completion of the training

Work readiness skill goal – work readiness skills include world of work awareness, labor market knowledge, occupational information, values clarification and personal understanding, career planning and decision making, and job search techniques (resumes, interviews, applications, and follow-up letters). They also encompass survival/daily living skills such as using the phone, telling time, shopping, renting an apartment, opening a bank account, and using public transportation. They also include positive work habits, attitudes, and behaviors such as punctuality, regular attendance, presenting a neat appearance, getting along and working well with others, exhibiting good conduct, following instructions and completing tasks, accepting constructive criticism from supervisors and co-workers, showing initiative and reliability, and assuming the responsibilities involved in maintaining a job. This category also entails developing motivation and adaptability, obtaining effective coping and problem-solving skills, and acquiring an improved self image.

8/4/2000

xxi

February 2007

Appendix D Customer Satisfaction Measures Customer satisfaction measures have been established for two groups of customers: WIA participants and employers receiving services from one-stop centers. USDOL has chosen to use the copyrighted American Customer Satisfaction Index (ACSI) and scoring methodology. It has purchased a license to use the ACSI for 500 completed participant and 500 completed employer surveys per state, per year. The State will be responsible for conducting these surveys. Because this number of surveys is not sufficient for valid measurement of customer satisfaction at the local level, additional surveys will need to be conducted. We will work with local areas to address issues such as design of survey instruments, data collection methodology, funding, and State and local responsibilities. 1. Who will be surveyed?

Random samples will be drawn from two customer groups: • • Title I-B participants who have exited from WIA Employers who have received a substantial service involving personal contact with OneStop staff

The “substantial service” criterion excludes employers who request a brochure or standard mailing, ask a question that is answered with little expenditure of staff time, or use electronic selfservices. Examples of substantial services include staff-facilitated job orders, customized labor market information requests, and on-the-job or customized training activities. 2. How many surveys must be completed?

Each state must complete 500 participant surveys and 500 employer surveys a year. Completed surveys must be generated from data collection processes achieving response rates of 50 percent or higher. A completed survey is defined as a survey in which all three questions regarding customer satisfaction have been answered. USDOL has been silent on the issue of the number of surveys that must be completed within each local area, but, clearly, 500 surveys statewide will not provide data that can be used to calculate valid measures of local customer satisfaction. We will consider the issue of sample size after local plans have been received and we have some sense of the number of people local areas expect to serve. 3. How will customer satisfaction data be collected?

The responses are to be obtained using a uniform telephone methodology. According to USDOL, estimates of the cost of telephone surveys nationwide run an average of $15 per completed survey. Although USDOL has not yet issued guidance on cost allocation, it is our understanding that the

8/4/2000

xxii

February 2007

cost of collecting customer satisfaction data will be a program cost, rather than an administrative cost. 4. When will surveys be conducted?

The surveys will be conducted on a rolling basis within the following time frames: Participants will be contacted within 60 days of the exit date or the date that an exit date has been determined. This means either 60 days after the date of an exit interview or 60 days after the 90 days have elapsed since the last service date. Employers will be contacted within 60 days of the completion of the service or 30 to 60 days after a job order has been listed where no referrals have been made. 5. What types of questions will be asked?

Following an introduction used to determine whether the interviewer is speaking to the right person and establish the appropriate frame of reference, respondents will be asked to rate the service they received on three dimensions: • • • Their overall satisfaction with the service received The extent to which the service met his or her expectations How close the service was to the respondent’s ideal for such services

Ratings will be made on a scale from 1 to 10. 6. How will customer satisfaction scores be calculated?

A customer’s ACSI score is the weighted average of his or her responses to the three questions. This score is then transformed to an index reported on a 0-100 scale. An overall State or local customer satisfaction score is simply the average of the individual index scores, rounded to the nearest whole number.

8/4/2000

xxiii

February 2007

Appendix E Baseline Performance Data 1. Where did the baseline data come from?

Although USDOL mandated the use of JTPA data as a baseline for WIA performance, Georgia also submitted data for Employment Service customers who received the equivalent of WIA intensive services from a field service office (FSO) during a six-month period. In general, four types of customers received such services: UI Claimants receiving services under the Claimant Assistance Program Veterans served by veterans’ programs TANF recipients and applicants served under the TANF contract Other individuals who were disadvantaged in the workplace in some way, including people with disabilities, the economically disadvantaged, and those lacking a high school diploma UI claimants were excluded from these baseline data because they are not typical of the population likely to be served under WIA and because their need for assistance will continue to be met through the Claimant Assistance Program. The outcomes of the remaining recipients of intensive services are seen in the entries for Adults and Older Youth listed under “Intensive Labor Exchange” in the table below. All of the dislocated workers included in the baseline data were JTPA Title III customers. However, performance for “Education and Training” is for those served by SDAs, while performance for “Intensive Labor Exchange” is for those receiving basic readjustment services from FSOs. Baseline data from both JTPA and the Employment Service were matched against UI data so that performance could be calculated in terms of the WIA performance measures. In calculating the combined baseline figures, we assumed a mix between intensive services only and training of 50/50 for adults and older youth (age 19 – 21). This mix was chosen simply because we had no basis on which to make any assumptions about what the actual mix would be. The mix for dislocated workers was set at 80% intensive services to only 20% education and training, based on the historic ratio of participation between basic readjustment services and training. These assumptions will be reconsidered after we receive local plans.

8/4/2000

xxiv

February 2007

U. S. Department of Labor Projected Performance and Georgia Baseline Data

Performance Indicator Adults Entered Employment Rate Employment Retention Rate Earnings Change in Six Months Employment and Credential Rate Dislocated Workers3 Entered Employment Rate Employment Retention Rate Earnings Replacement Rate Employment and Credential Rate Older Youth (Age 19 – 21) Entered Employment Rate Employment Retention Rate Earnings Change in Six Months Credential Rate Younger Youth (Age 14 – 18) Skill Attainment Rate Diploma or Equivalent Attainment Rate Retention Rate
1

USDOL Projected Average

Georgia Baseline Data Intensive Education Labor And Exchange2 Combined Training1

71% 78% $3,700 60%

71% 81% $3,506 ***

56% 80% $2,965 ***

64% 80% $3,235 60%

77% 85% 92% 60%

76% 91% 120% ***

71% 89% 88% ***

72% 89% 94% 60%

63% 77% $3,150 50%

75% 81% $3,037 ***

64% 81% $1,917 ***

70% 81% $2,477 50%

72% 55% 54%

*** *** 60%

N/A N/A ***

72% 55% 60%

Based on customers exiting from JTPA during the period from October 1997 through September 1998. 2 Based on job seekers receiving services similar to WIA intensive services and exiting from Georgia’s field service offices during the period from September 1997 through March 1998. 3 Based entirely on JTPA participants served by SDAs (Education and Training) and FSOs (Intensive Labor Exchange) *** Cannot be calculated using available data.

8/4/2000

xxv

February 2007

2.

The USDOL projected average levels for earnings gains seem very high. Can we really expect to see average increased earnings of over $3000?

Performance on the earnings measures depends very substantially on prior earnings. When we looked at the outcomes of all FSO customers during the baseline period, the average change was very low – less than $300. However, among those receiving intensive services, there was a clear split between UI claimants and all other customers. Earnings remained essentially the same for the claimants, but increased substantially (the $2,965 seen in the table, above) for other recipients of intensive service. That is, the UI claimants had solid earnings before losing their jobs, and when they returned to work, they earned about what they had earned before. The other Employment Service customers provided with intensive services had very low earnings in the months prior to receiving services. With help, they improved their employment situation and their earnings increased substantially. We see the same thing in the JTPA baseline data. JTPA serves individuals who are economically disadvantaged. That is, they are people with very low earnings in the period prior to enrollment. Following participation, their employment situation improves, and their earnings also increase substantially. The USDOL projected average levels for the two earnings gain measures have the effect of targeting WIA services on individuals who have been unemployed, or underemployed, for an extended period prior to their registration for WIA. 3. It appears that customers who received education and training fared better than those receiving intensive services.

Yes. Entered employment rates were consistently higher for those who received training than for those receiving intensive services only, although employment retention rates were essentially the same for the two groups. Similarly, performance on the earnings change measure for adults and older youth (age 19 – 21) and the earnings replacement rate for dislocated workers was higher among those who received education and training than among those receiving intensive services only. It is not surprising that outcomes are generally better for recipients of training. After all, more time and resources were invested in these customers, and the skills they learned enhanced their competitiveness in the job market. However, not everyone registered under WIA will be either willing or able to enter training. In many cases, intensive services will be the services most responsive to our customers’ needs.

8/4/2000

xxvi

February 2007

Appendix F WIA Performance Measure Baseline Data by State Service Delivery Regions Background. Although the WIA performance measures are different from the measures used in assessing the performance of programs funded under JTPA and Wagner-Peyser, the WIA measures that are based on Unemployment Insurance (UI) wage data can be calculated for people served under those programs. The U. S. Department of Labor instructed states to do just that, using JTPA terminees for the period from October 1997 through September 1998, in order to set baseline performance levels for WIA. Georgia chose to add individuals who received services similar to WIA intensive services from the state’s Career Centers to its baseline data so that it would be more representative of the scope of services to be provided under WIA. The Career Center customers were active during the period from October 1997 through March 1998. Findings from the baseline data are presented in the tables and charts that follow. They are organized by the twelve state service delivery regions. To a large extent, these regions coincide with local WIA areas. Where they do not, small WIA areas have generally been carved out of larger regions for administrative purposes. In most small areas, we do not have enough data to provide a valid basis for planning and negotiation. However, the regions reflect the labor markets within which these local areas will operate. In addition, labor market data that may be important in understanding variation between areas − and useful in negotiating local levels of performance − are readily available by region. Local areas may also provide additional, more localized data to inform the negotiation process. In the tables and charts, comparisons are made between “training” and “intensive services only.” Data on training for Adults and Older Youth are based on JTPA Title II terminees. Data on “intensive services only” for the same groups are based on Career Center customers, excluding UI claimants, who received services similar to the intensive services authorized under WIA. Claimants were excluded because they will continue to receive such services through the Claimant Assistance Program. The majority of the Career Center customers in the baseline data were economically, educationally, or physically disadvantaged and therefore were similar to those served under Title II of JTPA. Performance for Adults and Older Youth are shown both separately and combined because outcomes for the two groups were very similar and because the number of cases available for Older Youth is very small. Data on Dislocated Workers are based entirely on terminees from Title III of JTPA. Those who received “training” were served by SDAs, while those shown under “intensive services only” received basic readjustment services from a Career Center. Factors Influencing Performance. In general, the factors that are likely to have an effect on performance include labor market conditions, the background and characteristics of the people served, the types of services provided, and the quality of those services. Ideally, negotiated performance measures should take the first three types of factors into consideration and let quality shine through. However, unlike the JTPA performance standard adjustment process, which finetuned performance expectations to actual labor market conditions and participants served, negotiation of the WIA performance measures takes place during the planning process, and the

8/4/2000

xxvii

February 2007

resulting levels remain fixed. Therefore, emphasis should be placed on factors that reflect enduring conditions or fundamental decisions regarding program focus. We are currently doing analyses of the relationships between Georgia’s baseline performance and factors that we believe may influence performance. Examples include regional factors such as unemployment rates, average annual wages, or indicators of economic growth or decline, and registrant characteristics such as educational attainment or economic disadvantage. We will share what we learn in a future package on negotiating local performance measures. However, the baseline data may help you become familiar with the wage-based measures mandated under WIA. They may also help you think about the implications of program design choices for performance. Baseline Performance The wage-based performance measures, which are calculated separately for each of the groups listed, include the following: • • • • Entered employment rate (Adults, Older Youth, and Dislocated Workers) Employment retention rate (Adults, Older Youth, and Dislocated Workers) Earnings change (Adults and Older Youth) Earnings replacement rate (Dislocated Workers)

The operational definitions of these measures are technically complex, especially with regard to who is included in each calculation. However, all of them are based on either the presence or amount of wages reported to the Unemployment Insurance program during a particular quarter or set of quarters. (See Table 3, Calculating the WIA Performance Measures, on page 9 of this Guide.) In looking at the baseline data, it is useful to think about variation, both across regions and between services, and whether it appears to be “systematic” or “random.” Systematic variation is that part of the differences between regions that we can account for. For example, one might expect regions with higher unemployment rates to have lower entered employment rates. Random variation is that part of the differences between regions that we cannot explain. It is there, but we don’t know why, and it should not be the basis for differences in performance expectations. In general, we do not see strong evidence of large, systematic geographic differences. However, we will explore this issue further by examining relationships between labor market conditions and performance, and we expect that an understanding of geographic differences will be useful in finetuning performance expectations. On the other hand, outcomes were generally better for individuals who received training than for those who received intensive services only. Therefore, we would expect to see higher levels of performance from areas that provide training to more of their WIA registrants. However, for many customers, intensive services may be the best, and most cost effective, form of help a local area can offer. In order to avoid creating a disincentive to providing intensive services to those who need them, service mix will need to be taken into consideration in the negotiation process.
8/4/2000
xxviii

February 2007

Among the measures, employment retention rates showed the least variation, both across regions and between services − especially among Dislocated Workers. (One reason why there is little variation is that only those where were employed in the first quarter following exit are included in the measure, so it is unaffected by differences in entered employment rates.) As we learn more about the retention measures, we may discover that performance expectations for these measures should vary within a narrow range, if at all. We have the most to learn about the earnings measures. The earnings replacement rate for dislocated workers shows that training made a strong contribution to earnings. But while earnings replacement rates were very consistent across regions for those who received intensive services only, there was a great deal of variation across regions for those who received training. We know that the character of dislocated worker programs can change dramatically, depending on the types of workers affected by dislocation. Therefore, while the very high earnings replacement rates seen in some areas are cause for celebration, they are probably not very helpful when it comes to setting realistic performance expectations. The baseline earnings changes for Adults and Older Youth vary widely across regions, and, although earnings changes were generally higher for those who received training, the relationship between the type of service received and earnings change is not consistent across areas. We do know, however, that earnings changes tend to be greatest for those customers with the lowest preparticipation earnings, and that, therefore, this measure has the effect of targeting services on those most in need of assistance. While it will probably be important to take characteristics of the population being served into consideration in negotiating this measure, the negotiation process should not be used to shift services away from those who need them most. Finally, in thinking about the implications of wage-based WIA measures for local programs, it is important to remember that the state will also be held accountable for its performance. That performance can only come from the performance of local areas. Across all areas, negotiated performance levels will need to “roll up” to a reasonable expectation that the state will meet its negotiated levels of performance.

8/4/2000

xxix

February 2007

Appendix G Data for Use in Negotiating WIA UI Wage-Record Based Performance Measures Background Under WIA, proposed state and local performance levels are to be negotiated between federal and state partners and between state and local partners. The challenge is how to make the negotiation process valid and equitable, resulting in performance levels that reflect a realistic understanding of differences among states and between local areas within states. What follows does not provide a standard "formula" for coming up with planned performance. However, it does give you some background information and concepts to think about as you develop reasonable performance expectations. The national JTPA performance standard system addressed this issue by using a statistical modeling process that adjusted local standards based on a set of factors shown to have an effect on local performance. For example, if an area served a high percentage of participants who lacked a high school diploma, its performance standards were lowered because this population was harder to place and less likely to retain their jobs. Although this approach is not being used under WIA, it can help us understand the factors that effect performance so that we can consider them in the negotiation process. The information that follows is based on the outcomes of customers who either terminated from JTPA or exited from FSOs during the baseline period. The outcomes are expressed in terms of the WIA UI wage-record based measures. Training (SDAs) and “intensive services only” (FSOs) were examined separately. Statistical models could not be developed for outcomes for older youth because there were so few members of this group in the baseline data. However, findings regarding the adult measures suggest factors that could be considered in negotiating performance levels for older youth. The factors tend to be related to one another. Therefore, some factors that you might expect to be important in explaining outcomes were not because other, related, and more powerful factors did the job. For example, one of the customer characteristics tested was whether or not the customer was receiving TANF. Because factors for gender and prior earnings level accounted for the differences between TANF recipients and other customers, the TANF factor did not add anything to our understanding of outcomes. The same thing happened with unemployment rate, which was related to all of the other labor market factors tested. The outcomes associated with various levels of important factors are shown in the tables that follow. We hope that this information will help you think about the ways in which your area and your customers differ from those of other areas, and the implications of those differences for your expected performance. We expect that findings from the baseline data will inform the negotiation process but will not, by themselves, determine the resulting performance levels.

8/4/2000

xxx

February 2007

Factors Type of Services Provided. In Appendix F of this Guide, we saw that, for most measures, differences between training and intensive services within service delivery regions tended to be greater than differences between service delivery regions. Under JTPA, type of services provided was excluded from the performance standard adjustment process. However, JTPA was primarily a training program, serving individuals who were willing and able to participate in training. The exception to this focus on training was JTPA's ability to provide basic readjustment services under Title III. Georgia found that a substantial majority of the dislocated workers served under JTPA chose basic readjustment services only. When we compare the outcomes of the baseline dislocated workers who received training with those who received basic readjustment services using the WIA measures, we see that those who received training did somewhat better. This difference was especially true in the case of earnings replacement rate. That is, an investment of participant time and JTPA funds in training helped the trainees get better wages. That does not mean, however, that basic readjustment services (which are similar to WIA intensive services) were not helpful in facilitating the transition from dislocation to reemployment. It simply means inexpensive, shortterm services cannot be expected to generate the same results as longer-term, more costly services. We believe that both intensive services and training are needed, and the outcomes for both should be considered in setting state and local standards. Therefore, planned mix of services should be taken into consideration in negotiating performance levels. (See "Statewide Baseline Outcomes by Type of Service," below.)
Statewide Baseline Outcomes by Type of Service Intensive Services Only 73% 77% 76% 81% 81% 91% $3506 $3037 120% 56% 58% 71% 80% 79% 89% $2985 $2279 89%

Performance Measure Entered Employment Rate Adults Older Youth Dislocated Workers Employment Retention Rate Adults Older Youth Dislocated Workers Earnings Change Adults Older Youth Earnings Replacement Rate Dislocated Workers

Training

8/4/2000

xxxi

February 2007

While there is no prescribed methodology for predicting WIA performance, a good place to start in thinking about performance is the statewide baseline performance levels. Planned mix of services between training and intensive services can be considered by calculating a weighted average of the two. For example, suppose you expect that 40 percent of your adult WIA registrants will enroll in training, while the remaining 60 percent will receive intensive services only. Applying these percentages to the statewide baselines (multiplying the weights times the baseline rates and adding the results) would give you a projected adult entered employment rate of approximately 63%.

Service Weight Baseline Product Training 40% 73% 29.2% Intensive Services 60% 56% 33.6% Predicted Performance (Sum) 62.8%

Customer Characteristics. In general, the baseline data include two types of individuals: dislocated workers and people who had some disadvantage in the labor force. Those served under Title II of JTPA were economically disadvantaged or had other barriers to employment. When claimants and dislocated workers were excluded from those provided intensive services by FSOs, the data showed that the majority of these customers were also disadvantaged. The table below summarizes customer characteristics in the baseline period.
Statewide Baseline Customer Characteristics by Program Area and Type of Service Adults Characteristic Training Gender Male Female High School Graduate Yes No Person with Disability Yes No Earnings in the 3rd Quarter Before Entry: Less than $1000 $1000 to $2499 $2500 and Higher Earnings in the 3rd Quarter Before Entry: Less than $2500 $2500 to $4999 $5000 and Higher 30% 70% 75% 25% 5% 95% Intensive Services 40% 60% 76% 24% 3% 97% Dislocated Workers Intensive Training Services 37% 63% 91% 9% 2% 98% 36% 64% 87% 13% 1% 99%

70% 15% 15%

64% 12% 24%

37% 33% 30%

23% 38% 39%

8/4/2000

xxxii

February 2007

Among the customer characteristics examined for their relationship to WIA outcomes were gender, high school diploma or GED, disability, and receipt of TANF. In addition, level of UIreported wages in the third quarter before receipt of services was examined. The 3rd quarter prior to participation was chosen because earnings typically decline just before individuals seek employment assistance or training. Going back in time provides a better picture of the individual’s “normal” employment and earnings and helps distinguish between those with temporary versus enduring employment problems. In meaning, this factor is similar to the more familiar characteristics of economic disadvantage or poor work history. As you will see, lower prior earnings are associated with lower entered employment and retention rates but much higher earning gains and earnings replacement rates. In considering whether characteristics of your customers should be considered in setting your performance levels, the first question to ask is whether your customers are significantly different from those of other areas. (See table, "Statewide Baseline Customer Characteristics by Program Area and Type of Service, above.) For example, if you expect that about 25 percent of your adult customers will lack a high school diploma or GED, you are typical. There is no need to adjust your expected performance levels for that fact. However, if you expect that, based on coordination relationships and service goals in your area, 15% of your adult trainees will be persons with disabilities, you will probably want to take that fact into consideration. Again, calculating a weighted average will help you see the likely effect of this commitment on your performance.
Persons with Disabilities Weight Baseline Yes 15% 61% No 85% 72% Predicted Performance (Sum) Product 9.2% 61.2% 70.4%

Labor Market Factors. Because a variety of labor market factors had been tested over the years for use in the JTPA performance standard adjustment process, we attempted to look at the relationships between those factors and WIA outcomes. Some of those factors could not be calculated from available data, and substitutions were made. Factor values were linked to individual customer data based on county of residence. The following factors were tested: • • • • • • • • Annual unemployment rate (1997) Average weekly wage (1997) Average annual earnings in wholesale and retail trade (1997) Percentage employed in agriculture, manufacturing, or mining (1997) Percentage of population below poverty level (1995) Employee/resident worker ratio (1997) Percentage change in population (1990 to 1997) 1993 Beale Code Definitions of Metro/Nonmetro Status

8/4/2000

xxxiii

February 2007

To take the effect of a labor market factor into consideration, look at the difference between the baseline value and the value that applies to your area and apply that difference to your baseline, or modified baseline, figures. (The most current available data on the factors that proved to be important in explaining outcomes will be provided separately.) Levels of Performance Associated with Significant Factors The tables that follow show the levels of performance associated with the customer characteristics and labor market factors that had a significant effect on performance. Levels of the labor market factors have been selected based on their distribution in the baseline population. Different levels of earnings in the third quarter prior to receipt of services have been chosen for adults and dislocated workers because the range of earnings was substantially higher for the dislocated workers than for the customers included in the baseline data for adults.

In some cases, effects of the factors on performance are modest, and may not be worth considering. In other cases, they are dramatic. In particular, earnings in the 3rd quarter prior to participation helps explain the very substantial differences in earnings change and earnings replacement rate that we saw between service delivery regions in Appendix F. Regions that served customers with lower prior earnings had higher earnings gains and replacement rates. Decisions that you make regarding targeting of services based on work and earnings history will have a substantial effect on your performance on these measures.

8/4/2000

xxxiv

February 2007

WIA Outcomes by Customer Characteristics: Training Recipients Earnings Replacement Rate: Dislocated Workers

Characteristic

Entered Employment Rate Dislocated Adults Workers

Employment Retention Rate Dislocated Adults Workers

Earnings Change: Adults

Gender Male Female High School Graduate Yes No Disability Yes No Earnings in 3rd Quarter Before Entry: Less than $1000 $1000 - $2499 $2500 and Higher Earnings in 3rd Quarter Before Entry Less than $2500 $2500 to $4999 $5000 and Higher * Based on fewer than 50 individuals.

72% 78%

73% 84% 84% 71%

87% 93%

$4,028 $3,287 $3,698 $2,896

121% 123%

123% 107%

61% 72%

* 75% 91%

68% 77% 84%

76% 88% 90%

$4,638 $2,300 $454

86% 92% 95%

464% 110% 79%

8/4/2000

xxxv

February 2007

WIA Outcomes by Labor Market Factors: Training Recipients Earnings Replacement Rate: Dislocated Workers

Labor Market Factor

Entered Employment Rate Dislocated Adults Workers

Employment Retention Rate Dislocated Adults Workers

Earnings Change: Adults

Average Weekly Wage Less than $400 $400 to $499 $500 and Higher % of Population in Poverty Less than 13% 13% to 22% 23% and Higher Beale Code Metropolitan County Nonmetropolitan Cnty * Based on fewer than 50 individuals.

74% 74% 79% $4,027 $3,050 $3,515 75% 78%

8/4/2000

xxxvi

February 2007

WIA Outcomes by Customer Characteristics: Intensive Services Only Entered Employment Rate Characteristic Adults Dislocated Workers 69% 72% Adults Dislocated Workers Employment Retention Rate Earnings Change: Adults Earnings Replacement Rate: Dislocated Workers

Gender Male

$3,943 $2,363

Female High School Graduate Yes No Disability Yes No Earnings in 3rd Quarter Before Entry: Less than $1000 $1000 - $2499 $2500 and Higher Earnings in 3rd Quarter Before Entry Less than $2500 $2500 to $4999 $5000 and Higher

59% 47% 41% 56%

72% 70%

82% 71%

89% 85%

$3,345 $1,629

52% 63% 64%

75% 84% 87%

$5,053 $1,612 -$603

86% 89% 90%

157% 90% 77%

8/4/2000

xxxvii

February 2007

WIA Outcomes by Labor Market Factors: Recipients of Intensive Services Only Earnings Replacement Rate: Dislocated Workers

Labor Market Factor

Entered Employment Rate Dislocated Adults Workers

Employment Retention Rate Dislocated Adults Workers

Earnings Change: Adults

% Employed in Manufacturing, Agriculture, & Mining Less than 20% 20% to 39% 40% and Higher % of Population in Poverty Less than 13% 13% to 22% 23% and Higher Beale Code Metropolitan County Nonmetropolitan Cnty

74% 69% 67% $3,750 $2,760 $2,700 58% 52%

8/4/2000

xxxviii