NAEP Assessment Sample Design

A new sample design for every year that the National Assessment of Educational Progress (NAEP) is given. Because each year's design differs, this section of the website discusses each separately; however, some general facts can be said about all NAEP assessments. In each assessment year, a sample of students in designated grades within both public and private schools is selected for assessment. The geographic scope always covers the entire United States and—in some assessment years—U.S. territories and possessions. Since 1990, the samples of public schools and their students in each state have been large enough to provide state-level estimates. In all cases, the selection process has utilized a probability sample design in which every school and student has a non-zero chance of being selected, and standard errors can be calculated for the derived estimates. The selection of a sample of students for the assessment involves a complex multistage sampling design with the following stages: • • • selection of schools, both public and private, within the designated areas; selection of students in the relevant grades within the designated schools; and allocation of selected students to the subject areas being assessed by NAEP in the given year.

The Common Core of Data (CCD) file—a comprehensive list of operating public schools in each jurisdiction that is compiled each school year by the National Center for Education Statistics (NCES)—is used as the sampling frame for the selection of sample schools. The CCD also contains information about grade span and enrollment of each school, as well as its location. As part of the selection process, in years when relatively large samples are selected, public schools are combined into groups known as strata on the basis of various school characteristics related to achievement. These characteristics include the physical location of the school, extent of minority enrollment, state-based achievement scores, and median income of the area in which the school is located. Stratification of public schools occurs within each state. Combining schools within strata provides a more ordered selection process with improved reliability of the assessment results because each stratum group is adequately represented in the sample. Generally, a sample of approximately 100 grade-eligible public schools is selected within each jurisdiction (or about 7,000 to 10,000 all together); within each school, about 60 students are selected for assessment. Both of these numbers may vary somewhat, depending on the number and enrollment size of the schools in a jurisdiction, as well as the scope of the NAEP program in the particular assessment year. Students are sampled from a roster of individual student names, not by entire classrooms. The total number of schools is a function of the number of grades to be assessed, the number of subject areas to be assessed, and the number of states that choose to participate in the program. The national sample of private schools in all grades is drawn from a list compiled through the Private School Survey (PSS), which is a mail survey of all U.S. private schools carried out biennially by the Census Bureau under contract to NCES. In years in which relatively large samples are to be selected, private schools are classified by type (e.g., Roman Catholic, Lutheran), and are grouped for sampling by geography (Census division), degree of urbanization of location, and minority enrollment. About 700 private schools, on average, are included in a national sample, with up to 60 students per school selected for assessment. When the samples are to be representative only at the national level, the procedure for selecting both public and private schools typically involves a somewhat different process. Rather than selecting schools directly from lists of schools, the first stage sampling involves selecting a sample of some 50 to 100 geographic areas called primary sampling units (PSUs). A PSU is a geographic area comprising a consolidated metropolitan statistical area (CMSA), a metropolitan statistical area, a New England County Metropolitan Area (NECMA), a county, or a group of contiguous counties in the United States (including Alaska, Hawaii, and the District of Columbia.) The entire Untied States was divided into 1,027 PSUs in
1

2001. Each PSU met a minimum size requirement and was contained within one of four geographic regions. Within the set of selected PSUs, public and private school samples are selected using similar procedures to those described above for the direct sampling of schools from lists. The samples are clustered geographically which results in a more efficient data collection process. The selection of PSUs is not necessary when the sample sizes are large in each state.

2

Sample Design for the 2000 Assessment
The 2000 NAEP assessment sample consisted of public and nonpublic school students drawn throughout the United States and its jurisdictions. The sample was used to generate estimates that measured the nation's student population and various student groups of interest. The national main assessment utilized multistage probability sampling that split the United States into geographical primary sampling units (PSUs). The state assessment also used multistage probability sampling, although with fewer stages, within each participating state or jurisdiction.

3

National Main Assessment Sample Design in 2000
National samples were selected using multistage probability based sample designs for the 2000 national main assessment. Samples represented fourth-, eighth-, and twelfth-grade students in public and private schools in the 50 states and the District of Columbia. At grade 4, mathematics, reading, and science were assessed. At grades 8 and 12, only mathematics and science were assessed. The 2000 sample design oversampled specific student groups to enhance the reliability of NAEP estimates for the following three groups: • • • Black and Hispanic; students with disabilities and limited-English-proficient students; and private school.

The oversampling of these groups was achieved at either the school or student sampling stages and, in the case of Black and Hispanic students, at both stages. For all 2000 assessment samples other than the grade 8 public school sample: • • • first stage sampling units were individual counties or groups of counties called primary sampling units (PSUs); second stage units were elementary or secondary schools; the third stage of sampling was the assignment of sample types and session types to sampled schools. In NAEP assessments, the "session type" identifies assessment subject, while the "sample type" identifies whether accommodations were provided and the type of inclusion criteria applied for a given assessment; and the fourth and final stage was selection of students within schools and their assignment to session types (assessment subject area).

The eighth-grade public school sample was part of a pseudo-integrated design with the eighth-grade state sample, which allowed state and national samples to be combined after data collection to produce a larger sample that would yield somewhat more precise state estimates and substantially more precise national estimates. This design provided data necessary for NCES to explore the feasibility of conducting an integrated national and state sampling operation in assessments beginning in 2002. Should this design prove successful, future assessments would no longer require a separate national sample in years during which state assessments were being conducted. The basic grade 8 public school sample involved a stratified sample with state as the primary stratification. Except in six sparsely populated states, the national main eighth-grade public school sample used a three-stage probability sample design, modeled after the state sample: • • • the first stage of selection was the sampling of schools; the second stage of sampling was the assignment of session type and sample type to schools; and the third stage was student sampling.

For efficiency, the sample design in the six sparse states—Alaska, Idaho, Montana, North Dakota, South Dakota, and Wyoming—used geographic clustering (PSUs) at the first stage, similar to the grade 4 and 12 public school samples.

4

Target Population and Sample Sizes of the 2000 National Main Assessment
The target population for the 2000 national main assessment consisted of approximately 10 million students: 3.7 million fourth-graders, 3.5 million eighth-graders, and 2.8 million twelfth-graders enrolled in public and private elementary and secondary schools in the United States and the District of Columbia. This population included Bureau of Indian Affairs (BIA) and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS).
Target population of students in public schools, by grade, national main assessment: 2000 Grade Total 4 8 12 Number of public school students 9,965,000 3,686,400 3,497,000 2,781,600 Number of public schools 95,000 49,600 25,600 19,700

NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

The total sample size for the national main assessment was 106,500 students: • • • 45,500 at grade 4, 31,500 at grade 8, and 29,500 at grade 12.

The table below provides the student sample size goals by type of school, grade, and subject area.
Student sample size goals, by school type, grade, and subject, national main assessment: 2000 Student sample size goals Public school Private school sample sample Total 106,500 13,800 8,000 15,800 15,800 15,800 13,800 15,800 63,000 7,000 6,000 9,000 9,000 9,000 7,000 9,000 43,500 6,800 2,000 6,800 6,800 6,800 6,800 6,800

Grade and subject area Total Grade 4 mathematics Grade 4 reading Grade 4 science Grade 8 mathematics Grade 8 science Grade 12 mathematics Grade 12 science

NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

5

Primary Sampling Units for the 2000 National Main Assessment
The first stage of selection for all national main samples—except the grade 8 samples—was the selection of primary sampling units (PSUs). A PSU is a geographic area comprising an individual county or a group of contiguous counties. The entire United States (the 50 states and the District of Columbia) was divided into 1,027 PSUs. Each PSU met a minimum size requirement and was contained entirely within one of four geographic regions, specifically defined for NAEP, known as NAEP regions. These regions were used in stratifying the PSUs and ensured that each region was adequately represented in the various assessments. The primary sampling units (PSUs) used in the national main assessment of 2000 were selected from the PSU sampling frame designed and used for the assessments from 1994 through 2000. The basic PSU sample design for this NAEP period is a stratified probability sample with one PSU selected per stratum with probability proportional to population. The population size used was the from the 1990 Census. A PSU can be of a Consolidated metropolitan statistical area (CMSA), a metropolitan statistical area (MSA), a New England County Metropolitan Area (NECMA), a county, or a group of contiguous counties in the United States (including Alaska, Hawaii, and the District of Columbia). A total of 94 PSUs per survey year were selected. Some large PSUs were sampled with certainty and are common to all five survey years. The sampling frame used from 1994–2000 was constructed by partitioning all the counties in the United States into 1,027 non-overlapping groups following these specifications. • Each 1990 CMSA, and MSA that was not part of a CMSA, was considered a separate PSU. In New England, NECMAs were the metropolitan PSU unit (In New England a MSA may be part of county; but a NECMA may consist only of whole counties). Non-MSA PSUs were made to consist only of non-MSA counties. Whenever possible, each nonMSA PSU contained geographically contiguous counties with a minimum 1990 total population of 60,000 persons in the Northeast and Southeast regions and 45,000 persons in the Central and West regions. Region boundaries were not crossed in the definition of a PSU not even in the case of MSAs. If an MSA straddled regions, it was split into two PSUs, each falling entirely within regional boundaries.

The three pages that link from this page describe • • • how the 2000 national main PSUs were stratified, how the PSUs were selected, and how PSUs were formed and selected in six sparsely populated states in the grade 8 public school sample

to improve the efficiency of the field operations.

6

Definition of Primary Sampling Unit Strata
Primary sampling units (PSUs) on the PSU frame were grouped into eight major strata by crossing NAEP region and metropolitan statistical area (MSA) status. Each of the four NAEP regions (Northeast, Southeast, Central, and West) contained about one-fourth of the U.S. population at the time the frame was constructed. These regions, as listed below, were defined primarily by state. • • • • The Northeast region included Connecticut, Delaware, District of Columbia, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, Vermont, and Virginia. The Southeast region included Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia. The Central region included Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South Dakota, and Wisconsin. The West region included Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oklahoma, Oregon, Texas, Utah, Washington, and Wyoming.

Note that those counties and independent cities in Virginia which are part of the Washington, DC-MD-VA MSA are included in the Northeast region. The remainder of Virginia is included in the Southeast region. The 22 largest PSUs on the sampling frame were included in the PSU sample with certainty (guaranteed to be in the sample). The inclusion of these PSUs in the sample with certainty provided an approximately optimum, cost efficient sample of schools and students when samples were drawn within them at the required national sampling rate. The remaining smaller PSUs were not guaranteed to be selected for the sample. These PSUs were grouped into a number of noncertainty strata within the eight major strata. In all, there were 72 noncertainty strata (see the number of noncertainty strata in each major stratum by NAEP region). Further stratification was achieved by ordering the noncertainty PSUs within each major stratum according to certain socioeconomic characteristics discussed below. The noncertainty strata were defined so that the aggregate of the measures of size of the PSUs in a stratum was approximately equal for each stratum. The size measure used was the population from the 1990 Census. The characteristics, available for all PSUs, that were used to define strata were the percent minority population, the percentage change in total population since 1980, the per capita income, the percent of persons age 25 or over with college degrees, the percent of persons age 25 or over who completed high school, and the civilian unemployment rate. Up to four of these characteristics were used in any one major stratum. For each major stratum, the characteristics used were chosen by modeling NAEP PSU-level mean reading proficiency scores for the assessment years 1988 through 1992 and were the best predictors of PSU-level mean reading proficiency scores. An adjustment was made to rescale 1988 data to match 1989, 1990, and 1992. (NAEP did not conduct an assessment in 1991.) PSU-level mean reading scores were not needed for all PSUs in stratum. The scores were modeled from characteristics that were available for all PSUs, so that those characteristics could be used to form efficient strata for estimating proficiency scores.

7

Selection of PSUs in the 2000 National Main Assessment
A sample of 94 primary sampling units (PSUs) was drawn for each of the survey years 1994, 1996, 1998, and 2000. Of these 94 PSUs, 22 were selected with certainty because they had the largest populations in the PSU universe. These 22 certainty PSUs appeared in the set of 94 sample PSUs for each of the survey years. Within each of the 72 noncertainty strata, one PSU was selected with probability proportionate to its 1990 population for each survey year (i.e., within each stratum, a PSU's probability of being selected was proportional to its population). The PSUs were selected with probability proportional to size (PPS) with the twin aims of obtaining approximately self-weighting samples of students and having approximately equal workloads in each PSU. PSUs were drawn to minimize overlap of the PSUs from one assessment to the next, except that certainty PSUs were retained in each assessment year and some of the larger noncertainty PSUs are in the sample for more than one of these years. Each sample of 94 PSUs was drawn from the 1,027 PSUs on the frame. Primarily because of the use of MSAs as PSUs, PSUs varied considerably as to their probability of selection, because they varied greatly in size. In 2000, the 36 selected noncertainty MSA PSUs had probabilities of selection ranging from .03 to .58, while the 36 selected non-MSA PSUs had probabilities from .03 to .11. Parts of 42 states were included in the sample PSUs. Because one PSU was selected from each noncertainty stratum, the distribution of the noncertainty PSUs is the same as the noncertainty strata. The order of the PSUs was randomized within each stratum. Selection of PSUs within a stratum was not independent among the survey years. Ordering the PSUs within a stratum by size, geography, or other variables could have resulted in unintended and possibly detrimental correlations between survey estimates across years. The PSU ordering has no effect on sampling variance because only one PSU is selected for a given year. For each PSU within a stratum, a normalized measure of size was calculated by dividing the PSU's 1990 population by the aggregate 1990 population of all PSUs in the stratum. Next, a cumulative count of normalized measures of size was calculated for each PSU within a stratum. Where k is the PSU and i is the stratum, the cumulative count Cik is calculated as

Cik =

∑ NM
j=1

k

ij

where NMij is the normalizing measure for the jth PSU in the ith stratum. A random number between zero and one was generated for each stratum. With the use of this random number, denoted r, the following sequence of sample designation numbers was generated for the 5 survey years.
Selection of primary sampling units, by assessment year and sample designation number, national main assessment: 1994, 1996, 1998, 2000 Assessment year Sample designation number 1994 1996 1998 2000

r

r + 0.4

r + 0.8

r + 0.2

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

Only the noninteger part of any number in the sequence that exceeded 1.0000 was retained. For example, if r was equal to 0.326743, then r + 0.8 was equal to 1.126743 and 0.126743 became the sample designation number for 1998.
8

The first PSU in the stratum whose cumulative count was equal to or greater than r was designated the 1994 sample PSU. Similarly, the first PSUs in the stratum whose cumulative counts were equal to or greater than the noninteger portions of r + 0.4, r + 0.8, and r + 0.2 were designated the 1996, 1998, and 2000 main sample PSUs, respectively. The sample designation numbers for 1996, 1998, and 2000 were functions of r to minimize the overlap among the sets of main sample PSUs chosen for the various survey years. In strata with smaller numbers of PSUs, some PSUs had large enough normalized measures of size so that they were drawn for two and sometimes even three survey years. As the spacing between the sample designation numbers for any two consecutive survey years was 0.4, selecting the same PSU in two consecutive survey years was rare. For example, only one noncertainty PSU was selected for both the 1998 and the 2000 assessments.

9

Number of noncertainty strata in each major stratum, by total strata and number of strata per MSA or non-MSA PSU, national main assessment: By region, 2000 NAEP region Total Northeast Southeast Central West Total strata 72 10 24 20 18 Number of strata for Number of strata for MSA PSUs non-MSA PSUs 36 6 12 8 10 36 4 12 12 8

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

10

Sparsely Populated States in the 2000 National Main Assessment
Primary sampling units (PSUs) were formed and selected in six sparsely populated states in the grade 8 public school sample to improve the efficiency of the field operations. The basic grade 8 public school sample involved a stratified sample of schools, with state as the primary stratification, and an allocation of sampled schools to states proportionate to population size. If this design were applied to every state, only two or three schools would be assessed in states with the smallest populations. It would be extremely inefficient from a field operations point of view to assess unclustered samples of two or three schools in states such as Alaska and Wyoming, especially with the assessments taking place in February. In a few states, clustering the grade 8 public school sample in a few geographic PSUs, while increasing variance, did improve field work efficiency. Three PSUs were selected for the sample from PSUs formed in six sparsely populated states—Alaska, Idaho, Montana, North Dakota, South Dakota, and Wyoming. Because the six states were from two NAEP regions, and as national main sample designs use region as the first stratification characteristic (and region is a primary reporting variable), it was important to have at least one PSU from each region. In other national samples, each noncertainty PSU represents approximately 0.8 percent of the U.S. population, and this seemed a reasonable guide for the amount that should be represented by each of the PSUs in this case. For the sparsely populated states involved, North and South Dakota combined made up 0.53 percent of the U.S. population in 1992, while Alaska, Idaho, Montana, and Wyoming together made up 1.2 percent. On this basis, it was reasonable to select one PSU from the central NAEP region (North and South Dakota) and two from the west NAEP region (Alaska, Idaho, Montana, and Wyoming). Therefore there were a total three PSUs selected to represent sparsely populated states.

11

Sampling Frame for the 2000 National Main Assessment
The public school frame was constructed upon three sources: • • The primary source was the 1997–98 NCES Common Core of Data (CCD) Public Elementary and Secondary School Universe file. The secondary source was the Quality Education Data (QED) file, from which school location addresses were obtained. QED was also used to compile a list of Bureau of Indian Affairs (BIA) schools and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS). The coverage of BIA schools on CCD was incomplete, and it was preferable to simply drop all such records and replace them with QED data. CCD offered no coverage of DDESS schools. The third source was the Department of Defense Education Activity (DoDEA) website, which further refined the list of DDESS schools.

The NAEP private school frame was based on the 1997–98 Private School Survey (PSS) frame developed by the Census Bureau for NCES. PSS is a biennial mail survey of all private schools in the 50 states and the District of Columbia. The 1997–98 PSS frame of schools comprises a list frame and an area frame. The list frame is a conglomeration of the 1995–96 PSS frame and more up to date lists from state education agencies, private school associations, and other easily accessible sources. To improve the coverage of the PSS list frame, the Census Bureau conducted a more labor-intensive search for private schools in a random sample of areas throughout the United States, which were single counties or groups of counties sampled from an area frame constructed from all counties in the Nation. A complete list of private schools within each area was gathered using information from phone books, religious institutions, local education agencies, Chambers of Commerce and local government offices. Schools not already on the list frame were identified and added. The probabilities of selection for schools on the PSS file ranged from 0.02 to 1.00. A weight component was computed so that area frame schools represent all schools absent from the list frame, not just those in the selected areas.
Number of schools in the original sample by public, private, and new schools by grade, national main assessment: 2000 Grade 4 8 12 Total 1,400 1,200 940 Public 570 480 320 Private 820 720 620 New (public)

<5 <5 <5

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

School Frame Information Only regular schools offering grades 4, 8, or 12, including charter schools and statewide magnet schools, were eligible for inclusion in the NAEP frame. Regular schools are schools with students who are classified as being in a specific grade (as opposed to schools having only "ungraded" classrooms). Strictly ungraded schools, special education, prison and hospital schools, and vocational schools whose students were officially enrolled somewhere else were all purged from the frame. Number of Schools by Frame Source shows the numbers of schools included in the various sampling frame components. An independent sample of schools was selected for each of the grades. The population of eligible schools was restricted to the 94 PSUs selected for NAEP for all but the grade 8 public school sample. Estimates were made of the number of students who were eligible based on grade enrollment for each school in the frame. The CCD, QED, and PSS files give total enrollment and the grade range for each school and provide the means to calculate the average per grade enrollment. Any school not reporting total enrollment had the average per grade enrollment imputed as 20. For public schools, the average per grade enrollment was used as the estimated number of eligibles for whichever of grades 4, 8, and 12 were offered by the school. For private schools, the actual grade 4, 8, and 12 enrollment reported to PSS
12

was used, unless the number reported was zero. Under that circumstance, if the grade was offered, the average per grade enrollment was used. A school was assumed to offer a particular grade if the grade fell within the grade range, even if the enrollment reported for that specific grade was zero. However, any private school terminating in grade 11 was assumed also to offer grade 12. Private schools not responding to the PSS were assumed to offer all of grades 4, 8, and 12 to avoid bias. A school appeared in the frame for a particular grade without regard to its eligibility status for either of the two other designated grades, so there is considerable overlap among the three grade-level frames. There is considerable overlap among the three grade-level frames because schools could (and did) appear on more than one frame.

13

Selection of New Public Schools for the 2000 National Main Assessment
The 1997–98 NCES Common Core of Data (CCD) files do not contain data on schools that opened between 1998 and the assessment dates. Special procedures were implemented ensure that the NAEP assessment represented students in these more recently opened public schools. Small school districts— those with only one eligible school for a given grade—were handled differently from large school districts—those with more than one eligible school for a given grade. In small school districts, the schools selected for a given grade were thought to contain all students in the district that were eligible for the assessment. Officials of each sampled school district were asked if other schools with the appropriate grades for the assessment existed, and if so, the other schools were automatically included in the assessment. For large school districts, a district-level frame was constructed from the schools on the CCD file. Districts were sampled systematically with probabilities proportional to a measure of size. In most cases, the measure of size was total district enrollment, but in districts with very small size measures, a minimum measure of size was used. Each sampled district was asked to update the list of eligible schools derived from information on the CCD file. Frames of eligible new schools were then constructed at each grade, and samples of new schools were selected systematically with probability proportional to eligible enrollment using the same sampling rates as for the CCD schools. As a result of this process, two new public schools were selected at grade 4, four at grade 8, and one at grade 12.

14

School Substitution for Adjusting School Nonresponse in the 2000 National Main Assessment
NAEP assessments traditionally use substitute schools to compensate for school-level nonresponse and to improve the overall yield of assessed students. A substitute school is a school that can replace an original school if that original school refuses to participate in the assessment. Therefore, a nonresponding original sample school is replaced with another school that can be anticipated as yielding similar assessment results. If this is really the case, then using the substitute will reduce nonresponse bias. Potential substitute schools were selected for all sampled schools in the national main assessment where a close match could be identified. A new procedure was introduced for the 2000 assessment to identify substitutes. No sampled school was assigned more than one substitute, and no school was assigned to be a substitute for more than one school. The criteria for assigning substitutes were quite strict. Many sampled schools were not assigned substitutes at all, as there were no schools that met the necessary criteria to be a substitute. The potential pool of substitute schools for the national main assessment for a given grade was restricted to schools with that grade that were not part of any of the NAEP samples (state or main) for any grade. The substitutes were drawn in a single step from a joint pool of unsampled schools, with twelfth-grade sampled schools receiving substitutes first, eighth-grade sampled schools next, and fourth-grade schools last. There were a number of other "absolute" boundaries, as follows: • Only schools within the same school type group could serve as substitutes. The school type groups in the 2000 national main assessment comprised o o o o o o o • regular public schools, Bureau of Indian Affairs schools, Department of Defense Education Activity schools, other public schools, Catholic schools, non-Catholic religious schools, and other private schools.

Only those public schools that o o o were located in the same PSU, had the same locality type, and had the same minority status

as the originally sampled school(s) could serve as substitute schools. • Catholic schools could only have as substitutes schools in the same district (usually diocese).

Within the various cells defined by these absolute boundaries, a distance measure was computed for each sampled school and its potential matches. The distance measure was computed based on percent Blacks, percent Hispanics, and the square root of the grade enrollment, and used a standardized Euclidean distance Dij as follows:

Dij =

( Bi − Bj ) 2 ( Hi − Hj ) 2 ( Ei − Ej ) 2 + + VHB VHB VSQE

15

where • • •

Dij is the distance between the ith sampled school and the jth candidate substitute; Bi is the percentage of Black students in sampled school i, Hi the percentage of Hispanic students in sampled school i, Ei the school grade enrollment in sampled school i; Bj is the percentage of Black students in candidate substitute school j, Hj the percentage of Hispanic students in candidate substitute school j, Ej the school grade enrollment in candidate substitute school j; VBH is the average of the mean variance of Black percentages across all schools within the grade and the mean variance of Hispanic percentages across all schools within the grade; and VSQE is the mean variance of the square root of estimated grade enrollment across all schools
within the grade.

• •

Two passes were carried out for each grade. The first pass was a matching of regular public schools to schools outside their district. From a field viewpoint, it is preferable to have substitutes in different districts as many public school districts refuse to allow substitution. NAEP took an out-of-district substitute j for a sampled public school i if the pair had a Dij value less than 0.65. If there was more than one candidate for a particular sampled school i, then the j-school with the smallest Dij was selected. (Note that a particular j-school might be the best substitute for two or more sampled schools. In this case, the school was matched with the sampled school in which it had the smallest Dij value. Because of this, some sampled schools did not receive as substitutes their best matches, as those schools might have already been assigned to other sampled schools.) All matched schools in the first pass were removed from further consideration. The second pass was then a matching of regular public schools within their districts and of all private schools (Catholic schools within their dioceses, religious non-Catholic schools in one cell, and nonreligious private schools in one cell). The Dij limit was 0.65: if a sampled school had no potential substitute lower than this limit, then that sampled school received no substitute. (Note that a school might have a j-school with a Dij value less than 1.00 that is "lost" to another school and may thereby have no substitute.) Although the selected cutoff point of 0.65 on the distance measure is somewhat arbitrary, it was decided upon based on experience with this algorithm in the 1994 Trial State Assessment (Mazzeo, Allen, and Kline 1995) by a group of statisticians finding consensus on the distance measures at which substitutes began to appear unacceptable.

16

Original Sample Schools With and Without Substitutes in the 2000 National Main Assessment
The table below shows the number of schools with and without substitutes by grade and school type. A nonparticipating original school was replaced by a substitute when the nonparticipating school was considered a final refusal.
Original sample schools with and without substitutes and number of participating substitute schools, by grade and school type, national main assessment: 2000 School counts All original sample schools With assigned substitutes Without assigned substitutes Participating substitute schools Total 3,540 2,000 1,540 120 Grade 4 Public Private 580 440 130 20 820 440 380 30 Grade 8 Public Private 480 310 170 10 720 370 350 30 Grade 12 Public Private 320 90 230 10 620 350 270 30

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

17

Number of Schools by Frame Source for the 2000 National Main Assessment
The table below shows the number of public and private schools included in the 2000 national main assessment's frame components. All numbers are within the 94 selected primary sampling units (PSUs) for NAEP unless otherwise noted.
Number of schools by frame source and by grade, national main assessment: 2000 Grade 4 8 12
1 2

CCD-public1 22,190 24,5203 6,820

QED-public2 20 1304 10

PSS-private 10,570 9,260 4,270

Public schools, including state-run schools. Bureau of Indian Affairs (BIA) and Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS). 3 Within 44 states and the District of Columbia, plus 3 PSUs representing AK, ID, MT, ND, SD and WY. 4 Within all 50 states and the District of Columbia. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

18

Sampling of Schools in the 2000 National Main Assessment
The second stage of the national main design was the selection of the school sample for all but the grade 8 public school sample. For this sample, which did not select primary sampling units (PSUs) except in six sparsely populated states, the selection of schools was the first stage of sampling. Schools, obtained from several sources, were selected with probability proportional to size at rates targeted to meet required student sample sizes.

19

Assigning Measure of Size and Selecting School Samples for the 2000 National Main Assessment
To increase cost efficiency in sampling, the 2000 national main samples were designed to include schools with a relatively high number of students and more public schools with higher percentages of minority students. A measure of size was assigned to each school based on two factors—school type (public and private) and grade: School measure of size assignment scheme by school type and grade, national main assessment: 2000
Public schools School measure of size School measure of assignment scheme for size assignment grades 4 and 12 scheme for grade 8 0.25 ki , if Si < 6 0.5 ki , if Si < 10 School measure of size assignment scheme for private schools (grades 4, 8, and 12) 0.25 ki , if Si < 5

ki Si ÷ 20, if 6 Si < 20 ki , if 20 Si < 100 kiSi ÷ L, if Si ≥ 100

kiSi ÷ 20, if 10 Si < 20 ki , if 20 Si < 65 kiSi ÷ L, if Si ≥ 65

kiSi ÷ 20, if 5 Si < 20 ki , if 20 Si < 65 kiSi ÷ L, if Si ≥ 65

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

In the table above, Si is the estimated number of grade-eligible students in a given school i; L is the target within-school student sample size for a given grade and school type; and ki is the ith school's oversampling factor; where

L=

{

100, for grade 4 and 12 public schools for grade 8 public schools and all private schools

60,

and

ki =

{

2,

for high-minority public schools for low-minority schools and all private schools

1,

This procedure was used to obtain approximately self-weighting samples of students (i.e., students selected with approximately equal overall probabilities) at each grade and school type within oversampling domains. Two aspects of the school-level measure-of-size cause the resulting student weights to vary: • • Schools with fewer than 20 estimated grade-eligible students were assigned somewhat lower measures of size and thus lower probabilities of selection. This was designed to increase cost efficiency. Each public school designated as high-minority was given double the measure of size of a public school not designated high minority of similar size. Such high-minority schools were oversampled to enlarge the sample of Black and Hispanic students, which enhanced the reliability of estimates for these groups.

20

For each grade and public/private school sample, schools were selected (without replacement) within the appropriate sample areas systematically from a sorted list with probabilities proportional to assigned measures of size. Schools were ordered to achieve an implicit stratification on school characteristics related to school achievement and to ensure that the sampled schools represented a variety of population student groups. To meet specific sample requirements, the school sampling procedures differed by • • • grade 4 and 12 public schools; grade 8 public schools; and grades 4, 8, and 12 private schools.

21

Grade 4 and 12 Public School Samples for the 2000 National Main Assessment
Grade 4 and 12 public school samples were similar to past national main designs. The actual school sample from the school frame for each grade was a systematic sample using the assigned measures of size. Schools in the frame for each grade were ordered first by dividing into two subsets: certainty PSU schools and noncertainty PSU schools. The certainty PSU schools were ordered as follows: 1. NAEP region; 2. locale (locale is a Common Core of Data (CCD) field attached to each school defining the type of locality of the community: central city large city, central city mid-sized city, urban fringe large city, urban fringe mid-sized city, large town, small town, rural (note that the two CCD rural groups are collapsed together here); 3. minority status, whether the school was designated as high minority; 4. PSU stratum (see Assigning Measure of Size and Selecting School Samples for a definition of the PSU strata); 5. school type (school type for public schools includes the categories: regular, state-run, Bureau of Indian Affairs (BIA), and the Department of Defense's Domestic Dependent Elementary and Secondary Schools (DDESS); and 6. estimated grade enrollment, which approximates the number of students enrolled in the assessed grade.

The noncertainty PSU schools were ordered by PSU stratum first, type of locality second, minority status third, school type fourth, and estimated grade enrollment fifth. The sort order was "serpentine" at each level: an alternation of ascending to descending and descending to ascending within each higher level group (e.g., for certainty PSU public schools, the sort order for estimated grade enrollment within school type was ascending to descending for the first school type, descending to ascending for the next school type). This sort order is designed to ensure that adjacent schools in the sorted list have similar characteristics, even at variable cut points. For the grade 4 and 12 public school samples, the number of schools to be selected was not determined separately for each PSU in the national main samples. Instead, an overall sample size was determined so that the national target number of students to be assessed would be selected (adjusted for nonparticipation of schools and students and exclusion of students) and that the probability of selection of a student would be uniform over the United States (exceptions are students from very small schools and high minority schools), given that all students in schools with fewer than a specified number of eligible students would be selected and that only this specified number of students would be selected in larger schools. The number of schools selected for the grade 4 and 12 public school samples were 575 and 320 schools, respectively. They were based on a target sample of 29,500 students at grade 4 and 16,000 students at grade 12. These samples were inflated to 41,500 and 28,000, respectively, to offset student attrition due to school ineligibility, student exclusion, and student nonresponse. The inflation factors used in 2000, as shown below, were based on experience with the national main NAEP sample.

22

Inflation factor used to offset attrition in public school student participation by school type, by grade, the national main assessment: 2000 Grade 4 8 12 School type Catholic Other Catholic Other Catholic Other Inflation factor 1.062 1.333 1.075 1.449 1.161 1.783

NOTE: The inflation factor is an inverse of participation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

23

Grade 8 Public School Sample for the 2000 National Main Assessment
The eighth-grade national main public school sample was part of an integrated design with the eighthgrade state assessment sample. The integrated design allowed state and national main samples to be combined after data collection to produce a larger sample that would yield somewhat more precise state estimates and substantially more precise national estimates. In the grade 8 public school sample, states were treated as sampling strata, and a separate sample of schools was selected from most states. Fortysix jurisdictions, including 45 states and the combination of Department of Defense Domestic Dependent Elementary and Secondary Schools (DDESS) and Bureau of Indian Affairs (BIA) schools, were treated as separate strata, and a sample of schools was selected from each. Three geographic primary sampling units (PSUs) were selected from six low-population states (North Dakota, South Dakota, Alaska, Idaho, Montana, and Wyoming). Schools on the grade 8 public school frame, within each explicit sampling stratum, were sorted in a serpentine fashion by the following: 1. school size (school size for grade 8 public schools is a dichotomous variable defined as having at most 24 grade 8 students or at least 25 grade 8 students); 2. urbanicity stratum (urbanicity stratum is defined in terms of large or midsize central city, urban fringe of large or midsize city, large town, small town, and rural areas; separately within each sampling stratum, the CCD locale categories were collapsed to form urbanicity stratum); 3. minority stratum (minority stratum is defined within urbanization strata using the two largest minority enrollment percentages among Black, Hispanic, Asian and Pacific Islander, and Native American in each district); 4. minority status, whether the school was designated as high minority; and 5. estimated grade enrollment, which approximates the number of students enrolled in the assessed grade. A target number of schools were selected from each of the sampling strata. The allocation of the sample schools for the nonsparse states was approximately proportional to the assigned measure of size, which included an adjustment to oversample high minority schools.

24

Distribution of public schools in the sample, by sampling strata, grade 8 national main assessment: By jurisdiction, 2000 Sampling strata Total Alabama Arizona Arkansas California Colorado Connecticut Delaware District of Columbia Florida Georgia Hawaii Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia Washington See notes at end of table. Sampled schools 480 # 10 # 62 # # # # 30 14 # 20 # # # # 10 # # # 14 # # # # # # 12 # 26 14 14 # # 14 # # # 44 # # 12 10

25

Distribution of public schools in the sample, by sampling strata, grade 8 national main assessment: By jurisdiction, 2000 (continued) Sampling strata West Virginia Wisconsin Other jurisdictions BIA/DDESS1 Other PSUs # Rounds to fewer than 5 schools. 1 Bureau of Indian Affairs/Department of Defense Domestic Dependent Elementary and Secondary Schools. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment. Sampled schools # # # 12

26

Grade 4, 8, and 12 Private School Samples for the 2000 National Main Assessment
The 2000 national main assessment expanded the oversampling of private schools to allow the reporting of assessment results for up to six groups defined by religious orientation and/or private school association membership. Schools on the private school frame were grouped into seven strata for each grade separately. The seven strata corresponded to the six reporting groups plus an additional stratum for those private schools whose religious orientation and private school association membership were unknown. The first five reporting groups listed are mutually exclusive and exhaustive. Student assessment data from an independent school potentially contributes both to an independent school estimate and to the estimate for one of the other five groups (Over 70 percent of independent schools are nonsectarian. An additional 22 percent would be classified as "Other Religious.") In creating the sampling strata, National Association of Independent Schools (NAIS) membership took precedence over any other affiliation. Also because only a very small percentage of Catholic schools were NAIS members, Catholic schools for which NAIS membership was unknown were designated to the Catholic stratum, not to the NAIS stratum. Schools within each explicit sampling stratum for three grade-specific private school frames were sorted in a serpentine fashion by the following: 1. school size (school size for private schools is a dichotomous variable defined as having at most 19 grade-eligible students or at least 20 grade-eligible students); 2. PSU stratum; 3. school type (school type for private schools includes Catholic, other religious, and nonsectarian); and 4. estimated grade enrollment. For grades 4, 8, and 12 private school samples, schools were sampled at a rate designed to meet specific student-level sample sizes for each sampling stratum except for the unknown affiliation stratum. An equal probability sample of 25 schools was selected from the unknown affiliation stratum for each grade. In the remaining six sampling strata, school sample sizes were determined so that the national target of students to be assessed would be selected (adjusted for nonparticipation of schools and students and exclusion of students) and that the probability of selection of a student would be uniform over the United States (except students from very small schools), given that all students in schools with fewer than a specified number of eligible students would be selected and that only this specified number of students would be selected in larger schools. These school-level sample sizes also took into account that very large schools could be sampled more than once in grade 8 (2 times) and grade 12 (up to 3 times), as discussed in the Initial Student Sample. The target number of completed assessment booklets for Catholic schools was 3,000 for mathematics assessment and 3,000 for science at each of grades 4, 8, and 12. The target was 750 per subject and grade for each of the other five reporting groups. At grade 4 alone, a reading assessment and a special study for the mathematics assessment were conducted, with overall targets of 2,000 and 1,000 completed booklets, respectively. The samples for these last two subjects were allocated in rough proportion to the national distribution of fourth-graders across the six private school reporting groups. Because not all sampled schools and students were expected to participate, some allowance was made for attrition: the target student sample sizes were inflated to offset loss due to school ineligibility, student exclusion and student nonresponse. In years past, private school sample attrition has been consistently greater at the higher grades and for non-Catholic schools. Consequently, the inflation factors used to set the before-attrition student sample size targets for the 2000 assessment, and based on NAEP 1998 survey experience, varied by grade and Catholic affiliation.

27

Definition of private school reporting groups, national main assessment: 2000 Reporting group "Independent" Catholic Lutheran Includes Membership in: National Association of Independent Schools Orientation or affiliation: Roman Catholic Orientation or affiliation: Lutheran Church - Missouri Synod Evangelical Lutheran Church in America Wisconsin Evangelical Lutheran Synod Other Lutheran Membership in: Accelerated Christian Education American Association of Christian Schools Association of Christian Schools International Oral Roberts University Educational Fellowship Has religious orientation or affiliation, but is not Catholic, Lutheran or Conservative Christian No religious orientation or affiliation

Conservative Christian

Other religious Nonsectarian

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

28

Sample Type Assignment to Schools in the 2000 National Main Assessment
To determine the effect of using different inclusion criteria for students who might otherwise be excluded from the assessment, two different sample types (S2 and S3) were assigned to the sampled schools. For schools assigned to sample type S3, certain accommodations were offered to students with disabilities (SD) and students who were limited English proficient (LEP) to enable participation in the NAEP assessment. In sample type S2 schools, no such assessment accommodations were offered to SD/LEP students. Because of the offered accommodations, a larger proportion of the student population was able to participate in NAEP than when accommodations were not available. Sample type was assigned to schools separately for each grade so that 50 percent of the schools were assigned sample type 2 and 50 percent were assigned sample type 3. Schools that were sampled for more than one grade were assigned only one sample type, which was used then for all sampled grades. Schools selected for both the national main and state samples were assigned only one sample type as well to enable a school to provide the same assessment accommodations and to implement the same exclusion criteria regardless of grade or NAEP sample.

29

Assignment of Sessions to Schools for the 2000 National Main Assessment
Session types assigned to a school determined the assessment subject or subjects administered at the school. In the 2000 national main assessment, schools could be assessed in at least one subject and up to four different subjects depending on grade, school type (public or private), and student grade enrollment. There were four assessment subjects in grade 4: mathematics, reading, science, and marketbasket/short-form mathematics; and two for grades 8 and 12: mathematics and science. Schools were first assigned one, two, three, or four sessions depending on school type and student grade enrollment estimated from the frame. The cutoffs were as follows: Grade 4 Public Schools • • • • Schools with estimated grade enrollment of 25 or less: one session; Schools with estimated grade enrollment of 26 to 50: two sessions; Schools with estimated grade enrollment of 51 to 75: three sessions; and Schools with estimated grade enrollment of 76 or more: four sessions.

Grade 8 and 12 Public Schools • • Schools with estimated grade enrollment of 25 or less: one session; and Schools with estimated grade enrollment of 26 or more: two sessions.

Grade 4, 8, and 12 Private Schools • • Schools with estimated grade enrollment of 24 or less: one session; and Schools with estimated grade enrollment of 25 or more: two sessions.

Each session for the school was then assigned a session type based on the following probabilities: • • • • • Fourth-grade public schools: 7/29 mathematics, 6/29 reading, 9/29 science, and 7/29 short-form mathematics; Eighth-grade public schools: 1/2 mathematics and 1/2 science; Twelfth-grade public schools: 7/16 mathematics and 9/16 science; Fourth-grade private schools: varied by sampling strata and subject; and Eighth- and twelfth-grade private schools: 1/2 mathematics and 1/2 science.

If a school had significantly fewer students than expected (based on frame data) at the time of the assessment, the session count was adjusted downward. This adjustment was automatically made in the student sampling procedure as discussed in Updating Number of Sessions.

30

Session assignment probabilities for private schools by subject and sampling stratum, grade 4 national main assessment: 2000 Sampling stratum Roman Catholic Lutheran Conservative Christian Other Religious Nonsectarian Independent1 Unknown Affiliation
1

Mathematics 3,000/7,650 750/1,650 750/1,890 750/1,920 750/1,740 750/1,680 6,750/16,530

Science 3,000/7,650 750/1,650 750/1,890 750/1,920 750/1,740 750/1,680 6,750/16,530

Reading 1,110/7,650 90/1,650 270/1,890 270/1,920 150/1,740 120/1,680 2,010/16,530

A member of the National Association of Independent Schools (NAIS). NOTE: The numeric data above expresses the probability that a type of school (e.g., Roman Catholic) was assigned to the assessment. For example, the probability that a Roman Catholic school was assigned to the national main mathematics assessment was 3,000 / 7,650 = 0.392157. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

31

Student Sample Selection for the 2000 National Main Assessment
The final sampling stage for the national main designs is the selection of students within the sampled schools and the allocation of these students to session types. The overall student sample had three parts: the initial sample, the oversample of Black and Hispanic students in low minority public schools, and the oversample of SD/LEP students in public and private schools. The student samples were drawn and allocated to session types using a computer-based system and were specified by the use of session assignment forms (SAF). NAEP field supervisors completed student sampling a week before the assessment. Student Listing Forms (SLF) were prepared in each participating school. All enrolled students for the sampled grade were entered on the SLF in any order. The various student samples were drawn and allocated to session types using a computer-based system and were specified through SAFs. The sampled students were identified on the SAF by the SLF line numbers organized by session type.

32

Initial Student Sample for the 2000 National Main Assessment
Every student whose name appeared on the student listing form (SLF) had an equal chance of being selected in the initial sample. Based on the number of students on the SLF, the student-sampling program first determined the student sample size for the initial sample and then systematically generated a sample of SLF line numbers of the required size. The sampling process was systematic in that every kth line number was selected after a random start. The sampling interval used in the systematic sample was obtained by dividing the number of students on the listing form by the calculated sample size. The random start was a number between one and the sampling interval. The students selected for the initial sample were identified by the selected SLF line numbers. The target student sample size within a school for the initial sample varied by school type and grade. The target sample size was 100 students for grades 4 and 12 public schools and 60 students for grade 8 public schools. Because private schools—which generally have small enrollment—were heavily oversampled, they were allowed to be selected multiple times in grade 8 (up to two times) and in grade 12 (up to three times). The use of multiple selections reduced the number of sampled schools needed to achieve the student sample. The target sample size for private schools was 60 students per selection. Thus, large private schools with three selections had 180 students in sample. In general, the initial student sample consisted of all students if the number of students on the SLF was less than a maximum cutoff. Otherwise, the number of students in the initial sample was the target sample size. The maximum cutoff value was slightly larger than the target sample size to avoid samples that excluded only a handful of students in a grade. The maximum cutoff was • • • 110 students for grades 4 and 12 public schools, 70 students for grade 8 public schools, and 62 students per selection for all private schools.

33

Within-School Sampling in the 2000 National Main Assessment
The students selected for inclusion in the sample (in either the initial sample, Black and Hispanic oversample, or the SD/LEP oversample) were allocated to a session based on the final number of sessions assigned to the school and whether the school was public or private. Except for grade 12 public schools, students in schools with two or more assigned sessions were equally distributed across the sessions. For example, if a grade 4 public school was assigned four sessions—one in mathematics, one in reading, and two in science—and had a sample size of 100 students, 25 students would be assigned to mathematics, 25 to reading, and 50 to science. Grade 12 public schools with two assigned sessions distributed students unequally in order to meet the desired subject-specific sample sizes. Seven out of 16 students were assigned to mathematics and 9 out of 16 students were assigned to science.

34

Black and Hispanic Oversample for the 2000 National Main Assessment
In public schools with high minority (Black and Hispanic) enrollments, schools were assigned a measure of size twice the size of other [low minority] schools, therefore increasing their probability of selection, and indirectly increasing the number of Black and Hispanic students in the sample. In public schools with low minority enrollment, an oversample of Black and Hispanic students was selected. The procedure was as follows. After the initial sample was selected, the nonselected Black and Hispanic students were identified and listed. The extra Black and Hispanic students were sampled to a total that, in expectation, was the same number of Black and Hispanic students as were already selected. In practice, if the number of nonselected students was less than the number of selected students, then all nonselected Black and Hispanic students were to be assessed as well. Otherwise, Black and Hispanic students were sampled so that their overall within-school probability of selection was twice the rate of other students. The sampling of additional Black and Hispanic students was carried out using designated line numbers, as indicated on the session assignment form used to generate the samples of students in each school. In this way, the necessary information about the selection probability of each student was retained for use in weighting. Since the aim was to oversample by a factor of two where possible, but never more than two, the overall rate of oversampling of Black and Hispanic students was instead less than two. That is because in smaller low minority schools there were no students remaining who had not already been assigned to a session. The weighting procedures ensured that the results were not biased as a result of the relative under-representation of Black and Hispanic students from smaller low minority public schools.

35

SD/LEP Oversample for the 2000 National Main Assessment
The procedures for assessing students with disabilities (SD) and limited-English-proficient (LEP) students varied by sample type. SD/LEP students in sample type 3 were offered various accommodations not available to either non-SD/LEP students or SD/LEP students in sample type 2. As a measure to ensure an adequate sample size of SD/LEP students from both sample types 2 and 3, oversampling procedures were applied to SD/LEP students at all three grades. In this way, comparisons of the effect of offering accommodations to students have enhanced power to detect effects. The general intent of oversampling within each school was to select SD/LEP students at twice the rate at which non SD/LEP students were sampled (or to include all SD/LEP students if there were not sufficient numbers to permit sampling at twice the rate.) There was no oversampling of schools as part of the procedure. The procedure was as follows. In each school where oversampling of SD/LEP students was to occur, the initial student sample selection was drawn from the full list of eligible students. In addition, in public schools in low minority areas, oversampling of Black and Hispanic students occurred. Among those students not selected for either of the two prior sampling operations for this school, the SD/LEP students were identified. A sample from among these was drawn, using a sampling rate that would achieve the double sampling rate required overall. In most grade 4 cases, this involved selecting all such SD/LEP students in the school. The weighting procedures ensured that the results were not biased as a result of the relative under-representation of SD/LEP students from smaller schools.

36

Updating Number of Sessions in the 2000 National Main Assessment
If a school originally assigned two or more sessions was found to have significantly fewer students than was expected at the time of assessment, then the number of sessions originally assigned to the school was reduced as follows. • • • In general, sessions were dropped if a school did not have at least 12 students per session for the sampled grade. For grade 8 public schools and all private school samples, which had at most two sessions assigned to a school, the enrollment size cutoff for dropping a session was 24 students. For grade 12 public schools, which also were assigned at most two sessions, the cutoff was slightly higher (28) because of their differential student allocation rate. Instead of equally distributing students among the sessions assigned to a school as was done for all the other samples, grade 12 public schools required a student allocation of 7/16 to mathematics and 9/16 to science. For grade 4 public schools, which could have up to four assigned sessions, the rules for dropping sessions and the number of sessions to drop were more detailed, but still were based on same general principle of a minimum session size of 12 students.
Rules for dropping sessions for grade 4 public schools, national main assessment: 2000 Original number of assigned sessions 4 Number of sessions to drop 1 2 3 1 2 1

Cutoff rules 36 ≤ NSLF < 48 24 ≤ NSLF < 36 NSLF < 24

3 2

24 ≤ NSLF < 36 NSLF < 24 NSLF < 24

NOTE: NSLF = The number of students listed on the student listing form for a given school. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

In schools where sessions were reduced, the types of sessions were dropped proportionally to the session type allocation discussed in Assignment of Sessions to Schools. For example, in grade 12 the probability of dropping mathematics and science session in public schools was 7/16 and 9/16, respectively; for private schools, the probabilities for dropping mathematics and science were both 1/2.

37

Session Assignment Form for the 2000 National Main Assessment
As part of the computer-based sampling system, a session assignment form (SAF) was generated for each school where sampling was carried out and specified the students selected for the sample. The SAF contained the following information: • • • • • the number of students selected in the initial sample; the types of sessions that were to be administered at the school; whether the school was eligible for Black and Hispanic oversampling; line numbers for students selected for the Black and Hispanic oversample and/or for the SD/LEP oversample organized by session type; and special instructions as appropriate for the 2000 SD/LEP questionnaire.

38

Students Excluded from the 2000 National Main Assessment
In NAEP assessments, some students have been excluded from the assessment because school officials believed that either they could not participate meaningfully in the assessment or that they could not participate without assessment accommodations that the program did not, at the time, make available. These students fall into the general categories of students with disabilities (SD) and limited-English­ proficient students (LEP). Some identified students fall within both of these categories. For the 2000 assessment, schools were asked to complete a questionnaire for each student identified as having a disability or its equivalent (SD) and/or being limited English proficient (LEP). School personnel determined if any of these students should be excluded from the assessment as shown in Determining the SD/LEP Student's Capability to Participate in the Assessment. The tables Weighted Percentages of Students Excluded (S2) and Weighted Percentages of Students Excluded (S3) show rates of excluded SD/LEP students from the S2 and S3 samples. As Sample Type Assignment to Schools explains, students in sample type 2 (S2) schools were not offered accommodations, whereas students in sample type 3 (S3) schools were offered accommodations. The two tables show the distributions of excluded students by reason for exclusion. The dominant reason for exclusion from NAEP across all grades and subjects was a student disability. The proportion attributable to student disability increased with grade, whereas the proportion attributable to limited English proficiency, the second reason, decreased with grade. The table Student Exclusion Rates by School Type summarizes the overall rates of exclusion for each reporting population, grade, subject, and school type. As expected, the exclusion rates for theR3 reporting population are significantly lower than those for the R2 reporting population because accommodations were offered to students in the R3 population if necessary. It also shows that the exclusion rates for the private schools, which generally have very few SD or LEP students, are much lower than for the public schools. The exclusion rates for the grade 12 samples are significantly lower than those for the grades 4 and 8 samples.

39

Determining the SD/LEP Student's Capability to Participate in the Assessment
Some of the students identified on the administration schedule as a student with a disability (SD) or as limited English proficient (LEP) or may be incapable of participating meaningfully in the assessment. For the 2000 assessment, school staff members knowledgeable about the SD/LEP student, followed these guidelines in determining the student's capability to participate in the assessment. 1. Student With a Disability A student who is identified on the Administration Schedule as a student with a disability (SD) should be included in the NAEP assessment unless: • • the student's cognitive functioning is so severely impaired that he or she cannot participate; or the student's Individualized Education Program (IEP) requires that the student be tested with an accommodation that NAEP does not permit, and the student cannot demonstrate his or her knowledge of the subject area in which he/she will be assessed without that accommodation.

2. Limited-English-Proficient Student A student who is identified on the Administration Schedule as limited English proficient (LEP) and who is a native speaker of a language other than English should be included in the NAEP assessment unless: • • the student has received reading or mathematics instruction primarily in English for less than three school years including the current year; and the student cannot demonstrate his or her knowledge of reading or mathematics in English even with an accommodation permitted by NAEP.

3. Consult With School Staff The decision regarding whether any of the students identified as SD or LEP cannot be included in the assessment should be made in consultation with knowledgeable school staff. The following questions in the2000 SD/LEP Questionnaire (113KB) pertain to this decision as well: • Section A for students with disabilities included such questions as "Which of the following best describes this student’s primary disability?" "What is the degree of this student’s disability?" "Does the student’s IEP state that he or she cannot participate in assessments such as NAEP, even with accommodations?" "What grade level of instruction is this student currently receiving in reading? mathematics" science?" "Is this student participating in the same curriculum content as nondisabled students receiving the same grade level of instruction in reading? mathematics? science?" "Are any accommodations or adaptations used for district or statewide achievement testing for this student?" "If so, which accommodations?" (questions 1 through 17). Section B for students with limited English proficiency included such questions as "What is this student’s first or native language?" "Including the current school year, how long has this student been receiving academic instruction in reading primarily in English?" "..in mathematics primarily in English?" "..in science primarily in English?" "During this school year, what percentage of this student’s academic instruction is provided in his/her native language?" "What grade level of instruction is this student currently receiving in English reading? math? science?" "Are any accommodations or adaptations used for district or statewide achievement testing for this student?" "Which accommodations?" "How would this student most appropriately participate in the NAEP mathematics assessment?" "..reading assessment?" "..science assessment?" (questions 18 through 30)

40

Weighted percentages of students excluded (SD and LEP) from assessment in the S2 sample, by grade and subject area, national main assessment: 2000 Percentage of students identified as SD or LEP 14.98 16.42 2.53 15.26 16.78 1.67 14.44 15.89 2.21 14.41 15.53 3.28 14.37 15.57 2.86 9.22 9.84 2.62 9.10 9.71 2.71 Percentage of Percentage of students students identified excluded as SD 6.77 7.47 0.74 7.14 7.89 0.42 7.07 7.84 0.61 6.77 7.40 0.51 6.74 7.35 0.92 4.26 4.54 1.30 3.91 4.18 1.06 10.84 11.83 2.26 11.20 12.28 1.51 10.19 11.16 1.97 11.29 12.12 2.93 11.52 12.44 2.73 7.38 7.85 2.39 7.53 8.02 2.45 Percentage of Percentage of students Percentage of students that were that were students excluded identified excluded and SD and LEP with LEP 5.48 6.03 0.72 5.86 6.48 0.34 5.74 6.35 0.57 5.85 6.39 0.51 5.86 6.38 0.82 3.95 4.21 1.19 3.64 3.89 0.99 5.08 5.61 0.44 4.86 5.37 0.29 5.11 5.68 0.31 3.76 4.10 0.38 3.53 3.88 0.20 2.16 2.29 0.77 1.99 2.10 0.79 2.05 2.28 0.08 1.87 2.07 0.08 2.13 2.37 0.05 1.40 1.54 0.03 1.42 1.55 0.18 0.58 0.58 0.61 0.60 0.61 0.58

Grade 4

Subject area Mathematics

School type Overall Public Private Overall Public Private Overall Public Private Overall Public Private Overall Public Private Overall Public Private Overall Public Private

Reading

Science

8

Mathematics

Science

12

Mathematics

Science

NOTE: Students who are both SD and LEP are included in all columns that apply to them. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

41

Weighted percentages of students excluded (SD and LEP) from assessment in the S3 sample, by grade and subject area, national main assessment: 2000 Percentage of students identified as SD or LEP 16.70 18.31 2.57 16.82 18.34 3.38 16.24 17.78 3.00 13.20 14.23 3.17 12.50 13.53 2.84 9.34 9.88 3.61 8.63 9.08 3.96 Percentage of Percentage of students students identified excluded as SD 3.83 4.24 0.22 5.61 6.20 0.35 4.26 4.71 0.34 3.67 4.02 0.17 3.33 3.62 0.65 2.24 2.40 0.53 1.95 2.10 0.37 11.95 13.02 2.56 11.52 12.47 3.13 11.45 12.44 2.99 10.32 11.07 2.98 9.95 10.74 2.55 7.16 7.55 3.05 6.62 6.91 3.61 Percentage of Percentage of students Percentage of students that were that were students excluded identified excluded and SD and LEP with LEP 3.12 3.45 0.22 4.06 4.49 0.25 3.31 3.65 0.34 3.23 3.54 0.16 2.86 3.10 0.64 2.03 2.19 0.34 1.75 1.89 0.34 5.32 5.92 0.03 5.91 6.55 0.25 5.51 6.14 0.08 3.37 3.69 0.19 2.95 3.23 0.33 2.41 2.59 0.56 2.26 2.43 0.50 0.96 1.07 0.00 1.98 2.19 0.10 1.35 1.50 0.06 0.78 0.86 0.01 0.66 0.73 0.05 0.32 0.33 0.18 0.39 0.41 0.14

Grade Subject area 4 Mathematics

School type Overall Public Private Overall Public Private Overall Public Private Overall Public Private Overall Public Private Overall Public Private Overall Public Private

Reading

Science

8

Mathematics

Science

12

Mathematics

Science

NOTE: Students who are both SD and LEP are included in all columns that apply to them. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

42

Student exclusion rates by school type, by reporting population, subject, and grade, national main assessment: 2000 Reporting population R2 Accommodations not permitted Subject Mathematics Grade 4 8 12 4 8 12 4 4 8 12 4 8 12 4 Public 7.47 7.40 4.54 7.84 7.35 4.18 7.89 4.24 4.02 2.40 4.71 3.62 2.10 6.20 Private 0.74 0.51 1.30 0.61 0.92 1.06 0.42 0.22 0.17 0.53 0.34 0.65 0.37 0.35 Overall 6.77 6.77 4.26 7.07 6.74 3.91 7.14 3.83 3.67 2.24 4.26 3.33 1.95 5.61

Science

Reading R3 Accommodations permitted Mathematics

Science

Reading

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

43

Participation Results for the 2000 National Main Assessment
School Participation Results See the tables on the next page. Student Participation Results The table "Weighted Student Participation Rates by School Type and Subject Area" shows the weighted student participation rates of eligible students. The set of eligible students consists of the selected students, after removing the excluded students. For a given session, a makeup session was called for when, for various reasons, more than a predetermined tolerable number of eligible students were absent from the originally scheduled session to which they were assigned. The participation rates given in the table express the number finally assessed as a percentage of those initially eligible in the participating schools. Participation rates are shown for public and private schools separately. The combined impact of school nonparticipation and student absenteeism from sessions within participating schools is summarized in the table "Weighted Participation Rates by Subject Area, Grade, and Reporting Population." It shows the school participation rates before and after cooperating substitute schools are added and the student participation rates after absentee sessions are added. The two overall participation rates are the student participation rates multiplied to the before-substitution school participation rate and the after-substitution school participation rates respectively. The results show that, consistent with earlier rounds of NAEP, the overall level of participation decreases substantially with the increase in the grade of the students. Analysis is typically performed separately by grade and subject type, and NCES standards regarding acceptable potentials for bias are expressed in terms of weighted participation rates. The school and student participation rates decrease as grade increases for each subject, while participation rates by reporting population are essentially the same. The procedures for substituting for nonparticipating schools or imputing for them through weighting and the procedures for imputing for absent students through weighting were designed to reduce the biases resulting from school and student nonparticipation. These procedures are discussed in the Weighting Procedures and Variance Estimation section for the 2001 assessment.
Weighted school and student participation rates, by grade, reporting population, and subject area, national main assessment: 2000 Reporting population R2 R3 R2 R3 R2 R3 R2 R3 R2 R3 R2 R3 R2 R3 School participation Before After substitution substitution 85.4 85.4 82.8 82.8 78.2 78.2 84.3 84.3 84.7 84.7 82.4 82.4 77.4 77.4 88.7 88.7 85.5 85.5 82.4 82.4 87.4 87.4 88.4 88.4 85.2 85.2 81.9 81.9 Student participation 95.6 95.5 92.2 91.7 76.6 77.2 95.7 95.8 95.7 95.6 92.0 91.9 75.9 75.9 Overall participation 84.8 84.7 78.8 78.4 63.2 63.6 83.6 83.7 84.6 84.5 78.4 78.3 62.2 62.2

Subject area Mathematics

Grade 4 8 12

Reading Science

4 4 8 12

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

44

School Participation in the 2000 National Main Assessment
The table below shows a detailed breakdown of school participation status for the 2000 national main assessment; participation rates are also shown for previous national main assessments.
Summary of school participation experiences by grade and school type, unweighted, national main assessment: 2000 Grade 4 Grade 8 Grade 12 Number of Number of Number of Number of Number of Number of public private public private public private schools schools schools schools schools schools 580 20 20 30 30 480 20 500 820 60 0 § 130 630 30 650 480 30 10 30 40 380 10 400 720 90 0 § 120 500 30 530 320 10 20 30 240 10 250 620 120 0 § 140 360 30 390

Participation experience Total original sample Out-of-scope school1 District refused District refused for this school only School refused Cooperating Cooperating replacements for refusals (2000) Total participating schools

§ Fewer than 10 schools. 1 School did not have eligible grade; had eligible grade but no enrolled students; closed school; was not a regular school. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment. Cooperation rate of schools before the addition of substitute schools to the sample, by grade and school type, unweighted, national main assessment: Various years, 1992–2000 Grade 4 Grade 8 Grade 12 Percent of Percent of Percent of Percent of Percent of Percent of public private public private public private schools schools schools schools schools schools 86 81 86 86 86 82 84 81 83 86 85 80 81 75 79 79 81 72

Cooperation rate Cooperation rate before substitution (2000) Cooperation rate before substitution (1998) Cooperation rate before substitution (1996) Cooperation rate before substitution (1994) Cooperation rate before substitution (1992)

NOTE: The figures for 1992–98 are for public and private schools combined. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), various years, 1992–2000 Assessments. Cooperation rate of schools before and after the addition of substitute schools to the sample, by grade and school type, unweighted, national main assessment: 2000 Grade 4 Grade 8 Grade 12 Percent of Percent of Percent of Percent of Percent of Percent of public private public private public private schools schools schools schools schools schools 86 89 82 86 84 86 80 84 81 86 72 78

Cooperation rate Cooperation rate before substitution Cooperation rate after substitution

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

45

The cooperation rates shown in the table above are computed using the counts in the table as follows: Before substitution rate = (100 × cooperating) /(cooperating + district refused + district refused for this school only + school refused) For example, grade 4 public: 100 × 500 / (480 + 30 + 30 + 20) = .86 After substitution rate =(100 × total participating schools) / (cooperating + district refused + district refused for this school only + school refused) For example, grade 4 public: (100 × (496) / (480 + 30 + 30 + 20) = .89 The difference is between these two equations is that the rate after substitution includes cooperating substitutes.

46

Weighted Student Participation Rates by School Type and Subject Area, National Main Assessment (Accommodations Not Permitted [R2] and Accommodations Permitted [R3]): 2000
Weighted student participation rates by school type and subject area, national main assessment (accommodations not permitted [R2]): 2000 Public Private Overall Number Participation Number Participation Number Participation Subject area Grade eligible rate eligible rate eligible rate Mathematics 4 8 12 4 4 8 12 7,400 10,300 9,200 6,200 9,600 10,400 11,600 95.6 91.8 75.5 95.6 95.6 91.6 74.7 6,700 6,600 7,400 2,000 6,800 6,600 7,300 95.9 96.2 87.6 96.5 96.5 95.9 88.6 14,100 16,800 16,600 8,300 16,400 17,000 19,00 95.6 92.2 76.6 95.7 95.7 92.0 75.9

Reading Science

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

Weighted student participation rates by school type and subject area, national main assessment (accommodations permitted [R3]): 2000 Public Private Overall Number Participation Number Participation Number Participation Grade eligible rate eligible rate eligible rate 4 8 12 4 4 8 12 7,800 10,500 9,400 6,400 9,900 10,500 11,900 95.5 91.3 76.1 95.7 95.5 91.5 74.7 6,700 6,600 7,400 2,000 6,900 6,600 7,400 96.0 96.2 87.5 96.7 96.4 95.9 88.6 14,500 17,100 16,900 8,400 16,800 17,100 19,300 95.5 91.7 77.2 95.8 95.6 91.9 75.9

Subject Mathematics

Reading Science

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

47

State Assessment Sample Design in 2000
The 2000 state assessment sample design utilized a multistage probability sampling approach for all participating jurisdictions. The assessment tested fourth- and eighth-graders in the subject areas of mathematics and science. The first stage sampled public schools from the participating jurisdiction. After designating assessment sessions for the selected schools, the final stage sampled students from those schools and assigned them to sessions. Before sampling began, the state assessment defined its target population for the state or jurisdiction and created a school sampling frame. A stratification of schools within the sampling frame enabled the school sample to represent a variety of student groups. Next, schools were selected and assigned sessions. Within the sampled schools, the student sample selection identified which students would participate in each assessment session.

48

Target Population of the 2000 State Assessment
The NAEP state assessment targets public school students enrolled in the fourth or eighth grades. All 50 states and the District of Columbia are eligible to participate in the state assessment, as well as territories and Bureau of Indian Affairs (BIA) schools. The state assessment also includes separate jurisdictions of Department of Defense Education Activity schools. One jurisdiction covers the department's domestic schools (DDESS), while another jurisdiction covers its international schools (DoDDS). Participation in the 2000 assessment included 41 states, the District of Columbia, DDESS, DoDDS, and three U.S. territories, as detailed on the web page Number of Schools and Enrollment of All Students and Testable Students for Participating Jurisdictions.
Sampling parameters for the grades 4 and 8 public school frames, state assessment: By jurisdiction, 2000 Jurisdiction Total Alaska Alabama Arkansas Arizona California Colorado Connecticut Delaware Florida Georgia Hawaii Iowa Indiana Illinois Indiana Kansas Kentucky Louisiana Massachusetts Maryland Maine Michigan Minnesota Missouri Mississippi Montana North Carolina North Dakota Nebraska New Hampshire New Jersey New Mexico Nevada New York See notes at end of table. Grade 4 Grade 8 Number of schools Number of students Number of schools Number of students 50,500 380 790 540 820 5,210 850 610 90 1,750 1,090 180 750 340 2,400 1,150 800 830 820 1,080 820 390 2,120 1,120 1,130 490 460 1,210 320 880 260 1,310 400 280 2,270 3,593,000 10,000 59,200 36,200 65,100 474,000 52,900 42,300 9,100 174,300 104,800 14,500 35,800 18,100 152,800 77,600 34,700 51,200 59,000 76,900 62,600 16,700 134,500 65,600 69,300 41,000 12,100 99,300 8,400 21,500 16,900 101,000 25,000 23,000 218,700 25,900 300 520 360 480 2,130 390 240 60 950 440 60 420 200 1,580 490 440 440 490 430 280 240 1,000 820 680 340 290 600 230 510 130 660 180 110 1,070 3,542,000 10,400 56,500 37,000 62,700 425,500 53,700 38,800 8,700 183,900 104,300 14,000 38,800 19,400 149,800 76,200 36,600 51,600 58,900 70,700 61,400 17,300 132,100 74,100 70,640 40,600 13,200 94,500 9,200 23,600 16,200 92,600 26,000 22,400 203,600

49

Sampling parameters for the grades 4 and 8 public school frames, state assessment: By jurisdiction, 2000 (continued) Grade 4 Number of schools 2,190 940 770 1,810 190 570 400 950 3,540 440 1,090 240 1,280 1,150 530 220 100 40 100 20 20 Grade 8 Number of schools 1,030 620 360 760 50 270 290 560 2,070 170 420 130 700 570 220 100 30 10 60 10 10

Jurisdiction Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Virginia Vermont Washington Wisconsin West Virginia Wyoming Other jurisdictions District of Columbia DDESS1 DoDDS2 Guam Virgin Islands
1 2

Number of students 139,500 49,200 42,500 140,410 12,300 51,700 9,700 73,100 290,400 35,690 88,200 8,200 78,400 60,900 21,900 7,100 6,200 3,300 6,300 2,600 1,700

Number of students 146,100 50,200 42,900 139,700 11,600 52,300 10,900 66,800 297,900 38,500 87,900 8,300 82,000 66,600 22,900 8,100 4,000 1,700 4,600 2,400 1,900

Department of Defense Domestic Dependent Elementary and Secondary Schools. Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

50

Sampling Frame for the 2000 State Assessment
Both the National Center for Education Statistics (NCES) and Quality Education Data, Inc. (QED), an educational research company, supplied school data files to create the 2000 state assessment's school sampling frame. Creating the school sampling frame required comprehensive lists of public schools in each participating jurisdiction. The lists needed accurate school information such as • • • possible stratification variables (urbanization and minority enrollment.); school grade span and enrollment data; and school identification information (school name and school address).

The Common Core of Data (CCD) public school file, maintained by NCES, served as the primary school sampling frame. This database covered elementary and secondary schools in the 50 states, District of Columbia, and outlying areas. It also included Department of Defense Dependents Schools (DoDDS). QED maintains a national education database school file that lists public and nonpublic schools within the United States. This database provides DoDDS for the school sampling frame.

51

Number of Schools and Enrollment of All Students and Testable Students for Participating Jurisdictions in the 2000 State Assessment
The table below shows the distribution of schools with fourth- or eighth-grade in the 2000 state assessment's school sampling frame. For all participating and non-participating jurisdictions, the table lists the number of schools containing each grade and their combined grade-specific enrollments. Estimates of grade-specific enrollment for each school results from dividing total school enrollment by the number of grades in the school. This information comes from the 1997-1998Common Core of Data (CCD) public school file.
Distribution of schools and enrollment for participating jurisdictions by grade, number of schools, and number of students enrolled, state assessment: By jurisdiction, 2000 Fourth-grade schools Total Total schools enrollment 42,410 790 820 540 5,210 610 1,090 180 335 2,403 1,146 753 796 828 819 389 819 1,079 2,124 1,122 489 1,130 458 876 284 403 2,266 1,210 317 2,188 944 771 185 567 949 3,541 3,001,500 59,200 65,100 36,100 474,000 42,300 104,800 14,500 18,085 152,801 77,584 35,808 34,705 51,186 58,981 16,698 62,576 76,931 134,525 65,576 41,021 69,339 12,087 21,491 22,973 24,805 218,649 99,310 8,380 139,463 49,187 42,479 12,260 51,660 73,129 290,379 Eighth-grade schools Total Total schools enrollment 21,300 520 480 360 2,130 240 440 58 185 1,579 491 † 436 437 489 236 275 429 1,003 819 341 678 290 508 112 179 1,072 595 225 1,031 620 357 52 273 551 2,070 2,906,100 56,500 62,700 37,000 425,500 38,800 104,300 13,997 19,410 149,770 76,223 † 36,600 51,580 58,943 17,287 61,420 70,665 132,145 74,086 40,603 70,636 13,161 23,550 22,363 25,942 203,577 94,498 9,151 146,057 50,221 42,931 11,555 52,288 66,817 297,939

Jurisdiction Total Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas See notes at end of table.

52

Distribution of schools and enrollment for participating jurisdictions by grade, number of schools, and number of students enrolled, state assessment: By jurisdiction, 2000 (continued) Fourth-grade schools Total Total schools enrollment 444 238 1,092 528 1,145 223 22 106 40 96 23 24 35,685 8,168 88,245 21,948 60,884 7,083 1,172 6,164 3,316 6,317 2,593 1,728 Eighth-grade schools Total Total schools enrollment 173 132 417 216 572 96 22 32 14 59 7 7 38,464 8,267 87,860 22,935 66,644 8,077 1,120 4,019 1,707 4,620 2,366 1,893

Jurisdiction Utah Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa District of Columbia DDESS1 DoDDS2 Guam Virgin Islands

† Iowa did not participate in the eighth-grade state assessment. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

53

Stratification of Schools in the Sampling Frame for the 2000 State Assessment
Before school selection begins for the state assessment, the school sampling frame undergoes a hierarchical sort of schools based on selected demographic variables. This stratification process ensures that the state assessment represents a variety of school and student groups. Stratified sampling also improves the precision of the NAEP estimates by allowing separate estimates of population parameters for each stratum and removing the variation between strata. Stratification variables provide information about a school, its students, and its environment. The state assessment uses such stratification variables as: • • • • urbanization, minority enrollment, achievement data, and median household income.

54

School Stratification Variables for the 2000 State Assessment
The school sampling frame undergoes two levels of stratification. Primary stratification breaks down the school sampling frame by selected demographic detail. Implicit stratification provides a measure of control over additional school variables after the primary stratification occurs. Primary stratification variables for public schools, listed in hierarchical order, are as follows: • • • • small or large school district class, school size class, urbanization classification, and minority classification.

Implicit stratification variables for public schools include achievement data and median household income. If available for a participating jurisdiction and grade, achievement data becomes the implicit stratification variable. If no viable achievement data are found, the median household income of the school location's ZIP code serves as the implicit stratification variable. Prior to the selection of the school samples, the public schools are sorted by the four primary stratification variables in an order such that changes occur on only one variable at a time. Serpentine sorting accomplishes this feat by alternating between ascending and descending sort order on each variable successively through the sort hierarchy. For example, if schools are to be sorted by urbanicity (urban, suburban, rural) and size (enrollment): • • • Urban schools are sorted in ascending order of enrollment. Suburban schools are sorted in descending order of enrollment. Rural schools are sorted in ascending order of enrollment.

This sorting pattern places the large urban schools next to large suburban schools and the small suburban schools next to small rural schools. A traditional sort places the large urban schools next to small suburban schools and the large suburban schools next to small rural schools, which is less desirable for variance estimation purposes. The implicit stratification of public schools by either achievement data or median household income maintains the established serpentine order. This final sorting stage results in implicit stratification of either variable.

55

Urbanization Classification for the 2000 State Assessment
The National Center for Education Statistics (NCES) type of location variable defines the urbanization of the school's location. The type of location variable has seven urbanization categories: • Large Central City: A central city of a metropolitan statistical area (MSA) with a population greater than or equal to 400,000, or a population density greater than or equal to 6000 persons per square mile; Mid-Size Central City: A central city of an MSA but not designated as a large central city; Urban Fringe of Large City: A place within an MSA of a large central city and defined as urban by the U.S. Census Bureau; Urban Fringe of midsize City: A place within an MSA of a mid-sized central city and defined as urban by the U.S. Census Bureau; Large Town: A place not within an MSA, but with a population greater than or equal to 25,000 and defined as urban by the U.S. Census Bureau; Small Town: A place not within an MSA, with a population less than 25,000, but greater than or equal to 2,500, and defined as urban by the U.S. Census Bureau; and Rural: A place with a population of less than 2,500 and defined as rural by the U.S. Census Bureau.

• • • • • •

Each urbanization classification within a participating jurisdiction must have a minimum of 10 percent of the jurisdiction's eligible students. Some jurisdictions must have their type of location categories collapsed together until this requirement is met. Specific rules govern the collapsing of type of location categories. Initial collapses merge the central city categories, urban fringe categories, and/or town categories. If needed, additional collapses merge together all central city and urban fringe categories, or both town and the rural categories. For the 2000 state assessment, all public school jurisdictions requiring urbanization classification contained no missing urbanization classification values. Note that the urbanization classification variable is more detailed than the type of location variable used for reporting data. The reported type of location variable contains only three categories: urban, suburban, and rural.

56

Minority Enrollment Classification for the 2000 State Assessment
A school's minority classification depends on its urbanization classification and the percentages of its two largest minority student populations. Minority student populations include Blacks, Hispanics, Asians and Pacific Islanders, and Native Americans. Urbanization classifications with small minority populations do not require minority stratification. Urbanization classifications with larger minority populations are divided into three or four minority enrollment classes, described in Cases 2 and 3 below. Based on its urbanization classification and student race/ethnicity percentages, each school falls into one of three minority classification schemes. These schemes are as follows: • Case 1: An urbanization classification with less than 10 percent of students in its largest minority group and 7 percent of students in its second largest minority group. No stratification by minority enrollment occurs within this classification. Case 2: An urbanization classification with greater than or equal to 10 percent of students in its largest minority group or 7 percent of students in its second largest minority group, but not more than 20 percent in either group. This classification sorts schools by the combined percent student enrollment of its two largest minority groups, then divides the schools into three groups (Low, Medium, High) with similar numbers of students per minority classification. Case 3: An urbanization classification with greater than 20 percent of students in both its largest and second largest minority groups. This classification designates the minority group with the highest concentration as the primary stratification variable, and the other minority group as the secondary stratification variable. After sorting schools by the primary stratification variable, the schools split into two groups with approximately equal numbers of grade-specific students. Each resulting group sorts its schools by the secondary stratification variable, then splits into two subgroups of schools with approximately equal numbers of grade-specific students. This procedure creates four minority classifications: low primary/low secondary; low primary/high secondary; high primary/low secondary; and high primary/high secondary.

The minority classifications exist solely for creating efficient stratification design at this sampling stage. They also reduce sampling errors for achievement-level estimates. The analysis and reporting of state assessment data, however, do not use these minority classifications. Methods of imputing missing public school minority enrollment data include: • • • Assigning the average minority enrollment percentages of schools within the school district; Assigning the average minority enrollment percentages of schools within the school's five digit ZIP code; and Assigning the average minority enrollment percentages of schools within the school's three digit ZIP code prefix.

For the 2000 state NAEP assessment, all schools in Idaho had missing minority enrollment data. The Common Core of Data (CCD) public school file contained no student race/ethnicity data for all Idaho schools, preventing any imputation of missing minority enrollment data.

57

Achievement Data for the 2000 State Assessment
Where available, achievement data served as the implicit stratification variable for public schools. Each participating jurisdiction with its own educational assessment program submitted the results for each school. These school achievement results varied by jurisdiction, grade, and testing subject. The achievement data files were processed in order to select one achievement measure that could be used in stratification for each jurisdiction and grade. A jurisdiction that used achievement data for stratification assigned schools with missing achievement data the median achievement result within the school's urbanization and minority classifications. For each combination of jurisdiction and grade where achievement data were not available or usable, median household income served as the implicit stratification variable.

58

Usable Achievement Data for the 2000 State Assessment
Usable school achievement measures met the following criteria: • • • • An achievement measure reflected the assessment's targeted grades and subjects. An achievement measure came from the most recent educational assessments. An achievement measure appeared for at least 70 percent of schools within the jurisdiction and grade. An achievement measure adequately differentiated schools (e.g., district-level measures or pass/fail indicators tended not to make effective sort variables).

For the 2000 state assessment, usable achievement data included average school test scores for fourthand eighth-grade mathematics and science. Third or fifth grade mathematics and science data could substitute for fourth grade data, while seventh grade mathematics and science data could substitute for eighth grade data. Achievement data came from the 1996, 1997, or 1998 educational assessments. Thirty-eight participating jurisdictions in the 2000 state NAEP assessment submitted school achievement results. Of these 38 jurisdictions: • • • 30 jurisdictions used school achievement results for stratification for at least one grade. 29 jurisdictions used school achievement results for stratification of their fourth-grade schools. 28 jurisdictions used school achievement results for stratification of their eighth-grade schools.

59

Implicit Stratification Variables for Jurisdictions from the 2000 State Assessment
The table below shows whether the 2000 state assessment's participating jurisdictions and grades used school achievement data or median household income as their implicit stratification variable. The table includes only those 41 states that required stratification. It does not include any jurisdiction that required the sampling of all of its schools. District of Columbia, Guam, Virgin Islands, and the Department of Defense schools are the jurisdictions that required the sampling of all schools. Note that Hawaii and Rhode Island required sampling of all eighth-grade schools, while Iowa did not participate in the eighth grade state assessment. Alaska, Colorado, Delaware, Florida, New Hampshire, New Jersey, Pennsylvania, South Dakota, and Washington did not participate in the NAEP 2000 state assessment.
Implicit stratification variables for state assessment, by grade and state: 2000 Fourth grade School achievement data * * — * * * * * * * — * * — * * * * — * * — — * * — * * — Median household income — — * — — — — — — — * — — * — — — — * — — * * — — * — — * Eighth grade School achievement data * — — * * * — * * * — * * — * * * * — * * * — — * — * * — Median household income — — * — — — — — — — — — — * — — — — * — — — * * — * — — *

State Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio

See notes at end of table.
60

Implicit stratification variables for state assessment, by state and grade: 2000 (continued) Fourth grade School achievement data * * — * * — — * * * * — Median household income — — * — — * * — — — — * Eighth grade School achievement data * * — * * — — — * * * — Median household income — — — — — * * — — — — *

State Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming

* This variable was used for implicit stratification. — This variable was not used for implicit stratification. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

61

Median Household Income for the 2000 State Assessment
For each combination of jurisdiction and grade with neither available nor usable achievement data, median household income served as the implicit school stratification variable. A school's median household income resulted from matching its ZIP code against a geographic-based median household income file derived by Donnelly Marketing Analytical Services from 1990 Census data. Any school still missing an income value after the match received the average median household income calculated for the three-digit ZIP code prefix or county of its location.

62

School Sample Selection for the 2000 State Assessment
A systematic random sample selects grade-eligible schools from each participating jurisdiction's stratified frame of public schools. The school sample design uses a selection probability proportional to the estimated grade-specific enrollment of the schools. As a result, large schools are more likely to be selected than small schools. Each cooperating sampled school administers one or more assessment sessions. The number of sessions administered depends on the school's estimated grade-specific enrollment. Numerous features highlight the detailed school sample selection process: • • • • • • • • estimated grade enrollment and measure of size, district size and school size, school sorting and sample selection, sparse state sample option, selection of schools in small jurisdictions, control of overlap of school samples for national educational studies, selection of new public schools, and school substitution and retrofitting.

63

Estimated Grade Enrollment and Measure of Size for the 2000 State Assessment
A school's estimated grade enrollment approximates the number of students enrolled in the assessed grade (either fourth or eighth grade). The measure of size standardizes a school's estimated grade enrollment for calculating the school sampling probability. Dividing a school's total student enrollment by its number of grades results in its estimated grade enrollment. Schools with missing total student enrollment values receive estimated grade enrollments equal to 20 students. The value 20 was chosen because it leads to a measure of size that yields a selfweighting sample of students within the school. A school's measure of size depends on its estimated grade enrollment. The purpose of this measure is twofold. First, selecting schools with probability proportional to size (PPS) sampling reduces the variances in NAEP. Second, sampling very small schools less frequently than their size would dictate reduces the cost per student of data collection. The table below provides the formula to calculate a school's measure of size.
Calculation of a school's measure of size, NAEP state assessment: 2000 Estimated grade enrollment (EGE) EGE < 10 10 = EGE < 20 20 = EGE < 65 EGE = 65 EGE unknown Measure of size 30 3 x EGE 60 EGE 60

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

64

District Size and School Size for the 2000 State Assessment
District size and school size are two stratification variables used in the systematic sorting of the state NAEP school sampling frame. The classification of schools by district size and school size occurs before school sample selection. Both district size and school size have two possible values: "small" and "large". Small school districts contain less than 20 percent of the jurisdiction's student population, while large school districts contain 20 percent or more of the jurisdiction's student population. The small school size class represents all schools with an estimated grade enrollment of less than 25 students, while the large school size class represents all schools with an estimated grade enrollment of 25 students or more.

65

School Sorting and Sample Selection in the 2000 State Assessment
The state NAEP school sampling frame undergoes a systematic sorting before school sampling begins. This sorting ensures that the sampled schools represent a variety of population subgroups. The sorting variables, listed in their hierarchical sort order, are as follows: • • • • • small or large district status, small or large school size class, urbanization classification, minority enrollment classification, and achievement data or median household income.

For each participating jurisdiction, a sample of schools comes from the sorted school sampling frame, and each school's probability of selection is proportional to its measure of size. The table below shows the mean, minimum, and maximum number of schools sampled by grade and school type within the 2000 state assessment's participating jurisdictions.
Mean, minimum, and maximum number of schools sampled, by grade, state assessment: 2000 Mean number of sampled schools 112 99 Minimum number of sampled schools 22 7 Maximum number of sampled schools 209 150

Schools Fourth-grade schools Eighth-grade schools

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

66

Sparse State Sample Option for the 2000 State Assessment
Jurisdictions with student populations largely concentrated in small schools find standard NAEP sample design requirements burdensome. These jurisdictions have to select large numbers of schools to reach the required student sample sizes. They bear a large burden in school recruitment and assessment administration, but they do not qualify for reductions in student sample sizes. The Sparse State Sample Option addresses these problems while maintaining adequate sampling standards for representation and precision. Any participating jurisdiction with at least 120 schools selected in the full school sample qualifies for the Sparse State Sample Option. This option selects a proportional sample of schools that reduces both school and student sample sizes. The new sample meets the following criteria: • • • • The number of selected schools equals at least 115; The number of selected schools for each assessment subject equals at least 80; The sampling probability of each school equals at least half of its probability in the full sample; and The sample retains the jurisdiction's largest schools and their student sample sizes.

The last two conditions imply retaining all of a jurisdiction's large schools and at least half of its small schools. In practice, jurisdictions exercising this option see their school samples drop to 115 schools. Student samples within the schools retain at least half, and often more, of their original sizes.

67

Selection of Schools in Small Jurisdictions in the 2000 State Assessment
The table below contains the participating jurisdictions that had all schools selected in the 2000 state assessment by grade. The symbol § indicates that the jurisdiction had all schools selected for the state assessment for the indicated grade, while the symbol ~ indicates that the jurisdiction did not have all schools selected for the state assessment for the indicated grade. The table only contains those participating jurisdictions that had at least one grade where all schools were selected.
Jurisdictions with all schools selected, state assessment: By jurisdiction, 2000 Jurisdiction Hawaii Rhode Island District of Columbia DDESS1 DoDDS2 Guam Virgin Islands Grade 4 ~ ~ § § § § § Grade 8 § § § § § § §

~ The jurisdiction did not have all schools selected for the state assessment for the indicated grade. § The jurisdiction had all schools selected for the state assessment for the indicated grade. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

68

Effect of Sparse State Sample Option on Jurisdictions in the 2000 State Assessment
Eligible jurisdictions for the Sparse State Sample Option in the 2000 state assessment included: • • Grade 4: Alaska, Arkansas, Colorado, Idaho, Illinois, Iowa, Kansas, Kentucky, Maine, Michigan, Minnesota, Missouri, Montana, Nebraska, New Hampshire, New Mexico, North Dakota, Ohio, Oklahoma, Oregon, South Dakota, Vermont, Washington, West Virginia, Wisconsin, Wyoming. Grade 8: Alaska, Kansas, Minnesota, Montana, Nebraska, North Dakota, Oklahoma, South Dakota.

Of the 26 jurisdictions eligible for the option among fourth-grade schools, 14 jurisdictions exercised the option. Of the eight jurisdictions eligible for the option among eighth-grade schools, four jurisdictions exercised the option. Note that Alaska requested the Sparse State Sample Option for the 2000 state assessment, but later declined to participate in the assessment. The table below shows the effects of the Sparse State Sample Option on sample sizes in those 2000 state assessment jurisdictions that exercised the option.
Effect of sparse state sample option on jurisdictions, by grade, state assessment: By jurisdiction, 2000 Reduced student sample as percentage of original student sample 88 94 77 95 87 91 57 54 94 95 90 69 88 67 93 91 91 81

Grade 4

Jurisdiction Idaho Illinois Kansas Michigan Minnesota Missouri Montana Nebraska New Mexico Ohio Oregon Vermont Wisconsin Wyoming Kansas Minnesota Montana Nebraska

Original school sample 131 123 149 121 132 126 199 213 123 121 128 173 130 175 124 126 133 147

Reduced school sample 115 115 115 115 115 115 115 115 115 115 115 115 115 115 115 115 115 115

8

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

69

Control of Overlap of School Samples for National Educational Studies in the 2000 State Assessment
A comparison of joint national main and state assessments usually finds an overlap of sampled schools. Schools selected for both assessments spend a fair amount of time preparing and administering two sets of educational tests. Concern about school sample overlap arises each assessment cycle. NAEP has a policy of avoiding national and state sample overlap where possible. This policy avoids excess burden on individual schools without distorting the resulting school samples with either bias or substantial variance. To control overlap between state and national main samples, a procedure is used that conditions on the national NAEP primary sampling unit (PSU) sample. National school selection probabilities (for the national main NAEP assessment) that are conditional on the selection of national main sample PSUs are used in determining state assessment school selection probabilities. Participating jurisdictions not containing national main PSUs do not have their state assessment school selection probabilities adjusted. The overlap control procedure reduces the variance of the state samples, although it leads to a greater degree of sample overlap than using unconditional national selection probabilities. However, this degree of sample overlap is less than would be achieved without any overlap control at all. The procedure additionally recognizes the impact of within-PSU sampling in noncertainty PSUs in some jurisdictions. Schools necessary for the state sample are not subject to overlap control; such schools are selfrepresenting in the state sample. Excluding such schools on a random basis adds undue variance to the state estimates. The state assessment's conditional selection probabilities generated by the overlap control procedure tend to be smaller than the unconditional state selection probabilities for schools selected for the national main sample. Unconditional probabilities obtained by integrating over the national sampling process equal the required state assessment probabilities without any overlap control implementation. Such probabilities mean that a school's unconditional probability of state assessment selection stays the same regardless of overlap control implementation. More information about the sample overlap procedures appears in the Technical Report of the NAEP 1994 Trial State Assessment Program in Reading (Mazzeo, Allen, and Kline 1995).

70

Number of Schools Selected for Both the State and National Main Assessments in 2000
The table below lists the number of schools selected for both the 2000 state and national main assessments. Note that for the 2000 national main assessment, 573 non-new schools were selected at the fourth-grade level, 480 at the eight-grade level, and 319 at the twelfth-grade level.
Number of schools selected for both state and national main assessments: 2000 State assessment schools Fourth-grade schools Eighth-grade schools National main assessment schools Fourth-grade Eighth-grade Twelfth-grade schools schools schools 5 6 4 34 3 4

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

71

Selection of New Public Schools for the 2000 State Assessment
A supplemental sample of new public schools reflects additions to the target population that occur after the creation of the sampling frame. New public schools include: • • recently opened schools not listed on the original sampling frame; and schools existing on the sampling frame that changed grade spans to include an eligible grade.

Because public schools are organized into districts, and district personnel (not school personnel) will be knowledgeable about the existence of new schools, a sample of school districts is selected in order to compile lists of new schools. A school district level file constructed from the Common Core of Data (CCD) public school file forms the district sampling frame. The CCD public school file, maintained by the National Center for Education Statistics, contains information about elementary and secondary schools in the 50 states, District of Columbia, and outlying areas. The district sampling frame separates school districts into two groups: • • Small districts: Districts that contain at most three schools and no more than one fourth-grade, one eighth-grade, and one twelfth-grade school; and Medium/large districts: Districts that contain four or more schools, two or more fourth-grade schools, two or more eighth-grade schools, or two or more twelfth-grade schools.

The selection of new schools excludes small school districts. School recruitment during the state assessment identifies any new schools in the small school districts. The district sampling frame yields a sample of school districts to contact regarding the existence of new schools. The school district sample includes all school districts from those participating jurisdictions that sample all eligible schools. For each of the other participating jurisdictions, the school district sample includes a combination of 10 medium and large school districts. To create this school district sample for a participating jurisdiction, the medium and large school districts in the jurisdiction are first sorted by student enrollment, from the smallest enrollment to the largest enrollment. The student enrollments of the sorted school districts are summed, starting with the district with the smallest school enrollment. Summing is stopped at the school district that causes the cumulative student enrollment to exceed 20 percent of the jurisdiction's student enrollment. All school districts in the sorted list above (but not including) that school district forms the smaller district sampling group, while all other school districts form the larger district sampling group. Two school districts are selected from the group of smaller school districts with equal probability. Eight school districts are selected from the group of larger school districts with probability proportional to student enrollment. For jurisdictions with less than eight school districts in its larger district group, all school districts from the larger district group are selected, as well as enough districts from the smaller district group to make a total sample of 10 districts. Each selected school district receives a list of its public schools that appear on the original school sampling frame. These districts, in return, provide information about either new schools or new gradeeligible schools not included on the lists. These new school data form the sampling frame for the selection of new schools.

72

From each participating jurisdiction's new school data comes a sample of new schools. The following formula gives the probability of the selection of a school within a jurisdiction:

⎡ sampling rate × measure of size ⎤ minimum ⎢ , 1⎥ P(district) ⎣ ⎦
where • • • sampling rate equals the sampling rate used for the particular jurisdiction in the original school selection sample; measure of size equals a standardized measure of the school's estimated grade enrollment (see Estimated Grade Enrollment and Measure of Size for more information); and P(district) equals the probability of selection of a school district within the particular jurisdiction.

Newly grade-eligible schools within the new school sample keep the measure of size calculated during the original school sample. The new school sample includes two additional groups of schools. First, it adds all new and newly gradeeligible schools found during the field period for medium and large school districts in participating jurisdictions that sample all eligible schools. Second, it adds all new and newly grade-eligible schools coming from small school districts that have a school selected in the original sample.

73

Public School Sampling Frame and New-School Sample Sizes
The table below shows the number of new schools from participating jurisdictions for the 2000 state assessment. Jurisdictions not listed in these tables did not have any new schools selected as part of the sampling procedure.
Public school sampling frame and new-school sample sizes, grades 4 and 8: By jurisdiction, 2000 Fourth-grade Eighth-grade Common Common Core Core of Data Fourth-grade Total fourthof Data Eighth-grade Total eighth(CCD) school new school grade school (CCD) school new school grade school sample sample sample sample sample sample 5,220 117 116 123 111 114 107 111 115 115 117 141 103 125 117 156 110 116 114 115 116 115 115 115 113 115 109 112 207 115 137 116 114 110 116 113 111 113 112 62 3 0 1 1 1 2 0 0 0 0 0 3 0 4 0 1 0 0 1 2 2 0 0 4 0 0 0 2 1 0 0 6 0 0 1 1 0 1 5,280 120 120 120 110 120 110 110 120 120 120 140 110 130 120 160 110 120 110 120 120 120 120 120 120 120 110 110 210 120 140 120 120 110 120 110 110 110 110 4,429 114 114 113 110 108 107 59 95 117 108 † 105 115 114 113 107 107 113 115 112 119 115 115 65 100 108 110 150 109 127 115 52 106 112 115 101 107 109 64 4 0 0 2 1 2 2 2 0 0 † 0 0 2 1 2 0 2 1 2 2 0 0 3 0 1 1 0 1 0 0 3 0 0 0 0 0 0 4,490 120 110 110 110 110 110 60 100 120 110 † 110 120 120 110 110 110 120 120 110 120 120 120 70 100 110 110 150 110 130 120 60 110 110 120 100 110 110

Jurisdiction Total Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia See notes at end of table.

74

Public school sampling frame and new-school sample sizes, grades 4 and 8: By jurisdiction, 2000 Fourth-grade Eighth-grade Common Common Core Core of Data Fourth-grade Total fourthof Data Eighth-grade Total eighth(CCD) school new school grade school (CCD) school new school grade school sample sample sample sample sample sample 157 115 115 22 106 40 96 23 24 2 1 2 0 15 0 1 4 0 160 120 120 20 120 40 100 30 20 111 113 83 22 32 14 59 7 7 2 0 12 1 13 0 1 1 0 110 110 100 20 50 10 60 10 10

Jurisdiction West Virginia Wisconsin Wyoming Other jurisdictions American Samoa District of Columbia DDESS1 DoDDS2 Guam Virgin Islands

† Iowa did not participate in the state assessment at grade 8. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). NOTE: Nonparticipating jurisdictions are not included in this table. These jurisdictions are Alaska, Colorado, Delaware, Florida, New Jersey, New Hampshire, Pennsylvania, South Dakota, and Washington. The table includes a new school from a small district that was discovered during the data collection period (Maine grade 8). Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Assessment.

75

School Substitution and Retrofitting in the 2000 State Assessment
Substitution for sampled schools that do not participate in the assessment provides a means to reduce the nonresponse bias that may result from school nonparticipation, and to maintain an acceptable yield of assessed students. Substitution cannot eliminate school nonresponse bias. However, to the extent that a nonparticipating school can be replaced in the sample by a school, not originally selected in the sample, that has similar characteristics, nonresponse bias is likely to be reduced. Thus the 2000 NAEP state assessment included a procedure for associating a potential substitute school with each school in the sample, where feasible. An automated substitute selection mechanism assigns each sampled school a substitute school, if possible, before the field period begins. Schools and their substitutes share the same jurisdiction and urbanization classification. If possible, substitutes for public schools come from public schools in other school districts. This different district strategy arises from the fact that public school nonresponse may be due to district-level refusal. A distance measure between a sampled school and each potential substitute school determines how well the schools match each other. This measure incorporates differences between the paired schools among four component variables: • • • • Estimated grade enrollment; Achievement data or median household income; Largest student minority enrollment; and Second largest student minority enrollment.

Each difference was squared and standardized to the population standard deviation for the component variable across all grade-eligible schools and jurisdictions. The distance measure equals the sum of the four resulting squared standardized differences. For example, compare the following two schools from the NAEP 2000 state assessment: School A: • • • • Estimated grade enrollment = 49 Achievement score = 221 Largest minority enrollment percentage = 29 Second largest minority enrollment percentage = 24

School B: • • • • Estimated grade enrollment = 49 Achievement score = 223 Largest minority enrollment percentage = 25 Second largest minority enrollment percentage = 21

Using the population standard deviation calculated for each component variable in the NAEP 2000 assessment, the distance measure between schools A and B equals 0.40. School B would be assigned as a substitute school for school A if no other school had a smaller distance measure. The automated substitute selection mechanism chooses the potential substitute with the smallest distance measure as the substitute for a given school. The mechanism assigns only one substitute school to a sampled school. Once a school has been assigned as a substitute, it can not be used again. For the first pass, the maximum acceptable distance is 0.60, and public school substitutes must come from a different school district than the sampled school. Some sampled schools do not receive an assigned
76

substitute school in the first pass. Either the sampled schools outnumber the substitute schools, or the distance measure exceeds 0.60 for the remaining substitute schools. For the second pass, the mechanism raises the maximum acceptable distance measure to 0.75. It also lifts the different district constraint for public schools. These steps produce a small number of additional assigned substitutes. Note the selected distance measure cutoff points of 0.60 and 0.75 started with the 1994 Trial State Assessment. Before that assessment, statisticians reviewed a large number of school listings, calculated the distance measures between school pairs, and agreed on the cutoff points at which substitute schools appear unacceptable. The selected cutoff points have been used since the 1994 assessment. More information can be found in Technical Report of the NAEP 1994 Trial State Assessment Program in Reading (Mazzeo, Allen, and Kline 1995). When substitutes were assigned, it was sometimes the case that there was just a single school that was a suitable substitute for two sampled schools. Thus the sampled school that had the smaller distance measure to that potential substitute (school A) received it as an assigned substitute, while the second sampled school (school B) received no substitute. Once the field period began, on occasion it transpired that school A participated, while school B did not. In a case such as this, the substitute school was reassigned, and became a substitute for school B. In this way substitute schools were assigned to as many sampled schools as possible.

77

Number of Substitute Schools in the 2000 State Assessment
The links to the right connect the user to tables that provide the number of original and participating schools in the 2000 state assessment samples, together with weighted school participation rates, within each jurisdiction by grade and assessment subject. These tables also show the number of substitute schools in each jurisdiction associated with nonparticipating original schools and the number of those substitute schools that participated. The table below includes information about the number of substitute schools provided in each participating 2000 state assessment jurisdiction by grade. In jurisdictions where all of the eligible schools were included in the original sample, there were no schools available to serve as substitutes. This is indicated by a zero entry in the table.
Number of substitute schools provided in each jurisdiction by grade, state assessment: By jurisdiction, 2000 Jurisdiction Total Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon See notes at end of table.
78

Grade 4 4,454 116 111 106 112 111 109 53 105 112 117 137 113 124 115 127 110 114 113 108 102 116 108 111 89 92 107 110 92 112 132 115

Grade 8 3,072 96 77 73 110 65 102 0 44 112 99 — 87 97 96 69 92 98 113 95 76 109 76 78 12 21 94 105 70 104 106 90

Number of substitute schools provided in each jurisdiction by grade, state assessment: By jurisdiction, 2000 (continued) Jurisdiction Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa District of Columbia DDESS1 DoDDS2 Guam Virgin Islands Grade 4 66 101 114 112 112 94 110 149 116 81 0 0 0 0 0 0 Grade 8 0 81 99 110 20 25 91 61 106 13 0 0 0 0 0 0

— Not available; Iowa did not participate in the state assessment at grade 8. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 State Assessment.

79

Distribution of school sample, grade 4 mathematics state assessment: By jurisdiction, 2000 Weighted percent of school participation Before After substitution substitution 87.3 87.5 86.7 75.9 100.0 99.1 99.1 74.0 73.7 71.0 70.5 71.3 91.5 100.0 86.0 100.0 99.0 72.5 82.7 98.1 96.4 75.2 97.3 100.0 93.0 71.4 100.0 88.4 81.9 94.4 87.5 86.7 75.9 100.0 99.1 99.1 74.6 73.7 71.0 70.5 71.3 94.4 100.0 86.0 100.0 99.0 84.8 82.7 98.1 96.4 77.1 97.3 100.0 93.0 71.4 100.0 88.4 81.9 Number of schools in the original sample Total Not eligible 117 111 117 110 112 109 109 105 109 114 129 106 115 117 129 110 113 107 102 115 108 87 87 112 109 106 110 154 111 3 3 3 3 6 1 0 4 5 3 2 3 5 8 1 1 7 7 9 5 3 5 6 3 2 1 3 6 6 Participated 100 95 99 81 106 107 108 76 78 80 90 79 101 109 108 109 105 73 77 108 101 59 79 109 100 76 107 131 86 Number of substitute schools for nonparticipating originals Provided 14 12 12 26 0 1 1 22 25 31 36 23 9 0 19 0 1 27 16 0 4 20 2 0 6 28 0 3 18 Participated 8 0 0 0 0 0 0 1 0 0 0 0 3 0 0 0 0 12 0 0 0 2 0 0 0 0 0 0 0 Total participated 108 95 99 81 106 107 108 77 78 80 90 79 104 109 108 109 105 85 77 108 101 61 79 109 100 76 107 131 86

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio

See notes at end of table.

80

Distribution of school sample, grade 4 mathematics state assessment: By jurisdiction, 2000 (continued) Weighted percent of school participation Before After substitution substitution 100.0 72.9 100.0 96.6 96.7 96.9 100.0 70.3 100.0 100.0 66.6 100.0 100.0 99.3 100.0 100.0 96.6 100.0 100.0 74.0 100.0 96.6 96.7 99.0 100.0 70.3 100.0 100.0 68.8 100.0 100.0 99.3 100.0 100.0 96.6 100.0 Number of schools in the original sample Total Not eligible 116 106 119 108 112 110 109 88 109 134 107 98 16 121 40 87 27 23 2 2 7 0 4 8 0 1 3 11 2 4 0 10 0 1 2 0 Participated 114 77 112 104 104 99 109 61 106 123 68 94 16 110 40 86 25 23 Number of substitute schools for nonparticipating originals Provided 0 26 0 3 4 3 0 20 0 0 37 0 0 0 0 0 0 0 Participated 0 1 0 0 0 2 0 0 0 0 2 0 0 0 0 0 0 0 Total participated 114 78 112 104 104 101 109 61 106 123 70 94 16 110 40 86 25 23

Jurisdiction Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa District of Columbia DDESS1 DoDDS2 Guam Virgin Islands
1 2

Department of Defense Domestic Dependent Elementary and Secondary Schools. Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Mathematics Assessment.

81

Distribution of school sample, grade 8 mathematics state assessment: By jurisdiction, 2000 Weighted percent of school participation Before After substitution substitution 81.8 76.1 87.4 71.5 99.1 99.0 91.2 78.4 74.8 72.8 70.6 94.5 100.0 83.0 98.1 99.0 71.1 73.8 97.9 92.3 73.9 98.5 100.0 91.0 69.9 99.0 90.5 90.7 91.3 76.1 87.4 71.5 99.1 99.0 91.2 78.4 74.8 72.8 70.6 95.5 100.0 84.4 98.1 99.0 80.7 73.8 97.9 94.2 75.0 98.5 100.0 91.0 69.9 99.0 90.5 90.7 Number of schools in the original sample Total Not eligible 115 108 110 109 108 109 59 91 111 107 105 110 112 102 108 107 110 104 110 115 92 92 65 95 109 108 108 108 3 4 3 3 4 6 6 9 8 2 1 8 8 2 1 7 7 17 7 5 2 8 2 4 3 3 1 12 Participated 91 79 94 76 104 102 51 66 78 76 74 96 104 82 105 99 75 64 101 102 64 83 63 83 74 104 95 87 Number of substitute schools for nonparticipating originals Provided 20 19 10 29 0 1 0 2 25 27 17 5 0 9 1 1 28 15 1 6 16 0 0 0 29 1 3 7 Participated 11 0 0 0 0 0 0 0 0 0 0 1 0 2 0 0 10 0 0 2 1 0 0 0 0 0 0 0 Total participated 102 79 94 76 104 102 51 66 78 76 74 97 104 84 105 99 85 64 101 104 65 83 63 83 74 104 95 87

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio

See notes at end of table.

82

Distribution of school sample, grade 8 mathematics state assessment: By jurisdiction, 2000 (continued) Weighted percent of school participation Before After substitution substitution 99.4 74.7 100.0 91.2 88.8 92.9 100.0 81.9 100.0 100.0 65.3 100.0 100.0 100.0 100.0 100.0 100.0 100.0 99.4 74.7 100.0 92.2 90.8 95.7 100.0 81.9 100.0 100.0 73.4 100.0 100.0 100.0 100.0 100.0 100.0 100.0 Number of schools in the original sample Total Not eligible 115 111 54 106 107 111 99 92 107 108 110 81 15 45 13 54 8 6 1 2 3 3 2 2 3 0 2 4 1 10 1 10 3 1 0 Participated 113 81 51 94 93 101 96 76 105 104 70 71 14 34 13 51 7 6 Number of substitute schools for nonparticipating originals Provided 1 20 0 4 11 8 0 1 0 0 36 0 0 0 0 0 0 0 Participated 0 0 0 1 2 3 0 0 0 0 9 0 0 0 0 0 0 0 Total participated 113 81 51 95 95 104 96 76 105 104 79 71 14 34 13 51 7 6

Jurisdiction Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa District of Columbia DDESS1 DoDDS2 Guam Virgin Islands
1 2

Department of Defense Domestic Dependent Elementary and Secondary Schools. Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Mathematics Assessment.

83

Distribution of school sample, grade 4 science state assessment: By jurisdiction, 2000 Weighted percent of school participation Before After substitution substitution 87.0 86.8 85.1 75.9 100.0 99.1 98.4 74.9 73.1 70.0 70.6 91.6 100.0 85.5 100.0 99.0 71.0 83.0 98.0 96.5 75.9 95.7 100.0 92.9 72.2 100.0 88.9 94.4 86.8 85.1 75.9 100.0 99.1 98.4 74.9 73.1 70.0 70.6 94.4 100.0 85.5 100.0 99.0 83.5 83.0 98.0 96.5 77.5 95.7 100.0 92.9 72.2 100.0 88.9 Number of schools in the original sample Total Not eligible 117 112 113 110 113 109 108 104 108 113 129 116 116 131 111 113 106 103 114 109 89 87 112 107 109 110 150 2 2 3 3 6 1 0 1 5 3 4 5 8 4 1 6 7 9 6 2 1 11 3 2 1 2 4 Participated 100 95 93 81 107 107 106 78 77 78 89 102 108 107 110 106 71 78 106 103 66 73 109 98 79 108 129 Number of substitute schools for nonparticipating originals Provided 15 14 13 26 0 1 1 22 25 32 35 9 0 19 0 1 28 16 0 4 19 3 0 6 28 0 5 Participated 9 0 0 0 0 0 0 0 0 0 0 3 0 0 0 0 12 0 0 0 1 0 0 0 0 0 0 Total participated 109 95 93 81 107 107 106 78 77 78 89 105 108 107 110 106 83 78 106 103 67 73 109 98 79 108 129

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota

See notes at end of table.

84

Distribution of school sample, grade 4 science state assessment: By jurisdiction, 2000 (continued) Weighted percent of school participation Before After substitution substitution 81.7 98.6 73.1 100.0 96.6 96.7 96.9 100.0 74.7 100.0 100.0 65.2 100.0 100.0 100.0 100.0 95.6 100.0 81.7 98.6 74.2 100.0 96.6 96.7 99.0 100.0 74.7 100.0 100.0 67.3 100.0 100.0 100.0 100.0 95.6 100.0 Number of schools in the original sample Total Not eligible 111 123 108 117 107 112 111 110 88 111 134 108 98 17 39 85 26 22 7 1 3 7 0 3 10 0 0 3 8 2 5 0 0 1 2 0 Participated 85 120 78 110 103 105 98 110 66 108 126 67 93 17 39 84 23 22 Number of substitute schools for nonparticipating originals Provided 17 2 26 0 3 4 3 0 16 0 0 39 0 0 0 0 0 0 Participated 0 0 1 0 0 0 2 0 0 0 0 2 0 0 0 0 0 0 Total participated 85 120 79 110 103 105 100 110 66 108 126 69 93 17 39 84 23 22

Jurisdiction Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa DDESS1 DoDDS2 Guam Virgin Islands
1 2

Department of Defense Domestic Dependent Elementary and Secondary Schools. Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

85

Distribution of school sample, grade 8 science state assessment: By jurisdiction, 2000 Weighted percent of school participation Before After substitution substitution 82.3 75.9 87.2 72.1 100.0 99.0 91.2 78.0 75.0 73.0 94.4 100.0 83.4 97.2 99.0 71.5 73.3 97.9 92.3 73.3 98.4 100.0 90.6 70.7 98.1 90.7 90.7 100.0 91.8 75.9 87.2 72.1 100.0 99.0 91.2 78.0 75.0 73.0 95.5 100.0 84.8 97.2 99.0 81.0 73.3 97.9 94.2 74.4 98.4 100.0 91.3 70.7 98.1 90.7 90.7 100.0 Number of schools in the original sample Total Not eligible 115 109 108 110 107 108 58 87 111 106 108 112 103 107 106 109 103 111 113 91 94 65 96 108 109 106 108 115 4 3 3 5 3 5 6 8 6 2 7 8 1 1 6 5 22 8 3 4 5 1 3 3 4 1 11 1 Participated 91 80 92 76 104 102 50 63 80 76 95 104 84 103 99 76 59 101 102 61 87 64 84 74 103 93 88 114 Number of substitute schools for nonparticipating originals Provided 19 20 10 28 0 1 0 2 25 26 5 0 9 2 1 28 15 1 6 15 0 0 1 28 2 4 7 0 Participated 11 0 0 0 0 0 0 0 0 0 1 0 2 0 0 10 0 0 2 1 0 0 1 0 0 0 0 0 Total participated 102 80 92 76 104 102 50 63 80 76 96 104 86 103 99 86 59 101 104 62 87 64 85 74 103 93 88 114

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma

See notes at end of table.

86

Distribution of school sample, grade 8 science state assessment: By jurisdiction, 2000 (continued) Weighted percent of school participation Before After substitution substitution 73.9 100.0 91.2 89.6 91.4 100.0 80.2 100.0 100.0 66.3 100.0 95.7 100.0 100.0 100.0 100.0 73.9 100.0 92.2 91.6 94.3 100.0 80.2 100.0 100.0 74.5 100.0 95.7 100.0 100.0 100.0 100.0 Number of schools in the original sample Total Not eligible 109 54 105 109 110 98 92 108 108 109 74 18 14 55 8 7 2 2 2 3 3 3 0 3 6 1 10 1 0 2 1 0 Participated 78 52 94 95 97 95 74 105 102 71 64 16 14 53 7 7 Number of substitute schools for nonparticipating originals Provided 21 0 4 10 10 0 2 0 0 34 0 0 0 0 0 0 Participated 0 0 1 2 3 0 0 0 0 9 0 0 0 0 0 0 Total participated 78 52 95 97 100 95 74 105 102 80 64 16 14 53 7 7

Jurisdiction Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa DDESS1 DoDDS2 Guam Virgin Islands
1 2

Department of Defense Domestic Dependent Elementary and Secondary Schools. Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

87

Assignment of Subject, Sample Type, Monitor Status, and Drop Flag in the 2000 State Assessment
Each sampled school holds one or more assessment sessions. Determining what assessment sessions each school administers depends on four session characteristics: • • • • Assessment subject indicates the tested academic subject for the session. These subjects vary by grade and NAEP assessment. Sample type indicates whether or not the session allows accommodations for students with disabilities or limited English proficiency. Monitor status indicates whether or not a NAEP representative monitors the assessment session. A drop flag indicates which assessment subject to drop if a school assigned assessment sessions in both subjects can only handle a single session.

88

Assessment Subject in the 2000 State Assessment
The 2000 state assessment focused on the students' knowledge of mathematics and science. All schools with 25 or more eligible students participated in both mathematics and science assessments. Students from such schools participated in either the mathematics assessment or the science assessment, but not both assessments. All eighth-grade schools with fewer than 25 eligible students participated in a randomly selected assessment.

89

Sample Type in the 2000 State Assessment
Each state assessment uses the inclusion rules (established in 1996) for students with disabilities (SD) and limited-English-proficient (LEP) students. Sample type denotes whether or not a session may allow such accommodations. Sample type 2 (S2) provides no accommodations for SD/LEP students, while the sample type 3 (S3) allows SD/LEP accommodations. For an assessment subject that allows accommodations, sample type assignment begins by considering all schools assigned an assessment session in the subject. After sorting the schools by jurisdiction, sampled grade, and assessment subject, the assignment of sample type (S2 or S3) to the schools occurs in an alternating pattern. For the 2000 state assessment, both the mathematics and science subjects allowed SD, LEP, and SD/LEP accommodations. The 2000 state assessment student counts by sample type are provided on the web pages Participating Fourth-Grade School Counts and Participating Eighth-Grade School Counts.

90

Monitor Status in the 2000 State Assessment
A random subsample of selected schools determines which schools NAEP representatives monitor during the assessment field period. This subsample enables reliable comparisons between sessions administered with and without monitoring. The monitoring rate for each participating jurisdiction depends whether or not the jurisdiction took part in an earlier state assessment. Jurisdictions participating in earlier state assessments have a 25 percent school monitoring rate, while jurisdictions new to state assessments have a 50 percent school monitoring rate. Monitoring assignment starts first by sorting all sampled schools in the following order: • • • • jurisdiction; assessment subject; sample type (S2 or S3); and school identification number.

A systematic sample, determined by the jurisdiction's school monitoring rate, assigns monitoring status (monitored or not monitored) to the sorted schools. For the 2000 state assessment, American Samoa had a 50 percent school monitoring rate. All other participating jurisdictions had a 25 percent school monitoring rate.

91

Drop Flag Assignment in the 2000 State Assessment
Schools assigned sessions in both assessment subjects occasionally turn out to be smaller than expected. As students may only be assessed in one subject, these schools are only large enough to administer one assessment subject. For these schools, the drop flag indicates what assessment subject session to drop. Schools assigned only a single session ignore the drop flag. The drop flag assignment starts by sorting all sampled schools in the following order: • • • • • jurisdiction; assessment subject; number of assessment sessions (in descending order); sample type (S2 or S3); and school identification number.

An alternating pattern assigns an assessment subject to each sorted school as its drop flag.

92

Student Sample Selection in the 2000 State Assessment
Each sampled school provides a list of grade-eligible enrolled students. From each list, a systematic random sample draws 30 students (if possible) for each assessment session assigned to the school. This student sample design produces both aggregate estimates and estimates of various demographic populations of interest. Most sampled schools followed the established student sampling and participation procedures to select the students for its assessment sessions. Participating jurisdictions with multiple assessment sessions in their sampled schools could opt for a reduced sample option to shrink their student sample sizes.

93

Student Sampling and Participation for the 2000 State Assessment
A systematic random sample assigns students to a sampled school's assessment sessions from a list of grade-eligible students provided by the school. The school also keeps track of students that miss the assessment sessions, conducting makeup sessions if needed. For student sample selection, each school sends a Student Listing Form (SLF), a complete list of its fourth- or eighth-grade students, to a designated sampler. A school's form must include all grade-eligible students, although not in any particular order. The sampler enters the number of students listed on the school's Student Listing Form and other necessary information into a sampling algorithm program. The program generates a random start number, calculates the skip pattern, and systematically selects 30 sample line numbers for each session. To compensate for new enrollees not listed on the Student Listing Form, the sampling algorithm program also generates extra line numbers for a supplemental sample of new students. School administrators review student samples and identify students incapable of taking an assessment because of either disability (SD) or limited English proficiency (LEP). These inclusion criteria were revised in 1996 to communicate more clearly a presumption of inclusion except under special circumstances, thereby allowing more SD or LEP students into the assessment. After an assessment session, each school makes a count of the non-excluded students who did not attend the session. If this count exceeds three students, the school conducts a makeup session for all students absent from the initial session.

94

Weighted student participation rates, exclusion rates, and sample counts, grade 4 mathematics state assessment (accommodations not permitted [R2]): By jurisdiction, 2000 Weighted percentage student Number of Number of nonparticipation after students accommodated makeups sampled students1 assessed 95 94 95 94 96 95 94 96 94 95 95 96 95 96 95 95 96 94 94 95 95 95 94 94 95 94 95 96 95 95 93 95 96 96 96 94 95 96 95 96 95 2,700 2,500 2,600 1,900 2,900 3,100 2,900 1,900 1,900 2,100 2,100 1,700 2,600 2,900 2,400 3,100 2,600 2,200 2,000 3,200 2,700 1,200 1,600 3,000 2,300 2,000 2,900 2,700 2,200 2,700 1,800 2,900 2,800 2,800 2,600 3,000 1,300 2,800 2,800 1,700 1,900 2,400 2,100 2,300 1,700 2,500 2,700 2,400 1,700 1,600 1,900 1,900 1,600 2,300 2,500 2,100 2,600 2,300 1,900 1,800 2,800 2,300 1,100 1,400 2,500 1,900 1,800 2,400 2,500 1,900 2,300 1,600 2,400 2,500 2,500 2,200 2,600 1,200 2,400 2,400 1,500 1,700 Weighted percentage of Total Weighted number of students percentage of students identified as SD students assessed or LEP excluded 2,400 2,100 2,300 1,700 2,500 2,700 2,400 1,700 1,600 1,900 1,900 1,600 2,300 2,500 2,100 2,600 2,300 1,900 1,800 2,800 2,300 1,100 1,400 2,500 1,900 1,800 2,400 2,500 1,900 2,300 1,600 2,400 2,500 2,500 2,200 2,600 1,200 2,400 2,400 1,500 1,700 13 25 14 33 15 11 19 16 17 11 15 16 12 16 16 12 19 11 16 6 15 12 18 20 31 16 16 12 12 20 18 23 17 11 25 14 15 16 13 19 15 6 12 7 9 10 7 10 6 10 7 10 7 8 8 10 9 10 8 6 4 10 5 8 10 12 12 13 6 10 10 8 12 7 4 15 7 11 11 10 12 6

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming

See notes at end of table.

95

Weighted student participation rates, exclusion rates, and sample counts, grade 4 mathematics state assessment (accommodations not permitted [R2]): By jurisdiction, 2000 (continued) Weighted percentage student participation after makeups 94 94 95 94 95 95 Number of students sampled 500 2,700 1,700 3,300 1,200 800 Weighted Number of nonTotal percentage of Weighted accommodated number of students percentage of identified as students1 students students assessed assessed SD or LEP excluded 500 2,300 1,300 2,800 1,000 800 500 2,300 1,300 2,800 1,000 800 15 19 11 11 26 8 14 9 5 5 12 6

Jurisdiction Other jurisdictions American Samoa District of Columbia DDESS2 DoDDS3 Guam Virgin Islands
1 2

Since accommodations were not permitted, no accommodated students were assessed in this sample. Department of Defense Domestic Dependent Elementary and Secondary Schools. 3 Department of Defense Dependents Schools (Overseas). NOTE: R2 is the non-accommodated reporting sample. Sampled students that have neither a student disability (SD) nor a limited English proficiency (LEP), plus SD/LEP students from sessions in which accommodations were not allowed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Mathematics Assessment.

96

Weighted student participation rates, exclusion rates, and sample counts, grade 4 mathematics state assessment (accommodations permitted [R3]): By jurisdiction, 2000 Weighted percentage Number student of participation students after makeups sampled 95 95 96 94 96 95 94 95 94 95 95 95 95 96 94 94 96 94 94 95 95 95 95 94 95 94 96 96 95 94 94 95 96 96 96 93 95 95 95 97 95 2,800 2,400 2,600 1,900 2,800 3,100 2,900 1,900 1,900 2,100 2,200 1,800 2,600 2,900 2,400 3,000 2,600 2,100 2,000 3,200 2,700 1,200 1,600 3,100 2,400 2,100 2,800 2,700 2,100 2,700 1,900 2,800 2,800 2,800 2,600 3,000 1,400 2,900 2,800 1,700 1,900 Number of nonaccommodated students assessed 2,500 2,000 2,200 1,600 2,500 2,700 2,400 1,700 1,600 1,900 1,900 1,600 2,300 2,400 2,100 2,700 2,300 1,900 1,800 2,800 2,300 1,100 1,400 2,500 1,900 1,700 2,400 2,400 1,900 2,300 1,600 2,400 2,500 2,500 2,200 2,700 1,200 2,500 2,400 1,500 1,700 Weighted Number of Total percentage of Weighted accommodated number of students percentage of students students identified as students assessed assessed SD or LEP excluded ‡ 106 ‡ 93 ‡ ‡ ‡ 62 83 66 86 ‡ ‡ 153 77 76 132 ‡ 71 ‡ 98 ‡ ‡ 85 128 95 106 ‡ ‡ ‡ 74 147 67 ‡ 86 ‡ 75 102 111 80 ‡ 2,500 2,100 2,300 1,700 2,600 2,700 2,400 1,700 1,700 1,900 2,000 1,600 2,300 2,600 2,200 2,700 2,400 1,900 1,800 2,900 2,400 1,100 1,500 2,600 2,000 1,800 2,500 2,500 1,900 2,400 1,700 2,600 2,500 2,500 2,300 2,700 1,200 2,600 2,500 1,500 1,800 13 25 14 33 15 11 19 16 17 11 15 16 12 16 16 12 19 11 16 6 15 12 18 20 31 16 16 12 12 20 18 23 17 11 25 14 15 16 13 19 15 3 4 4 6 5 3 9 2 3 2 2 3 3 3 5 2 3 3 2 3 3 2 3 7 6 5 5 1 5 5 3 3 5 3 7 3 3 4 3 5 2

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming

See notes at end of table.

97

Weighted student participation rates, exclusion rates, and sample counts, grade 4 mathematics state assessment (accommodations permitted [R3]): By jurisdiction, 2000 (continued) Weighted Weighted percentage Number Number of nonNumber of Total percentage student of accommodated accommodated number of of students participation students students students students identified as after makeups sampled assessed assessed assessed SD or LEP 94 94 95 93 95 95 600 2,700 1,700 3,300 1,300 800 500 2,300 1,300 2,800 1,100 800 ‡ 90 ‡ ‡ ‡ ‡ 500 2,400 1,300 2,800 1,100 800 15 19 11 11 26 8 Weighted percentage of students excluded 4 5 4 2 6 4

Jurisdiction Other jurisdictions American Samoa District of Columbia DDESS1 DoDDS2 Guam Virgin Islands

‡ Reporting standards not met. Sample size is insufficient. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Mathematics Assessment.

98

Weighted student participation rates, exclusion rates, and sample counts, grade 4 science state assessment (accommodations not permitted [R2]): By jurisdiction, 2000 Weighted percentage student Number of Number of nonparticipation after students accommodated makeups sampled students1 assessed 96 93 95 94 96 95 95 95 95 95 96 95 95 95 95 95 94 95 95 95 95 94 94 94 93 95 96 93 95 94 95 96 95 96 95 95 96 95 95 95 2,800 2,600 2,500 2,000 2,900 3,100 2,800 1,900 1,800 2,000 2,100 2,600 2,800 2,400 3,100 2,600 2,100 2,100 3,100 2,700 1,300 1,500 3,000 2,300 2,100 2,900 2,600 2,200 2,800 1,900 2,800 2,800 2,800 2,500 3,000 1,400 2,800 2,900 1,600 1,900 2,500 2,100 2,200 1,700 2,500 2,600 2,400 1,700 1,600 1,800 1,900 2,200 2,500 2,100 2,600 2,300 1,900 1,900 2,800 2,400 1,200 1,300 2,500 1,900 1,800 2,400 2,300 1,900 2,400 1,600 2,400 2,400 2,500 2,100 2,700 1,200 2,500 2,500 1,400 1,700 Weighted percentage of Total Weighted number of students percentage of students identified as SD students assessed or LEP excluded 2,500 2,100 2,200 1,700 2,500 2,600 2,400 1,700 1,600 1,800 1,900 2,200 2,500 2,100 2,600 2,300 1,900 1,900 2,800 2,400 1,200 1,300 2,500 1,900 1,800 2,400 2,300 1,900 2,400 1,600 2,400 2,400 2,500 2,100 2,700 1,200 2,500 2,500 1,400 1,700 12 24 13 33 15 11 19 16 16 12 14 12 16 18 13 20 11 16 6 15 13 16 20 30 17 17 14 12 20 18 23 17 11 26 14 15 15 13 20 14 6 11 6 11 11 8 9 6 9 7 10 8 8 11 9 11 9 7 4 10 5 6 11 13 13 14 6 10 10 8 12 8 4 15 7 11 10 10 13 6

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming

See notes at end of table.

99

Weighted student participation rates, exclusion rates, and sample counts, grade 4 science state assessment (accommodations not permitted [R2]): By jurisdiction, 2000 (continued) Weighted percentage student participation after makeups 93 95 95 95 96 Number of students sampled 600 1,700 3,300 1,200 700 Number of nonaccommodated students1 assessed 500 1,300 2,800 1,000 700 Total number of students assessed 500 1,300 2,800 1,000 700 Weighted percentage of Weighted students percentage of identified as students SD or LEP excluded 17 11 11 26 7 15 7 5 10 5

Jurisdiction Other jurisdictions American Samoa DDESS2 DoDDS3 Guam Virgin Islands
1 2

Since accommodations were not permitted, no accommodated students were assessed in this sample. Department of Defense Domestic Dependent Elementary and Secondary Schools. 3 Department of Defense Dependents Schools (Overseas). NOTE: R2 is the non-accommodated reporting sample. Sampled students that have neither a student disability (SD) nor a limited English proficiency (LEP), plus SD/LEP students from sessions in which accommodations were not allowed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

100

Weighted student participation rates, exclusion rates, and sample counts, grade 4 science state assessment (accommodations permitted [R3]): By jurisdiction, 2000 Weighted percentage Number Number of nonNumber of Total student of accommodated accommodated number of participation students students students students after makeups sampled assessed assessed assessed 96 93 95 94 95 94 95 95 95 95 95 95 95 94 94 95 94 95 95 94 95 95 94 94 93 95 97 93 94 95 95 96 95 96 95 95 96 95 96 95 93 96 2,800 2,400 2,500 1,900 2,800 3,000 2,800 1,900 1,900 2,100 2,100 2,600 2,800 2,400 3,100 2,600 2,100 2,100 3,100 2,700 1,300 1,500 3,100 2,300 2,100 2,800 2,600 2,200 2,800 1,900 2,800 2,800 2,800 2,600 3,000 1,400 2,900 2,900 1,700 2,000 600 1,600 2,500 2,000 2,200 1,600 2,500 2,600 2,400 1,700 1,600 1,800 1,900 2,200 2,400 2,100 2,700 2,200 1,900 1,800 2,800 2,400 1,200 1,300 2,500 1,900 1,800 2,400 2,300 1,900 2,400 1,600 2,400 2,400 2,500 2,200 2,600 1,200 2,500 2,500 1,400 1,700 500 1,300 ‡ 76 ‡ 100 ‡ ‡ ‡ ‡ ‡ ‡ 70 64 159 106 85 128 ‡ 69 ‡ 113 ‡ ‡ 71 80 94 96 ‡ ‡ 66 64 146 ‡ ‡ 66 63 75 89 113 80 75 ‡ ‡ 2,800 2,400 2,500 1,900 2,800 3,000 2,800 1,900 1,900 2,100 2,100 2,600 2,800 2,400 3,100 2,600 2,100 2,100 3,100 2,700 1,300 1,500 3,100 2,300 2,100 2,800 2,600 2,200 2,800 1,900 2,800 2,800 2,800 2,600 3,000 1,400 2,900 2,900 1,700 2,000 600 1,600 Weighted percentage of students identified as SD or LEP 12 24 13 33 15 11 19 16 16 12 14 12 16 18 13 20 11 16 6 15 13 16 20 30 17 17 14 12 20 18 23 17 11 26 14 15 15 13 20 14 17 11 Weighted percentage of students excluded 4 6 4 5 6 3 8 2 4 3 3 4 2 4 3 4 3 3 2 1 3 5 7 6 4 6 1 4 4 4 4 5 2 8 4 3 5 3 5 1 7 5

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa DDESS1

See notes at end of table.

101

Weighted student participation rates, exclusion rates, and sample counts, grade 4 science state assessment (accommodations permitted [R3]): By jurisdiction, 2000 (continued) Weighted percentage Number student of participation after students Jurisdiction makeups sampled DoDDS2 Guam Virgin Islands 96 95 96 3,200 1,200 800 Number of nonaccommodated students assessed 2,800 1,000 700 Weighted Number of Total percentage of Weighted accommodated number of students percentage of identified as students students students assessed assessed SD or LEP excluded ‡ ‡ ‡ 3,200 1,200 800 11 26 7 2 6 4

‡ Reporting standards not met. Sample size is insufficient. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

102

Weighted student participation rates, exclusion rates, and sample counts, grade 8 mathematics state assessment (accommodations not permitted [R2]): By jurisdiction, 2000 Weighted percentage student participation after makeups 92 91 93 91 92 90 90 93 93 93 92 94 90 91 90 93 88 93 92 92 92 92 92 89 90 92 95 91 93 90 91 93 90 93 92 92 92 92 92 93 97 87 Number of nonTotal accommodated number of students1 students assessed assessed 2,300 1,800 2,200 1,600 2,500 2,500 2,300 2,000 1,700 1,900 1,700 2,300 2,400 2,100 2,400 2,300 2,000 1,500 2,400 2,300 1,700 1,900 2,600 1,900 1,600 2,400 2,200 2,100 2,400 1,800 2,300 2,300 2,200 2,300 2,500 2,000 2,500 2,500 1,800 2,600 400 1,600 2,300 1,800 2,200 1,600 2,500 2,500 2,300 2,000 1,700 1,900 1,700 2,300 2,400 2,100 2,400 2,300 2,000 1,500 2,400 2,300 1,700 1,900 2,600 1,900 1,600 2,400 2,200 2,100 2,400 1,800 2,300 2,300 2,200 2,300 2,500 2,000 2,500 2,500 1,800 2,600 400 1,600 Weighted percentage of students identified as SD or LEP 14 19 14 27 16 11 20 14 15 12 14 14 13 15 13 19 11 15 11 15 12 13 16 25 16 16 11 11 15 17 20 13 13 20 14 17 15 15 17 13 14 15

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa District of Columbia See notes at end of table.

Number of students sampled 2,700 2,200 2,600 1,900 2,900 3,100 2,800 2,300 2,000 2,100 2,000 2,700 2,900 2,500 3,000 2,700 2,400 1,700 2,900 2,800 2,000 2,200 3,200 2,400 2,000 2,900 2,500 2,500 2,900 2,100 2,800 2,700 2,600 2,800 2,900 2,400 3,000 3,000 2,100 3,000 500 2,000

Weighted percentage of students excluded 5 9 8 9 10 7 7 5 8 7 6 9 6 9 11 12 7 5 7 9 5 4 10 12 13 14 4 9 9 6 12 7 5 10 6 10 10 11 10 4 12 9

103

Weighted student participation rates, exclusion rates, and sample counts, grade 8 mathematics state assessment (accommodations not permitted [R2]): By jurisdiction, 2000 (continued) Weighted percentage student participation after makeups 92 94 92 94 Number of students sampled 900 2,300 1,200 700 Number of nonTotal accommodated number of students1 students assessed assessed 600 2,000 1,000 600 600 2,000 1,000 600 Weighted percentage of students identified as SD or LEP 13 8 13 11 Weighted percentage of students excluded 11 3 5 11

Jurisdiction DDESS2 DoDDS3 Guam Virgin Islands
1 2

Since accommodations were not permitted, no accommodated students were assessed in this sample. Department of Defense Domestic Dependent Elementary and Secondary Schools. 3 Department of Defense Dependents Schools (Overseas). NOTE: R2 is the non-accommodated reporting sample. Sampled students that have neither a student disability (SD) nor a limited English proficiency (LEP), plus SD/LEP students from sessions in which accommodations were not allowed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Mathematics Assessment.

104

Weighted student participation rates, exclusion rates, and sample counts, grade 8 mathematics state assessment (accommodations permitted [R3]): By jurisdiction, 2000 Weighted Weighted percentage Number Number of nonNumber of Total percentage student of accommodated accommodated number of of students participation students students students students identified as after makeups sampled assessed assessed assessed SD or LEP 92 91 93 92 92 90 91 93 92 92 92 94 90 91 91 93 88 92 92 93 92 91 92 89 90 92 94 91 92 91 90 93 91 93 92 92 91 91 91 93 98 88 2,700 2,200 2,600 1,900 2,900 3,000 2,700 2,300 2,000 2,100 1,900 2,700 2,900 2,500 2,900 2,700 2,400 1,800 2,900 2,800 2,000 2,200 3,200 2,400 2,000 2,900 2,500 2,400 2,900 2,100 2,900 2,700 2,600 2,800 2,900 2,300 3,000 3,000 2,100 3,000 500 2,100 2,300 1,800 2,200 1,600 2,500 2,500 2,200 2,000 1,700 1,900 1,600 2,300 2,300 2,100 2,500 2,300 2,000 1,500 2,400 2,300 1,700 1,900 2,600 1,900 1,700 2,400 2,200 2,100 2,500 1,800 2,400 2,300 2,300 2,300 2,500 2,000 2,500 2,500 1,800 2,600 400 1,600 ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ 80 ‡ ‡ 115 ‡ ‡ ‡ 88 ‡ ‡ 73 ‡ 66 93 ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ 101 68 ‡ ‡ 66 2,300 1,800 2,200 1,700 2,500 2,500 2,200 2,000 1,800 1,900 1,700 2,400 2,400 2,200 2,500 2,400 2,000 1,600 2,400 2,400 1,800 1,900 2,700 1,900 1,700 2,500 2,300 2,100 2,500 1,800 2,400 2,300 2,300 2,300 2,500 2,100 2,500 2,600 1,800 2,700 400 1,700 14 19 14 27 16 11 20 14 15 12 14 14 13 15 13 19 11 15 11 15 12 13 16 25 16 16 11 11 15 17 20 13 13 20 14 17 15 15 17 13 14 15 Weighted percentage of students excluded 6 3 2 4 6 5 5 2 5 3 3 5 3 3 3 3 4 2 5 3 2 4 4 7 4 5 2 4 4 3 3 4 2 8 3 3 6 3 4 1 4 6

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa District of Columbia

See notes at end of table.
105

Weighted student participation rates, exclusion rates, and sample counts, grade 8 mathematics state assessment (accommodations permitted [R3]): By jurisdiction, 2000 (continued) Weighted percentage Number student of participation after students Jurisdiction makeups sampled DDESS1 DoDDS2 93 Guam Virgin Islands 92 94 Number of nonaccommodated students assessed 700 2,000 ‡ 600 ‡ Weighted Number of Total percentage of Weighted accommodated number of students percentage of identified as students students students assessed assessed SD or LEP excluded ‡ ‡ 700 2,000 1,000 600 13 8 13 11 3 0 6 9

900 2,300 1,100 1,000 94 700

‡ Reporting standards not met. Sample size is insufficient. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Mathematics Assessment.

106

Weighted student participation rates, exclusion rates, and sample counts, grade 8 science state assessment (accommodations not permitted [R2]): By jurisdiction, 2000 Weighted percentage student participation after makeups 94 91 92 93 91 92 90 93 94 93 94 91 94 89 93 91 92 93 93 92 91 92 89 89 92 93 92 92 90 91 93 91 93 92 93 91 93 91 93 Number of students sampled 2,700 2,200 2,500 1,900 2,900 3,000 2,700 2,200 2,000 2,100 2,700 2,900 2,500 2,900 2,700 2,400 1,600 2,900 2,700 2,000 2,200 3,300 2,400 2,000 2,900 2,500 2,500 2,900 2,100 2,800 2,700 2,600 2,700 2,900 2,400 3,000 2,900 2,100 2,900 Number of nonTotal accommodated number of students1 students assessed assessed 2,400 1,800 2,100 1,700 2,500 2,600 2,300 2,000 1,800 1,900 2,300 2,400 2,200 2,300 2,300 2,000 1,400 2,500 2,300 1,700 1,900 2,700 1,900 1,600 2,300 2,200 2,100 2,500 1,800 2,400 2,300 2,200 2,300 2,400 2,000 2,400 2,400 1,800 2,600 2,400 1,800 2,100 1,700 2,500 2,600 2,300 2,000 1,800 1,900 2,300 2,400 2,200 2,300 2,300 2,000 1,400 2,500 2,300 1,700 1,900 2,700 1,900 1,600 2,300 2,200 2,100 2,500 1,800 2,400 2,300 2,200 2,300 2,400 2,000 2,400 2,400 1,800 2,600 Weighted percentage of students identified as SD or LEP 13 18 15 26 14 11 20 14 15 11 13 13 16 14 20 11 15 8 13 13 15 14 26 18 15 13 11 14 17 19 14 14 19 12 19 15 16 15 12 Weighted percentage of students excluded 4 9 8 9 9 7 8 5 11 6 9 5 9 10 13 8 5 5 8 6 4 9 13 14 13 4 8 8 6 10 8 6 9 6 11 10 11 9 4

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming

See notes at end of table.

107

Weighted student participation rates, exclusion rates, and sample counts, grade 8 science state assessment (accommodations not permitted [R2]): By jurisdiction, 2000 (continued) Weighted percentage student participation after makeups 97 94 94 90 90 Number of nonTotal accommodated number of students1 students assessed assessed 400 700 2,000 900 600 400 700 2,000 900 600 Weighted percentage of students identified as SD or LEP 15 16 8 17 11

Jurisdiction Other jurisdictions American Samoa DDESS2 DoDDS3 Guam Virgin Islands
1 2

Number of students sampled 500 900 2,300 1,100 700

Weighted percentage of students excluded 12 13 4 5 11

Since accommodations were not permitted, no accommodated students were assessed in this sample. Department of Defense Domestic Dependent Elementary and Secondary Schools. 3 Department of Defense Dependents Schools (Overseas). NOTE: R2 is the non-accommodated reporting sample. Sampled students that have neither a student disability (SD) nor a limited English proficiency (LEP), plus SD/LEP students from sessions in which accommodations were not allowed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

108

Weighted student participation rates, exclusion rates, and sample counts, grade 8 science state assessment (accommodations permitted [R3]): By jurisdiction, 2000 Weighted percentage Number student of participation students after makeups sampled 93 91 92 93 91 92 91 93 93 93 94 90 94 89 92 91 92 93 93 93 90 91 89 89 91 92 91 93 90 90 93 91 92 92 92 90 92 91 93 2,700 2,200 2,500 2,000 3,000 3,000 2,700 2,200 2,100 2,100 2,700 2,800 2,500 2,900 2,700 2,400 1,700 3,000 2,800 2,000 2,200 3,300 2,400 2,000 2,900 2,500 2,500 2,900 2,100 2,900 2,700 2,600 2,800 2,900 2,300 3,000 3,000 2,200 2,900 Number of nonaccommodated students assessed 2,400 1,800 2,100 1,700 2,500 2,500 2,300 2,000 1,800 1,900 2,300 2,300 2,200 2,400 2,300 2,000 1,400 2,500 2,300 1,700 1,900 2,700 1,900 1,600 2,400 2,200 2,100 2,500 1,700 2,400 2,300 2,200 2,300 2,400 2,000 2,400 2,500 1,800 2,500 Weighted Number of Total percentage of Weighted students percentage of accommodated number of students students identified as students SD or LEP assessed assessed excluded ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ 80 ‡ 101 ‡ ‡ ‡ 71 ‡ ‡ ‡ ‡ 79 74 ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ ‡ 63 70 114 ‡ ‡ 2,400 1,800 2,100 1,700 2,600 2,600 2,300 2,000 1,800 1,900 2,400 2,400 2,300 2,400 2,400 2,000 1,500 2,500 2,400 1,700 1,900 2,700 2,000 1,700 2,500 2,200 2,200 2,500 1,800 2,400 2,300 2,300 2,300 2,500 2,000 2,500 2,600 1,900 2,600 13 18 15 26 14 11 20 14 15 11 13 13 16 14 20 11 15 8 13 13 15 14 26 18 15 13 11 14 17 19 14 14 19 12 19 15 16 15 12 5 4 5 4 6 4 5 3 5 3 3 3 2 3 4 5 4 4 2 1 4 4 6 7 5 1 4 4 4 4 6 4 6 3 3 5 3 4 1

Jurisdiction Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont Virginia West Virginia Wisconsin Wyoming

See notes at end of table.

109

Weighted student participation rates, exclusion rates, and sample counts, grade 8 science state assessment (accommodations permitted [R3]): By jurisdiction, 2000 (continued) Weighted percentage Number Number of nonNumber of Total student of accommodated accommodated number of participation students students students students after makeups sampled assessed assessed assessed 97 95 94 90 89 500 900 2,300 1,100 800 500 700 2,000 900 600 ‡ ‡ ‡ ‡ ‡ 500 700 2,000 900 600 Weighted percentage of students identified as SD or LEP 15 16 8 17 11 Weighted percentage of students excluded 3 2 1 9 9

Jurisdiction Other jurisdictions American Samoa DDESS1 DoDDS2 Guam Virgin Islands

‡ Reporting standards not met. Sample size is insufficient. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

110

Reduced Sample Option
The reduced sample option adjusts the number of assessment sessions for participating schools with multiple sessions in some participating jurisdictions, easing their testing burdens. Assessment estimates obtained from jurisdictions that exercise this option, however, prove more variable than the estimates obtained without the option. A school is eligible for the reduced sample option if it meets the following criteria: • • its jurisdiction contains fewer than 100 schools; and it has more than two fourth-grade assessment sessions or three eighth-grade assessment sessions.

Under this option, participating jurisdictions reduce student sample sizes by adjusting the number of sessions within the eligible schools. The reduction is subject to the following constraints: • • • the minimum number of sessions per school equals one session; the maximum number of sessions per school equals two fourth-grade sessions or three eighthgrade sessions; and the expected student size from the reduced sample equals one-half or more of the original student sample size.

For the 2000 state assessment, no participating jurisdictions exercised the reduced sample option.

111

Participation Results for the 2000 State Assessment
School and student participation results in the 2000 assessment are reported here. The assessment collected data from samples of students where assessment accommodations for students with disabilities (SD) or limited-English-proficient (LEP) students were not permitted (R2) and samples of students where accommodations were permitted (R3). If an SD/LEP student did not meet the criteria for inclusion, he or she did not participate; all other students were asked to participate in the assessment under standard conditions. School Participation Results The links below connect you to tables that provide school counts. Participating Grade 4 School Counts Participating Grade 8 School Counts

Student Participation Results In every NAEP assessment, the student sample falls into the following groups: • • • • • • withdrawn students, excluded students with disabilities, excluded limited English proficiency students, students assessed in the initial session, students assessed in makeup sessions, and students absent from both sessions (not excluded but not assessed).

Withdrawn students are those students who have left the school before the assessment data have been collected. Excluded students are those who cannot be assessed even with an accommodation because they are disabled (SD) or have limited English proficiency (LEP). Assessed students are either assessed in the initial session, or if they were absent for that session, perhaps in the makeup session. The last category in the list includes students who were not excluded (i.e., "were to be assessed"), but were not assessed, either due to absence (from both sessions) or a refusal to participate. Assessed students are also classified as assessed without an accommodation or assessed with an accommodation. The latter group can be divided into SD students assessed with an accommodation, LEP students assessed with an accommodation, or students who are both SD and LEP and accommodated (students neither SD nor LEP can only be assessed without an accommodation). Note that some SD and LEP students are assessed without accommodations.

112

Fourth-Grade School Participation in the 2000 State Assessment
This table shows counts of participating 2000 state assessment fourth-grade schools by jurisdiction, subject, and sample type.
Number of participating fourth-grade schools, by subject area and sample type, state assessment: By jurisdiction, 2000 Mathematics Jurisdiction Total Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah Vermont See notes at end of table.
113

Science Accommodations not permitted S2 2,011 54 48 45 41 51 53 51 37 39 37 46 — 52 51 52 54 52 44 37 55 53 34 37 54 49 39 54 63 44 61 39 56 53 52 50 55 33 Accommodations permitted S3 2,036 55 47 48 40 56 54 55 41 38 41 43 — 53 57 55 56 54 39 41 51 50 33 36 55 49 39 54 66 41 59 40 54 50 53 50 55 33

Accommodations not permitted S2 2,115 54 48 48 41 51 53 54 38 39 38 44 38 52 52 53 54 50 43 38 56 50 32 41 55 51 38 53 70 43 59 38 57 54 51 52 54 28

Accommodations permitted S3 2,121 54 47 51 40 55 54 54 39 39 42 46 41 52 57 55 55 55 42 39 52 51 29 38 54 49 37 54 61 43 55 40 55 50 53 49 55 33

Number of participating fourth-grade schools, by subject area and sample type, state assessment: By jurisdiction, 2000 (continued) Mathematics Jurisdiction Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa District of Columbia DDESS1 DoDDS2 Guam Virgin Islands Accommodations not permitted S2 52 61 34 50 6 54 22 43 12 11 Accommodations permitted S3 54 62 36 44 10 55 18 43 12 12 Science Accommodations not permitted S2 53 61 33 47 8 — 21 42 12 9 Accommodations permitted S3 55 65 36 46 9 — 18 42 11 13

— District of Columbia did not participate in the 2000 science assessment at grade 4. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 State Assessment.

114

Eighth-Grade School Participation in the 2000 State Assessment
This table shows counts of participating 2000 state assessment eighth-grade schools by jurisdiction, subject, and sample type.
Number of participating eighth-grade schools, by subject area and sample type, state assessment: By jurisdiction, 2000 Mathematics Science

Jurisdiction Total Alabama Arizona Arkansas California Connecticut Georgia Hawaii Idaho Illinois Indiana Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Rhode Island South Carolina Tennessee Texas Utah

Accommodations Accommodations Accommodations Accommodations not permitted permitted not permitted permitted S2 S3 S2 S3 1,786 51 40 47 38 52 54 25 33 38 34 39 48 52 40 54 49 44 30 52 52 29 42 32 41 35 52 49 44 59 40 26 47 47 50 48 1,807 51 39 47 38 51 48 26 33 40 42 35 49 52 44 51 50 41 34 49 52 36 41 31 42 39 52 46 43 54 41 25 48 48 54 48 1,723 52 39 46 37 51 54 25 33 39 34 — 48 53 40 53 49 43 28 52 51 30 44 33 40 35 51 45 44 60 38 27 47 50 49 47 1,749 50 41 46 39 53 48 25 30 41 42 — 48 51 46 50 50 43 31 49 53 32 43 31 45 39 52 48 44 54 40 25 48 47 51 48

See notes at end of table.

115

Number of participating eighth-grade schools, by subject area and sample type, state assessment: By jurisdiction, 2000 (continued) Mathematics Science

Jurisdiction Vermont Virginia West Virginia Wisconsin Wyoming Other jurisdictions American Samoa District of Columbia DDESS2 DoDDS1 Guam Virgin Islands

Accommodations Accommodations Accommodations Accommodations not permitted permitted not permitted permitted S2 S3 S2 S3 40 53 51 34 35 7 16 6 26 ‡ ‡ 36 52 53 45 36 7 18 7 25 ‡ ‡ 40 53 48 35 33 7 — 7 27 ‡ ‡ 34 52 54 45 31 9 — 7 26 ‡ ‡

— Kansas and District of Columbia did not participate in the 2000 science assessment at grade 8. ‡ Reporting standards not met. Sample size is insufficient. 1 Department of Defense Domestic Dependent Elementary and Secondary Schools. 2 Department of Defense Dependents Schools (Overseas). SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 State Assessment.

116

Comparison of National Main and State Sample Designs
The national main and state assessments' sample designs differ in target population, assessment subjects, and number of required sampling stages. The national main assessment generally targets and samples from a larger student population than the state assessment. More sampling stages occur in the national main assessment than in the state assessment.

117

Target Populations
2000 National Main Assessment: • • • Public and nonpublic school students enrolled in fourth and twelfth grades Public school students enrolled in eighth grade Sample represents all 50 states, the District of Columbia, and domestic Department of Defense Education Activity (DoDEA) schools.

2000 State Assessment: • • Public school students enrolled in fourth or eighth grade Sample represents participating states/jurisdictions, including: o Department of Defense Education Activity (DoDEA) schools comprised of two separate jurisdictions: domestic (DoDEA/DDESS) and international (DoDEA/DoDDS), o American Samoa, o Guam, and o Virgin Islands

Sources of school data for the assessments include: • • • • Quality Education Data, Inc. (QED) National Education Database; National Center for Education Statistics' Common Core of Data (CCD); National Center for Education Statistics' Private School Survey (PSS); and 1990 Census median household income database created by Donnelley Marketing.

118

Assessment Subjects
Each NAEP assessment tests students on a variety of academic subjects. For the 2000 assessments, subject tests included mathematics, reading, and science. 2000 National Main Assessment: • • • Mathematics - grades 4, 8, and 12 Reading - grade 4 only Science - grades 4, 8, and 12

2000 State Assessment: • • Mathematics - grades 4 and 8 Science - grades 4 and 8

Sample designs for both assessments ensure that school and student samples for each subject represent their respective target populations.

119

Sampling Stages
2000 National Main Assessment • Fourth- and twelfth-grade students o Selected geographic primary sampling units o Selected schools from primary sampling units o Assigned sessions to schools o Selected students from schools Eighth-grade students o Selected schools within the integrated national and state samples o Assigned sessions to schools o Selected students from schools

2000 State Assessment • • • • Sampling began within a participating state/jurisdiction Selected schools from jurisdiction Assigned sessions to schools Selected students from schools

The integrated approach of the 2000 national main and state assessment for eighth-grade school samples combined the national and state samples after data collection. This approach produced a larger sample that yielded more precise estimates. Because the state assessment focused on public schools, no private school students appeared in the 2000 national main assessment eighth grade.

120

Sample Design for the 2001 Assessment
For 2001, the National Assessment of Educational Progress (NAEP) sampling followed a multistage approach: • • • sampling of geographic-based primary sampling units (PSUs), sampling of schools within PSUs, and sampling of students within schools.

Because NAEP state assessments are administered in even-numbered years, no state assessment was given in 2001.

http://cmspreview.naepims.org/nationsreportcard/tdw/sample_design/2000_2001/2001.asp

121

National Main Assessment Sample Design in 2001
The sample for the 2001 national main assessment in geography and U.S. history was a multistage probability-based sample. This was a national sample in which counties or groups of counties, also known as primary sampling units (PSUs), were the first-stage sampling units, and elementary and secondary schools were the second-stage units. The third stage of sampling consisted of the assignment of sample types to sampled schools. The fourth and final stage involved selection of students within schools and their assignment to session types (geography or U.S. history). The 2001 sample design oversampled specific student groups to enhance the reliability of NAEP estimates for the following groups: • • • Black and Hispanic students; students with disabilities and limited-English-proficient students; and private school students.

The oversampling of these groups was achieved at either the school or student sampling stages and, in the case of Black and Hispanic students, at both stages. In addition, the 2001 national main samples included substitute schools to compensate for school-level nonresponse and to improve the overall number of assessed students. A total of 94 PSUs were included in the sample. The sample of schools that participated in the assessment included 370 schools at the fourth-grade level, 370 schools at the eighth-grade level, and 370 schools at the twelfth-grade level. Assessments were administered in these schools to 15,700 fourthgraders, 22,700 eighth-graders, and 21,800 twelfth-graders for a total of 60,200 assessed students overall.

http://cmspreview.naepims.org/nationsreportcard/tdw/sample_design/2000_2001/2001_m.asp

122

Target Population and Sample Sizes of the 2001 National Main Assessment
The target population for the 2001 national main assessment consisted of fourth-, eighth-, and twelfthgrade students enrolled in public and private elementary and secondary schools throughout the United States and the District of Columbia. This population included the Bureau of Indian Affairs (BIA) and the Department of Defense Domestic Dependents Elementary and Secondary Schools (DDESS). The table below shows the target number of students to be assessed and the realized number of assessed students. The realized sample sizes exceeded the targets by 20 to 30 percent, with the highest ratio in fourth grade. The unusually high ratio was the result of conservative estimates of response and eligibility rates in the projections computing sample sizes. Sample size goals, by and subject area and grade, national main assessment: 2001
Grade All All All 4 4 8 8 12 12 Subject area Total Geography U.S. history Geography U.S. history Geography U.S. history Geography U.S. history Sample size goal 48,000 22,000 26,000 6,000 6,000 8,000 10,000 8,000 10,000 Realized sample size 60,100 27,600 32,600 7,900 7,800 10,000 12,700 9,700 12,100

NOTE: Detail may not sum to total because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

http://cmspreview.naepims.org/nationsreportcard/tdw/sample_design/2000_2001/2001_m_targpop.asp

123

Primary Sampling Units (PSUs) for the 2001 National Main Assessment
The samples for the 2001 national main assessment were selected using a complex multistage sample design involving the sampling of students from selected schools within 94 selected geographic areas, called primary sampling units (PSUs), across the United States. The samples were designed to represent fourth-, eighth-, and twelfth-grade students enrolled in public and nonpublic elementary and secondary schools.

124

Primary Sampling Unit Definition and Frame Creation for the 2001 National Main Assessment
The basic primary sampling unit (PSU) sample design for the 2001 assessment is a stratified probability sample with one PSU selected per stratum with probability proportional to population. A PSU can consist of the following: • • • • • a consolidated metropolitan statistical area (CMSA), a metropolitan statistical area (MSA), a New England county metropolitan area (NECMA), a single county in the United States (including Alaska, Hawaii, and the District of Columbia), or a group of contiguous counties in the United States.

A total of 94 PSUs per round were selected. PSUs that, because of their size, were drawn into the sample with certainty are common to the five rounds. The PSU sampling frame for the 2001 assessment was constructed by grouping counties as specified by the following rules: • Each 1990 CMSA, and each MSA that was not part of a CMSA, was considered a separate PSU. In New England, NECMAs were the metropolitan PSU unit. (In New England, counties may split between two or more MSAs. A NECMA is a definition that approximates MSAs but consist of whole counties.) Non-MSA PSUs were made to consist only of non-MSA counties. Whenever possible, each nonMSA PSU contained geographically contiguous counties with a minimum 1990 Census total population of 60,000 persons in the Northeast and Southeast regions, and 45,000 persons in the Central and West regions. Region boundaries were not crossed in the definition of a PSU, not even in the case of MSAs. If a county in an MSA was in a separate region, it was taken out of the MSA and grouped with other contiguous counties in its region to define a PSU.

Checks were made to ensure that every county was included in one and only one PSU. The 2001 PSU sampling frame contained 1,027 PSUs: 290 MSA PSUs, and 737 non-MSA PSUs.

125

Definition of PSU Strata for the 2001 National Main Assessment
Eight major strata were formed by crossing region and metropolitan statistical area (MSA) status. The primary sampling units (PSUs) were classified into four regions, each containing about one-fourth of the U.S. population. These regions, as listed below, were defined primarily by state. • • • • Northeast: Connecticut, Delaware, District of Columbia, Maine, Maryland, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island, Vermont, and Virginia. Southeast: Alabama, Arkansas, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Virginia, and West Virginia. Central: Illinois, Indiana, Iowa, Kansas, Michigan, Minnesota, Missouri, Nebraska, North Dakota, Ohio, South Dakota, and Wisconsin. West: Alaska, Arizona, California, Colorado, Hawaii, Idaho, Montana, Nevada, New Mexico, Oklahoma, Oregon, Texas, Utah, Washington, and Wyoming.

Note that those counties and independent cities in Virginia which are part of the Washington, DC-MD-VA MSA are included in the Northeast region. The remainder of Virginia is included in the Southeast region. The 22 largest PSUs were included with certainty. The inclusion of these PSUs in the sample with certainty provided an approximately optimum, cost-efficient sample of schools and students when samples were drawn within them at the required national sampling rate. The 22 largest PSUs are presented below by region: • Northeast: Baltimore, MD MSA; Boston-Lawrence-Salem-Lowell-Brockton, MA NECMA; New York-Northern New Jersey-Long Island, NY-NJ CMSA (excluding the part in CT); PhiladelphiaWilmington-Trenton, PA-NJ-DE-MD CMSA; Pittsburgh-Beaver Valley, PA CMSA; and Washington, DC-MD-VA MSA. Southeast: Atlanta, GA MSA; Miami-Fort Lauderdale, FL CMSA; and Tampa-St. PetersburgClearwater, FL MSA. Central: Chicago-Gary-Lake County, IL-IN-WI CMSA; Cleveland-Akron, OH CMSA; Detroit-Ann Arbor, MI CMSA; Minneapolis-St. Paul, MN-WI MSA; and St. Louis, MO-IL MSA. West: Dallas-Fort Worth, TX CMSA; Denver-Boulder, CO CMSA; Houston-Galveston-Brazoria, TX CMSA; Los Angeles-Anaheim-Riverside, CA CMSA; Phoenix, AZ MSA; San Diego, CA MSA; San Francisco-Oakland-San Jose, CA CMSA; and Seattle-Tacoma, WA CMSA.

• • •

The remaining smaller PSUs were not guaranteed to be selected with certainty for the sample. These were grouped into a number of noncertainty strata, and one PSU was selected from each stratum. In each region, noncertainty PSUs were classified as MSA (metropolitan) or non-MSA (non-metropolitan). Within each major stratum, further stratification was achieved by ordering the noncertainty PSUs according to several additional socioeconomic characteristics, yielding 72 strata. The strata were defined so that the aggregate of the measures of size of the PSUs in a stratum was approximately equal for each stratum. The size measure used was the population from the 1990 Census. The characteristics, available for all PSUs, that were used to define strata were as follows: • • • • the percent minority population, the percentage change in total population since 1980, the per capita income, the percent of persons age 25 or over with college degrees,

126

• •

the percent of persons age 25 or over who completed high school, and the civilian unemployment rate.

Up to four of these characteristics were used in any one major stratum. For each major stratum, the characteristics used were chosen by modeling NAEP PSU-level mean reading proficiency scores for the years 1988–1992 (e.g., for a particular major stratum, the characteristics might be per capita income and percent minority population, for another, civilian unemployment rate, percent of persons with college degrees, and percent change in total population, depending on what registered as significant in regression models with reading proficiency in that particular major stratum). The characteristics chosen were the best predictors of PSU-level mean reading proficiency scores in these models. An adjustment was made to re-scale 1988 data to match data from years 1989, 1990, and 1992. PSU-level mean reading scores were not needed for all PSUs in stratum. The point was to model the scores from characteristics that were available for all PSUs, so that those characteristics could be used to form efficient strata for gathering proficiency scores.

127

Number of Noncertainty Strata in Each Major Stratum in the 2001 National Main Assessment
The following table lists the number of strata for metropolitan statistical area (MSA) and non-MSA PSUs for the 2001 national main assessment by region. Number of noncertainty strata in each major stratum, national main assessment: By NAEP region, 2001
NAEP region Total Northeast Southeast Central West Total strata 72 10 24 20 18 Number of strata for Number of strata for MSA PSUs non-MSA PSUs 36 6 12 8 10 36 4 12 12 8

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

128

Selection of Primary Sampling Units in the 2001 National Main Assessment
A sample of 94 primary sampling units (PSUs) was drawn for the 2001 sample. Of the 94 PSUs, 22 were selected with certainty because they had the largest populations in the PSU universe. Within each of the 72 noncertainty strata, one PSU was selected with probability proportionate to its 1990 Census population. That is, within each stratum, a PSU's probability of being selected was proportional to its population. The PSUs were selected with probability proportional to size (PPS) with the twin aims of obtaining approximately self-weighting samples of students, and having approximately equal workloads in each PSU. PSUs were drawn to minimize overlap of the PSUs from one assessment to the next, except that certainty PSUs were retained in each assessment year, and some of the larger noncertainty PSUs are in the sample for more than one of these assessment years. Primarily because of the use of MSAs as PSUs, PSUs varied considerably as to their probability of selection, since they varied greatly in size. In 2001, the 36 selected noncertainty MSA PSUs had probabilities of selection ranging from 0.046 to 0.506, while the 36 selected non-MSA PSUs had probabilities ranging from 0.031 to 0.137. Within each stratum, the order of the PSUs was randomized. The selection of PSUs within a stratum was not independent among the survey years. Ordering the PSUs within a stratum by size, geography, or other variables could have resulted in unintended and possibly detrimental correlation between survey estimates across years. Since only one PSU is selected for a given year, the PSU ordering has no effect on sampling variance. For each PSU within a stratum, a normalized measure of size was calculated for each PSU within a stratum, where k is the PSU and i is the stratum. The cumulative count Cik is calculated as:

Cik =

∑ NM
j=1

k

ij

where NMij is the normalizing measure for the jth PSU in the ith stratum. A systematic sample was carried out using the measures, with five different sample designation numbers. These five sample designation numbers covered the five NAEP assessments between 1994 and 2001 (1994, 1996, 1998, 2000, 2001). The first sample designation number, denoted r, was selected as a random number between 0 and 1. This number was used for the NAEP 1994 assessment. Subsequent sample designation numbers added an additional factor to the original r. The sample designation number for the NAEP 2001 assessment equaled r + 0.6. Only the noninteger part of any number in the sequence which exceeded 1.0000 was retained. For example, if r equals 0.626743, then r + 0.6 equals 1.126743, and 0.126743 becomes the sample designation number for NAEP 2001. The first PSU in each stratum whose cumulative count was equal to or greater than the noninteger portions of r + 0.6 were designated the 2001 main sample PSUs. The purpose of having the sample designation numbers for 2001 be a function of r was to attempt to minimize the overlap among the sets of main sample PSUs chosen for the five NAEP assessments between 1994 and 2001 that involved PSU sampling. In strata with smaller numbers of PSUs, some PSUs had large enough normalized measures of size so that they were drawn for two and sometimes even three survey years. By setting the spacing between the sample designation numbers for any two consecutive survey years to 0.4, selecting the same PSU in two consecutive survey years was unlikely (had a low probability of occurring).

129

Sampling Frame for School Sampling Within Selected PSUs
The 2001 national main assessment surveys utilized the same original frame of schools as that used for the 2000 surveys. This frame was created by combining the 1997–1998Common Core of Data (CCD) frame of public schools, and the 1997–1998 Private School Survey (PSS) file of nonpublic schools. More details about the construction of this frame can be found in the 2000 assessment sampling frame pages. For the 2001 assessment, a differing set of primary sampling units (PSUs) was selected so that the final school frame was not identical to that used in the 2000 assessment. The population of eligible schools for each grade was restricted to the selected 94 PSUs. Any school having one or more of the eligible grades, and located within an appropriate PSU, was included in the sampling frame of schools (the list of schools from which the samples of schools were drawn) for a given sample. An independent sample of schools was selected for each of the grades. For each school in each frame, the number of assessable students was provided by school officials. (See "Students Excluded from the 2001 National Main Assessment" for more information of those students deemed unable to participate in the assessment.) The CCD and PSS files give total enrollment, enrollment by grade, and the grade range for each school, thus providing the average enrollment per grade. A school could appear on more than one grade-level frame. As a result, there is considerable overlap among the three grade-level frames. Before selecting schools, high-minority public schools were identified for oversampling. The school was classified as high-minority if the percentage of Hispanic and Black students was reported to be greater than 10 percent (15 percent for grade 12) and if the number of Hispanics and Black students was reported to be at least 10 (15 for grade 12). Otherwise the school was classified as low-minority.

130

Number of Schools Within the 94 Selected PSUs by School Type in the 2001 National Main Assessment
The table below shows the numbers of schools for each grade-specific sample within the 94 selected primary sampling units (PSUs) for the NAEP 2001 national main assessment.
Number of schools within the 94 selected PSUs, by school type and grade, national main assessment: 2001 Grade 4 8 12
1

Public 21,672 9,735 7,500

Other nonprivate1 14 10 8

Private2 10,528 9,275 4,158

Other nonprivate schools include Department of Defense Education Activity schools and Bureau of Indian Affairs schools. 2 Private schools include Catholic schools, non-Catholic religious schools, and non-religious private schools. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

131

Each public school designated as high-minority was given double the measure of size of a public school not designated high-minority of similar size. Such high-minority schools were oversampled in order to enlarge the sample of Black and Hispanic students, thereby enhancing the reliability of estimates for these groups. Each public school designated as high-minority was given double the measure of size of a public school not designated high-minority of similar size. Such high-minority schools were oversampled in order to enlarge the sample of Black and Hispanic students, thereby enhancing the reliability of estimates for these groups. For a given overall size of sample, this procedure reduces somewhat the reliability of estimates for all students as a whole and for those not Black or Hispanic.

The actual school sample from the school frame for each grade was a systematic sample using these measures of size, with schools ordered to achieve an implicit stratification on school characteristics judged to be related to school achievement. The NAEP 2001 ordering was the same as in NAEP 2000. The frame schools for each grade were ordered first by dividing into three subsets: certainty PSU public schools, certainty PSU nonpublic schools, and noncertainty PSU schools. The certainty PSU public schools were ordered by NAEP region first, type of locality second, minority status third, PSU stratum fourth, and estimated grade enrollment fifth. The certainty PSU nonpublic schools were ordered by NAEP region first, school type second (Catholic, non-Catholic religious, non-religious private), PSU stratum third, and estimated grade enrollment fourth. The noncertainty PSU schools were ordered by PSU stratum first, and then by school type (public and nonpublic). The public schools were then ordered by type of locality first, minority status second, and estimated grade enrollment third. The nonpublic schools were ordered by school type first (Catholic, nonCatholic religious, non-religious private), and estimated grade enrollment second. The sort order was 'serpentine' at each level: an alternation of ascending to descending and descending to ascending within each higher level group. For example, for certainty PSU public schools, the sort order for estimated grade enrollment within PSU stratum was ascending to descending for the first PSU stratum, descending to ascending for the next PSU stratum, and so on. Once the measures of size had been determined, the school samples were selected.

132

Assigning Measures of Size and Selecting School Samples for the 2001 National Main Assessment
For each grade-level sample, schools were selected (without replacement) across all primary sampling units (PSUs), systematically from a sorted list with probabilities proportional to assigned measures of size. The sorting variables included the following variables: • • • • • • • certainty/noncertainty PSU classification, NAEP region (Northeast, Southeast, Central, and West), public/nonpublic school classification, type of location, high/low minority classification, PSU stratum, and estimated grade enrollment.

Type of location is a Common Core of Data (CCD) field attached to each school defining the type of locality of the community: central city large city, central city mid-sized city, urban fringe large city, urban fringe mid-sized city, large town, small town, and rural. (Note that the two CCD rural subgroups are collapsed together here.) The order of the sort was different depending on public and private school classification and certainty/noncertainty PSU classification. To increase cost-efficiency in sampling, samples were designed to include more nonpublic schools, highminority public schools, and relatively large schools. Therefore, a measure of size was assigned to each school according to the following scheme. Let Si denote the estimated number of grade-eligible students in school i. Let L = 60 for the assessment of grade 4, L = 100 for the assessment of grade 8, and L = 100 for the assessment of grade 12. The measure of size was defined as follows: • • • • 0.25Ki, if Si was five or less;

KiSi /20, if Si was between six and nineteen; Ki, if Si was between 20 and L; and KiSi /L, if Si was greater than L;

where Ki equals • • 2.8, for nonpublic schools on the fourth and eighth grade frames (other than Bureau of Indian Affairs and Department of Defense domestic schools); 2.875, for nonpublic schools on the twelfth grade frame (other than Bureau of Indian Affairs and Department of Defense domestic schools); 2, for high minority public schools; and 1, for low minority public schools.

This procedure was used so as to obtain approximately self-weighting samples of students (i.e., students selected with approximately equal overall probabilities) at each grade within oversampling domains. Three aspects of the school-level measure-of-size cause the resulting student weights to vary. • Schools with fewer than 20 estimated grade-eligible students were assigned somewhat lower measures of size, and thus lower probabilities of selection. This was designed to increase cost efficiency.
133

Selecting School Samples for the 2001 National Main Assessment
Once the measures of size had been assigned, the actual school sample from the school frame for each grade was a systematic sample using these measures of size, with schools ordered to achieve an implicit stratification on school characteristics judged to be related to school achievement. The 2001 NAEP assessment frame ordering was the same as that of the 2000 assessment. The frame schools for each grade were ordered first by dividing into three subsets: • • • certainty primary sampling unit (PSU) public schools; certainty PSU nonpublic schools; and noncertainty PSU schools.

The certainty PSU public schools were ordered by • • • • • NAEP region first; type of locality second; minority status third; PSU stratum fourth; and estimated grade enrollment fifth.

The certainty PSU nonpublic schools were ordered by NAEP region first, school type second (Catholic, non-Catholic religious, non-religious private), PSU stratum third, and estimated grade enrollment fourth. The noncertainty PSU schools were ordered by PSU stratum first, and then by school type (public and nonpublic). The public schools were then ordered by type of locality first, minority status second, and estimated grade enrollment third. The nonpublic schools were ordered by school type first (Catholic, nonCatholic religious, non-religious private), and estimated grade enrollment second. The sort order was 'serpentine' at each level: an alternation of ascending to descending and descending to ascending within each higher level group. For example, for certainty PSU public schools, the sort order for estimated grade enrollment within PSU stratum was ascending to descending for the first PSU stratum, descending to ascending for the next PSU stratum, and so on.

134

School Substitution for Adjusting School Nonresponse for the 2001 National Main Assessment
NAEP assessments traditionally use substitute schools to compensate for school-level nonresponse and to improve the overall yield of assessed students. Potential substitute schools were selected for all sampled schools in the 2001 national main assessment where a close match could be identified. The procedure for identifying substitutes followed the new procedure first used for the 2000 assessment. No sampled school was assigned more than one substitute, and no school was assigned to be a substitute for more than one school. The criteria for assigning substitutes was quite strict; many sampled schools were not assigned substitutes at all as there were no schools that met the necessary criteria to be a substitute. All schools with either a fourth, eighth, or twelfth grade that were not in any of the samples were the potential pool of substitutes. The substitutes for the national main assessment schools and the mathematics online special study schools were drawn in a single step from a joint pool of unsampled schools. The twelfth -grade sampled schools received substitutes first, followed by eighth-grade sampled schools (for both the national main and the special study samples) next, and fourth grade schools last (for both the national main and the special study samples). Twelfth-grade sampled schools were assigned substitutes first, as there are fewer schools available as substitutes at that grade. It should be noted that a national main assessment school that was sampled for one grade was eligible to be a substitute for a national main assessment school in another grade, if it had that grade. A school with a particular sample type was not eligible to be a substitute for a school with a different sample type, as this would add expenses to the accommodations process. Mathematics online special study schools for one grade were also eligible to be substitutes for mathematics online special study schools in the other grade. National main assessment schools were not eligible to be substitutes for mathematics online special study schools, and vice-versa. There were a number of other 'absolute' boundaries, as follows: • Sampled schools could only have as substitutes schools in the same school type group, with school type groups defined as regular public, Bureau of Indian Affairs, Department of Defense, other public, Catholic, non-Catholic religious, and other private. Public schools could only have as substitutes schools in the same primary sampling unit (PSU) and with the same locality type. Catholic schools could only have as substitutes schools in the same district (usually diocese).

• •

Within the various cells defined by these absolute boundaries, distance measures were computed for each sampled school and its potential matches. The distance measure was computed based on percent Black students, percent Hispanic students, and the square root of the grade enrollment. It used a th standardized Euclidean distance Dij as follows (with Dij the distance between the i sampled school and th the j candidate substitute):

Dij =

( Bi − Bj ) 2 ( Hi − Hj ) 2 ( Ei − Ej ) 2 + + VHB VHB VSQE

135

where • • • • • • •

Bi = the percentage of Black students in school i, Bj = the percentage of Black students in out-of-district substitute school j, Hi = the percentage of Hispanic students, Hj = the percentage of Hispanic students in out-of-district substitute school j, Ei = the school grade enrollment, VBH = the average of the mean variance of Black percentages across all schools within the grade and the mean variance of Hispanic percentages across all schools within the grade, and VSQE = the mean variance of the square root of estimated grade enrollment across all schools
within the grade.

Two passes were carried out for each grade. The first pass was a matching of regular public schools to schools outside their district. From a field operations viewpoint, it is preferable to have substitutes in different districts, as many public school districts refuse for all of their schools. An out-of-district substitute j was taken for a sampled public school i if the pair had a Dij value less than 0.65. If there was more than one candidate for a particular sampled school i, then the j-school with the smallest Dij was selected. Although the selected cutoff point of 0.65 on the distance measure is somewhat arbitrary, statisticians in the 1994 Trial State Assessment (Mazzeo, Allen, and Kline 1995) selected this as the distance measure at which substitutes began to appear unacceptable. This cutoff point was also utilized in the2000 assessment. Note that a particular j-school might be the best substitute for two or more sampled schools. In this case, the school was matched with the sampled school in which it had the smallest Dij value. Because of this rule, some sampled schools did not receive as substitutes their best matches, as those schools might have already been assigned to other sampled schools. All matched schools in the first pass were removed from further consideration. The second pass was then a matching of regular public schools within their districts, and of all private schools (Catholic schools within their dioceses, religious non-Catholic schools in one cell, and non-religious private schools in one cell). The Dij limit was 1.00; if a sampled school had no potential substitute lower than this limit, then that sampled school received no substitute. Note that a school might have a j-school with a Dij value less than 1.00 that is used by another school as its substitute, and may thereby have no substitute.

136

Substitute Frame Standard Deviations for the 2001 National Main Assessment
In the process of school substitution to adjust for nonresponse, distance measures were computed for each sampled school and its potential matches. These distance measures for pairs of schools were computed by dividing the sum of squares of the differences of three values for the pair of schools divided by the corresponding variances, which can be seen as 'average' squared differences over all pairs of schools for each school. A pair of schools will register a large difference if any of these squared differences exceed considerably the corresponding variance. The distance measure was computed based on percent of Black students, percent of Hispanic students, and the square root of the grade enrollment. Note that values for the first pass were slightly different from the values for the second pass because of changes in the three variances: the second pass variance computations excluded the schools already selected as substitutes in the first pass.
Substitute frame standard deviations, national main assessment: 2001 Measure Percent of Black students Percent of Hispanic students Mean percent Black-Hispanic Square root grade enrollment Fourth-grade Eighth-grade Twelfth-grade standard deviation standard deviation standard deviation 28.7 23.6 26.3 3.0 27.9 21.6 24.9 5.8 25.5 19.9 22.9 7.4

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

137

Originally Sampled Schools With and Without Substitutes for the 2001 National Main Assessment
This table presents the numbers of schools in the 2001 national main assessment that were assigned substitutes, and how many schools were not assigned substitutes.
Originally sampled schools with and without substitutes, national main assessment: 2001 School counts Total originally sampled schools With assigned substitutes Without assigned substitutes Fourthgrade 440 343 97 Eighthgrade 462 345 117 Twelfthgrade 522 339 183

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

138

Ineligible Schools for the 2001 National Main Assessment
The NCES Common Core of Data (CCD) public school file from which most of the sampled schools were drawn corresponds to the 1997-98 school year, two years prior to the assessment school year. During the intervening period, some of these schools either closed or no longer offered the grade of interest. In other cases, records in the CCD file were sampled and found to be special schools, such as adult education programs, subsidiary vocational programs, schools for blind and deaf students, and correctional facilities. In such cases, the sampled school was coded as ineligible for the NAEP assessment. The upper portion of the table on the web page School Participation for the 2001 National Main Assessment provides counts of the schools that were not eligible to participate in the 2001 national main assessment.

139

Sampling Within Schools in the 2001 National Main Assessment
The second stage of the national main sample design was the selection of the school sample. Schools were selected with probability proportional to size at rates targeted to meet required student sample sizes. The students selected for inclusion in the sample (in either the initial sample, Black and Hispanic oversample, or the SD/LEP oversample) were allocated to a session based on the final number of sessions assigned to the school and whether the school was public or private.

140

Assignment of Sample Types of Schools in the 2001 National Main Assessment
In order to determine the effect of using different criteria for excluding students from the assessment, two different sample types (S2 and S3) were assigned to the schools sampled for the U.S. history and geography study. In sample type 3 schools, accommodations were offered to students with disabilities (SD) and limited-English-proficient (LEP) students who met the criteria for inclusion in the assessment. In sample type 2 schools, no assessment accommodations were offered to SD/LEP students. Sample type was assigned to schools separately for each grade so that 50 percent of the schools were assigned sample type 2 and 50 percent were assigned sample type 3. It should be noted that schools which were sampled for more than one grade were assigned only one sample type, which was used then for all sampled grades.

141

Assignment of Sessions to Schools for the 2001 National Main Assessment
In the 2001 assessment, schools were assigned two types of sessions: "main"—for geography and U.S. history, and "field test"—for future NAEP assessments. The field tests were not weighted. Schools were assigned 1, 2, 3, or 4 sessions depending on their estimated grade enrollment from the frame. The cutoffs were as follows: • • • • Schools with 25 or less estimated grade enrollment: 1 session; Schools with 26 to 50 estimated grade enrollment: 2 sessions; Schools with 51 to 75 estimated grade enrollment: 3 sessions; and Schools with 76 or more estimated grade enrollment: 4 sessions.

If the school was found to have significantly fewer students than was expected at the time of assessment, then the session count was adjusted downwards. Each session for the school was then assigned as main or field based on the following probabilities: • • • • Fourth and eighth grade public: 10/11 chance of main, 1/11 chance of field; Fourth and eighth grade nonpublic: 30/31 chance of main, 1/31 chance of field; Twelfth grade public: 15/16 chance of main, 1/16 chance of field; and Twelfth grade nonpublic: 45/46 chance of main, 1/46 chance of field.

Note that, in some cases, a school with only one session could have that session assigned as a field session. In that case, the school drops out of the sample for the history and geography assessment. The weights of schools with only one session that are assigned to the main assessment are adjusted upwards to account for these schools which are assigned solely to the field test. The following table presents the session results for the participating schools in the history/geography assessment by grade.
Distribution of main and field test sessions among sampled schools by grade, national main assessment: 2001 Session sampling outcomes Total participating schools All field test sessions All geography/U.S. history sessions 1:1 Geography/U.S. history to field test sessions 2:1 Geography/U.S. history to field test sessions 3:1 Geography/U.S. history to field test sessions 4:1 Geography/U.S. history to field test sessions
1 2

Fourth grade 365 2 308 551 0 0 0

Eighth grade 369 1 279 3 122 73 1

Twelfth grade 374 1 301 3 3 66 0

Nine of these schools had two sessions each. The rest had one session each. Two of these schools had six sessions altogether. The rest had three sessions altogether. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

142

Student Sample Selection for the 2001 National Main Assessment
The student sample selection within sampled schools for the national main assessment was systematically drawn from school-prepared lists of eligible students. (Ineligible students were those who had been deemed by school officials to be unable to participate meaningfully in the assessment or to be unable to participate without assessment accommodations that the program did not, at the time, make available. These ineligible students; therefore, were excluded from the assessment.) Students who are not eligible are given in the Excluded Students webpage). Student Listing Forms (SLF) were prepared for each participating school in a given grade; all enrolled students of the specified grade were to be entered on the SLFs. Student samples also included oversampling of Black and Hispanic students in low minority public schools, and oversampling SD/LEP students in all schools. The samples were specified through the use of Session Assignment Forms. There were a number of schools, primarily in the Southwest, which are year-round schools, where at any given time a certain percentage of the students will be on vacation. At the time of assessment, a certain percentage of students were off-track—in their out-of-school period. Assignment to tracks was generally a random process; the sample design treated the on-site students as an equal probability sample from the full set of grade-eligible students. In the assessed sample, there were 11 fourth-grade schools in this category, three eighth-grade schools, and two twelfth-grade schools. The percentages of off-track students in these schools ranged from a low of 25 percent to a high of 33 percent.

143

Student Sampling Procedure for the 2001 National Main Assessment
In the 2001 assessment, 'e-sampling' was used for the first time. Under this process, participating schools sent in their student lists via the Internet to the sampling statisticians, who drew the student samples. If the field personnel found new students added to the grade-eligible list at the time of assessment, then a supplementary sample was drawn of the new students. This sample was drawn using the same sampling rates as for the original list, but with no oversampling of minority or SD/LEP students. If the number of new students exceeded 20 percent of the earlier student list, the field personnel were asked to discard the original sample and draw a new sample in the field. Some schools did not participate in 'e-sampling'. Field personnel drew the samples for these schools in the field.

144

Within-School Sampling Rates for the 2001 National Main Assessment
Let L equal the maximum allowable sample size from an individual school (60 for grade 4; 100 for grade 8; and 100 for grade 12). Let Gi be the revised estimate of grade eligible students for school i. Then the sampling rate, denoted by R, applied to the list of eligible students to select the sample was defined as follows: • •

R = L/Gi, if Gi > (L + 10); or R = 1, otherwise.

It should be noted that many schools requested that all of their grade-eligible students should be taken even though they were over the maximum limits (70 for fourth grade; 110 for eighth and twelfth grades). These schools were accommodated in their requests. The total number of schools in this category were as follows: • • • 14 fourth-grade schools (with a mean value of 88.6 grade-eligible students), 7 eighth-grade schools (with a mean value of 128.7 grade-eligible students), and 7 twelfth-grade schools (with a mean value of 140 grade-eligible students).

Three schools (1 in fourth grade and 2 in eighth grade) requested that whole classrooms be selected rather than students. These schools were accommodated in their request. A without-replacement simple random sample was taken of the classrooms in these schools. The reciprocals of the classroom selection rates were incorporated into the student weights for these schools.

145

Oversampling Black and Hispanic Students for the 2001 National Main Assessment
In public schools with high minority (Black and Hispanic) enrollments, schools were assigned a measure of size twice the size of other low minority schools, therefore increasing their probability of selection, and indirectly increasing the number of Black and Hispanic students in the sample. In public schools with low minority enrollment, an oversample of Black and Hispanic students was selected. The procedure was as follows. After the initial sample was selected, the non-selected Black and Hispanic students were identified and listed. All such extra Black and Hispanic students were sampled to a total that, in expectation, was the same number of Black and Hispanic students as were already selected. In practice, if the number of non-selected students was less than the number of selected students, then all non-selected Black and Hispanic students were to be assessed also. Otherwise, Black and Hispanic students were sampled so that their overall within-school probability of selection was twice the rate of other students. The sampling of additional Black and Hispanic students was carried out using designated line numbers, as indicated on the session assignment form used to generate the samples of students in each school. In this way, the necessary information as to the selection probability of each student was retained for use in weighting. No reliance was placed on information generated in the field. Field supervisors had only to follow the pre-specified sampling instructions. Since the aim was to oversample by a factor of two where possible, but never more than two, the overall rate of oversampling of Black and Hispanic students was instead less than two. That is because in smaller low minority schools there were no students remaining who had not already been assigned to a session. The weighting procedures ensured that the results were not biased as a result of the relative under-representation of Black and Hispanic students from smaller low minority public schools.

146

Oversampling SD/LEP Students for the 2001 National Main Assessment
The procedures for assessing students with disabilities (SD) and limited-English-proficient (LEP) students varied by sample type. SD/LEP students in sample type 3 were offered various accommodations not available to either non-SD/LEP students or SD/LEP students in sample type 2. As a measure to ensure an adequate sample size of SD/LEP students from both sample types 2 and 3, oversampling procedures were applied to SD/LEP students at all three grades. In this way, comparisons of the effect of offering accommodations to students have enhanced power to detect effects. The general intent of oversampling within each school was to select SD/LEP students at twice the rate at which non SD/LEP students were sampled (or to include all SD/LEP students if there were not sufficient numbers to permit sampling at twice the rate.) There was no oversampling of schools as part of the procedure. The procedure was as follows. In each school where oversampling of SD/LEP students was to occur, the initial desired student sample selection was drawn from the full list of eligible students. In addition, in public schools in low minority areas, oversampling of Black and Hispanic students occurred. Among those students not selected for either of the two prior sampling operations for this school, the SD/LEP students were identified. A sample from among these was drawn, using a sampling rate that would achieve the double sampling rate required overall. In most grade 4 cases, this involved selecting all such SD/LEP students in the school. The weighting procedures ensured that the results were not biased as a result of the relative under-representation of SD/LEP students from smaller schools.

147

Updating Number of Sessions in the 2001 National Main Assessment
If a school originally assigned two or more sessions was found to have significantly fewer students than was expected at the time of assessment, then the number of sessions originally assigned to the school was reduced as follows. • • • In general, sessions were dropped if a school did not have at least 12 students per session for the sampled grade. For grade 8 public schools and all private school samples, which had at most two sessions assigned to a school, the enrollment size cutoff for dropping a session was 24 students. For grade 12 public schools, which also were assigned at most two sessions, the cutoff was slightly higher (28) because of their differential student allocation rate. Instead of equally distributing students among the sessions assigned to a school as was done for all the other samples, grade 12 public schools required a student allocation of 7/16 to mathematics and 9/16 to science. For grade 4 public schools, which could have up to four assigned sessions, the rules for dropping sessions and the number of sessions to drop were more detailed, but still were based on same general principle of a minimum session size of 12 students.
Rules for dropping sessions for grade 4 public schools, national main assessment: 2001 Original number of assigned sessions 4 Number of sessions to drop 1 2 3 1 2 1

Cutoff rules 36 ≤ NSLF < 48 24 ≤ NSLF < 36 NSLF < 24

3 2

24 ≤ NSLF < 36 NSLF < 24 NSLF < 24

NOTE: NSLF = The number of students listed on the student listing form for a given school. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

In schools where sessions were reduced, the types of sessions were dropped proportionally to the session type allocation discussed in Assignment of Sessions to Schools. For example, in grade 12 the probability of dropping mathematics and science session in public schools was 7/16 and 9/16, respectively; for private schools, the probabilities for dropping mathematics and science were both 1/2.

148

Students Excluded From the 2001 National Main Assessment
In NAEP assessments, some students have been excluded from the assessment because school officials believed that either they could not participate meaningfully in the assessment or that they could not participate without assessment accommodations that the program did not, at the time, make available. These students fall into the general categories of students with disabilities (SD) and limited-English­ proficient students (LEP). Some identified students fall within both of these categories. Schools were asked to complete a questionnaire for each student identified as having a disability or its equivalent (SD) and/or being limited English proficient (LEP). School personnel determined if any of these students should be excluded from the assessment, as shown on the page "Determining the SD/LEP Student's Capability to Participate in the Assessment."

149

Weighted percentages of students excluded (SD and LEP) from the assessment, national main assessment: 2001 Percentage Percentage Percentage Percentage of students Percentage of students of students Percentage of students that were of students that were excluded excluded School identified as of students identified identified type SD or LEP excluded as SD and SD as LEP and LEP 16.04 17.58 2.92 15.63 17.03 2.57 10.77 11.20 5.53 15.50 17.03 2.96 15.73 17.11 2.82 10.63 11.06 5.18 17.37 18.80 5.34 16.01 17.38 3.29 9.96 10.59 1.95 17.59 19.29 3.26 16.59 17.94 4.15 9.73 10.39 1.15 7.65 8.42 1.06 7.55 8.18 1.63 4.77 5.02 1.75 6.89 7.67 0.41 7.85 8.51 1.79 4.39 4.61 1.60 4.12 4.45 1.34 3.72 4.10 0.17 2.32 2.51 0.00 3.34 3.66 0.64 3.42 3.78 0.14 2.44 2.63 0.00 10.60 11.50 2.92 12.25 13.33 2.13 8.42 8.68 5.17 9.94 10.80 2.86 12.27 13.35 2.19 8.08 8.32 5.09 12.65 13.55 5.06 12.41 13.41 3.12 7.75 8.21 1.95 13.03 14.22 2.98 12.92 13.88 4.02 7.66 8.15 1.15 6.13 6.72 1.06 6.67 7.26 1.19 4.16 4.38 1.40 5.35 5.95 0.41 6.99 7.62 1.16 3.83 4.01 1.51 2.62 2.81 1.05 2.77 3.07 0.00 2.02 2.18 0.00 2.21 2.41 0.48 2.46 2.72 0.00 2.17 2.34 0.00 6.01 6.71 0.00 3.87 4.24 0.44 2.73 2.92 0.35 6.18 6.90 0.25 3.98 4.34 0.63 2.81 3.02 0.08 5.49 6.10 0.28 4.39 4.84 0.17 2.43 2.62 0.00 5.52 6.15 0.28 4.33 4.78 0.14 2.27 2.45 0.00 1.92 2.15 0.00 1.22 1.30 0.44 0.85 0.89 0.35 1.85 2.07 0.00 1.21 1.27 0.63 0.75 0.81 0.08 1.68 1.84 0.28 1.19 1.30 0.17 0.41 0.45 0.00 1.43 1.58 0.16 1.13 1.23 0.14 0.35 0.38 0.00

Student population

Subject area Grade 4

Population Geography for which accommodations were not allowed1

Overall Public Nonpublic 8 Overall Public Nonpublic 12 Overall Public Nonpublic Overall Public Nonpublic 8 Overall Public Nonpublic 12 Overall Public Nonpublic Overall Public Nonpublic 8 Overall Public Nonpublic 12 Overall Public Nonpublic Overall Public Nonpublic 8 Overall Public Nonpublic 12 Overall Public Nonpublic 4 4 4

U.S. history

Population Geography for which accommodations were allowed2

U.S. history

1

R2 Sampled students that neither have a student disability (SD) nor are limited English proficient (LEP), plus SD/LEP students from sessions in which accommodations were not allowed. 2 R3 Sampled students that neither have a student disability (SD) nor are limited English proficient (LEP), plus SD/LEP students from sessions in which accommodations were allowed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

150

Student Exclusion Rates by School Type for the 2001 National Main Assessment
The table below presents the overall rates of exclusion for each grade, subject, sample type and school type. In general, the exclusion rates decrease as grade increases. Likewise, the sample type 3 (S3) exclusion rate was lower than that of sample type 2 (S2) because accommodations were offered.
Weighted student exclusion rates by grade and school type, by reporting population and subject area, national main assessment: 2001 Reporting sample Sample for which accommodations were not permitted (S2) Subject area Grade Geography 4 8 12 4 8 12 4 8 12 4 8 12 Percentage of public school students excluded 8.42 8.18 5.02 7.67 8.51 4.61 4.45 4.10 2.51 3.66 3.78 2.63 Percentage of Percentage of students nonpublic school excluded (both public- and students excluded nonpublic-school students) 1.06 1.63 1.75 0.41 1.79 1.60 1.34 0.17 0.00 0.64 0.14 0.00 7.65 7.55 4.77 6.89 7.85 4.39 4.12 3.72 2.32 3.34 3.42 2.44

U.S. history

Sample for which accommodations were permitted (S3)

Geography

U.S. history

NOTE: 'S2' sampled students that neither have a student disability (SD) nor are limited English proficient (LEP), plus SD/LEP students from sessions in which accommodations were not allowed. 'S3' sampled students that neither have a student disability (SD) nor are limited English proficient (LEP), plus SD/LEP students from sessions in which accommodations were allowed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

The table on the following page shows the distribution of excluded students by reason for exclusion for the three grades. The dominant reason for exclusion from NAEP across all grades and subjects was a student disability. In general, the proportion attributable to student disability and/or limited English proficiency decreased with grade.

151

Assigning Assessments as U.S. History and Geography for the 2001 National Main Assessment
The Common Core of Data (CCD) files do not contain schools that opened between 1998 and the assessment dates. Therefore, special procedures were implemented to be sure that the NAEP assessment represented students in new public schools. (The NAEP sampling contractor did not do this for private schools, because there isn't a district structure that can be readily utilized, except for Catholic schools. Catholic schools were in fact included for the first time in the new school procedure in 2003.) Small school districts, those which contained only one eligible school for a given grade, were handled differently from large school districts, which contained more than one eligible school for a given grade. In small school districts, the schools selected for a given grade were thought to contain all students in the district that were eligible for the assessment. Districts containing these schools were asked if other schools with the appropriate grades for the assessment existed, and if so, they were automatically included in the assessment. For large school districts, a district-level frame was constructed from the schools on the CCD file. Districts were then sampled systematically with probabilities proportional to a measure of size. In most cases, the measure of size was total district enrollment, but in very small districts, a minimum measure of size was used. Each sampled district was asked to update the list of eligible schools derived from information on the CCD files. Frames of eligible new schools were then constructed at each grade, and samples of new schools were selected systematically with probability proportional to eligible enrollment using the same sampling rates as for the CCD schools. As a result of this process, one new school was selected in grade 12, and one school newly eligible for grade 8 was selected in that grade.

152

Participation Results for the 2001 National Main Assessment
The National Center for Education Statistics (NCES) statistical standards require that weighted response rates exceed 85 percent. In some cases, the response rates for the 2001 assessment were achieved without the need to recruit substitute schools, and, in some cases, substitutes needed to be recruited. Weighted aggregations for each set of schools were calculated by dividing the frame enrollment by the school probability of selection. The weighted aggregations for the eligible schools for each jurisdiction are an estimate of the total population of students in the grade within each jurisdiction.

153

School Participation for the 2001 National Main Assessment
The table below provides a detailed breakdown of all schools contacted for the 2001 national main assessment by their participation status. It also includes the 1998 and 2000 participation rates, which are based on the same computations. Overall, the 2001 before-substitution school participation rates were lower than school participation rates encountered in previous years. However, the after-substitution participation rates were higher than in previous years.
Unweighted summary of school participation experience, national main assessment: 2001 School participation Total original sample Out-of-scope1 District refused District refused for this school only School refused Participating Participation rate before substitution 2001 national main (percent) 2000 mathematics online (percent) 1998 national main (percent) Participating replacement for refusals Total participating schools Participation rate after substitution (percent)
1

Grade 4 440 20 20 10 50 340 82 85 81 20 370 88

Grade 8 470 50 20 10 40 340 82 83 81 30 370 88

Grade 12 520 60 30 20 80 350 75 78 75 30 370 81

Does not have eligible grade; has eligible grade but no enrolled students; closed school; not a regular school. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

154

Student Participation Results for the 2001 National Main Assessment
The following table shows the unweighted student participation rates of invited students by grade and school type for the 2001 national main assessment. The set of invited students consists of the selected students that remained after the removal of excluded students. When, for various reasons, more than a predetermined tolerable number of invited students were absent from the originally scheduled session to which they were invited, a makeup session was held. The participation rates given in the table express the number finally assessed as a percentage of those initially invited in the participating schools. Participation rates are shown for public and nonpublic schools separately. In each of 10 schools (3 in eighth grade and 7 in twelfth grade), the school allowed only a subset of its grade-eligible students to be included in the sample. In each case the students who were excluded from the sampling frame for the school were assigned a 'pseudo-nonrespondent' code in the NAEP data records. In all steps of weighting and participation rate calculations, pseudo-nonrespondent students were treated as absent students.
Student participation counts and (unweighted) rates, by school type and grade, national main assessment: 2001 Public school students Number of public school students Grade invited 4 8 12 14,300 21,600 26,200 Nonpublic school students Number of nonpublic Nonpublic school school students participation invited rate 2,200 3,000 2,600 96.8 95.9 90.6 Combined number invited

Public school participation rate 95.1 91.8 74.5

Combined number invited 16,400 24,600 28,700

Overall participation rate 95.3 92.3 75.9

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

The combined impact of school nonparticipation and student absenteeism from sessions within participating schools is summarized in the table below. The table shows weighted participation rates by grade, assessment, and reporting population. The table shows the school participation rates before and after cooperating substitute schools are added, and the student participation rates after absentee sessions are added. The two overall student participation rates are the student participation rates multiplied by the before-substitution school participation rate and the after-substitution school participation rates, respectively. The procedures for substituting for nonparticipating schools or imputing for them through weighting and the procedures for imputing for absent students through weighting were designed (so far as feasible) to reduce the biases resulting from school and student nonparticipation. These procedures are discussed in Weighting Procedures and Variance Estimation for the 2001 National Main Assessment.

155

Results from the table below show that, consistent with earlier rounds of NAEP, the overall level of participation decreases substantially with the increase in the grade of the students.
Weighted school and student participation rates, by subject area and grade, national main assessment: 2001 Grade 4 Participation School participation Before substitution After substitution Student participation Overall participation Before school substitution After school substitution School participation Before substitution After substitution Student participation Overall participation Before school substitution After school substitution School participation Before substitution After substitution Student participation Overall participation Before school substitution After school substitution U.S. history/R2 percent 83.3 88.3 96.0 79.9 84.7 79.5 87.1 93.3 74.2 81.3 72.5 80.1 77.6 56.2 62.2 U.S. history/R3 percent 83.3 88.3 95.8 79.8 84.6 79.5 87.1 92.8 73.8 80.9 72.5 80.1 77.4 56.1 62.0 Geography/R2 percent 83.3 88.3 95.2 79.2 84.0 79.5 87.1 92.5 73.5 80.6 72.5 80.1 77.5 56.1 62.0 Geography/R3 percent 83.3 88.3 95.2 79.2 84.0 79.5 87.1 92.1 73.2 80.2 72.5 80.1 77.1 55.9 61.8

8

12

SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Assessment.

156

Master your semester with Scribd & The New York Times

Special offer for students: Only $4.99/month.

Master your semester with Scribd & The New York Times

Cancel anytime.