Professional Documents
Culture Documents
Standard Setting Report
Standard Setting Report
Overview of Standard Setting Process 1 2010 Maine PAAP Standard Setting Report
Chapter 1. OVERVIEW OF STANDARD SETTING
The purpose of this report is to summarize the activities of the standard setting meeting for Maine’s
Personalized Alternate Assessment Portfolio (PAAP) in reading and mathematics (grades 2–7, 10, and 11),
science (grades 5, 8, and 11) and writing (grades 4, 7, and 11). The PAAP standard setting meeting was held
between June 27 and 29, 2010. In all, there were 14 panels with 70 panelists participating in the process.
Eight panels met for two days, and each panel established cuts for two grade level combinations (either two
reading grades or two mathematics grades). The remaining six panels met for one day and established cuts for
a single grade and content area combination. The configuration of the panels is shown in Table 1-1. Note that
some panelists participated in multiple content areas.
Table 1-1. 2010 PAAP Standard Setting: Configuration of Standard Setting Panels
Number of June 28 June 29 June 30
Panel
panelists
Panel 1 8 Math 2 Math 3
Panel 2 8 Math 4 Math 5
Panel 3 10 Math 6 Math 7
Panel 4 9 Math 10 Math 11
Panel 5 11 Reading 2 Reading 3
Panel 6 9 Reading 4 Reading 5
Panel 7 8 Reading 6 Reading 7
Panel 8 7 Reading 10 Reading 11
Panel 9 7 Science 5
Panel 10 7 Science 8
Panel 11 7 Science 11
Panel 12 10 Writing 4
Panel 13 9 Writing 7
Panel 14 8 Writing 11
A modified version of the body of work method was used for setting standards for the PAAP. The
body of work standard setting method was developed specifically for use with assessments that are designed to
allow for a range of student responses, such as portfolio- or performance-based assessments (Kingston, Kahl,
Sweeney, & Bay, 2001). A modified version of the method has been in use for a number of years that
substantially reduces the logistical burden of the procedure and has been found to yield reasonable and
defensible cutpoints. In the body of work method, panelists are presented with samples of actual student work
(in this case, student portfolios) and make their judgments based on those samples. Specifically, panelists
examine each student portfolio and determine which achievement level best matches the particular
knowledge, skills, and abilities the student exhibits through his or her performance on the work sample. This
report is organized into three major sections, describing tasks completed prior to, during, and after the
standard setting meeting.
Overview of Standard Setting Process 2 2010 Maine PAAP Standard Setting Report
Chapter 2. TASKS COMPLETED PRIOR TO STANDARD SETTING
2.1 Creation of Achievement Level Descriptors
The Achievement Level Descriptors (ALDs) describe the set of knowledge, skills, and abilities that
students in each achievement level are expected to display. Staff at the Maine Department of Education
(MDOE) created these draft descriptors prior to the standard setting meeting, where they were presented to
the panelists. The draft ALDs are provided in Appendix A.
▪ Meeting agenda
▪ Nondisclosure agreement
▪ ALDs
▪ Samples of student portfolios
▪ Rating forms
▪ Evaluation forms
Copies of the ALDs, meeting agenda, nondisclosure form, a sample rating form, and evaluation forms
are included in Appendices A through E.
Tasks Prior to Standard Setting 3 2010 Maine PAAP Standard Setting Report
2.4 Preparation of Presentation Materials
The PowerPoint presentation used in the opening session was prepared prior to the meeting. The
presentation was designed to give panelists an overview of the assessment and how it is scored as well as a
preview of what to expect throughout the standard setting process. A copy of the presentation is included in
Appendix F.
Tasks Prior to Standard Setting 4 2010 Maine PAAP Standard Setting Report
disabilities. In addition, to the extent possible, panels were assembled so as to reflect a balance of gender,
race/ethnicity, and geographic location. A list of the panelists and their affiliations is included in Appendix H.
Tasks Prior to Standard Setting 5 2010 Maine PAAP Standard Setting Report
Chapter 3. TASKS COMPLETED DURING STANDARD SETTING
3.1 Overview of Body of Work Method
The body of work standard setting method was developed specifically for use with assessments that
are designed to allow for a range of student responses, such as portfolio- and performance-based assessments.
For a number of years a modified version of the method has been in use that substantially reduces the logistical
burden of the procedure and has been found to yield reasonable and defensible cutpoints. Panelists were asked
to evaluate each work sample from a holistic perspective before classifying it into a single achievement level.
3.2 Orientation
With regard to panelist training, Standards for Educational and Psychological Testing states the
following:
Care must be taken to assure that judges understand what they are to do. The
process must be such that well-qualified judges can apply their knowledge
and experience to reach meaningful and relevant judgments that accurately
reflect their understanding and intentions. (AERA/APA/NCME, 1999, p. 54)
The training of the panelists began with a general orientation at the start of the standard setting
meeting. The purpose of the orientation was to ensure that all panelists received the same information about
the need for and goals of standard setting and about their part in the process. First, the MDOE provided some
pertinent context about the PAAP program and an introduction to the issues of standard setting. Second, the
PAAP program manager provided an overview of the assessment, including alternate grade level expectations,
task bank selection, administration, and scoring. Next, a Measured Progress psychometrician presented a brief
overview of the body of work procedure and the activities that would occur during the standard setting
meeting. Once the general orientation was complete, each panel convened in a breakout room, where the
panelists received more detailed training and completed the standard setting activities.
Tasks During Standard Setting 6 2010 Maine PAAP Standard Setting Report
3.4 Review of Achievement Level Definitions
The second step in the process, once the panelists convened into their content area and grade level
groups, was to discuss the Achievement Level Descriptors (ALDs). This important step was designed to ensure
that panelists thoroughly understood the knowledge, skills, and abilities needed for students to be classified
into achievement levels (Substantially Below Proficient, Partially Proficient, Proficient, and Proficient
With Distinction). Panelists first reviewed the ALDs on their own and then participated in group discussion
of the ALDs, clarifying the description for each achievement level. The discussions focused on the knowledge,
skills, and abilities that differentiated adjacent achievement levels. Bulleted lists of characteristics for each
level were generated based on the group discussion and were posted in the room for panelists to refer to during
the rounds of ratings.
Tasks During Standard Setting 7 2010 Maine PAAP Standard Setting Report
Table 3-1. 2010 PAAP Standard Setting: Round 1 Mathematics
Average Standard Raw score Percent of
Grade Achievement level
cut error Min Max students
PD 62.6 1.6 63 69 3.4
2 Proficient 45.9 2.1 46 62 21.0
Partially Proficient 24.9 2.5 25 45 50.0
SBP NA NA 0 24 25.6
PD 70.2 1.6 NA NA NA
Proficient 52.9 2.8 53 69 35.6
3
Partially Proficient 25.8 1.9 26 52 41.5
SBP NA NA 0 25 22.9
PD 65.4 1.1 66 69 12.0
Proficient 42.8 2.3 43 65 50.9
4
Partially Proficient 20.9 0.9 21 42 28.6
SBP NA NA 0 20 8.5
PD 65.5 1.3 66 69 36.4
Proficient 52.0 2.4 53 65 27.3
5
Partially Proficient 25.2 1.0 26 52 20.9
SBP NA NA 0 25 15.5
PD 85.5 1.6 86 99 10.1
Proficient 57.0 2.1 58 85 49.8
6
Partially Proficient 25.7 1.7 26 57 26.0
SBP NA NA 0 25 14.1
PD 91.4 1.0 92 99 19.8
Proficient 51.1 1.5 52 91 58.5
7
Partially Proficient 25.0 0.9 25 51 13.5
SBP NA NA 0 24 8.2
PD 120.3 0.8 121 129 7.9
Proficient 83.6 4.3 84 120 33.5
10
Partially Proficient 36.3 3.2 37 83 39.8
SBP NA NA 0 36 18.8
PD 125.9 5.8 126 129 9.5
Proficient 83.2 4.3 84 125 34.4
11
Partially Proficient 24.1 11.9 25 83 42.3
SBP NA NA 0 24 13.7
Tasks During Standard Setting 8 2010 Maine PAAP Standard Setting Report
Grade Average Standard Raw score Percent of
Achievement level
cut error Min Max students
Partially Proficient 14.9 1.2 15 29 25.9
4
SBP NA NA 0 14 8.4
PD 43.3 1.0 44 46 42.3
Proficient 24.5 1.1 25 43 38.9
5
Partially Proficient 12.6 0.6 13 24 15.4
SBP NA NA 0 12 3.4
PD 61.9 0.9 62 66 5.1
Proficient 50.7 1.6 51 61 31.2
6
Partially Proficient 31.6 1.4 32 50 33.3
SBP NA NA 0 31 30.3
PD 63.1 0.6 64 66 24.8
Proficient 32.8 1.6 33 63 51.8
7
Partially Proficient 17.1 0.7 18 32 14.7
SBP NA NA 0 17 8.7
PD 85.5 0.3 86 86 1.6
Proficient 65.8 5.5 66 85 27.7
10
Partially Proficient 21.8 2.7 22 65 53.7
SBP NA NA 0 21 17.0
PD 87.1 2.1 NA NA NA
Proficient 56.2 2.3 57 86 45.9
11
Partially Proficient 32.5 1.2 33 56 30.6
SBP NA NA 0 32 23.6
Tasks During Standard Setting 9 2010 Maine PAAP Standard Setting Report
Average Standard Raw score Percent of
Grade Achievement level
cut error Min Max students
4 SBP NA NA 0 8 29.0
PD 32.8 0.1 33 33 11.7
Proficient 15.7 0.6 16 32 60.2
7
Partially Proficient 11.4 0.2 12 15 14.6
SBP NA NA 0 11 13.6
PD 42.8 2.2 43 43 4.3
Proficient 19.5 3.9 20 42 56.0
11
Partially Proficient 8.8 2.2 9 19 24.1
SBP NA NA 0 8 15.5
Tasks During Standard Setting 10 2010 Maine PAAP Standard Setting Report
Grade Average Standard Raw score Percent of
Achievement level
cut error Min Max students
PD 71.6 1.9 NA NA NA
Proficient 51.5 1.2 52 69 37.6
3
Partially Proficient 25.6 1.4 26 51 39.5
SBP NA NA 0 25 22.9
PD 68.1 0.0 69 69 7.3
Proficient 45.7 1.6 46 68 51.7
4
Partially Proficient 21.5 0.0 22 45 32.1
SBP NA NA 0 21 9.0
PD 66.3 0.9 67 69 33.2
Proficient 51.2 0.4 52 66 31.0
5
Partially Proficient 27.0 0.0 27 51 19.3
SBP NA NA 0 26 16.6
PD 85.0 1.2 86 99 10.1
Proficient 55.5 1.8 56 85 52.0
6
Partially Proficient 25.2 0.5 26 55 23.8
SBP NA NA 0 25 14.1
PD 91.3 0.9 92 99 19.8
Proficient 49.9 1.0 50 91 58.5
7
Partially Proficient 24.9 0.6 25 49 13.5
SBP NA NA 0 24 8.2
PD 119.2 0.9 120 129 7.9
Proficient 87.9 2.3 88 119 31.9
10
Partially Proficient 38.0 2.3 38 87 40.8
SBP NA NA 0 37 19.4
PD 122.4 1.8 123 129 10.8
Proficient 77.1 1.0 78 122 39.0
11
Partially Proficient 31.8 3.0 32 77 34.0
SBP NA NA 0 31 16.2
Tasks During Standard Setting 11 2010 Maine PAAP Standard Setting Report
Grade Average Standard Raw score Percent of
Achievement level
cut error Min Max students
5 Partially Proficient 13.0 0.3 13 23 13.0
SBP NA NA 0 12 3.4
PD 60.6 0.6 61 66 7.7
Proficient 46.6 0.8 47 60 35.5
6
Partially Proficient 28.2 0.6 29 46 30.3
SBP NA NA 0 28 26.5
PD 62.9 0.7 63 66 25.7
Proficient 34.5 1.0 35 62 49.1
7
Partially Proficient 16.6 0.6 17 34 17.0
SBP NA NA 0 16 8.3
PD 85.5 0.0 86 86 1.6
Proficient 54.7 0.8 55 85 48.4
10
Partially Proficient 29.8 0.3 30 54 27.7
SBP NA NA 0 29 22.3
PD 84.5 0.0 85 86 5.0
Proficient 57.4 0.4 58 84 40.9
11
Partially Proficient 35.2 0.8 36 57 27.3
SBP NA NA 0 35 26.9
Tasks During Standard Setting 12 2010 Maine PAAP Standard Setting Report
Average Standard Raw score Percent of
Grade Achievement level error students
cut Min Max
7 SBP NA NA 0 11 13.6
PD 42.1 0.7 43 43 4.3
Proficient 24.3 0.5 25 42 37.5
11
Partially Proficient 11.2 0.6 12 24 42.2
SBP NA NA 0 11 15.9
Tasks During Standard Setting 13 2010 Maine PAAP Standard Setting Report
Average Standard Raw score Percent of
Grade Achievement level
cut error Min Max students
3 SBP NA NA 0 26 24.4
PD 61.1 0.4 62 69 20.5
Proficient 41.2 0.3 42 61 42.3
4
Partially Proficient 21.5 0.0 22 41 28.2
SBP NA NA 0 21 9.0
PD 66.9 0.6 67 69 33.2
Proficient 51.7 0.3 52 66 31.0
5
Partially Proficient 26.8 0.2 27 51 19.3
SBP NA NA 0 26 16.6
PD 84.8 1.2 85 99 10.1
Proficient 55.9 1.5 56 84 52.0
6
Partially Proficient 24.4 0.5 25 55 24.7
SBP NA NA 0 24 13.2
PD 91.3 0.9 92 99 19.8
Proficient 49.9 1.0 50 91 58.5
7
Partially Proficient 24.9 0.6 25 49 13.5
SBP NA NA 0 24 8.2
PD 120.1 1.3 121 129 7.9
Proficient 85.6 1.6 86 120 32.5
10
Partially Proficient 35.5 2.4 36 85 41.4
SBP NA NA 0 35 18.3
PD 122.4 1.8 123 129 10.8
Proficient 77.5 0.6 78 122 39.0
11
Partially Proficient 29.4 2.3 30 77 34.4
SBP NA NA 0 29 15.8
Tasks During Standard Setting 14 2010 Maine PAAP Standard Setting Report
Table 3-10. 2010 PAAP Standard Setting: Round 3 Reading
Average Standard Raw score Percent of
Grade Achievement level
cut error Min Max students
PD 45.2 0.1 46 46 4.5
Proficient 31.7 0.7 32 45 23.7
2
Partially Proficient 17.7 0.3 18 31 49.5
SBP NA NA 0 17 22.2
PD 45.5 0.1 46 46 7.7
Proficient 28.8 0.6 29 45 42.3
3
Partially Proficient 17.4 0.3 18 28 25.0
SBP NA NA 0 17 25.0
PD 42.2 0.2 43 46 21.5
Proficient 24.4 0.3 25 42 54.2
4
Partially Proficient 11.5 0.4 12 24 21.9
SBP NA NA 0 11 2.4
PD 41.6 0.4 42 46 42.3
Proficient 23.9 0.2 24 41 41.3
5
Partially Proficient 13.0 0.3 14 23 12.0
SBP NA NA 0 13 4.3
PD 59.1 1.0 60 66 8.5
Proficient 30.8 0.8 31 59 62.8
6
Partially Proficient 18.9 0.7 19 30 15.0
SBP NA NA 0 18 13.7
PD 64.0 0.5 64 66 24.8
Proficient 37.3 1.5 38 63 44.0
7
Partially Proficient 17.1 0.7 18 37 22.5
SBP NA NA 0 17 8.7
PD 85.5 0.0 86 86 1.6
Proficient 55.6 0.0 56 85 45.7
10
Partially Proficient 29.5 0.0 30 55 30.3
SBP NA NA 0 29 22.3
PD 84.5 0.0 85 86 5.0
Proficient 56.8 0.0 57 84 40.9
11
Partially Proficient 36.5 0.0 37 56 25.2
SBP NA NA 0 36 28.9
Tasks During Standard Setting 15 2010 Maine PAAP Standard Setting Report
Table 3-11. 2010 PAAP Standard Setting: Round 3 Science
Average Standard Raw score Percent of
Grade Achievement level
cut error Min Max students
PD 65.9 1.3 66 69 19.3
Proficient 44.6 0.7 45 65 44.8
5
Partially Proficient 23.5 0.5 24 44 22.1
SBP NA NA 0 23 13.8
PD 92.3 0.7 93 99 8.7
Proficient 58.0 0.5 58 92 59.4
8
Partially Proficient 32.9 0.6 33 57 22.8
SBP NA NA 0 32 9.1
PD 126.1 1.2 127 129 3.5
Proficient 86.5 1.3 87 126 32.0
11
Partially Proficient 49.1 1.4 50 86 34.2
SBP NA NA 0 49 30.3
Tasks During Standard Setting 16 2010 Maine PAAP Standard Setting Report
Figure 3-1. 2010 PAAP Standard Setting: Mathematics: Round 3 Results
0
0
.....-
0
co
C"
0
0)
Q.)
+""'
ro 0
u CD
c
CJ)
+""'
c
Q.)
""C
:::J 0
+""'
CJ) "¢
'+-
0
+""'
c
Q.)
u
I-
Q.)
a_ N0
02 03 04 05 06 07 10 11
Proficient with Distinction
Proficient
Partially Proficient
Substantially Below Proficient
Mathematics: Round 3 Results
Tasks During Standard Setting 17 2010 Maine PAAP Standard Setting Report
Figure 3-2. 2010 PAAP Standard Setting: Reading: Round 3 Results
0
0
........
0
ro
0
0)
Q)
.......
co 0
u ID
c
·CJ) -
........
c
Q)
"0
..:..:.:J. 0
CJ) "¢
'+-
0
........
c
Q)
u
Q)
a_ 0
N
02 03 04 05 06 07 10 11
Proficient with Distinction
Proficient
Partially Proficient
Substantially Below Proficient
Reading: Round 3 Results
Tasks During Standard Setting 18 2010 Maine PAAP Standard Setting Report
Figure 3-3. 2010 PAAP Standard Setting: Science: Round 3 Results
0
0
......
0
co
0
0)
Q)
.......
co
(.)
0
CD
c
·CJ) -
........
c
Q)
"0
..:..:.:J. 0
CJ) "¢
'+-
0
........
c
Q)
(.)
Q)
0
11.. N
05 08 11
Proficient with Distinction
Proficient
Partially Proficient
Substantially Below Proficient
Science: Round 3 Results
Tasks During Standard Setting 19 2010 Maine PAAP Standard Setting Report
Figure 3-4. 2010 PAAP Standard Setting: Writing: Round 3 Results
0
0
.....--
0
CX)
0
0)
Q)
+""'
co
(.) 0
<D
c
(/)
+""'
c
Q)
""0
::J
+""' 0
(/) "¢
'+-
0
+""'
c
Q)
(.)
I-
Q)
a_ 0C'\1
04 07 11
Proficient with Distinction
Proficient
Partially Proficient
Substantially Below Proficient
Writing: Round 3 Results
Tasks During Standard Setting 20 2010 Maine PAAP Standard Setting Report
3.12 Repeat Process for Second Grade and Content Area
(Mathematics and Reading Only)
As mentioned above, eight panels each recommended cutpoints for two grade levels in mathematics
and reading. For those panels, once they had completed the entire process for the first grade level, they (1)
completed a process evaluation, giving their perceptions of the standard setting process and results thus far,
and (2) repeated the entire process (except for the training evaluation) for the second grade level.
3.13 Evaluation
The measurement literature sometimes considers the evaluation process to be another product of the
standard setting process (e.g., Reckase, 2001). To provide evidence of the participants’ views of the standard
setting process, panelists were asked to complete questionnaires throughout the process. The results of the
evaluations are presented in Appendix I.
Tasks During Standard Setting 21 2010 Maine PAAP Standard Setting Report
Chapter 4. TASKS COMPLETED AFTER STANDARD SETTING
Upon conclusion of the standard setting meeting, several important tasks were completed. These tasks
centered on reviewing the standard setting process and addressing anomalies that may have occurred in the
process or in the outcomes, presenting the results to MDOE and making any final revisions or adjustments.
Tasks After Standard Setting 22 2010 Maine PAAP Standard Setting Report
Table 4-1. 2010 PAAP Standard Setting: Mathematics: Adjusted Results
Average Standard Raw score Percent of
Grade Achievement level
cut error Min Max students
PD 61.1 NA 62 69 3.4
Proficient 41.2 NA 42 61 30.7
2
Partially Proficient 21.5 NA 22 41 58
SBP NA NA 0 21 8
PD 61.1 NA 62 69 14.1
Proficient 41.2 NA 42 61 34.6
3
Partially Proficient 21.5 NA 22 41 40
SBP NA NA 0 21 11.2
PD 61.1 0.4 62 69 20.5
Proficient 41.2 0.3 42 61 42.3
4
Partially Proficient 21.5 0 22 41 28.2
SBP NA NA 0 21 9
PD 66.9 0.6 67 69 33.2
Proficient 51.7 0.3 52 66 31
5
Partially Proficient 26.8 0.2 27 51 19.3
SBP NA NA 0 26 16.6
PD 84.8 1.2 85 99 10.1
Proficient 55.9 1.5 56 84 52
6
Partially Proficient 24.4 0.5 25 55 24.7
SBP NA NA 0 24 13.2
PD 91.3 0.9 92 99 19.8
Proficient 55.9 NA 56 91 53.1
7
Partially Proficient 24.9 0.6 25 55 18.8
SBP NA NA 0 24 8.2
PD 121.25 NA 122 129 9
Proficient 81.55 NA 82 121 35.4
HS
Partially Proficient 32.45 NA 33 81 38.4
SBP NA NA 0 32 17.1
Tasks After Standard Setting 23 2010 Maine PAAP Standard Setting Report
Table 4-2. 2010 PAAP Standard Setting: Reading: Adjusted Results
Average Standard Raw score Percent of
Grade Achievement level
cut error Min Max students
PD 42.2 NA 43 46 7.6
Proficient 24.4 NA 25 42 53
2
Partially Proficient 11.5 NA 12 24 36.9
SBP NA NA 0 11 2.5
PD 42.2 NA 43 46 17.3
Proficient 24.4 NA 25 42 45.2
3
Partially Proficient 11.5 NA 12 24 34.6
SBP NA NA 0 11 2.9
PD 42.2 0.2 43 46 21.5
Proficient 24.4 0.3 25 42 54.2
4
Partially Proficient 11.5 0.4 12 24 21.9
SBP NA NA 0 11 2.4
PD 41.6 0.4 42 46 42.3
Proficient 23.9 0.2 24 41 41.3
5
Partially Proficient 13 0.3 14 23 12
SBP NA NA 0 13 4.3
PD 59.1 1 60 66 8.5
Proficient 30.8 0.8 31 59 62.8
6
Partially Proficient 18.9 0.7 19 30 15
SBP NA NA 0 18 13.7
PD 64 0.5 64 66 24.8
Proficient 37.3 1.5 38 63 44
7
Partially Proficient 17.1 0.7 18 37 22.5
SBP NA NA 0 17 8.7
PD 85 NA 85 86 4.2
Proficient 56.2 NA 57 84 40.2
HS
Partially Proficient 33 NA 33 56 30.9
SBP NA NA 0 32 24.7
Tasks After Standard Setting 24 2010 Maine PAAP Standard Setting Report
Figure 4-1. 2010 PAAP Standard Setting: Mathematics: Adjusted Results
0
0
......
0
co
C"
0
0)
Q.)
+""'
ro 0
u c.o
c
CJ)
+""'
Q.)
""C
:::J 0
+""'
CJ) 'V
'+-
0
+""'
Q.)
u
I-
Q.)
a_ 0 N
2 3 4 5 6 7 HS
Proficient with Distinction
Proficient
Partially Proficient
Substantially Below Proficient
Mathematics: Adjusted Results
Tasks After Standard Setting 25 2010 Maine PAAP Standard Setting Report
Figure 4-2. 2010 PAAP Standard Setting: Reading: Adjusted Results
0
0
.....-
0
CX)
C"
0
0)
Q.)
+""'
ro 0
u CD
c
CJ)
+""'
c
Q.)
""C
:::J 0
+""'
CJ) "¢
'+-
0
+""'
c
Q.)
u
I-
Q.)
a_ N0
2 3 4 5 6 7 HS
Proficient with Distinction
Proficient
Partially Proficient
Substantially Below Proficient
Reading: Adjusted Results
Tasks After Standard Setting 26 2010 Maine PAAP Standard Setting Report
After carefully considering the above information, the MDOE remained concerned about a few of the
cut scores in mathematics and writing. Consequently, the department decided to make a final policy adjustment
to some of the cut scores. The resulting DOE-approved operational cut scores, and a more detailed explanation
outlining how the final cut scores were established can be found in Appendix J.
Tasks After Standard Setting 27 2010 Maine PAAP Standard Setting Report
REFERENCES
American Educational Research Association, American Psychological Association, & National Council on
Measurement in Education (1999). Standards for educational and psychological testing. Washington,
DC: American Educational Research Association.
Cizek, G. J., & Bunch, M. B. (2007). Standard setting: Establishing and evaluating performance standards
on tests. Thousand Oaks, CA: Sage Publications.
Kingston, N., Kahl, S., Sweeney, K., & Bay, L. (2001). Setting performance standards using the body of
work method. Setting performance standards: concepts, methods, and perspectives (pp. 219–
248). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.
Reckase, M.D. (2001). Innovative methods for helping standard-setting participants to perform their task:
The role of feedback regarding consistency, accuracy, and impact. In G. J. Cizek (Ed.) Setting
performance standards: concepts, methods, and perspectives (pp. 159-173). Mahwah, NJ: Lawrence
Erlbaum.
U.S. Department of Education (2009). Standards and assessments peer review guidance: Information and
examples for meeting requirements of the No Child Left Behind Act of 2001. Washington, DC: U.S.
Department of Education Office of Elementary and Secondary Education. Retrieved June 10, 2010
from the World Wide Web: www2.ed.gov/policy/elsec/guid/saaprguidance.pdf.
Appendix A—Achievement Level Definitions 1 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 2
Reading achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate reading standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade two is clarified by the level of complexity within the standard.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary
and/or demonstrate understanding of word meaning. The work also generally demonstrates the
ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Partially Proficient
The student’s performance demonstrates the inconsistent ability to identify and decode
unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also
demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary
OR informational texts appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 3 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 3
Reading achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate reading standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade three is clarified by the level of complexity within the
standard.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary
and/or demonstrate understanding of word meaning. The work also generally demonstrates the
ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Partially Proficient
The student’s performance demonstrates the inconsistent ability to identify and decode
unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also
demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary
OR informational texts appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 4 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 4
Reading achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate reading standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade four is clarified by the level of complexity within the standard.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary
and/or demonstrate understanding of word meaning. The work also generally demonstrates the
ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Partially Proficient
The student’s performance demonstrates the inconsistent ability to identify and decode
unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also
demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary
OR informational texts appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 5 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 5
Reading achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate reading standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade five is clarified by the level of complexity within the standard.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary
and/or demonstrate understanding of word meaning. The work also generally demonstrates the
ability to read, comprehend, and interpret literary OR informational texts appropriate to the
student’s instructional level.
Partially Proficient
The student’s performance demonstrates the inconsistent ability to identify and decode
unfamiliar vocabulary and/or demonstrate understanding of word meaning. The work also
demonstrates the incomplete or inconsistent ability to read, comprehend, and interpret literary
OR informational texts appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 6 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 6
Reading achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate reading standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade six is clarified by the level of complexity within the standard.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or
demonstrate understanding of word meaning. The work also generally demonstrates the ability to read,
comprehend, and interpret literary OR informational texts appropriate to the student’s instructional level.
Partially Proficient
The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar
vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
incomplete or inconsistent ability to read, comprehend, and interpret literary OR informational texts
appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 7 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ Grade 7
Reading achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate reading standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade seven is clarified by the level of complexity within the standard.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or
demonstrate understanding of word meaning. The work also generally demonstrates the ability to read,
comprehend, and interpret literary OR informational texts appropriate to the student’s instructional level.
Partially Proficient
The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar
vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
incomplete or inconsistent ability to read, comprehend, and interpret literary OR informational texts
appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 8 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Reading ~ High School
Reading achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate reading standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in reading.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at high school level is clarified by the level of complexity within the standard.
Proficient
The student’s performance demonstrates the ability to identify and decode unfamiliar vocabulary and/or
demonstrate understanding of word meaning. The work also generally demonstrates the ability to read,
comprehend, and interpret literary OR informational texts appropriate to the student’s instructional level.
Partially Proficient
The student’s performance demonstrates the inconsistent ability to identify and decode unfamiliar
vocabulary and/or demonstrate understanding of word meaning. The work also demonstrates the
incomplete or inconsistent ability to read, comprehend, and interpret literary OR informational texts
appropriate to the student’s instructional level.
Appendix A—Achievement Level Definitions 9 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 2
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade 2 is clarified by the level of complexity within the standard.
Proficient
The student’s work demonstrates an understanding of basic concepts in mathematics and
connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient
The student’s work demonstrates an incomplete understanding of basic concepts in mathematics
and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Appendix A—Achievement Level Definitions 10 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 3
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade three is clarified by the level of complexity within the
standard.
Proficient
The student’s work demonstrates an understanding of basic concepts in mathematics and
connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient
The student’s work demonstrates an incomplete understanding of basic concepts in mathematics
and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Appendix A—Achievement Level Definitions 11 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 4
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade four is clarified by the level of complexity within the standard.
Proficient
The student’s work demonstrates an understanding of basic concepts in mathematics and
connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient
The student’s work demonstrates an incomplete understanding of basic concepts in mathematics
and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Appendix A—Achievement Level Definitions 12 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 5
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade five is clarified by the level of complexity within the standard.
Proficient
The student’s work demonstrates an understanding of basic concepts in mathematics and
connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient
The student’s work demonstrates an incomplete understanding of basic concepts in mathematics
and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Appendix A—Achievement Level Definitions 13 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 6
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade six is clarified by the level of complexity within the standard.
Proficient
The student’s work demonstrates an understanding of basic concepts in mathematics and
connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient
The student’s work demonstrates an incomplete understanding of basic concepts in mathematics
and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Appendix A—Achievement Level Definitions 14 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ Grade 7
Mathematics achievement level definitions describe the quality of a student’s responses on statelevel
alternate assessments in relation to the alternate mathematics standards. These definitions
serve as the foundation for achievement level definitions for the alternate grade level
expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade seven is clarified by the level of complexity within the
standard.
Proficient
The student’s work demonstrates an understanding of basic concepts in mathematics and
connections among central ideas. The student’s responses demonstrate basic ability to solve
problems, including performing procedures and providing solutions. The student’s work may
contain minor errors.
Partially Proficient
The student’s work demonstrates an incomplete understanding of basic concepts in mathematics
and inconsistent connections among central ideas. The student’s responses demonstrate limited
ability to solve problems, including performing procedures and providing solutions. Problem
solving strategies may be flawed and procedures preformed inaccurately.
Appendix A—Achievement Level Definitions 15 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Mathematics ~ High School
Mathematics achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate mathematics standards. These definitions serve as the
foundation for achievement level definitions for the alternate grade level expectations in mathematics.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at the high school level is clarified by the level of complexity within the
standard.
Proficient
The student’s work demonstrates an understanding of basic concepts in mathematics and connections
among central ideas. The student’s responses demonstrate basic ability to solve problems, including
performing procedures and providing solutions. The student’s work may contain minor errors.
Partially Proficient
The student’s work demonstrates an incomplete understanding of basic concepts in mathematics and
inconsistent connections among central ideas. The student’s responses demonstrate limited ability to solve
problems, including performing procedures and providing solutions. Problem solving strategies may be
flawed and procedures preformed inaccurately.
Appendix A—Achievement Level Definitions 16 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Science ~ Grade 5
Science achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate science standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in science.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade five is clarified by the level of complexity within the standard.
Proficient
The student’s work demonstrates a general understanding of essential concepts in science and
connections among central ideas. The student’s response demonstrates the ability to utilize
information and solve problems and explain central concepts. Student work may contain minor
errors.
Partially Proficient
The student’s work demonstrates incomplete understanding of essential concepts in science and
inconsistent connections among central ideas. The student’s response demonstrates some ability
to utilize information and solve problem. The quality of the responses is inconsistent. Explanation
of concepts may be incomplete or unclear.
Appendix A—Achievement Level Definitions 17 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Science ~ Grade 8
Science achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate science standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in science.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade eight is clarified by the level of complexity within the standard.
Proficient
The student’s work demonstrates a general understanding of essential concepts in science and
connections among central ideas. The student’s response demonstrates the ability to utilize
information and solve problems and explain central concepts. Student work may contain minor
errors.
Partially Proficient
The student’s work demonstrates incomplete understanding of essential concepts in science and
inconsistent connections among central ideas. The student’s response demonstrates some ability
to utilize information and solve problem. The quality of the responses is inconsistent. Explanation
of concepts may be incomplete or unclear.
Appendix A—Achievement Level Definitions 18 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Science ~ High School
Science achievement level definitions describe the quality of a student’s responses on state-level alternate
assessments in relation to the alternate science standards. These definitions serve as the foundation for
achievement level definitions for the alternate grade level expectations in science.
Skills associated with these achievement level definitions may include some of the following:
• sorting objects into groups using • describing ways in which the needs of
physical properties a plant and/or animal are met by its
• describing physical properties of objects environment
and materials • sorting living things based on external
• using observable characteristics to features
describe physical changes • matching organisms to the
• identifying chemical and physical environment in which they live
changes • identifying organisms that once lived
• identifying organisms that are similar on Earth but no longer exist
and different based on external features • identifying examples of fossils and/or
• describing how plants and/or animals explaining how fossils are used to
look help us understand the past
Appropriate performance at the high school level is clarified by the level of complexity within the
standard.
Proficient
The student’s work demonstrates a general understanding of essential concepts in science and
connections among central ideas. The student’s response demonstrates the ability to utilize
information and solve problems and explain central concepts. Student work may contain minor
errors.
Partially Proficient
The student’s work demonstrates incomplete understanding of essential concepts in science and
inconsistent connections among central ideas. The student’s response demonstrates some ability
to utilize information and solve problem. The quality of the responses is inconsistent. Explanation
of concepts may be incomplete or unclear.
Appendix A—Achievement Level Definitions 19 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Writing ~ Grade 4
Writing achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate writing standards. These definitions also serve
as the foundation for achievement level definitions for the alternate grade level expectations in
writing.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade four is clarified by the level of complexity within the standard.
Proficient
The student’s performance demonstrates an understanding of basic concepts in writing. The
student’s work demonstrates the ability to compose a response that is organized, accurate and
focused. Some errors may occur but do not interfere with meaning.
Partially Proficient
The student’s performance demonstrates an incomplete understanding of basic concepts in
writing. The student’s work demonstrates the ability to compose a response that may be
inconsistent and/or limited in its organization, accuracy and/or focus. Some errors may occur that
interfere with meaning.
Appendix A—Achievement Level Definitions 20 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Writing ~ Grade 7
Writing achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate writing standards. These definitions also serve
as the foundation for achievement level definitions for the alternate grade level expectations in
writing.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at grade seven is clarified by the level of complexity within the
standard.
Proficient
The student’s performance demonstrates an understanding of basic concepts in writing. The
student’s work demonstrates the ability to compose a response that is organized, accurate and
focused. Some errors may occur but do not interfere with meaning.
Partially Proficient
The student’s performance demonstrates an incomplete understanding of basic concepts in
writing. The student’s work demonstrates the ability to compose a response that may be
inconsistent and/or limited in its organization, accuracy and/or focus. Some errors may occur that
interfere with meaning.
Appendix A—Achievement Level Definitions 21 2010 Maine PAAP Standard Setting Report
Personalized Alternate Assessment Portfolio (PAAP)
Achievement Level Definitions for Writing ~ High School
Writing achievement level definitions describe the quality of a student’s responses on state-level
alternate assessments in relation to the alternate writing standards. These definitions also serve
as the foundation for achievement level definitions for the alternate grade level expectations in
writing.
Skills associated with these achievement level definitions may include some of the following:
Appropriate performance at the high school level is clarified by the level of complexity within
the standard.
Proficient
The student’s performance demonstrates an understanding of basic concepts in writing. The
student’s work demonstrates the ability to compose a response that is organized, accurate and
focused. Some errors may occur but do not interfere with meaning.
Partially Proficient
The student’s performance demonstrates an incomplete understanding of basic concepts in
writing. The student’s work demonstrates the ability to compose a response that may be
inconsistent and/or limited in its organization, accuracy and/or focus. Some errors may occur that
interfere with meaning.
Appendix A—Achievement Level Definitions 22 2010 Maine PAAP Standard Setting Report
Appendix B—AGENDAS
Name (printed)
Name (signature)
Date
Appendix D—Sample Rating Form 1 2010 Maine PAAP Standard Setting Report
Round: ID Number:
Appendix D—Sample Rating Form 3 2010 Maine PAAP Standard Setting Report
Appendix E—S AMPLE EVALUATION
Undecided
Disagree
Disagree
Strongly
Strongly
Agree
Agree
I understand the goals of the standard setting meeting. □ □ □ □ □
I understand the procedures we are using to set standards. □ □ □ □ □
I understand how to use the standard setting materials. □ □ □ □ □
I understand the differences between the achievement levels. □ □ □ □ □
I understand how to make the cut score judgment. □ □ □ □ □
I know what tasks to expect for the remainder of the meeting. □ □ □ □ □
I am confident in my understanding of the standard setting task. □ □ □ □ □
Please indicate any areas in which you would like more information before you continue.
Please indicate any questions you may have about the remainder of the standard setting meeting.
Undecided
Disagree
Disagree
Strongly
Strongly
Agree
Agree
I understood how to make the cut score judgments. □ □ □ □ □
I understood how to use the materials provided. □ □ □ □ □
I understood how to record my judgments. □ □ □ □ □
I think the procedures make sense. □ □ □ □ □
I am sufficiently familiar with the assessment. □ □ □ □ □
I understand the differences between the achievement levels. □ □ □ □ □
Please indicate any questions you may have about the remainder of the standard setting meeting.
Extremely
influential
influential
Not at all
Undecided
Disagree
Disagree
Strongly
Strongly
Agree
Agree
I understood the goals of the standard setting meeting. □ □ □ □ □
I understood the procedures we used to set standards. □ □ □ □ □
The facilitator helped me understand the process. □ □ □ □ □
The materials contained the information needed to set standards. □ □ □ □ □
I understood how to use the materials provided. □ □ □ □ □
The achievement level descriptors were clear. □ □ □ □ □
I understood how to make the cut score judgments. □ □ □ □ □
I understood how to use the feedback provided after each round. □ □ □ □ □
I understood how to use the impact data. □ □ □ □ □
I understood how the cut scores were calculated. □ □ □ □ □
The facilitator was able to get answers to my questions. □ □ □ □ □
Sufficient time was allotted for training on the standard setting tasks. □ □ □ □ □
Sufficient time was allotted to complete the standard setting tasks. □ □ □ □ □
Extremely
Not at all
useful
useful
The opening session. □ □ □ □ □
The small group activities. □ □ □ □ □
Becoming familiar with the assessment. □ □ □ □ □
Articulating the differences between the achievement levels. □ □ □ □ □
Discussions with other participants. □ □ □ □ □
Extremely
influential
influential
Not at all
Appendix F—Opening Session PowerPoint 1 2010 Maine PAAP Standard Setting Report
Slide 1
Maine’s Personalized
Alternate Assessment
Portfolio (PAAP)
Standard Setting
Slide 2
Introductions
Maine’s Department of Education
- Susan Fossett, PAAP and Accommodations
Coordinator
- Peter Bernard, PAAP Assistant
Slide 3
Introductions (cont.)
Data
Writing
Grade 4: Amanda Breitmaier
Grade 7: Susan Izard
Grade 11: Jake Goldsmith
Slide 6
Agenda
Slide 9
2009-10 Test Blueprint
Slide 10
Visual Guide to PAAP Requirements
Slide 12
Slide 13
Slide 14
Slide 15
Slide 16
Slide 17
1. Entry Slip
2. Task Description
4. Task Summary
Slide 19 The Entry Slip
• Teacher fills in Student
Name and Grade.
• The Content Area,
AGLE, Level of
Complexity (LoC), and
the AGLE/Indicator is
pre-populated.
• By design, text provided
in gray shaded cells is
information for the
teacher and is pre-
populated.
• Areas with white or no
shading indicates the
teacher should fill in the
missing information.
C
sleeping
sleeping sleeping
smiling
X
sleeping
sleeping smiling
s miling
Option 2
C
sleeping
sleeping sl eeping
smiling
C
eating
person laughing eating eating
C
eating
eating
person laughing eating
eating
eating X
laughing
person laughing
Slide 25 The Completed Work Template
1. Teacher fills in Liam 12/11/10
C
2. Teacher corrects
C
student responses
C
using the Responses
X
Expected from
Student section on
the Task Description.
Task 1 Summary
Liam 12/11/10
67
Unscorable:
● Hand-over-Hand
● Altering Items/Tasks (task no longer connects to
the AGLE.)
• Independent
• Providing encouragement
• Completing Task by using
augmentative/alternative means of
communication
• Repeating directions
• Reacting to a student
• Rereading a passage (except for required
reading)
• Reminding a student to stay focused
X .
√ Level of Complexity
√ Level of Accuracy
√ Level of Assistance
Slide 34 Who scored the PAAPs?
● Measured Progress scorers scored 2,024
PAAPs in early May using the PAAP Task
Scoring Rubric. Each PAAP was scored twice
and sometimes a third time if scores between
Scorer #1 and #2 were not an exact match.
● Measured Progress provided training.
● All scorers passed a qualifying test.
● MDOE staff were present at scoring and
available to answer questions.
Note:
This session is intended to be an overview.
Slide 41
Content Achievement
vs.
Standards Standards
Content standards (AGLE) = “What”
Describe the knowledge and skills students are expected to
demonstrate by content area and grade span
Slide 42
What is Your Job?
Slide 44
Achievement Continuum
Slide 45
Based on Achievement Level
Descriptors, you will recommend cut
scores…
Data-collection
Policy-making/Decision-making
Slide 47
Final Recommendations
Slide 48
Slide 50
How: The Body of Work Method
Slide 54
Achievement Level Descriptors
Individual review of Achievement Level Descriptors
Slide 57
Slide 58
Slide 59
Please Note:
Slide 60
Round 1
Working Individually:
Review each portfolio
Focus on the knowledge, skills and abilities being
demonstrated in the portfolio
Determine which Achievement Level Descriptor best
matches the knowledge, skills and abilities
demonstrated in the portfolio
Classify the portfolio into the appropriate
achievement level
Complete the rating form
Slide 61
Round 2
Working as a Group:
Discuss your portfolio classifications in relation to
The average round 1 results
The other panelists
The knowledge, skills and abilities
Working Individually:
Determine which Achievement Level Descriptor best
matches the knowledge, skills and abilities
demonstrated in the portfolio
Classify the portfolio into the appropriate
achievement level
Complete the rating form
Slide 62
Round 3
Working as a Group:
Discuss your portfolio classifications in relation to
The round 2 results & impact data
The other panelists
The knowledge, skills and abilities
Working Individually:
Determine which Achievement Level Descriptor best
matches the knowledge, skills and abilities
demonstrated in the portfolio
Classify the portfolio into the appropriate
achievement level
Complete the rating form
Slide 65
Evaluation
Slide 66
Slide 69
Top 10 Misconceptions About Standard
Setting
1. Disagreement is bad.
Slide 70
What Next?
☺ Some meeting logistics
☺ After this session, you will break into grade
groups and complete the standard setting
process!
– First grade/content
• Review the portfolios
• Discuss the Achievement Level Descriptors
• Round 1, 2, 3
– Repeat for second grade/content
– Cross grade policy forum
– Evaluation
Slide 71
Room Assignments
Overview
The Reading and Mathematics groups will each be setting standards for two grade levels. The
panels will complete the standard setting activities for the first grade level – discussing the
Achievement Level Descriptors and completing the three rounds of ratings – then will repeat the
entire process for the second grade level. For the first grade level, the panelists will complete two
evaluation forms: a training evaluation before starting round 1 and a procedural evaluation after
round 3. For the second grade level, the panelists will not need to fill out either of these
evaluations; instead, the panelists will complete the final evaluation after the Cross Grade Policy
Forum, which occurs at the very end of the process.
Introductions
Review Materials
1) Have the panelists take a few minutes to briefly look through the AGLEs
2) Have the panelists briefly review about every fifth portfolio, noting the increasing level of
performance.
3) When they are done, allow a minute or two for comments or questions.
4) Familiarize the panelists with the rating sheet and explain how to complete it:
a. Place one (and only one) “X” in each row
b. They can place the X’s at the low or high end of the box, but they must clearly be
within one box: no straddling!
Appendix G—Facilitators’ Scripts 3 2010 Maine PAAP Standard Setting Report
Discuss Achievement Level Descriptors
The purpose of this activity is for the panelists to come to consensus about what characterizes
students in each of the four achievement level categories. This activity is critical since the ratings
panelists will be making in Rounds 1 through 3 will be based on these understandings.
Activities:
1. Introduce task. In this activity they will:
a. Individually review the Achievement Level Descriptors that describe the main
characteristics that define students in each achievement level category; and
b. discuss Descriptors as a group.
2. Have panelists individually review the Achievement Level Descriptors for each level. They
can make notes if they like. The goal here is for the panelists to come to a common
understanding of what it means to be in each achievement level. It is not unusual for
panelists to disagree with the Descriptors they will see; almost certainly there will be some
panelists who will want to change them. However, the task at hand is for panelists to have a
common understanding of what knowledge, skills, and abilities are described by each
Achievement Level Descriptor.
3. After individually reviewing the Descriptors, have the panelists discuss each achievement
level as a group, starting with Partially Proficient. The panelists will discuss the
characteristics a student must demonstrate in order to be classified in the Partially Proficient
category. Or, put another way, the most important characteristics that distinguish a
Substantially Below Proficient student from a student in the Partially Proficient category. They
will then repeat this process for the Proficient and Proficient with Distinction categories. The
purpose of this step is to have a collegial discussion in which to bring up/clarify any issues
or questions that any individual may have and to reach consensus on an understanding of
the Descriptors.
4. Have the panelists identify the most important characteristics describing students at each
achievement level and record those as bulleted lists on chart paper. These should be posted
on the walls for panelists to refer to as they complete the three rounds of rating.
After completing the discussion of the Achievement Level Descriptors for the first grade level, have
panelists fill out the training evaluation form before proceeding to Round 1. Before you start the
Round 1 activities, scan the completed evaluations to see if there are any problems or concerns that
need to be addressed before proceeding. Return the completed evaluations to the data analysis
room at the next convenient opportunity. It is not necessary to complete the training evaluation
form for the second grade level.
Appendix G—Facilitators’ Scripts 4 2010 Maine PAAP Standard Setting Report
Round 1 Ratings
Overview of Round 1: The primary purpose of Round 1 is to ask the panelists to make their initial
determination as to which achievement level category each portfolio should be classified into. In
this round, panelists will be working individually, without discussion with their colleagues.
Activities:
1. Make sure panelists have the following materials:
a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
2. Orient panelists to the set of portfolios. Point out that the portfolios are presented in order,
from lowest scoring to highest. Make sure panelists understand that, even though the
portfolios are presented from lowest- to highest-scoring, their own ratings do not need to be
in strictly increasing order.
3. Starting with the first portfolio, the panelists will review each portfolio in turn. As they are
reviewing the portfolios, the panelists should keep in mind the Achievement Level
Descriptors. They should consider the knowledge, skills, and abilities demonstrated by
each and how they relate to the definitions of the achievement levels. The purpose of this
step is for panelists to make their initial determinations as to how the portfolios should be
categorized into the four achievement levels. The panelists are free to make notes on the
portfolios, sort them into piles, use sticky notes, or use whatever system helps them to keep
track of their categorizations.
4. Panelists may want to take notes as they work if there are particular points they would like
to discuss with their colleagues in Round 2.
a. Have panelists write their ID and round number on the rating form. The ID number
is on their name tags.
b. Briefly remind them how to fill in the rating form.
c. Answer questions the panelists may have about the work in Round 1.
d. Once everyone understands what they are to do in Round 1, tell them to begin.
5. As panelists complete the task, ask them to carefully inspect their rating forms to ensure
they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio must be assigned to one and only one achievement level.
c. Reiterate that although the portfolios are presented in order from lowest- to highest-
scoring, the panelists’ category assignments do not need to be in strictly increasing
order.
6. Facilitators should bring all the completed rating forms together to the data analysis work
room for tabulation. Prior to submitting them, however, using a show of hands, indicate on
a piece of chart paper how many panelists assigned each portfolio to each achievement level
category. This chart will be used for the Round 2 discussions.
Overview of Round 2: In Round 2, the panelists will have an opportunity to discuss their Round 1
placements and to revise their ratings on the basis of that discussion. Prior to beginning the Round
2 discussions, the psychometrician will share the group average cut points based on the Round 1
ratings.
Focusing on any portfolios for which there was disagreement as to how they should be categorized,
the panelists will discuss why they categorized each portfolio as they did, making sure that all
different points of view are included in the discussion.
Once panelists have reviewed and discussed the Round 1 categorizations, the panelists will make
their Round 2 ratings.
Activities:
1. Make sure panelists have the following materials:
a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
2. Have panelists write their ID number and Round 2 on the rating form.
3. Provide an overview of Round 2. Paraphrase the following:
a. As in Round 1, the primary purpose is to categorize each portfolio into the
achievement level category where you believe it belongs.
b. Each panelist needs to base his/her judgments on his/her experience with the
content area, understanding of students, the definition of each achievement level
category, discussions with other panelists, and the knowledge, skills, and abilities
required to answer each item.
4. Show the panelists how the portfolios would be categorized based on the room average
Round 1 cut point placements.
5. Remind panelists that they will be discussing each portfolio with their colleagues, but that
they will be categorizing the portfolios individually. It is not necessary for the panelists to
reach consensus about how to categorize each portfolio.
6. Give panelists an opportunity to ask questions about the task for Round 2.
7. Beginning with the first portfolio for which there is disagreement as to how it should be
categorized, the panelists should begin discussing the categorization of the portfolios
according to the Round 1 ratings.
a. Panelists only need to discuss those portfolios for which there was disagreement as
to how they should be categorized.
b. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
c. If the panelists hear a logic/rationale/argument that they did not consider and that
they feel is compelling, then they may adjust their ratings to incorporate that
information.
d. On the basis of the discussions and the feedback presented, panelists should make
their round 2 ratings.
Appendix G—Facilitators’ Scripts 6 2010 Maine PAAP Standard Setting Report
e. The group does not have to achieve consensus. If panelists honestly disagree, that is
fine. We are trying to get the best judgment of each panelist. Panelists should not feel
compelled or coerced into making a rating with which they disagree.
Encourage the panelists to use the discussion and feedback to assess how stringent or
lenient a judge they are. If a panelist is categorizing portfolios consistently higher or
lower than the group, he or she may have a different understanding of the Achievement
Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,
but that disagreement should be based on a common understanding of the
Achievement Level Descriptors.
8. When the group has completed their Round 2 ratings, collect the rating forms. When you
collect the rating forms, carefully inspect them to ensure they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio must have one (and only one) rating.
9. Facilitators should bring all the completed rating forms together to the data analysis work
room for tabulation. Prior to submitting them, however, using a show of hands, indicate on
a piece of chart paper how many panelists assigned each portfolio to each achievement level
category. This chart will be used for the Round 3 discussions.
Appendix G—Facilitators’ Scripts 7 2010 Maine PAAP Standard Setting Report
Round 3 Ratings
Overview of Round 3: In Round 3, the panelists will have a final opportunity to discuss their
Round 2 placements and to revise their ratings on the basis of that discussion. Prior to beginning
the Round 3 discussions, the psychometrician will share the Round 2 results with the group,
including the group average cut points and impact data, i.e., the approximate percentage of
students who would be classified into each achievement level category based on the room average
cut points from Round 2.
Once panelists have reviewed and discussed the Round 2 categorizations, they will make their final
ratings.
Activities:
1. Make sure panelists have the following materials:
a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
2. Have panelists write their ID number and Round 3 on the rating form.
3. Provide an overview of Round 3. Paraphrase the following:
a. As in Round 2, the primary purpose is to categorize each portfolio into the
achievement level category where you believe it belongs.
b. Each panelist needs to base his/her judgments on his/her experience with the
content area, understanding of students, the definition of each achievement level
category, discussions with other panelists, and the knowledge, skills, and abilities
required to answer each item.
4. Review the feedback information with the panelists.
a. Show the panelists how the portfolios would be categorized based on the room
average Round 2 cut point placements.
b. Go over the impact data, explaining that if the Round 2 ratings were to be used to set
the final cut points, these are the approximate percentages of students who would be
classified into each achievement level category.
5. Remind panelists that they will be discussing each portfolio with their colleagues, but that
they will be categorizing the portfolios individually. It is not necessary for the panelists to
reach consensus about how to categorize each portfolio.
6. Give panelists an opportunity to ask questions about the feedback information or about the
task for Round 3.
7. Beginning with the first portfolio for which there is disagreement as to how it should be
categorized, the panelists should begin discussing the categorization of the portfolios
according to the Round 2 ratings.
a. Panelists should discuss the portfolios for which there was disagreement as to how
they should be categorized, focusing in particular on thoe portfolios around the cuts.
b. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
c. If the panelists hear a logic/rationale/argument that they did not consider and that
they feel is compelling, then they may adjust their ratings to incorporate that
information.
Appendix G—Facilitators’ Scripts 8 2010 Maine PAAP Standard Setting Report
d. On the basis of the discussions and the feedback presented, panelists should make
their final ratings.
e. The group does not have to achieve consensus. If panelists honestly disagree, that is
fine. We are trying to get the best judgment of each panelist. Panelists should not feel
compelled or coerced into making a rating with which they disagree.
Encourage the panelists to use the discussion and feedback to assess how stringent or
lenient a judge they are. If a panelist is categorizing portfolios consistently higher or
lower than the group, he or she may have a different understanding of the Achievement
Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,
but that disagreement should be based on a common understanding of the
Achievement Level Descriptors.
8. When the group has completed their final ratings, collect the rating forms. When you collect
the rating forms, carefully inspect them to ensure they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio for Round 3 must have one (and only one) rating.
c. Return the completed rating forms to the data analysis work room.
After the panelists have completed the three rounds of ratings and filled in the procedural
evaluation, they will then repeat the standard setting activities (except the training and procedural
evaluations) for the second grade level: discussing the Achievement Level Descriptors and the
three rounds of ratings.
Introductions
Review Materials
5) Have the panelists take a few minutes to briefly look through the AGLEs
6) Have the panelists briefly review about every fifth portfolio, noting the increasing level of
performance.
7) When they are done, allow a minute or two for comments or questions.
The purpose of this activity is for the panelists to come to consensus about what characterizes
students in each of the four achievement level categories. This activity is critical since the ratings
panelists will be making in Rounds 1 through 3 will be based on these understandings.
Appendix G—Facilitators’ Scripts 10 2010 Maine PAAP Standard Setting Report
Activities:
5. Introduce task. In this activity they will:
c. Individually review the Achievement Level Descriptors that describe the main
characteristics that define students in each achievement level category; and
d. discuss Descriptors as a group.
6. Have panelists individually review the Achievement Level Descriptors for each level. They
can make notes if they like. The goal here is for the panelists to come to a common
understanding of what it means to be in each achievement level. It is not unusual for
panelists to disagree with the Descriptors they will see; almost certainly there will be some
panelists who will want to change them. However, the task at hand is for panelists to have a
common understanding of what knowledge, skills, and abilities are described by each
Achievement Level Descriptor.
7. After individually reviewing the Descriptors, have the panelists discuss each achievement
level as a group, starting with Partially Proficient. The panelists will discuss the
characteristics a student must demonstrate in order to be classified in the Partially Proficient
category. Or, put another way, the most important characteristics that distinguish a
Substantially Below Proficient student from a student in the Partially Proficient category. They
will then repeat this process for the Proficient and Proficient with Distinction categories. The
purpose of this step is to have a collegial discussion in which to bring up/clarify any issues
or questions that any individual may have and to reach consensus on an understanding of
the Descriptors.
8. Have the panelists identify the most important characteristics describing students at each
achievement level and record those as bulleted lists on chart paper. These should be posted
on the walls for panelists to refer to as they complete the three rounds of rating.
Round 1 Ratings
Overview of Round 1: The primary purpose of Round 1 is to ask the panelists to make their initial
determination as to which achievement level category each portfolio should be classified into. In
this round, panelists will be working individually, without discussion with their colleagues.
Activities:
7. Make sure panelists have the following materials:
a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
8. Orient panelists to the set of portfolios. Point out that the portfolios are presented in order,
from lowest scoring to highest. Make sure panelists understand that, even though the
portfolios are presented from lowest- to highest-scoring, their own ratings do not need to be
in strictly increasing order.
9. Starting with the first portfolio, the panelists will review each portfolio in turn. As they are
reviewing the portfolios, the panelists should keep in mind the Achievement Level
Descriptors. They should consider the knowledge, skills, and abilities demonstrated by
each and how they relate to the definitions of the achievement levels. The purpose of this
step is for panelists to make their initial determinations as to how the portfolios should be
Appendix G—Facilitators’ Scripts 11 2010 Maine PAAP Standard Setting Report
categorized into the four achievement levels. The panelists are free to make notes on the
portfolios, sort them into piles, use sticky notes, or use whatever system helps them to keep
track of their categorizations.
10. Panelists may want to take notes as they work if there are particular points they would like
to discuss with their colleagues in Round 2.
a. Have panelists write their ID and round number on the rating form. The ID number
is on their name tags.
b. Briefly remind them how to fill in the rating form.
c. Answer questions the panelists may have about the work in Round 1.
d. Once everyone understands what they are to do in Round 1, tell them to begin.
11. As panelists complete the task, ask them to carefully inspect their rating forms to ensure
they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio must be assigned to one and only one achievement level.
c. Reiterate that although the portfolios are presented in order from lowest- to highest-
scoring, the panelists’ category assignments do not need to be in strictly increasing
order.
12. Facilitators should bring all the completed rating forms together to the data analysis work
room for tabulation. Prior to submitting them, however, using a show of hands, indicate on
a piece of chart paper how many panelists assigned each portfolio to each achievement level
category. This chart will be used for the Round 2 discussions.
Overview of Round 2: In Round 2, the panelists will have an opportunity to discuss their Round 1
placements and to revise their ratings on the basis of that discussion. Prior to beginning the Round
2 discussions, the psychometrician will share the group average cut points based on the Round 1
ratings.
Focusing on any portfolios for which there was disagreement as to how they should be categorized,
the panelists will discuss why they categorized each portfolio as they did, making sure that all
different points of view are included in the discussion.
Once panelists have reviewed and discussed the Round 1 categorizations, the panelists will make
their Round 2 ratings.
Activities:
10. Make sure panelists have the following materials:
d. Rating form
e. set of portfolios
f. Achievement Level Descriptors
11. Have panelists write their ID number and Round 2 on the rating form.
12. Provide an overview of Round 2. Paraphrase the following:
a. As in Round 1, the primary purpose is to categorize each portfolio into the
achievement level category where you believe it belongs.
b. Each panelist needs to base his/her judgments on his/her experience with the
content area, understanding of students, the definition of each achievement level
category, discussions with other panelists, and the knowledge, skills, and abilities
required to answer each item.
13. Show the panelists how the portfolios would be categorized based on the room average
Round 1 cut point placements.
14. Remind panelists that they will be discussing each portfolio with their colleagues, but that
they will be categorizing the portfolios individually. It is not necessary for the panelists to
reach consensus about how to categorize each portfolio.
15. Give panelists an opportunity to ask questions about the task for Round 2.
16. Beginning with the first portfolio for which there is disagreement as to how it should be
categorized, the panelists should begin discussing the categorization of the portfolios
according to the Round 1 ratings.
f. Panelists only need to discuss those portfolios for which there was disagreement as
to how they should be categorized.
g. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
h. If the panelists hear a logic/rationale/argument that they did not consider and that
they feel is compelling, then they may adjust their ratings to incorporate that
information.
i. On the basis of the discussions and the feedback presented, panelists should make
their round 2 ratings.
Appendix G—Facilitators’ Scripts 13 2010 Maine PAAP Standard Setting Report
j. The group does not have to achieve consensus. If panelists honestly disagree, that is
fine. We are trying to get the best judgment of each panelist. Panelists should not feel
compelled or coerced into making a rating with which they disagree.
Encourage the panelists to use the discussion and feedback to assess how stringent or
lenient a judge they are. If a panelist is categorizing portfolios consistently higher or
lower than the group, he or she may have a different understanding of the Achievement
Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,
but that disagreement should be based on a common understanding of the
Achievement Level Descriptors.
17. When the group has completed their Round 2 ratings, collect the rating forms. When you
collect the rating forms, carefully inspect them to ensure they are filled out properly.
c. The ID and round number must be filled in.
d. Each portfolio must have one (and only one) rating.
18. Facilitators should bring all the completed rating forms together to the data analysis work
room for tabulation. Prior to submitting them, however, using a show of hands, indicate on
a piece of chart paper how many panelists assigned each portfolio to each achievement level
category. This chart will be used for the Round 3 discussions.
Appendix G—Facilitators’ Scripts 14 2010 Maine PAAP Standard Setting Report
Round 3 Ratings
Overview of Round 3: In Round 3, the panelists will have a final opportunity to discuss their
Round 2 placements and to revise their ratings on the basis of that discussion. Prior to beginning
the Round 3 discussions, the psychometrician will share the Round 2 results with the group,
including the group average cut points and impact data, i.e., the approximate percentage of
students who would be classified into each achievement level category based on the room average
cut points from Round 2.
Once panelists have reviewed and discussed the Round 2 categorizations, they will make their final
ratings.
Activities:
9. Make sure panelists have the following materials:
a. Rating form
b. set of portfolios
c. Achievement Level Descriptors
10. Have panelists write their ID number and Round 3 on the rating form.
11. Provide an overview of Round 3. Paraphrase the following:
a. As in Round 2, the primary purpose is to categorize each portfolio into the
achievement level category where you believe it belongs.
b. Each panelist needs to base his/her judgments on his/her experience with the
content area, understanding of students, the definition of each achievement level
category, discussions with other panelists, and the knowledge, skills, and abilities
required to answer each item.
12. Review the feedback information with the panelists.
a. Show the panelists how the portfolios would be categorized based on the room
average Round 2 cut point placements.
b. Go over the impact data, explaining that if the Round 2 ratings were to be used to set
the final cut points, these are the approximate percentages of students who would be
classified into each achievement level category.
13. Remind panelists that they will be discussing each portfolio with their colleagues, but that
they will be categorizing the portfolios individually. It is not necessary for the panelists to
reach consensus about how to categorize each portfolio.
14. Give panelists an opportunity to ask questions about the feedback information or about the
task for Round 3.
15. Beginning with the first portfolio for which there is disagreement as to how it should be
categorized, the panelists should begin discussing the categorization of the portfolios
according to the Round 2 ratings.
a. Panelists should discuss the portfolios for which there was disagreement as to how
they should be categorized, focusing in particular on thoe portfolios around the cuts.
b. Panelists should be encouraged to listen to their colleagues as well as express their
own points of view.
c. If the panelists hear a logic/rationale/argument that they did not consider and that
they feel is compelling, then they may adjust their ratings to incorporate that
information.
Appendix G—Facilitators’ Scripts 15 2010 Maine PAAP Standard Setting Report
d. On the basis of the discussions and the feedback presented, panelists should make
their final ratings.
e. The group does not have to achieve consensus. If panelists honestly disagree, that is
fine. We are trying to get the best judgment of each panelist. Panelists should not feel
compelled or coerced into making a rating with which they disagree.
Encourage the panelists to use the discussion and feedback to assess how stringent or
lenient a judge they are. If a panelist is categorizing portfolios consistently higher or
lower than the group, he or she may have a different understanding of the Achievement
Level Descriptors than the rest of the group. It is acceptable for panelists to disagree,
but that disagreement should be based on a common understanding of the
Achievement Level Descriptors.
16. When the group has completed their final ratings, collect the rating forms. When you collect
the rating forms, carefully inspect them to ensure they are filled out properly.
a. The ID and round number must be filled in.
b. Each portfolio for Round 3 must have one (and only one) rating.
c. Return the completed rating forms to the data analysis work room.
Please indicate any areas in which you would like more information before you continue.
Cut Scores? Have we discussed that?
Please indicate any questions you may have about the remainder of the standard setting meeting.
I think I understand – not sure
Please indicate any questions you may have about the remainder of the standard setting meeting.
Very few of the tasks lend themselves to establishing extension to show PWD.
Still struggle with PAAPs incorrectly scored either inaccuracy or in levels of assistance.
N Mean % SD %D %U %A % SA
I understood the goals of the standard
8 4.38 0 0 0 62 38
setting meeting.
I understood the procedures we used to
8 4 0 0 25 50 25
set standards.
The facilitator helped me understand the
8 4.25 0 0 12 50 38
process.
The materials contained the information
8 3.38 0 38 12 25 25
needed to set standards.
I understood how to use the materials
8 4.12 0 0 12 62 25
provided.
The performance level descriptors were
8 4.25 0 0 12 50 38
clear.
I understood how to make the cut score
8 3.88 0 12 25 25 38
judgments.
I understood how to use the feedback
8 4.12 0 12 0 50 38
provided after each round.
I understood how to use the impact data. 7 4 0 0 14 71 14
I understood how the cut scores were
8 3.62 12 12 0 50 25
calculated.
The facilitator was able to get answers to
8 4.38 0 0 12 38 50
my questions.
Sufficient time was allotted for training on
8 4.25 0 0 12 50 38
the standard setting tasks.
Sufficient time was allotted to complete
8 4.12 0 12 0 50 38
the standard setting tasks.
The facilitator helped the standard setting
8 4.62 0 0 0 38 62
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Examples should include those with correctly scored items and correctly identified level of
accuracy marked because we were told they were corrected when Measured Progress scored
but we didn’t have that info.Examples shouldn’t have missing tasks or zeros ( at the upper
limits) makes more difficult to give accurate reading limits) makes more difficult to give
accurate reading. It was unique to see I know that no highest level will create the need for
policy level discussion.
Samples had numerous errors-scoring errors, level of assistance errors, task with scores
below 33 on LOC 4 tasks at “top” of samples (30+) at gr. 3
Better quality of examples. Felt pressured to raise some students’ standings for better cut
scores.
Use tests that are scored correctly. Provide raw scores.
Please indicate any areas in which you would like more information before you continue.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
None were more influential than others
Level of complexity, Level of accuracy, level of assistance
Student responses +my experience
Student work & core skills assessed for grade level
N Mean % SD %D %U %A % SA
I understood the goals of the standard
8 4.62 0 0 0 38 62
setting meeting.
I understood the procedures we used to
8 4.62 0 0 0 38 62
set standards.
The facilitator helped me understand the
8 4.88 0 0 0 12 88
process.
The materials contained the information
8 4.62 0 0 0 38 62
needed to set standards.
I understood how to use the materials
8 4.62 0 0 0 38 62
provided.
The performance level descriptors were
8 4.62 0 0 0 38 62
clear.
I understood how to make the cut score
8 4.62 0 0 0 38 62
judgments.
I understood how to use the feedback
8 4.75 0 0 0 25 75
provided after each round.
I understood how to use the impact data. 8 4.5 0 0 0 50 50
I understood how the cut scores were
8 4.5 0 0 0 50 50
calculated.
The facilitator was able to get answers to
8 4.88 0 0 0 12 88
my questions.
Sufficient time was allotted for training on
8 4.62 0 0 0 38 62
the standard setting tasks.
Sufficient time was allotted to complete
8 4.25 0 12 12 12 62
the standard setting tasks.
The facilitator helped the standard setting
8 4.88 0 0 0 12 88
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Sample sizes leading to accuracy was different and impacted decisions.
Super Facilitator
Please indicate any areas in which you would like more information before you continue.
Cut score further explained
Just need to put process to work.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
I believe I will learn what further questions I have from doing the actual work.
None at this time.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
The criteria the group set for the steps of the rubric. This help keep me focus when personal
bias came in to play
The combination of LOC & LoA w/ & of accuracy as well as overall & outcome.
Achievement level descriptors helped me to better see subtle difference between levels.
Descriptors
The student PAAP samples and seeing their work (Loc, level of accuracy and level of
assistance). Combining that info with the descriptors helped me place each PAAP.
Defining more specifically the levels of proficiency using the Maine PAAP definitions.
N Mean % SD %D %U %A % SA
I understood the goals of the standard
10 4.9 0 0 0 10 90
setting meeting.
I understood the procedures we used to
10 4.8 0 0 0 20 80
set standards.
The facilitator helped me understand the
10 4.9 0 0 0 10 90
process.
The materials contained the information
10 4.8 0 0 0 20 80
needed to set standards.
I understood how to use the materials
10 4.8 0 0 0 20 80
provided.
The performance level descriptors were
10 4.6 0 0 0 40 60
clear.
I understood how to make the cut score
10 4.7 0 0 0 30 70
judgments.
I understood how to use the feedback
10 4.7 0 0 0 30 70
provided after each round.
I understood how to use the impact data. 10 4.7 0 0 0 30 70
I understood how the cut scores were
10 4.6 0 0 0 40 60
calculated.
The facilitator was able to get answers to
10 4.8 0 0 0 20 80
my questions.
Sufficient time was allotted for training on
10 4.9 0 0 0 10 90
the standard setting tasks.
Sufficient time was allotted to complete
10 4.9 0 0 0 10 90
the standard setting tasks.
The facilitator helped the standard setting
10 4.9 0 0 0 10 90
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Maybe the intro sessions with lots of repetition + “you’ll learn more about this in small
groups later” could be shorter
Enjoyed the process very much
Please indicate any areas in which you would like more information before you continue.
Not a question, but a comment, I would have began the achievement levels discussion with
the proficient level instead of the substantially below level proficiency.
How to apply these cuts to the tasks
A bit more time with the materials and I’ll have it I’m almost there.
Let’s go through one together.
Have learned much so far
Please indicate any questions you may have about the remainder of the standard setting
meeting.
I don’t know if I will be able to do this successfully. It seems a little confusing.
I think if we did the first one together, everyone would be on the same page about what we’re
looking for.
Am visual – will feel more confident as one is actually done.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Would have liked to do practice task first.
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Level descriptions
The descriptions of skills (prior knowledge) in the task descriptions. How specific do the ach.
level descriptions need to be followed when determining the standard?
Difficulty level and content standards
The grade level complexities and how the students performed on those tasks were most
influential to me.
LOC mainly as it loosely corresponded to the skills in the 4 levels.
N Mean % SD %D %U %A % SA
I understood the goals of the standard
9 4.11 0 11 0 56 33
setting meeting.
I understood the procedures we used to
9 4 0 11 0 67 22
set standards.
The facilitator helped me understand the
9 4.22 0 11 0 44 44
process.
The materials contained the information
9 3.78 0 0 33 56 11
needed to set standards.
I understood how to use the materials
9 4.44 0 0 0 56 44
provided.
The performance level descriptors were
9 3.89 0 0 33 44 22
clear.
I understood how to make the cut score
9 4.33 0 0 0 67 33
judgments.
I understood how to use the feedback
9 4.33 0 0 0 67 33
provided after each round.
I understood how to use the impact data. 9 4.11 0 0 11 67 22
I understood how the cut scores were
9 3.78 0 11 11 67 11
calculated.
The facilitator was able to get answers to
9 4.33 0 0 0 67 33
my questions.
Sufficient time was allotted for training on
9 3.89 0 22 0 44 33
the standard setting tasks.
Sufficient time was allotted to complete
9 3.89 0 11 22 33 33
the standard setting tasks.
The facilitator helped the standard setting
9 4.44 0 0 0 56 44
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
I didn’t like the SBP PP P PD skills we listed. I found it was easier to go by LOC’s and
student scores along with the need for assistance.
I think a sample task would be very helpful to go over before starting the evaluations within
your own smaller group.
Process reversed to have open discussion for example of standard setting expectations.
Please indicate any areas in which you would like more information before you continue.
Is Proficiency based on how well a student did on this particular assessment or upon where
he should be?
Difference between achievement levels. More about cut scores
Not sure how the indicators we established are helpful when scoring level ½ assessments.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
I just wanted to have the conversation with group to see if we are seeing it in a similar
manner.
Note: It is helpful to have done standard setting before.
How does substantially below proficient and partially proficient affect AYP?
Can a student be considered proficient if they received 100/83 at locs?
Please indicate any questions you may have about the remainder of the standard setting
meeting.
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Understanding of the student’s body of work, level of complexities. I know how difficult it
is for students to complete these tasks and try to give them the benefit of the doubt.
Level of complexity
Level of assistance and achievement level descriptors.
Great facilitators- questions were answered.
N Mean % SD %D %U %A % SA
I understood the goals of the standard
11 4.36 0 0 0 64 36
setting meeting.
I understood the procedures we used to
11 4.45 0 0 0 55 45
set standards.
The facilitator helped me understand the
11 4.36 0 0 9 45 45
process.
The materials contained the information
11 4.45 0 0 0 55 45
needed to set standards.
I understood how to use the materials
11 4.36 0 0 0 64 36
provided.
The performance level descriptors were
11 4.36 0 0 0 64 36
clear.
I understood how to make the cut score
11 4.27 0 9 0 45 45
judgments.
I understood how to use the feedback
11 4.36 0 0 9 45 45
provided after each round.
I understood how to use the impact data. 11 4.09 0 0 27 36 36
I understood how the cut scores were
11 4 0 9 18 36 36
calculated.
The facilitator was able to get answers to
11 4.55 0 0 0 45 55
my questions.
Sufficient time was allotted for training on
11 4.45 0 0 0 55 45
the standard setting tasks.
Sufficient time was allotted to complete
11 4.45 0 0 0 55 45
the standard setting tasks.
The facilitator helped the standard setting
11 4.64 0 0 0 36 64
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Please indicate any areas in which you would like more information before you continue.
As a reg. ed Teacher I wish I had prepared myself more on PAAPs because I have realized
that I really knew very little. Thank you for the brief introduction.
The training provided was sequential and with a clearly stated outcome. At this point, I need
to get my feet wet with the process. Maybe ?’s will arise as I am completing the task.
Example of how to score demonstrated would be helpful on first task as a group.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
The procedures make sense but I feel like we could have had more time. Explanations of
different ability levels would have been helpful for Regular Ed. Teachers.
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Discussion- as a reg. ed teacher I expected the students to be able to read for the reading
section but they didn’t have to. Hard not to compare to the NECAP.
Discussion among facilitator and group; more time should be allowed for this process- felt
very rushed.
The more info we have about student achievement, the better decisions we can make.
Discussions were helpful; did feel rushed at times to get through the stack!
LOC; level of assistance; for successive rounds, listening to others’ feedback
LOCs and the student work with score sheets; more info (specific info) would have helped
with regards to the level of assistance and actual students responses, not just correct or
incorrect.
N Mean % SD %D %U %A % SA
I understood the goals of the standard
9 4.22 0 0 0 78 22
setting meeting.
I understood the procedures we used to
9 4.22 0 0 11 56 33
set standards.
The facilitator helped me understand the
9 4.11 0 0 11 67 22
process.
The materials contained the information
9 4 0 0 11 78 11
needed to set standards.
I understood how to use the materials
8 4.25 0 0 0 75 25
provided.
The performance level descriptors were
9 4 0 0 11 78 11
clear.
I understood how to make the cut score
9 3.67 0 0 33 67 0
judgments.
I understood how to use the feedback
9 4.33 0 0 0 67 33
provided after each round.
I understood how to use the impact data. 9 3.56 0 0 56 33 11
I understood how the cut scores were
8 3.12 0 25 38 38 0
calculated.
The facilitator was able to get answers to
9 4.44 0 0 0 56 44
my questions.
Sufficient time was allotted for training on
9 3.89 0 11 11 56 22
the standard setting tasks.
Sufficient time was allotted to complete
9 3.89 0 11 11 56 22
the standard setting tasks.
The facilitator helped the standard setting
9 4.44 0 0 0 56 44
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Devise some sort of graphic organizer to record notes as we revised the portfolios (to keep
track of our reactions + inputs)
Please indicate any areas in which you would like more information before you continue.
Cut score
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Using the LOCs and TLC evidence of growth influences my cut score rating.
The LOC levels
It appears as if the Loc’s do not match the tasks. Ex an A1 LOC the student must be using
phoenic awareness/word parts and context clues, yet the task does not require the student
read. I feel this needs to be examined more closely.
It became obvious during/prior to Round 3 what Measured Progress was “looking for” this
influenced the decisions made causing me to question the validity of this entire process.
Difficulty of the task along with student responses.
N Mean % SD %D %U %A % SA
I understood the goals of the standard
8 4 0 0 0 100 0
setting meeting.
I understood the procedures we used to
8 3.75 0 0 25 75 0
set standards.
The facilitator helped me understand the
8 3.88 0 0 12 88 0
process.
The materials contained the information
8 3.62 0 0 50 38 12
needed to set standards.
I understood how to use the materials
8 3.88 0 0 12 88 0
provided.
The performance level descriptors were
8 3.38 0 0 62 38 0
clear.
I understood how to make the cut score
7 3.71 0 0 29 71 0
judgments.
I understood how to use the feedback
8 3.88 0 0 12 88 0
provided after each round.
I understood how to use the impact data. 8 3.75 0 0 25 75 0
I understood how the cut scores were
8 3.75 0 0 25 75 0
calculated.
The facilitator was able to get answers to
8 3.75 0 0 25 75 0
my questions.
Sufficient time was allotted for training on
8 4.25 0 0 0 75 25
the standard setting tasks.
Sufficient time was allotted to complete
8 4.25 0 0 0 75 25
the standard setting tasks.
The facilitator helped the standard setting
8 3.88 0 0 25 62 12
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
The achievement level definitions made this process difficult. They do not match the tasks,
therefore made this process mute, in my opinion.
Please indicate any areas in which you would like more information before you continue.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
Please indicate any questions you may have about the remainder of the standard setting
meeting.
(referring to “I think the procedures make sense”; subject checked “agree”): This seems
counter!
Subjectivity?
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Group discussion about student work. Initial setting of achievement indicators.
Fantastic process.
Group discussions and skills at each level.
Achievement Level Descriptors, definitions of the four levels- handout
Please indicate any questions you may have about the remainder of the standard setting
meeting.
What materials, information, or procedures were most influential in your placement of the cut
scores? Why?
Excellent Process
Description of 4 levels SBP,PP,P,PWD + achievement level descriptors
The discussions and standards
Student work and discussion about it. Achievement indicators
N Mean % SD %D %U %A % SA
I understood the goals of the standard
7 4.57 0 0 0 43 57
setting meeting.
I understood the procedures we used to
7 4.43 0 0 14 29 57
set standards.
The facilitator helped me understand the
7 4.57 0 0 0 43 57
process.
The materials contained the information
7 4.57 0 0 0 43 57
needed to set standards.
I understood how to use the materials
7 4.57 0 0 0 43 57
provided.
The performance level descriptors were
7 4.57 0 0 0 43 57
clear.
I understood how to make the cut score
7 4.43 0 0 14 29 57
judgments.
I understood how to use the feedback
7 4.57 0 0 0 43 57
provided after each round.
I understood how to use the impact data. 7 4.57 0 0 0 43 57
I understood how the cut scores were
7 4.14 0 0 29 29 43
calculated.
The facilitator was able to get answers to
7 4.71 0 0 0 29 71
my questions.
Sufficient time was allotted for training on
7 4.57 0 0 0 43 57
the standard setting tasks.
Sufficient time was allotted to complete
7 4.57 0 0 0 43 57
the standard setting tasks.
The facilitator helped the standard setting
7 4.57 0 0 0 43 57
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Many teacher still are not following the process-fair amount of teacher error rather than
student error
It was more difficult than need to due to lack of higher LOC’s at some indicators
Much better samples than for math!
( Referring to “I understood how to make cut score judgments.”) Seriously, I don’t feel I
make cut score judgments
If this were to be done again, it would be helpful to have a summary sheet for each students
with task , LOL,LOA for all tasks on 1 sheet. This would save a lot of time, also ever
consider going paperless. Some of this could definitely be paperless
N Mean % SD %D %U %A % SA
I understood the goals of the standard
7 4.71 0 0 0 29 71
setting meeting.
I understood the procedures we used to
7 4.71 0 0 0 29 71
set standards.
The facilitator helped me understand the
7 4.43 0 0 14 29 57
process.
The materials contained the information
7 4.86 0 0 0 14 86
needed to set standards.
I understood how to use the materials
7 4.86 0 0 0 14 86
provided.
The performance level descriptors were
7 4.57 0 0 14 14 71
clear.
I understood how to make the cut score
7 4.57 0 0 0 43 57
judgments.
I understood how to use the feedback
7 4.57 0 0 0 43 57
provided after each round.
I understood how to use the impact data. 7 4.57 0 0 0 43 57
I understood how the cut scores were
7 4.43 0 0 14 29 57
calculated.
The facilitator was able to get answers to
7 4.86 0 0 0 14 86
my questions.
Sufficient time was allotted for training on
7 4.86 0 0 0 14 86
the standard setting tasks.
Sufficient time was allotted to complete
7 4.86 0 0 0 14 86
the standard setting tasks.
The facilitator helped the standard setting
7 4.71 0 0 0 29 71
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
I enjoyed the process very much.
Hope the blue print and fall trainings can give clarity to next years grade level expectations
and that a full task bank will provide more evaluation options.
N Mean % SD %D %U %A % SA
I understood the goals of the standard
7 4.29 0 0 0 71 29
setting meeting.
I understood the procedures we used to
7 4 0 0 14 71 14
set standards.
The facilitator helped me understand the
7 4.14 0 0 0 86 14
process.
The materials contained the information
7 4 0 0 29 43 29
needed to set standards.
I understood how to use the materials
7 4.14 0 0 14 57 29
provided.
The performance level descriptors were
7 3.71 0 0 29 71 0
clear.
I understood how to make the cut score
7 4 0 0 14 71 14
judgments.
I understood how to use the feedback
7 4.14 0 0 14 57 29
provided after each round.
I understood how to use the impact data. 7 4.29 0 0 0 71 29
I understood how the cut scores were
7 4.29 0 0 0 71 29
calculated.
The facilitator was able to get answers to
7 4.14 0 0 0 86 14
my questions.
Sufficient time was allotted for training on
7 4.29 0 0 14 43 43
the standard setting tasks.
Sufficient time was allotted to complete
7 4.43 0 0 0 57 43
the standard setting tasks.
The facilitator helped the standard setting
7 4.29 0 0 0 71 29
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Process between facilitators seemed at times different. Possible that this is due to different
groups? Content areas.
N Mean % SD %D %U %A % SA
I understood the goals of the standard
9 4.44 0 0 0 56 44
setting meeting.
I understood the procedures we used to
9 4.67 0 0 0 33 67
set standards.
The facilitator helped me understand the
9 4.56 0 0 0 44 56
process.
The materials contained the information
9 4.56 0 0 0 44 56
needed to set standards.
I understood how to use the materials
9 4.67 0 0 0 33 67
provided.
The performance level descriptors were
9 4.67 0 0 0 33 67
clear.
I understood how to make the cut score
9 4.67 0 0 0 33 67
judgments.
I understood how to use the feedback
9 4.56 0 0 0 44 56
provided after each round.
I understood how to use the impact data. 9 4.44 0 0 0 56 44
I understood how the cut scores were
10 4.2 0 0 20 40 40
calculated.
The facilitator was able to get answers to
9 4.67 0 0 0 33 67
my questions.
Sufficient time was allotted for training on
9 4.56 0 0 0 44 56
the standard setting tasks.
Sufficient time was allotted to complete
9 4.67 0 0 0 33 67
the standard setting tasks.
The facilitator helped the standard setting
9 4.78 0 0 0 22 78
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Since I was unfamiliar with what the student was expected to do, it would have been more
helpful to see the tasks before we designed our rubrics.
N Mean % SD %D %U %A % SA
I understood the goals of the standard
9 4.67 0 0 0 33 67
setting meeting.
I understood the procedures we used to
9 4.67 0 0 0 33 67
set standards.
The facilitator helped me understand the
9 4.67 0 0 0 33 67
process.
The materials contained the information
9 4.67 0 0 0 33 67
needed to set standards.
I understood how to use the materials
9 4.67 0 0 0 33 67
provided.
The performance level descriptors were
9 4.56 0 0 0 44 56
clear.
I understood how to make the cut score
9 4.56 0 0 0 44 56
judgments.
I understood how to use the feedback
9 4.56 0 0 0 44 56
provided after each round.
I understood how to use the impact data. 9 4.56 0 0 0 44 56
I understood how the cut scores were
9 4.56 0 0 0 44 56
calculated.
The facilitator was able to get answers to
9 4.67 0 0 0 33 67
my questions.
Sufficient time was allotted for training on
9 4.67 0 0 0 33 67
the standard setting tasks.
Sufficient time was allotted to complete
9 4.67 0 0 0 33 67
the standard setting tasks.
The facilitator helped the standard setting
9 4.67 0 0 0 33 67
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Awesome 3 days! This is the best I have been to in 16 yrs!
N Mean % SD %D %U %A % SA
I understood the goals of the standard
8 4.25 0 0 0 75 25
setting meeting.
I understood the procedures we used to
8 4.25 0 0 0 75 25
set standards.
The facilitator helped me understand the
8 4.38 0 0 0 62 38
process.
The materials contained the information
8 4 0 0 25 50 25
needed to set standards.
I understood how to use the materials
8 4.25 0 0 0 75 25
provided.
The performance level descriptors were
8 4.12 0 0 12 62 25
clear.
I understood how to make the cut score
8 4.25 0 0 0 75 25
judgments.
I understood how to use the feedback
8 4.25 0 0 0 75 25
provided after each round.
I understood how to use the impact data. 8 4 0 0 25 50 25
I understood how the cut scores were
8 4 0 12 0 62 25
calculated.
The facilitator was able to get answers to
8 4.38 0 0 0 62 38
my questions.
Sufficient time was allotted for training on
8 4.25 0 0 0 75 25
the standard setting tasks.
Sufficient time was allotted to complete
8 4 0 12 0 62 25
the standard setting tasks.
The facilitator helped the standard setting
8 4.5 0 0 0 50 50
process run smoothly.
SD = strongly disagree; D = disagree; U = undecided; A = agree; SA = strongly agree
Please provide any additional comments about the standard setting process or suggestions as
to how the training and process could be improved.
Would like guidelines of LOA to be present in future.
Efficiently done; leadership/guidance great help and support
Jake was great to work with!
Need more clarity on level of assistance- Scribe/- more info provide on how each teacher did
it.
I believe that we needed more information regarding the level of assistance. Some teachers
were not clear. Training for this may need to be more specific.
2 3 4 5 6 7 HS
Proficient with Distinction
Proficient
Partially Proficient
Substantially Below Proficient Mathematics: Policy Results
Appendix J—Policy Adjustments 5 2010 Maine PAAP Standard Setting Report
100
80
Percent of students in category
60
40
20
0
4 7 11
Proficient with Distinction
Proficient
Partially Proficient
Substantially Below Proficient
Writing: Policy Results