PISA 2012 NPM Meeting, Singapore

10 14 October 2011 Monday 10 October 2011

Welcome and Introduction Welcome from Singapore authorities Progress report since last meeting and PISA update (Secretariat, Consortium) Review of tasks and agenda for the meeting Target: All NPMs

DRAFT AGENDA as at 29 April 2011

Tuesday 11 October 2011
Computer-based components: preparations for main survey. Target: NPMs from countries intending to participate in Problem Solving only or Problem Solving and CBAL for the main survey

Wednesday 12 October 2011

Questionnaire discussion groups Groups will discuss questionnaire proposals Target: All NPMs

Thursday 13 October 2011

Financial literacy preparation, operational issues, item selection Target: NPMs from countries participating in Financial Literacy option

Friday 14 October2011
National centre presentation:

Singapore school visits

Target: All NPMs

BREAK 11:0012:30
Field trial overview and review; implementation of PISA standards lessons learned from both the paper-based and computerbased components Target: All NPMs

BREAK
Field trial outcomes 2 Overview of data outcomes from questionnaires Target: All NPMs

BREAK
Questionnaire discussion groups (continued) Target: All NPMs

BREAK
Adaptation and verification of cognitive test materials and questionnaire materials for main survey Main Survey test design Target: All NPMs

BREAK
PISA test operations for Main Survey Site visits and quality monitoring for main survey Target: All NPMs

12:3014:00 14:00-15:30

HOSTED LUNCH at PAN PACIFIC HOTEL

Field trial outcomes 1 Overview of data outcomes from cognitive tests Target: All NPMs

LUNCH
Main survey questionnaire proposals Target: All NPMs

LUNCH
Sampling for the main survey Presentation from Westat KeyQuest in the main survey Target: All NPMs

LUNCH
Round table responses from each national centre

LUNCH
Response to Questionnaire discussion groups, and to issues from the round table session. Conclusions (Consortium, Secretariat) Target: All NPMs

15:30-16:00 16:00-17:30-

BREAK
Main survey cognitive item proposals (for Mathematics, both paper-based and computer-based; and for Problem Solving) Target: All NPMs

BREAK
Main survey questionnaire proposals, continued Concluding with formation of groups and task assignment for tomorrow. Target: All NPMs

BREAK
National Centre Presentations (Italy, Denmark and the US) Target: Interested NPMs

BREAK
Round table discussion continued

BREAK
Reading components preparation, operational issues, item selection Target: NPMs from countries participating in Reading Components option

HOSTED DINNER AND NIGHT SAFARI TOUR Sampling consultations (including KeyQuest): all national centres, scheduled on Monday, Tuesday and Thursday Translation and verification consultations: at national centre request, Tues 11-12:30, 14:00-17:30; Wed 9:00-12:30, 14:00-17:30. 1

OECD Programme for International Student Assessment PISA 2012 NPM Meeting, Singapore
10 14 October 2011

Draft Agenda
[Note that sampling consultations for all national centres will be scheduled throughout the week NCs will be given an appointment time; and that national centres wanting to consult with consortium translation verification staff will be invited to sign up for a specified appointment.] Monday 10h October, 2011 1 9:00 - 10:30 2 1.2 Welcome and Introduction Singaporean authorities will welcome NPMs to the meeting. Secretariat and Consortium staff will provide an update of activities and developments since the last meeting (Secretariat, Consortium) NPM(1010)Record 1.3 3 2.1 Review the proposed tasks for the meeting and confirm the agenda. Consortium staff from ACER will provide an overview of the general field trial conduct and operational procedures from both the paper and computer-based components, issues related to the PISA technical standards, and lessons learned for main survey planning. NPM(1110)2 Lunch at Pan Pacific Hotel hosted by Singaporean National Centre. 4 14:00 - 15:30 Field Trial Outcomes 1 Cognitive components 5 Consortium staff from ACER will provide an overview of field trial data outcomes from the cognitive tests. NPM(1110)3 6 16:00 - 17:30 Main survey cognitive item proposals 4.1 Consortium staff from ACER will introduce proposals for the cognitive item selections for the Main Survey for Mathematics, both paper-based and computer-based items, and for Problem Solving. NPM(1110)4 Tuesday 11h October, 2011 7 9:00 - 10:30 Computer-based components for countries intending to participate in Problem Solving or Problem Solving and CBAL for the Main Survey 8 Consortium staff will review field trial outcomes for the computer-based components, and the preparation and implementation tasks for the Main Survey, for countries implementing Problem Solving alone or both Problem Solving and CBAL components. NPM(1110)5 11:00 - 12:30 Field Trial overview and review, implementation of PISA Standards

11:00 - 12:30 Field Trial Outcomes 2 Questionnaire components 2.1 Consortium staff from DIPF and ACER will provide an overview of field trial data outcomes from questionnaires. NPM(1110)6

10

14:00 - 15:30 Main Survey Questionnaire Proposals 11 Consortium staff from DIPF and ACER will present options and proposals for the content of questionnaire instruments for the main survey. NPM(1110)7

12

16:00 - 17:30 Main Survey questionnaire proposals continued. 13 The session will continue, and will conclude with consortium staff outlining arrangements for discussion groups for tomorrow to consider the proposed main survey questionnaire content.

Wednesday 15th October 2011 14 9:00 - 10:30 15 16 17 Questionnaire discussion groups Groups will be formed to discuss the options and proposals on the contents of the PISA 2012 main survey questionnaire instruments.

11:00 - 12:30 Questionnaire discussion groups continued 14:00 - 15:30 Sampling for the main survey 18 Consortium staff from Westat will provide a briefing on all matters related to main survey sampling, and staff from ACER will discuss changes in KeyQuest since the field trial that are relevant to sampling. NPM(1110)11

19

16:00 - 17:30 National Centre presentations 20 A presentation from the Italian national centre (see summary in PISA12 Newsletter Issue 3, December 2010 on MyPISA). The presentation will report a study of missing responses from Italys PISA 2006 data, designed to investigate the characteristics of students failing to respond. A presentation by Lena Lindenskov and Peter Weng from the Danish national centre, reporting on a study of Danish student answers to 15 released mathematics tasks from PISA 2003, designed to stimulate interest in PISA results among teachers and trainee teachers.

21

12.3 A presentation from the US national centre to demonstrate the system used in the US to report PISA data and results back to sampled schools. Dinner and Night Safari Tour hosted by Singaporean National Centre. Thursday 13th October, 2011 22 9:00 10:30 23 Financial Literacy for countries participating in Financial Literacy Option Consortium staff from ACER will give a presentation on preparation, operational issues and proposed item selection for the Financial Literacy Option. NPM(1110)13 24 11:00 12:30 Assessment Operations 25 Adaptation and verification of main survey cognitive test materials and questionnaire materials Consortium staff from cApStAn and HallStat will give a presentation on consortium changes in the national version of main survey instruments and for 3

documenting national changes, verification of clusters and booklet shells, final optical check of booklets and verification of coding guides for cognitive test materials and questionnaire materials. NPM(1110)14a 14.2 Main survey test design Consortium staff from ACER will outline the proposed main survey test design and the options that are available to match the various optional components. NPM(1110)14b 26 14:00 15:30 Round Table Responses from each national centre 15.1 Each national centre will be invited to make comments about the field trial, changes proposed for the main survey, and especially about the item selections proposed for the questionnaires and in each cognitive domain. 27 16:00 17:30 Round Table Responses continued

Friday 14th October, 2012 28 8:00 11:00 National centre presentation: Singapore school visits 17.1 Arrangements will be made to pick interested participants up at the hotel at 8.00 am for a visit to one or more local schools. Participants will be dropped back at the meeting venue at 11.00am in time for the remaining meeting sessions. Details to be provided later. 18. 11:00 12:30 Main survey operations 18.1 Test operations for main survey Consortium staff will give a presentation on test operations for the main survey. Matters to be covered will include materials preparation, test administration and field operations arrangements, data capture procedures, arrangements for response coding, data submission arrangements. NPM(1110)18a 18.2 Site visits and quality monitoring Consortium staff will give a briefing on the proposed arrangements for site visits and for the training and work of PISA Quality Monitors during the main survey. NPM(1110)18b 19. 14:00 15:30 Consortium reaction to Round Table and discussion group input 19.1 Consortium staff will provide a response to issues and proposals that came up in Wednesdays questionnaire discussion groups and in the round-table discussion on Thursday. 19.3 Responses, conclusions and next steps 20. Consortium staff and Secretariat to give brief summary. 16:00 17:30 Reading Components for countries participating in this option. 20.1 A training workshop will be conducted by Consortium staff from ETS and ACER to prepare countries that are implementing the optional component related to Reading Components. NPM(1110)20

Meeting paper list

Paper Number NPM(1010)Record NPM(1110)2 NPM(1110)3 NPM(1110)4 NPM(1110)5 NPM(1110)6 NPM(1110)7 MPM(1110)11 NPM(1110)13 NPM(1110)14a NPM(1110)14b NPM(1110)18a NPM(1110)18b NPM(1110)20 Description Record of previous NPM meeting (Budapest, October 2010) Field trial overview and implementation of technical standards Field trial outcomes cognitive components Main survey item proposals cognitive components Field trial outcomes computer-based components Field trial outcomes questionnaire components Main survey item proposals questionnaire components Main survey - sampling Main survey item proposals financial literacy option Main survey adaptation and verification cognitive questionnaire components Main survey test design and options Main survey test operations Main survey quality monitoring Main survey test operations reading components

and