You are on page 1of 14

# Chapter 9: Data

processing
Weighting

202
First European Survey on Language Competences: Technical Report

Weights are used for several reasons. and variance estimation. i.1 Motivation and overview Survey statistics are practically always weighted. an example of a population total might be the number of students in a country that are studying a particular FL and have reached a certain level of proficiency. population totals. the data for any individual in the sample is given a weight that is. While scientific research may be more interested in structural aspects of the data as expressed in averages. our estimates for the population totals will decrease. or regressions. so they influence the estimation of all statistics. non-responding schools and students may differ in important aspects from those who respond. To avoid bias due to unequal sampling probabilities. there may be different adjustments for non-response. possibly causing the sample results to deviate from what is true for the population. A general principle is that the final 203 First European Survey on Language Competences: Technical Report . policy makers typically have to deal with absolute numbers. To calibrate sample totals to the population totals. All these operations rely on weights. not just the overall magnitude of totals. i. we redistribute the weights of the non-respondents among those who did respond. To arrive at correct estimates of the population total. It has a negative impact on both aims discussed above. In addition. To counteract.Weighting This chapter deals with sampling weights. 9. (i) The computation of sampling weights is a rather complicated procedure that can involve many steps. proportions. in principle. so their absence may bias the results. and that would slant the results towards persons who had a higher probability to be selected. schools at the first stage. any measurement or response for a person is given a specific weight when calculating the statistic. Furthermore.9 Data processing . When persons drop out. further adjustment may be needed). Without weighting. of which the following are the most salient. Sampling itself may proceed in several stages (for instance. To counteract.e. equal to the inverse of the inclusion probability for the person (in practice. Non-response is an unwelcome but unavoidable complication in all surveys. but among similar schools and students who did respond. and students within schools at the second).. weights will differ across persons. etc. adjusting weights for non-response. statistics would assign the same weight to the each person. As long as the probability is known.e.. we try to redistribute their weights not in a general fashions. In the context of SurveyLang. In this second function. we can still get unbiased results by weighting. Members of the population are seldom sampled with the same probability.

and as a result of stratified sampling with disproportional allocation. If all students from the schools in the sample were to be tested. However. In this chapter. 204 First European Survey on Language Competences: Technical Report . which differ dramatically in population size but are represented with samples of approximately the same size. the weighting procedures used for ESLC follow the standards of the Best Practices used for this type of complex survey. students from large schools would be overrepresented. This means that. This property is described with the name self-weighting sample. A standard technique involves taking many subsamples from the sample and inferring the standard error (SE) of the sample from the variability among the subsamples. we could use non-weighted statistics. weights can vary considerably even in a design that is originally self-weighting. a problem that can be easily fixed by using appropriate weights. product. there is stratification by school size and other school characteristics for details on the stratified sampling design in SurveyLang. TIMSS (Third International Mathematics and Science Study) and PIRLS (Progress in International Reading Literacy Study). large or small. If it really holds. The appropriate method does depend on the fact that inclusion probabilities are not necessarily equal. at the second stage. of various The sampling design of SurveyLang involves two stages. Another important issue is how to estimate the statistical precision of the point estimates. Following the principle that sampling weights at different stages are multiplied to produce the final weight. Similar procedures were used in other international studies of this nature including PISA (Programme for International Student Assessment). we concentrate on the following topics: the computation of base school weights and base student weights the various adjustments made to the base weights to compensate for nonresponse at the school and student level the estimation of standard errors. (i) In general. and population totals could be obtained by multiplying all means and proportions with a constant factor. Within countries. the participating countries. sometimes rather long.weight giv components and adjustment factors. In the first stage. To increase precision and simplify logistics. the inclusion probability for the individual would then be about the same in the final run. we have discussed the importance of weights for determining the statistic itself (the point estimate). students are sampled with a probability inversely proportional to the school size. Because of nonresponse and disproportional allocation. schools are sampled with a probability proportional to their size. see chapter 4 on Sampling. sampling is done not from the entire population but separately for subpopulations (strata) that may have different sampling probabilities and different nonway. Reality is not that simple because of the inevitable problems with non-response mentioned above. So far. the second stage samples the same number of students in each school.

a census was conducted and no sampling of schools was 205 First European Survey on Language Competences: Technical Report . The formula applies to non-certainty selections. and the index shows the lowest (most detailed) level at which weights or adjustment factors differ.2.2 Base weights Since sampling for the ESLC involves two stages. schools with measure of size large enough to make the right hand side of the equation less than 1). there are four weights per student. the index p for persons. and this fourth factor is the adjustment for sub-sampling students for two out of three cognitive tests Prior to any adjustments for non-response. schools whose probability to be sampled was less than 1. We now explain the computation of the four elements in more detail. a sample is drawn from each stratum separately. we use the index s for schools. which is the inverse of the sampling probability for the student within his or her school. Two additional factors are added to the product: C. the student weight is therefore As × Bp× Cs × Dk Throughout this chapter. The complex design of SurveyLang expects each student in the sample to complete a Student Questionnaire and to be tested in two out of three skills.. and B.. The formulae are shown in a maximally simplified form. For schools selected with certainty (i. 9. i. A base weight for the student. the weight attached to the responses or properties of each individual student is the product of at least two components: A. and the formula above applies to the stratum. For some countries and some languages. When explicit strata are used. and the index k for skills. the school base weight was set to 1. moss is a measure of size for school s. A base weight for the school.e. which is inverse of the sampling probability for the school. the school base weight can be calculated as: As = M / (n × moss) where n is the sample size (of schools). even if there is no explicit indexing. and M is the total of the measures of size. As a consequence.1 A: School base weight The school base weight for school s is the inverse of the probability of selection of that school in the school sample.9.e. A trimming factor for the schools is intended to compensate for in the sampling frame D. Based on a PPS (Probability Proportional to Size) sampling scheme.

2 Student base weight (B) The student base weight is the reciprocal of the probability for a student to be sampled within his or her school. whichever of the two was larger. All schools from these countries (and languages) were assigned a base weight of 1. student base weights were calculated separately in each sub-stratum. When students within a school had to be further sub-stratified (in situations where there were students eligible for sampling in both languages within the same school). Note that the student base weight is based on actual enrolment. it is obtained as the actual number of eligible students in the school divided by the number of sampled students in the school. Obviously. 9.2. For the purpose of completing school sampling on time. This was found necessary for schools that turned out to be significantly larger than what was expected at the time of school sampling based on their estimated size as of that point of time. Bs. such estimates cannot be completely accurate.2. might become excessively large. the student base weight. moss. so it was decided to replace the original measure of size. In other words. In such situations. As. or the target cluster size (TCS) for the stratum. In most countries. Listening 206 First European Survey on Language Competences: Technical Report .2. by 3 × max (TCS. the number of instances where weight trimming had to be used was relatively small. Since the measure of size tended to underestimate actual enrolment in most countries. these estimates had to be generated in advance and were primarily based on similar enrolment figures from the previous year. 9. whenever the conditions described above were met. moss).3 School weight trimming factor (C) Trimming of school base weights was undertaken to avoid unexpectedly large values of the weights. in which case the student base weight is equal to 1. Put simply.undertaken at all. 9. Trimming was done for schools where the revised enrolment (RENR) at the time of sampling exceeded at least 3 times the original measure of size (moss). they were found to overestimate the population of students eligible for the survey. in the formula for the school base weight. each sampled student was assigned to two of the three skill tests (Reading Comprehension. not on the measure of size used to select the sample of schools. The measure of school size (moss) used for the PPS sampling is based on the estimated enrolment of eligible students in specific grades. the number of students learning the language in the grade below the eligible grade was used as the best estimate.4 Student sub-sampling weight for skills (D) Once the student sample within the school was selected. The value of the student base weight was therefore always larger than 1 unless all students were sampled.

so 20 missing students in the sample will decrease the population total by 10000. the index p denotes persons.3 Adjusting weights for non-response When schools or persons do not participate in the survey. k. A correction factor for schools that dropped out of the survey and could not be replaced. In a large country like France or Spain. was calculated as the ratio of the total number of students sampled to the number of students assigned to a specific test. D = 1 for this 9. we try to redistribute the weights for the non-responding schools or students not in general. the formula for the individual weight becomes As × Bp Cs × Dk × Es × Fs × Gp × Hp As explained. and the index k denotes skills. a sampled student may represent 500 students in the population. and we only show the index for the most detailed level at which weight components differ. To counteract. F. but among those schools or students that are as similar as possible to the non-responding ones.Comprehension and Writing) at random. This is a problem that affects not only totals but all kinds of statistics. we add four adjustment factors: E. The value of Dk is hence about 3/2 but may vary somewhat because of the integer arithmetic involved. we redistribute the weights assigned to non-responding students among those who did respond. A correction factor for a small number of students who were supposed to be excluded from the sample but were nonetheless sampled. the index s denotes schools. This means that. To account for this sub-sampling. We now explain the computation of the four correction factors in more detail. Unless non-responding students are a perfectly random part of the sample. A trimming factors for student weights. a weighting adjustment factor. their responses are lost. non-response can lead to biased estimates. to the four components of the individual student weight discussed above. In the final run. G. Since all students were expected to complete a Student Questionnaire. This has at least two important consequences: (i) (ii) Population totals will be underestimated. 207 First European Survey on Language Competences: Technical Report . A correction factor for students who did not participate. H. which they can hardly be expected to be. To avoid underestimation. Dk.

the school is classified as non-participating. and a 93.3 2 72 97.3 3 89 96.0 0 9 100.0 8 98 92.5 Bulgaria 3 74 96.7 18 57 76. we had a 93. the criterion of 85% participation at school level is comfortably met everywhere.7 Malta 2 55 96. Table 28 shows mode detailed data on school participation (after replacement of initial refusals) by country and target language (as far as sampling is concerned.0 11 66 85.0 24 55 69.9 7 67 90.9.6 8 70 89.6% rate of participation for schools in target language 1.5 Spain 0 78 100.7 5 55 91.5 The extent of non-response SurveyLang has adopted rather strict quality criteria with respect to coverage and response rates at both school and student level.6 France Belgium (French Community) Netherlands Greece 25 24 Note that in cases where within a sampled school students completed the cognitive tests but no students completed the Questionnaires.0 Croatia 0 75 100.3.5 4 70 94. and acceptable response rates for students as at least 80% of all sampled students.0 0 76 100. The Technical Standards define acceptable response rates for schools as at least 85% of sampled schools.9 Belgium (Flemish Community) 5 70 93. Table 28 Number of participating and non-participating schools24 and percentage of participating schools per country and target language First target language Second target language No Yes % No Yes % Belgium (German Community) 0 9 100. Overall.4 Portugal 3 72 96.3% rate of participation for schools in target language 2.0 8 71 89.0 3 71 95.7 9 66 88.5 2 55 96.7 Slovenia 2 71 97.0 Sweden 3 72 96. 209 First European Survey on Language Competences: Technical Report . the two target languages count as two separate surveys). 25 Note that the figures presented for the French Community of Belgium should read: for target language 2 (German) 4 schools out of 59 did not participate in the survey (and not 5 out of 60 as it is mentioned in the table).0 1 76 98.3 Poland 8 81 91.1 2 75 97. Except for one country.0 0 82 100.0 Estonia 0 79 100.