You are on page 1of 11

Bambang Ruby S, NIM: 0203513037, Smester 1

PPs Unnes Pendidikan Bahasa Inggris (Kelas Khusus) 2013/2014

HANDOUT 4
Research in English Language Teaching
(The Psychometric Tradition)

Activity 1
What are variables, samples, and populations, and why are they important in
research?
Variable

: Nunan (1992) stated that a variable, as the term itself suggests, is
anything which does not remain constant. In our case, it includes
language proficiency, aptitude, motivation, and so on.

Population

: Saleh (2012) stated that population is the group of people or
objects that the researcher wants to know about by conducting
the study.

Sample

: Saleh (2012) stated that the sample is representative of the
population defined.

They are important components of research because of the significant impact
that they can have on the quality of the results/findings. The variables used as
main resources while population and sample used to know the influence of
variables.
What are the basic principles of sound experimental design?
The basic principles of experimental designs are randomization, replication and
local control. Each of them is described in the following subsections.
(1) Randomization. The first principle of an experimental design is randomization,
which is a random process of assigning treatments to the experimental units. The
random process implies that every possible allotment of treatments has the
same probability. An experimental unit is the smallest division of the
experimental material and a treatment means an experimental condition whose
effect is to be measured and compared. The purpose of randomization is to

Balancing means that the treatments should he assigned to the experimental units in such a way that the result is a balanced arrangement of the treatments. Hence the treatments must be assigned at random to the experimental units. The word control in experimental design is used for a treatment. This type of variation can be removed by using a number of experimental units. (3) Local Control. A replication is used to: (i) Secure more accurate estimate of the experimental error. We therefore perform the experiment more than once. which are not controllable. It has been observed that all extraneous sources of variation are not removed by randomization and replication. In other words. The number. In all experiments. (2) Replication. which is a repetition of the basic experiment. . i. we make use of local control. a term referring to the amount of balancing. An individual repetition is called a replicate.remove bias and other sources of extraneous variation. The second principle of an experimental design is replication. Which does not receive any treatment but we need to find out the effectiveness of other treatments through comparison.e. blocking and grouping of the experimental units. Another advantage of randomization (accompanied by replication) is that it forms the basis of any valid statistical test. some variation is introduced because of the fact that the experimental units such as individuals or plots of land in agricultural experiments cannot be physically identical. it is a complete run for all the treatments to be tested in the experiment. which is a measure of the variability of the experimental error.. we repeat the basic experiment. This necessitates a refinement in the experimental technique. The main purpose of the principle of local control is to increase the efficiency of an experimental design by decreasing the experimental error. The point to remember here is that the term local control should not be confused with the word control. In other words. and (iii) Obtain more precise estimate of the mean effect of a treatment. or by drawing numbered balls from a well-shaken container or by using tables of random numbers. For this purpose. we need to choose a design in such a manner that all extraneous sources of variation are brought under control. Randomization is usually done by drawing numbered cards from a well-shuffled pack of cards. since . (ii) Decrease the experimental error and thereby to increase precision. the shape and the size of replicates depend upon the nature of the experimental material. a term which represents the differences that would be observed if the same treatments were applied several times to the same experimental units. where denotes the number of replications.

He might try lecturing for one group of students. If he only had these two groups. an ANOVA is exactly the same as a t-test. For example. The first type of inferential statistic is a t-test. but it can analyze multiple groups at once. In other words. or an ANOVA test. Correlation: when researcher need to determine relationship between two variables. Analysis of variance (ANOVA): when researcher need to compares the average scores between three or more different groups in a study to see if the groups are different from each other. Chi-square: when researcher need to compares observed frequencies to expected frequencies. The second basic type of inferential statistics is called an analysis of variance or ANOVA.What do we mean by inferential statistics? Inferential statistics are data which are used to make generalizations about a population based on a sample. which was having the students learn the material on their own and then teach it to each other. a teacher believes that different teaching styles result in different scores when children take a test over the material. . They rely on the use of a random sampling technique designed to ensure that a sample is representative. Chi-square? T-test: when researcher need to compare the average scores between two different groups in a study to see if the groups are different from each other. When is it appropriate to use the following statistical procedures: t-test analysis variance. he would use an analysis of variance. An analysis of variance is a test that compares the average scores between three or more different groups in a study to see if the groups are different from each other. versus worksheets with a second group of students. However. he would use a t-test to compare the scores. He then gives everyone the same test. A t-test is used to compare the average scores between two different groups in a study to see if the groups are different from each other. and wants to compare the results. If he now wants to compare all three teaching styles to each other. correlation. if he wanted to add a third teaching style.

it is customary to distinguish between the two variables by giving them different labels. as the term itself suggests. In our case. independent variable that is teaching method. and freeexperiments? True experiments: Tuckman (1978) stated that it provides completely adequate controls for all sources of internal invalidity. . The teacher want to persuade his/her new method is better than traditional method. The variable upon which the independent variable is acting is called the dependent variable. So. In such a case. They represent no compromise between experimental design requirements and the nature and reality of the situation in which a study is being undertaken. quasi-experiments. The kind of situation requires experiment as an appropriate way of gathering data The teacher found the new innovative method in teaching and learning English. First. it is clearly known that variables in English language teaching and learning divide into two categories. Secondly. and so on. Free-experiments: Tuckman (1978) stated that free experimental designs do not qualify as legitimate experimental designs because they do not control adequately against the sources of internal invalidity.What is the difference between true experiments. Based on the explanation above. he/she should collect evidences from the teaching method and it correlation with the students’ development. dependent variable that is the result of the students’ achievement. is anything which does not remain constant. motivation. Quasi-experiments: Tuckman (1978) stated that quasi experimental design exist for situations in which complete experimental control is difficult or impossible. it includes language proficiency. aptitude. He/she has used it and made development to his/her students. Activity 2 The Two Ways of Classifying Variables Nunan (1992) stated that a variable. The label given to the variable that experimenter expects to influence the other is called the independent variable.

giving us information on the extent to which a set of scores varies in relation to the mean. . we know that there is some factor apart from dependent and independent variables. standard deviation. There is always possibility that some factor other than the experimental materials has brought about the observed differences in the scores. for example. those are: Mean. With only that information. we have two classes whose mean reading scores are the same. We can’t conclude the result only from the correlation of both variables. one must be familiar with the following statistical concepts: Mean. In this situation we will need to have teaching strategies more challenged. and standard error. The explanation above shows us to know the solution to conduct the experiment in a school. For example. normal distribution. Standard deviation helps researcher in measuring the variability of a mean. it is still no rigorous enough to allow you to claim that there is a casual relationship between the independent variable and the dependent variable. After we done the concepts we could reasonably claim that our research is in true experiment. it means we don't have any challenged students. and our teaching strategy will be entirely different. we have many students throughout the entire range of performance in the first class. We should follow the statistical concepts logically. But in the second class they show low Standard deviation. Based on the explanation. and standard error. The important of standard deviation for studying numerical data Nunan (1992) stated that the standard deviation on the other hand. standard deviation. is the most important measure of dispersion. Nunan (1992) stated that in order to appreciate the logic behind the procedures. They're all average. we would be inclined to teach the two classes in the same way. It used in evaluating values in records set to the mean and measuring of dispersion. but there is possibility we should observed some factor which make our research acceptable. In our research. From the explanation above.The difficulties and solutions to conduct experiment in a school setting Nunan (1992) argued that while your research design is becoming more rigorous. normal distribution.

Activity 3 Intact.ANOVA as the best technique of data analysis Nunan (1992) stated that when comparing more than two means. ANOVA can test more than one treatment is a major advantage over other statistical analysis such as the t-test. The ability of ANOVA is to test the significance of interactions between different variables.group comparison Improved control for history but weak on external validity Pretest – posttest control group design does not control for history or selection Posttest – only control group design True design with no testing effect One. Based on the explanation above.shot case study does not control for selection Equivalent time-samples design True design with possible testing effect Separate-sample pretest posttest one – group design repeated twice Nonequivalent control group design Imperfect control of selection but better than nondesign Posttest-only control group design true design with no pretest bias Time-series design inadequate control of history One-group pretest-posttest design imperfect control of history Factorial design true design for dealing with multiple independent variables Patched-up design combination of two non design . ANOVA’s use an F-ratio as its significance statistic which is variance because it is impossible to calculate the sample means difference with more than two samples. it opens up many testing capabilities. the appropriate test is the f-test. which is based on a procedure called analysis of variance (ANOVA). or more than two groups.

This design is identical to the previous one in all . Based on the prediction above we should be better to use quasi experimental design because the situations in which complete experimental control is difficult or impossible. There are times when a comparison or control group cannot be included in an experiment. It can be diagramed as follows: O1O2O3O4 X O5O6O7O8 Prediction: students given programmed math instruction will gain more in math achievement than students not given this instruction. Construct an experimental design to test this prediction. One kind of quasi experimental design that suitable for this case is time series design. Construct an experimental design to test this prediction. but this effect will be more pronounced among high math aptitude students than among low.Three design in terms of controlling for history bias Most adequate pretest-posttest control group design Next most adequate time-series design Least adequate one-group pretest-posttest design Four design in terms of controlling for selection bias Most adequate posttest-only control group design Next most adequate patched-up design Next Least adequate nonequivalent control group design Least adequate intact-group comparison Prediction: student teachers who are randomly assigned to urban schools for experience are more likely to choose urban schools for their first teaching assignment than student teachers who are randomly assigned to nonurban schools. There is possibility of bias because there is not enough randomized of Ss. Based on the prediction above it can be concluded that the best design is Nonequivalent Control Group Design.

a. One kind of quasi experimental design that suitable for this case is time series design. Which validity threat is not controlled? Maturation and History Prediction: student teachers who choose urban schools for experience are more likely to choose urban schools for their first teaching assignment than student teachers who choose nonurban schools. It has ability in controlling of selection of bias. creating a control problem in terms of selection bias. It can be diagramed as follows: O1O2O3O4 X O5O6O7O8 A school decides to implement a dental hygiene program for all its students. It predicts that cavities will be reduced as a result of this program. Which of the following circumstances necessitate the use of a quasiexperimental design? Experimenter cannot assign Ss to condition A patched up design has been created where this years’ first graders serve as the control group for a treatment being tried on this years’ second graders. .that is. It can be diagramed below: O1 X O2 ----------------------O3 O4 The procedures of this design are the same as for a true design except that intact groups rather than randomly assigned ones are used.respects except for the random assignment of subjects to conditions. especially by using a pretest. Why must a quasi-experimental design be employed to test this prediction? Because there is only a single group is available for study and the group pattern of the experience with the treatment is highly predetermined. There are times when a comparison or control group cannot be included in an experiment. Based on the prediction above we should be better to use quasi experimental design because the situations in which complete experimental control is difficult or impossible.

The following week the teacher checks student’s teeth (O 1). b.the researcher must expose the group to the treatment on some systematic basis.O2. The analysis of the data in this study is set up as shown above. Illustrated by the directive versus nondirective teacher study. construct one The criterion group design can be diagramed as follows. construct one The design used for the prediction above is Equivalent Time – samples Design.group design because the context of the prediction is suitable for the requirements of this design. Why must a criterion-group design be employed to test this prediction? We must use criterion. The interaction between the four measurements provides a check on differential changes overtime. Then the teacher ask a dentist to check again student’s teeth (X0) followed by the teacher’s checking on the next week (O2). And it is done simultanously until (O4). the criterion group approach might better have called a naturalistic study. b. An attempt is made to determine what characteristics are associated with the criterion group and have presumably preceded and thereby caused the criterion behavior. Prediction : children from broken homes will create a greater discipline problem in school ( as evidenced by demerits) than children from intact homes. A comparison of O1. a.O3 and O4 allows the teacher to compare the experiences. C O1 O1 C O2 C O1 . It is diagramed below X1 O1 X0 O2 X1 O3 X0 O4 The school takes the dental hygiene programme regularly for the week(X 1).

A researcher has just designed a special program to increase verbal I. It is a series of classroom lesson.Q. a. the purpose is to introduce a new procedure that is not anticipated to have specific effects related to the effects of the treatment or intervention being evaluated. unrelated intervention is deliberately introduced in order to create the Hawthorne effect which is often associated with intervention. why does the testing of this prediction call for Hawthorne control? The prediction above calls for Hawthorne control because the experimental intervention were after-school dance program for first graders and dependent variables are physical skills and social skills. . Hawthorne control (irrelevant experience) Positive teacher expectation Neutral teacher expectation created. would involve no contact whatever between experimenter and subjects. The nontreatment control condition. X H Ep En R O Ep O1 R X En O2 R H Ep O3 R H En O4 Experimental (relevant) treatment. Then Hawthorne intervention might take the form of technique in teaching dance to the children during the same period of the time that the experimental group was experiencing it. on the other hand. In Hawthorne control experiment an irrelevant. She wants to try it out in some schools.Prediction : an after-school dance program will improve the physical skills and social skills of first graders. b. Why would a Hawthorne control be a good idea? A Hawthorne control is a good idea because it represents a systematic intervention and interaction on the part of experimenter with the subject. Construct a design for testing it. a.

c. Why would teacher expectancy controls be a good idea? Teacher expectancy controls be a good idea because the teacher would believe that the experimental innovation would be successful.b. . Construct a design to test this program R O Ep O1 R X En O2 R H Ep O3 R H En O4 X Experimental (relevant) treatment. The outcome then would be a combination of the treatment plus the teacher’s expectation to success. H Hawthorne control (irrelevant experience) Ep Positive teacher expectation En Neutral teacher expectation created.