Matching Statistics with the Research Design

GO TO PART ONE OF RESEARCH DESIGNS This document focuses on how to select the correct statistical procedure for your research study. This is an important element of your research design. The choice of how the data is analyzed effects the number of subjects and the way you measure the dependent variables. To avoid using too few subjects or gathering the wrong type of data, you need to decide on the data analysis method during the design phase of your research. A key element to selecting the correct statistical test is to fit the design one is using into one of four patterns. The patterns are made by the sequence of events; the Xs and Os of the design. You must evaluates the design diagram by looking at the pattern of Xs and Os for group 1, by counting the number of Os for group 1, and by knowing the number of groups. The simplest pattern is pattern 0. This pattern consists of at least one group and one observation and is used for descriptive and relationship research questions.
EVENT 1 ------------------GROUP 1 O -------------------

(Pattern 0)

The remaining three patterns are used for differences research questions. Pattern 1 is an experimental design that is all observations and has one or more groups.
EVENT 1 ------------------GROUP 1 O GROUP 2 O GROUP 3 O ------------------EVENT 1 2

(Pattern 1)

to deduce the design's pattern. The DV may be observed more than once and still the design is assigned to pattern 2. Pattern 3 designs are assigned to analysis of covariance when the DV uses interval measurement and there are three or more observations of one group. It is usually assumed that the first observation is used as a control measure. Once you have the pattern you can use Table 2 to locate the correct statistical test. event 2 indicates that group 3 is a control or no treatment group. TABLE 2 Relating Research Designs to Appropriate Statistical Analyses ------------------------------------------------------------------DESIGN STATISTICAL TEST ------------------------------------------------------------------DIFFERENCES RESEARCH QUESTION 1. Basic two-group design 1. Pre-test and post-test 2. EVENT 1 2 3 -------------------------GROUP 1 X1 O1 O2 GROUP 2 X2 O1 O2 -------------------------- (Pattern 2) Pattern 3 is for experimental designs that have the sequence of events: observe.----------------------GROUP 1 O O ----------------------- (Pattern 1) Pattern 2 is an experimental design that has a pattern of events where first a treatment is done and then the DV is observed. but you only need to count the total number of events and observations. observe. In this case. EVENT 1 2 3 ----------------------------GROUP 1 O1 X1 O2 GROUP 2 O1 X2 O2 GROUP 3 O1 O2 ----------------------------- (Pattern 3) The blank cell for group 3. or two or more observations of two or more groups. t-test . a. and examine the first three events. treat. Mann-Whitney U test (Ordinal data) c. Designs can have any number of events. t-test . a. the first (pretreatment) observation is used to predict the post treatment observations and then the residuals are analyzed for differences due to the treatment.independent means (Pattern 1 or 2) (Interval or ratio data)* b. Chi-square (nominal data) 2.dependent .

Analysis of variance (Interval) b. One-group t-test (Interval) known population. One-group sample from a 6. Another example. Lambda Beta. Repeated measures analysis of variance OR Analysis of co-variance (Interval) b. Type of research question: Differences . Chi-square goodness-of-fit test (Nominal) RELATIONSHIPS RESEARCH QUESTION (Pattern 0) 7. Phi coefficient or Chi-square (Nominal) ___________________________________________________________________ * Refers to the way the dependent variable is measured. and one group) (Interval) b. Kolmogorov-Smirnov test for goodness-of-fit (Ordinal) c.2 or 3) 5. Interrupted time-series analysis (interval) a. Kruskal-Wallis AOV (Ordinal) c. Continuous variables use interval or ratio measurement Categorical variables use nominal or ordinal measurement For DESCRIPTIVE RESEARCH QUESTION (Pattern 0) 6. Chi-square test for K independent groups (Nominal) 3. (Pattrn 3) (non-independent) means (Interval) b. McNemar test (Nominal) 3. Spearman's rank order correlation. Given the two-group design shown below. 4. a. 4. Pearson product moment (Two or more variables correlation coefficient. will need the t-test for dependent means. (Pattern 1 or 3) 5. Three or more groups design (Pattern 1. b. with interval measurement of the DV. Kendall's Tau (Ordinal) c. Time-Series or Single Subject (Pattern 3) Covariance. Friedman's AOV by ranks (Ordinal) c. or repeated measures design. Correlational study 7. a. then you would use the Wilcoxon or Sign test instead of the t-test. EVENT 1 2 3 -------------------------GROUP 1 O1 X O2 -------------------------(Example 1) If you changed the scale of measurement used by the DV to ordinal. a one group pre-post design (Example 1). Wilcoxon or Sign test (Ordinal) c. Cochran's Q (Nominal) a.

These tests are appropriate for large sample sizes. and if you have two IVs it is a two-factor AOV.Number of independent variables: Type of variable : Number of dependent variables : Type of variable : Number of Events Number of Groups : 3 : 2 1 categorical 1 continuous EVENT 1 2 3 -----------------------GROUP 1 O1 O2 O3 GROUP 2 O1 O2 O3 -----------------------Statistical test to use: Use a one factor repeated measures analysis of variance.Use this test when you have two-groups and a single measurement of the dependent variable (DV). Categorical IV and Continuous DV All the tests in this section are for differences research questions. and 3) if there is a difference between the group means (An interaction effect.Used when you have three or more groups and measure the DV once. Table 2 provides a summary of this section. see Chapter 4 for more details). Some of them can be used with smaller group sizes (10 to 15 subjects per group). It tells you if the mean of the first measurement is different from the mean of the second.e. The two-factor AOV tells you three things: 1) if there is a difference between the DV means for the first IV. Analysis of variance (AOV) .Used when you have one group and you measure the DV twice.. This test tells you if the mean of one group is different from the mean of the other. It tells you if one of the group means is different from the other means. AOV designs (and RAOV and . Remember all continuous variables use interval or ratio measurement. 2. Catalog of Statistical Tests This section describes the uses for the statistical tests that are available to the researcher for data analysis. but the validity of the test results can be reduced when a small group size is used. t-test for dependent means . If you have one independent variable (IV) it is a one-factor AOV. An AOV design with two factors has one group for each combination of the categories of the two IVs. twenty or more subjects per group. i. If one IV has 2 categories and the second IV has 3 categories then the study will have 6 groups (3 x 2). 2) if there is a difference between the DV means for the second IV. 3. t-test for independent means . 1.

. 4. 1. or AOCovar. Can have one or more IVs. they are both measuring the same trait) but that the immediate measurement is effected more by exercise level while resting heart rate is effected more by overall physical condition. have two or more groups. Categorical IV and DV The tests in this section are called non-parametric tests because they are based on categorical data. Analysis of covariance (AOCovar) . and have one or more after treatment observations. Since you have only one group you do NOT have an independent variable. 6. If you have only two groups you use Hotelling's T-squared statistic. In addition to the information provided by an AOV design (see item 3 above) it tells you if the groups are different across the several measurements of the DV. two observations equivalent of the AOCovar. This test allows you to see how the DVs relate to each other. and a single measure of the DV you can see if your sample mean is different from the mean of the population (that is you can if you know the population mean). A multivariate AOV will tell you about the effects of the exercise levels on each heart rate measurement. 7. Also appropriate for three or more observations of one group.If you only have one group. The t-test for dependent means is the one group. This test is useful to see if your sample is representative of the population. You can have one or more IVs. For example. One group t-test . . Repeated measures analysis of variance (RAOV) . you may use two different measures for heart rate (resting and immediately after exercise) as DVs and your single IV may have two categories of exercise levels.Used when you have three or more groups and measure the DV two or more times. Your results will show that the resting heart rate and heart rate immediately after exercise are correlated (e. 5. a descriptive research question. They can be used with small sample sizes between five and twenty per group. When data are frequencies (counts of the number of times an observation falls into one of several nominal categories) use these methods.Used when you measure the DV before and after the treatment.When you have two or more DVs and three or more groups you can also run a multivariate AOV. Analysis of covariance is most useful when the groups are not randomly assigned and you want to control for any initial differences between the groups. Multivariate analyses . RAOV.AOCovar below) can be used with as small as four subjects per group and still obtain valid results. It tells you if the DVs overlap in terms of what they measure or if they each measure a separate trait or concept.g.

It is the nonparametric parallel to Pearson's product moment correlation coefficient.Used with two observations of one group. McNemar test . Lambda beta . one or more IVs.Used when you have three or more observations of a nominal variable and one group. e. yes/no or 1 / 0). d. a. This type of variable is called a dichotomous variable. This measure is not the same as Person's product moment correlation coefficient. and one group you can use the Chi-square goodness-of-fit test to see if your sample frequencies are the same as the population frequencies. Cochran's Q . Nonparametric equivalent of the t-test for dependent means. and one or more DVs you can see if a relationship exists between the variables or the groups. When data are scores (ordinal measurement) use these methods. c. If there is a relationship. here we will discuss three uses: i) When you have one set of observations. These can be used with interval data as well by converting the interval data to ranks. If there is no relationship then the groups are the same. nominal data. The frequencies in the categories of the DV are different is a systematic way if a relationship exists.a. Chi-square . ii) When you have one set of observations and one or more groups. one DV. Similar to a correlation but it only tells you if a relationship exists not the strength of the relationship (as does a correlation coefficient).This is a measure of relationship used when you have two nominal variables that each have two or more categories. iii) You can also use Chi-square for the same purpose as a t-test for independent means or a one-way AOV when you only have frequency data. .This is a measure of relationship (association) used when you have two ordinal (rank order) variables. Phi coefficient . A Chi-square test can be used for both a relationship question and a differences question because at this level both questions have the same answer. This test is the non-parametric parallel to the repeated measures AOV. This is similar to the one-group t-test. This measure gives the same result as Pearson's product moment correlation coefficient. Lambda beta is a more general measure of association than the Phi coefficient.g. then you must use your knowledge of group membership to interpret the results. If a relationship exists then the groups are different.There are many uses for Chi-square tests.. Spearman's rank order correlation . b. 2.This is a measure of relationship used when you have two variables that each only have two categories (e.

signed-ranks test.Used to compare your sample distribution to the population distribution.Also called univariate regression analysis because there is only one DV used. The equation looks like this: . This test is not as powerful as the Wilcoxon because it only uses direction of change (+ or -) and not the actual ranks. Wilcoxon Matched-pairs. This test is the non-parametric equivalent of the t-test for independent means. Sign test . Kolmogorov-Smirnov test for goodness-of-fit .Used when you have two observations of one group. Friedman analysis of variance by ranks .. It is best to use Kendall's method when you have ten or fewer subjects. f. This test is sometimes called a two-way AOV because the subjects are considered as the second factor. Kendall's Tau . Smaller sample sizes can be used (10 to 15 per group) but the validity of the results can be reduced.b.This method provides the same measure of relationship as the Spearman. ordinal data. ordinal data. e. Regression analysis . signed-ranks test .Used when you have three or more observations of one ordinal DV and one group.Used with pairs of continuous variables. This test is the non-parametric parallel to the AOV. d. h. This is a mathematical procedure not a statistical test. c. twenty or more per group. Use this test when you have ordinal (rank order) data. 2. 1.Used when you have three or more groups and one observation of an ordinal variable. i. Mann-Whitney U-test . This test uses the ranks as data. It produces an equation that lets you predict an individual's score on the DV from his score on the IV. It provides the same information as Cochran's Q or the Repeated measures AOV except that it is used with rank order data. It is not a true two factor design. Continuous IV and DV All of these procedures are used with relationships research questions and require large sample sizes. Kruskal-Wallis one-way analysis of variance by ranks . This test provides the same information as the one-group t-test or Chi-square goodness-of-fit test. Pearson's product moment correlation coefficient .Used for exactly the same data as the Wilcoxon Matched-pairs.Used when you have two groups and observe the DV once.e. It is similar in function to the t-test for dependent means and the McNemar test for nominal data. It provides a measure of the degree of relationship between the two variables. g.

Non-parametric Methods of Quantitative Analysis by Jean D. Multiple regression analysis also provides a measure of relationship called Multiple R. SAS-Statistical Analysis System from SAS Institute Inc. and BMD-Biomedical Computer Programs by W. Gibbons. Multivariate Multiple Regression Analysis . Winer. Jean G. 4.cdc. There are several computer programs available that ease the pain of computing statistics by hand. Three good books for this purpose are: Statistical Principles in Experimental Design by B. Glass and Julian C. Three major ones are: SPSS-Statistical Package for the Social Sciences by Norman H. There are no statistical tests for designs with continuous IVs and categorical DVs. Jenkins. Epi Info.. . This procedure provides a set of prediction equations (one for each DV). The computational procedures for these statistical procedures can be found in many books on statistics.http://www. 3.This procedure combines the functions of Pearson's correlation and regression analysis when you use two or more IVs to predict one DV. The regression equation does not tell you the accuracy of your prediction. Multiple regression analysis . J. Nie. J. You can read about it here . This Multiple R (or R squared) tells you the accuracy of prediction of the DV from the several IVs.DV = I + IV(S) Where I is called the intercept and S is called the slope. B2. The correlation coefficient tells you the accuracy of the prediction. and Dale C Hadlai Hull. The beta weights are values that correspond in function to the slope used in the univariate regression analysis. or both continuous. Karen Steinbrenner.e. Some on-line statistical programs are available from the US Center for Disease Control. tells you the accuracy of prediction of each DV from the IVs. Bent.This procedure is used when you have several IVs (like multiple regression above) and you have several DVs. Dixon. Relationship designs require that the IVs and DVs both use the same type of measurement. Stanley. The procedure provides a regression (prediction) equation that looks like this: DV = I + (IV1 * B1) + (IV2 * B2) + (IVn * Bn) Where I is the intercept. a statistics package which is available for free from the CDC web site. and finally tells you the degree to which the DVs are related and lets you study the structure of the relationship. and Bn are called beta weights (Bn is the nth beta weight for the nth IV). and B1.. both categorical. i. and Statistical Methods in Education and Psychology by Gene V.

Lohnes and Computational Handbook of Statistics by James L. Cooley and Paul R. Kintz . L. Bruning and B.Two excellent books that combine statistical procedures with computer programs you can use to compute statistical tests is Multivariate Data Analysis by William W.

Sign up to vote on this title
UsefulNot useful