How to Use the Power Calculators

Disclaimer: This tutorial shows you how to use an online power calculator created by Dr. Russ Lenth at the University of Iowa for two purposes: (a) to calculate the target sample size for a planned study using a generic Cohens d effect size, and (b) to calculate post hoc power after a study has been completed, to determine what the chances were of finding a significant difference if one was really there. It should be noted that the creator of these calculators specifically disagrees with the use of power analysis for both of these purposes (see his note on the web page about two very wrong things that people try to do with his software). Please note that it is always advisable to do a pilot study and collect data on the actual between-group and within-group variability for the specific measures that you are interested in. The Cohens d effect size essentially short-cuts this process by using a ratio of the two quantities and labeling that ratio as small, medium, or large. However, it is sometimes useful to take this shortcut. Therefore, with apologies to the author of the website, this tutorial is an unauthorized commentary on how to use his very helpful software to accomplish pragmatic goals. How to Use the Power Calculators First, access the power calculators web page.

Next, choose the statistical test that you are planning to perform from this list by clicking on one of the entries, and then click on the run selection button. UCDHSC Center for Nursing Research Page 1 of 10 Updated 5/18/06

Your choices are:

Confidence Interval for 1 proportion Test of 1 proportion Test comparing 2 proportions (Fishers exact test) Confidence Interval for 1 mean One-sample t test (or dependent t-test) Two-sample t test (independent t-test) ANOVA (F-test)

Each of these options is described separately, below. Printing Your Results You will see the output of your power calculation on-screen (the calculators move the sliding bar for power or sample size immediately as you are making changes, so there is no run button to push in order to run the analysis). When you have a result that you are satisfied with, and you want to save it for future reference, do this: 1. Hold down the Control (Ctrl) key on your keyboard. 2. While still holding down Control, hit the print screen button (found at the top of your keyboard, to the right of the function keys but to the left of the number keypad). Nothing will appear to happen when you hit this key, but whatever is visible on your monitor will be copied into the clipboard. 3. Open a new Word document. 4. Select paste from the edit menu, or use Ctrl-V to paste. The picture of your monitor screen will appear inside the Word document. 5. Save the Word document that shows the image of your power analysis results. Citing the Power Calculators Dr. Lenth requests that you cite his website if you use his calculators for any published work or grant proposals. The requested citation is: Lenth, R. V. (2006). Java Applets for Power and Sample Size [Computer software]. Retrieved month day, year, from http://www.stat.uiowa.edu/~rlenth/Power. Please do not cite the CNR website as your source for these power calculators, because we didnt develop them. This wonderful tool is provided by Dr. Russ Lenth at the University of Iowa, and we are just providing a link to them.

UCDHSC Center for Nursing Research Page 2 of 10

Updated 5/18/06

If you are reporting on the prevalence of something in a population (e.g., the % of parolees who recidivate after release on parole), then you should use the confidence interval for one proportion. (x% plus or minus z% of criminals recidivate).

1) select the confidence level you want (e.g., 0.95 for 95% confidence) 2) type in the number of people in the population that your sample came from [e.g., all criminals. This is large N, as opposed to small n, the number of people in your actual sample]. If you cant quantify exactly how many people are in the larger population that you want to generalize to, you can un-check this check box to estimate results for an infinite population. 3) type in the proportion that you want a confidence interval around (e.g., if you found 20% recidivism in your sample, or if youre still just planning the study and thats what you expect to find, type in 0.20 here). 4) use the scroll bars: a. if you are planning a study, select the margin of error that you are willing to live with, and look at the bottom scroll bar to find out what sample size n you will need in order to get there. b. if you have already completed your study and want to know what confidence interval you can report, use the bottom scroll bar to select your actual sample size n, and look at the top scroll bar to find out what your margin of error is.

UCDHSC Center for Nursing Research Page 3 of 10

Updated 5/18/06

If you are testing whether a proportion is equal to some specific value (e.g., you found that 20% of people in your sex offenders group were abused as children, you know that only 5% of people in the general population were abused as children, and you want to know if this difference is statistically significant or might just be due to chance), then you should use the test of one proportion.

1) Use the top scroll bar to indicate what the population value is for the proportion you want to test (in the example above, 5% or .05). 2) Use the next scroll bar to indicate what the obtained value in your sample is for the proportion (in the example above, 20% or .20). 3) Generally, the three drop-down menus can be left at their default settings (the null hypothesis is that the sample proportion equals the population proportion; the alpha level is .05; and the method to be used is based on the standard normal curve). 4) Use the other two scroll bars to get your results: a. If you are planning a study, select the power level you want (usually at least .80), and see what n is needed for your sample. b. If you have already completed the study, select the actual sample size n that you have, and see what level of power you have to avoid making a Type II error. UCDHSC Center for Nursing Research Page 4 of 10 Updated 5/18/06

If you are testing whether two groups have different rates of something (e.g., whether teenagers with poor social skills have a higher suicide rate than teenagers with good social skills), then you should use the test of equality of two proportions (also known as a Fishers Exact Test).

1) Use the first two scroll bars to put in the (actual or expected) proportions for the two groups. Your effect size is determined by how far apart the proportions are. 2) Generally you can leave these drop-down menus on their default settings: the null hypothesis is that the two proportions are equal, and the alpha level equals .05. 3) Use the other two scroll bars to get your results: a. If you are planning a study, select the power you want (usually at least .80), and see how large an n you need in each group for your study. b. If you have already completed your study, select the actual n in each group (uncheck the equal ns checkbox if there are different numbers in the two groups), and find out how much power you had to avoid a Type II error.

UCDHSC Center for Nursing Research Page 5 of 10

Updated 5/18/06

If you are reporting on the average value of something in a population (e.g., the average IQ score among people with a first-degree relative who has bipolar disorder is x, plus or minus z IQ points), you should use the confidence interval for one mean.

1) select the confidence level you want (e.g., 0.95 for 95% confidence) 2) type in the number of people in the population that your sample came from [e.g., all criminals. This is large N, as opposed to small n, the number of people in your actual sample]. If you cant quantify exactly how many people are in the larger population that you want to generalize to, you can un-check this check box to estimate results for an infinite population. 3) Use the top scroll bar to indicate the standard deviation of your sample (if you dont know, take a guessif your average is larger, your standard deviation may be larger as well). 4) Use the other two scroll bars to get your results: a. if you are planning a study, select the margin of error that you are willing to live with, and look at the bottom scroll bar to find out what sample size n you will need in order to get there. b. if you have already completed your study and want to know what confidence interval you can report, use the bottom scroll bar to select your actual sample size n, and look at the top scroll bar to find out what your margin of error is.

UCDHSC Center for Nursing Research Page 6 of 10

Updated 5/18/06

If you are testing for the differences between average pre- and post-test scores for a single group of people (e.g., average depression score before vs. after treatment), or if you are testing differences between two groups where the members are connected with each other (e.g., scores for husbands vs. wives), you should use the one-sample t test.

1) your alpha level (in the drop-down menu) will usually be set at .05. 2) Use the default for a two-tailed test if you arent sure which of the two groups will have a higher mean than the other, or if you have a specific theory, you can uncheck this box to get a one-tailed test. 3) To enter an effect size, use the top two scroll bars. The Cohens d effect size is defined as the true difference between means divided by the pooled standard deviationso, if you leave the standard deviation (sigma) set at 1, the Cohens d effect size will be equal to the true difference between the means. Leave the top scroll bar alone, and use the second scroll bar to select a Cohens d effect size. a. If you think you will find a small difference between groups, use .5 b. If you think you will find a moderate difference between groups, use .75 c. If you think you will find a large difference between groups, use 1.00 d. If you think you will find a very large difference between groups, use 1.25 4) Use the other two scroll bars to: a. Enter Power (usually at least .80), and solve for sample size n, or b. Enter your actual sample n, and find out how much Power your study had to avoid a Type II error. UCDHSC Center for Nursing Research Page 7 of 10 Updated 5/18/06

If you are testing for differences in scores on some measure between two independent groups of people (i.e., the groups are not correlated with each other), you should use the two-sample t test.

1) your alpha level (in the drop-down menu) will usually be set at .05. 2) Use the default for a two-tailed test if you arent sure which of the two groups will have a higher mean than the other, or if you have a specific theory, you can uncheck this box to get a one-tailed test. 3) To enter an effect size, leave the two scroll bars marked sigma set at 1 (also leave the equal sigmas checkbox checked), and use the true difference of means scroll bar to select a Cohens d effect size. a. If you think you will find a small difference between groups, use .5 b. If you think you will find a moderate difference between groups, use .75 c. If you think you will find a large difference between groups, use 1.00 d. If you think you will find a very large difference between groups, use 1.25 4) Use the other scroll bars to: a. Enter Power (usually at least .80), and solve for the sample size n that will be required in each group to get the desired level of power, or b. Enter your actual sample n (which can be different for the two groups), and find out how much Power your study had to avoid a Type II error. UCDHSC Center for Nursing Research Page 8 of 10 Updated 5/18/06

If you are testing for differences between three or more groups, you should use the Balanced ANOVA (F-test) power calculator. The F-test selection takes you through a preliminary screen before you get to the actual power calculator:

If you are doing a basic F-test (one-way ANOVA), just click the F tests button to go on (you can change the other values later). If you are doing a more complicated test (like a two-way ANOVA, discussed in the advanced statistics class), select the appropriate test from the drop-down menu before clicking the F tests button to go on. Note that you can use this same screen to examine two-way designs (by specifying additional variables in the model line), or to examine ANCOVA designs (by specifying covariates in the random factors line). For MANOVA designs, you can test the power of the test on one individual dependent variable, but the power of the multivariate test will be influenced by the degree of multicollinearity between the dependent variables, and the direction of the effect is not always the same: Power for the multivariate test is highest when the pooled within-cell correlation among two DVs is high and negative. The multivariate test has much less power when the correlation is positive, zero, or moderately negative. An interesting thing happens, however, when one of the two DVs is affected by the treatment and the other is not. The higher the absolute value of the correlation between the two DVs, the greater the power of the multivariate test.Tabachnik & Fidell, Multivariate Statistics, 4th ed. (2001), p. 329. Therefore, additional consideration is needed for MANOVA designs. UCDHSC Center for Nursing Research Page 9 of 10 Updated 5/18/06

This screen shows the power analysis setup for a basic F-test (one-way ANOVA):

1) Select the number of groups you are comparing, using the levels[treatment] scroll bar. The default is two groups; but you can select three or more. Leave the radio button set to fixed effects. 2) Select an alpha level for your test (usually .05). Remember that if you are performing multiple tests (e.g., if you are testing for differences between groups on more than one variable), you should use a more conservative alpha level (such as .01) due to the Bonferroni correction. 3) To select an effect size, leave the SD[Within] (which represents random error) set at 1, and move the SD[treatment] (which reflects the average difference between group means) to a level that represents your predicted effect size. When the SD[Within] is 1, then the SD[treatment] will give you a Cohens d. a. If you think you will find a small difference between groups, use .5 b. If you think you will find a moderate difference between groups, use .75 c. If you think you will find a large difference between groups, use 1.00 d. If you think you will find a very large difference between groups, use 1.25 4) Use the other scroll bars to: a. Enter Power (usually at least .80), and solve for the sample size n that will be required in each group to get the desired level of power, or b. Enter your actual sample n (which can be different for the two groups), and find out how much Power your study had to avoid a Type II error.

UCDHSC Center for Nursing Research Page 10 of 10

Updated 5/18/06