Attribution Non-Commercial (BY-NC)

141 views

Attribution Non-Commercial (BY-NC)

- 1234
- Cupcake Height
- Chapter 04 Answers
- Engg Mathematics - 4 July 2011
- fitp
- iyama1
- yepp ah
- M4 All Slides
- IJIEC_2013_32
- Lab10 Augmented Lattice
- Ken Black QA 5th chapter14 Solution
- 13047_2014_Article_55.pdf
- Stata Commands
- Study Session
- Tests of Normality MB LENI
- Professional Realty Word
- EffectSize - CBU Statistics Wiki
- Accounts
- Introduction to ANOVA
- STAT6140_Assignment4_2014

You are on page 1of 11

4) 1

Random-, and Fixed-Eﬀects Models

Wolfgang Viechtbauer

University of Maastricht

http://www.wvbauer.com/

March 8, 2006

Contents

1 Meta-Analytic Models 2

1.1 Mixed-Eﬀects Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Random-Eﬀects Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.3 Fixed-Eﬀects with Moderators Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.4 Fixed-Eﬀects Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

4 Example 4

6 Categorical Moderators 8

12 References 11

MiMa Function Tutorial (version 1.4) 2

1 Meta-Analytic Models

Given are i = 1, . . . , k independent eﬀect size estimates, each estimating a corresponding eﬀect

size. This can be expressed algebraically by

Yi = θi + i , (1)

where i is the sampling error for the ith study. The sampling errors are assumed to be nor-

mally distributed with mean zero and variance vi . Now assume that p moderator variables are

inﬂuencing the eﬀect sizes in a linear fashion. Let Xij denote the value of the j th moderator

variable for the ith eﬀect size. We then write

where ui is assumed to follow a normal distribution with mean zero and variance τ 2 . The

average eﬀect size is given by β0 +β1 Xi1 +. . .+βp Xip , while the study-speciﬁc eﬀect size is given

by (2). Therefore, τ 2 denotes the amount of residual heterogeneity, that is, the amount of excess

or unexplained variability in the eﬀect sizes after accounting for the amount of heterogeneity

we would expect based on the moderator variables that are included in the model. Combining

Equations 1 and 2 yields the mixed-eﬀects model, which is given by

Therefore, βj denotes by how much the average eﬀect size changes for a one unit increase in

the corresponding moderator variable when holding all other moderator values constant, while

β0 denotes the average eﬀect size when all moderator variables are equal to zero.

When no moderators are included in the model, then the mixed-eﬀects model simpliﬁes to

Yi = µ + ui + i , (4)

where β0 has been replaced with µ. According to the random-eﬀects model, all of the hetero-

geneity among the eﬀect sizes is random and cannot be accounted for by moderators. There-

fore, τ 2 in the random-eﬀects model simply denotes the amount of total heterogeneity, that is,

the amount of variability in the eﬀect sizes.

When moderators are included in the model and the moderators can account for all of the

heterogeneity among the eﬀect sizes, then the ﬁxed-eﬀects with moderators model applies,

which is given by

Yi = β0 + β1 Xi1 + . . . + βp Xip + i . (5)

Therefore, this model is again just a special case of the mixed-eﬀects model, where τ 2 = 0.

MiMa Function Tutorial (version 1.4) 3

When all of the eﬀect sizes are homogeneous (i.e., no moderators are operating and there is no

random heterogeneity among the eﬀect sizes), then θ1 = θ2 = . . . = θk = θ and the appropriate

model is given by

Yi = θ + i , (6)

using θ in place of β0 .

Once estimates of β0 through βp and τ 2 have been obtained (i.e., once the model has been

ﬁtted), we can conduct a variety of hypothesis tests:

from zero (i.e., H0 : τ 2 = 0). For this, we use the QE statistic, which follows a chi-square

distribution under the null hypothesis with degrees of freedom equal to k − p − 1.

2. We can also test whether any of the β1 , . . . , βp parameters in the model are signiﬁcantly

diﬀerent from zero, or in other words, whether at least one moderator is inﬂuencing the

eﬀect sizes (i.e., H0 : β1 = β2 = . . . = βp = 0). For this, we use the QME statistic, which is

distributed chi-squared with p degrees of freedom (i.e., equal to the number of parameters

tested) under the null hypothesis.

3. We can also test individual parameters, by dividing the parameter estimate by its stan-

dard error. This ratio follows a standard normal distribution under the null hypothesis.

Conﬁdence intervals for each parameter are also easily obtained by inverting these tests.

Notes: (1) The sampling variances (i.e., the vi values) are usually not exactly known. In-

stead, the vi values are replaced with corresponding estimates (v̂i ). Therefore, the distribu-

tions of the test statistics above are only exact when the within-study sample sizes are large

(i.e., when the error in the sampling variance estimates is negligible). (2) Moreover, when ﬁt-

ting a mixed/random-eﬀects model, then τ 2 must be estimated (τ̂ 2 ). The estimate is then

simply treated as a known constant. Therefore, the QME statistic and the individual param-

eter tests assume that k is suﬃciently large, such that τ 2 is estimated with high precision

and the error in the estimate is negligible. (3) Finally, most eﬀect size measures are not ex-

actly normally distributed, as assumed under the meta-analytic models. However, the normal

distribution approximation usually becomes more accurate as the within-study sample sizes

increase. Therefore, large within-study sample sizes are needed to be certain that the tests

perform as desired (speciﬁcally, that the Type I error rate is controlled). Reﬁned moderator

tests that provide better control of the Type I error rate when these assumptions are violated

have been developed and will be included in the MiMa function later on.

MiMa Function Tutorial (version 1.4) 4

The S-PLUS/R function “mima” can be used for ﬁtting the various meta-analytic models.

Usage:

verbose="no", digits=4, fe="no", out="no")

Required Arguments:

vi -- a vector of sampling variances

mods -- a vector/matrix of moderator values

Optional Arguments:

possible methods are: "REML", "ML", "HE", "DL", "SH", or "EB";

the default is "REML" (see below for more details)

threshold -- precision of the Fisher scoring algorithm when using either

ML, REML, or EB estimation; the default is 0.00001

maxiter -- maximum number of iterations of the algorithm when using either

ML, REML, or EB estimation; the default is 100

alpha -- to set the width of the confidence interval for individual

parameters (default is .05, which yields a 95% CI)

verbose -- set to "yes" to obtain information on the evolution of the

iterative algorithm when using either ML, REML, or EB estimation

digits -- number of decimals for rounding of results; the default is 4

fe -- set to "yes" to force fitting a fixed-effects model

out -- see below for explanation

Several methods for estimating the amount of (residual) heterogeneity are available. The de-

fault is restricted maximum-likelihood (REML) estimation and this is the recommended method

(Viechtbauer, 2005). Regular maximum-likelihood (ML) estimation is also available, but it is

not recommended, because the estimate of τ 2 tends to be too small (negatively biased). Two

method of moment estimators are also available (HE and DL, respectively); the ﬁrst is an ex-

tension of the estimator suggested by Hedges (1983, 1989) and the other an extension of the

estimator suggested by DerSimonian and Laird (1986). More details on these two estimators

can be found in Raudenbush (1994). Finally, the estimator suggested by Sidik and Jonkman

(2005a, 2005b) and the empirical Bayes estimator (Berkey, Hoaglin, Mosteller, & Colditz, 1995;

Morris, 1983) are also available (SH and EB, respectively). The ML, REML, and EB estimation

methods make use of the Fisher scoring algorithm, which is robust to poor starting values and

usually converges quickly (Harville, 1977; Jennrich & Sampson, 1976).

4 Example

Table 1 provides the results for k = 16 studies examining the eﬀectiveness of massage therapy

for relieving state anxiety. Listed are the sample sizes of each study (nEi and nC

i ), the eﬀect size

MiMa Function Tutorial (version 1.4) 5

estimate (Yi ), the estimated sampling variance (v̂i ), the minutes per session of massage therapy

provided in each study, whether a fully trained massage therapist or a layperson provided the

therapy, and the mean age of the sample. The data were adopted from Moyer, Rounds, and

Hannum (2004), leaving out three studies with missing data and two studies where the duration

of massage therapy provided was less than 10 minutes.

Table 1

Results for 16 Studies on the Eﬀectiveness of Massage Therapy for Reducing State Anxiety

Sample Sizes Eﬀect Size Sampling Minutes/ Trained Mean

Study nEi nCi Estimate (Yi ) Variance (v̂i ) Session Therapist Age

1 30 30 0.444 0.068 30 0 28

2 46 39 −0.495 0.049 10 0 42

3 15 15 0.195 0.134 20 1 31

4 10 10 0.546 0.207 40 1 39

5 12 12 0.840 0.181 20 1 17

6 10 10 0.105 0.200 30 1 51

7 26 24 0.472 0.082 15 1 26

8 18 14 −0.205 0.128 10 0 64

9 12 12 1.284 0.201 45 1 48

10 12 12 0.068 0.167 30 1 40

11 15 15 0.234 0.134 30 1 52

12 12 12 0.811 0.180 30 1 33

13 15 15 0.204 0.134 30 1 20

14 18 18 1.271 0.134 60 1 27

15 15 15 1.090 0.153 45 1 52

16 43 35 −0.059 0.052 10 1 61

Note. Adopted from Moyer, Rounds, and Hannum (2004). Minutes/Session = minutes of therapy provided per

session; Trained Therapist: 0 for a layperson providing the therapy and 1 for a trained therapist; Mean Age =

mean age of the sample.

The data are available in the S-PLUS dataset ﬁle “dat.sdd”. The names of the variables are:

n1, n2, yi, vi, minutes, trained, and meanage. In order to use the mima function, we must

ﬁrst create a matrix with the moderator values that we want to include in the model. For this,

load the dataset, open up the command window (click on “Window” in the menu bar and choose

“Command Window”), and type:

> attach(dat)

> mods <- cbind(minutes, trained, meanage)

The attach command makes the variables in the dataset available (technically, it adds the

dataset to the S-PLUS search path). The second command creates a matrix, called “mods”,

where the columns are composed of the minutes, trained, and meanage variables.

Next, load the mima function by double-clicking on the ﬁle called “mima.ssc” (if asked

whether to overwrite the existing script, then this means that the function is already loaded,

but to be certain that the newest version of the function is available, select ok). Once the script

MiMa Function Tutorial (version 1.4) 6

is loaded, run it (i.e., make it available) by hitting F10 (you can then close the script window).

Finally, to ﬁt the mixed-eﬀects model to the massage therapy data, simply type:

QE = 9.1278

df = 12

p-value = 0.692

Parameter Estimates:

[,1]

intrcpt -0.2971

minutes 0.0250

trained 0.3013

meanage -0.0059

intrcpt 0.1218 -0.0013 -0.0112 -0.0018

minutes -0.0013 0.0000 -0.0005 0.0000

trained -0.0112 -0.0005 0.0382 -0.0001

meanage -0.0018 0.0000 -0.0001 0.0000

QME = 28.9176

df = 3

p-value = 0

intrcpt -0.2971 0.3490 -0.8513 0.3946 -0.9811 0.3869

minutes 0.0250 0.0067 3.7172 0.0002 0.0118 0.0381

trained 0.3013 0.1954 1.5422 0.1230 -0.0816 0.6842

meanage -0.0059 0.0063 -0.9488 0.3427 -0.0182 0.0063

MiMa Function Tutorial (version 1.4) 7

The output shows that no residual heterogeneity remains when all three moderators are

included in the model (the estimate of residual heterogeneity is zero). The test for residual

heterogeneity is also far from signiﬁcant (QE = 9.13, df = 12, p = .692). The QME statistic is

highly signiﬁcant (QME = 28.92, df = 3, p < .0001), indicating that at least one moderator vari-

able is related to the eﬀect sizes. The individual parameter tests suggest that it is the minutes

moderator that is signiﬁcant (95% conﬁdence interval bounds for this parameter: (0.01, 0.04),

z = 3.72, p = .0002).

One can also obtain a conﬁdence interval for the amount of (residual) heterogeneity in the

model with the tau2cime.r function available also on the author’s website. Simply open that

script in S-Plus and source it with F10. Then type:

95% CI: (0, 0.144)

which shows that the bounds for a 95% conﬁdence interval for τ 2 are given by (0, 0.144). For

more information about this function, see the author’s website.

Once a model has been ﬁtted, it is easy to obtain predicted/estimated eﬀect sizes from the

output. Recall that the output for the mixed-eﬀects model ﬁtted earlier included the following

parameter estimates:

[,1]

intercept -0.2971

minutes 0.0250

trained 0.3013

meanage -0.0059

Therefore, the predicted/estimated eﬀect size for 30 minutes of therapy provided by a fully

trained therapist to a group with an average age of 40 is:

We can use S-PLUS/R to carry out these calculations. We ﬁrst ﬁt the model again, using the

out="yes" option. With this argument set to yes, the estimate of residual heterogeneity, the

parameter estimates, and the variance-covariance matrix of the parameter estimates are given

in list form for further processing. Speciﬁcally, if we write:

then the output is put into the object “res”. Next, we select some moderator values and then

multiply the moderator values with the parameter estimates to obtain a predicted eﬀect size.

For example, for minutes = 30, trained = 1, and meanage = 40, we use:

> t(mvals) %*% res$b

[,1]

[1,] 0.5157254

MiMa Function Tutorial (version 1.4) 8

The “1” in front of the “30” on the line starting with mvals is needed, because of the intercept

parameter in the model. The variance/standard error of this estimate can also be obtained. To

obtain the variance, we type:

[,1]

[1,] 0.0103143

and for the standard error (the square root of the variance), we use:

[,1]

[1,] 0.1015593

An approximate 95% conﬁdence interval is then given by adding and subtracting 1.96 times the

standard error from the predicted eﬀect size:

6 Categorical Moderators

Categorical moderators can be included in the model, in the same way that linear models can

include appropriately coded categorical independent variables (in the example above, the vari-

able specifying whether a layperson or a trained therapist provided the treatment was dummy

coded). See, for example, Neter, Kutner, Nachtsheim, and Wasserman (1996) for more details

on how to code categorical independent variables in the context of linear models.

One can also ﬁt the random-eﬀects model with the mima function. The random-eﬀects model

is simply a special case of the mixed-eﬀects model, which does not include any moderator

variables. For this, create an empty moderator matrix and then run the mima function as

follows:

> mima(yi, vi, mods)

MiMa Function Tutorial (version 1.4) 9

Parameter Estimates:

[,1]

intrcpt 0.3772

intrcpt

intrcpt 0.0183

QME = NA df = 0 p-value = NA

intrcpt 0.3772 0.1354 2.7855 0.0053 0.1118 0.6425

Note that the estimate of residual heterogeneity now indicates the amount of (total) het-

erogeneity in the eﬀect size estimates over and beyond that which we would expect based on

sampling variability alone. The estimate is quite large (τ̂ 2 = 0.1648). The test for residual

heterogeneity is now equivalent for the usual Q-test for heterogeneity and also reveals that

the eﬀect sizes are heterogeneous (QE = 38.05, df = 15, p = .0009). The QME statistic is

not needed here (since there are no moderators in the model), but we can test whether the

average eﬀect size (i.e., µ) is signiﬁcantly diﬀerent from zero, which turns out to be the case

(z = 2.79, p = .0053). The bounds for a 95% conﬁdence interval for µ are (0.11, 0.64). However,

given that at least one moderator appears to be inﬂuencing the eﬀect sizes, the results from

the random-eﬀects model are incomplete at best. Again, a conﬁdence interval for τ 2 can also

be obtained with the tau2cime.r function available on the author’s website:

95% CI: (0.035, 0.524)

which shows that the bounds for a 95% conﬁdence interval for τ 2 are given by (0.035, 0.524).

Except when using the Sidik and Jonkman estimator (SH), it is possible that the heterogeneity

estimate turns out to be negative when using any of the estimation methods. However, negative

estimates of τ 2 are outside of the parameter space. For the method of moments estimators,

negative estimates are therefore simply truncated to zero. For ML, REML, and EB estimation,

the Fisher scoring algorithm used in the mima function was modiﬁed using step halving to

guarantee a non-negative estimate (see Harville, 1977, and Jennrich & Sampson, 1976, for more

details on ML and REML estimation and penalty techniques).

MiMa Function Tutorial (version 1.4) 10

Two problems can occur when ﬁtting the mixed-eﬀects model with the mima function:

1. First of all, there is a chance that the Fisher scoring algorithm used for ML, REML, and

EB estimation does not converge. One can try increasing “maxiter” (the default is 100

iterations), but there are cases where the algorithm will cycle between several estimates

and never converge, no matter how many iterations are run (by setting verbose = "yes",

one can see how the iterative algorithm is progressing). In this case, one can try a diﬀerent

estimation method thatn the default, which is REML estimation.

2. Another problem that can occur is the following error message when running the function:

“Problem in solve.qr(a): apparently singular matrix”. This problem can occur for at least

three reasons.

(a) When there are linear relationships between the moderator variables (i.e., the values

of one moderator variable can be given as a linear function of the other moderator

variables), then this error will occur. In this case, one must remove moderators

variables from the model until such linear relationships are no longer present.

(b) When some of the moderators are highly correlated, then this error can also occur

(a correlation of exactly ±1 indicates a linear relationship between two moderators

and we are back to the ﬁrst case). Highly correlated moderators could be argued to

be redundant anyway and therefore one should probably remove one of them.

(c) The error can also occur even when the moderators are not highly correlated, but

the values of the moderators are scaled very diﬀerently (e.g., the values of one mod-

erator are between 0 and 1, while the values of another moderator are between 1

and 10,000). Such diﬀerential scaling can cause numerical problems with the matrix

inversions used as part of the algorithm (right now, “brute force inversion” is used

in the iterative procedures – in the future, this may be replaced with more elegant

methods). The solution here is to make sure that the scaling is not too dissimilar (it

does not have to be exactly the same). One could try dividing the values of a moder-

ator with very large values by 10 or 100 and re-running the model. For example, to

re-scale the meanage moderator, use:

> meanage <- meanage/10

> mods <- cbind(minutes, trained, meanage)

> mima(yi, vi, mods)

The re-scaling will change the parameter estimate and variance for this moderator,

but the z-value and p-value should remain unchanged. The other parameter esti-

mates should not be aﬀected by the re-scaling.

One can also force the mima function to ﬁt a ﬁxed-eﬀects model (with or without moderators).

To do this, set fe="yes". The estimate of (residual) heterogeneity will automatically be set

MiMa Function Tutorial (version 1.4) 11

to zero. This is not recommended, because the resulting moderator tests can have severely

inﬂated Type I error rates (i.e., moderators that are not actually signiﬁcant may appear to be

signiﬁcant). This option is available only for demonstration purposes and for conducting a

sensitivity analysis, but should not be used to actually test the moderators.

The function and this manual/tutorial should be cited (APA style) as follows:

ﬁxed-eﬀects models [Computer software and manual]. Retrieved from http://www.wvbauer.

com/.

12 References

Berkey, C. S., Hoaglin, D. C., Mosteller, F., & Colditz, G. A. (1995). A random-eﬀects regression

model for meta-analysis. Statistics in Medicine, 14, 395-411.

DerSimonian, R., & Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials,

7, 177–188.

Harville, D. A. (1977). Maximum likelihood approaches to variance component estimation

and to related problems. Journal of the American Statistical Association, 72, 320–338.

Hedges, L. V. (1983). A random eﬀects model for eﬀect sizes. Psychological Bulletin, 93,

388–395.

Hedges, L. V. (1989). An unbiased correction for sampling error in validity generalization

studies. Journal of Applied Psychology, 74, 469–477.

Jennrich, R. I., & Sampson, P. F. (1976). Newton-Raphson and related algorithms for maxi-

mum likelihood variance component estimation. Technometrics, 18, 11–17.

Morris, C. N. (1983). Parametric Empirical Bayes inference: Theory and applications (with

discussion). Journal of the American Statistical Association, 78, 47-65.

Moyer, C. A., Rounds, J., & Hannum, J. W. (2004). A meta-analysis of massage therapy

research. Psychological Bulletin, 130, 3–18.

Neter, J., Kutner, M. H., Nachtsheim, C. J., & Wasserman, W. (1996). Applied linear statistical

models (4th. ed.). Irwin: Chicago.

Raudenbush, S. W. (1994). Random eﬀects models. In H. M. Cooper & L. V. Hedges (Eds.),

The handbook of research synthesis (pp. 301–321). New York: Russell Sage.

Sidik, K., & Jonkman, J. N. (2005a). A note on variance estimation in random eﬀects meta-

regression. Journal of Biopharmaceutical Statistics, 15, 823-838.

Sidik, K., & Jonkman, J. N. (2005b). Simple heterogeneity variance estimation for meta-

analysis. Journal of the Royal Statistical Society, Series C, 54, 367-384.

Viechtbauer, W. (2005). Bias and eﬃciency of meta-analytic variance estimators in the

random-eﬀects model. Journal of Educational and Behavioral Statistics, 30, 261-293

- 1234Uploaded byakhil_rao_13
- Cupcake HeightUploaded byJulia Turpo Suarez
- Chapter 04 AnswersUploaded bysadfj545
- Engg Mathematics - 4 July 2011Uploaded byPrasad C M
- fitpUploaded byanhthigl25
- iyama1Uploaded bydusan6666
- yepp ahUploaded byTomasLieffering
- M4 All SlidesUploaded byJosua Pardede
- IJIEC_2013_32Uploaded byaghosh704
- Lab10 Augmented LatticeUploaded byItalo Granato
- Ken Black QA 5th chapter14 SolutionUploaded byRushabh Vora
- 13047_2014_Article_55.pdfUploaded byvina
- Stata CommandsUploaded byKwao Lazarus
- Study SessionUploaded byChiragDahiya
- Tests of Normality MB LENIUploaded byDefri Syahputra SKep Ns
- Professional Realty WordUploaded byKamal Ahmmad Munna
- EffectSize - CBU Statistics WikiUploaded byÁlvaro Arancibia Muñoz
- AccountsUploaded byZohab Baig
- Introduction to ANOVAUploaded byFernando Josafath A
- STAT6140_Assignment4_2014Uploaded byPaul Gokool
- project2-jdongUploaded byapi-280866825
- Report5.rtfUploaded byel.onar09
- Out Put SpssUploaded byBobby Indra Laksana
- Ch.18_Regression ExampleUploaded byEmma Lu
- Time Series Assignment for UploadUploaded byash curt
- Chapter 7Uploaded byPavithran
- 07a2 AnsUploaded byPETER
- price fairnessUploaded byShaifaliChauhan
- MS-8dec-10Uploaded bydebaditya_hit326634
- atjak2015Uploaded byWissam Khair

- Chapter6.pptxUploaded byRobert
- skittles project finalUploaded byapi-271348941
- MGB Solution Chapter 1 Q4Uploaded bySeow Fan Chong
- Lecture markov chain monte carloUploaded byPolito Pogba
- AnovaUploaded byHongYu Hui
- Fundamentals of Quality Control and ImprovementUploaded byEngr Zubair Ahmed
- 21 2 SamplingUploaded byasd
- Econometrics BruceUploaded byWendel Mirbel
- RM Session 5.pptUploaded byvaistorm
- BA 2606 Summer 2014 Chapter 11 Introduction to Hypothesis TestingUploaded byDennyseOrlido
- Module-6-for-students-1-sampling.pdfUploaded bySeeb Sabaki
- Intro to Probability and Statistics.pdfUploaded byDragos-Ronald Rugescu
- r07220101 Probability and StatisticsUploaded byandhracolleges
- MIT18 650F16 Bayesian StatisticsUploaded byEdd
- lasell sph245 homework 1Uploaded byapi-296534570
- Important Information About Relative Risk and Odd RatioUploaded byLibyaFlower
- Ge mathUploaded byKristel Unay
- Lecture 13 - Random VariablesUploaded bySaji Jimeno
- LR_Updated.pdfUploaded byNipun Goyal
- Descriptive StatistcsUploaded byRenuga Subramaniam
- Quantitative Methods 1Uploaded byjohn goodpasture
- Chi Square Calculation MethodUploaded byananthakumar
- 31-Article Text-43-1-10-20190302Uploaded byAlam Langit Brewok
- SPSSUploaded byAmrita Dhasmana
- Stock_Watson_3U_ExerciseSolutions_Chapter03_Students.pdfUploaded byHenry Feng
- Multiple range MCQs on Introductory statisticsUploaded byKenny
- BS-5Uploaded byimtiazbulbul
- T TestUploaded byMc Ortega
- Homework-pvalUploaded bydavparan
- Chapter 3 Review QuestionsUploaded bydhfbbbbbbbbbbbbbbbbbh