J. R. Statist. Soc. A (2014) 177, Part 1, pp.

237257

Multilevel factor analytic models for assessing the relationship between nurse-reported adverse events and patient safety
Luwis Diya,
KU Leuven and Universiteit Hasselt, Diepenbeek, Belgium

Baoyue Li,
Erasmus Medical Center, Rotterdam, The Netherlands

Koen Van den Heede and Walter Sermeus

KU Leuven, Belgium

and Emmanuel Lesaffre

KU Leuven and Universiteit Hasselt, Diepenbeek, Belgium, and Erasmus Medical Center, Rotterdam, The Netherlands
[Received June 2011. Final revision December 2012] Summary. We explore health outcomes and nurse stafng data that are multivariate multilevel structured. These data can be used to relate latent constructs such as patient safety to hospital, nursing unit, nurse and patient characteristics by using factor analytic models. It is important that the multilevel nature of the data is taken into account; otherwise it can lead to invalid inferences. We explore the relationship between patient safety and nurse-reported adverse events from the Belgian chapter of the Europe Nurse Forecasting Survey. The data were split into a learning and a validation data set. Since no a priori factor structure has been proposed in the literature, we establish the factor structure by using a frequentist exploratory factor analysis on the learning data set and validate the factor structure proposed by using a Bayesian conrmatory factor analysis on the validation data set. Multivariate analysis-of-variance discrepancy measures were used to assess the need for multilevel factor analysis. We establish that there was substantial between-nursing-unit, but not between-hospital, variability to warrant the use of multilevel factor analyses. The nal model was a two-level (nurse level and nursing unit level) factor analytic model with two factors at both levels. The Bayesian approach offers more exibility in tting the multilevel conrmatory factor analysis. To avoid double usage of the data the validation and learning data sets were used to t and assess the goodness of t of the multilevel conrmatory factor analysis respectively. Keywords: Bayesian inference; Data augmentation; Factor analysis; Markov chain Monte Carlo methods; Multilevel structure; Multivariate analysis

1.

Introduction

Health outcomes and nurse stafng data are often collected to quantify latent constructs such as patient safety, quality of care and nurses working environment. Usually univariate responses
Address for correspondence: Luwis Diya, Interuniversity Institute for Biostatistics and Statistical Bioinformatics, KU Leuven, Blok D, Bus 7001, Kapucijnenvoer 35, B3000 Leuven, Belgium. E-mail: luwis.diya@yahoo.co.uk
2013 Royal Statistical Society

09641998/14/177237

238

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

are used as proxies of these constructs; for instance a single adverse event, say mortality, can be used to represent patient safety. However, univariate analyses fall short in addressing the multi-dimensional facets of most healthcare measures (Diya et al., 2011); hence multivariate responses can be used. The relationship between a multivariate response vector and covariates of interest can then be established by using a multivariate analysis. A special kind of multivariate analysis which is gaining ground in the eld of health outcomes and nursing research is factor analysis (FA) (Lake, 2002; Bruyneel et al., 2009; Gajewski et al., 2010). FA is used to quantify the relationship between a multivariate response vector and a latent construct (or its domains), i.e. in an FA model the multivariate response vector is regressed on latent variables and covariates. Examples of the use of FA can be found in nursing research (Lake, 2002), bioinformatics (Hochreiter et al., 2006), psychology (Norris and Lecavalier, 2010) and educational research (Naicker, 2010). In this paper the healthcare measure of interest is patient safety. Patient safety is dened as
the absence of the potential for, or occurrence of, health care associated injuries to patients that occurs by avoiding medical errors as well as by taking actions to prevent errors from causing injury

(Agency for Healthcare Research and Quality, 2004). Since patient safety is a latent construct, nurse-reported adverse events obtained from the Belgian chapter of the Europe Nurses Forecasting Survey will be used as its proxy. Adverse events are injuries caused by medical management rather than by the underlying disease conditions (Van den Heede, 2008). In the survey several adverse events are recorded from nurses (nested within nursing units within hospitals), i.e. not only do we have multivariate responses but also the data are multilevel in structure. Neglecting the multilevel structure of the data in the analysis can be problematic since the fundamental independence assumption underlying many commonly used statistical techniques, including FA techniques, is violated (Goldstein and McDonald, 1988). When the degree of dependence is substantial, violation of the independence assumption biases parameter estimates (and standard errors) and affects the power of statistical tests (Bliese and Hanges, 2004; Kenny and Judd, 1986). Using wrongly single-level FA may lead to the atomistic fallacy, which is incorrectly assuming that the relationship between variables observed at the observational level also holds at the cluster level. Aggregating the data at the cluster level does lead to independent observations but at the risk of committing the ecological fallacy, which is incorrectly assuming that the relationship between variables at the cluster level also holds at the observational level (Robinson, 1950). So, to minimize the chances of these fallacies and to attain valid inferences for multilevel data, models which take the multilevel structure of the data into account should be considered (Longford and Muth en, 1992; Ansari and Jedidi, 2000; Goldstein and Browne, 2005; Grilli and Rampichini, 2007). Another consequence of neglecting the multilevel structure of the data in FA is that an invalid factor structure can be proposed and the interpretation of the inferences in FA is level dependent. It might be useful to check that there is a multilevel structure in the data before embarking on the more complex multilevel FA modelling. Univariate techniques such as the intraclass correlation coefcient ICC or the F -discrepancy measure that was proposed by Yan and Sedransky (2007), in a Bayesian context, can be used to verify the multilevel structure quickly. ICC is the ratio of the variation between the clusters (level 2 units) and the total variation of the observed or latent (for categorical observed responses) response. ICC-values range from 0 to 1 with larger values indicating greater variability between clusters compared with withincluster variability. As a rule of thumb, when a response has ICC > 0:10 then we can argue that there is enough level 2 variability to warrant a multilevel analysis (Dyer et al., 2005; Gajewski et al., 2010). The F -discrepancy measure that was proposed by Yan and Sedransky (2007)

Multilevel Factor Analytic Models

239

uses analysis-of-variance (ANOVA) concepts in a Bayesian context to diagnose the need to account for a hierarchical structure. This tool may be more interesting in multilevel data sets like the Europe Nurse Forecasting Survey data with more than two levels of analysis. However, in the nurse survey data interest is on a multivariate response; hence the F -discrepancy measure neglects the dependence between responses. We propose here to extend this measure to the multivariate setting by making use of multivariate analysis-of-variance (MANOVA) concepts. The objective of this paper is to t a multilevel FA model to establish the relationship between patient safety and nurse-reported adverse events from the Belgian nurse survey data, and to develop Bayesian diagnostic techniques for detecting hierarchical structures in multivariate data sets. The remainder of this paper is organized as follows. Section 2 describes the Europe Nurse Forecasting Survey data set, with particular emphasis on the Belgian chapter. A brief review of FA is presented in Section 3 and in Section 4 model checking is presented. In Section 5 the application of the Bayesian diagnostic techniques on the Belgian nurse survey data is presented. The paper ends with a discussion in Section 6. Details pertaining to a simulation study to compare the MANOVA and the ANOVA discrepancy measures, and details of the modelling tting for FA models (MPLUS and JAGS codes) are provided in the on-line supplemental document and available from http://www.blackwellpublishing.com/rss 2. The Europe Nurse Forecasting Survey data set

The Europe Nurse Forecasting project aims at introducing innovative forecasting methods by addressing not only volumes but also quality of staff and its effect on patient care. The project involves researchers from 12 European countries, the USA, China, South Africa and Botswana. Data collection was only done in 12 European countries, namely Belgium, Finland, Germany, Greece, Ireland, Poland, Spain, Sweden, Switzerland, the Netherlands, the UK and Norway. Most of the data were collected in 2009 and 2010. In the project two multicountry surveys were conducted, namely the nurse survey and patient satisfaction survey. Hospital discharge data were also collected. The various data sets were targeted at investigating how nursing qualications, demographics, workload, wellbeing and practice environment can affect productivity, patient safety and patient outcomes. This research focuses on the Belgian chapter of the nurse survey data. It contains information about the nature of a nurses job (e.g. nursephysician relationships), quality and safety issues, the nurses most recent shift and demographics. The nurse survey seeks to establish the perception of nurses with regard to their work conditions and their relationship with either patients or other healthcare professionals. In particular the focus is on relating nurse-reported adverse events to the characteristics of the nurse, nursing unit and hospital. The main question that we looked at here was How often would you say each of the following incidents occurs involving you or your patients?. For this question six incidents were considered: (a) (b) (c) (d) (e) (f) patients received the wrong medication, time or dose, pressure ulcers after admission, patient falls with injury, urinary tract infections, bloodstream infections and pneumonia.

240

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

The main question (for each incident type) had seven potential answers measured on a Likert scale, i.e. 0, never, 1, a few times a year or less, 2, once a month or less, 3, a few times a month, 4, once a week, 5, a few times a week, and 6, every day. In this paper new responses were generated by discretizing the Likert-scaled variables because of software considerations. (It is currently not feasible to implement the proposed latent variable or data augmentation approach for categorical responses with more than two categories in JAGS.) A nurse was judged to have indicated that his or her nursing unit had a substantial number of events if he or she selected the values greater than 2. This led to binary responses, namely (a) (b) (c) (d) (e) (f) wrong medication, time or dose (wrong), pressure ulcers (pressure), patient falls with injury (falls), urinary tract infections (uti), bloodstream infections (infections) and pneumonia.

The data that are considered here are multilevel in structure, i.e. nurses are nested within nursing units which are in turn nested within hospitals. The data consisted of 3151 nurses from 269 nursing units in 66 hospitals. 23 and 43 hospitals respectively were and were not technologically advanced and four and 62 hospitals respectively were and were not university hospitals. Of the 269 nursing units, 112 were surgical units, 125 were medical units and 32 were mixed surgical and medical units. Of the nurses who responded to the nurse survey, 1333 worked in surgical units, 1471 worked in the medical units and 347 worked in mixed surgicalmedical units. 2167 of the nurses reported Flemish as their language of communication whereas 984 reported French. The covariates of interest in this paper were measured at the nurse level, nursing unit level and hospital level. At the hospital level the covariates were the number of beds (beds), whether the hospital is technologically advanced (tech = 1; otherwise tech = 0) and whether the hospital was a university hospital (university = 1; otherwise university = 0). At the nursing unit level the covariate of interest was the unit specication (1, surgical, 2, medical, and 3, mixed). At the nurse level the covariates considered were gender (male = 1 if gender is male; 0 otherwise), age, time in years as a registered nurse (expn), time in years at the current hospital (exph), language (language = 1 if Flemish; 0 otherwise) and whether a nurse was satised with his or her career choice (satised = 1 if a nurse was satised; 0 otherwise). In this paper we shall focus on the nurse and the nursing unit level. The hospital level (third level) is dropped because empirical evidence did not support its inclusion, i.e. there was low variability at the hospital level; see Section 5.2.2. Thus the role of hospitals is explored in a non-hierarchical way.

3.

Factor analysis models

3.1. Single-level factor analysis We rst consider a single-level FA model for continuous responses. Let Yj = .Yj 1 , . . . , Yjp / be a p-dimensional continuous response vector for the j th individual (j = 1, . . . , N), Fj = .Fj 1 , . . . , Fjk/ be a k-dimensional (k p) vector of latent constructs and be a p k matrix of factor loadings; then the single-level FA model can be written as Yj = j + Fj + j , .1/

Multilevel Factor Analytic Models

241

where j = .j 1 , . . . , jp / is the mean vector, the common factors are multivariate normally distributed with mean 0 and covariance matrix .Fj Nk .0, // and the unique factors are multivariate normally distributed with mean 0 and covariance matrix (j Np .0, /). Note that the mean of the sth (s = 1, . . . , p) response Yjs can be modelled as a function of covariates, i.e. js = Xj s . Factor loadings (elements of ) are parameters which relate the unobserved factors to the observed responses, i.e., the larger the value of a factor loading, the more the corresponding response is said to load on the corresponding factor. Factor loadings are allowed to vary across survey items but not across individuals. Assuming orthogonality of all common and unique factors, the marginal covariance matrix is given by var.Yj / = U = + , i.e. the observed covariances matrix can be decomposed into a component that is attributable to the underlying factors and measurement error. Two special cases of FA are (a) if Fj is known then the FA model can be viewed as a multivariate regression model with the factor loadings playing the role of regression coefcients and (b) if the factor loadings are known then the FA model is equivalent to a two-level randomslope model. There are two modes of FA, namely exploratory FA (EFA) and conrmatory FA (CFA). EFA is used when there is no a priori information about the factor structure, i.e. choosing the number of factors, k, and establishing the variables that are related to a given factor domain. In that case all factor loadings are free parameters. In both the frequentist and the Bayesian contexts, EFA can be done by tting several models with varying numbers of factors and using goodness-of-t indices (discrepancy measures) to select the best model (see Section 4.2). In a Bayesian context, Lopes and West (2004) advocated the use of the reversible jump Markov chain Monte Carlo (MCMC) approach. The ultimate objective of EFA is to come up with the best factor structure for a given data set. In contrast, CFA is used to validate a priori hypothesized factor structures. In the literature, the most common CFA structure is the simple factor structure, i.e. the factor loadings are allowed to load on one and only one latent construct and all the other factor loadings (cross-loadings) are set equal to 0. The above FA model caters for continuous responses assumed to be multivariate normally distributed. For categorical or binary responses, we could assume that they are discretized versions of latent Gaussian variables, i.e. consider the binary response vector given by Yj ; then >0/ where Y , s = 1, . . . , p, is a latent Gaussian response. For identiability, the Yjs = I.Yjs js variance of the error term ("js ) for the sth response, ss , is set to 1 such that all the diagonal elements of are 1. The matrix is called the polychoric or tetrachoric correlation matrix as it is the correlation matrix of the underlying latent Gaussian responses. Additional constraints are required for both categorical and continuous response FA models to be identiable. For instance, for the single-level FA in equation (1) there are p.p + 1/=2 unique elements in but as many as pk + k.k + 1/=2 + p.p + 1/=2 elements in + , implying that there are more parameters to estimate than available information. In this paper we shall constrain the measurement error for different responses to be independent (off-diagonal elements of equal to 0) and the common factors to be orthogonal. In CFA, an additional constraint that is employed is to adopt a simple factor structure, i.e. a variable loads on only one factor, whereas in an EFA this constraint is relaxed. FA models can be tted by using maximum likelihood estimation or (robust) weighted least squares estimation; see Longford (1993). From equation (1), the total marginal likelihood of the data is

242

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

L., , , |Y/ =

.2 /pN=2 |U |N=2 exp

1 N 1 .Yj / U .Yj / 2 j =1

f.Fj / dFj ,

.2 /

where = X. For binary or categorical data the (latent) tetrachoric or polychoric correlation matrix can be used. Computational strategies for obtaining maximum likelihood estimates have been discussed in Mardia et al. (1979) and Lawley and Maxwell (1971). Another estimation technique which can be used is weighted least squares. Alternatively, one could estimate the parameters of FA models by using the Bayesian approach. Estimation is done via sampling using MCMC techniques. In the Bayesian approach the likelihood is combined with prior information to obtain the posterior distribution of the parameters of interest, i.e. p., , , |Y/ L., , , |Y/ p., , , /, . 3/

where p., , , / is the prior distribution. For an FA on binary responses, we used in this paper the latent variable technique called the data augmentation approach (Albert and Chib, 1993; Azevedo et al., 2011), where a latent Gaussian response vector is assumed that is discretized to yield the binary responses. The complete marginal data likelihood function for the binary single-level FA model is given by L., , , |Y/ where
>0/ I.Y =1/ + I.Y |Yjs / = I.Yjs .Yjs js js 0/ I.Yjs =0/ N p .Yjs ; js ; 1/ .Yjs |Yjs / f.Fj / f.Y / dFj dY ,

j =1 s=1

.4 /

; ; 1/ represents the value of the normal probability density, is the conditional likelihood. .Yjs js with being the conditional with mean js and standard deviation SD = 1, at the point Yjs js mean, i.e. js = js + s Fj , where s is the sth row of . Note that the distribution of the multivariate response vector can now be presented as a product of univariate distributions because responses are conditional independent (conditional on the common factor). The EFA models in this paper used the frequentist approach and were implemented by using MPLUS (Muth en and Muth en, 2011). A frequentist approach was necessary since methods for Bayesian covariance selection are scarce and computationally intense. CFA models can be easily tted by using both the frequentist and the Bayesian approaches (using link functions) in MPLUS. However, these CFA models are restricted in term of exibility. Hence, the CFA models in this paper were tted by using the Bayesian approach, i.e. we used the data augmentation approach called the latent variable approach in JAGS (Plummer, 2003). In particular, JAGS was run from R (R Development Core Team, 2012) by using the R2jags (Su and Yajima, 2012) package. The R package CODA was used to assess MCMC convergence (Plummer et al., 2006). In this paper we restricted the convergence diagnosis to the trace plots and the Gelman, Brooks . The quick mixing of MCMC chains and values of R close to 1 (say and Rubin diagnostic R < 1:1) were considered as indicative of no convergence problems. R The merit of the Bayesian approach is that it allows for a formal inclusion of prior opinions, thus allowing sensitivity analyses on prior opinion to be carried out. In addition, Bayesian computational methods often easily allow departures from standard assumptions, such as the normality of the factor scores. Furthermore, in a Bayesian context, it is relatively easy to generalize FA models by allowing for non-linear factor effects and multiplicative factor schemes and

Multilevel Factor Analytic Models

243

in a CFA the cross-loadings can be given priors concentrated around 0 instead of xing them to 0. The data augmentation approach has interesting features because it replaces the binary response with a latent continuous response; therefore statistical techniques which are used for continuous responses can be applied to the latent continuous response. In addition, it allows quite general link functions by replacing the classical link functions (e.g. normal distributions (probit link)) by quite general forms of distributions (e.g. skewed normal distributions). 3.2. Multilevel factor analysis When the data have a multilevel structure, multilevel FA approaches are needed. This implies that the multivariate continuous response vector for an individual has attributes emanating from the observational and the cluster levels. Let Yij = .Yij 1 , . . . , Yijp / be a p-dimensional continuous response vector for the j th individual (j = 1, . . . , ni ) in cluster i (i = 1, . . . , m), .1/ .1/ .1/ .2 / = Fij = .Fij 1 , . . . , Fijk1 / be a k1 -dimensional (k1 p) vector of latent constructs at level 1, Fi .2 / .2 / .Fi1 , . . . , F ik2 / be a k2 -dimensional (k2 p) vector of latent constructs at level 2, .1/ be a p k1 matrix of factor loadings at level 1 and .2/ be a p k2 matrix of factor loadings at level 2; then the multilevel FA model is given by Yij = ij + .1/ Fij + ij + .2/ Fi + i ,
. 1/ .1/ .2 / .2 /

.5/

where ij is the mean vector. The vector of unique factors or measurement errors at the .1/ observational level is multivariate normally distributed with mean 0 and variance .1/ (ij . 1 / Np .0, /), the vector of unique factors or measurement errors at the cluster level is multi.2 / variate normally distributed with mean 0 and variance .2/ (i Np .0, .2/ /), the vector of common factors at the observational level is multivariate normally distributed with mean 0 . 1/ and variance .1/ (Fij Nk1 .0, .1/ /) and the vector of common factors at the cluster level .2 / is multivariate normally distributed with mean 0 and variance .2/ (Fi Nk1 .0, .2/ /). Assuming orthogonality of all common and unique factors, the marginal covariance matrix consists of a within and between component, var.Yij / = M = W + B . The within-covariance matrix is given by W = .1/ .1/ .1/ + .1/ and the between-covariance matrix is given by B = .2/ .2/ .2/ + .2/ . Note that both the within and the between covariances have a component coming from the common factors (factor scores) and unique factors (measurement errors or random effects). Multilevel FA can be estimated by using the frequentist approach but this requires multi-dimensional integration to obtain a marginal likelihood which will then be optimized to obtain parameters of interest. The Bayesian approach will bypass the multidimensional integration by using sampling-based algorithms (Ansari and Jedidi, 2000; Goldstein and Browne, 2005). 4. Model checking

After the model has been tted it is necessary to monitor the quality of the statistical model. This implies that we are interested in detecting whether there are any systematic differences between the model and the observed data. In a frequentist context this entails checking whether certain functions of the data, test statistics, follow the distribution that is implied by the model. In a Bayesian context, posterior predictive checks (PPCs) are used. PPCs are obtained by generating replicated data sets from the posterior predictive distribution of the tted model (Gelman et al., 2004). These data sets are then compared with the observed data set with respect to the feature of interest by using discrepancy measures. We rst discuss how to check for the multilevel structure

244

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

in Section 4.1, followed by model assessment (e.g. normality of both the common and the unique factors, and the orthogonality of the common and unique factors) in Section 4.2. 4.1. Detecting the multilevel structure The ICC-values can be used to assess informally whether there is substantial dependence to warrant the use of multilevel models. In contrast, Yan and Sedransky (2007) proposed a Bayesian approach for checking a multilevel structure in the data by making use of an F -statistic inspired by classical ANOVA concepts. When there are two or more responses the approach of Yan and Sedransky (2007) neglects to take the dependence between responses into account, thus leading to a reduction in power to detect hierarchical structures. We propose the use of diagnostic checks based on MANOVA. To test for the independence assumption (no hierarchical structure) the relative sizes of the between (or total) and within sums of squares and cross-products are considered. In a frequentist context, the following test statistics are considered: 1 (a) the LawleyHotelling trace, LH = tr.BW /; = tr.BT1 /; (b) Pillais trace, PT (c) Wilks , = |W|=|T|.
WL

Here the total, between and within sums of squares and cross-products are given by T = ni ni m m m i=1 j =1 .Yij Y/.Yij Y/ , B = i=1 ni .Yi Y/.Yi Y/ and W = i=1 j =1 .Yij Yi /.Yij Yi / respectively. Assuming multivariate normality of the responses, all test statistics are asymptotically equivalent. When comparing two clusters all the test statistics reduce to a statistic which follows a Hotelling T 2 sampling distribution. Wilks WL is the most frequently used MANOVA test statistic as it is related to the likelihood ratio criterion and its exact sampling distribution can be derived for some special cases; for example see chapter 6 of Johnson and Wichern (2002). In this paper we propose to use these test statistics as discrepancy measures in a Bayesian context. The posterior predictive P - (PPP-) values for the three multivariate discrepancies are computed by taking the proportion of times that the predicted discrepancy exceeds the observed discrepancy. If only the data structure is misspecied and not other model assumptions then our approach does indeed diagnose the hierarchical structure. In cases where other model assumptions do not hold then the discrepancy measures proposed might indicate a statistically important difference even when the between-cluster variability is low. In such cases our approach acts as a global goodness-of-t check. Caution must be exercised when using PPCs or PPP-values since the data are used twice, i.e. in tting the model and in model checking, hence leading to some degree of bias (Bayarri and Castellanos, 2004). This can be avoided by using a learning and a validation data set, where the learning data set is used to estimate the model and a validation data set is used to assess the model t. We employed this approach in this paper. The approach proposed can be applied only to continuous multivariate responses. It cannot be directly applied to categorical multivariate responses, i.e. when using link functions, but it can be applied to the latent continuous response vector when the data augmentation approach is used. 4.2. Model assessment After having checked the multilevel structure in the data, we can either select the single-level

Multilevel Factor Analytic Models

245

or multilevel FA depending on the evidence that is presented. The next step entails model assessment, for instance checking the normality assumption of both the common and the unique factors, and validating the orthogonality assumption. In this paper we shall adhere to global model checks using PPCs in a Bayesian context and variants of the 2 likelihood ratio statistic in the frequentist context. Recall that the general goal of an FA model is to test the hypothesis that the observed covariance matrix is equal to the model-based covariance matrix, i.e. S = ./, . 6/ where S represents the covariance matrix of the observed response vector (a latent continuous response vector for categorical data) and ./ represents the model-based covariance matrix. In a frequentist context, a multitude of goodness-of-t indices have been proposed. In this paper we shall look at three indices which are functions of the 2 likelihood ratio statistic, i.e. the 2 likelihood ratio statistic, the comparative t index CFI (Bentler, 1990) and the Tucker Lewis index TLI (Tucker and Lewis, 1973) and two indices which are based on how well a given model approximates the true model, i.e. the root mean square of approximation RMSEA (Browne and Cudeck, 1992; Steiger and Lind, 1980) and the standardized root-mean-square residual SRMSR. These indices check whether the model-based covariance matrix is close to the covariance matrix of the observed response vector. If the model ts the data well then (a) the 2 likelihood ratio statistic should be small, (b) both CFI and TLI should have a value 1 and (c) RMSEA and SRMSR should be close to 0. As a rule of thumb, a model with CFI- and TLI-values of at least 0.90 are deemed acceptable and values of at least 0.95 are indicative of a good t. Alternatively, we could use RMSEA and SRMSR for which values less than 0.08 indicate an acceptable model t and values less than 0.05 are indicative of a good t (Hu and Bentler, 1999). Multiple t indices are used because each t index has its own limitations and there is no agreed-on method for evaluating whether the lack of t of a model is substantively important. These indices are used for both single and multilevel FA models. In a Bayesian context, model assessment can focus on the likelihood and the priors. PPCs are used to check how well the model ts the data; however, the discrepancy measures are now different from those used in diagnosing the neglected hierarchical structure. From these PPCs, PPP-values can be computed. In this paper the 2 discrepancy measure (Gelman et al., 1996) was used. The deviance information criterion DIC (Spiegelhalter et al., 2002) can be used to compare models on the basis of different likelihood and prior specications. The model having the smallest DIC-value is then a good candidate; see Spiegelhalter et al. (2002). However, this measure can be misleading as latent variables, e.g. factor scores and random effects, are treated as parameters. This has led to numerous discussions in the literature with no conclusive suggestions (Celeux et al., 2006); hence we shall use the mean-square prediction error MSPE instead. MSPE measures how well the model ts the data and is given by MSPE.Y/ = E
m n1 i=1 j =1

ij /2 , .Yij Y

ij is the posterior predicted response for nurse j in cluster i. Note that all model assesswhere Y ments are carried out on a validation data set (independent of the learning data set that is used to t the model).

246

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

5.

Application to Europe Nurse Forecasting Survey data

In this section, we want to establish the relationship between the latent construct patient safety and six nurse-reported adverse events in the Belgian chapter of the Europe Nurse Forecasting Survey. Hereby we wish to control for nurse, nursing unit and hospital characteristics, i.e. nurses are seen as informants of not only their own patient safety record but also informants of patient safety culture in their nursing unit. The response of interest is a multivariate vector of nursereported adverse events, i.e. Yij = .Yij 1 , . . . , Yij 6 / where Yij 1 denotes wrong medication, or dosage or time, Yij 2 denotes pressure ulcers, Yij 3 patient falls, Yij 4 urinary tract infections, Yij 5 bloodstream infections and Yij 6 pneumonia. Since no a-priori -specied factor structure is proposed for the adverse events in this paper and patient safety, there is a need to explore the factor structure. So, to avoid double usage of the data, the data were equally divided into the learning and validation data sets by using simple random sampling (within a cluster). The factor structure was explored by using the learning data set and the validation data set was used in conrming the factor structure. The learning data set was then reused in the model assessment for the CFA model. EFA was implemented by using a frequentist approach by making use of MPLUS since the Bayesian approach is computationally intense. The Bayesian module of MPLUS is not sufciently exible to allow for the computations of the discrepancy measures that are proposed in this paper; hence CFA models were tted in JAGS by using R2jags. 5.1. Descriptive statistics The descriptive statistics in this section pertain to the entire (learning plus validation) Belgian nurse survey data. The learning and validation data sets had similar characteristics (as they are random samples from the entire data set). Table 1 shows the percentage of nurse-reported adverse events, the number of missing responses and ICC. From Table 1, there are a few missing values for all the responses. The most common nurse-reported adverse event is urinary tract infection, followed by wrong medication, or time or dose and the least reported is patient fall accidents. The ICC-values are all greater than 0.1, indicating the possible need for a multilevel FA model. The Spearman correlation coefcient matrix of the percentages in a nursing unit of nursereported adverse events shows that there were moderate-to-high correlations between the nurse-reported adverse events. The largest correlation was between bloodstream infections and pneumonia and the smallest correlation was between wrong medication, or time or dose and pneumonia:
Table 1. Belgian nurse survey data: percentage of nurse-reported adverse events and ICC Type of adverse event Wrong medication, dose or time Pressure ulcers Falls Urinary tract infection Infection Pneumonia % adverse event 20.55 11.60 9.08 21.54 9.66 11.40 Number missing 70 98 89 110 160 151 ICC 0.11 0.13 0.19 0.21 0.18 0.27

Calculated for the non-missing responses.

Multilevel Factor Analytic Models

247

University hospitals were more likely to have frail patients compared with the non-university hospitals. Thus this might lead to a higher incidence of nurse-reported adverse events. University hospitals had higher numbers of nurse-reported adverse events than non-university hospitals except for wrong medication, or dose or time and patient fall accidents. 5.2. Factor analysis In this section we present the results from FA. All EFA models (single level and multilevel) were implemented on the learning data set by using a frequentist approach in MPLUS. The use of a frequentist approach here was necessitated by the fact that methods for Bayesian covariance selection approaches are scarce in the literature, no software is available and these methods are computationally intense. Since we are assuming latent Gaussian responses and the responses are binary a probit link was assumed and parameter estimates were obtained by using the maximum likelihood approach. The Bayesian approach was used to t all CFA models (single level and multilevel). In particular the latent variable approach was used, which enabled the easy computation of MANOVA discrepancy measures which are instrumental in detecting hierarchical structures in the data. 5.2.1. Single-level factor analysis 5.2.1.1. Single-level exploratory factor analysis. Here we explored the factor structure, i.e. the number of factors and the outcomes on which the chosen factors load, by using EFA in MPLUS. It is customary in EFA that one starts by aiming for a simple factor structure and on doing a CFA modify the structure accordingly. Fig. 1(a) shows a scree plot of the singlelevel EFA. The scree plot indicates that two factors were adequate to account for much of the variability in the six responses. The two-factor solution was further supported by the results shown in Table 2, which gives measures of t for various factor structures. The measures of t are all within acceptable limits; see Section 4.2. Models with more than three factors were not identiable as the number of parameters to estimate exceeded the available information. We also looked at the factor loadings to assess their classical statistical signicance. On the basis of these results and on Fig. 1(a) and Table 2 we concluded that a two-factor solution was a reasonably good single-level factor structure for the Belgian nurse survey data. Recall that, in Section 2, the responses were generated by discretizing the Likert-scaled variables by using 2 as a cut-off point. We carried out a sensitivity analysis by choosing other cut-off points, i.e. 1, 3 and 4, and redoing the EFA. The proposed factor structures for all these cut-off points were similar to the factor structure that is presented here. We also explored the factor

248

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

3.5 Variance (Eigenvalue) 0.0 0.5 1.0 1.5 2.0 2.5 3.0

Number of factors

(a)
3.5 Variance (Eigenvalue) 0.0 0.5 1.0 1.5 2.0 2.5 3.0

Number of factors

(b)

Fig. 1. within;

Belgian nurse survey datascree plots for (a) the single-level and (b) the multilevel EFA: , between

Multilevel Factor Analytic Models

Table 2. Belgian nurse survey data: measures of t for single-level EFA 2 p-value CFI TLI RMSEA SRMR

249

Number of factors 1 2 3

< 0:01 0.09 < 0:01

0.977 0.999 1.000

0.961 0.994 1.000

0.068 (0.054;0.083) 0.026 (0.000;0.052) 0.000 (0.000;0.000)

0.078 0.022 0.001

structure without discretizing the data, using ordinal EFA in a frequentist way. As a conclusion, the factor structure remained the same. So we maintained the cut-off point of 2 for all six responses. 5.2.1.2. Single-level conrmatory factor analysis. Taking the proposed structure, a singlelevel CFA was tted on the validation data set by using the data augmentation approach (Albert and Chib, 1993; Azevedo et al., 2011). The Bayesian approach was preferred here over the frequentist approach as it allows greater exibility. For instance the common factors can be assumed to be t distributed, cross-loadings can be given priors concentrated around 0 and the latent continuous responses can be assumed to be skewed. All CFA models, in this paper, were tted by using the Bayesian approach. The missing response values (see Table 1) were imputed from the posterior predictive distribution of the data. We assumed vague or non-informative priors for the regression coefcients Nq .0, 106 Iq /. The factor variances were xed to 1; hence all factor scores were generated from the standard normal distribution. The factor loadings were assumed to follow a truncated normal distribution with mean 0 and variance 20, i.e. the kth factor loading (k = 1, 2) on the sth response (s = 1, . . . , 6), ks N.0, 20/T.0,/, where T (0), is a truncation function which permits only positive values to be sampled, thus avoiding the label switching problem (Bartholomew et al., 2011). Three chains were considered, each with 300000 MCMC iterations (with a burn-in of 50000 iterations). Thinning of one in 10 iterations was implemented. This led to summary measures for all the based on 25 000 sampled values. The Gelman, Brooks and Rubin diagnostics R parameters were below 1.1 and the chains mixed well. The FA structure proposed resulted in a good t to the data (2 discrepancy measure PPP-value 0.497), i.e. wrong medication, or time or dose, pressure ulcers and patient fall accidents load on the rst factor whereas urinary tract infections, bloodstream infections and pneumonia load on the second factor. We went on to assess whether we could neglect the hierarchical structure. Table 3 shows the PPP-values for the MANOVA and ANOVA discrepancy measures. It was clear that, when univariate discrepancy measures were used, all the PPP-values had values above 0.1 or less than 0.9, i.e. none of these values were in the region of extreme values; however, three of the values were slightly above 0.1. Thus if we were to use univariate discrepancy measures and the 2 discrepancy measure, then a single-level CFA model might have been considered as the nal model. In contrast, all MANOVA discrepancies indicated that there was an omitted hierarchy in the data set and hence there was evidence in support of tting multilevel FA models. 5.2.2. Multilevel factor analysis Classical univariate multilevel probit regression models were used to establish the relationship between the adverse events and the covariates. This was used as a tool to select the covariates to

250

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

Table 3. Belgian nurse survey data: PPP-values for discrepancy measures to diagnose hierarchical structures Discrepancy
LH PT WL F1 F2 F3 F4 F5 F6

PPP-value 0.03 0.03 0.97 0.14 0.58 0.12 0.12 0.66 0.29

include in the mean structure of the multilevel FA model. A covariate was selected for inclusion in the FA model if it was signicant at the 5% level for at least two adverse events. From this exercise the following variables were considered: (a) (b) (c) (d) (e) (f) age, gender, satised, unit type (medical or mixed, with surgical being the default), university hospital and beds.

For computational reasons the continuous covariates were centred and standardized (SD = 1). 5.2.2.1. Multilevel exploratory factor analysis. Using MPLUS a frequentist two-level EFA was tted to the learning data set. Here, the observational level consists of nurses who are nested in the nursing unit (cluster level). Fig. 1(b) shows the scree plot for the multilevel factor analysis. The scree plot indicates that two factors are needed at each level. The within-factor structure remained the same as that suggested by the single-level EFA but, for the betweenfactor structure, the factor indicators wrong medication, or time or dose and pressure ulcers loaded on the rst between-nursing unit factor whereas the remainder of the factor indicators loaded on the second between-nursing unit factor, as shown in Fig. 2. The two-factor solution was further supported by the measures of t for the two-factor solution at both levels. The 2 goodness-of-t p-value was 0.5887, RMSEA (and the 95% condence interval) is given by 0.000 (0.000; 0.026), both CFI and TFI had a value of 1 and the within and between SRMR are 0.018 and 0.057 respectively. Hence all the goodness-of-t indices were within acceptable limits. These goodness-of-t statistics were better than for any other factor solution. 5.2.2.2. Multilevel conrmatory factor analysis. To validate the factor structure proposed (from multilevel EFA) we tted a two-level (nurse and nursing unit levels) CFA model, controlling for nurse, nursing unit and hospital characteristics, by using the data augmentation approach in JAGS. We assumed similar priors to those for the single-level FA in Section 5.2.1. With regard to the variance components for the random effects we used parameter expansion,

Multilevel Factor Analytic Models

251

Fig. 2.

Belgian nurse survey data: multilevel factor structure

i.e. we had a gamma distribution with scale and shape parameters equal to 0.5 as the prior for the precision and a normal distribution with mean 0 and variance 106 for the parameter expansion term. The variances of the random effects were obtained through dividing the squared parameter expansion term by the precision. We considered three chains, each having 300 000 MCMC iterations (with a burn-in of 50 000 iterations). We thinned the chains by selecting one in 10 iterations. Hence, values in Table 4 are based on 25 000 sampled values. (This is so for all the models in this section, including models used for sensitivity analysis.) The Gelman, Brooks and Rubins diagnostics for all parameters were less than 1.1 and the MCMC chains mixed well, hence indicating no convergence problems. Table 4 shows the posterior means (with SDs in parentheses) and the 95% credible intervals for the factor loadings and the variance components of the second-level unique factors (random effects). Older nurses reported fewer adverse events than younger nurses, except for wrong medication, or time or dose. Male nurses reported more incidences of wrong medication, or time or dose than their female counterparts. Nurses who were satised with their career choice reported fewer patient fall accidents and incidences of wrong medication, or time or dose. Flemishspeaking nurses reported more incidences of urinary tract infections, bloodstream infections and pneumonia than French-speaking nurses. There were no statistically important differences between the reported adverse events in the surgical, medical and mixed units. The factor loadings that are presented here were standardized and can be interpreted as the correlation between the factor domain and the factor indicators (responses), i.e. the factor loadings when the total variability (per response) is xed to 1. All the posterior means of

252

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

Table 4. Belgian nurse survey data: results of the multilevel CFA model Posterior mean (SD) 95% credible interval

Parameter

Within-level factor loadings .1/ 1,1 0.407 (0.053) .1/ 1,2 0.947 (0.025) .1/ 1,3 0.557 (0.054) .1/ 2,4 0.828 (0.035) .1/ 2,5 0.853 (0.037) .1/ 2,6 0.804 (0.040) Between-level factor loadings .2/ 1,1 0.182 (0.098) .2/ 1,2 0.224 (0.100) .2/ 2,3 0.405 (0.102) .2/ 2,4 0.226 (0.072) .2/ 2,5 0.323 (0.067) .2/ 2,6 0.381 (0.090) Residual variances 2 1 2 2 2 3 2 4 2 5 2 6 Fit indices 2 MSPE 0.117 (0.063) 1.117 (2.035) 0.226 (0.191) 1.652 (1.059) 0.199 (0.226) 0.919 (0.724) 0.349 (0.477) 17206.897 (262.432)

(0.300; 0.508) (0.889; 0.986) (0.449; 0.659) (0.756; 0.892) (0.777; 0.921) (0.722; 0.876) (0.014; 0.375) (0.023; 0.403) (0.183; 0.574) (0.082; 0.363) (0.190; 0.450) (0.201; 0.546) (0.015; 0.256) (0.012; 8.066) (0.008; 0.715) (0.453; 4.531) (0.008; 0.824) (0.045; 2.786) (0.000; 1.000) (17030.075; 17724.202)

.v/ 2 is k,s is the kth standardized factor loading for variable s at level v. s the variance component for variable s. s: 1 denotes the wrong medication, or time or dose; 2 denotes pressure ulcers; 3 denotes patient falls; 4 denotes urinary tract infection; 5 denotes bloodstream infection; 6 denotes pneumonia response.

the rst-level factor loadings were greater than 0.4, implying that there was moderate-to-high correlation between the factor domains and the factor indicators. The nursing unit level factor loadings were between 0.18 and 0.41. Though these factor loadings were low to moderate they were still statistically important. The 2 goodness-of-t Bayesian p-value was 0.50, suggesting that there was no major discrepancy between the data and the model. The model MSPE was 17206.897. A sensitivity analysis was done by (a) altering the common factor distribution to have a Student t-distribution with 4 degrees of freedom, (b) altering the common factor distribution to have a t-distribution with unspecied degrees of freedom and

Multilevel Factor Analytic Models

253

(c) xing the rst factor loading instead of the factor variance to 1. The three models resulting from these alterations had larger MSPE-values than the initial multilevel FA model, i.e. the MSPE-values were 17209.677, 17207.970 and 17207.180. We also assessed whether there was substantial variability at the hospital level after accounting for the nursing unit level. All the MANOVA discrepancy measures had values that were close to 0.5, hence indicating that there was not that much dependence between nurses in a hospital after accounting for the nursing unit. This was used as evidence for not extending the FA model to three levels (hospital level, nursing unit level and nurse level). Hence the effect of hospital was explored in only a non-hierarchical way. The within-factor structure was the same as that attained under the single-level FA. The nurse-reported adverse events wrong medication, or time or dose, pressure ulcers and patient fall accidents loaded on one factor whereas the remaining nurse-reported adverse events loaded on another factor. This implied that nurses who reported that they encountered many incidences of wrong medication, or time or dose were also more likely to report high numbers of pressure ulcers and patient fall accidents (adhering to protocol and safety guidelines) whereas nurses who reported that they encountered many incidences of urinary tract infections were also more likely to have reported many incidences of bloodstream infections and pneumonia (hygiene or hand hygiene). These three adverse events can also be considered as indicators of adherence to safety guidelines, for instance, safe handling of perfusion lines (which if not done properly can be a source of infection). However, these three adverse events are not as directly related to the nursing care process as the adverse events which load on the rst factor. We should note that the nurses here were informants of not only their own experience but also that of the nursing unit at large. The between-factor structure was reective of what happens at the nursing unit level. Wrong medication, or time or dose and pressure ulcers loaded on the rst nursing unit factor, and the remainder of the adverse events loaded on the second nursing unit factor. Clearly the rst factor is an indicator of whether the nursing unit administers medication correctly, in time and is attentive to the physical need of immobile patients. The second factor is a mixture of, for example, whether the right equipment is accorded to the patient and the general hygiene condition of the nursing unit.

6.

Discussion

In this paper we have illustrated how to address complex research questions by using FA. FA models are used to relate the latent constructs to the responses of interest, e.g. adverse events. These models are becoming popular in many elds, especially in health outcomes and nursing research where they are used to inform policy decisions. The data that are collected in health outcomes and nursing research are often huge and for this reason, coupled with limitations in available software, health outcome and nurse researchers tend to t single-level FA models based on simplistic assumptions, e.g. the independence assumption. However, most data in this area of research are multilevel in structure; hence neglecting the data structure in the statistical analysis can lead to invalid inferences which might lead to wrong policy recommendations. Before embarking on complex multilevel FA modelling which is computationally intense there should be an assessment of whether there is substantial dependence between subjects in the same cluster. If the dependence between subjects in the same cluster is low the inference that is obtained from the single-level FA is more likely to be valid, thus leading to simple conclusions, i.e. we aim for parsimony, not only in the covariates that are included in these models but also in the level of analysis. However, if there is substantial between-cluster variability, inferences from

254

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

the single-level FA will be invalid because the level of analysis has a bearing on the interpretation of the factor structure and the inferences coming from such a structure. We proposed the use of the MANOVA discrepancy measures to address formally whether there is substantial dependence (between-cluster variability) in our multivariate multilevel structured data set. These measures are handy as they have more power than the ANOVA discrepancy measures and intraclass correlation coefcients to diagnose the hierarchical structures. It is crucial that the factor structure is correctly specied when diagnosing the hierarchical structure as misspecication of the factor structure implies that the discrepancy measure proposed will then become sensitive not only to the hierarchical structure but also to the misspecied factor structure. Though it can be argued that all potential models, i.e. single-level and multilevel models, can be tted and then compared, for large data sets that are encountered in health outcomes, nursing and epidemiological research (e.g. healthcare registers or healthcare administrative databases) this entails a huge computational burden. Instead a single-level FA model can be tted and from this model MANOVA discrepancies used to assess whether there is a need to do multilevel FA. The MANOVA discrepancy measures that are presented in this paper are not limited to FA models only but can also be used when other types of multivariate model are under consideration. The MANOVA concepts can also be used in a frequentist context where MANOVA test statistics can be used. A potential area for further research is the development of diagnostic measures to check omitted non-hierarchical multilevel structures, for instance in multiple-membership and cross-classied models. One can easily implement simple FA models, using either the frequentist approach or the Bayesian approach, in MPLUS. However, with regard to the Bayesian approach, the MPLUS Bayesian module is quite limited, i.e. MPLUS cannot t multilevel FA models with more than two levels and it is not feasible to implement the latent variable approach or to obtain discrepancy measures. In this paper the EFA employed the frequentist approach as the Bayesian approach (Lopes and West, 2004) is computationally intensive. From a practical point of view the EFA is easy to implement in a frequentist context compared with the Bayesian context. Even researchers with limited methodological comprehension can implement EFA in a frequentist context. The Bayesian approach was adopted in tting CFA models because it offers exibility. It is fairly easy to deviate from common assumptions when using the Bayesian approach compared with the frequentist approach; for example cross-loadings can be given priors concentrated around 0 instead of xing them to 0. The Bayesian approach in this paper, the latent variable approach, offers additional advantages over using link functions (e.g. logit and probit links) in that distributional assumptions can be relaxed; for example a truncated skewed normal distribution can be used and various discrepancy measures for continuous responses can be used for discrete data. The PPP-values from the 2 discrepancy measures for both the single-level and the multilevel FA suggested that the two models were good ts for the data at hand. It is clear that this discrepancy measure failed to capture the misspecication of the data structure. Relying on global goodness-of-t measures without looking at specic aspects of the model can lead to the wrong inference with regard to the compatibility of the model to the data. Thus, various measures should be considered. PPCs use the data twice, i.e. in tting the model and in checking model adequacy, hence leading to some degree of bias. To eliminate this bias cross-validation PPCs (Marshall and Spiegelhalter, 2003) can be used; however, they are computationally intense. When a model is correctly specied, the PPP-values should have a uniform distribution. This characteristic of the PPP-values has been discussed elsewhere (Hjort et al., 2006; Steinbakk and Storvik, 2009). Hjort et al. (2006) proposed recalibrating the PPCs, resulting in recalibrated PPP-values which have a uniform distribution under the correct model specication. However,

Multilevel Factor Analytic Models

255

the PPP-values in this paper are not recalibrated as computing the recalibrated PPP-values is extremely computer intensive, requiring a double-simulation approach. So the data were split into a learning data set and a validation data set, where the model tting is done by using the learning data set and model assessment by using the validation data set. Another potential approach which is less computationally demanding is to compute the calibrated PPP-values by using integrated nested Laplace approximations (Rue et al., 2009). This is an area which the authors are currently exploring. The factor structure for the single-level FA and the nurse level of the multilevel FA is in line with expectations. Indeed, medication errors, pressure ulcers and patient fall accidents form a logical group of adverse events. Belgian hospitals traditionally focus on these three adverse events when setting up actions to improve patient safety in which nurses are involved. After all, actions to prevent these adverse events are closely related to the nursing process (for example, turning patients to prevent pressure ulcers, most medication is administered by nurses and patient fall accidents can be prevented if patients are monitored and wear the correct type of shoes). Urinary tract infections, bloodstream infections and pneumonia are all infections that are potentially related to nursing care (for example, urinary tract infections can be prevented by patients not having in-dwelling catheters for longer than needed). These can also be explained by the severity of illness of the patients as well as by medical interventions (e.g. sterile insertion central venous catheters). At the nursing unit level the variable patient fall accidents is now grouped with the infections. The rst two adverse events clearly reect the nursing process at the nursing unit level. The second nursing unit factor consists of patient fall accidents and infections. This factor might be indicative of collective efforts taken at the nursing unit to prevent injury (falls and infections) to patients, i.e. making sure that the environment that a patient is exposed to is free from conditions which might increase the risk of infections or patient fall accidents (e.g. clearing spillages from the oor or dropped food, clearing medical equipment from nursing units and picking up dropped linen). However, the authors advocate further research to comprehend the grouping of factor indicators at the nursing unit level and also to explore whether this structure is unique to Belgium or whether it can be established in other European countries. As a sensitivity analysis the responses were discretized by using different cut-off points. The results that are reported here were robust to changes in the cut-off points. Acknowledgements The research leading to these results has received funding from the European Unions seventh framework programme (FP7/2007-2013) under grant agreement 223468. The rst and last authors were partially supported by the Interuniversity Attraction Poles program P6/03, Belgian State Federal Ofce for Scientic, Technical and Cultural Affairs. References
Agency for Healthcare Research and Quality (2004) AHRQs patient safety initiative: building foundations, reducing risk. Interim Report 04-Rg005. Agency for Healthcare Research and Quality, Rockville. Albert, J. H. and Chib, S. (1993) Bayesian analysis binary and polychotomous response data. J. Am. Statist. Ass., 88, 669679. Ansari, A. and Jedidi, K. (2000) Bayesian factor analysis for multilevel binary observations. Psychometrika, 65, 475496. Azevedo, C. L., Bolfarine, H. and Andrade, D. F. (2011) Bayesian inference for a skew-normal IRT model under the centred parameterization. Computnl Statist. Data Anal., 55, 353365. Bartholomew, D., Knott, M. and Moustaki, I. (2011) Latent Variable Models and Factor Analysis: a Unied Approach, 3rd edn, sect. 2.10, pp. 3037. Chichester: Wiley.

256

L. Diya, B. Li, K. Van den Heede, W. Sermeus and E. Lesaffre

Bayarri, M. J. and Castellanos, M. E. (2004) Bayesian checking of the second levels of hierarchical models. Statist. Sci., 22, 322343. Bentler, P. M. (1990) Comparative t indices in structural models. Psychol. Bull., 107, 238246. Bliese, P. D. and Hanges, P. J. (2004) Being both too liberal and too conservative: the perils of treating grouped data as though they were independent. Organiznl Res. Meth., 7, 400417. Browne, M. W. and Cudeck, R. (1992) Alternative ways of assessing model t. Sociol. Meth. Res., 21, 230258. Bruyneel, L., van Den Heede, K., Diya, L., Aiken, L. and Sermeus, W. (2009) Predictive validity of the international hospital outcomes study questionnaire: an RN4CAST pilot study. J. Nursng Schol., 41, 202210. Celeux, G., Forbes, F., Robert, C. and Titterington, D. (2006) Deviance information criteria for missing data models. Am. Statistn, 1, 651674. Diya, L., Van den Heede, K., Sermeus, W. and Lesaffre, E. (2011) The relationship between in-hospital mortality, readmission into the intensive care nursing unit and/or operating theater and nurse stafng levels. J. Adv. Nursng, 68, 10731081. Dyer, N. G., Hanges, P. J. and Hall, R. J. (2005) Applying multilevel conrmatory factor analysis techniques to the study of leadership. Lead. Q., 16, 149167. Gajewski, B. J., Boyle, D. K., Miller, P. A., Oberhelman, F. and Dunton, N. (2010) A multilevel conrmatory factor analysis of the practice environment scale: a case study. Nursng Res., 59, 147153. Gelman, A., Carlin, J. B., Stern, H. S. and Rubin, D. (2004) Bayesian Data Analysis. London: Chapman and Hall. Gelman, A., Meng, X. and Stern, H. (1996) Posterior predictive assessment of model tness via realized discrepancies. Statist. Sin., 6, 733807. Goldstein, H. and Browne, W. J. (2005) Multilevel factor analysis models for continuous and discrete data. In Contemporary Psychometrics: a Festschrift for Roderick P. McDonald (eds A. Maydeu-Olivares and J. J. McArdle), pp. 453475. London: Erlbaum. Goldstein, H. and McDonald, R. P. (1988) A general model for the analysis of multilevel data. Psychometrika, 53, 455467. Grilli, L. and Rampichini, C. (2007) Multilevel factor analysis for ordinal variables. Struct. Equn Modlng, 14, 125. Hjort, N. L., Dahl, F. A. and Steinbakk, G. H. (2006) Post-processing posterior predictive p values. J. Am. Statist. Ass., 101, 11571174. Hochreiter, S., Clevert, D.-A. and Obermayer, K. (2006) A new summarization method for affymetrix probe level data. Bioinformatics, 22, 943949. Hu, L. and Bentler, P. (1999) Cutoff criteria for t indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct. Equn Modlng, 6, 155. Johnson, R. A. and Wichern, D. W. (2002) Applied Multivariate Statistical Analysis. Upper Saddle River: Pearson Education International. Kenny, D. A. and Judd, C. M. (1986) Consequences of violating the independence assumption in analysis of variance. Psychol. Bull., 99, 422431. Lake, E. T. (2002) Development of the practice environment scale of the nursing work index. Res. Nursng Hlth, 25, 176188. Lawley, D. N. and Maxwell, A. E. (1971) Factor Analysis as a Statistical Method, 2nd edn. London: Butterworth. Longford, N. T. (1993) Random Coefcient Models. New York: Oxford University Press. Longford, N. T. and Muth en, B. O. (1992) Bayesian posterior predictive checks for complex models. Psychometrika, 57, 581597. Lopes, H. F. and West, M. (2004) Bayesian model assessment in factor analysis. Statist. Sin., 14, 4167. Mardia, K. V., Bibby, J. M. and Kent, J. T. (1979) Multivariate Analysis. New York: Academic Press. Marshall, E. C. and Spiegelhalter, D. J. (2003) Approximate cross-validatory predictive checks in disease mapping models. Statist. Modllng, 22, 16491660. Muth en, L. K. and Muth en, B. O. (2011) Mplus Users Guide, 6th edn. Los Angeles: Muth en and Muth en. Naicker, V. (2010) Educators pedagogy inuencing the effective use of computers for teaching purposes in classrooms: lessons learned from secondary schools in South Africa. Educ. Res. Rev., 5, 674689. Norris, M. and Lecavalier, L. (2010) Evaluating the use of exploratory factor analysis in developmental disability psychological research. J. Autsm Develpmntl Disordrs, 40, 820. Plummer, M. (2003) JAGS: a program for analysis of Bayesian graphical models using Gibbs sampling. In Proc. 3rd Int. Wrkshp Distributed Statistical Computing, Vienna, Mar. 20th22nd (eds K. Hornik, F. Leisch and A. Zeileis). Plummer, M., Best, N., Cowles, K. and Vines, K. (2006) Coda: convergence diagnosis and output analysis for mcmc. R News, 6, 711. R Development Core Team (2012) R: a Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing. Robinson, W. S. (1950) Ecological correlations and the behavior of individuals. Am. Sociol. Rev., 15, 351357. Rue, H., Martino, S. and Chopin, N. (2009) Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations (with discussion). J. R. Statist. Soc. B, 71, 319392. Spiegelhalter, D. J., Best, N. G., Carlin, B. P. and van der Linde, A. (2002) Bayesian measures of model complexity and t (with discussion). J. R. Statist. Soc. B, 64, 583639.

Multilevel Factor Analytic Models

257

Steiger, J. H. and Lind, J. C. (1980) Statistically based tests for the number of common factors. A. Spring Meet. Psychometric Society, Iowa City. Steinbakk, G. H. and Storvik, G. O. (2009) Posterior predictive p-values in Bayesian hierarchical models. Scand. J. Statist., 36, 320336. Su, Y.-S. and Yajima, M. (2012) R2jags: a package for running JAGS from R. R Package Version 0.03-06. Tucker, C. and Lewis, C. (1973) A reliability coefcient for maximum likelihood factor analysis. Psychometrika, 38, 110. Van den Heede, K. (2008) Nurse-stafng levels and patient safety in acute hospitals: analyzing administrative data at the nursing unit level. PhD Thesis. Centre for Health Services and Nursing Research, Department of Public Health, Katholieke Universiteit Leuven. Yan, G. and Sedransky, J. (2007) Bayesian diagnostic techniques for detecting hierarchical structure. Baysn Anal., 2, 735760.

Supporting information Additional supporting information may be found in the on-line version of this article: Supplementary Material: Multilevel factor analytic models for assessing the relationship between nurse reported adverse events and patient safety.