This action might not be possible to undo. Are you sure you want to continue?
Fall 2010 ISSN 1948-822X DOI: 10.5243/jsswr.2010.8
Author Guidelines for Reporting Scale Development and Validation Results in the Journal of the Society for Social Work and Research
Peter Cabrera-Nguyen Washington University in St. Louis
Reliable and valid measurement is critical for advancing social work research and evidence-based social work practice. However, the quality of available evidence is largely determined by the study designs used in intervention research. A well-designed intervention study includes optimal sampling techniques, data analysis using appropriate statistical methods, steps to enhance internal and external validity, and psychometrically sound assessment instruments. Ensuring intervention studies include assessment instruments with strong psychometric properties may, ultimately, enable practitioners to assess client target problems with greater precision. Further, increasing the availability of instruments with demonstrated reliability and validity may also help practitioners select evidence-based interventions that best match the needs of their clients. Therefore, it is important for social work researchers engaged in scale development and validation to conduct their research and report their findings in a reliable manner that allows other researchers to replicate those findings. The guidelines presented in this article are intended to assist JSSWR authors in this endeavor. These guidelines are limited to the latent variable approach (i.e., exploratory factor analysis [EFA] and confirmatory factor analysis [CFA]) to scale development and validation, and these suggestions do not cover item-response theory approaches. There is little consensus in the literature regarding these guidelines. Indeed, some aspects are still intensely debated. Therefore, these guidelines should be interpreted as a framework for reporting findings— from early-stage scale development to later-stage scale validation research. Peter Cabrera-Nguyen is a PhD student and Chancellor's Fellow at the George Warren Brown School of Social Work, Washington University in St. Louis. Correspondence regarding this manuscript should be sent to the author at firstname.lastname@example.org
Guidelines 1. Precisely define the target construct. 2. Justify the need for your new measure. For example, if measures of the construct exist in the literature, explain the value added by your new scale. How might the new measure enhance the substantive knowledge base or social work practice? 3. Indicate that you have submitted your initial pool of items to expert review (Worthington & Whittaker, 2006). Report (a) the number of items in the preliminary pool; (b) the number of expert reviewers and their qualifications; and (c) any major changes to your initial item pool following the review (e.g., a substantial decrease in the number of items, changes to the original item response format, overhaul of item pool due to experts’ assessment regarding content validity). 4. Report the name and version of the statistical software package used for all analyses. 5. Identify and justify the sampling strategy (e.g., convenience, snowball) and sampling frame. Report standard sample demographic characteristics as well as other salient sample characteristics (e.g., ―participants were advancedstanding MSW students at a large public Midwestern university concentrating in social service administration‖). 6. Discuss relevant data preparation and screening procedures. For instance, do the data meet the appropriate assumptions for factor analysis? If not, what actions were taken? Report tests of factorability if appropriate (e.g., report Bartlett’s test of sphericity). 7. Provide all dates of data collection. 8. Avoid use of principal components analysis (PCA) as a precursor to CFA (Costello & Osborne, 2005; Worthington & Whittaker, 2006). Instead, start with EFA to assess the underlying factor structure and refine the item pool. EFA should be
Journal of the Society for Social Work and Research
1 ▪ There are legitimate arguments for using different approaches to EFA and CFA in scale construction and validation.an active online discussion group about all things related to structural equation modeling (SEM. EFA followed by CFA is one of the most common approaches to scale development and validation (Worthinton & Whittaker. 9. Authors should use these guidelines as a roadmap.. but not necessarily both methods. Describe the matrix (or matrices) you analyzed (e.ats. Authors should provide an empirically based rationale for their choice of a particular approach. does your four-factor model of acculturation fit the data better than a two-factor model or a onefactor model)? Identify the preferable model based on appropriate fit statistics (e. 2006). In contrast. Brown (2006) recommended using ―EFA in a CFA framework‖ as an intermediate step between EFA and CFA. Worthington and Whittaker (2006) recommended starting with EFA. For a review of practices for handling missing data. This site offers a wide range of online seminars on topics such as conducting EFA/CFA using various versions of Mplus. and (d) the next appropriate step in the scale development and validation process. and Horton and Kleinman (2007). Henson & Roberts. the lines between EFA and CFA are becoming increasingly blurred.. otherwise.html UCLA Academic Technology Services – an invaluable resource for an array of methodologies including EFA/CFA/SEM. and carrying out multiple imputation of missing data using a range of software packages. note potential repercussions of suboptimal sampling techniques. 12. Do competing models fit the data better or worse than your model (e.ncsu. and then moving to CFA using a different sample. discuss implications of using listwise deletion to handle missing data instead of multiple imputation or FIML).g. Gillaspy. authors interested in cross-cultural scale development and validation are encouraged to refer to classic articles by Alegría and colleagues (2004) and Canino and Bravo (1994).g.edu/archives/semnet.1 For CFA. correlation). (Costello & Osborne.g. 2006. Guidelines for reporting EFA results are presented in Table 1. parsimony. Discuss directions for future research (e. It is difficult to develop a decision tree that adequately specifies the entire universe of choices. see Sterne and colleagues (2009). Worthington & Whittaker. (b) model fit statistics. Include your scale (items and response options) in an appendix. if appropriate.. (c) statistical tests for comparing competing models. Include matrices in the manuscript if feasible. The following resources may be useful for researchers involved in scale development and validation: ▪ SEMNET . and relevant theory. Guidelines for reporting presented in Table 2. Kline (2005) recommended using either EFA or CFA.edu/stat/seminars/default. 16. Rose & Fraser.htm StatNotes – a good.ua. ▪ Finally.htm 11. Rose and Fraser (2008). Journal of the Society for Social Work and Research 100 . Akaike information criterion for non-nested models). indicate these data are available upon request. Report the amount of missing data and describe how missing data were handled. Provide a rationale for your approach to handling missing data.ucla. CFA results are 14. covariance. Report how methodological limitations may have impacted findings regarding your measure’s psychometric properties (e. chi-square difference test for nested models. An overview of factor analysis is available atxxxxxxxxxxxxxxxxxxxxxxx http://faculty. Indeed. but they should also be familiar with ongoing developments and debates in the psychometric literature. Compare your CFA model with the alternative or competing models.. general resource for quantitative data analysis. 2009). For instance.g. Authors are encouraged to consider using multiple imputation or model estimation with full-information maximum likelihood (FIML.edu/garson/PA765/factor. 15. including CFA) for novices and experts alike. 10. 13.. 2006). 2008). authors should specify an a priori hypothesized model and a priori competing models (Jackson. Available atxxxxxxxxxxxxxxxxxxxx http://alabamamaps. & Purc-Stephenson.chass. However. testing your scale for measurement invariance by conducting CFA on different populations). Available atxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx http://www. Conclusion Scale development and validation using EFA and CFA is a complex process involving many choices regarding (a) data screening procedures.g. 2005.AUTHOR GUIDELINES: SCALE DEVELOPMENT AND VALIDATION RESULTS followed by CFA using a different sample (or samples) to evaluate the EFA-informed a priori theory about the measure’s factor-structure and psychometric properties.
This can. iterative. including all of the items in the analysis. especially if there are other items with factor loadings of . (Worthington & Whittaker. Criteria for number of factors. 2. Report the number of factors retained and justify this decision using multiple criteria (eigenvalue > 1. Report if additional data collection was necessary due to inadequate sample size. 8. Similarly. When in doubt.3). Report the variance explained by the factors. Reporting the eigenvalue > 1 rule alone is inadequate because it is among the least accurate criteria for assessing factor retention (Costello & Osborne. etc).4|. Justify the decision to use an orthogonal or oblique solution. ―it is very important that researchers [and reviewers] be able to independently evaluate the results obtained in an EFA study.50 or there are 10:1 items per factor with factor loadings of roughly |. t tests).. . it is probably best to describe what you do and defend your approach on substantive grounds. p. It is recommended that authors report this information in a chart following the example provided by Henson and Roberts (2006) on page 411. 2006). Always report the whole factor pattern/structure matrix.32 on at least two factors should generally be candidates for deletion. 2006). 6. 5. report the new participants’ sociodemographic characteristics and test for differences between groups using standard statistical procedures (e. Report any deleted items and the criteria used for deletion. 2006). noting the appropriate sample size is dependent on the features of the gathered data. occur on two levels. 2005). empirically assess factor intercorrelations. Another rule of thumb is that N = 300 is usually acceptable (Worthington & Whittaker. Journal of the Society for Social Work and Research 101 . some researchers have criticized these sample size rules of thumb. However. 7. 2006). parallel analysis. Criteria for deleting (crossloaded) items. 2005. Run a preliminary EFA to determine if further data collection is required based on the following criteria: (a) If communalities are greater than . Worthington & Whittaker. Rerun the EFA each time an item is deleted. or even employ a CFA‖ (Hensen & Roberts. EFA is a complex. and report them before deciding upon a final rotation solution (Henson & Roberts.60 or there are a minimum of 4:1 items per factor with factor loadings above |. rejection of a factor with fewer than 3 items. 2006). 2005). if possible. Therefore. Conversely. Given the lack of consensus. One recommendation is to always begin with an oblique rotation. Given the myriad subjective decisions necessary in EFA. Report the whole factor pattern/structure. 2005. 4. Run EFA .. Worthington & Whittaker.g. 2006. . Henson & Roberts.g. 2006. Some researchers argue oblique rotation is always the best approach because (a) factor intercorrelations are the norm in social sciences and (b) both approaches yield the same result if the factors happen to be uncorrelated (Costello & Osborne. In general.6|. p. and subjective process. independent researchers should be able to accurately replicate the study on new data. varimax. Every decision should be thoroughly reported and justified. or not. 2006. describe your decisions. other researchers contend that orthogonal rotation is preferable because fewer parameters are estimated—orthogonal rotation is more parsimonious and amenable to replication (Henson & Roberts. If so. 2006). Others state that doing so is ―fine‖ (Costello & Osborne. geomin). 3.50 or greater (Costello & Osborne. (b) If communalities are all at least . err on the side of over reporting. then a sample size of 150 to 200 is likely to be adequate. scree test. Crossloading items with values ≥ . then even smaller sample sizes may suffice. independent researchers should be able to evaluate the analytic choices of authors in the reported study. 400). Report the specific rotation strategy used (e. 2006). and should. These researchers recommend obtaining the largest possible sample because the adequacy of the sample size cannot be determined until after the data have been analyzed (Henson & Roberts. some researchers warn against relying on a statistical software package’s default settings to determine the appropriate type of oblique rotation (Henson & Roberts. How large a sample? One common rule of thumb is to ensure a person-to-item ratio of 10:1. Give EFA details.CABRERA-NGUYEN Table 1 Reporting Guidelines for Exploratory Factor Analysis 1. Explained variance. Second.
Respecification: Caution! Report any post-hoc respecifications to improve model fit based on modification indices. 3. 6. 2006). 2. df > 0. RMSEA. 2004. Use multiple fit indices. Other measures. Comment on the new measure’s convergent and discriminant validity based on parameter estimates. 2009).g. . RMSEA ≤ . 9.96 (for p < . Respecification to allow for correlated errors is not supportable without strong pragmatic justification (e. their p values. Justify the respecifications on theoretical or conceptual grounds (Jackson et al.. Report the absence of localized strain. free parameters.. report a priori competing models. WLSMV) and justify your choice based on distributional assumptions. Assessing the validity of the factor solution. a Authors should consult the current SEM literature to stay abreast of the ongoing debate over appropriate cut-off values. 2006. Provide theoretical and empirical justification (e. CFI/TLI. strong factor loadings that do not crossload may indicate good convergent validity. & Wen. After estimating a model. scaling of factors.95 b. authors are referred to Yu (2002).g.80 may indicate poor discriminant validity (Brown. results of preliminary EFAs) for your hypothesis. In addition. even those that are nonsignificant (Brown. SRMR ≤ . and constrained parameters. Note that respecification precludes comparing the model with your a priori specified competing models. always report multiple fit indices (e. Report hypothesized factor structure. Provide a comprehensive description of the a priori parameter specification. authors should generally use the cut-off values recommended by Hu and Bentler (1999) and endorsed by Brown (2006). 12. When in doubt. What is acceptable fit? For model fit indices.AUTHOR GUIDELINES: SCALE DEVELOPMENT AND VALIDATION RESULTS Table 2 Reporting Guidelines for Confirmatory Factor Analysis 1. authors should consult Brown’s (2006) CFA text or Kline’s (2005) SEM text for guidance. Report improvements in appropriate model fit indices for respecified models (e. not just those favorable to your hypotheses (Jackson et al. CFI/TLI ≥ . otherwise. Select an estimator based on distributional patterns and assumptions. Journal of the Society for Social Work and Research 102 ..g. always report the unstandardized estimates. 4. Include a figure of each CFA model being tested using Kline’s (2005) graphical conventions if feasible.. ML. 11. include the standardized estimates when appropriate. indicate if you freed the errors of any items to correlate.. Marsh. Include a figure.. In addition.40 are weak and factor loadings ≥ . For example.... 2010). and the standard errors. 5. model X2. Report all appropriate fit indices.05) indicate areas of strain (Harrington. 10.g.g. Be sure to report all parameter estimates. Describe the parameterization. the two-indicator rule). For instance. It is not appropriate to report that you relied on your statistical software’s default setting.g. chi-square difference test). Identification. Describe and justify the theoretical model. df.g. assuming ML estimation:a a. factor correlations ≥ .60 are strong (Garson. note localized areas of strain by reporting the relevant standardized residuals. Demonstrate model identification (e. do not report acceptable CFI and TLI scores while omitting a relevant fit index with a subobtimal value.. When reporting factor loadings and other parameter estimates. p. Necessary and sufficient conditions for model identification may vary for certain types of CFA models. Identify fixed parameters. Localized strain? When reporting model fit. 2009). Hau.g. if appropriate. One rule of thumb is that factor loadings < . Report squared multiple correlations and comment on the measure’s reliability (e. Standardized residuals greater than 1. For example. Jackson et al. 2009). Fan & Sivo.. 2007). include an assessment for localized areas of strain by examining standardized residuals. assess and report the ―t-rule‖. Parameter estimates and SEs.06 c. SRMR). items contain similar words or phrases).08 8. 7. 2009). For recommended cut-off scores across a variety of circumstances. In addition. Authors who choose to use lower cutoffs should provide a thorough rationale supported by ample citations from the recent SEM literature (e. report Raykov’s Rho if appropriate). Report the estimator used (e.
Retrieved from http://statmodel2. Multiple imputation for missing data in epidemiological and clinical research: Potential and pitfalls. J.. P. D. 171-178. & Purc-Stephenson. M. White.181 Brown.ncsu.naswpressonline. Z. doi:10. Los Angeles).. Retrieved from http://pareonline.. D. doi:10.. J. American Statistician. Research and Evaluation. A. Woo. C. 66. Scale development research: A content analysis and recommendations for best practices. Carlin. H. 339(7713).. Takeuchi. Evaluating cutoff criteria of model fit indices for latent variable models with binary and continuous outcomes (Doctoral dissertation. (2008). International Review of Psychiatry. & Roberts. Hau. doi:10.. The adaptation and testing of diagnostic and outcome measures for cross-cultural research. Educational and Psychological Measurement. Spratt. & Kleinman. (1999).. X. Confirmatory factor analysis for applied research. Confirmatory factor analysis..com/download/Yudissertation.1037/a0014694 Kline. Cultural relevance and equivalence in the NLAAS instrument: Integrating etic and emic in the development of cross-cultural measures for a psychiatric epidemiology and services study of Latinos. International Journal of Methods in Psychiatric Research. 393-416. (2002). & Bentler. Kenward.CABRERA-NGUYEN References Alegría.…Carpenter. New York. & Bravo. R. Royston. 10(7). 42(3)..1080/00273170701382864 Garson. I.. Canino. Multivariate Behavioral Research. D.1177/0011000006288127 Yu. Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. doi:10.b2393 Worthington. T. …Shrout. (2004). Structural Equation Modeling.1002/mpr. Statnotes: Topics in Multivariate Analysis: Factor Analysis. N.3109/09540269409023267 Costello. doi:10. R.org/vl=2149329/cl=1 1/nw=1/rpsv/cw/nasw/10705309/v32n3/s5/p171 Sterne. doi:10. Reporting practices in confirmatory factor analysis: An overview and some recommendations. K. Henson. doi:10. 6(4).1136/bmj. 270-288. (2006). D. New York. Practical Assessment.pdf Harrington. M... 34. Counseling Psychologist. 11..chass. J. (1994). 6. G. M. J. (2009).. 32(3). 509-529. L.. 806-838. Vila. M. P.htm Jackson. 13(4). 623.. M. 320-341. doi:10. R. (2005). 79-90. New York. doi:10. & Whittaker.edu/garson/pa765/statnote.1198/000313007X172556 Hu. Psychological Methods. R. (2007). 157-160. BMJ: British Medical Journal. Retrieved from http://titania. (2006). J. Gillaspy. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. R. K. doi:10. A. Sensitivity of fit indices to model misspecification and model types. NY: Guilford Press. G. 1-55. M... Much ado about nothing: A comparison of missing data methods and software to fit incomplete data regression models. D. 14. (2009).).1080/10705519909540118 Journal of the Society for Social Work and Research 103 . & Sivo.1177/0013164405282485 Horton. Retrieved from http://faculty. (2006). T. NY: Guilford Press. NY: Oxford University Press. (2005). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Canino.asp?v=10&n=7 Fan. Vera. University of California. Principles and practice of structural equation modeling (2nd ed. S. Marsh.. Social Work Research. Structural Equation Modeling. & Osborne.1207/s15328007sem1103_2 Rose. A simplified framework for using multiple imputation in social work research... P. M. & Wen. J. 61(1). & Fraser. (2010). In search of golden rules: Comment on hypothesis-testing approaches to setting cutoff values for fit indexes and dangers in overgeneralizing Hu and Bentler's (1999) findings. (2007). (2004).. 281-286.net/getvn.. (2009).
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue listening from where you left off, or restart the preview.