Retrieved From:

htm Last Updated: 25/8/2005

Evaluation Tool for Mixed Method Studies
Building on work within a project exploring the feasibility of undertaking systematic reviews of research literature on effectiveness and outcomes in social care, a set of evaluation tools have been developed to assist in the critical appraisal of research studies. The evaluation tool for mixed studies allows appraisal of both the qualitative data collection and analysis component and the wider quantitative research design. It is applicable where the aim of the qualitative component is to draw out the informants’ understandings and perceptions. The tool has seven sub-sections: study evaluative overview; study and context (setting, sample and outcome measurement); ethics; group comparability; qualitative data collection and analysis; policy and practice implications; and other comments. Review Area Key Questions (1) STUDY OVERVIEW Bibliographic 0. Author, title, source (publisher and place of publication), year Details Purpose 1. What are the aims of this paper? 2. If the paper is part of a wider study, what are its aims? Key Findings 3. What are the key findings of the study? Evaluative 4. What are the strengths and weaknesses of the study and theory, policy and Summary practice implications? (2) STUDY, SETTING, SAMPLE AND ETHICS The Study 5. What type of study is this? 6. What is the intervention? 7. What was the comparison intervention? 8. Is there sufficient detail given of the nature of the intervention and the comparison intervention? 9. What is the relationship of the study to the area of the topic review? Context I: Setting 10. Within what geographical and care setting is the study carried out? 11. What is the rationale for choosing the setting? 12. Is the setting appropriate and/or sufficiently specific for examination of the research question? 13. Is sufficient detail given about the setting? 14. Over what time period was the study conducted? Context II: 15. What was the source population? Sample 16. What were the inclusion criteria? 17. What were the exclusion criteria? 18. How was the sample (events, persons, times and settings) selected? (For example, theoretically informed, purposive, convenience, chosen to explore contrasts) 19. Is the sample (informants, settings and events) appropriate to the aims of the study? 20. If there was more than one group of subjects, how many groups were there, and how many people were in each group?

What are the key characteristics of the sample (events. includes original/raw data extracts.Context III: Outcome Measurement (3) ETHICS Ethics 21. length and timing of observation work. Are the findings interpreted within the context of other studies and theory? Researcher's 40. participant Potential Bias observer) 41. To what setting are the study findings generalisable? (For example. times and settings)? 23. interviewer. checking back with informants over interpretation) 39. assumptions and possible biases outlined? (Indicate how these could affect the study. appropriateness and availability for independent analysis) 35. is the setting typical or representative of care settings and in what respects? If the setting is atypical. sampling procedure. to allow reproduction. evidence of iterative analysis. will this present a stronger or weaker test of the hypothesis?) 43. Were there other important confounding variables controlled for in the study design or analyses and what were they? 33. Is adequate evidence provided to support the analysis? (For example. matching. Is there sufficient breadth (e.g. persons. What was the researcher's role? (For example.searching for negative evidence. Was Ethical Committee approval obtained? 27. account of how the data were elicited. data triangulation). Is the conclusion justified given the conduct of the study (For example. How are the data analysed? 37. If there was more than one group was analysed. randomisation. interview guide. note taking) Data Analysis 36. representative evidence presented.g. in particular. How have ethical issues been adequately addressed? (4) GROUP COMPARABILITY Comparable 29. were the groups comparable Groups before the intervention? In what respects were they comparable and in what were they not? 30. Did the authors take these into account in their interpretation of the findings? (5) QUALITATIVE DATA COLLECTION AND ANALYSIS Data Collection 34. measures of outcome used and results achieved?) . time and settings. Whose perspectives are addressed (professional. service. steps taken to guard against selectivity) 38. contrast of two or more perspective) and depth (e. efforts to establish validity . What data collection methods are used in the study? (Provide insight into: data Methods collected. How were important confounding variables controlled (e. reliability/consistency (over researchers. What outcome criteria were used in the study? 24. How adequate is the description of the data analysis? (For example. user. or in the analysis stage)? 31. To what population are the study’s findings generalisable? 44. Was informed consent obtained from participants of the study? 28. type and range of questions. Is the process of fieldwork adequately described? (For example. Are the researcher’s own position. carer)? 25.g. use of multiple sources. insight into a single perspective)? 26. the analysis and interpretation of the data) (6) POLICY AND PRACTICE IMPLICATIONS Implications 42. Was this control adequate to justify the author's conclusions? 32. Is the achieved sample size sufficient for the study aims and to warrant the conclusions drawn? 22.

What are the implications for service practice? (7) OTHER COMMENTS Other Comments 47. Greater Manchester M6 6PU. List other study references Reviewer 50. Review date School of Nursing. Salford. Name of reviewer 51. Are there any other noteworthy features of the study? 49.45. UK Telephone: +44 (0)161 295 2211 | Fax: +44 (0) 161 295 2432 | Email: SchoolWebmaster | site help / accessibility statement | privacy statement | . What were the total number of references used in the study? 48. What are the implications for policy? 46. University of Salford.