You are on page 1of 8

Reliability in Qualitative Research

External reliability Would independent researchers discover the same phenomena or generate similar constructs in similar settings? Internal reliability Given a set of previously generated constructs, would other researchers match data in the same way as the original researcher? Five ways to reduce threats to internal reliability: Mechanical recording Low inference descriptions (verbatim quotes and detailed accounts) Multiple researchers Participant researchers Peer examination Goetz, J. & LeCompte, M. D. (1984). Ethnography and qualitative design in educational research. Orlando, FL: Academic Press.

Reliability (pp. 205-207) Can the findings be replicated? Or more fitting with qualitative research, are the results consistent with data collected? Making studies dependable: The investigator’s position with regard to assumptions & frameworks and in relation to those studied is made explicit. Triangulation (multiple methods of data collection and analysis) Audit trail Merriam, S. (1998). Qualitative research and case study applications in education (2nd ed.). San Francisco: Jossey-Bass.

Reliability ‘refers to the degree of consistency with which instances are assigned to the same category by different observers or by the same observer on different occasions’ (Hammersley, 1992, 67). For reliability to be calculated, it is incumbent on the investigator to document his or her procedure and to demonstrate that categories have been used consistently (Silverman, 2000, p. 188). Reliability and observation Short notes made at the time Expanded notes made as soon as possible after each field session

1

Reliability and transcripts Standardization of transcript conventions Group data analysis (with audio. text. and interaction.A fieldwork journal to record problems and ideas that arise during each state of fieldwork A provisional running record or analysis and interpretation (Spradley. 2 . CA: Sage. (1993). Interpreting qualitative data: Methods for analysing talk. 1979) Reliability and texts Categories should be applied in a standardized way.or video-data) Silverman. Inter-rater reliability: involves giving the same data to a number of analysts (or raters) and asking them to analyze it according to an agreed set of categories. London & Thousand Oaks. D. Reports are examined and any differences are discussed.

event.) Are investigators observing and measuring what they think they are measuring? Triangulation (with multiple researchers. J. describes how typical the program. or methods of confirmation) Member checks Long-term observation Peer examination Participatory or collaborative modes of research Researcher’s biases/subjectivities – clarifying assumptions. sample is compared with the majority of others in the same class so that others can make comparisons with their own situations) Multi-site designs Reader or user generalizability Merriam. & LeCompte. theoretical framework of study External validity (pp. M. multidimensional and ever changing. (1998).Validity in Qualitative Research Internal validity The extent to which scientific observations and measurements are authentic representations of some reality External validity The degree to which such representations can be legitimately compared across groups – “typicality” of phenomena Validity increases by: Increased length of time in site reduces artificial responses Triangulation of data through multiple sources Goetz. sources of data. thick description Typical or modal category (eg. Internal validity (pp. San Francisco: Jossey-Bass. D. S. Ethnography and qualitative design in educational research. (1984). 207-212) What is the extent to which the findings can be applied to other situations? How generalizable are the results? Rich. worldview. FL: Academic Press. 201-205) How closely do research findings match reality? (Reality for qualitative researchers is holistic. Qualitative research and case study applications in education (2nd ed.). Orlando. 3 .

Reality is always viewed through particular perspectives. Silverman. Reality is assumed to be independent of the claims that researchers make about it. D. and interaction. Interpreting qualitative data: Methods for analysing talk. 1993. Methods of testing hypotheses 3. (1993). p. Hammersley suggests that researchers can address issues of validity by adopting a ‘subtle form of realism’. The use of simple counting procedures (Silverman. 50-51). text. 57). 1992. Silverman (2000. I mean truth: interpreted as to the extent to which an account accurately represents the social phenomena to which it refers (Hammersley. ie: Validity is identified with confidence in our knowledge but not certainty.Validity ‘By validity. Methods of generalising to a larger population 2. 166). 4 . 188) argues that we cannot say that the claims of a research study are valid when: Only a few exemplary instances are reported The criteria or grounds for including certain instances and not others are not provided The original form of the materials is unavailable Silverman (1993) rejects triangulation and members’ validation as appropriate methods for validating qualitative studies. p. 1990. London & Thousand Oaks. CA: Sage. He suggests instead: 1. hence our accounts represent reality they do not reproduce it (Hammersley.

M. analysis. The validity of an account or interpretation is judged in terms of the consensus about words. listen a lot Record accurately. 5 . immediately after or during events Begin writing early. sources. 169). interpretation. include primary data in final accounts Report fully. 337373). (1990). H. in their words. yet an account is always fallible’ (p. CA: Sage. On seeking and rejecting – validity in qualitative research. (1994). and interpretation – either positively or negatively Intellectual rigor Patton. Relativized validity ‘validity of an account is relative to the standards of a particular community at a particular place and time. analysis. Qualitative evaluation methods. Thousand Oaks: Sage. deal with discrepant cases Be candid. concepts.Techniques of analysis for enhancing the quality and validity of qualitative data Testing rival explanations Negative cases Triangulation (methods. (pp. Transforming qualitative data: Descriptions. standards and so on in a given community of interpreters’ (p. 169). analyst. Different positions taken by scholars on issue of ‘validity’: Fallibilistic validity ‘validity is understood as a test of whether an account accurately represents the social phenomena to which it refers…Defenders of this view hold that one can have good reasons for accepting an account as true or false. Thousand Oaks. then reread the draft Write accurately – write for technical accuracy. theory/perspective) Keep data in context Techniques for enhancing the credibility of the researcher Report any personal and professional information that may have affected data collection. internal consistency with generalizations grounded in what is seen/heard Wolcott. share drafts with others knowledgeable about the setting Let readers see for themselves. Wolcott. In H. Wolcott describes ‘getting it right’ or trying not to ‘get it all wrong’ by the following activities: Talk little. sees subjectivity as a strength of qualitative research Seek feedback Try to achieve a balance – return to site or field notes to reread the data.

D. 170). (1997). Silverman. empowerment – a criterion of action (Kvale. London & Thousand Oaks. Comprehensive data treatment All cases of data are incorporated in the analysis Deviant-case analysis Actively seek out and address deviant cases Using appropriate tabulations Use of simple counting techniques. Schwandt. Silverman (2000) suggests 5 ways of thinking critically about qualitative data analysis in order to aim at more valid findings: The refutability principle Seek to refute assumed relations between phenomena (Popper’s ‘critical rationalism’) The constant comparative method Attempt to find another case through which to test out a provisional hypothesis. CA: Sage. theoretically derived and based on members’ own categories can offer a means to survey the whole data corpus. No single account can be judged superior to others – a needless interplay of different interpretations’. and interaction. Thousand Oaks. (1993). Lather as described by Schwandt. also can be valid if it leads to change. Interpreting qualitative data: Methods for analysing talk. text. p. Qualitative inquiry: A dictionary of terms. 6 . Guba & Lincoln. T. Nonepistemic validity Validity may be interpreted as a criterion of good communication or dialogue. CA: Sage.No validity The most radical of postmodernists would argue that it is meaningless to talk of a ‘true account of the work – there are only different linguistically mediated social constructions.

investigators. empowers to action Lincoln. 7 . Denzin.Other criteria for establishing quality in qualitative research Trustworthiness How can an inquirer persuade his/her audiences that findings of an inquiry are worth paying attention to. & Guba. dependability. Newbury Park. E. to make applications to other situations Dependability (qualitative research equivalent to reliability) Triangulation Multi-site research design/teams of researchers in close communication Audit trail Confirmability (qualitative research equivalent to objectivity) Reflexive journal Audit trail Triangulation From a constructivist perspective: Trustworthiness: credibility. stimulates to action. methods. confirmability Authenticity: fairness. transferability. leads to improved understanding of constructions of others. (1985). 1978) Peer debriefing Negative case analysis Referential adequacy (archived recorded data for others to reference) Member checks Transferability (qualitative research equivalent to external validity) Thick description Adequate data base for others to make transferability judgements possible. theories. Y. CA: Sage. enlarges personal constructions. worth taking into account? Trustworthiness is established by the following criteria/activities: Credibility (qualitative research equivalent to internal validity) Prolonged engagement Persistent observation Triangulation (sources. Naturalistic inquiry.

theory triangulation: approach data with several hypotheses in mind to see what happens 4. methodological triangulation: (eg ethnography + interviews. Use researcher’s concepts to predict members’ descriptions 2. team research. (1999). engagement in continued discussion 3. Regard successful action research as a form of member validation. Triangulation: 1. investigator triangulation: multiple observers in the field. 3. mixed methods) Member validation: 1. C. Ask members to judge the adequacy of the researcher’s account: Strong version: members evaluate the final report Weak version: members comment on the accuracy of some interim document (eg. Seale.Triangulation & Member Validation These techniques were originally proposed to assist in converging on a ‘single true version. diverse sources of data (find phenomenon in several different settings. different points in space & time) 2. The quality of qualitative research. 4. London: Sage. 8 . Show that the researcher’s account can lead to successful ‘passing’ as a member. interview transcript).’ They may also generate multiple perspectives.