Paradigm Wars - Validity and Reliability in Qualitative Research
development of alternative quality indicators within them are highlighted. Since the usefulness of this multitude of indicators is questionable, reconciliation is attempted by consolidating them. The concepts of “core validity” and “core reliability”, which can be specified according to the researcher’s paradigm, are introduced for this task. These concepts underline the relevance and applicability of validity and reliability as quality indicators in qualitative research. Furthermore, qualitative research has developed strategies and methods, which enable the researcher to address negative influences on validity and reliability and achieve high degrees of both.
Validity and reliability as quality indicators have an uneasy standing in qualitative research and are subject to numerous debates. Researchers from different paradigmatic backgrounds expressed a variety of views, the extremes ranging from a complete denial of the possibility of valid and reliable qualitative research on one hand to the rejection of validity and reliability as meaningful quality indicators on the other. The following essay acknowledges the diverging assumptions underlying the different paradigms associated with quantitative and qualitative research. However, it denies that validity and reliability are inherently connected to predetermined ontological or epistemological assumptions and argues for their general use as quality indicators. To clarify this claim, a selection of different paradigms and the development of alternative quality indicators within them are highlighted. Since the usefulness of this multitude of indicators is questionable, reconciliation is attempted by consolidating them. The concepts of “core validity” and “core reliability”, which can be specified according to the researcher’s paradigm, are introduced for this task. These concepts underline the relevance and applicability of validity and reliability as quality indicators in qualitative research. Furthermore, qualitative research has developed strategies and methods, which enable the researcher to address negative influences on validity and reliability and achieve high degrees of both.
The pursuit of scientific research is characterized by different paradigms, meaning “the entire constellation of beliefs, values, techniques and so on shared by members of a given community” (Kuhn 1970). While the preoccupation of researchers is to conduct research in accordance to quality standards promoting acceptance within their community (Rodwell and Byers 1997), these standards often differ between paradigms. Furthermore, social scientists from different paradigmatic backgrounds show a clear orientation towards either the use of qualitative or quantitative research methods. Validity and reliability as central quality indicators originated in quantitative research, which is dominated by the positivist paradigm (Golafshani 2003). Disagreements with the basic assumptions of positivist research and allegations that this paradigm assumed a restrictive hegemonic position in social sciences led some qualitative researchers to refute the utility of validity and reliability (Denzin 2009, Smith 1984). Instead, they created a multitude of alternative quality indicators (Morse et al. 2002). The partial renunciation of validity and reliability in qualitative research and the resulting inconsistent standards produced the perception that qualitative research cannot be properly valid and reliable (Maxwell 2002). Did the two quality indicators thus become victims of what Gage dubbed the “paradigm wars” (Gage 1989)? This essay argues that the notion of conflict and its consequences obscure the fact that validity and reliability are central elements of research quality, regardless of the chosen approach. They are not inherently connected to positivist or quantitative assumptions (Morse et al. 2002). Conversely, when connected with the theoretical assumptions of the researcher’s paradigm, they can be used in any research endeavour. This fully applies to qualitative research, which is equipped with a variety of methods to address challenges to validity and reliability. Consequent application of these methods throughout the entire research process can undoubtedly produce properly valid and reliable qualitative research.
In order to establish this claim, it is essential to understand how doubts about the validity and reliability of qualitative research arose. Using two frameworks to assess scientific paradigms, positivism is introduced and the development of validity and reliability within quantitative research depicted. Ontological and epistemological differences to realism and constructivism, two paradigms associated mainly with qualitative research on which this essay will focus, led researchers from these backgrounds to create alternative quality indicators, distancing themselves from validity and reliability. In the following section the essay reconciles this divergence using the concepts of “core validity” and “core reliability”, which can be adapted to the theories underpinning the researcher’s scientific paradigm. Quality indicators from both qualitative and quantitative research can be allocated to these concepts, underlining the general importance of validity and reliability. Qualitative research still faces a number of challenges to validity and reliability, a selection of which is explained in the final section. However, by addressing these challenges throughout the whole research process using adequate methods, their negative effects on research quality can be substantially reduced. While “perfect” validity and reliability remain an illusion for any kind of research (Carmines and Zeller 1979), the arguments in this essay strongly support the claim that qualitative research can be properly valid and reliable.
The quality conflict - indicators and paradigms
In order to illustrate the conflict between quantitative and qualitative understanding of research quality, the following section will firstly introduce two frameworks for assessing scientific paradigms. Secondly, the emergence of reliability and validity as quality indicators for quantitative research within a largely homogeneous positivist research environment is depicted. This contrasts with the heterogeneity of different paradigms associated with qualitative research. The end of the section illustrates how researchers from realist and constructivist backgrounds developed alternative quality standards and partially refuted reliability and validity as meaningful concepts, giving rise to claims that qualitative research fails to ensure properly reliable and valid results (Morse et al. 2002).
The first framework by Guba and Lincoln differentiates “inquiry paradigms” using three fundamental questions: the ontological question about the form and nature of reality, the epistemological question about the relationship between the knower and what can be known and the methodological question about the ways to find out what can be known. The answer to the first question is restraining the answer to the second, and these two answers in turn restrain the answer to the third (1994). On a more practical level, LeCompte and Goetz identify distinctions in three significant areas. The formulation of problems encompasses the definition of the research area, research design and investigative methods. The nature of goals reflects the position of creation and use of theory in the research process. The application of results is the question between generalizing the findings to a larger population or limiting them to the researched case (1982). These two frameworks are useful in order to approach the different paradigms and their view on reliability and validity.
Reliability and validity as indicators of research quality originated within a homogeneous research environment marked by the positivist paradigm and the almost exclusive use of quantitative methods (Simco and Warin 1997). Positivist researchers share an objective ontology characterized by the assumption that social phenomena have an existence and a meaning independent of social actors (Bryman 2008). In line with this ontology, the positivist epistemology states that knowledge is derived from observable phenomena, which can be recorded value-free – i.e. objective – by an independent researcher (Matthews and Ross 2010). The dominant methodological approaches are based on the measurement of quantitative data, the application of statistical techniques and the identification of causal relationships, verified by hypothesis tests (Carmines and Zeller 1979). Research problems are analyzed through the manipulation of dependent and independent variables, trying to emulate the characteristics of an experimental setting (Welman et al. 2005). Theory is created prior to the analysis of the data with the goal of testing it deductively through validation or falsification of hypotheses. Through the experimental design and statistical methods the aim of the researcher is to generalize the results to a wider population (Black 1999). Four central elements of positivist research can be identified from the previous application of the two frameworks: the emphasis on objectivity, measurement, causality and generalizability. The definitions of reliability and validity consequently address these elements. A common definition for reliability is the consistency of measurements of a concept, using an identical measurement procedure, and the replicability of the findings (Lee 1999). It is often subdivided into internal reliability and external reliability. Internal reliability describes the consistency of the indicators used in the research, generally expressed as a correlation value between them. External reliability indicates whether the findings can be generalized beyond the research context (Black 1999). Assessing reliability is carried out through mathematical-statistical procedures, e.g. test-retest for stability estimates and Cronbach’s alpha for internal reliability (Carmines and Zeller 1979). Turning to validity, three types can be seen as central in positivist research: construct or measurement validity, internal validity and external validity (Black 1999). Measurement validity is the extent to which a measure really reflects the corresponding concept. Internal validity serves to ensure that assumed causal connections between independent and dependent variables are actually responsible for the observed phenomena. Finally, external validity expresses the generalizability of the findings beyond the specific research context (Bryman 2008). Validity is therefore a way to ensure quality in measurement, attest expected causalities and allow generalization. Measuring the degree of validity is similar to reliability largely based on numerical calculations (Lee 1999), although forms of peer-review are also used (Litwin 1995). Validity and reliability are essential indicators for quality in the largely quantitative positivist research community. As this community became dominant in social sciences research, especially in the United States (Riley 2007), social scientists from other paradigmatic backgrounds often took opposing views to the positivist assumptions.
While quantitative research is dominated by the positivist paradigm, paradigms associated mainly with qualitative methods are heterogeneous in their characteristics. Between quantitative and qualitative researchers there has been ongoing debate over the legitimacy of qualitative research as proper science (Denzin 2009). In their criticism, quantitative researchers focussed especially on missing standards of reliability and validity (Maxwell 2002). Qualitative researchers responded differently to this challenge. While those sharing positivist assumption tried to emulate the procedures of quantitative research (King et al. 1994), some researchers other paradigmatic backgrounds took opposing views (Denzin 2009, Morse et al. 2002). This essay concentrates on the realist and the constructivist paradigm. The critical paradigm, encompassing among others feminism and neo-Marxism, shares many assumptions with the constructivist paradigm (Guba and Lincoln 1994) and will not be individually analyzed.
The realist paradigm is placed closer to positivist assumptions than the constructivist paradigm (Bryman 2008). The realist ontology accepts the existence of an actor-independent reality, but assumes an additional hidden dimension of non-observable generative mechanisms. This structural dimension is only comprehensible through its effects on the actors’ behaviour (Matthews and Ross 2010). The epistemological view regards research as a process towards objectivity, but due to the hidden structural features of the world only an approximation of this goal can be attained. Methodologically the realist paradigm is advocating “critical multiplism”, the idea of approximating reality through the use of multiple ways of inquiry (Guba and Lincoln 1994). According to Healy and Perry, realist research adapts the formulation of problems to “critical multiplism”. As social phenomena are contingent, causal relationships are not primary. While realist research does incorporate quantitative alongside mainly qualitative methods, theory is derived predominantly inductive during the research process and later subjected to hypothesis tests. Generalizing the findings of the research after successful theory tests is an aim (2000). This definition of the realist paradigm allows an assessment of how the four central elements of the positivist research identified above are perceived. Objectivity and generalizability are ideals, but in accordance with the realist ontology these can only be approximated (Madill et al. 2000). Discrepancies are obvious in the attitude towards measurements, which are mainly qualitative and centred around a multi-method approach. Furthermore, causality only plays a subordinate role. Stressing the differences between the two paradigms, Healy and Perry propose six quality criteria for realist research, as according to them “research done within a paradigm has to be judged by its own paradigm’s terms” (2000:121-122). Ontological appropriateness means ensuring the conducting of research according to the realist ontology. Contingent validity concerns the generative mechanisms under research and their embedding in relevant contexts. Reference to critical multiplism is made by multiple perceptions, which stresses the use of triangulation. Methodological trustworthiness is a transparency criterion, calling for the creation of auditable research through standardized and extensive documentation. Analytic generalisation stresses inductive approaches to theory-building rather then deductive theory-testing. Finally, construct validity parallels positivist measurement validity as an indicator for the degree to which the measurement reflects the measured concept (Healy and Perry 2000). The six quality indicators thus do not entirely refute positivist validity and reliability preoccupations, but the realist ontological and epistemological positions are pronounced.
 See section three of the essay for a short definition.