ISSUES & OPINIONS

WHEN STATISTICAL SIGNIFICANCE IS NOT ENOUGH:

INVESTIGATING RELEVANCE, PRACTICAL SIGNIFICANCE,
1
AND STATISTICAL SIGNIFICANCE
Kaveh Mohajeri
IESEG School of Management, 3 rue de la Digue,
59000 Lille, FRANCE {k.mohajeri@ieseg.fr}

Mostafa Mesgari
College of Business Administration, Loyola Marymount University, 1 LMU Drive,
Los Angeles, CA 90045-2659 U.S.A. {mostafa.mesgari@lmu.edu}

Allen S. Lee
School of Business, Virginia Commonwealth University, 301 West Main Street,
Richmond, VA 23208-4000 U.S.A. {allenslee@alum.mit.edu}

The notions of significance and relevance have provoked much controversy and confusion among those who
conduct and those who are intended to be informed by quantitative research in the information systems (IS)
field. The history of quantitative research in the IS field and beyond reveals not only disputes over the
adequacy of statistical significance to warrant the scientific merits of research, but also pleas for drawing
attention to practical significance, as well as a lack of distinction between relevance and practical significance.
This essay offers a remedial, overarching account. We establish the position that statistical significance,
practical significance, and relevance are distinct qualities, where the latter two transcend mere statistical
concerns and respectively refer to the distinct matters of research impressiveness and real-world usefulness.
Furthermore, we draw attention to the importance of proper communication of quantitative/statistical analyses
through a detailed examination of published IS research. Our examination gives rise to three major issues.
The three issues are concerned with the proper communication of (1) research rigor, (2) practical significance,
and (3) research relevance. We express our opinions with respect to the three issues and provide a number
of recommendations.

Keywords: Research communication, rigor, relevance, nonacademic stakeholders, practical significance,

statistical significance

Introduction1 sion/factor analysis, to structural equation modeling (SEM),

Quantitative research has a long history in the information
systems (IS) field. The methods employed by quantitative IS
researchers have significantly evolved from simple regres-
1
Jason Thatcher was the accepting senior editor for this paper. Stacie Petter
served as the associate editor.
and nowadays, to advanced data analytics. The practice of
quantitative research, however, has remained vastly un-
changed over the course of the field’s history in terms of at
least one particular aspect: the quest for statistical signifi-
cance. Quantitative IS researchers, as with their peers in
many other social science fields, have been used to con-
sidering statistical significance a cornerstone of “scientific”
inference (e.g., Boudreau et al. 2001; Gefen et al. 2000;

DOI: 10.25300/MISQ/2020/13932 MIS Quarterly Vol. 44 No. 2 pp. 525-559/June 2020 525
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
Straub 2009; Straub et al. 2004). Nevertheless, the debates
over the notion of statistical significance have long estab-
lished that statistical significance has little, or nothing, to do
with scientific inference, and researchers need to change their
perspective toward what statistical significance is and what it
really indicates (e.g., Daniel 1977; Kirk 1998; Nickerson
2000; Selvin 1957). In the IS field, the mere reliance on
statistical significance and p-values has been seriously ques-
tioned as well, particularly over the past few years with the
increasing popularity of data analytics research, where the
issue described as “the deflated p-value problem” (Lin et al.
2013, p. 907) has drawn much attention. The issue is that
large datasets, which are common in data analytics research,
almost always lead to statistically significant p-values for
research results (e.g., Abbasi et al. 2016; George et al. 2014;
Lin et al. 2013; Müller et al. 2016).
There have been remedies suggested to address the situation
portrayed above. In particular, drawing on the concept of
practical significance (in contrast to statistical significance),
quantitative researchers have long been encouraged in many
disciplines (e.g., Chatfield 1995; Cohen 1977; Daniel 1977;
Fisher 1925; Kerlinger and Pedhazur 1973; Kirk 1996;
Nickerson 2000; Pearson 1900; Selvin 1957) to supplement,
if not replace, their reporting of statistical significance with
interpretations or judgments of the magnitude (and corre-
sponding confidence intervals) of their obtained research
results. In the IS field, this turn toward practical significance
is much more evident, again, over the past few years with the
prevalence of data analytics research. Based on long-
established lessons from other disciplines, data analytics IS
researchers are now strongly advised to convey the scientific
merits of their research studies through giving, as much as
possible, objective and clear presentations and interpretations
of their obtained effect sizes, reporting confidence intervals,
and employing graphical charts (see Abbasi et al. 2016; Lin
et al. 2013; Müller et al. 2016).
Despite the seemingly positive developments in the IS litera-
ture with respect to encouraging quantitative researchers to go
beyond statistical significance, the observations we share in
the current essay indicate that the status-quo research practice,
widespread among quantitative IS researchers, is still far from
satisfactory. We observe that, in either the more traditional
statistical research involving comparatively small samples or
the recent data analytics research utilizing large samples, the
majority of IS research studies still tend to only focus on
reporting the statistical significance of the results. In other
words, the majority of IS research studies still delegate the
task of interpreting the magnitude of results to their academic
or nonacademic audience. In our view, this unpleasant situa-
tion in quantitative IS research can be partially attributed to
526 MIS Quarterly Vol. 44 No. 2/June 2020
conceptual confusion concerning the three concepts of statis-
tical significance, practical significance and relevance. For
instance, we searched for the usage of the term practical signi-
ficance throughout the history of MIS Quarterly and found
that, out of 24 papers and editorials mentioning the term, only
10 papers2 use it in the context of statistical analysis and in
contrast to statistical significance, and the rest mostly use the
term to imply the real-world usefulness of research in contrast
to its theoretical importance. On this basis, we believe there
is still considerable misunderstanding in the IS community
with respect to such fundamental questions as: What does
statistical significance have to do with scientific inference?
What is practical significance? Why is reporting on practical
significance a necessary practice in all forms of statistically
conducted research (not merely data analytics)? What should
appropriate reporting on practical significance in a research
study look like? And, why does obtaining and reporting
research results bearing practical significance not necessarily
mean that a research study bears relevance to the real world?
Against the outlined backdrop, the contribution that the cur-
rent essay makes is twofold. On the one hand, this essay
offers illustrative, yet critical, material concerning the three
notions of statistical significance, practical significance and
relevance, all in one place, in order to clarify the conceptual
distinctions among them for IS researchers. On the other
hand, the current essay advances a coherent account on the
proper communication of practical significance and, espe-
cially, research relevance, in the context of quantitative IS
research. In the next two sections, we make distinctions
among the three concepts of statistical significance, practical
significance and relevance in quantitative research, so that IS
researchers may consider this essay a reference point for their
understanding of the three concepts—including the distinction
between relevance and practical significance. Next, we
examine the extent to which practical significance and
relevance have been considered in the published IS research.
Finally, in the fifth section, we discuss three major issues
arising from our review of published IS research and highlight
additional considerations and tasks that IS researchers must
take into account as a part of modern, rigorous quantitative
research practice.
2
Adams et al. (1992), Banerjee et al. (1998), Bapna et al. (2017); Brown and
Venkatesh (2005), Chan et al. (1993), Hong et al. (2013), Igbaria et al.
(1994), Kettinger et al. (1994), Petter et al. (2007), and Venkatesh and Morris
(2000).
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
Some Background on Significance
and Relevance
The work in this essay essentially revolves around two
general notions we consider to be pertinent to statistically
conducted IS research: significance3 and relevance.4 These
two notions, however, have developed in two separate intel-
lectual traditions. On the one hand, the notion of significance
(covering both statistical significance and practical signi-
ficance) has developed in the statistical school of thought
exclusively concerning the quantitative results of statistical
analyses. On the other hand, the notion of relevance, as far as
the management-school disciplines are concerned, has devel-
oped in connection with much broader concerns regarding the
real-world usefulness of management research (including
research models, constructs, variables, and results)—where
research employing statistical analysis is only one form—both
in practice and for society in general. We review the back-
ground of these two notions in the following paragraphs.
One major issue that led to the emergence of the notion of
significance was to address the issue that researchers con-
ducting inferential statistical analysis had to make sure that
results induced from only a sample of a population are suffi-
ciently reliable to show that hypothesized effects/relationships
actually exist in the population. Although the idea of statis-
tical significance tests dates back to the 1700s (see Hacking
1965), most of the developments happened in the early 1900s.
Fisher (1921, 1925) and Neyman and Pearson (1928) are
considered pioneers who worked out the underlying ideas,
concepts, and methods in order to enable researchers to show
whether their reported results can be considered sufficiently
3
Different forms of significance are discussed/envisaged in the literature
(e.g., economic significance or theoretical significance). However, in the
context of statistically conducted research, statistical significance and
practical significance are the only two forms of significance that have long
been recognized and established among the majority of statisticians and those
who conduct quantitative research in many disciplines. Therefore, in the
current essay, we only focus on these two forms of significance.
4
In our view, the notion of relevance offers the general case of which some
other notions such as clinical significance only represent a special form. In
particular, as for the notion of clinical significance (e.g., Campbell 2005;
Kazdin 1999; Thompson 2002), it is known that the notion “refers to the
practical or applied value or importance of the effect of” (Kazdin 1999, p.
332) therapeutic interventions. To extrapolate to the world of IS research, we
know that not all forms of quantitative IS research involve real-world inter-
ventions. On the other hand, clinical significance, as conceived of in the
fields of counseling, medicine and clinical psychology, is merely associated
with the results of research, not the model(s) or variables employed in a
research. On this basis, when it comes to the real-world usefulness of quan-
titative IS research, we prefer to adhere to the long established notion of
relevance in order to offer a broad enough account on the matter.
discernible5 (i.e., significant) in statistical terms. On this
basis, one may tend to say that, in the history of statistics, the
notion of significance was initially and mainly perceived as
statistical significance.
Controversy around the notion of statistical significance is
almost as old as the notion itself (see Berkson 1938). Statis-
tical significance has long been shown to be an inadequate
measure for scientific reasoning and credibility due to a
variety of reasons. However, our aim here is not to bring in
all the prolonged and detailed debates on the notions of
statistical significance and null hypothesis significance testing
(for a coherent account see Nickerson 2000). Yet, it is critical
for our purpose in this essay to emphasize a very fundamental
criticism of statistical significance. As explained by Kirk
(1996), a major criticism is that statistical significance, and
the procedure to establish it in statistically conducted
research, do not “tell researchers what they want to know” (p.
747). According to Kirk (p. 747):
[N]ull hypothesis significance testing and scientific
inference address different questions. In scientific
inference, what we want to know is the probability
that the null hypothesis (H0) is true given that we
have obtained a set of data (D); that is, p(H0|D).
What null hypothesis significance testing tells us is
the probability of obtaining these data or more
extreme data if the null hypothesis is true, p(D|H0).
As Falk (1998) argues, p(H0|D) and p(D|H0) can be equal, but
that happens only under rare mathematical conditions. There-
fore, as Kirk (p. 747) explains:
Researchers reason incorrectly that if the p value
associated with a test statistic is suitably small, say,
less than .05, the null hypothesis is probably false.
This form of deductive reasoning has been referred
to by Falk and Greenbaum (1995) as the “illusion of
probabilistic proof by contradiction.”
In the IS field, reactions to the inadequacy of statistical signi-
ficance have emerged much more recently (e.g., Abbasi et al.
2016; Lin et al. 2008, 2013; Müller et al. 2016; Shmueli and
Koppius, 2011). What has triggered IS researchers to attend
to the matter seems to stem chiefly from one of the classic
criticisms of statistical significance, namely, the issue that
5
This term comes from Wonnacott and Wonnacott (1984, pp. 259-261).
Basically, the authors suggest that the term statistically significant be
replaced by statistically discernible. That is, a theorized difference (e.g.,
between two means or between a particular value and zero) would be said to
be statistically discernible through a sample.
MIS Quarterly Vol. 44, No. 2/June 2020 527
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
research results nearly always turn out to be statistically signi-
ficant when the sample size is large enough (see Chatfield
1995; Nickerson 2000). This classic criticism has drawn
much attention in the IS field particularly because of the
increasing prevalence of what is known as data analytics
research. In data analytics research, IS researchers usually
exploit datasets—drawn from the web, social media, mobile
devices, or sensor networks—that contain a huge number of
observations. Relying on very large sample sizes bears many
advantages, but comes at the high price of encountering what
is termed the deflated p-value problem (see Lin et al. 2013).
In other words, in research studies that employ large datasets,
the p-values associated with research results can go quickly
to zero, creating the illusion that the research results are scien-
tifically supported, while the magnitude of the research results
may have no particular meaning or significance in research or
real-world terms.
Most of the remedies offered in the IS field and beyond with
respect to the inadequacy of statistical significance revolve
around another form of the notion of significance (i.e.,
practical significance). Historically speaking, along with
placing emphasis on the statistical discernibility of results,
statisticians were also aware of another demanding issue of a
different nature: the research impressiveness6 of results. In
other words, researchers conducting inferential statistical
analysis also had to make sure that the magnitude of their
reported results are sufficiently impressive so that they could
consider their hypothesized effects/relationships to actually
matter in the research area they are concerned about. Kirk
(1996) explains that Pearson (1900) and then Fisher (1925)
were among the first to react to the issue of practical signifi-
cance. In particular,
Fisher (1925) proposed that researchers supplement
the [statistical] significance test in analysis of
variance with the correlation ratio or eta, which
measures the strength of the association between the
independent and dependent variables (Kirk 1996, p.
748).
Later reactions to this issue can also be identified in the
behavioral and educational literatures (e.g., Cohen 1977;
Daniel 1977; Kerlinger and Pedhazur 1973), and the socio-
logical literature (e.g., Selvin 1957), among others.
6 statistics and psychology is beyond the scope of this essay. For a coherent
We acknowledge the fact that “in some cases effect sizes are most
impressive by being as small as possible” (Kelley and Preacher 2012, p.
148). Therefore, we prefer to use the term impressiveness (as suggested by
Kelley and Preacher 2012) in lieu of the term largeness, when referring to the
concept of practical significance.
528 MIS Quarterly Vol. 44 No. 2/June 2020
In the IS field, as with many other disciplines, the effort to
encourage researchers to address practical significance is
mainly focused on promoting the practice of reporting and
interpreting effect sizes (and their corresponding confidence
intervals)7 (e.g., Abbasi et al. 2016; Aguirre-Urreta and
Rönkkö 2018; Lee and Mohajeri 2012; Lin et al. 2013). This
was once described as “the effect size movement” (Robinson
et al. 2003, p. 51). For instance, in the case of large-sample
studies in IS, Lin et al. (2013, p. 909)
suggest that in addition to the traditional statistical
tests, researchers should, as much as possible, be
objective and clear in helping readers understand the
meaning of the coefficient estimates [i.e., effect size
values] within the study context.
To do so, the authors discuss the utility of what they call
“marginal analysis,” through which the sensitivity of a depen-
dent variable to changes in an independent variable may be
reported by researchers. The same kind of analysis is advo-
cated by Lee and Mohajeri (2012). Furthermore, along the
same lines, Aguirre-Urreta and Rönkkö (2018) argue that IS
researchers should offer interpretations of their obtained
research results (i.e., effect sizes and their corresponding con-
fidence intervals) in terms of what those results mean for the
literature under examination. However, as stated earlier, we,
in this essay, make a contribution by distinguishing between
practical significance (i.e., research impressiveness of statis-
tical results) and relevance (i.e., real-world usefulness of
research and its results). In fact, despite what seems to be a
major turn toward practical significance in the IS literature, it
appears that what is referred to as practical significance in our
field is vastly conflated with the notion of relevance (as
indicated in the “Introduction” section).
Reviewing the background of the notion of relevance suggests
that Koontz (1961) is perhaps one of the earliest works
reacting to the issue of usefulness of management research.
Koontz, arguing for the clarification of management theory,
emphasized the usefulness and understandability that such
theory should have in the eyes of practitioners. Thirty two
years later, Hambrick (1994), in his Academy of Management
presidential address,8 voiced substantially similar concerns,
criticized the minimal innovation, visibility, and impact of
management research, and called for making “a significant
contribution to the solution of major problems facing our
7
A full elaboration on effect size measures as offered in such fields as
account on effect size measures see Breaugh (2003), Huberty (2002), Kelley
and Preacher (2012), and Vacha-Haase and Thompson (2004).
8
The presidential address was given in 1993.
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
society and its value-creating enterprises” (p. 15). Concerns
for the usefulness or impact of management research are still
raised in the 21st century, where reports, such as one from the
AACSB International (2008), draw our attention again to the
criticality of enhancing the value and visibility of manage-
ment scholarship and research. In the IS field, the most
notable works on the notion of relevance, in our view, are the
1999 MIS Quarterly essays on “rigor vs. relevance” where IS
scholars provide their own definitions of the concept of
relevance, argue for or against the lack of relevance in IS
research, and recommend ways by which IS research rele-
vance could be improved. Prior to, and following, the 1999
MIS Quarterly discussion, we also have some invaluable
works and commentaries such as the ones by Dennis (2001),
Desouza et al. (2006), Hassan (2014), Robey and Markus
(1998), Rosemann and Vessey (2008), and Straub and Ang
(2008, 2011). Although our review of the notion of relevance
in the IS literature shows maturation in the perspectives
developed on the notion over time (see Mohajeri and Leidner
2017), we did not find any coherent account particularly
addressing relevance in the context of statistically conducted
research.
The foregoing review supports our position that there exist
major distinctions among the concepts of statistical signi-
ficance, practical significance and relevance. To better illus-
trate our contention, we highlight all the key distinctions
among the three concepts in Table 1.
As demonstrated in Table 1, besides the fact that statistical
significance, practical significance and relevance each refer
to a different quality, it is clear based on our review that
whereas the first two concepts are only concerned with the
quantitative results of a particular form of research (i.e., statis-
tically conducted research), the third concept is concerned
with research as a whole not just its results, whether that
research is statistically conducted or not.
To develop a more nuanced view on the distinction between
practical significance and relevance, another point we wish to
emphasize is the difference in the entities making judgments
about practical significance and relevance. As shown in
Table 1, making judgments about practical significance falls
upon the shoulders of researchers. This can be inferred from
the effort statisticians have been historically making to scale
practical significance (e.g., certain thresholds specified for
small, medium, and large effects; see Cohen 1992; Kirk
1996). Kirk (2001, p. 214) has a well-expressed statement in
this regard:
I believe that researchers have an obligation to make
this kind of judgment. No one is in a better position
than the researcher who collected and analyzed the
data to decide whether the effects are trivial or not.
It is a curious anomaly that researchers are trusted to
make a variety of complex decisions in the design
and execution of an experiment, but in the name of
objectivity they are not expected to nor even encour-
aged to decide whether the effects are practically
significant.
On the other hand, in accordance with our review presented
above, we attribute judgments about research relevance to
nonacademic research stakeholders. Nevertheless, there
seems to be a common perspective in the literature that does
not recognize the distinction between the two different types
of entities (i.e., academics vs. nonacademics) in relation to
making judgments about practical significance and relevance.
For instance, Aguinis et al. (2010), drawing on Vaske et al.
(2002), point out that “practical significance involves a value
judgment made by the consumer of research” (p. 530). It is
clear from Aguinis et al. that “the consumer of research” in
fact refers to “practitioners” (see p. 530). Another example,
from the IS literature, is Lin et al. (2013), where the authors
indicate that the practical significance of research results
depends on the point of view of the general readership (i.e.,
nonacademics) of a research study (see p. 909). As men-
tioned before, we see this as an opportunity for us to make a
contribution by making explicit the difference between
relevance and practical significance.
The last point, again on the distinction between practical
significance and relevance, is that there exists a tendency in
the literature (e.g., Kelley and Preacher 2012; Kirk 1996) to
equate the research impressiveness of quantitative results with
the real-world usefulness of research or its results, and hence
not to make a full distinction between practical significance
and relevance. One might speculate this occurs, in part, due
to the fact that the relevance of the conceptual aspects (e.g.,
research models, hypotheses, variables, etc.) of statistically
conducted research is often a taken-for-granted quality among
statisticians. On this basis, statisticians mostly presume that
the only remaining condition to achieve relevance for the
entire research is to obtain impressive results. In the next
section, by illustrating the distinction between relevance and
practical significance, we indicate that such a presumption
can be questionable.
An Illustration of Relevance,
Practical Significance, and
Statistical Significance
In principle, a given academic study may bear any one, any
two, or all three of the qualities of statistical significance,
practical significance and relevance (although a condition for
MIS Quarterly Vol. 44, No. 2/June 2020 529
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance

Table 1. The Key Distinctions among Statistical Significance, Practical Significance and Relevance
Statistical Significance
Practical Significance
Relevance
Corresponding Quality Applicability Scope Primary Adjudicator
Researchers/
Statistical Discernibility Only Quantitative Findings
Academic Audience
Researchers/
Research Impressiveness Only Quantitative Findings
Academic Audience
Research Findings
plus Practitioners/
Real-World Usefulness
Research Models/ Nonacademic Audience
Constructs/Variables

Figure 1. The Distinctions and Overlaps among Relevant, Practically Significant, and Statistically
Significant Research

academic publication has traditionally been that statistical X1: organizational role of BI user9
significance be present). This idea is illustrated in the Venn X2: length of BI user training
diagram shown as Figure 1. å: the error term

We use two running examples to illustrate different scenarios
where academic research studies would fall under the seven
regions shown in the Venn diagram. Such an examination
also allows us to elaborate on the distinctions among statis-
tical significance, practical significance and relevance.
Consider the following statistical model (Model A) posited in
a study that a firm commissions researchers to conduct for the
purpose of investigating factors associated with the individual
proficiency in the usage of the firm’s business intelligence
(BI) program. (We will later put forward an alternative,
Model B.)
Model A: Y = â0 + â1 X1 + â2 X2 + å
Region 1: The research study falls in Region 1 in Figure 1 if
it has the profile shown in Table 2.
The scenario represented by Region 1 is useful for illustrating
that while research results can bear statistical significance,
practical significance, and relevance at the same time, the
three qualities are in fact distinct in terms of what they refer
to.
Regarding statistical significance, the research study reports
that the probability of obtaining results equal to, or more
extreme than, â^1 and â^2 is very low (e.g., below 5%) “if there
[are] no factors operating but chance” (Nickerson 2000, p.
263) and if the null hypothesis (H0) is true (Kirk 1996, pp.

Where:
Y: individual BI usage proficiency 9
X1 is a binary categorical variable with two possible values: 1 for mana-
gerial roles, and 0 for non-managerial roles.

530 MIS Quarterly Vol. 44 No. 2/June 2020

 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance

Table 2. The Profile of the Example Research Study in Region 1

The research is statistically significant because the p-values corresponding to the
T statistical significance ^ and â ^ are both small (p < .05).
estimated coefficients â 1 2

^ and â
The research is practically significant because the estimated coefficients â ^ have
1 2

T practical significance
T relevance
certain magnitudes to constitute what the researchers consider to be impressive effect
size values.
The research is relevant to the firm because:
• For the firm, “individual BI usage proficiency” is an understandable and noteworthy*
matter.
• For the firm, “organizational role of BI user” and “length of BI user training” are
understandable and actionable factors.
• For the firm, the magnitudes of the research results corresponding to “organizational
role of BI user” and “length of BI user training” are satisfactory to make a difference
to its question of interest, which pertains to “individual BI usage proficiency.”

*This means that “individual BI usage proficiency,” for whatever reason, is a matter worth paying attention to from the firm’s perspective.

746-747; Nickerson 2000, p. 262). As indicated earlier, this
probability, which is the measured level of statistical signi-
ficance, can be stated as p(D|H0), or what is commonly known
as the p-value; here, it is the probability of obtaining results
(D) equal to, or more extreme than, â^i if the null hypothesis
(H0) is true, where one null hypothesis is â1 = 0 and the other
is â2 = 0. In other words, the research study is ruling out
chance as a likely factor producing the values obtained for â^1
and â^2, so that the “organizational role of BI user” and the
“length of BI user training” are most likely related to
“individual BI usage proficiency.” In this way, the research
can be considered to have statistical significance.
Regarding practical significance, the researchers consider the
magnitude of each of the estimated coefficients â^1 and â^2 to
denote a sufficiently impressive effect size value for the
research also to claim practical significance.
Regarding relevance, the firm deems the magnitude of the
research findings corresponding to “organizational role of BI
user” and “length of BI user training” to be satisfactory to
make a difference to “individual BI usage proficiency” and
overall considers the research model’s variables to be under-
standable, noteworthy, and actionable.
Region 2: The research study falls in Region 2 in Figure 1 if
it has the profile shown in Table 3.
The case of Region 2 illustrates the situation in which statis-
tical significance and relevance can be present, even when
practical significance is absent.
Regarding Model A, the researchers conducting the statistical
study consider the research results to lack practical signi-
ficance because the estimated coefficients â^1 and â^2 do not
have magnitudes to constitute what the researchers consider
to be impressive effect size values with regard to the
dependent variable.
In this scenario, the firm nonetheless considers the research to
be relevant. Not only does the firm consider Model A to be
understandable and actionable, but also, despite the lack of
what the researchers consider to be an impressive effect size
value, the firm deems the magnitude of research results corre-
sponding to “organizational role of BI user” and “length of BI
user training” to be satisfactory to make a difference to
“individual BI usage proficiency.”
Regions 3 and 4: These two regions, which also apply Model
A, involve reasoning similar to the reasoning just presented
for Regions 1 and 2. The full text of the reasoning pertaining
to Regions 3 and 4 can be found in Appendix A.
Now, as for illustrating the scenarios under the remaining
regions in Figure 1, suppose that the research study hypothe-
sizes the following statistical model, Model B, in place of
Model A:
Model B: Y = â0 + â1 X1 + â2 X2 + å
Where:
Y: individual BI usage proficiency
X1: BI user’s age
X2: BI user’s gender
å: the error term

MIS Quarterly Vol. 44, No. 2/June 2020 531

Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance

Table 3. The Profile of the Example Research Study in Region 2

T statistical significance
X practical significance
T relevance
The research is statistically significant because the p-values corresponding to the
^ and â ^ are both small (p < .05).
estimated coefficients â 1 2
^ and â
The research is not practically significant because the estimated coefficients â ^
1 2
do not have magnitudes to constitute what the researchers consider to be impressive
effect size values.
The research is relevant to the firm because:
• For the firm, “individual BI usage proficiency” is an understandable and
noteworthy matter.
• For the firm, “organizational role of BI user” and “length of BI user training” are
understandable and actionable factors.
• For the firm, the magnitudes of the research results corresponding to “organiza-
tional role of BI user” and “length of BI user training” are satisfactory to make a
difference to its question of interest, which pertains to “individual BI usage
proficiency.”

Table 4. The Profile of the Example Research Study in Region 5

T statistical significance
T practical significance
The research is statistically significant because the p-values corresponding to the
^ and â ^ are both small (p < .05).
estimated coefficients â 1 2
^ and â
The research is practically significant because the estimated coefficients â ^
1 2
have certain magnitudes to constitute what the researchers consider to be impressive
effect size values.
The research is not relevant.

The firm does not consider the factors, indicated by the independent variables, to be
X relevance
actionable. Specifically, the firm does not consider it feasible to change an em-
ployee’s age or gender, or consider it ethical to select an employee based on age or
gender.

Region 5: The research study, which involves Model B, falls
in Region 5 in Figure 1 if it has the profile shown in Table 4.
Region 5 highlights the general situation where the commis-
sioned researchers have conducted research that yields
statistically and practically significant results, but the research
involves a model (here, “individual BI usage proficiency” =
â0 + â1 “BI user’s age” + â2 “BI user’s gender” + å) whose
relationships, variables, or parameters the firm does not
accept to be able to inform any action. In other words, even
if the research is statistically significant and practically signi-
ficant in the eyes of the researchers, the research can still lack
relevance. In this scenario, the firm does not consider it
feasible to change an employee’s age or gender, or consider
it ethical to select an employee based on age or gender. 10
The scenario represented by Region 5 is helpful for demon-
strating the fact that practically significant results do not
necessarily imply that research bears real-world usefulness or
relevance. This fact goes against many definitions of practi-
cal significance offered in the statistics literature where the
usefulness is assumed to unquestionably follow from the
practical significance of research results (e.g., Kirk 1996), or
where real-world noteworthiness or actionability is assumed
to immediately follow from academic judgments on the
magnitude of results (e.g., Kelley and Preacher 2012).
Regions 6 and 7: These two regions, which also apply Model
B, involve reasoning similar to the reasoning just presented
for Region 5. The full text of the reasoning pertaining to
Regions 6 and 7 can be found in Appendix A.

10
It is possible that a different firm could interpret the results differently. For
instance, a different firm might use the results to institute a training program, individual BI usage proficiency. Of course, the point remains that relevance,
targeting participants by age and gender, with the goal of enhancing their however construed, is a different matter from practical significance.

532 MIS Quarterly Vol. 44 No. 2/June 2020

 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
The seven different regions in Figure 1 using two research
models illustrate the differences among the concepts of statis-
tical significance, practical significance and relevance. Two
points are worthy of some elaboration.
The first point is that, across the seven regions, any determin-
ation of statistical significance, practical significance, and/or
relevance requires human judgment; there is no purely objec-
tive rule that can render the determination by itself. Even in
the case of statistical significance, the reliance on a level of
significance of .05 or .01 is only a convention and, therefore,
is itself only a judgment. As for practical significance, inter-
pretation of obtained effect sizes is a matter of human judg-
ment and also relies on convention. “Researchers study
varied phenomena using a plethora of paradigms and method-
ologies” (Cortina and Landis, 2009, p. 306). Consequently,
depending on the context and research domain/design, there
can certainly be situations where a small effect size can
denote a large practical impact and vice versa (e.g., see Hsieh
et al. 2012; Lewandowsky and Maybery 1998; Prentice and
Miller 1992; Rosenthal 1990; Rosenthal and Rubin 1979). On
this basis, factors such as population and sample size, the
particular effect size measures involved, and the nature of the
phenomenon in question, are in fact determinants of how the
magnitude of effect sizes obtained in a given research study
should be interpreted.11 Therefore, as Grissom and Kim
(2005) state, “knowledge of a result’s effect size can [only]
inform a judgment about practical significance” (p. 4). More-
over, not a convention but nonetheless requiring judgment
across the seven regions are decisions made by the non-
academic stakeholder about relevance, namely, whether the
dependent variable is understandable and noteworthy,
whether the independent variables are understandable and
actionable, and whether the magnitudes of the associations
between the latter and the former are satisfactory to make a
difference to their question of interest.
The second point is that the seven regions are sufficient for
illustrating the concepts of practical significance and rele-
vance, but do not necessarily exhaust all possible cases or
scenarios. For instance, the seven regions illustrate just one
form of relevance, which we name “action lever” relevance.
We are referring to Robey and Markus (1998) who define
“action levers” as “factors that decision-makers can influence
through direct action, example, or policy” (p. 13)—where the
action levers are the independent variables. In addition, there
is at least one other form of relevance, which we name
“general” relevance; it is a quality of research that makes a
difference to the nonacademic stakeholder’s question of
interest even without any independent variables that the non-
11
For interesting examples of small effect sizes that tell a “big story” and of
large effect sizes that do not, see Cortina and Landis (2009).
academic stakeholder can use as action levers. For example,
a statistical study establishing that telemedicine is beneficial
to patients can be considered relevant, even though the non-
academic stakeholder (a patient) has no control over the
presence or absence of the independent variable (tele-
medicine).
The Consideration of Practical Signi-
ficance and Relevance in Published
IS Research
In the previous sections, we examined the concepts of sta-
tistical significance, practical significance and relevance from
a theoretical standpoint in an effort to shed light on the dis-
tinctions among them, as well as to explicate situations under
which research may bear any one, any two or all three of
them. However, when thinking of published research, a dis-
tinction needs to be made between whether a research study
bears statistical significance, practical significance and rele-
vance, on the one hand, and how the same research study
actually reports or communicates the statistical significance,
practical significance and relevance, on the other hand. For
instance, a research study could have yielded very large statis-
tical results that are of practical significance, but the paper
reporting that research might not consider and discuss the
practical significance of the research’s results at all. In this
light, we take an empirical stance in the current section in
order to investigate the extent to which published IS research
considers and communicates the two critical qualities of prac-
tical significance and relevance. Our investigation is
primarily focused on all statistically conducted research
published in MIS Quarterly in 2015, where published IS
research is well represented. The protocol we use to review
the MIS Quarterly papers involves two steps. In the first step,
we present a bird’s-eye view of a total of 27 MIS Quarterly
papers published in 2015. The outcome of the first step is
presented in Table 5 below. Then, in the second step, we pick
three papers (the ones shown with a gray background in Table
5) reviewed during the first step in order to offer an in-depth
review of them, and to highlight pivotal lessons we have
learned from our review. In addition, we supplement, and
provide further support for the results of, our investigation
using randomly picked examples of research published in MIS
Quarterly, Information Systems Research, and Journal of the
Association for Information Systems in 2016. Finally, we
provide extra evidence from the only identified review of the
communication of practical significance in published IS
research (i.e., Lin et al. 2013).
As shown in Table 5, the 27 MIS Quarterly papers are evalu-
ated in terms of seven aspects (i.e., seven columns) which we
MIS Quarterly Vol. 44, No. 2/June 2020 533
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance

Table 5. The Consideration of Practical Significance and Relevance in MIS Quarterly 2015 Articles
The Consideration of
Practical Significance The Consideration of Relevance
Does the paper
indicate how its Does the
independent paper indicate
Does the paper (including how the
indicate how its moderating) magnitude of
dependent and mediating the RES w ould
Are there variables are variables are make a
Statistically Does the paper offer any separate paper relevant in the relevant in the difference in
Conducted Research judgment, w ith supporting sections on eyes of the eyes of the the eyes of the
Published in MISQ rationale, indicating W ho are the identified implications for identified identified identified
2015 w hether the RES are nonacademic nonacademic nonacademic nonacademic nonacademic
(Vol. 39) W hat are the reported effect sizes (RES)? practically significant? stakeholders? stakeholders? stakeholders? stakeholders? stakeholders?
Correlation Coefficients † , Total R≤, ÷≤,
Maruping and Magni Deviance, Path Coefficients, Indirect Effect No Managers Yes No No No
Coefficients
Managers, Industrial
Tian and Xu Regression Coefficients, Adj. R≤, Cohen’s ƒ≤ Only for Cohen’s ƒ≤ values Consortia, and Yes No No No
Policymakers
Only for one
Correlation Coefficients, Regression Only for one regression Health Policy Makers, IS
Salge et al. No No independent No
Coefficients, W ald ÷≤, R≤ coefficient Professionals
variable
Only for one
W ang et al. Regression Coefficients, Hazard Ratios No Security Managers Yes No Yes
hazard ratio
R≤, Path Coefficients, Cohen’s ƒ≤, Correlation Managers, Security
Johnston et al. No Yes No No No
Coefficients Managers
R 2pv , R 2pe , Regression Coefficients, AIC, Log-
Scherer et al. No Managers Yes No No No
Likelihood
Correlation Coefficients, Path Coefficients, R≤,
Park et al. ÷≤, ÷≤/df, NFI, CFI, RMSEA, AIC, Indirect Effect No Hospital Managers Yes No No No
Coefficients, Cohen’s ƒ≤
For the coefficients related
Vance et al. Direct Effect Coefficients, AIC, BIC Managers No No No No
to the research hypotheses
Correlation Coefficients, Regression
Faraj et al. No Firms No No No No
Coefficients, Log-Likelihood, W ald ÷≤
GFI, RMSEA, SRMR, CFI, NFI, TLI, Practitioners, Companies,
Hoehle and Venkatesh Correlation Coefficients, R≤, Unique Variance No Mobile Application Yes No No No
Explained, Path Coefficients Developers
Sykes Correlation Coefficients, R≤, Path Coefficients No Practitioners, Organizations Yes No No No
W u et al. R≤, Path Coefficients, Formative Item W eights No Managers Yes No No No
Correlation Coefficients, Regression Only for one regression Healthcare Policy Makers,
Gao et al. Yes Yes Yes No
Coefficients, R≤ coefficient Patients
Miranda et al. Correlation Coefficients, R≤ No Managers, Vendors Yes No No No
Executives, Firms,
RMSEA, AIC, BIC, CFI, TLI, R≤, SRMR, CD, ÷≤,
Iyengar et al. No Franchisees, and No No No No
Ä÷≤, Correlation Coefficients, Path Coefficients
Franchisors
Correlation Coefficients, Path Coefficients,
Only for two ÄCFI
Kankanhalli et al. Formative Item W eights, R≤, ÷≤, IFI, NNFI, CFI, Firms, Administrators Yes No No No
measurements
ÄCFI, RMSEA
Companies, W ebsite
Shen et al. Regression Coefficients, Log-Likelihood, R≤ No No No No No
Designers, Retailers
Path Coefficients, R≤, Correlation Coefficients, Organizations, HR
Armstrong et al. No No Yes Yes No
Cohen’s ƒ≤, VAF Managers, IS Professionals
Leonardi Regression Coefficients, fi≤, R≤, Adj. R≤ No None N/A N/A N/A N/A
Healthcare Organizations,
Healthcare Practitioners,
Yeow and Goh Regression Coefficients, ÷≤, Log-Likelihood No Yes No Yes No
Managers of Telemedicine
Systems
Regression Coefficients, Correlation Software Development
Ramasubbu et al. No Yes No No No
Coefficients, Adj. R≤, ÷≤ Organizations
Senior Business and IT
Sabherwal and Jeyaraj Regression Coefficients, R≤, Adj. R≤, ÄR≤ No Yes No No No
Executives
RMSEA, CFI, TLI, R≤, CD, ÷≤, ÷≤/df, Correlation
Boss et al. No Practitioners Yes No No No
Coefficients, Path Coefficients
Borrowers who use Peer-to-
Odds Ratio Coefficients, Pseudo R≤, Log-
Peer Lending Platforms,
Liu et al. Likelihood, W ald ÷≤, Adj. R≤, Correlation No No No No No
Peer-to-Peer Lending
Coefficients
Platforms
Correlation Coefficients, Regression
IT Executives and Sourcing
Rai et al. Coefficients, R≤, ÄR≤, Partial R≤, ÷≤, Log- No Yes No No No
Managers
Likelihood
Correlation Coefficients, Regression Venture Capital Industry,
Singh et al. No No No No No
Coefficients, Adj. R≤, Adj. R≤/Log-Likelihood Venture Capitalists
Regression Coefficients, W ald ÷≤, Odds Ratio University Administrators,
Han et al. Coefficients, Log-Likelihood, ÷≤, Nagelkerke No Campus Emergency No No No No
R≤, Mean Contrasts Managers
Yes: 65% No: 92% No: 81% No: 96%
Total: 27 No: 81% (22) None: 4% (1)
(17/26*) (24/26*) (21/26*) (25/26*)

†In our review of all the papers, we only considered the reported pairwise correlation coefficients between independent/mediating and dependent variables.
*Leonardi (2015) is not counted.

534 MIS Quarterly Vol. 44 No. 2/June 2020

 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
believe are pertinent to the issues of practical significance and
relevance. These seven aspects along with the results of our
review are discussed in the following paragraphs.
1. What are the reported effect sizes (RES)? In order for a
paper to consider the issue of practical significance, one
or more effect sizes must be stated in the first place.
Following Kirk (1996) and Kelley and Preacher (2012),
we consider a broad range of statistical measures as
effect size measures. This includes path/regression coef-
ficient, correlation coefficient, fit indices, measures of
variance explained and so forth.12 According to Table 5,
all of the papers have reported effect sizes of at least two
different types.
2. Does the paper offer any judgment, with supporting
rationale, indicating whether the RES are practically
significant? To merely report effect sizes, as captured by
the first aspect, however, does not suffice if a paper
intends to consider the issue of practical significance. As
explained in the previous sections, researchers should
offer judgments on the magnitude of effect sizes as well.
A quick look at Table 5 shows that, in fact, no paper
offers this kind of judgment with respect to all the effect
sizes that it reports. The table also shows that while 81%
of papers give no consideration to the issue of practical
significance at all, most of the remaining papers consider
and discuss the issue of practical significance with regard
to only a small portion of their reported effect sizes.
3. Who are the identified nonacademic stakeholders? In
order for a paper to consider the issue of relevance, one
or more nonacademic research stakeholders should be
identified in the paper in the first place. In our review of
MIS Quarterly papers, we noticed that the vast majority
of papers (i.e., 96%) identify, and target, at least one
group of nonacademic stakeholders.
4. Are there separate paper section(s) on implications for
nonacademic stakeholders? The fact that almost all of
the papers we reviewed identify at least one nonacademic
stakeholder should be viewed as a result of the editorial
policies in effect at MIS Quarterly (and also some other
leading IS academic journals) which have established the
expectation that IS research studies, including statisti-
cally conducted ones, should identify one or more non-
academic stakeholders and provide some implications for
them (e.g., see Straub and Ang, 2008, 2011; Te’eni et al.
2015). These implications can be presented in separate
paper section(s), or can be part of a general discussion
12
For an exhaustive record on effect size measures see Kelley and Preacher
(2012).
section or implications section. Based on curiosity, we
reviewed the MIS Quarterly papers to find out the per-
centage of papers which devote separate paper section(s)
to implications for nonacademic stakeholders. The result
of our review indicates that 65% (17 out of 26) of articles
contain separate paper sections covering the nonaca-
demic implications.
5. Does the paper indicate how its dependent variables are
relevant in the eyes of the identified nonacademic stake-
holders? We reviewed the MIS Quarterly papers to find
out the percentage of papers that offer content (e.g.,
statements from interviews with nonacademic stake-
holders) on how nonacademic stakeholders find the
papers’ dependent variables (as operationally defined
measured) relevant. Consistent with the conceptualiza-
tion offered in the section with the Venn diagram illus-
tration, we consider the relevance of a dependent variable
to be concerned with the understandability and note-
worthiness of the variable in the eyes of nonacademic
research stakeholders. As seen in Table 5, despite the
fact that almost all of the papers target at least one non-
academic stakeholder group and the majority of them
devote separate paper sections to implications for non-
academic stakeholders, our review shows that 92% (24
out of 26) of papers do not indicate how their dependent
variables are relevant in the eyes of the identified non-
academic stakeholders.
6. Does the paper indicate how its independent (including
moderating) and mediating variables are relevant in the
eyes of the identified nonacademic stakeholders? We
included this aspect in order to find out the percentage of
MIS Quarterly papers that offer content (e.g., statements
from interviews with nonacademic stakeholders) on how
nonacademic stakeholders find the papers’ independent
(including moderating) and mediating variables (as
operationally defined/measured) relevant. Consistent
with the conceptualization offered in the section with the
Venn diagram illustration, we consider the relevance of
an independent/mediating variable to be concerned with
the understandability, and when appropriate, action-
ability, of the variable in the eyes of nonacademic stake-
holders. Actionability, as mentioned, was a contingent
criterion in our review of the MIS Quarterly papers due
to the possibility of invoking a “general” relevance, as
opposed to an “action lever” relevance (see the section
with the Venn diagram). Table 5 shows that, despite the
fact that almost all of the papers target at least one non-
academic stakeholder group and the majority of them
devote separate paper sections to implications for non-
academic stakeholders, 81% (21 out of 26) of papers do
not indicate how their independent/mediating variables
MIS Quarterly Vol. 44, No. 2/June 2020 535
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
are relevant in the eyes of the identified nonacademic
stakeholders.
7. Does the paper indicate how the magnitude of the RES
would make a difference in the eyes of the identified non-
academic stakeholders? Aspect 7 was another review
aspect we employed to examine the consideration of the
issue of relevance in the MIS Quarterly papers. What
motivated us to include this aspect was the recognition,
as explained in all three preceding sections of this essay,
that nonacademic stakeholders may have different judg-
ments compared to researchers on whether the magni-
tudes of effect sizes reported in a research study make
any difference in real-world terms. Nonacademic stake-
holders’ judgments can be different in part because they
might consider other criteria, which may not be of
interest to researchers. For instance, nonacademic stake-
holders might apply a financial cost-benefit criterion,
which may often have no place in the researchers’ inter-
pretation of the practical significance of effect size
measurements. Our review of the MIS Quarterly papers
suggests that, despite the fact that almost all of the papers
target at least one nonacademic stakeholder group and
the majority of them devote separate paper sections to
implications for nonacademic stakeholders, 25 out of 26
papers (i.e., 96%) do not provide any content on the rele-
vance of the magnitude of their reported effect sizes.
Only one paper provides such content, albeit only for one
of the effect size measurements that the paper reports.
What is presented so far is concerned with the overall profile
of the MIS Quarterly papers in terms of the consideration of
the issues of practical significance and relevance. A more in-
depth view however is needed to reveal the nuances of this
kind of consideration in the published IS research. For that
purpose, we selected the three papers shown with a gray back-
ground in Table 5. This essay, however, is not intended to be,
and should not be taken as, a negative criticism of the three
excellent articles. In this essay, we are synthesizing ideas that
were not synthesized at the publication time of those papers;
therefore, it would not be reasonable to hold the authors of the
three articles to the standards or criteria implied in a frame-
work that was not even proposed at the time of their research.
Rather, we regard these papers as providing a solid foundation
upon which to provide more details on the underlying logic
we used to review the 27 MIS Quarterly papers, as well as to
highlight some critical observations we have made.
The Case of Park, Sharman, and Rao
The study by Park, Sharman, and Rao (2015) investigates the
effect of an individual’s “perceptions of sociotechnical safety
536 MIS Quarterly Vol. 44 No. 2/June 2020
factors” on the individual’s “perceived usefulness of hospital
information systems (HIS).” The authors also posit that the
aforementioned effect is moderated by the “individual’s
disaster experience” (i.e., whether a person has had an experi-
ence of natural or human-made disasters in the past when
working with an HIS). The individual’s “perceptions of
sociotechnical safety factors” in fact is an umbrella term for
the three distinct independent variables that the study
incorporates “perceived system risk,” “perceived resilience”
and “perceived information assurance.”
Park et al. administer three surveys: a survey distributed
among hospital personnel who have had an experience of a
snowstorm in the past when working with an HIS (i.e., the
experience group), a second survey distributed among the
same group of people but after showing them a message that
stimulated the personnel to recall the snowstorm they had
experienced (i.e., the recall group), and finally, a survey
among hospital personnel who had not experienced the
snowstorm (i.e., the no-experience group). Using the survey
data, Park et al. report many different types of effect sizes (as
shown in Table 5). Primary among these effect sizes are path
coefficients estimated using PLS, R2 measurements, and
varying types of statistical fit measurements.
We will not summarize Park et al.’s entire analysis here, but
instead focus on some particular observations we have made
in relation to the consideration of the issues of practical
significance and relevance.
It is clear that Park et al. offer a comprehensive and appro-
priately conducted statistical analysis. Nevertheless, we argue
that the authors could have made a much greater contribution
to their research area of interest by addressing the practical
significance of the results they have obtained. We found that,
beyond simply reporting numerous effect sizes, the authors
give no additional consideration to the issue of practical
significance. This is particularly evident when Park et al.
discuss their obtained path coefficients. The authors provide
statements such as “systems risk had negative effects on
perceived usefulness (â = -0.240, p < 0.001 and -0.160, p <
0.01)” (p. 331), or “the effect of perceived information
assurance (H2) was also significant for the perceived
resilience (â = 0.249, p < 0.001 and 0.318, p < 0.001) in both
the experience and recall groups” (p. 331). In other words,
the authors merely draw attention to the sign and statistical
significance of the path coefficients, without providing any
interpretation of their own regarding the magnitude of those
coefficients. This of course does not necessarily mean that
the path coefficients obtained by Park et al. are not of prac-
tical significance. The point we make here only is that the
authors, in their writing, could have addressed the issue of
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
practical significance as well.13 Moreover, we notice that
Park et al. (2015) in many places, including the second state-
ment quoted above, use the term “significant” in an ambig-
uous way, insofar as the term can connote an effect with
considerable magnitude, whereas what the authors actually
mean is a statistically significant effect.
With respect to the issue of relevance, our review suggests
that Park et al. target a highly-regarded nonacademic stake-
holder group (i.e., hospital managers). They well elaborate on
the “practical implications” of their study for hospital
managers in more than a full page. In the “practical impli-
cations” section, the authors legitimately argue that the
findings of their study “suggest that hospitals seeking to
increase their successful HIS implementation may pursue” (p.
338) some strategies such as focusing “on managing em-
ployees’ attitudes toward and perceptions of risk and
resilience in a normal context” (p. 338), or adopting “one
process to enhance resilience, and a separate process to
reduce perceived system risk” (p. 338). Nevertheless, the
authors could have provided a much stronger basis for their
recommendations by indicating how the independent vari-
ables (i.e., “perceived system risk,” “perceived resilience,”
and “perceived information assurance”), upon which hospital
managers are encouraged to act, are understandable and
actionable in their eyes in the first place. More importantly,
the authors could have indicated the empirical basis on which
they presume that their dependent variable of interest (i.e., the
research construct, “perceived usefulness of HIS”) is a matter
of interest or importance to, or even understandable by,
hospital managers. On a side note, we argue that the rele-
vance of the study by Park et al. would have been better
established if the authors could have shown how, and at what
level, the magnitude of the effect sizes they obtained can
actually support assertions such as the following ones and
make them important in the eyes of hospital managers. On
page 338, the authors state that:
The findings of our studies suggest that employees’
perceptions of risks and information assurance are
important not only for maintaining resilience and
effective operation of HIS, but also for facilitating
employees’ HIS related work.
Or, on the same page, the authors argue that:
13
In the “Discussion and Recommendations” section, we will demonstrate
how Park et al. could have done a better job communicating the practical
significance of their research findings.
As the results of the present study show, perception
of the strength of information assurance increases
belief in resilience.
The above quotations are examples highlighting the fact that
there was an opportunity for Park et al. to make references to
the magnitude of their obtained effect sizes in order to
precisely specify why the factors they are concerned about are
actually important in the practice of managing hospitals, or
why levels of increase/decrease in such factors can actually
make a difference in the eyes of hospital managers. Again,
our intention here is not to imply that the study by Park et al.
is not relevant. The point is that, in order for the nonacademic
stakeholders (i.e., hospital managers) to acknowledge the
novelty and importance of Park et al.’s paper, the authors
could have considered and discussed the relevance of their
work in greater depth and specificity.14
The above observations regarding the issues of practical
significance and relevance, however, are not unique to the
Park et al.’s paper. We have made many similar observations
of other MIS Quarterly papers also appearing in Table 5.
The Case of Armstrong, Brooks,
and Riemenschneider
The second article we examine is by Armstrong, Brooks, and
Riemenschneider (2015). In their article, they draw attention
to what they call an individual’s “exhaustion from IS career
experience (EISCE)” and investigate its effect on what is
called the individual’s “intention to turn away from an IS
career” (i.e., turn-away intention or TAI). Furthermore, Arm-
strong et al. incorporate in their study four other independent
variables as antecedents to EISCE. The variables are
“perceived workload” (PW), “career–family conflict” (CFC),
“fairness” (FAIR), and “control of career” (CTRL). The
authors also posit that the relationship between EISCE and
TAI can be mediated by another variable called “affective
commitment to the IS profession” (ACISP).
In our view, Armstrong et al. present a very well-conducted
study. As shown in Table 5, the authors estimate and report
many types of effect sizes. The case of Armstrong et al.
shows considerable similarity to the case of Park et al., where
the authors merely draw attention to the sign and/or statistical
significance of their obtained effect sizes. The case of Arm-
strong et al. is, however, useful for highlighting another prac-
14
Again, in the “Discussion and Recommendations” section, we will
demonstrate how Park et al. could have done a better job communicating the
relevance of their research.
MIS Quarterly Vol. 44, No. 2/June 2020 537
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
tice we found common in our reviewed MIS Quarterly papers.
The practice is to offer interpretations, without solid sup-
porting rationales, of the magnitude of obtained effect sizes.
In our review of the MIS Quarterly papers, we did not count
such interpretations as adequate discussions of practical
significance. For instance, Armstrong et al. offer an interpre-
tation of the magnitude of one R2 measurement, which
corresponds to the TAI variable and has the magnitude of
0.555. Referring to this R2 measurement, Armstrong et al.
state that “the model had reasonable explanatory power
accounting for just over half of the variance in TAI” (p. 720).
Strictly speaking, this statement constitutes only a partial
treatment of the issue of practical significance with respect to
the concerned R2 measurement. Although Armstrong et al.
offer a judgment on the magnitude of the R2 measurement by
using the term “reasonable,” which may connote “medium in
size,” the authors do not provide any supporting rationale on
why 0.555 might indicate a “reasonable explanatory power”
within the authors’ research domain/context.
With regard to the issue of relevance, we notice that, different
from Park et al., Armstrong et al. do not devote a separate
paper section to implications for nonacademic stakeholders.
However, we believe Armstrong et al. provide a better treat-
ment of the issue of relevance. The reason is that the authors
are concerned about the relevance of their variables of
interest, at least in the eyes of one of the nonacademic stake-
holder groups they target, namely, IS professionals.
Armstrong et al. report on focus group interviews with 29 IS
professionals, conducted with the aim of determining the
applicability of their research model to the particular context
of an IS career. Armstrong et al. point out (p. 715):
Based on the comments from the focus group
interviews, we realized the importance of con-
sidering an individual’s career instead of a specific
job when looking at TAI. Individuals in the focus
groups highlighted the importance of exhaustion and
turn-away for IS professionals and identified several
of Ahuja et al.’s [2007] constructs [the main litera-
ture’s constructs on which Armstrong et al. draw] as
relevant across their ISCE [IS career experience].
The exploratory focus group sessions revealed the
need to study the concepts and helped identify key
constructs that were then complemented with our
own understanding of the literature.
Nevertheless, we believe the authors could have avoided the
same pitfall, as we encountered in Park et al.’s paper, when it
comes to the discussion of implications for nonacademic
stakeholders. Precisely speaking, Armstrong et al. could have
drawn attention to the magnitudes of obtained effect sizes
when making assertions with respect to the nonacademic
538 MIS Quarterly Vol. 44 No. 2/June 2020
implications of their research. In the “Discussion and Recom-
mendations” section, we demonstrate how this could have
been achieved in Armstrong et al.’s paper.
The Case of Liu, Brass, Lu, and Chen
The study by Liu, Brass, Lu, and Chen (2015) examines the
effect of online/offline friendship relationships on lending
decisions in a large peer-to-peer (P2P) lending website. The
authors’ main argument is that whether or not to offer a loan
on P2P lending websites “is affected by the friendship
between the potential lender and the borrower (the pipe
effect), by a bid from the borrower’s friend (the prism effect),
and by a bid from the potential lender’s friend (the relational
herding effect)” (p. 730). The study by Liu et al. is distinct
from the previous two cases as it belongs to the growing body
of data analytics research in the IS field. The authors ob-
tained a large dataset from PPDai, one of the largest P2P
lending platforms in China. The dataset includes all the bid-
ding activities of more than 2,100 lenders, 7,800 borrowers,
and 12,500 listings, and incorporates a total of 2,546,799
lending decisions.
Liu et al. present a solid statistical analysis. They estimate a
total of eight different conditional logit models and conduct
Wald tests, using nearly 50% of their obtained dataset (N =
1,250,426 lending decisions). As shown in Table 5, the
authors report various types of effect sizes including odds
ratio coefficients, log-likelihood, Pseudo R2, Wald ÷2,
Adjusted R2, and correlation coefficients. Unsurprisingly
(considering the volume of data), almost all of the statistical
results reported by the authors are statistically significant.
However, the audience of data analytics studies (like the one
conducted by Liu et al.) often look for qualities far beyond
mere statistical significance. Data analytics studies are gener-
ally expected to leverage large amounts of data in order to
“detect smaller, subtler, and more complex effects” (Lin et al.
2013, p. 906), as well as generate novel understandings of
real-world phenomena (see Lycett 2013; Sharma et al. 2014)
on a broad empirical landscape, which is usually not available
to the classic, survey-based quantitative studies. On this
basis, the proper communication of results in data analytics
research demands that researchers supply, among other
material, sound interpretations of the magnitudes of estimated
effects as well as justified explanations on the importance of
those magnitudes.
Contrary to our expectation from a data analytics study, our
review of Liu et al. shows that the research communication
language adopted by the authors predominantly relies on the
issue of statistical significance. This is in fact similar to what
we observed previously in the two rather classic cases of
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
statistically conducted IS research (i.e., the papers by Park et
al. and Armstrong et al.). Liu et al. discuss their obtained
odds ratio coefficients (reported in the paper’s Table 3), for
the most part, based on the two criteria of statistical signifi-
cance and whether the coefficients are greater or less than 1.
The authors, for instance, make the following inferences (p.
739):
As shown in Table 3, the lending probability in-
creases with the number of days passed, the percent-
age funded, the number of prior bids, and co-
location of the borrower and the lender in the same
city. The lending probability decreases with 100
percent funding status, gender (female), education,
and the number of friends the lender has.…The
number of large bids has a [statistically] significant
negative effect on subsequent lending probabilities.
What the authors conclude based on their reported results is
statistically appropriate. However, we argue that the paper
could have yielded a much greater contribution if, instead of
using such terms as “increase,” “decrease,” or “[statistically]
significant negative effect” and so forth, the authors could
have explicitly mentioned the magnitude of the effects they
estimated and offered their own judgments (along with
corresponding rationale) on those magnitudes.
The authors also provide three relatively short paragraphs (on
page 739) to address the results associated with the three
hypothesized effects in the study (i.e., the pipe effect, the
prism effect, and the relational herding effect). A constructive
practice the authors pursue here though is that they convert
the obtained odds ratio coefficients (reported in the paper’s
Table 3) to percentages. For example, the authors point out
(p. 739):
Each prior bid increases the likelihood of lending by
2.1 percent (Model 3), suggesting an anonymous
herding effect among potential lenders (H4 is
supported).
To mention the magnitude of statistical results in the form of
percentages is certainly helpful because it makes the results
more comprehensible and transparent in the eyes of the audi-
ence of the research. However, the authors could have
contributed to greater appreciation of the statistical results by
the audience if they could have offered, in addition, an assess-
ment (with supporting rationale) of the magnitude of the
estimated effect sizes such as “2.1 percent” in the research
area concerning P2P lending platforms. As indicated before,
to address the magnitude of estimated effect sizes (and hence
to address practical significance) is particularly important in
the case of data analytics studies due to the possibility of the
existence of the deflated p-value problem.
With respect to the relevance issue, the paper correctly
identifies P2P lending platforms and the borrowers who use
them as major groups of a nonacademic audience. Moreover,
the authors raise interesting implications and “suggest that …
[P2P platforms] can increase bidding activities by making it
easy for lenders to follow their offline friends,” or state that
“P2P lending platforms may prioritize loan and friend-bid
notifications by friendship types” (p. 740). However, the
paper’s recommendations could have had more impact if the
authors could have shown that the variables they studied are
in fact understandable, noteworthy and/or actionable in the
eyes of the nonacademic stakeholders they identify for their
paper. The authors could have demonstrated the relevance of
the study by providing quotations or other evidence from
practitioners to report their reactions to the employed depen-
dent and independent variables. Additionally, as just recom-
mended for the other two papers, in order to present a much
more objective and convincing case as to why Liu et al.’s
study might be important for P2P lending platforms and the
borrowers who use them, the authors could have made
reference to, and discussed, the significance of the magnitude
of the results they obtained. Indeed, it is particularly impor-
tant for data analytics studies (due to the highly sophisticated
analytics models and algorithms they often employ) to discuss
their findings in a way that is conveniently interpretable for
the nonacademic, nonexpert audience. As a matter of fact, the
interpretability of research results has regularly been brought
up as a critical aspect of communicating data analytics
research in a number of recent commentaries in the IS litera-
ture (e.g., see Abbasi et al. 2016; Lycett 2013; Müller et al.
2016; Sharma et al. 2014).
Additional Cases
Apparently, what we have observed in the case of papers
published in MIS Quarterly in 2015 is not exceptional at all.
In other words, the lack of attention to the communication of
practical significance and relevance in quantitative IS
research seems to be an issue spanning a range of different
outlets and a series of years. The support for this contention
follows.
In addition to our review of 2015 MIS Quarterly papers, we
did an extra investigation of published IS research. We ran-
domly selected papers published in 2016, two each from the
journals of MIS Quarterly, Information Systems Research,
and Journal of the Association for Information Systems.15
These randomly selected papers present outstanding statistical
15
The randomly selected papers are Benaroch et al. (2016); Greenstein and
Zhu (2016); Goh et al. (2016); Tripp et al. (2016); Venkatesh et al. (2016);
and Wang et al. (2016).
MIS Quarterly Vol. 44, No. 2/June 2020 539
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
analyses. However, our review suggests that there still are
major overlooked opportunities for the papers to address prac-
tical significance and relevance. Overall, despite the fact that
a large number of various effect sizes such as regression
coefficients, correlation coefficients, R2, adjusted R2, AIC,
and BIC are reported in the papers, the practical significance
of research results receives partial, minimal, or no attention.
In our view, out of the six papers, only one (i.e., Greenstein
and Zhu 2016) offers a partial treatment of the issue of the
practical significance of research results. The rest of the
papers, as with the 2015 MIS Quarterly papers, primarily rely
on statistical significance and the sign of obtained effect sizes
for the matter of discussing research results. On the other
hand, as for the issue of relevance, we found that all of the
studies identify at least one nonacademic stakeholder group
as their audience, and each offers statements on how the
identified nonacademic stakeholder group(s) may benefit from
the research study. Still, our review suggests that none of the
papers draw attention in a meaningful way to the relevance of
the employed variables and the magnitude of reported results
in the eyes of the identified nonacademic stakeholder groups.
As a matter of fact, the findings from the above review of
published IS research (i.e., the 2015 MIS Quarterly papers as
well as the six additional papers), particularly with respect to
the issue of practical significance, are in accordance with the
indications made in the only relevant review of published IS
research we found in the literature (i.e., Lin et al. 2013). Lin
et al. (2013) conduct a survey of large-sample papers (n >
10,000) published in leading IS journals and conferences from
2004 to 2010. Based on the survey, the “overall conclusion
is that information systems research that is based on large
samples might be over-relying on p-values to interpret
findings” (p. 908). Lin et al. also indicate that “about half of
the recent papers with large samples rely almost exclusively
on low p-values and the sign of the coefficient” (p. 908).
Furthermore, as shown by Lin et al., only one out of four
reviewed MIS Quarterly papers, and only two out of eight
reviewed Information Systems Research papers, offer conclu-
sions that rely on practical significance.
Discussion and Recommendations
There have been three interrelated themes in this essay. The
first theme pertains to the position that there are unques-
tionably distinctions among statistical significance, practical
significance and relevance. These distinctions proved to be
completely recognizable and valid both when one examines
the literature concerning the notions of significance and
relevance (as shown in our review of the background of these
notions), as well as when one ponders some hypothetical
examples/scenarios where research may bear any one, any
540 MIS Quarterly Vol. 44 No. 2/June 2020
two, or all three of the qualities of statistical significance,
practical significance and relevance (as shown in the section
with the Venn diagram). The second theme relates to the
position, which we developed in the preceding section, where
we argued that it is not sufficient for research to bear statis-
tical significance, practical significance and relevance, but it
is also critical for research to address another issue of a
different nature, namely, to explicitly communicate the status
of its statistical significance, practical significance and rele-
vance. Finally, the third theme, regarding our findings from
reviewing published IS research, is that the issues of practical
significance and relevance have been considered, and reported
on, far less than the issue of statistical significance.
The third theme concerns us the most in that it shows, in our
view, the need for raising awareness about some critical
issues concerning statistical practice in IS research. In the
following paragraphs, and in line with our purpose in the
current essay, we will enumerate three issues we believe are
of high importance followed by our opinions on how future IS
research can deal with the issues and communicate their
significance and relevance in a much more adequate manner.
Issue 1: Communicating rigor transcends merely reporting
statistical significance and also requires addressing practical
significance.
The predominant exclusive reliance on statistical significance,
as our review of high quality published IS research suggests,
can imply that there is still a widespread notion-in-use in our
community presuming that reporting statistical significance
alone is sufficient to establish and communicate research
rigor. However, reviewing the literature and history of statis-
tics indicates otherwise. Statistical significance and null
hypothesis significance testing have been criticized for many
decades and from several different angles as misunderstood
and misplaced concepts incapable of satisfying the require-
ments of scientific practice in general (e.g., see Kirk 1996;
Nickerson 2000, 2011).
Our opinion is that a shift in perspective should occur with
regard to how IS researchers perceive and practice quanti-
tative research. The long history of controversy around the
notion of statistical significance, and the respective remedies
offered in the literature, offer a clear message for IS
researchers. The message is that if IS researchers want to
pursue rigorous quantitative research that matters to their own
self-chosen questions of interest, whether the research is of a
classic statistical type or data analytics type, they should view
practical significance not as an optional notion to be taken
into account, or as an antithesis to statistical significance, but
as a necessary and complementary quality for their research
to address. Practical significance is as equally important a
measure as statistical significance when it comes to rigorous
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
quantitative research. Thus, addressing practical significance
can be regarded a major step toward broadening the notion-in-
use of “rigor” in statistically/computationally conducted IS
research. In other words, we posit that rigor should be per-
ceived as composed of at least the two equally important
measures of statistical significance and practical significance
among those who practice statistically/computationally con-
ducted IS research.
Nonetheless, our position by no means should be interpreted
as a call for obtaining the largest possible statistical results
with regard to practical significance. We once again empha-
size the fact that the matter of practical significance, and the
matters of statistical significance and relevance, refer to
qualities heavily dependent on human judgment, research
context and conventions. What we do want to promote
through this essay however is the practice of communicating
practical significance, however large or small the obtained
research results are.
Issue 2: Communicating practical significance transcends
merely presenting effect magnitudes and also requires their
interpretation in context.
We acknowledge that many researchers in IS do recognize the
importance of the notion of practical significance and its role
in conducting sound statistical analysis. However, there
seems to be a prevailing issue about how one can, or should,
communicate the practical significance of research results.
With the advent of statistical/computational software pack-
ages that provide advanced functionalities, researchers
nowadays, particularly in the data analytics domain, can
conveniently and effortlessly generate and report many
different types of effect sizes. A dubious belief, however, is
that as long as numbers known as “effect size” are reported in
the text or tables of a research paper, the task of commu-
nicating the practical significance of research results is
accomplished.
Scientific research goes much beyond simply reporting
numbers. Researchers are also expected to deliver interpre-
tations and insights on the basis of quantitative research
results. This is an essential task given the fact that one’s
interpretations of the magnitude of the same effect size
measure can dramatically change from one research study to
another depending on the particular circumstances under
which the research is conducted. Therefore, to communicate
the practical significance of research results is more a matter
of offering interpretations about the magnitude of research
results, rather than merely reporting them. The proper com-
munication of the practical significance of findings is even of
much higher importance in data analytics research. As stated
earlier, whereas large datasets common in data analytics
research can comfortably turn findings statistically significant
(i.e., the deflated p-value problem), the practical significance
of the findings, as a distinct matter from their statistical
significance, cannot be taken for granted and certainly
demands an explicit and justifiable treatment (see George et
al. 2014; Lin et al. 2013).
Recommendations on the Proper Commu-
nication of Practical Significance
In general, we recommend that IS researchers offer clear
interpretations about the magnitude of research results and
bring those interpretations to the fore, making them explicit
in their research articles. This in our view is essential to make
sure that the findings and conclusions drawn from our
quantitative IS scholarship are transparently interpretable. In
practice, we believe that the proper communication of the
practical significance of results (particularly, those that are
directly associated with research hypotheses) should incor-
porate the following four elements: effect magnitudes, confi-
dence intervals, interpretations of magnitude, and rationales
supporting the interpretations of magnitude.
(I) Effect Magnitudes: The proper communication of practi-
cal significance begins with an explicit presentation of effect
size measurements. We particularly emphasize this for all
effect sizes (e.g., path coefficients) directly related to research
hypotheses. The measurements can be presented in a variety
of different ways such as in the text, in tables, or on charts
graphs. The important point is that the quantitative magnitude
of effect sizes should be stated clearly. Authors should avoid
using such vague terms as “positive impact” or “non-zero
effect.” Our review of published IS research suggests that all
of the papers pursue our recommendation and provide quan-
titative magnitudes of all effect sizes they deal with.
(II) Confidence Intervals: Whenever the data and statistical
analysis warrant the calculation of confidence intervals, we
emphasize the need for IS researchers to present confidence
intervals in their papers. In fact, within the context of com-
municating practical significance, the reporting of confidence
intervals “has been advocated as an alternative, or adjunct, to
NHST [null hypothesis significance testing] by many writers,
including both critics and proponents of NHST” (Nickerson
2000, p. 278). A confidence interval delivers all the same
information provided by NHST (Kirk 1996, 2001). A confi-
dence interval (surrounding the point estimate of an effect
size) that includes 0 (where 0 is the point corresponding to the
null hypothesis) suggests that the null hypothesis cannot be
rejected. In other words, “the smaller the confidence interval
that includes 0, the stronger the evidence for the null
hypothesis” (Nickerson 2000, p. 261). However, a confidence
interval provides more important information than what is
afforded by NHST. A confidence interval shows “a range of
MIS Quarterly Vol. 44, No. 2/June 2020 541
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
values within which the effect parameter is likely to lie” (Kirk
2001, p. 214), with the same unit of measurement as the data
(Kirk 1996, 2001). This means that calculating and reporting
confidence intervals can significantly boost researchers’
ability to offer meaningful interpretations of magnitude (see
element III). On the other hand, the practice of reporting
confidence intervals yields a means to develop a common
thread through which researchers in a community become
able to conduct purposeful synthesis and meta-analysis of
findings across different studies that investigate certain
variables/phenomena of interest (Cumming and Finch 2001;
Lin et al. 2013). Additionally, the frequent reporting of
confidence intervals in a research community allows re-
searchers to compare the magnitude, and the lower and upper
bounds, of their own estimated effect sizes with the ones
reported in similar studies. This therefore can aptly pave the
way to establish the required rationale(s) for the inter-
pretations of magnitude (see element IV).
The move toward reporting confidence intervals is quite wide-
spread across research domains and communities. We for
instance know that “medicine has moved away from sole
reliance on NHSTs” and almost “all medical studies now state
confidence intervals” (Schwab et al. 2011, p. 1111). In addi-
tion, a number of other disciplines have already announced
the mandate to report confidence intervals in research studies
(e.g., see American Educational Research Association 2006;
American Psychological Association, 2010). In the IS field,
we found some examples of published research where con-
fidence intervals have been reported. For instance, Ludwig et
al. (2014) present a research study on user communities where
two mixed multilevel Poisson regression models have been
utilized to conduct text mining on users’ posts in 37 com-
munities. The authors show the 95% confidence intervals for
an estimated effect size called incidence rate ratio (IRR). The
confidence intervals are presented along with the actual
magnitudes for the IRR effect size in a table (see Ludwig et
al. 2014, Table 3). In another IS research study, Saboo et al.
(2016) investigate the issue of marketing resource allocation
using a computational model called the time-varying effect
model (TVEM). The authors use “transaction data from a
Fortune 500 retailer [along] with demographic information
obtained from Acxiom Corp for over a quarter million cus-
tomers” (p. 911) to develop and test their proposed model.
The authors present 95% confidence intervals for parameter
estimates of their independent variables of interest. These
confidence intervals along with the magnitude of estimated
parameters are presented in three tables (see Saboo et al.
2016, Tables 3, 4, and 6). According to the authors (p. 925),
the confidence intervals are in fact reported to display the
authors’ conservative approach in presenting and interpreting
research results in reaction to “the deflated p-value problem”
concerning data analytics studies.
542 MIS Quarterly Vol. 44 No. 2/June 2020
Different computational/statistical software packages such as
R, SPSS, STATA or even Excel are available to calculate
confidence intervals. As shown in the cases of Ludwig et al.
and Saboo et al., it appears a common practice to report 95%
confidence intervals, but it should be noted that this is as
arbitrary a practice as setting 0.05 or 0.01 thresholds for p-
values. Moreover, the manner in which researchers may
present confidence intervals in their papers depends on the
particular type of research they conduct and/or the readership
they target. The simplest way to present confidence intervals
is perhaps the way we observed in the cases of Ludwig et al.
and Saboo et al., where confidence intervals are presented
along with effect magnitudes in the text or in tables. How-
ever, there are other, more sophisticated, suggestions on how
to present and take advantage of confidence intervals in
research studies. We offer one example here. In the case of
large-sample studies, Lin et al. (2013) argue for the use of a
chart called the confidence interval chart. The chart “displays
the confidence interval as a function of the sample size,
ranging from very small to the maximal large-sample size
itself” (p. 910). According to Lin et al., the chart is useful to
show the effect magnitude and its decreasing standard error.16
(III) Interpretations of Magnitude: Whereas the two pre-
ceding elements of the proper communication of practical
significance are, so to speak, “hard” elements, conveniently
achievable nowadays with the help of statistical computer
programs, the current element and the next one (i.e., rationales
supporting the interpretations of magnitude) are “soft” ele-
ments, which are often challenging to deliver because they
highly depend on researchers’ depth of understanding con-
cerning both statistics as well as the research area they are
dealing with. That is perhaps why the latter two “soft”
elements are often missing in some published research. Not-
withstanding, our review of published IS research suggests
positive examples of presenting interpretations of magnitude.
For instance, Salge et al. (2015), in their study of “behavioral
and institutional search mechanisms underpinning hospitals’
IS investment decisions,” state the following regarding an
obtained regression coefficient (p. 76):
According to hypothesis 3, we expected the coef-
ficient of the lagged dependent variable IS invest-
ment intensity in model 3 to be between 0 and 1 and
statistically significant. In line with our theoretical
arguments for institutionalized search, our analyses
yield a [statistically] significant estimate of 0.536,
indicating moderately high levels of temporal persis-
tence in hospitals’ IS investments.
16
For more detailed considerations about calculating and reporting confi-
dence intervals see sources such as Cumming and Finch (2001) and
Nickerson (2000).
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
In other words, Salge et al. consider 0.536, on a scale of 0 to
1, to constitute a moderately high effect size value. However,
the academic audience of the paper might still argue that the
number does not represent a moderately high effect size
value. This all goes back to the presence, or lack thereof, of
convincing rationales supporting the interpretations of
magnitude—a crucial issue that will be discussed later under
element IV.
The study by Greenstein and Zhu (2016) represents another
example of published IS research where interpretations of the
magnitude of obtained effect sizes are proffered to some
extent. The authors’ investigation of “whether the open-con-
tent production model helps achieve a neutral point of view
(NPOV)” (p. 618) uses data from Wikipedia’s articles on U.S.
politics. In one particular case, the regression analysis run by
Greenstein and Zhu yields estimated coefficients associated
with the two independent variables of “unique contributors”
and “total revisions to date” and a dependent variable, which
is the probability of Wikipedia articles having certain “code
words” related to U.S. politics. The authors interpret the
results this way (p. 630):
We find that having more unique contributors
reduces the probability of having code words, while
having more revisions works in the opposite direc-
tion, increasing the probability of having code
words. However, the value of the coefficients sug-
gests a small overall effect: a 10% increase in the
number of unique contributors reduces the chance of
having code words by 0.2%, and a 10% increase in
the number of revisions increases the chance by
0.1%.
It is worth noting that although Greenstein and Zhu offer their
own interpretation (i.e., small overall effect) of the magnitude
of the two coefficients, the exact basis on which the authors
offered such an interpretation may still remain an open ques-
tion for some segments of the paper’s audience. As stated
earlier, we discuss the matter of rationales supporting the
interpretations of magnitude under element IV.
The third example we draw here is Vance et al. (2015). The
authors propose a new solution to address the issue of com-
puter systems access-policy violations. Essentially, Vance et
al. offer “four user-interface (UI) design artifacts that raise
users’ accountability perceptions within systems and in turn
decrease access-policy violations” (p. 345). In their statistical
analysis of the matter, the authors investigate the relationship
between an independent variable (i.e., Äaccountability) and
the main dependent variable (i.e., Intention to Violate the
Access Policy), among others. In a footnote, the authors
present their interpretation of the magnitude of the obtained
coefficient for Äaccountability (p. 357):
The size of the effect of Äaccountability can be
understood in terms of the scale of the dependent
variable, the range of the difference of the change in
accountability, and the parameter estimate (-.187) in
the model. For every unit increase in Äaccount-
ability, intention was reduced by -.187. Because the
values for the change in accountability had a range
of 32, intention could be reduced by up to -5.76
(-.187 C 32) [-5.98 is correct]. Because intention
was measured on a 0 to 10 scale, this represents a
very large effect size.
We should emphasize that, as far as element III alone is con-
cerned, the three IS studies examined above exemplify what
we consider adequacy in the practice of offering interpreta-
tions of magnitude. However, these studies do not necessarily
satisfy all the requirements we propose in this essay as for the
proper communication of practical significance. For instance,
only one of the three studies (i.e., Vance et al.), albeit par-
tially, reports confidence intervals.
With regard to advice to facilitate the task of offering inter-
pretations of magnitude, we wish to draw attention to two
particular tactics here. The first tactic deals with utilizing a
type of analysis known as “marginal analysis,” whenever
possible. Indeed, the preceding quote from Vance et al.
vividly demonstrates this type of analysis. As stated earlier
in this essay, through marginal analysis, the sensitivity of a
dependent variable to changes in an independent variable may
be measured and reported by researchers (see Lee and
Mohajeri 2012; Lin et al. 2013; Vittinghoff et al. 2005).
Indeed, in the case of nonlinear models,
which are quite common in IS research, marginal
analysis is a more robust way—and sometimes the
only way—to interpret effect size, compared to
looking at the p-value or magnitude of the coeffi-
cient (Lin et al. 2013, p. 909).
Marginal analysis can therefore be instrumental to not only
deliver an interpretation of magnitude (i.e., element III), but
also to articulate and present a sound rationale (i.e., element
IV) for such an interpretation.
The other recommended tactic is to calculate and report
unstandardized regression/path coefficients when possible. In
fact, there can be different reasons as to why interpretations
of the magnitude of obtained effect sizes are often missing in
published research. In the case of path coefficients in SEM,
we believe a major reason is that statistical software packages
MIS Quarterly Vol. 44, No. 2/June 2020 543
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
usually generate the coefficients in standardized form, which
hardly lends itself to being meaningfully interpreted. Kirk
and other statisticians suggest a general remedy, which is to
“use the same unit of measurement as the data” (1996, p.
754) when calculating research results in order to facilitate the
task of interpreting them. Accordingly, we recommend that
IS researchers consider calculating and reporting unstan-
dardized path/regression coefficients, particularly when the
units of measurement of the data can be familiar to the outside
world (e.g., Likert scale,17 financial indicators/ratios, stock
market indices, etc.). In SEM, this can be achieved by esti-
mating path coefficients using the covariance matrix, rather
than the correlation matrix (Kelley and Preacher 2012, p.
144). Also, in regression tables, which are often reported in
papers, separate columns can be used to communicate both
standardized and unstandardized coefficients. We know that
“this is the way in which SPSS reports the output for linear
regression analysis, whereas with SAS the option STB in
PROC REG yields the same result” (Kelley and Preacher
2012, p. 148). Even if standardized path/regression coeffi-
cients are already obtained, there is a fairly straightforward
formula to transform them to unstandardized ones when one
conducts multiple regression or PLS analysis—which is
“conceptually and practically…similar to using multiple
regression analysis” (Hair et al. 2011, p. 140). The formula
can be demonstrated in the following way:
“An unstandardized effect size is one whose interpretation is
dependent on the units of the measurement instrument”
(Kelley and Preacher 2012, p. 144). With unstandardized
coefficients, authors are often in a better position to commu-
nicate the substantive importance of their obtained results to
both the reviewers/editors and the readership. Employing the
unstandardized magnitudes, authors can, for instance, conduct
a marginal analysis and show the actual change in a depen-
dent variable along the same scale as the collected data when
one unit of change happens for any independent variable of
interest, again along the same scale as the collected data. This
often looks more sensible in the eyes of the reviewers, and
particularly the readership, compared to an analysis that
draws on standardized coefficients, where the entire narrative
concerning the amount of change in the variables has to be
stated in units of standard deviations.
(IV) Rationales Supporting the Interpretations of Magnitude:
To offer interpretations of magnitude (element III) is a neces-
sary, but insufficient, condition in order to ensure the proper
17
For a case of calculating and analyzing unstandardized coefficients drawn
from a Likert-scale-based TAM research study, see Lee and Mohajeri (2012).
544 MIS Quarterly Vol. 44 No. 2/June 2020
communication of practical significance. Authors should also
clearly disclose the rationales with which they make certain
interpretations about the magnitude of their obtained effect
sizes. Nevertheless, we found that the IS articles pointing to
the issue of practical significance—in particular, those which
promote the practice of addressing practical significance in
data analytics research (e.g., Abbasi et al. 2016; Lin et al.
2013; Müller et al. 2016)—draw little or no attention to this
critical aspect. Essentially, the question is: How may
researchers establish sound rationales for their interpretations
of magnitude? As stated before, the classic statistics literature
has long suggested some conventional rationales to support
the interpretation of practical significance (i.e., to distinguish
among small, medium and large magnitudes) in the cases of
certain types of effect sizes such as Cohen’s ƒ2 or Cohen’s d
(Cohen 1992; Kirk 1996). However, to draw on such conven-
tions has been severely criticized, particularly over the past
decade (e.g., Aguinis et al. 2010, pp. 529-530; Cortina and
Landis 2009; Ellis, 2010). The reason is that any assessment
of the magnitude of obtained effect sizes in a research study—
even for some well-known effect size measures such as R2,
Cohen’s ƒ2 or Cohen’s d—must take into account the par-
ticular contextual circumstances under which the research
study has been conducted. As stated earlier in this essay,
depending on the research context, we can certainly envisage
situations where a small effect magnitude can in fact denote
a high substantive importance and vice versa (Lewandowsky
and Maybery 1998; Prentice and Miller 1992; Rosenthal
1990; Rosenthal and Rubin 1979).
In our review of published IS research, we found examples
displaying varying degrees of adherence to the, so to speak,
“principle of interpretation in context” described in the
preceding paragraph. For instance, Tian and Xu (2015), in
their interpretation of obtained magnitudes for the Cohen’s ƒ2
effect size, state the following (p. 51):
By convention, effect sizes of 0.02, 0.15, and 0.35
are termed small, medium, and large, respectively
(Cohen 1988). We chose R&D spending, which has
been generally considered to be a major risk factor,
as a benchmark (e.g., Kothari et al. 2002). Based on
the regression results reported in Column (3) of
Table 4, the effect size of R&D spending and the
effect size of ERP systems (including its primary
effect and interactions with uncertainty) were esti-
mated to be 0.0385 and 0.0161, respectively. Based
on the regression results reported in Column (2) of
Table 5, R&D spending and ERP system scope
variables had effect sizes of 0.0344 and 0.0376,
respectively. Hence, regarding their effect on firm
risk, ERP systems and R&D spending had com-
parable effect sizes, between a small and medium
effect size (Cohen 1988).
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
Evidently, Tian and Xu offer the least adherence to the prin-
ciple of interpretation in context and state a rationale that is
completely drawn on the classic conventions communicated
in the statistics literature (i.e., Cohen 1988). In contrast,
drawing another example from published IS research, we
believe the marginal analysis quoted earlier (under element
III) from Vance et al. largely satisfies the principle of inter-
pretation in context. The authors show an adequate articu-
lation of their logic for why they consider the parameter
estimate of -0.187 for the Äaccountability variable to con-
stitute a very large effect size. Another interesting example
in the IS literature, where adherence to the principle of
interpretation in context is implied, is Hsieh et al. (2012). In
fact, Hsieh et al. employ very appropriate language to convey
the message that their research findings are of substantive
importance, although the magnitude of findings may be
deemed small (p. 1075):
Importantly, while the marginal contribution of
UserSat to ESQ is small in absolute terms (the
correlation between UserSat and ESQ = 0.19, see
Table 3; ÄR≤ = 4.2% for the direct effect and ÄR≤ =
3.5% for the interaction effect, see Table 4), method-
ologists have noted that small effects in the social
sciences must be evaluated in the context of the
significance of the practical problem and the theo-
retical gaps, the nature of the research design, and
the measures (Ellis 2010; Rosnow and Rosenthal
2003). In our context (1) the problem domain is of
significant practical value, (2) the research design
involves multisourced data with a time lag, (3) the
independent variables and/or dependent variables are
cumulative in nature, and (4) the relationships are
theoretically important. As such, the identified
effects, though small in size, are indeed of consider-
able theoretical and practical value.
The rationale offered by Hsieh et al. could have been pre-
sented with much more elaboration regarding the four aspects
indicated in the quote above, but does hint at an appropriate
practice through which researchers may establish a rationale
to support their interpretations of magnitude. The practice is
to draw attention to the literature in a certain research domain.
Ideally speaking, researchers may juxtapose the findings of
their own study with the ones reported by comparable
research studies in order to establish a sound rationale for
their judgment of practical significance. As stated before, a
pertinent facilitator in this type of practice is to draw on
confidence intervals. Assuming that confidence intervals
associated with a researcher’s variables of interest are already
reported by some comparable studies in the literature, the
researcher will be able to draw and show comparisons in
terms of not only the magnitude of estimations, but also the
uncertainty surrounding, or the precision of, the estimations
(Cumming and Finch 2001; Schwab et al. 2011).
Table 6 provides a summary of our recommendations as for
the proper communication of practical significance.18
To conclude this part on the proper communication of
practical significance, we now return to the three MIS
Quarterly papers (i.e., Armstrong et al. 2015; Liu et al. 2015;
Park et al. 2015) we reviewed in detail earlier. In Table 7, we
see elements that have been already communicated by the
authors as well as the ones that remained overlooked. In other
words, Table 7 demonstrates the proper communication of
practical significance with respect to one example of effect
size measurement reported in each paper.
The material summarized in Tables 6 and 7 are useful not
only to authors for recognizing the key tasks they need to
accomplish when communicating the practical significance of
their findings, but also to editors/reviewers for examining the
elements that need to be communicated by authors in their
submitted papers. As stated before, the burden of assessing
the practical significance of research findings is on the
academic/research community. Authors of course have a
major stake in showing that their findings are of practical
significance. However, since we posited earlier (under Issue
1) that addressing practical significance should be regarded a
main constituent of rigor in quantitative IS research, we also
see a prominent role for editors/reviewers to assess, and chal-
lenge, if necessary, the authors’ interpretations of practical
significance. We envisage the possibility that IS editors/re-
viewers may ask authors to report confidence intervals (as has
already been required in some other research communities),
to provide more precise interpretations of magnitude, or to
offer more robust rationales supporting their interpretations of
magnitude.
Issue 3: Communicating relevance transcends merely
including an “implications for practice” section and also
requires an explicit treatment of the relevance of research
constructs/variables and quantitative results from the per-
spective of the intended nonacademic stakeholders.
18
Our recommendations remain applicable to quantitative research that
employs other, sometimes, simpler, or more classic, types of statistical
designs/analyses such as experimental design (involving ANOVA), and
univariate, bivariate, cluster, or principal component, analyses. In all such
types of designs/analyses, as long as estimations associated with the measures
known as effect size (see Huberty 2002 or Kelley and Preacher 2012 for the
definition, history, and typology of effect sizes) are calculated/reported,
researchers need to address, and communicate, the practical significance of
those estimations.
MIS Quarterly Vol. 44, No. 2/June 2020 545
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance

Table 6. The Proper Communication of the Practical Significance of Research Results

Elements to Be Examples of Where to Concerns or Exemplar Papers Reviewed
Communicated Communicate Recommended Practices from Published IS Research
Effect In the text, in tables/figures, • Present the exact quantitative N/A
Magnitudes on charts magnitudes. (No paper was reviewed because
this element is already widely
communicated in published IS
research)
Confidence In the text, in tables/figures, • To report 95% confidence Ludwig et al. (2014)
Intervals on charts (e.g., the intervals is as arbitrary a Saboo et al. (2016)
confidence interval chart, practice as setting 0.05 or 0.01
see Lin et al. 2013) thresholds for p-values.
Interpretations of In the text, in tables/figures • Conduct and present marginal Salge et al. (2015)
Magnitude analysis. Greenstein and Zhu (2016)
• Calculate and present Vance et al. (2015)
unstandardized coefficients
(when appropriate).
Rationales In the text, in tables • Pursue the principle of Tian and Xu (2015)
Supporting the interpretation in context. Vance et al. (2015)
Interpretations of • Conduct and present marginal Hsieh et al. (2012)
Magnitude analysis.
• Draw comparisons with com-
parable findings reported in the
literature.*
*This is not of course an exhaustive list. Other practices may also be pursued as long as they are in accordance with the general principle of
interpretation in context.

Table 7. A Demonstration of the Communication of Practical Significance in the Cases of

Park et al., Armstrong et al., and Liu et al.
Reviewed MIS
Quarterly Example of Confidence Interpretation of Rationale Supporting the
Papers Effect Magnitude Interval Magnitude Interpretation of Magnitude
Park et al. â = -0.240 Not Not Provided Not Provided
(2015) Provided The authors could have The use of marginal analysis could
used marginal analysis to also lead to establishing the required
present an interpretation of rationale for the authors’ interpre-
the coefficient’s magnitude. tation of the coefficient’s magnitude.
Armstrong et al. R2 = 0.555 Not “The model had Not Provided
(2015) Provided reasonable explanatory The authors could have disclosed the
power accounting for just rationale with which they believe,
over half of the variance in within their particular research con-
TAI” (p. 720) text, the magnitude of 0.555 for the
R2 measure can indicate a
“reasonable explanatory power.”
Liu et al. (2015) “Each prior bid Not Not Provided Not Provided
increases the Provided The authors could have The authors could have established a
likelihood of provided an assessment of sound rationale for their assessment
lending by 2.1 the substantive importance of the substantive importance of 2.1
percent” (p. 739) of 2.1 percent in their percent by, for instance, drawing
respective research area. comparisons with findings reported in
comparable studies.

546 MIS Quarterly Vol. 44 No. 2/June 2020

 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
It is completely valid to say that not all IS research studies
want or are conducted to directly inform, or offer implications
to, nonacademics. Part of the reason is that nonacademics are
neither the primary readership of IS academic journals nor
often involved in the review process at those journals.
However, we believe, as long as these journals require authors
to address relevance to nonacademics (e.g., Straub and Ang
2008, 2011; Te’eni et al. 2015), there is a significant need to
raise awareness among IS researchers that simply mentioning
the name of one or more nonacademic stakeholder groups and
embedding an implications for practice section in a research
paper are not sufficient for the task of communicating
research relevance. In fact, the lesson from our review of the
published IS research is that communicating research
relevance is not only a matter of form, but also, and more
importantly, a matter of content. The content should be
offered in a way that could appropriately enable nonacademic
stakeholders/readers (if they happen to read IS academic
journals) to make judgments about the impact and importance
of a research study.19 In particular, in data analytics research,
it is of high importance for nonacademic decision makers to
be able to comprehend the results of applying sophisticated
models and algorithms to large datasets (see Müller et al.
2016, p. 294). “[Real-world] insights do not emerge auto-
matically out of mechanically applying analytical tools to
data” (Sharma et al. 2014, p. 435), in part, because “mana-
gerial decisions are typically based on economic return, rather
than a statistically significant analytical model, and that is
where analytics can generate mistrust and thus fall short”
(Baesens et al. 2016, p. 815).
Recommendations on the Proper Communi-
cation of Research Relevance
The proper communication of research relevance is largely
absent in published quantitative IS research. Perhaps a major
reason, as indicated before, is that there is no coherent
account in the literature particularly characterizing the notion
of relevance in the context of quantitative research. We, in
this essay, have offered a remedy. The characterization of
research relevance we presented in the Venn diagram section
(see Table 2), as well as in our review of the 27 MIS Quar-
terly papers published in 2015 (see Table 5), indicates three
major elements in the notion of relevance in quantitative
research: the relevance of dependent variables, the relevance
19
We completely embrace the idea of communicating IS research to the
nonacademic readership through other, nonacademic mediums/outlets.
However, we believe that the considerations we put forth in this essay as for
how to properly communicate academic quantitative research to the non-
academics remain germane to the matter of publishing IS research findings
in nonacademic mediums/outlets.
of independent/mediating variables, and the relevance of the
magnitude of findings.20 Accordingly, we recommend that if
IS research papers are intended to provide nonacademics with
lessons and implications (and our review of published IS
research suggests that this is often the case), authors should
provide adequate content on each of the three aforementioned
elements in order to ensure the proper communication of
research relevance.
(I) The Relevance of Dependent Variables: The dependent
variables/constructs in quantitative research often represent
the principal phenomena of interest researchers are dealing
with. It is therefore crucial, regarding the communication of
relevance, to offer content on the significance of dependent
variables in the eyes of nonacademics. Based on the
perspective proffered in this essay so far, it is clear that non-
academics may not necessarily find subjects or phenomena,
which researchers choose to study, relevant. The manner in
which we characterized the relevance of dependent variables
in this essay (see the Venn diagram section) emphasizes two
aspects: understandability and noteworthiness. In other
words, in an ideal situation, authors in a research paper are to
clearly establish that the nonacademic audience of their
research not only understand, but also acknowledge the note-
worthiness of, their dependent variable(s) of interest.
We envisage that, in practice, authors may find themselves in
at least two distinct types of situations when striving for
establishing the relevance of their dependent variables of
interest. The first type of situation, which is quite common,
has to do with research studies that employ dependent vari-
ables/constructs derived from pure academic literature. In this
type of situation there is a high chance that the dependent
variables/constructs will not be recognized at all, or not fully
acknowledged, in the everyday life of nonacademic stake-
holders. One practice we recommend in this type of situation
is to collect empirical evidence from samples of the targeted
nonacademic stakeholders through methods such as inter-
views or questionnaire surveys. The evidence should incor-
porate the stakeholders’ views about the understandability and
noteworthiness of the researchers’ dependent variables of
interest. It is also expected that researchers draw on the
collected evidence (say, through triangulation with the related
academic literature) to build/refine their proposed research
model(s)/hypotheses. We found a compelling example of this
20
In the case of quantitative research that employs other, sometimes, simpler,
or more classic, types of statistical designs/analyses such as experimental
design (involving ANOVA), and univariate, bivariate, cluster, or principal
component, analyses, we still believe that our proposed conceptions and
recommendations with respect to the notion of relevance remain applicable.
In all these types of quantitative research, one can still identify the three
elements of the phenomena under study, the factors espoused being asso-
ciated with the phenomena, and the reported magnitude of findings.
MIS Quarterly Vol. 44, No. 2/June 2020 547
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
practice in the study by Armstrong et al. (2015). As discussed
earlier in our detailed review of Armstrong et al., the authors
report on focus group interviews with IS professionals as one
of the major nonacademic stakeholders they target. The
authors conducted the interviews during the theory develop-
ment phase with the aim of determining the applicability of
their proposed research model to the particular context of an
IS career. Particularly with respect to the relevance of their
two major dependent variables of interest (i.e., Exhaustion
from IS Career Experience and Turn-Away Intention),
Armstrong et al. point out that “individuals in the focus
groups highlighted the importance of exhaustion and turn-
away for IS professionals” (p. 715). The authors also provide
details on how they adjusted their research model based on
comments from the IS professionals. Notwithstanding the
example drawn from the IS literature, we understand that, in
general, the task of conducting fieldwork to collect evidence
in order to establish the relevance of dependent variables
might be too demanding a requirement for IS researchers.
Therefore, another appropriate practice we can recommend is
to look for evidence reported by other studies in the literature,
where the desired nonacademic stakeholder group(s) are
shown to be understanding and acknowledging the importance
of the researchers’ dependent variables/constructs of interest.
The second type of situation is concerned with research
studies that incorporate constructs/variables that are directly,
in one way or another, derived from industry reports, prac-
titioner outlets, or online user-generated data. In this type of
situation, the researchers’ dependent variables/constructs of
interest are very likely to be already in use, or fully acknowl-
edged, by nonacademic stakeholders. In that case, we
recommend that authors provide clear evidence from the
respective industry/online sources to communicate that their
dependent constructs/variables of interest are actually the
same, or directly related to the ones, already acknowledged
among their targeted nonacademic stakeholders. An exemplar
study is Gao et al. (2015). The authors provide plain evidence
right from the beginning of their paper that their research
variables directly reflect “population measures of physician
quality as rated by patients, constructed using rigorous stan-
dards set by the Agency for Healthcare Research and Quality
(AHRQ) of the U.S. Department of Health and Human Ser-
vices” (p. 566).
(II) The Relevance of Independent/Mediating Variables:
Apart from dependent variables, the relevance of independent/
mediating variables should not also be taken for granted. As
illustrated in the Venn diagram section, nonacademics may
not find certain factors—which are research-wise believed to
be contributing to, or associated with, a phenomenon—
relevant (see Table 4). This is of particular issue in data ana-
lytics research, where, even from the standpoint of research
548 MIS Quarterly Vol. 44 No. 2/June 2020
rigor and methodology, there are increasing “construct
validity/credibility concerns pertaining to variables derived
from user-generated, non-survey-based data sources, [which
are] often characterized by low veracity” (Abbasi et al. 2016,
p. xxiii).
Our characterization of the relevance of independent/medi-
ating variables (see the Venn diagram section) specifies the
two aspects of understandability and actionability. As men-
tioned before, actionability, in our view, is a contingent
criterion—due to the possibility that a research study may
merely invoke a “general” relevance, as opposed to an “action
lever” relevance (see the Venn diagram section). Therefore,
our recommendation is for authors to clearly establish and
communicate that their targeted nonacademic stakeholders
consider the independent/mediating variables at least under-
standable, if not actionable. The practicalities of establishing
and communicating the relevance of independent/mediating
variables though remain similar to that of dependent variables.
Authors in fact may face the very same two different types of
situations as described in the case of dependent variables.
And again, the studies by Armstrong et al. and Gao et al. can
definitely serve as exemplars of communicating the relevance
of research variables (whether dependent or independent/
mediating). However, we also wish to draw attention here to
the interesting case of Wang et al. (2015), where the rele-
vance of independent variables/constructs is established
during the operationalization of the research model—as
opposed to the cases of Armstrong et al. and Gao et al., where
the relevance of research variables was mainly established
during the development of the research model or even earlier.
Wang et al. develop hypotheses on how five theoretical con-
structs—derived from the routine activity theory (RAT) in the
criminology literature—concerning the characteristics of
computer applications are associated with the dependent risk
of unauthorized access attempts. In order to operationalize
their research model, the authors solicit input from their main
targeted nonacademic stakeholder group (i.e., security
managers). In particular, the authors “interviewed application
owners and security managers for relevant information
regarding the [computer] applications” (p. 97). In addition,
Wang et al. report on a Delphi-type exercise with “13 appli-
cation owners who were ultimately responsible for the secu-
rity and risk management of their applications” (p. A2). The
aim of the exercise was to “confirm the mapping and mea-
surement relationship between [the theoretical] constructs and
[the real-world computer] application characteristics” (p. 97).
To conclude our discussion on how to communicate the
relevance of research variables/constructs, it is important to
emphasize that, as shown through the exemplar studies, the
evidence on the relevance of research variables/constructs
might, and sometimes should, not merely be presented in an
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
“implications for practice” section of a paper. On many occa-
sions, authors may, and in fact need to, initiate communi-
cating the relevance of research variables/constructs much
earlier—for instance, in the theory development, measure-
ment, or introduction sections.
(III) The Relevance of the Magnitude of Findings: To estab-
lish and communicate the relevance of research variables/
constructs is a crucial task. However, the bottom-line aspect
of research relevance often has to do with the real-world
impact associated with research findings in the eyes of the
nonacademic audience. When it comes down to writing up
the section on practical implications, IS researchers should
establish and communicate research relevance in an objective
manner—with clear mentions, and comprehensible interpre-
tations, of the magnitude of research findings, rather than with
subjective, imprecise statements, which may often have no
readily apparent connection with those research findings. Our
emphasis on providing nonacademics with objective and
comprehensible interpretations of quantitative results also
turns out to be completely in line with some emerging con-
cerns in the data analytics domain—namely, that analytical
models and algorithms often yield “measures that can be
difficult to understand for end-users or nonexperts” (Baesens
et al. 2016, p. 814). As addressed in a number of recent com-
mentaries and editorials (e.g., Abbasi et al. 2016; Baesens et
al. 2016; Lycett 2013; Müller et al. 2016; Sharma et al. 2014),
the issue with the interpretability and accountability of the
outputs of analytical algorithms seems to be a serious bottle-
neck in the way of realizing (business) value from data
analytics. Although advanced analytics models are obviously
superior in terms of capturing complex relationships and
producing accurate results and predictions, there is clear
evidence from such areas as healthcare or finance (e.g.,
Breiman 2001; Martens et al. 2007), where the stakes are
high, that practitioners might still strongly prefer simpler,
parsimonious regression models with a limited number of
well-understood variables—“even though these models could
be easily outperformed by non-linear models working on
high-dimensional data sets” (Müller et al. 2016, p. 294).
Therefore, strictly speaking, in a choice between interpret-
ability and accuracy/comprehensiveness, it is not surprising
at all that nonacademics would prefer the former.
In a nutshell, our recommendation is for authors to write the
“implications for practice” section differently in the future.
Authors should clearly exhibit how the real-world insights,
advice, viewpoints or interpretations offered in that section
are grounded in the particular magnitudes of research
findings. Therefore, it is necessary to present the quantitative
magnitude of pertinent findings again in the “implications for
practice” section. To refer the readers back to a sophisticated
table or chart, where many analysis results are often reported,
to look for a particular research finding is not an appropriate
practice. The magnitude of research findings along with the
corresponding real-world insights or interpretations should be
as accessible as possible to the nonacademic readership. It is
also obvious that the use of language is a key. The language
employed to communicate research findings should be as
simple and straightforward as possible. That being said, other
appropriate practices to facilitate the comprehension of
research findings are to convert and present results in such
easily digestible forms as percentages (when possible), and to
visualize research findings using charts/graphs/plots. In a
broader perspective on how to present findings in an “implica-
tions for practice” section, we also envision that the non-
academic audience would often conveniently relate to the
cost-benefit type of analyses, where authors may present
alternative manipulations of factors studied in their research
in order to demonstrate possible varying returns in terms of
comprehensible real-world costs/benefits.
According to our review of published IS research, the study
by Wang et al. (2015) is the only one that shows a fair amount
of adherence to the practices outlined in the preceding
paragraph, albeit with respect to only one research finding. In
their discussion of practical implications, the authors refer to
one obtained hazard ratio and conclude the following (p. 108):
The study showed that one level of increase in
CSTR [i.e., control strength] decreased the hazard
rate of an application by 28 percent. This finding
becomes pertinent in practice owing to the fact that
each application works within several controls;
enhancing an existing or implementing a new
(exclusive or compensatory) control can have a far-
reaching impact on improving the overall security of
an application.
The above quote from Wang et al. in fact denotes that the
authors aim to pursue a form of cost-benefit analysis in order
to establish a sound rationale on why a 28% decrease in the
hazard rate of an application is significant in the world of
practice. The authors could also pursue other complementary
practices to further strengthen their rationale. One practice
we recommend is to solicit, and present, views and comments
from samples of targeted nonacademic stakeholders (in the
case of Wang et al., security managers) on the significance of
research findings (in the case of Wang et al., the 28%
decrease in the hazard rate of an application). Another suit-
able approach is to draw comparisons with findings reported
in practitioner outlets, where similar effect sizes, ratios or
measures have been reported for the concerned phenomena/
variables. Additionally, in some cases, the comparisons may
be made, in one way or another, between research findings
and certain industry-wide/global standard rates, market
averages, benchmarks, and so forth.
MIS Quarterly Vol. 44, No. 2/June 2020 549
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance

Table 8. The Proper Communication of Research Relevance

Exemplar Papers
Corresponding Reviewed/Noted from
Elements to Be Aspect(s) of Recommended Practices to Establish/Communicate Published IS
Communicated Relevance Relevance Research
The relevance Understandability, • Collect and present evidence (solicited from the Armstrong et al. (2015)
of dependent and targeted nonacademic stakeholders) on the relevance Gao et al. (2015)
variables Noteworthiness of dependent variables.
• Collect and present evidence (from similar research
studies) on the relevance of dependent variables.
• Collect and present evidence from nonacademic
outlets/reports to prove that the same variables are
already recognized in practice.
The relevance Understandability, • Collect and present evidence (solicited from the Armstrong et al. (2015)
of independent/ and Actionability targeted nonacademic stakeholders) on the relevance Gao et al. (2015)
mediating (when of independent/mediating variables. Wang et al. (2015)
variables appropriate) • Collect and present evidence (from similar research
studies) on the relevance of independent/mediating
variables.
• Collect and present evidence from nonacademic
outlets/reports to prove that the same variables are
already recognized in practice.
The relevance Real-World • Offer practical insights, advice and interpretations Wang et al. (2015)
of the Impact along with the corresponding quantitative
magnitude of magnitude of research findings.
findings • Convert and present research findings in such easily
digestible forms as percentages, when possible.
• Visualize research findings using charts/graphs/ plots.
• Conduct and present cost-benefit type of analyses.
• Solicit and present views on the importance of
research findings from the targeted nonacademic
stakeholders.
• Draw comparisons with practitioner outlets’ findings,
industry-wide/global standard rates, market averages,
benchmarks, etc.

Table 8 provides a summary of our recommendations for the
proper communication of research relevance.
As with our demonstration of addressing practical signifi-
cance in the cases of the three MIS Quarterly papers (i.e.,
Armstrong et al. 2015; Liu et al. 2015; Park et al. 2015) in
Table 7, we now return again to the three papers for a demon-
stration of the communication of research relevance. Table 9
shows elements that have been already communicated by the
authors as well as the ones that remained overlooked.
Generalizing Relevance, Practical Significance,
and Statistical Significance
To conclude our recommendations on how to properly com-
municate practical significance and research relevance, we
raise a point as to how to address the issue of generalizability.
May the interpretations made by a research study in a given
setting, or with a certain population, concerning the matters of
relevance, practical significance, and statistical significance
be generalized to a new setting or population? For instance,
for a particular research study, there could be the findings of
no relevance, medium practical significance, and high statis-
tical significance. May these interpretations be generalized to
a new setting or population? According to Lee and Basker-
ville (2003), in their statement on the generalizability of a
theory, “[t]he only way in which a researcher (or practitioner)
may properly claim that the theory [which has been tested] is
indeed generalizable to the new setting would be for the
theory to be actually tested and confirmed in the new setting”
(p. 237). Therefore, in a research study that, for instance,
involves SEM, the “theory” would refer to its operationali-
zation in the form of equations, which are fitted, using data

550 MIS Quarterly Vol. 44 No. 2/June 2020

 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
Table 9. A Demonstration of the Communication of Research Relevance in the Cases of
Park et al., Armstrong et al., and Liu et al.
Reviewed MIS Example of
Quarterly Papers Practical Implication
Park et al. (2015) “As the results of the present
study show, perception of the
strength of information
assurance increases belief in
resilience.” (p. 338)
Armstrong et al. “Our findings suggest that
(2015) perceptions of fairness and its
importance to individuals extend
beyond any one particular job
and across an individual’s ISCE
[i.e., IS career experience]. This
finding is particularly pertinent
as HR managers are limited in
that they are only able to control
job/organizational factors....HR
managers should be cognizant
of the importance of fairness
perceptions for new hires and
existing IS employees.” (p. 723)
Liu et al. (2015) “Our finding of the negative
prism effect suggests a distrust
of friend bid by third parties. A
P2P lending platform may
increase the number of bids by
removing friend bid labeling.” (p.
740)
The Relevance of Research
Variables
Not Communicated
The authors could have
solicited evidence from some
hospital managers (as the
main targeted nonacademic
stakeholder group) on the
relevance of the research
variables, i.e., “information
assurance” and “perceived
resilience.”
“Focus group interviews were
conducted with 29 IS profes-
sionals [as one of the main
targeted nonacademic
stakeholder groups] …
working in three different
organizations in the transpor-
tation, food, and technology
industries in an effort to
determine the applicability of
this [research] model to the
career context.” (p. 715)
Not Communicated
The authors could have
demonstrated the real-world
relevance of the research
constructs (i.e., the prism
effect and the probability of
lending) by providing empirical
evidence solicited from practi-
tioners in the online lending
industry or from some similar
studies in the literature.
MIS Quarterly Vol. 44, No. 2/June 2020
The Relevance of the
Magnitude of Findings
Not Communicated
The authors could have
explicitly mentioned the
magnitude of research finding
corresponding to the term
“increase.” Also, the authors
could have presented a cost-
benefit type of analysis and/or
evidence solicited from some
hospital managers to indicate
if the amount of the “increase”
makes any difference in real-
world terms.
Not Communicated
The authors could have
presented the magnitude of
the “findings” they are talking
about. In addition, the authors
could have provided evidence
solicited from some HR
managers and/or practitioner
outlets to support the claim
that the magnitude of the
“findings” associated with the
research variable/construct
“fairness” in fact constitute an
important impact in the eyes of
HR managers, or other
pertinent practitioners in the
real world.
Not Communicated
The authors could have
explicitly mentioned the
magnitude of the “finding of
the negative prism effect” and
thereby could have disclosed
the estimated amount of
“increase” in the number of
bids that a P2P lending
platform should expect to earn
from “removing friend bid
labeling.” These all in fact
could have been incorporated
in a solid cost-benefit type of
analysis, where the authors
could have justifiably estab-
lished that the “finding of the
negative prism effect” can
make a substantial difference
in real-world terms.
551
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
from a sample, to the population. Thus, the communication
of practical significance, relevance, and even statistical
significance, of results in the research study should also
include the caution that the interpretations associated with
these matters may not necessarily generalize to a new setting
or population—and that valid interpretations would require
generating new results by refitting the statistical equations to
the new setting or population.
Conclusion
To conduct statistical analysis merely to obtain statistically
significant results is too limited a goal. Statistical signifi-
cance simply has to do with statistical concerns as to whether
an obtained result may be considered discernible or not.
Returning to the question posed earlier (i.e., toward the end of
the “Introduction” section) on the relationship between statis-
tical significance and scientific inference, it is also clear that
statistical significance does not do any good under most
circumstances because, mathematically speaking, statistical
significance is concerned with p(D|H0), whereas through
scientific inference researchers are looking for p(H0|D).
In addition to statistical significance, there are the matters of
practical significance and relevance. Practical significance,
although a well-established concept in statistics and some
social science areas, is largely misunderstood in the IS field.
Concerning the question “What is practical significance?”
(posed toward the end of the “Introduction” section), both the
theoretical (see Table 1) and illustrative (see Table 2) material
in this essay indicate that practical significance refers to the
impressiveness of the magnitude of results—obtained through
statistical/computational analyses—in the research world.
Research results could well be significant in the statistical
sense but still may be of no or little substantive importance
among researchers. This long-established contrast between
statistical significance and practical significance, therefore,
clearly suggests—in response to the question (posed toward
the end of the “Introduction” section) on the necessity of
reporting on practical significance—that researchers need to
address, and report on, not only p-values, but also the prac-
tical significance of obtained effect sizes, no matter if those
effect sizes are reported in the more recent “data analytics”
type of research, or in the classic types such as ANOVA,
linear regression or SEM. Furthermore, a firm communica-
tion structure has to be in place for reporting practical
significance. To only present effect size measurements in
some bulky tables or charts would not suffice. At least three
other elements (see Table 6), in addition to effect magnitudes,
have to be communicated in IS research studies in order to
accomplish an appropriate reporting on practical significance.
552 MIS Quarterly Vol. 44 No. 2/June 2020
Nonetheless, a research study, where statistically and practi-
cally significant results are obtained and properly communi-
cated, may not necessarily bear relevance to the real world.
Relevance is a distinct matter. As conceptualized in this
essay, relevance pertains to the situation in which a non-
academic stakeholder finds the phenomenon under study, the
factors associated with the phenomenon, and the reported
magnitude of findings, to make a difference to the stake-
holder’s own real-world concerns or interests. Therefore, in
response to the last question posed in the “Introduction” sec-
tion, it is completely conceivable to think of situations where
research findings would be of considerable practical signifi-
cance among researchers, but the research as a whole would
not interest the targeted nonacademic audience (see Table 4).
In principle, it is possible for a study to bear any or all three
of the qualities of statistical significance, practical signifi-
cance and relevance. A pertinent question, therefore, is:
“Might IS researchers deliver more impactful research if their
studies would bear all the three qualities?” Whereas ac-
hieving all three qualities in a research study would be
desirable, we believe there still might be occasions where a
research study may or may not bear one or more of these three
qualities, but could still tell an interesting and impactful story.
Thus, our major motivation behind this essay is not to call on
producing more IS research bearing all the three qualities;
rather, we are motivated by establishing the lesson that all
three qualities should be appropriately addressed and reported
on by researchers. Therefore, this essay is not primarily
meant to address a problem of research methodology per se,
but to draw attention to an equally important problem,
namely, that of research communication.
Being concerned with the issue of research communication,
our review of published IS research suggests that statistical
significance (as expected) is always reported, but practical
significance and relevance are seldom fully reported in IS
research studies. This, in our view, is indicative of the lack of
attention to the development and promotion of proper
research communication practices in our community. Conse-
quently, our effort in this essay has been to put forth a body
of recommendations concerning how researchers can better
communicate the value of their research to both academic and
nonacademic stakeholders. This is a critical effort con-
sidering the fact that quantitative research has a major role in
shaping the profile and character of IS research as a whole.
Interestingly to note, our review of MIS Quarterly papers
published in 2015 established that 64% (27 out of 42) of the
articles involved statistical/computational analyses.
In summary, our recommendations for future quantitative IS
research are (1) to view practical significance as an essential
part of rigorous statistical practice and a necessary companion
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
to statistical significance; (2) to communicate the practical
significance of research results by offering explicit, context-
sensitive interpretations of magnitude and supporting ration-
ales; (3) to provide evidence on the relevance of variables
incorporated into the research; and (4) to discuss implications
of the research for nonacademics by making reference to the
magnitude of research findings, rather than merely the sign or
the statistical significance of those findings. Our recommen-
dations are readily implementable and do not place any
additional burden on researchers. Indeed, as exemplified
throughout this essay, there have been some past IS research
studies that have, in effect, followed our recommendations to
some extent. Furthermore, while not primarily intended to
serve as procedures or evaluative standards as to how one
may achieve or assess practical significance or relevance in IS
research studies, our recommendations can still give rise to
ideas with respect to such procedures or standards—a
research opportunity worthy to be pursued much further in the
future.
Acknowledgments
The authors wish to thank the senior editor, Jason Bennett Thatcher,
the associate editor, Stacie Petter, and the reviewers for constructive
comments and insights that have substantially improved this paper.
References
AACSB International. 2008. Final Report of the AACSB Task
Force on Impact of Research, The Association to Advance
Collegiate Schools of Business, Tampa, FL.
Abbasi, A., Sarker, S., and Chiang, R. H. L. 2016. “Big Data
Research in Information Systems: Toward an Inclusive Research
Agenda,” Journal of the Association for Information Systems
(17:2), pp. i-ii.
Adams, D. A., Nelson, R. R. and Todd, P. A. 1992. “Perceived
Usefulness, Ease of Use, and Usage of Information Technology:
A Replication,” MIS Quarterly (16:2), pp. 227-247.
Aguinis, H., Werner, S., Abbott, J. L., Angert, C., Park, J. H., and
Kohlhausen, D. 2010. “Customer-Centric Science: Reporting
Significant Research Results with Rigor, Relevance, and Prac-
tical Impact in Mind,” Organizational Research Methods (13:3),
pp. 515-539.
Aguirre-Urreta, M. I. and Rönkkö, M. 2018. “Statistical Inference
with PLSc Using Bootstrap Confidence Intervals,” MIS Quarterly
(42:3), pp. 1001-1020.
Ahuja, A., Chudoba, K. M., Kacmar, C. J., McKnight, H., and
George, J. F. 2007. “IT Road Warriors: Balancing Work–
Family Conflict, Job Autonomy, and Work Overload to Mitigate
Turnover Intentions,” MIS Quarterly (31:1), pp. 1-17.
American Educational Research Association. 2006. “Standards for
Reporting on Empirical Social Science Research in AERA
Publications,” Educational Researcher (35:6), pp. 33-40.
American Psychological Association. 2010. Publication Manual
of the American Psychological Association, American Psycho-
logical Association, Washington, DC.
Armstrong, D. J., Brooks, N. G., and Riemenschneider, C. K. 2015.
“Exhaustion from Information System Career Experience:
Implications for Turn-Away Intention,” MIS Quarterly (39:3),
pp. 713-727.
Baesens, B., Bapna, R., Marsden, J. R., Vanthienen, J., and Zhao,
J. L. 2016. “Transformational Issues of Big Data and Analytics
in Networked Business,” MIS Quarterly (40:4), pp. 807-818.
Banerjee, D., Cronan, T. P., and Jones, T. W. 1998. “Modeling IT
Ethics: A Study in Situational Ethics,” MIS Quarterly (22:1), pp.
31-60.
Bapna, R., Qiu, L., and Rice, S. 2017. “Repeated Interactions
Versus Social Ties: Quantifying the Economic Value of Trust,
Forgiveness, and Reputation Using a Field Experiment,” MIS
Quarterly (41:3), pp. 841-866.
Benaroch, M., Lichtenstein, Y., and Fink, L. 2016. “Contract
Design Choices and the Balance of Ex Ante and Ex Post
Transaction Costs in Software Development Outsourcing,” MIS
Quarterly (40:1), pp. 57-82.
Berkson, J. 1938. “Some Difficulties of Interpretation Encountered
in the Application of the Chi-Square Test,” Journal of the Ameri-
can Statistical Association (33), pp. 526-536.
Boudreau, M.C., Gefen, D., and Straub, D. 2001. “Validation in
Information Systems Research: A State-of-the-Art Assessment,”
MIS Quarterly (25:1), pp. 1-16.
Breaugh, J. A. 2003. “Effect Size Estimation: Factors to Consider
and Mistakes to Avoid,” Journal of Management (29:1), pp.
79-97.
Boss, S. R., Galletta, D. F., Lowry, P. B., Moody, G. D., and
Polak, P. 2015. “What Do Users Have to Fear? Using Fear
Appeals to Engender Threats and Fear that Motivate Protective
Security Behaviors,” MIS Quarterly (39:4), pp. 837-864.
Breiman, L. 2001. “Statistical Modeling: The Two Cultures,”
Statistical Science (16:3), pp. 199-231.
Brown, S. A., and Venkatesh, V. 2005. “Model of Adoption of
Technology in Households: A Baseline Model Test and Exten-
sion Incorporating Household Life Cycle,” MIS Quarterly (29:3),
pp. 399-426.
Campbell, T. C. 2005. “An Introduction to Clinical Significance:
An Alternative Index of Intervention Effect for Group Experi-
mental Designs,” Journal of Early Intervention (27:3), pp.
210-227.
Chan, H. C., Wei, K. K., and Siau, K. L. 1993. “User-Database
Interface: The Effect of Abstraction Levels on Query Perfor-
mance,” MIS Quarterly (17:4), pp. 441-464.
Chatfield, C. 1995. Problem Solving: A Statistician’s Guide (2nd
ed.), London: Chapman & Hall/CRC.
Cohen, J. 1977. Statistical Power Analysis for the Behavioral
Sciences (Rev. Ed.), New York: Academic Press.
Cohen, J. 1988. Statistical Power Analysis for the Behavioral
Sciences, Hillsdale, NJ: Lawrence Erlbaum Associates.
Cohen, J. 1992. “A Power Primer,” Psychological Bulletin (112),
pp. 155-159.
Cortina, J. M., and Landis, R. S. 2009. “When Small Effect Sizes
Tell a Big Story, and When Large Effect Sizes Don’t,” in
Statistical and Methodological Myths and Urban Legends:
MIS Quarterly Vol. 44, No. 2/June 2020 553
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
Doctrine, Verity and Fable in the Organizational and Social
Sciences, C. E. Lance and R. J. Vandenberg (eds.), New York:
Routledge, pp. 287-308.
Cumming, G., and Finch, S. 2001. “A Primer on the Under-
standing, Use, and Calculation of Confidence Intervals that are
Based on Central and Noncentral Distributions,” Educational and
Psychological Measurement (61:4), pp. 532-574.
Daniel, W. W. 1977. “Statistical Significance Versus Practical
Significance,” Science Education (61:3), pp. 423-427.
Dennis, A. R. 2001. “Relevance in Information Systems Research,”
Communications of the Association for Information Systems (6),
Article 10.
Desouza, K. C., El Sawy, O. A., Galliers, R. D., Loebbecke, C., and
Watson, R. T. 2006. “Beyond Rigor and Relevance Towards
Responsibility and Reverberation: Information Systems Re-
search That Really Matters,” Communications of the Association
for Information Systems (17), Article 16.
Ellis, P. D. 2010. The Essential Guide to Effect Size: An
Introduction to Statistical Power, Meta-Analysis and the Inter-
pretation of Research Results, Cambridge, UK: Cambridge
University Press.
Falk, R. 1998. “Replication—A Step in the Right Direction,”
Theory and Psychology (8), pp. 313-321.
Falk, R., and Greenbaum, C. W. 1995. “Significance Tests Die
Hard: The Amazing Persistence of a Probabilistic Misconcep-
tion,” Theory and Psychology (5), pp. 75-98.
Faraj, S., Kudaravalli, S., and Wasko, M. 2015. “Leading Colla-
boration in Online Communities,” MIS Quarterly (39:2), pp.
393-412.
Fisher, R. A. 1921. “On the ‘Probable Error’ of a Coefficient of
Correlation Deduced from a Small Sample,” Metron (l), pp. 1-32.
Fisher, R. A. 1925. Statistical Methods for Research Workers,
London: Oliver & Boyd.
Gao, G. G., Greenwood, B. N., Agarwal, R., and Jeffrey, S. 2015.
“Vocal Minority and Silent Majority: How Do Online Ratings
Reflect Population Perceptions of Quality?,” MIS Quarterly
(39:3), pp. 565-589.
Gefen, D., Straub, D., and Boudreau, M. C. 2000. “Structural
Equation Modeling and Regression: Guidelines for Research
Practice,” Communications of the Association for Information
Systems (4), Article 7.
George, G., Haas, M. R., and Pentland, A. 2014. “Big Data and
Management,” Academy of Management Journal (57:2), pp.
321-326.
Greenstein, S., and Zhu, F. 2016. “Open Content, Linus’ Law, and
Neutral Point of View,” Information Systems Research (27:3), pp.
618-635.
Goh, J. M., Gao, G., and Agarwal, R. 2016. “The Creation of
Social Value: Can an Online Health Community Reduce Rural–
Urban Health Disparities?,” MIS Quarterly (40:1), pp. 247-263.
Grissom, R. J., and Kim, J. J. 2005. Effect Sizes for Research: A
Broad Practical Approach, Mahwah, NJ: Erlbaum.
Hacking, I. 1965. Logic of Statistical Inference, New York:
Cambridge University Press.
Hair, J. F., Ringle, C. M., and Sarstedt, M. 2011. “PLS-SEM:
Indeed a Silver Bullet,” Journal of Marketing Theory and
Practice (19:2), pp. 139-152.
554 MIS Quarterly Vol. 44 No. 2/June 2020
Hambrick, D. C. 1994. “What if the Academy Actually
Mattered?,” Academy of Management Review (19:1), pp. 11-16.
Han, W., Ada, S., Sharman, R., and Rao, H. R. 2015. “Campus
Emergency Notification Systems: An Examination of Factors
Affecting Compliance with Alerts,” MIS Quarterly (39:4), pp.
909-929.
Hassan, N. R. 2014. “Value of IS Research: Is There a Crisis?”
Communications of the Association for Information Systems (34),
Article 41.
Hoehle, H., and Venkatesh, V. 2015. “Mobile Application
Usability: Conceptualization and Instrument Development,” MIS
Quarterly (39:2), pp. 435-472.
Hong, W., Hess, T. J., and Hardin, A. 2013. “When Filling the
Wait Makes it Feel Longer: A Paradigm Shift Perspective for
Managing Online Delay,” MIS Quarterly (37:2), pp. 383-406.
Hsieh, J. J. P. A., Rai, A., Petter, S., and Zhang, T. 2012. “Impact
of User Satisfaction with Mandated CRM Use on Employee
Service Quality,” MIS Quarterly (36:4), pp. 1065-1080.
Huberty, C. J. 2002. “A History of Effect Size Indices,” Educa-
tional and Psychological Measurement (62:2), pp. 227-240.
Igbaria, M., Parasuraman, S. and Badawy, M. K. 1994. “Work
Experiences, Job Involvement, and Quality of Work Life among
Information Systems Personnel,” MIS Quarterly (18:2), pp.
175-201.
Iyengar, K., Sweeney, J. R., and Montealegre, R. 2015. “Infor-
mation Technology Use as a Learning Mechanism: The Impact
of IT Use on Knowledge Transfer Effectiveness, Absorptive
Capacity, and Franchisee Performance,” MIS Quarterly (39:3),
pp. 615-641.
Johnston, A. C., Warkentin, M., and Siponen, M. T. 2015. “An
Enhanced Fear Appeal Rhetorical Framework: Leveraging
Threats to the Human Asset Through Sanctioning Rhetoric,” MIS
Quarterly (39:1), pp. 113-134.
Kankanhalli, A., Ye, H., and Teo, H. H. 2015. “Comparing Poten-
tial and Actual Innovators: An Empirical Study of Mobile Data
Services Innovation,” MIS Quarterly (39:3), pp. 667-682.
Kazdin, A. E. 1999. “The Meanings and Measurement of Clinical
Significance,” Journal of Consulting and Clinical Psychology
(67:3), pp. 332-339.
Kelley, K., and Preacher, K. J. 2012. “On Effect Size,” Psycho-
logical Methods (17:2), pp. 137-152.
Kerlinger, F. N., and Pedhazur, E. J. 1973. Multiple Regression in
Behavioral Research, New York: Holt, Rinehart and Winston.
Kettinger, W. J., Grover, V., Guha, S. and Segars, A. H. 1994.
“Strategic Information Systems Revisited: A Study in Sustain-
ability and Performance,” MIS Quarterly (18:1), pp. 31-58.
Kirk, R. E. 1996. “Practical Significance: A Concept Whose Time
Has Come,” Educational and Psychological Measurement (56:5),
pp. 746-759.
Kirk, R. E. 2001. “Promoting Good Statistical Practices: Some
Suggestions,” Educational and Psychological Measurement
(61:2), pp. 213-218.
Kothari, S. P., Laguerre, T. E., and Leone, A. J. 2002. “Capitali-
zation Versus Expensing: Evidence on the Uncertainty of Future
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
Earnings from Capital Expenditures Versus R&D Outlays,”
Review of Accounting Studies (7:4), pp. 355-382.
Koontz, H. 1961. “The Management Theory Jungle,” Academy of
Management Journal (4:3), pp. 174-188.
Lee, A. S., and Baskerville, R. L. 2003. “Generalizing Generali-
zability in Information Systems Research,” Information Systems
Research (14:3), pp. 221-243.
Lee, A. S., and Mohajeri, K. 2012. “Linking Relevance to Practical
Significance,” in Proceedings of the 45th Hawaii International
Conference on System Sciences, Los Alamitos, CA: IEEE
Computer Society Press.
Leonardi, P. M. 2015. “Ambient Awareness and Knowledge
Acquisition: Using Social Media to Learn ‘Who Knows What’
and ‘Who Knows Whom,’” MIS Quarterly (39:4), pp. 747-762.
Lewandowsky, S., and Maybery, M. 1998. “The Critics Rebutted:
A Pyrrhic Victory,” Behavioral and Brain Sciences (21), pp.
210-211.
Lin, M., Lucas Jr., H. C., and Shmueli, G. 2008. “Is More Always
Better? Larger Samples and False Discoveries,” Working Paper
RHS 06-068, Robert H Smith School of Business, University of
Maryland.
Lin, M., Lucas Jr., H. C., and Shmueli, G. 2013. “Research Com-
mentary—Too Big to Fail: Large Samples and the p-Value
Problem,” Information Systems Research (24:4), pp. 906-917.
Liu, D., Brass, D., Lu, Y., and Chen, D. 2015. “Friendships in
Online Peer-To-Peer Lending: Pipes, Prisms, and Relational
Herding,” MIS Quarterly (39:3), pp. 729-742.
Ludwig, S., De Ruyter, K., Mahr, D., Wetzels, M., Brüggen, E., and
De Ruyck, T. 2014. “Take Their Word for It: The Symbolic
Role of Linguistic Style Matches in User Communities,” MIS
Quarterly (38:4), pp. 1201-1217.
Lycett, M. 2013. “‘Datafication’: Making Sense of (Big) Data in
a Complex World,” European Journal of Information Systems
(22:4), pp. 381-386.
Martens, D., Baesens, B., Van Gestel, T., and Vanthienen, J. 2007.
“Comprehensible Credit Scoring Models Using Rule Extraction
from Support Vector Machines,” European Journal of Opera-
tional Research (183:13), pp. 1466-1476.
Maruping, L. M., and Magni, M. 2015. “Motivating Employees to
Explore Collaboration Technology in Team Contexts,” MIS
Quarterly (39:1), pp. 1-16.
Miranda, S. M., Kim, I., and Summers, J. D. 2015. “Jamming with
Social Media: How Cognitive Structuring of Organizing Vision
Facets Affects IT Innovation Diffusion,” MIS Quarterly (39:3),
pp. 591-614.
Mohajeri, K., and Leidner, D. E. 2017. “Towards a Typology of
Relevance,” in Proceedings of the 50th Hawaii International
Conference on System Sciences, Los Alamitos, CA: IEEE
Computer Society Press.
Müller, O., Junglas, I., vom Brocke, J., and Debortoli, S. 2016.
“Utilizing Big Data Analytics for Information Systems Research:
Challenges, Promises and Guidelines,” European Journal of
Information Systems (25:4), pp. 1-14.
Neyman, J., and Pearson, E. S. 1928. “On the Use and Interpreta-
tion of Certain Test Criteria for Purposes of Statistical Inference,”
Biometrika (29A), Part I: pp. 175-240; Part II: pp. 263-294.
Nickerson, R. S. 2000. “Null Hypothesis Significance Testing: A
Review of an Old and Continuing Controversy,” Psychological
Methods (5:2), pp. 241-301.
Nickerson, R. S. 2011. “Null-Hypothesis Significance Testing:
Misconceptions,” in International Encyclopedia of Statistical
Science, M. Lovric (ed.), Berlin: Springer.
Park, I., Sharman, R., and Rao, H. R. 2015. “Disaster Experience
and Hospital Information Systems: An Examination of Perceived
Information Assurance, Risk, Resilience, and HIS Usefulness,”
MIS Quarterly (39:2), pp. 317-344.
Pearson, K. 1900. “On the Correlation of Characters Not Quan-
titatively Measurable,” Philosophical Transactions of the Royal
Society of London, Series A, Containing Papers of a Mathe-
matical or Physical Character (195), pp. 1-47.
Petter, S., Straub, D., and Rai, A. 2007. “Specifying Formative
Constructs in Information Systems Research,” MIS Quarterly
(31:4), pp. 623-656.
Prentice, D. A., and Miller, D. T. 1992. “When Small Effects Are
Impressive,” Psychological Bulletin (112), pp. 160-164.
Rai, A., Arikan, I., Pye, J., and Tiwana, A. 2015. “Fit and Misfit of
Plural Sourcing Strategies and IT-Enabled Process Integration
Capabilities: Consequences of Firm Performance in the U.S.
Electric Utility Industry,” MIS Quarterly (39:4), pp. 865-885.
Ramasubbu, N., Bharadwaj, A., and Tayi, G. K. 2015. “Software
Process Diversity: Conceptualization, Measurement, and Analy-
sis of Impact on Project Performance,” MIS Quarterly (39:4), pp.
787-807.
Robey, D., and Markus, M. L. 1998. “Beyond Rigor and Rele-
vance: Producing Consumable Research about Information
Systems,” Information Resources Management Journal (11:1),
pp. 7-15.
Robinson, D. H., Whittaker, T. A., Williams, N. J., and Beretvas,
S. N. 2003. “It’s Not Effect Sizes So Much as Comments about
Their Magnitude That Mislead Readers,” Journal of Experi-
mental Education (72:1), pp. 51-64.
Rosemann, M., and Vessey, I. 2008. “Toward Improving the Rele-
vance of Information Systems Research to Practice: The Role of
Applicability Checks,” MIS Quarterly (32:1), pp. 1-22.
Rosenthal, R. 1990. “How Are We Doing in Soft Psychology?,”
American Psychologist (45), pp. 775-777.
Rosenthal, R., and Rubin, D. B. 1979. “A Note on Percent
Variance Explained as a Measure of the Importance of Effects,”
Journal of Applied Social Psychology (9), pp. 395-396.
Rosnow, R. L., and Rosenthal, R. 2003 “Effect Sizes for Experi-
menting Psychologists,” Canadian Journal of Experimental
Psychology (57:3), p. 221-237.
Sabherwal, R., and Jeyaraj, A. 2015. “Information Technology
Impacts on Firm Performance: An Extension of Kohli and
Devaraj (2003),” MIS Quarterly (39:4), pp. 809-836.
Saboo, A. R., Kumar, V., and Park, I. 2016. “Using Big Data to
Model Time-Varying Effects for Marketing Resource (Re)Allo-
cation,” MIS Quarterly (40:4), pp. 911-940.
Salge, T. O., Kohli, R., and Barrett, M. 2015. “Investing in Infor-
mation Systems: On the Behavioral and Institutional Search
Mechanisms Underpinning Hospitals’ IS Investment Decisions,”
MIS Quarterly (39:1), pp. 61-89.
MIS Quarterly Vol. 44, No. 2/June 2020 555
Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance
Scherer, A., Wünderlich, N., and von Wangenheim, F. 2015. “The
Value of Self-Service: Long-Term Effects of Technology-Based
Self-Service Usage on Customer Retention,” MIS Quarterly
(39:1), pp. 177-200.
Schwab, A., Abrahamson, E., Starbuck, W. H., and Fidler, F. 2011.
“Perspective—Researchers Should Make Thoughtful Assess-
ments Instead of Null-hypothesis Significance Tests,” Organi-
zation Science (22:4), pp. 1105-1120.
Selvin, H. C. 1957. “A Critique of Tests of Significance in Survey
Research,” American Sociological Review (22:5), pp. 519-527.
Sharma, R., Mithas, S., and Kankanhalli, A. 2014. “Transforming
Decision-Making Processes: A Research Agenda for Under-
standing the Impact of Business Analytics on Organisations,”
European Journal of Information Systems (23:4), pp. 433-441.
Shmueli, G., and Koppius, O. R., 2011. “Predictive Analytics in
Information Systems Research,” MIS Quarterly (35:3), p. 553.
Shen, W., Hu, Y. J., and Rees, J. 2015. “Competing for Attention:
An Empirical Study of Online Reviewers’ Strategic Behaviors,”
MIS Quarterly (39:3), pp. 683-696.
Singh, H., Aggarwal, R., and Cojuharenco, I. 2015. “Strike a
Happy Medium: The Effect of IT Knowledge on Venture
Capitalists’ Overconfidence in IT Investments,” MIS Quarterly
(39:4), pp. 887-908.
Straub, D. W. 2009. “Editor’s Comments: Why Top Journals
Accept Your Paper,” MIS Quarterly (33:3), iii-x.
Straub, D. W., and Ang, S. 2008. “Editor's Comments: Readability
and the Relevance Versus Rigor Debate,” MIS Quarterly (32:4),
pp. iii-xiii.
Straub, D. W., and Ang, S. 2011. “Editor’s Comments: Rigor and
Relevance in IS Research: Redefining the Debate and a Call for
Future Research,” MIS Quarterly (35:1), pp. iii-xi.
Straub, D. E., Boudreau, M.-C., and Gefen, D. 2004. “Validation
Guidelines for IS Positivist Research,” Communications of the
Association for Information Systems (13:1).
Sykes, T. A. 2015. “Support Structures and Their Impacts on
Employee Outcomes: A Longitudinal Field Study of an Enter-
prise System Implementation,” MIS Quarterly (39:2), pp.
473-495.
Te’eni, D., Rowe, F., Ågerfalk, P. J., and Lee, J. S. 2015.
“Publishing and Getting Published in EJIS: Marshaling Contri-
butions for a Diversity of Genres,” European Journal of
Information Systems (24:6), pp. 559-568.
Thompson, B. 2002. “‘Statistical,’‘Practical,’ and ‘Clinical’: How
Many Kinds of Significance Do Counselors Need to Consider?,”
Journal of Counseling & Development (80:1), pp. 64-71.
Tian, F., and Xu, S. X. 2015. “How Do Enterprise Resource
Planning Systems Affect Firm Risk? Post-Implementation
Impact,” MIS Quarterly (39:1), pp. 39-60.
Tripp, J. F., Riemenschneider, C., and Thatcher, J. B. 2016. “Job
Satisfaction in Agile Development Teams: Agile Development
as Work Redesign,” Journal of the Association for Information
Systems (17:4), pp. 267-307.
Vacha-Haase, T., and Thompson, B. 2004. “How to Estimate and
Interpret Various Effect Sizes,” Journal of Counseling Psych-
ology (51:4), pp. 473-481.
556 MIS Quarterly Vol. 44 No. 2/June 2020
Vance, A., Lowry, P. B., and Eggett, D. 2015. “Increasing
Accountability Through User-Interface Design Artifacts: A New
Approach to Addressing the Problem of Access-Policy Viola-
tions,” MIS Quarterly (39:2), pp. 345-366.
Vaske, J. J., Gliner, J. A., and Morgan, G. A. 2002. “Commu-
nicating Judgements about Practical Significance: Effect Size,
Confidence Intervals and Odds Ratios,” Human Dimensions of
Wildlife (7), pp. 287-300.
Venkatesh, V., and Morris, M. G. 2000. “Why Don’t Men Ever
Stop to Ask for Directions? Gender, Social Influence, and Their
Role in Technology Acceptance and Usage Behavior,” MIS
Quarterly (24:1), pp. 115-139.
Venkatesh, V., Thong, J. Y. L., Chan, F. K. Y., and Hu, P. J. H.
2016. “Managing Citizens’ Uncertainty in E-Government
Services: The Mediating and Moderating Roles of Transparency
and Trust,” Information Systems Research (27:1), pp. 87-111.
Vittinghoff, E., Glidden, D., Shiboski, S. C., McCulloch, C. E.
2005. Regression Methods in Biostatistics: Linear, Logistic,
Survival, and Repeated Measures Models, New York: Springer.
Wang, J., Gupta, M., and Rao, H. R. 2015. “Insider Threats in a
Financial Institution: Analysis of Attack-Proneness of Informa-
tion Systems Applications,” MIS Quarterly (39:1), pp. 91-112.
Wang, J., Li, Y., and Rao, H. R. 2016. “Overconfidence in
Phishing Email Detection,” Journal of the Association for Infor-
mation Systems (17:11), pp. 759-783.
Wonnacott, T. H., and Wonnacott, R. J. 1984. Introductory
Statistics for Business and Economics (3rd ed.), New York: John
Wiley & Sons.
Wu, S. P. J., Straub, D. W., and Liang, T. P. 2015. “How Infor-
mation Technology Governance Mechanisms and Strategic
Alignment Influence Organizational Performance: Insights from
a Matched Survey of Business and IT Managers,” MIS Quarterly
(39:2), pp. 497-518.
Yeow, A., and Goh, K. H. 2015. “Work Harder or Work Smarter?
Information Technology and Resource Allocation in Healthcare
Processes,” MIS Quarterly (39:4), pp. 763-785.
About the Authors
Kaveh Mohajeri is an assistant professor at IESEG School of
Management. He received both his B.Sc. degree in Industrial
Management and M.Sc. degree in IT Management from the
University of Tehran, Iran. He also has a Ph.D. in Information
Systems from Virginia Commonwealth University. His research
agenda mainly deals with advancing new research practices with a
focus on the issues of research relevance and theory testing. Before
starting his Ph.D. program, Kaveh worked in industry as an
entrepreneur and a project analyst for more than four years.
Mostafa Mesgari is an assistant professor of Information Systems
at Loyola Marymount University, Los Angeles. He received his
Ph.D. degree in business administration from Concordia University,
 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance

Montreal, Canada, in 2016. His research interests include data
analytics, open communities, and human–computer interaction. He
explores novel data-driven methodologies to study the socio-
technical aspects of information systems in organizations. His work
has appeared in leading outlets including European Journal of
Information Systems, IEEE Transactions on Software Engineering,
Journal of the Association for Information Science and Technology,
and Information Processing & Management. Before entering
academia, he worked as a computer systems analyst.
Allen S. Lee is professor emeritus of information systems at
Virginia Commonwealth University. He served as editor-in-chief of
MIS Quarterly and as a founding senior editor of MIS Quarterly
Executive. His research program has involved showing not only
how qualitative research can be done rigorously and scientifically,
but also how quantitative research equally needs to live up to the
requirements of science. He is a Fellow of the Association for
Information Systems, a LEO Award recipient, and a member of the
Circle of Compadres of the Information Systems Doctoral Students
Association of the KPMG Ph.D. Project.

Appendix A
An Illustration of Relevance, Practical Significance and
Statistical Significance (Continued)
Region 3: The research study falls in Region 3 in Figure 1 if it has the profile shown in Table A1.

Table A1. The Profile of the Example Research Study in Region 3

X statistical significance
T practical significance
T relevance
The research is not statistically significant because the p-values corresponding to the
^ and â ^ are both large (p > .05).
estimated coefficients â1 2
^ and â
The research is practically significant because the estimated coefficients â ^ have
1 2
certain magnitudes to constitute what the researchers consider to be impressive effect size
values.
The research is relevant to the firm because:
• For the firm, “individual BI usage proficiency” is an understandable and noteworthy matter.
• For the firm, “organizational role of BI user” and “length of BI user training” are
understandable and actionable factors.
• For the firm, the magnitudes of the research results corresponding to “organizational role
of BI user” and “length of BI user training” are satisfactory to make a difference to its
question of interest, which pertains to “individual BI usage proficiency.”

The scenario associated with Region 3 is useful for highlighting that research results can, at the same time, lack statistical significance but still
have practical significance and relevance. The absence of statistical significance does not, by itself, preclude practical significance from the
perspective of the researchers conducting the statistical analysis. Also, the absence of statistical significance does not, by itself, preclude the
firm from deeming the magnitudes of statistical results corresponding to “organizational role of BI user” and “length of BI user training” to
be satisfactory to make a difference to its question of interest, which pertains to “individual BI usage proficiency.” Moreover, the firm is aware
that, among researchers, the given threshold for the p-value in many disciplines is an arbitrary convention (see Kelley and Preacher 2012,
p. 139) and, therefore, does not necessarily see the lack of statistical significance in this case to be a negative criticism of the research model.

Region 4: The research study falls in Region 4 in Figure 1 if it has the profile shown in Table A2.

MIS Quarterly Vol. 44, No. 2/June 2020 557

Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance

Table A2. The Profile of the Example Research Study in Region 4

X statistical significance
X practical significance
T relevance
The research is not statistically significant because the p-values corresponding to the
^ and â^ are both large (p > .05).
estimated coefficients â1 2
^ and â
The research is not practically significant because the estimated coefficients â ^ do
1 2
not have magnitudes to constitute what the researchers consider to be impressive effect
size values.
The research is relevant to the firm because:
• For the firm, “individual BI usage proficiency” is an understandable and noteworthy
matter.
• For the firm, “organizational role of BI user” and “length of BI user training” are
understandable and actionable factors.
• For the firm, the magnitudes of the research results corresponding to “organizational
role of BI user” and “length of BI user training” are satisfactory to make a difference to
its question of interest, which pertains to “individual BI usage proficiency.”

Region 4 is useful for highlighting that research can still be relevant even when both statistical significance and practical significance are absent.
The researchers whom the firm commissioned to conduct the statistical study might very well caution the firm not to act on the basis of the
research model; however, the firm may also properly argue, as mentioned previously, that the insignificant findings do not necessarily
generalize to all settings in the firm, especially new settings that the researchers did not include in their study and in which the firm has an
interest in later applying the research model.

Region 6: The research study, which involves Model B, falls in Region 6 in Figure 1, if it has the profile shown in Table A3.

Table A3. The Profile of the Example Research Study in Region 6

X statistical significance
T practical significance
The research is not statistically significant because the p-values corresponding to the
^ and â ^ are both large (p > .05).
estimated coefficients â1 2
^ and â
The research is practically significant because the estimated coefficients â ^ have
1 2
certain magnitudes to constitute what the researchers consider to be impressive effect
size values.
The research is not relevant.

X relevance The firm does not consider the factors, indicated by the independent variables, to be
actionable. Specifically, the firm does not consider it feasible to change an employee’s
age or gender, or consider it ethical to select an employee based on age or gender.

This example shows the situation where researchers can consider the effect sizes to be impressive and where, at the same time, the lack of
statistical significance would indicate unreliability of the results (at 0.05 p-value level) and would therefore inhibit the researchers from
approving or recommending the research model. In any case, the lack of actionability of the research model would render it not relevant in
the eyes of the firm.

Region 7: The research study, which involves Model B, falls in Region 7 if it has the profile shown in Table A4.

558 MIS Quarterly Vol. 44 No. 2/June 2020

 Mohajeri et al./Investigating Relevance, Practical Significance, & Statistical Significance

Table A4. The Profile of the Example Research Study in Region 7

T statistical significance
X practical significance
The research is statistically significant because the p-values corresponding to the
^ and â ^ are both small (p < .05).
estimated coefficients â 1 2
^ and â
The research is not practically significant because the estimated coefficients â ^ do
1 2
not have magnitudes to constitute what the researchers consider to be impressive effect
size values.
The research is not relevant.

X relevance The firm does not consider the factors, indicated by the independent variables, to be
actionable. Specifically, the firm does not consider it feasible to change an employee’s
age or gender, or consider it ethical to select an employee based on age or gender.

Finally, the case of Region 7 draws attention to the situation where there is only statistical significance. In general, statistical significance, in
and of itself, should not be the goal of statistically conducted research in so far as statistically significant results are always obtainable as long
as the sample size is large enough (for an extensive explanation, see Nickerson 2000).

MIS Quarterly Vol. 44, No. 2/June 2020 559

Copyright of MIS Quarterly is the property of MIS Quarterly and its content may not be
copied or emailed to multiple sites or posted to a listserv without the copyright holder's
express written permission. However, users may print, download, or email articles for
individual use.