On the reliability and validity of digital
competences evaluation in Higher Education
Rafael Saltos Rivas Pavel Novoa-Herna´ndez
Facultad de Filosof´ıa, Letras Escuela de Ciencias Empresariales
y Ciencias Educacio´n Universidad Cato´lica del Norte
Universidad Te´cnica de Manab Coquimbo, Chile
´ı Portoviejo, Ecuador pavel.novoa@ucn.cl
w.saltos@utm.edu.ec
Abstract—The development and evaluation of digital compe-
tences in Higher Education has been in the focus of researchers
over the last two decades. In this context, some issues related
to the evaluation process itself remain unexplored. This is the
case of how reliability and validity assessments are conducted
and reported when an instrument is used for measuring digital
competences. In our opinion this is a paramount aspect that
clearly affects the achievement of meaningful results. In this
sense, this paper provides an overview about how the assessment
of reliability and validity have been conducted and reported.
Three research questions were addressed regarding to: 1) the
demographic and methodological features of the literature, 2)
typical assessments and reporting practices, and 3) association
of demographics and methodological features with the reporting
practice. Through a systematic mapping study, 60 primary stud-
ies published in period 2018-2020, were analyzed and classified.
Overall, results indicated that most studies were focused on
undergraduate students from Europe or Asia; about the 50%
of studies did not reported on these assessments; and both,
publishing in journals not indexed by the Journal Citation
Report and measuring only digital competences, are negatively
associated with the practice of reporting reliability and validity
assessments. We also discussed some important implications and
research opportunities derived from the results.
Index Terms—Digital competences, Higher Education,
Reliability, Validity, Systematic Mapping Study.
I. I NTRODUCTION
Our world is governed by Information and Communication
Technologies (ICTs) [1]. Currently, most of the essential
processes of modern society are computerized in one way or
another [2]. For this reason, it is essential that professionals
have sufficient ICT skills and competencies, adapting to the
demands of the modern working world [3], [4]. Higher
Education Institutions (HEIs) have an essential role in this
context [5], that is, by integrating different strategies to
provide these digital competences to their educational
community [6] and on the other hand, in building coherent
evaluation processes and instruments. The latter is particularly
important not only for diagnosing the educational community,
but also to verify the extent an intervention program has been
effective or not. Like happens in another scenarios from
Social Sciences, the development of an instrument for
measuring digital competences in Higher Education is a
complex process. About
Roc´ıo Serrano Rodr´ıguez
Departmento de Educacio´n
Universidad of Co´rdoba
Co´rdoba,
Espan˜a
m22seror@uco.es
15 steps have been identified in [7] for conducting such a
development. Among them, conducting assessment of
reliability and validity of the instrument are two crucial steps
in order to guarantee the achievement of meaningful results
[8]. However, although a number of experiences and research
related to the analysis of digital competences in Higher
Education has been accumulated over the last two decades
[9]–[11] some important issues remain unexplored regarding
the evaluation process itself. This is the case of reliability and
validity assessments. Following the suggestions given in [12],
when publishing their results researchers should provide
enough details about reliability and validity assessments in
order to provide strong evidences about the consistency and
accuracy of the employed instrument. So, a question arising
here is: can we trust in the available instruments used by
literature to evaluate digital competences in Higher
Education?
A related, more specific question is how spread is this
practice of reporting reliability and validity assessments in
research devoted to evaluate digital competences in Higher
Education? In our opinion, the benefits of answering this
question are twofold. In one hand, researchers will have an
overview about this topic thorough the current trends, which
will help to warn them about what have been done well and
what not. On the other hand, policy makers will benefit from
our results since a summary of studies reporting both type
of assessments is provided, which in turn contributes when
deciding the feasibility of using or not an existing instrument
for evaluating digital competences.
With aim at bring some answers to these questions, this
research focuses on the evaluation process of digital com-
petences in Higher Education by characterizing how current
research provides evidences about reliability and validity as-
sessments. A Systematic Mapping Study [13], was conducted
in order not only to provide an overview of the main trends
and current characteristics of this topic, but also to identify
which demographic and methodological features are related
to the practice of reporting such psychometric assessments.
 II. T HEORETICAL FRAMEWORK
A. Digital competences in Higher Education
Currently, no consensus exists about what is digital compe-
tences in the context of Higher Education [14]–[16]. This is
mainly because, as indicated in [16], the definition of digital
competence is context-dependent, and therefore, it is possible
to find various positions both in the scientific context [17],
and in the of the definition of government policies [3]. Added
to this is the fact that students and teachers demand specific
competences, which do not have to coincide as a whole [9].
Among the thirty definitions reviewed in [16] perhaps the
most complete is the one provided by [3] in the context
of defining policies. The author defined digital competences
as: “the set of knowledge, skills, attitudes, strategies and
awareness which are required when ICT and digital media
are used to perform tasks, resolve problems, communicate,
manage information, collaborate, create and share content,
and build knowledge in an effective, efficient and adequate
way, in a critical, creative, autonomous, flexible, ethical and a
sensible form for work, entertainment, participation, learning,
socialization, consumption and empowerment.” (p. 3).
In the same line, but in a scientific context [14] defined it as
the composition of: “(1) technical competence, (2) the ability
to use digital technologies in a meaningful way for working,
studying and in everyday life, (3) the ability to evaluate digital
technologies critically, and (4) motivation to participate and
commit in the digital culture.” (p. 655).
Another way to see what is meant by digital competence
is to approach this concept through the frameworks defined
both in the scientific context [9], and in that of government
policies [18]. A notable example in the latter case is the
Digital Competence Framework for Citizens (DigComp2.1)
promoted by the European Commission [19]. This framework,
in version 2.1, is the result of several refinement processes to
the first version that appeared in 2016. Although it is not
explicitly defined by [19], it is clear that according to
DigComp2.1, a citizen is digitally competent if he reaches
any of the 8 levels of competencies defined in this framework
within 5 general competency areas: 1) information and data
literacy, 2) communication and collaboration, 3) digital
content creation,
4) safety, and 5 ) problem solving.
From the above-mentioned definitions, it is easy to
conclude that measuring the degree or level of digital
competences involves at least three areas: knowledge, skills
and attitudes. In other words, the development of an
instrument for measuring digital competences should include
assessment items related to knowledge, skills and attitudes.
From a viewpoint of the Measurement Theory [7], assessing
items related to the first two areas can be classified as
Cognitive items, while those for measuring attitudes as Non-
cognitive items.
Literature includes several works related to our research
topic and methodology (review study). For example, [9]–[11],
[14]–[16], [20]–[24]. Among them only three works addressed
aspects related to the process of evaluating digital
competences in Higher Education [10], [16], [25].
In the case of [16], the authors included some relevant
factors for the evaluation process: such as the method of data
collection (instrument) and the study area of the participants.
However, it is not clear which works actually evaluated digital
competences. Regarding these two factors, the authors con-
cluded that most of the studies: use mixed methods or surveys
for measuring, and are based on populations from different
knowledge areas. The study of [25] concluded, from the 13
selected studies, that the programs and actions developed
by the HEIs leave out the development of competencies on
content creation and security. To identify this lack, the authors
used the DigComp 2.1 framework [19]. Finally, in the meta-
analysis developed by [10] the authors found that in the field
of Higher Education in Latin America, the proportion of
students and teachers with digital skills is moderate (64%),
with no notable differences between both type of
populations.
It is clear that other important aspects of the process of
evaluating digital competences remain unexplored by litera-
ture. One of them is related to the practice of assessment how
reliable and valid are the instrument used for conducting such
evaluation process.
B. Reliability and validity
When developing a measurement instrument it is important
to assess: 1) whether it provides the same (or similar) results
under similar conditions or inputs, and 2) whether it measures
what is supposed to measure [7]. While the first assessment
is known as reliability of the instrument, the second one is
known as the validity of the instrument [7]. Several types of
these assessments have been identified in the past [12]. A
detailed review of them is provided by [8], in which there
were identified four types of both reliability and validity
assessments. Regarding reliability, the following types were
defined:
1) Stability,
2) Internal Consistency,
3) Equivalence, and
4) Scalability.
In the case of validity, the following types were suggested in
[8]:
1) Face validity,
2) Content validity,
3) Criterion validity, and
4) Construct validity.
More details about the the methods for conducting these types
of assessments can be found in [8] and [12].
Ideally, an study developing or administering an instrument
should present enough details about the above mentioned
types of assessment. However, this is not always possible
because the existence of research limitations (e.g. time, lack of
another instrument to compare with, etc). In any case, it is
important to conduct an assessment of at least one of these
types in order to guaranteed a suitable degree of consistency
and accuracy for the instrument [7]. Even if the instrument
have been proposed and validated in a previous study it is a
good practice to check its reliability and validity [7].
 III. M ETHODOLOGY
This paper follows the methodology described in [13] for
conducting systematic mapping studies. In turn, this is a type
of secondary study [26]. It is also a correlation study since
we aimed to analyze the existing association between certain
variables under study.
According to the selected guide, the mapping is achieved
through three main steps: planning, conducting, and reporting.
Next sections describe how the first two steps were developed,
while the third ones is fulfilled by writing this paper.
A. Research questions
The main goal of this research is to provided an overview
about how the assessment of reliability and validity have been
conducted and reported by the current literature related to
the evaluation of digital competences in Higher Education.
Specifically, we were interested in answering the following
research questions:
• RQ1). What are the main demographic and methodolog-
ical features of the studies?
• RQ2) How assessment of reliability and validity are
reported?
– RQ2.1) What types of assessments are reported more
often?
– RQ2.2) To what extent are assessments reported?
• RQ3) What demographic and methodological features are
more likely associated to studies reporting reliability and
validity assessments?
B. Search
To design the search formula for finding the relevant
studies, we proceeded as follows. First, we used the PICO tool
[27] to identify the relevant terms according to the population
under study, intervention method, comparison group, and
outcomes. Second, carefully readings of similar reviews like
[10], [16] were useful in order to complement the results of
applying the PICO tool. As a result, the following search
formula was defined:
(”digital competence” OR ”digital literacy” OR
”digital literacies”) AND (undergraduate OR post-
graduate OR freshmen OR sophomore OR junior
OR senior OR preservice OR teacher OR junior
OR university OR ”higher education” OR college
OR tertiary OR ”academic staff” OR professor OR
lecturer) AND (evaluate OR assess OR appraise OR
validate OR evaluation OR assessment OR
appraisal OR validation OR evaluation OR
assessing OR appraising OR validating)
This formula was used to search in three relevant databases:
Scopus, Web of Science, and ERIC (Education Resources
Information Center). These databases cover a great part of the
scientific literature about Education Sciences and are widely
used in the review studies related to this field [15], [16].
The results obtained from applying the above formula are
shown in Table I. It is worth noting that the search was
conducted in July, 2020.
TABLE I
RESULTS OF THE SEARCH IN THE CONSIDERED DATABASES.
Database Search fields Studies
Scopus TITLE-ABS-KEY (Title, Abstract, and 466
Keywords)
Web of Science TS (Title, Abstract, and Keywords) 309
ERIC No specified. The query covered up the 239
title, abstract and descriptors of the indexed
documents.
Total 1014
C. Selection and quality assessment
During the selection we followed several steps, which are
summarized in Fig. 1. It is worth noting that we only covered
the period 2018-2020, because we are interest in
characterizing the current literature only. So, by focusing in
the last two and half years we get a good sample of this up-to-
date literature. The inclusion and exclusion criteria were as
follows:
Inclusion criteria:
• Studies using measuring digital competences quantita-
tively in the context of Higher Education.
• Studies published in the period 2018 to July, 2020.
• Studies published as journal article.
Exclusion criteria:
• Studies published in conference proceedings, book chap-
ters.
• Non-peer reviewed studies.
• Studies not accessible in full text.
• Books, technical reports, editorials.
• Duplicates of other studies.
Regarding quality assessment of the studies, we did not
con- duct an explicit evaluation of them previous to data
extraction. The main reason is that we are not interested in
selecting the studies with a suitable level of quality in order
to report meaningful results (e.g effect size of an
intervention). Our goal is to characterize the current literature
as such. Nevertheless, since we focused in peer-reviewed
journal articles that are indexed in quality databases, a
minimum quality is guaranteed.
D. Data extraction
Data extraction was conducted based on the template de-
picted in Table II. As it may be observed, 9 variables related
to demographic and methodological features of the study were
considered. Regarding the reporting style of reliability and
validity assessments, 4 different types were included [8]. For
each type, we collected how the study reported the
assessment, that is, No mentioned, Mentioned only,
Referencing (a previous study), and Details are provided.
Finally, a variable called Reported was defined in order to
perform association analysis. This variable take one of two
possible values: Yes (if the study at least mentioned the
assessment of both, reliability and validity), and No
(otherwise).
 Search in Inclusion and Title and Abstract Full-text Snowball Review of
Databases Exclusion Criteria Screening Reading Sampling Excluded Articles

+1014 -849 -99 -15 +9 0

Results = 1014 Results = 165 Results = 66 Results = 51 Results = 60 Results = 60

Fig. 1. Study selection process.

TABLE II
TEMPLATE USED FOR DATA EXTRACTION.

Group Variable Dimension Research question

Demographic Study ID Consecutive number RQ1
and
method- Year 2018,2019,2020 RQ1, RQ3
ological Continent Africa, Asia, Europe, North America, Oceania, South America, Intercontinental. RQ1, RQ3
features Participant Type Undergraduate Students, Academic Staff, Post-graduate Students, Mixed RQ1, RQ3
Discipline Natural Sciences, Engineering, Health Sciences, Agricultural Sciences, Social Sci- RQ1, RQ3
ences, Humanities, Multidisciplinary
Study Scope Administration, Development, Both (Administration and Development) RQ1, RQ3
Items Type Cognitive, Non-cognitive, Both (Cognitive and Non-cognitive) RQ1, RQ3
Assessment Context Effectiveness of an intervention, Diagnostic, Hypothesis testing, Proposal RQ1, RQ3
SJR* Quartile Q1, Q2, Q3, Q4, NQA(No quartile RQ1, RQ3
JCR** Quartile assigned) RQ1, RQ3
Reliability reporting Stability No mentioned, Mentioned only, RQ2
Internal Consistency Referencing (a previous study), RQ2
Equivalence Details are provided
RQ2
Scalability RQ2
Validity reporting Face Validity No mentioned, Mentioned only, RQ2
Content Validity Referencing (a previous study), RQ2
Criterion Validity Details are provided
RQ2
Construct Validity RQ2
Association analysis Reported Yes: if the study reported reliability and validity assessments, No: otherwise. R2, RQ3
*SCImago Journal Rank(https://www.scimagojr.com/), **Journal Citation Reports(https://clarivate.com/webofsciencegroup/solutions/journal-citation-reports/).

E. Analysis and classification
Analysis and classification of the studies was carried out
after data extraction. The obtained results were tabulated and
visually summarized as shown in the Sec. IV. A complete list
of the reviewed studies and their corresponding classification
according to the used data extraction template can be accessed
in the following link1.
F. Validity assessment
As suggested in [13] the validity assessment of our sys-
tematic mapping study was conducted by analyzing the main
threats occurring in the following validity types:
• Descriptive validity. In this type of validity, the main
thread is that subjective studies have less descriptive
validity than quantitative ones. However, our study is
based in the count of data using a well structured
template (see Table II). Consequently, we assumed that
this threat is under control.
1
https://docs.google.com/spreadsheets/d/1nyXtwyYMWeMGJkUjL
PU2X2clEQsyM4MKZlc3AOBPMM/edit?usp=sharing
• Theoretical validity. In our study, study selection and
data extraction are the important sources of threats of this
type. However, we employed snowballing sampling
(backward and forward) in order to mitigate the
possibility of not including relevant studies. Besides,
authors reviewed each other steps in order to control the
bias when selecting and extracting data. Regarding the
quality of the sample of studies, it is clear that it is high
since it comes from databases with great coverage of
high quality venues.
• Generalizability. To assess this type of validity, we have
to consider the internal and external generalizability of
the results and methodology. In the first case, it is clear
that results from this research can be generalized in
the context of digital competence assessment in Higher
Education. So, internal generalizability is guaranteed.
External generalizability is not guaranteed since digital
competences is not only measured in the context of
Higher Education. With respect to the methodology, we
consider that it can be generalized both, internally and
externally.
• Interpretive validity. In this case threats may exist here
 because authors have worked together in previous re- at the same time, and those no reporting any assessment at
search. So, it is possible that similar judgments when all. From these plots, it is clear that about 50% of the studies
selecting and analyzing the primary studies aroused. report assessments of reliability and validity at the same time.
• Reproducibility. We consider that reproducibility is guar-
anteed. This is because enough details are provided in
this paper, that is, by following a systematic guide like C. Association analysis
the proposed in [13].
With aims of understanding which demographics and
IV. RESULTS methodological features are associated to those study
In this section we describe the results achieved during reporting (or nor) reliability and validity assessments, we
our research. They were organized according to the research proceed with the Pearson’s Chi-Squared test of independence.
questions guiding this study. This is a non- parametric test widely used to analyze
association between categorical variables [28]. Specifically,
A. Demographic and methodological features we conducted this test for each combination of variables
The results of extracting the demographic and method- representing demographics and methodological features, and
ological features to the selected primary studies are visually the Reported variable.
summarized in Fig. 2. As it may noted in Fig. 2a, most of The results, in terms of χ2 statistic, degree of freedom, p-
the studies were published in 2019 (41.7%). Besides, from value, and Crame´r’s V are listed in Table III. This last
Fig. 2b) they mostly originated in Europe(36.7%) and Asia indicator (Crame´r’s V) quantifies the association (in the
(35.0%). Regarding the participant type (Fig. 2c) we can see range [0, 1]) between variables. The larger this indicator, the
that undergraduate students is the population from which more strong is the association between the variables. As Table III
results have been reported (63.3%). With a 46.7% Social shows, three variables are significantly associate to reporting
Sciences is the most studied discipline. practices (p-value ≤ 0.05): JCR.Quartile, Discipline and
Considering the scope of the study, Fig. 2e indicates that Study.Scope.
administration of an instrument is the most common practice In order to find out which dimension of these three variables
(73.3%). These instruments are mainly composed by cognitive cause such significant associations, we proceed with a post
items (38.3%) for assessing digital competences (Fig. 2f). The hoc analysis of the standardized residuals [29]. Table IV
context where the assessment took place is mostly of the summarized the results for this analysis. It is easy to observe
diagnostic type (45.0%) as it showed in Fig. 2g. that only two dimension were significantly associated with
Regarding the quality of the journal where the study was the reporting practice. In the case of JCR.Quartile, there is a
published, Fig. 2h shows clear differences between SJR and negative association between reporting the assessments and
JCR indicators. While for the SJR indicator the studies are journals without any quartile (NQA). In other words, not
more evenly distributed in the four quartiles, having a low reporting reliability and validity assessment is more likely a
proportion of studies published in journals without any practice of papers published in journal not indexed by JCR.
quartile (3.3%), for JCR ocurrs almost the opposite. Similarly, when studies are devoted to administer only an ex-
Concretely, 75% studies were published in journal without isting instrument, then it is highly probable that no assessment
any quartile assigned according to the JCR indicator. of reliability and validity is reported. Other interesting results
are also observable in Table IV (although not statistically
B. Reporting practices of reliability and validity assessments significant). See for instance that the positive association
In order to summarized both what types of assessments and between published in a journal with JCR quartile and
to what extend the assessments are reported, we relied in the reporting assessments. Additionally, in Social Science studies
bars plot depicted in Fig.3. In the first case (Fig.3a), it is clear is more likely to found this practice than in Humanities.
that the Internal Consistency is the most reported reliability
assessment with almost of the half of the studies providing TABLE III
details. In contrast, Stability, Equivalence and Scalability PEARSON’S CHI-SQUARED TEST AND CRAMER’S V FOR DEMOGRAPHIC AND
assessments are barely mentioned. METHODOLOGICAL VARIABLES VS. Reported VARIABLE.
Regarding validity assessments, Fig. 3b shows that Face 1
Demographic variable χ2 df p-value Cramer’s V
Validity, Content Validity and Construct Validity are the most
JCR.Quartile 18.710* 4 0.001 0.558
frequently reported. However, a large part of the studies Discipline 11.788* 5 0.038 0.443
reporting the first two types of validity assessments did not Study.Scope 8.127* 2 0.017 0.368
provide details. A different situation happens for Construct Assessment.Context 7.784 3 0.051 0.360
Validity. It is also worth noting that Criterion Validity is the Continent 7.535 5 0.184 0.354
less frequent reported assessment. SJR.Quartile 4.855 4 0.302 0.284
A more general summary is provided in Fig. 3c. In this Year 2.336 2 0.311 0.197
case, we divided the studies into two groups: those reporting Participant.Type 1.046 3 0.790 0.132
(at least mentioning) assessments for both reliability and Items.Types 0.024 2 0.988 0.020
validity 1
Degree of freedom, (*) Statistically significant (p-value ≤ 0.05)
 a) Publication Year b) Continent
Europe 36.7%
2019 41.7%
Asia 35.0%
North
America 11.7%
2018 33.3%
South
America 10.0%

Intercontinental 5.0%
2020 25.0%
Africa 1.7%

0 5 10 15 20 25 0 5 10 15 20
n n

c) Participant type d) Discipline

Social
Undergraduate 46.7%
63.3% Sciences
Students
Multidisciplinary 30.0%
Mixed 15.0%
Humanities 8.3%
Natural
Academic 5.0%
15.0% Sciences
Staff
Health
5.0%
Sciences
Post−graduate
Students 6.7%
Engineering 5.0%

0 10 20 30 0 10 20
n n

e) Study Scope f) Items Type

Administration 73.3% Cognitive 38.3%

Both 15.0% Non−cognitive 35.0%

11.7%
Development Both 26.7%

0 10 20 30 40
n 0 5 10 15 20
n
g) Assessment Context
h) Journal Rank

Diagnostic
45.0%

Intervention SJR 25.0% 26.7% 30.0% 3.3%

effectiveness 28.3%
Hypothesis
testing 21.7%
JCR 11.7% 3.3% 75.0%
Proposal
0 5.0% 10 20 0 20 40 60
n n

Quartile Q1
Q2 Q3 Q4 NQA

Fig. 2. Distribution of studies according to the demographic and methodological features.

V. C ONCLUSION paper we addressed two crucial aspects related to the eval-

In a world where information and communication technolo-
gies impact their individuals and relationships, it is necessary
to educate and train in the use of these technologies. Higher
Education plays an important role in this regard [29]. An
important part of this complex process of providing students
and academic staffs with the necessary digital skills is eval-
uating the extent to which they appropriate them. Although
much have been written about digital competences in Higher
Education in the past [15], [16], the evaluation process itself
have received little attention regardless its importance for
guarantying the achievement of meaningful results. In this
uation instrument: the assessment of reliability and validity.
A systematic mapping study covering the period of 2018 to
July, 2020 was conducted in order to answer three research
questions related to: 1) the demographic and methodological
features of scientific literature, 2) typical assessments and
reporting practices, and 3) association of demographics and
methodological features with reporting practices.
From the obtained results, we observed that evaluating
digital competences in Higher Education is a research topic of
increasing interest. Although we covered just the last two
years (including the first half of 2020), data indicates a
sustained
 TABLE IV
a) Reliability POST HOC ANALYSIS BASED ON STANDARDIZED RESIDUALS OF
100 PEARSON’S CHI-SQUARED TEST.
How it is reported
Percentage (%)

75 Variable Dimension Reported=No Reported=Yes

Details are provided
JCR.Quartile Q1 -2.723 2.723
50 Referencing
Q2 -1.719 1.719
Mentioned only
Q3 -1.391 1.391
25 No mentioned
Q4 -1.719 1.719
0 NQA 4.325* -4.325*
Study.Scope Administration 2.765* -2.765*
St Internal Equiv Scal
Consistency
a alence abilit Development -1.113 1.113
bil y
ity Both -2.424 2.424
Reliability type
Discipline Engineering and Tech- 1.837 -1.837
b) Validity nology
100 Humanities 2.415 -2.415
Medical and Health 0.652 -0.652
How it is reported
Percentage (%)

75 Sciences
Details are provided
Multidisciplinary -0.958 0.958
50 Referencing
Natural Sciences 0.652 -0.652
Mentioned only
Social Sciences -1.830 1.830
25 No mentioned
(*) Statistically significant (p-value ≤ 0.05)
0

Face Content Criterion Construct

Validit Validity Validity Validity than to the development new ones. This is in somehow expect since
y
Validity type as noted in [9], [20] several instruments exists for measuring digital
competences in Higher Education. It is interesting to see that these
c) Overall
instruments are mainly composed by items of a cognitive nature,
100
which indicates that digital competences have been more frequently
measure from the perspective of knowledge and skills rather than
Percentage (%)

75
Reported from attitudes. Additionally, diagnostic has been the assessment
50 Yes context more frequent in the studies we reviewed. It means that
No
researchers were more interested in characterizing than in
25
understanding. The analysis of typical assessments and reporting
0
practices revealed that in the case of reliability, Internal Consistency
is the most common reported assessment. A different situation occurs
Reli V B
abili a o for validity, in which Face Validity, Content validity and Construct
Assessment t
ty
d
li
h validity are the assessments more frequently re-
Fig. 3. Distribution of studiesit according to the type assessment of reliability ported. However, the extent of how authors provided
a), validity b) and whether yis reported or not c).
evidences in text varies considerably by the type of
assessment. While for Internal Consistency and Construct
Validity authors provided details very often, this is not the
growth rate as the years go by. Another important pattern we case for Face Validity and Content Validity . In these latter
found is that studies mostly originate in Europe or Asia and cases it is common that authors just mentioned that they
they frequently assess digital competences to undergraduate conduct them, that is, without any additional details. One
students. Regarding the discipline, the most studied one was possible explanation of this phenomena is that internal
Social Sciences, which includes Educational Sciences. These consistency and Construct Validity can be assessed from the
results are consistent with previous studies like [16] in the collected data and statistical methods, so it is relatively easy to
context of digital competences and with similar reviews in the obtain. In contrast, Face Validity and Content Validity require
field of e-learning [30]. With respect to the quality of the the presence of participants and experts, which is more
venue where the study was published we observe different difficult to achieve.
results from the SJR and JCR indicators. Overall, studies Overall, almost the half of the studies did not reported
published in journal without JCR quartile predominate. It reliability and validity assessments at the same time. These
indicate that current literature is most present in journals with results indicate that serious issues exists about this important
lower quality and reputation. practice. It seems however that this is not a problem typical to
Regarding the methodological aspects, we have seen that digital competence evaluation in Higher Education, but more
studies are mainly focused in administering instruments rather general one for Educational Sciences. Studies like [31] in the
context of medical education and more recently [32] in special
education alerted about similar problems.
In order to get insights about what demographic and
methodological features of the studies are more associate to
the practice of reporting (or not) these assessments, we con-
ducted specific statistical analysis. The results suggest that the
habit of not reporting reliability and validity assessments are
more likely associated to studies focused on measuring only
(administration) and to those published in journals without a
quartile according the JCR indicator. In other words, studies
published reputable journals or involving the development of
a new instrument, are more susceptible to include these type
of assessments.
In summary, results of this systematic mapping study are
a warning sign about reporting reliability and validity as-
sessment in the context of digital competences evaluation in
Higher Education. In addition to our suggestion of including
more evidence of this type of assessments in future studies,
we would like to highlight important opportunities for further
research. First, with respect to primary studies, we suggest
studying those populations that have been little or not reported
in the literature. This is the case of those coming from
South America and Africa. Academic staff and post-graduate
students are also in this group of populations. Secondly,
future secondary studies (like ours) should extend the results
obtained here in three main directions. First, by including
other variables with aims to get more insights about what
cause this phenomena, second, by expanding the number of
studies in order to get a more representative sample, and third,
by exploring other aspects related to the evaluation of digital
competences in Higher Education. Our future works will be
oriented to develop these research opportunities.
REFERENCES
[1] R. L. Konsbruck, “Impacts of Information Technology on Society
in the new Century,” Structure, pp. 1–6, 2002. [Online]. Available:
https://www.zurich.ibm.com/pdf/Konsbruck.pdf
[2] S. Makridakis, “The forthcoming Artificial Intelligence (AI) revolution:
Its impact on society and firms,” Futures, vol. 90, pp. 46–60, 2017.
[3] A. Ferrari, “Digital Competence in Practice: An Analysis of Frame-
works,” Joint Research Centre of the European Commission., Tech.
Rep., 2012.
[4] E. E. Gallardo-Echenique, J. M. de Oliveira, L. Marque´s, and F.
Esteve- Mon, “Digital competence in the knowledge society,” Journal
of Online Learning and Teaching, vol. 11, no. 1, p. 1, 2015.
[5] R. J. Krumsvik, “Teacher educators’ digital competence,” Scandinavian
Journal of Educational Research, vol. 58, no. 3, pp. 269–280, 2014.
[6] K. Ala-Mutka, Y. Punie, and C. Redecker, “Digital competence for life-
long learning,” European Commission, Joint Research Centre, Institute
for Prospective Technological Studies, p. 10, 2008.
[7] D. Bandalos, Measurement Theory and Applications for the Social Sci-
ences, ser. Methodology in the Social Sciences. Guilford Publications,
2018.
[8] K. Bannigan and R. Watson, “Reliability and validity in a nutshell,”
Journal of Clinical Nursing, vol. 18, no. 23, pp. 3237–3243, 2009.
[9] M. P. Prendes, I. Gutie´rrez, and F. Mart´ınez, “Competencia digital:
una necesidad del profesorado universitario en el siglo XXI,” Revista de
Educacio´n a Distancia (RED), no. 56, jan 2018.
[10] R. S. Rivas, P. Novoa-Herna´ndez, and R. S. Rodr´ıguez, “Evaluation
of the presence of digital competences in higher education institutions,”
RISTI - Revista Iberica de Sistemas e Tecnologias de Informacao, vol.
2019, no. E21, pp. 23–36, 2019.
[11] F. J. Palacios Hidalgo, M. E. Go´mez Parra, and C. A. Huertas
Abril, “Digital and media competences: Key competences for efl
teachers,” Teaching English with Technology, vol. 20, no. 1, pp. 43–59,
2020.
[12] R. O. Mueller and T. R. Knapp, “Reliability and Validity,” in The
Reviewer’s Guide to Quantitative Methods in the Social Sciences,
2nd ed., G. R. Hancock, L. M. Stapleton, and R. O. Mueller, Eds.
Routledge, 2019, pp. 397–401.
[13] K. Petersen, S. Vakkalanka, and L. Kuzniarz, “Guidelines for conduct-
ing systematic mapping studies in software engineering: An update,”
Information and Software Technology, vol. 64, pp. 1–18, 2015.
[14] L. Iloma¨ki, S. Paavola, M. Lakkala, and A. Kantosalo, “Digital
com- petence – an emergent boundary concept for policy and
educational research,” Education and Information Technologies, vol.
21, no. 3, pp. 655–679, aug 2014.
[15] F. Pettersson, “On the issues of digital competence in educational con-
texts – a review of literature,” Education and Information Technologies,
vol. 23, no. 3, pp. 1005–1021, 2018.
[16] M. Spante, S. S. Hashemi, M. Lundin, and A. Algers, “Digital compe-
tence and digital literacy in higher education research: Systematic
review of concept use,” Cogent Education, vol. 5, no. 1, pp. 1–21,
2018.
[17] N. Tsankov and I. Damyanov, “Education majors’ preferences on
the functionalities of e-learning platforms in the context of blended
learning,” International Journal of Emerging Technologies in Learning
(iJET), vol. 12, no. 05, p. 202, may 2017.
[18] L. Johnson, A. Levine, R. Smith, and S. Stone, The 2010 Horizon
Report. Austin, Texas: The New Media Consortium, 2010.
[19] S. Carretero, R. Vuorikari, and Y. Punie, The Digital Competence
Framework for Citizens With Eight Proficiency Levels and Examples
of Use. European Union, 2017.
[20] P. M. Henr´ıquez Coronel, M. Gisbert Cervera, and
I. Ferna´ndez Ferna´ndez, “La evaluacio´n de la competencia
digital de los estudiantesuna revisio´n al caso latinoamericano,”
Chasqui: Revista Latinoamericana de Comunicacio´n, no. 137, pp. 93–
112, 2018.
[21] P. F. Gibson and S. Smith, “Digital literacies: preparing pupils and
students for their information journey in the twenty-first century,”
Information and Learning Science, oct 2018.
[22] C. Reis, T. Pessoa, and M. Gallego-Arrufat, “Literacy and digital
competence in higher education: A systematic review,” REDU. Revista
de Docencia Universitaria, vol. 17, no. 1, pp. 45–58, 2019.
[23] J. A. Lopez Nun˜ez, M. N. Campos Soto, I. Aznar Diaz, and C.
Ro- driguez Jimenez, “Digital competence of the teaching staff to attend
to students with learning difficulties. a theoretical review,” Revista
Electro´nica Interuniversitaria de Formacio´n del Profesorado, vol.
23, no. 2, pp. 143–154, 2020.
[24] A. Sanchez-Caballe, M. Gisbert-Cervera, and F. Esteve-Mon, “The
digital competence of university students: a systematic literature
review,” Aloma, vol. 38, no. 1, pp. 63–74, 2020.
[25] N. Lopes Pereira, H. Aisenberg Ferenhof, and F. J. Spanhol, “Strategies
for the management of digital competences in higher education: a review
in the literature,” RELATEC: Revista Latinoamericana de Tecnolog´ıa
Educativa, vol. 18, no. 1, pp. 71–90, 2019.
[26] M. J. Grant and A. Booth, “A typology of reviews: an analysis of
14 review types and associated methodologies,” Health Information &
Libraries Journal, vol. 26, no. 2, pp. 91–108, may 2009.
[27] J. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. Page,
and V. Welch, Cochrane Handbook for Systematic Reviews of
Interventions, ser. Wiley Cochrane Series. Wiley, 2019. [Online].
Available: https://books.google.cl/books?id=G1SyDwAAQBAJ
[28] T. M. Franke, T. Ho, and C. A. Christie, “The chi-square test,”
American Journal of Evaluation, vol. 33, no. 3, pp. 448–458, nov
2011.
[29] D. Sharpe, “Chi-square test is statistically significant: Now what?”
Practical Assessment, Research, and Evaluation, vol. 20, no. 8, pp. 1–
10, 2015.
[30] A. Granic´ and N. Marangunic´, “Technology acceptance model in
ed- ucational context: A systematic literature review,” British Journal of
Educational Technology, vol. 50, no. 5, pp. 2572–2593, 2019.
[31] R. Speyer, W. Pilz, J. V. D. Kruis, and J. W. Brunings, “Reliability and
validity of student peer assessment in medical education: A systematic
review,” Medical Teacher, vol. 33, no. 11, pp. e572–e585, oct 2011.
[32] S.-H. Lee, Y.-R. Kim, and J. Park, “A review of instruments measuring
special educators' competencies,” British Journal of Special Education,
may 2020.