Article

Nordic Journal of Nursing Research

2018, Vol. 38(1) 28–37
! The Author(s) 2017
Critical thinking, research utilization and Reprints and permissions:
sagepub.co.uk/journalsPermissions.nav
barriers to this among nursing students DOI: 10.1177/2057158517704398
journals.sagepub.com/home/njn
in Scandinavia and Indonesia

Bodil Wilde-Larsson1,2, Ilyas Aiyub3, Hasan Hermansyah4,

Reidun Hov1,5, Sevald Høye1, Margrethe Valen Gillund1,
Kari Kvigne1,6, Abubakar Suwarni7 and Gun Nordström1

Abstract
The aim of this study was to describe and compare perceptions of critical thinking, attitudes to and availability of research,
research utilization and barriers to this among nursing students in Scandinavia and Indonesia. Data were collected at the
beginning, middle and end of education from nursing students in Norway, Sweden (bachelor’s diploma) and Banda Aceh
(bachelor’s diploma). Critical Thinking and Research Utilization Questionnaires were used along with the Barrier Scale.
Descriptive analyses, comparisons between and within groups were performed. At the end of education, all samples exhibited
positive attitudes to research and the main barrier was related to the setting. Scandinavian students reported higher critical
thinking. Indonesian students perceived greater barriers on two Barrier subscales. No differences were found between the
samples regarding research utilization. Significant changes over time varied among the samples except for the Norwegian
sample. Indonesian students (diploma) exhibited most changes over time. Teachers must support nursing students to strengthen
their critical thinking ability and develop professional competence.

Keywords
barriers to research, critical thinking, Indonesia, nursing students, research utilization, Scandinavia

Accepted: 22 March 2017

includes, for example, patient-centred care, and evidence-

Background
This article focuses on Scandinavian (Norwegian and
Swedish) and Indonesian (Banda Aceh) nursing students’
professional development in terms of critical thinking abil-
ity, attitudes to and availability of research, research util-
ization and barriers to this during their nursing education.
Nursing education in Norway and Sweden differs from
that in Indonesia in some respects. Since 2002 and 1993
respectively,1,2 Norwegian and Swedish nurses have been
educated at bachelor’s level. In Indonesia, nursing educa-
tion ranges from senior high school level, via diploma III
to bachelor’s level.3 Only a minority (1%) of nurses in
Indonesia have a bachelor’s degree.4,5
Newly graduated nurses are required to employ
research-based knowledge combined with clinical experi-
ence and patient values in their practice.6,7 The Institute
of Medicine (IOM) has pointed out that the nursing pro-
fession is important for providing high quality, safe patient
care.8 Recent studies have shown that low professional
competence among nurses leads to increased mortality in
hospital patients.9,10 The Faculty of Quality and Safety
Education for Nurses (QSEN) and The National
Advisory Board have defined quality and safety competen-
cies for nurses.11 This definition is based on the core pro-
fessional competencies described by the IOM8 and
based practice. Moreover, Kajander-Unkuri et al. have
identified competence areas for nursing students within
the EU.12 One area is research utilization.
The use of research findings in practice is an important
cornerstone of high-quality patient care.13 In many coun-
tries, utilization of research findings and evidence-based
practice (EBP) have become important subjects in nursing
education. In recent years nursing curricula have been
developed to a more academic level, despite this the
theory–practice gap still exists.14,15 Forsman et al. reported
that research use occurred in about half or less than half of
1
Inland Norway University of Applied Sciences, Faculty of Public Health,
Elverum, Norway
2
Faculty of Health, Science, and Technology, Department of Health Science,
Karlstad University, Sweden
3
Unsyiah Nursing Faculty, Aceh, Indonesia
4
Politeknik Kesehatan Kemenkes, Aceh, Indonesia
5
Centre for Development of Home Care Services, Hamar Municipality,
Norway
6
Nord University, Department of Nursing, Sandnessjøen, Norway
7
Akademi Keperawatan Ibnu Sina Kota Sabang, Aceh, Indonesia
Corresponding author:
Bodil Wilde-Larsson, Faculty of Health Science and Technology, Department
of Health Sciences, Karlstad University, 651 88 Karlstad, Sweden.
Email: bodil.wilde@kau.se
Wilde-Larsson et al.
the working shifts one and three years post graduation.14
Moreover, Wangensteen et al. found that one fourth of
newly graduated nurses were defined as research users,16
and Andersson et al. showed that newly graduated nurses
are not prepared for research utilization.17 A systematic
review18 regarding barriers to research utilization showed
that the most commonly reported items were related to the
setting and the research subscales. The barriers included,
for example, insufficient time to implement new ideas, and
lack of time to read research reports. The barriers were
consistent over time, geographic location and other condi-
tions, e.g. study setting.
In order to meet the requirements for EBP, nursing stu-
dents need to develop their critical thinking ability, as it
can result in nurses having the necessary skills and dispos-
itions, e.g. habits of mind, attitudes and traits, to support
EBP.13 Facione et al. described an ideal critical thinker as
inquisitive, well informed, open minded, willing to recon-
sider and orderly in complex matters.19 Thus, critical
thinking is an essential component of professional
accountability and quality nursing care. Wangensteen
et al. found that nearly 80% of newly graduated nurses
reported a strong disposition and positive inclination
towards critical thinking.20 However, mixed results are
shown when measuring critical thinking development
over time, i.e. before, in the middle and at the end of nur-
sing education.21
Studies have shown that a majority of nursing students
and newly graduated nurses have a positive inclination
towards critical thinking and positive attitudes to research
utilization. However, low research use and several barriers
to research utilization in practice have also been reported.
Longitudinal studies of nursing students regarding profes-
sional development have been requested. It would there-
fore be of interest to study these matters among nursing
students in different countries. The present study is a part
of a larger collaboration project about life-circumstances,
health status and professional development among nursing
students in Norway, Sweden and Indonesia. This article
highlights quantitative results regarding nursing students’
professional development.
The aim was to describe and compare perceptions of
professional development, i.e. critical thinking ability, atti-
tudes to and availability of research, research utilization
and barriers to this among nursing students in Scandinavia
(Norway, Sweden) and Indonesia (Banda Aceh) during
their nursing education.
Method
A descriptive, comparative, cross-sectional and longitu-
dinal cohort study.
Sample and procedure
The sample consisted of nursing students recruited from
one university college in Norway (bachelor’s), one univer-
sity in Sweden (bachelor’s), one university (bachelor’s) and
two nursing schools (diploma) in Banda Aceh, Indonesia.
29
Data were collected in 2012 (T1), 2013 (T2) and 2014
(T3). A questionnaire package was administered to the
students during the course of ordinary lessons. Teachers,
(not involved in the present study), informed the students
about the study and handed out the envelopes containing
the coded questionnaires. The students returned the ques-
tionnaires in a sealed envelope. This process was carried
out at the beginning (at the end of the first semester – T1),
in the middle (during the fourth semester – T2) and
at the end of their education (at the end of the sixth
semester – T3).
In Norway 206 questionnaire packages were handed out
at T1, 194 at T2 and 183 at T3. The analyses are based on
the following figures: T1 n ¼ 162 (79%), T2 n ¼ 75 (39%)
and T3 n ¼ 88 (48%).
In Sweden 71 questionnaire packages were handed out
at T1, 66 at T2 and 63 at T3. The analyses are based on the
following figures: T1 n ¼ 60 (85%), T2 n ¼ 59 (89%) and
T3 n ¼ 54 (86%).
In Indonesia (bachelor’s level) 127 questionnaire pack-
ages were handed out at T1, 127 at T2 and 141 at T3. The
analyses are based on the following figures: T1 n ¼ 124
(98%), T2 n ¼ 120 (94%) and T3 n ¼ 112 (79%).
In Indonesia (diploma level) 153 questionnaire packages
were handed out at T1, 139 at T2 and 114 at T3. The
analyses are based on the following figures: T1 n ¼ 152
(99%), T2 n ¼ 121 (87%) and T3 n ¼ 114 (100%).
Questionnaire package
The package included three questionnaires intended to
measure professional development: Critical Thinking
Questionnaire (CTQ), Research Utilization Questionnaire
(RUQ), the Barrier Scale and questions about characteris-
tics. The translation process followed the Eurostat
procedure.22
Critical thinking disposition was measured (T1, T2, T3)
by means of the CTQ, specifically developed for the pre-
sent study. The development of the questionnaire was
inspired by the California Critical Thinking Disposition
Inventory19 and Watson and Glaser’s Critical Thinking
Appraisal.23 The CTQ consists of 28 items pertaining to
critical thinking. The items employ a four-point scale:
1 ¼ completely disagree, 2 ¼ partly disagree, 3 ¼ partly
agree, 4 ¼ completely agree. The total score, ranging
from 28 to 112, was calculated by summarizing the item
scores of each respondent. Higher scores indicate a greater
inclination towards critical thinking.
Face and content validity were established by pilot test-
ing of the questionnaire with a group of Norwegian nurse
teachers and Swedish nursing students, after which some
minor changes to the wording were made. Cronbach’s
alpha, was .91 (T1), .66 (T2) and .78 (T3). The CTQ was
developed in Swedish, translated into Norwegian and
Bahasa.
Research utilization was measured (T1, T2, T3) by
means of the RUQ. The RUQ was developed by
Champion and Leach24 and further revised by Pettengrill
et al.25 and Humphris et al.26 Three RUQ indexes were
30
used: ‘Attitudes towards research’ (12 items), ‘Availability
and support to implement research results’ (8 items), and
‘Research use in daily practice’ (9 items) – 29 items in total.
At T1 only the attitude index was utilized. All three indexes
were utilized at T2 and T3. The items employ a five-point
Likert scale: 1 ¼ strongly disagree, 2 ¼ partly disagree,
3 ¼ don’t know/no opinion, 4 ¼ partly agree, and
5 ¼ strongly agree.
The indexes were calculated in two ways. A) The index
scores were summarized and then divided by the number of
items in the index. The index scores ranged from 1 to 5. B)
By summarizing each respondent’s item scores on the
index the scores ranged from 12 to 60, 8 to 40 and 9 to
45 respectively. Higher values indicate a more positive atti-
tude towards research, better availability of research and
more research use. Cronbach’s alpha, for the ‘Attitude
index’ was .74 (T1), .75 (T2) and .79 (T3). The alpha
values for the ‘Availability and support index’ were
.68 (T2) and .65 (T3) and for the ‘Research use index’
.80 (T2) and .76 (T3).
The RUQ had been translated into Swedish27 and
Norwegian.16 In the present study the RUQ was also trans-
lated into Bahasa.
Barriers to using research in practice was measured by
means of the Barrier Scale (T2, T3).28 The instrument con-
sists of 28 items divided into four subscales: the Nurse
subscale (8 items pertaining to nurses’ research values,
skills and awareness); the Setting subscale (8 items pertain-
ing to barriers and limitations); the Research subscale
(6 items pertaining to the quality of the research) and the
Presentation subscale (6 items pertaining to the presenta-
tion and accessibility of the research).
The items have a four-point scale where 1 ¼ to no
extent, 2 ¼ to a small extent, 3 ¼ to a moderate extent
and 4 ¼ to a great extent and a ‘no opinion’ alternative
was offered. The subscales were calculated in the following
ways: A) The subscale scores were summarized and then
divided by the number of items in the subscale. The scores
ranged from 1 to 5. B) Each respondent’s item scores on
the subscale were summarized and the scores ranged from
8 to 32, 8 to 32, 6 to 24 and 6 to 24 respectively. Higher
values indicate greater barriers. Cronbach’s alpha values
were, for the subscale ‘Nurse’ .84 (T2) and .84 (T3), for
‘Setting’ .79 (T2) and .57 (T3), for ‘Research’ .79 (T2) and
.84 (T3) and for ‘Presentation’ .81 (T2) and .76 (T3).
The Barrier Scale has been translated into Swedish18
and into Norwegian.29 The Barrier Scale was also trans-
lated into Bahasa for this study.
Statistical analyses
SPSS version 22.0 was used. Comparisons between the
samples on gender and marital status were made using
Pearson’s chi-square tests. Mean age comparisons were
conducted using one-way analysis of variance (ANOVA),
followed by Scheffé post hoc comparisons.
Between-group examinations at T1, T2 and T3 on the
CTQ, the RUQ and the Barrier Scale were carried out as
follows. Firstly, comparisons based on the raw scores were
Nordic Journal of Nursing Research 38(1)
made using ANOVA, followed by Scheffé tests. Secondly,
the same analyses were performed with analyses of covari-
ance (ANCOVA) to control for gender (as a factor) and
age (as a covariate). ANCOVAs yielding significant
F values were followed by Bonferroni tests for post hoc
comparisons.
Within-group comparisons on the CTQ, the RUQ and
the Barrier Scale across the three assessment occasions
(T1 vs T2 and T1 vs T3 or T2 vs T3) were performed by
means of paired t-tests.
Reliabilities were estimated by Cronbach’s alpha coeffi-
cients. Statistical significance was set at p < .05.
Ethical considerations
The study was approved by the Norwegian Social
Science Data Services (NSD) (No. 29212), the Regional
Ethical Review Board in Uppsala, Sweden (Dnr 2010/
462) and the Ethics Committee of the Nursing Faculty
of Syiah Kuala University Indonesia (Dnr 160811301).
The students gave their consent by completing the
questionnaires.
Results
The results are presented as follows:
Characteristics of the respondents from Norway (N),
Sweden (S), Indonesian bachelor’s (IB) and Indonesian
diploma (ID), descriptive results pertaining to the profes-
sional development scales, the RUQ and the Barrier Scale
at T3, between- and within-group comparisons for the
CTQ, the RUQ and the Barrier Scale over time.
Characteristics
Table 1 shows that the two Indonesian samples are signifi-
cantly younger than the two Scandinavian (Norwegian,
Swedish) samples. Regarding gender, there is an overall
statistically significant difference in the proportions
of men and woman (chi-square [3] ¼ 10.02, p ¼ 0.018).
Pairwise comparisons demonstrated that the Indonesian
bachelor’s sample had a higher proportion of men than
the other three samples.
The RUQ at T3: Descriptive data
All four samples reported positive attitudes towards
research on the ‘Attitude index’ with the following mean
values: 4.0 (N), 4.0 (S), 3.8 (IB) and 3.8 (ID) of a possible
5. Table 2 shows that 50% or more of the respondents
agreed (partly/strongly) on seven to nine items (of 12).
The mean values on the ‘Availability and support index’
were 3.5 (N), 3.1 (S), 3.7 (IB) and 3.6 (ID) of a possible 5.
Table 2 shows that 50% or more of the respondents agreed
(partly/strongly) on zero to seven items (of 8). The mean
values on the ‘Research use index’ were 3.6 (N), 3.5 (S), 3.6
(IB) and 3.6 (ID) of a possible 5. Table 2 shows that 50%
or more of the respondents agreed (partly/strongly) on
four to seven items (of 9).
Wilde-Larsson et al.
Table 1. Characteristics of the Norwegian, Swedish and Indonesian nursing students at the beginning of their nursing education (Time 1).
Mean values (M), Standard deviations (SD), percentages (%), p-values and post hoc comparisons are shown.
Norway Sweden
(bachelor’s) (A) (bachelor’s) (B)
n ¼ 162 n ¼ 60
Age Mean (SD) 24.7 (6.6) 24.9 (6.9)
Gender (%)
Women 86.4 84.8
Men 13.6 15.2
Marital status (%)
Married/Cohabiting 40.9 55.9
Living alone 59.1 44.1
a
ANOVA, post hoc comparisons (Scheffé); bChi-square
The Barrier Scale at T3: Descriptive data
The mean values on the Nurse subscale were 2.7 (N),
1.4 (S), 2.2 (IB) and 2.0 (ID) of a possible 5. Table 3
shows that 50% or more of the respondents reported mod-
erate/great barriers on one, two and three of 8 items. The
mean values on the Setting subscale were 1.7 (N), 1.7 (S),
2.3 (IB) and 2.2 (ID) of a possible 5. Table 3 shows that
50% or more of the respondents reported moderate/great
barriers on four and five of 8 items. The mean values on
the Research subscale were 1.3 (N), 0.9 (S), 2.3 (IB) and 2.0
(ID) of a possible 5. Table 3 shows that 50% or more of
the respondents reported moderate/great barriers on one,
two and four of 8 items. The mean values on the
Presentation subscale were 1.8 (N), 1.8 (S), 2.3 (IB) 2.2
(ID) of a possible 5. Table 3 shows that 50% or more of
the respondents reported moderate/great barriers on one,
three and four of 6 items.
The CTQ, the RUQ and the Barrier Scale:
Between-group comparisons
Table 4, which is based on raw scale scores, reveals that the
Indonesian samples scored significantly lower than the
Norwegian (4 of 10 comparisons) and the Swedish samples
(7 of 10) on scales designed to measure critical thinking
and research utilization. Regarding barriers to research,
the Scandinavian samples scored lower, indicating fewer
barriers than the Indonesian samples on five comparisons
(of 8). Almost no differences were found on the profes-
sional development scales between the Indonesian bach-
elor’s and Indonesian diploma samples (3 of 17), or
between the Norwegian and Swedish samples (3 of 18).
Table 5 shows the same between-group comparisons as
Table 4, now controlled for age and gender. Only statistic-
ally significant differences are presented in this table. As
can be seen, a different picture emerges, where most of the
above-mentioned statistically significant differences
between the two Indonesian and the Scandinavian samples
disappear. Critical thinking constitutes an exception.
31
Indonesia Indonesia
(bachelor’s) (C) (diploma) (D)
n ¼ 124 n ¼ 152 p-values Scheffé/Chi-square
18.5 (0.8) 18.6 (1.1) <.001a A vs C, B vs C
A vs D, B vs D
.018b A vs C, B vs C,
72.6 81.7 C vs D
27.4 8.3
<.001b
0.9
100 99.1
On the CTQ, significant differences were noted on all
three assessments occasions between the Indonesian
Diploma sample and the two Scandinavian samples, with
the latter scoring higher (see also Table 4), indicating a
higher inclination towards critical thinking. Significant dif-
ferences in the same direction were also found between the
Indonesian bachelor’s and the two Scandinavian samples
at T1. No significant differences were observed between the
two Indonesian samples or between the two Scandinavian
samples (not shown in the table).
As regards research utilization, the Swedish sam-
ple scored lower on the attitude index at T2, indicating a
less positive attitude towards research compared to
the Indonesian bachelor’s sample. On the Barrier Scale,
significant differences were noted on the Nurse and
Research subscales at T3. The Scandinavian samples
scored lower, i.e. perceived barriers in this area to a
lesser extent.
The CTQ, the RUQ and the Barrier Scale:
Within-group comparisons
Table 6 summarizes within-group comparisons across the
three measurement occasions (T1 vs T2 and T1 vs T3 or T2
vs T3). Only statistically significant differences are pre-
sented in this table. Most significant changes were found
in the Indonesian diploma sample. Thus, their scores (see
also Table 4) indicate increasing levels of critical thinking
disposition, less positive attitudes towards research and
more perceptions of barriers related to the Nurse,
Research and Presentation subscales. The Indonesian
bachelor’s sample also exhibited a higher level of critical
thinking disposition over time (T1 vs T2 and T1 vs T3) and
more perceived barriers (T3 vs T2) related to the Nurse and
Research subscales. The Swedish sample shows an
increased disposition for critical thinking between T1 and
T3 and less positive attitudes towards research between T1
and T2, but no significant difference between T1 and T3.
No significant changes over time were found on any of the
scales in the Norwegian sample.
32 Nordic Journal of Nursing Research 38(1)

Table 2. Norwegian, Swedish and Indonesian nursing students’ perceptions of research utilization at the end of their nursing education
(Time 3) – percentages (%) are shown.

Norway Sweden Indonesia Indonesia

(bachelor’s) (bachelor’s) (bachelor’s) (diploma)
Research Utilization Questionnaire (%) (%) (%) (%)

Attitude index
I wish to change my practice to make it based on research 82 77 87 75
I want to base my practice on research 83 91 87 80
Clinical practice should be based on research 84 94 83 86
Participating in research is waste of time 9 11 29 15
Understanding research helps me professionally 89 87 88 89
I think research is interesting 80 66 87 88
Research is stimulating 70 59 84 70
Research is understandable 85 79 71 59
Research is a dull, boring project 16 26 28 23
It is not relevant to use research findings in my day-to-day practice 3 2 37 19
Basing clinical practice on research is time-saving 49 52 51 44
Research findings are too complex to use in practice 18 8 51 26
Availability and support index
The clinical team I work with supports research utilization 53 36 70 61
My unit manager supports research utilization 33 19 71 57
The quality of research is not so good that it can be used in practice 9 9 37 15
I have access to research findings where I work 53 19 61 40
I have time to read about research while I am on duty 49 9 62 47
Research is performed in my work place 27 17 74 50
Research is performed in my community 33 40 81 74
Education in research is carried out in my work place 22 8 82 63
Research use index
I base my practice on research 75 76 78 66
My clinical practice is based on research 70 66 73 68
I do not use research in my day-to-day practice 9 13 39 33
I use research findings in my clinical practice 77 68 73 57
I apply research findings in my clinical practice 66 56 71 63
I help others to apply research in clinical practice 49 26 73 68
I use research to guide my clinical practice 58 47 72 63
I cannot apply research findings in my clinical practice 14 2 41 31
I seek out research related to my clinical practice 55 32 71 75
Bold figures show that 50% or more of the respondents agreed partly/strongly to the item.

Discussion
There is a global challenge for nurses to provide care based
on current scientific knowledge,6 and studies have shown
that EBP results in improved quality of care, better patient
outcomes and decreased costs.30 It is of the utmost import-
ance that educational institutions prepare nursing students
by providing them with training in research methods,
knowledge-seeking and critical appraisal.
The descriptive analyses demonstrated that at the end of
their education all samples reported positive attitudes
towards research on most items on the RUQ ‘Attitude
index’. These results are in line with results among
newly graduated nurses.16 According to Saunders and
Vehviläinen-Julkunen, nurses reported positive attitudes
towards EBP, but did not use best evidence in practice.31
Regarding the RUQ ‘Research use index’, the
respondents reported fairly high values on half of the
items. These figures are somewhat higher than those
reported among newly graduated nurses by
Wangensteen et al.16 Forsman et al. reported that 34%
of the nursing students intended to use research on more
than half or almost every working shift in their future
clinical practice.32
The ‘Availability and support index’ revealed the need
for improvement in several areas, e.g. support from the
unit manager. Further, support to implement research
findings has been shown to be a strong predictor of
research use.16 Support from unit managers is important
for the development of nursing care and there is a need to
create a culture that facilitates EBP.33
Wilde-Larsson et al. 33

Table 3. Norwegian, Swedish and Indonesian nursing students’ perceptions of barriers to research at the end of their nursing education
(Time 3) – percentages (%) are shown.

Norway Sweden Indonesia Indonesia

(bachelor’s) (bachelor’s) (bachelor’s) (diploma)
Barriers Scale (%) (%) (%) (%)

Nurse subscale
The nurse is isolated from knowledgeable colleagues 23 53 60 54
with whom to discuss research
The nurse is unaware of the research 37 28 46 38
The nurse sees little benefit for self 43 44 45 39
There is not a documented need to change practice 21 19 46 32
The nurse does not feel capable of evaluating the research 51 85 49 45
The nurse does not see the value of research for practice 25 23 51 45
The nurse feels the benefits of changing practice will be minimal 39 38 51 43
The nurse is unwilling to change/try new ideas 28 19 13 37
Setting subscale
The nurse does not have time to read research 81 96 37 48
There is insufficient time on the job to implement new ideas 86 85 62 59
The facilities are inadequate for implementation 62 76 67 65
The nurse does not feel she/he has enough authority to 48 84 53 57
change patient care procedures
The nurse feels results are not generalizable to own setting 32 47 54 51
Physicians will not cooperate with implementation 61 63 44 39
Other staff are not supportive of implementation 39 45 36 42
Research subscale
The research has not been replicated 26 64 59 52
Research reports/articles are not published fast enough 40 26 51 49
The nurse is uncertain whether to believe the results of the research 48 41 46 31
The literature reports conflicting results 38 52 50 46
The research has methodological inadequacies 58 6 53 39
The conclusions drawn from the research are not justified 6 3 47 34
Presentation subscale
Research reports/articles are not readily available 48 52 47 53
The relevant literature is not compiled in one place 50 71 56 56
Implications for practice are not made clear 32 26 65 56
The statistical analyses are not understandable 34 71 55 53
The research is not reported clearly and readably 34 50 38 46
The research is not relevant to the nurse’s practice 5 31 41 42
Bold figures show that 50% or more of the respondents agreed partly/strongly to the item.

Comparison of the results of the professional develop-
ment scales between the samples was initially based on the
raw scores showing that the Indonesian samples scored
significantly lower than the Scandinavian samples on the
CTQ and the RUQ, in addition to reporting greater bar-
riers to research utilization. After controlling for age and
gender most of the significant differences disappeared.
Critical thinking constituted an exception. One reason
for the lower CTQ values might be that the Indonesian
samples were younger. This result is supported by
Shinnick and Woo who found that age positively predicted
total critical thinking skills scores.34 However, contradict-
ing results have been reported by other authors.35,36
According to within-group comparisons related to the
RUQ ‘Attitude index’, the Indonesian diploma sample
reported less positive attitudes over time. This might be
due to these students developing a more critical view on
this aspect, which could be supported by the fact that they
scored significantly higher on the CTQ over time.
Regarding the Barrier scale, the descriptive analyses indi-
cated that the main barriers were related to the setting.
This was also shown in a systematic review by Nilsson
Kajermo et al.,18 and in studies by Tan et al.,37 and
Wang el al.38
When age and gender were controlled for, significant
differences were noted on the Barrier subscales Nurse
and Research. The Indonesian samples perceived barriers
to a greater extent. One reason might be limited resources,
e.g. teacher competence and availability of learning mater-
ial.39,40 According to the within-group comparisons of the
34 Nordic Journal of Nursing Research 38(1)

Table 4. Norwegian, Swedish and Indonesian nursing students’ perceptions of critical thinking, research utilization and barriers to research
during their nursing education. Mean values (M) Standard deviations (SD), p-values (ANOVA) and post hoc comparisons (Scheffé) are shown.

Norway Sweden Indonesia Indonesia

(bachelor’s) (A) (bachelor’s) (B) (bachelor’s) (C) (diploma) (D)

Questionnaire M (SD) M (SD) M (SD) M (SD) p-values Scheffé

Critical thinking (Score 28–112)

Time 1 87.9 (6.0) 86.4 (6.5) 61.3 (7.0) 59.7 (5.9) <.001 A vs C, B vs C,
A vs D, B vs D
Time 2 87.6 (5.3) 86.4 (6.9) 81.6 (6.7) 78.8 (7.0) <.001 A vs C, B vs C,
C vs D, A vs D, B vs D
Time 3 87.4 (5.8) 88.5 (7.4) 79.1 (7.1) 78.6 (7.8) <.001 A vs C, B vs C,
A vs D, B vs D
Research utilization
Attitude index (Score 12–60)
Time 1 47.6 (6.7) 46.3 (6.5) 46.8 (5.5) 48.0 (5.5) .273
Time 2 49.0 (5.7) 43.2 (7.1) 47.1 (5.4) 45.6 (6.9) <.001 B vs C, A vs B
Time 3 47.4 (7.5) 47.6 (7.5) 45.3 (6.1) 45.4 (5.4) .047
Availability/support index (Score 8–40)
Time 2 28.0 (4.3) 25.4 (5.0) 28.7 (4.4) 29.3 (4.4) <.001 B vs C, A vs B, B vs D
Time 3 27.7 (4.9) 24.8 (3.6) 29.8 (3.8) 28.7 (3.8) <.001 A vs B, B vs C,
B vs D, A vs C
Research use index (Score 9–45)
Time 2 32.7 (6.0) 30.4 (5.1) 33.7 (5.5) 32.6 (5.4) .004 B vs C
Time 3 32.2 (6.1) 31.9 (5.6) 32.4 (4.9) 32.3 (4.6) .949
Barriers scale
Nurse subscale (Score 8–32)
Time 2 13.3 (6.3) 11.6 (6.7) 13.6 (6.1) 14.1 (6.6) .103
Time 3 13.5 (7.8) 11.3 (5.8) 17.6 (6.0) 15.9 (5.9) <.001 A vs C, B vs C, B vs D
Setting subscale (Score 8–32)
Time 2 14.3 (6.7) 14.5 (6.8) .908
Time 3 13.4 (6.6) 13.7 (5.8) 18.1 (6.2) 17.4 (6.0) <.001 A vs C, B vs C,
A vs D, B vs D
Research subscale (Score 6–24)
Time 2 7.6 (4.6) 5.9 (4.8) 10.4 (5.0) 8.6 (5.4) <.001 A vs C, B vs C, C vs D
Time 3 7.9 (4.9) 5.1 (4.4) 13.8 (4.7) 11.9 (4.6) <.001 A vs C, B vs C,
C vs D, A vs D, B vs D
Presentation subscale (Score 6–24)
Time 2 12.2 (4.0) 11.7 (4.4) 12.5 (5.6) 10.2 (5.3) .003 C vs D
Time 3 10.6 (4.0) 11.0 (4.2) 13.6 (4.4) 13.0 (4.8) <.001 A vs C, B vs C
Statistically significant p-values are shown in italics.

Barrier Scale, most changes over time were found in the
Indonesian samples, who perceived greater barriers related
to the subscales Research and Presentation (only the dip-
loma sample). This was not the case among the
Scandinavian samples. One interpretation might be that
during their nursing education the Indonesian students
developed a more critical view over time.
To our knowledge, no longitudinal studies using the
Barrier Scale have been carried out. However, it is possible
to relate our findings to the results presented in an earlier
systematic review showing no significant differences in
mean values when comparing studies on barriers to
research utilization.18
Regarding critical thinking, the between-group com-
parisons revealed that the Scandinavian samples scored
higher on critical thinking compared to the Indonesian
samples. However, the within-group comparisons showed
significantly higher levels of critical thinking over time
among all samples except for the Norwegian sample.
Increasing levels of critical thinking scores over time
among nursing students are also reported by Pitt et al.21
One reason for the differences in critical thinking and
barriers to research utilization might be a divergence in the
competence level of Scandinavian and Indonesian teachers.
In the Scandinavian countries there are more teachers with
master’s degrees and PhDs, while in Indonesia nurses are
Wilde-Larsson et al. 35

Table 5. Swedish, Norwegian and Indonesian nursing students’ perceptions of critical thinking, research utilization and barriers to research
during their nursing education (between-group comparisons). Mean differences (M diff), controlled for age and gender (ANCOVA). Only
statistically significant differences are shown.

Sweden vs Indonesia Sweden vs Indonesia Norway vs Indonesia Norway vs Indonesia

(bachelor’s) (diploma) (bachelor’s) (diploma)

Questionnaire M diff p-values M diff p-values M diff p-values M diff p-values

Critical thinking
Time 1 26.739 <.001 24.833 <.001 27.147 <.001 25.241 <.001
Time 2 8.782 <.001 7.548 <.001
Time 3 11.780 <.001 10.294 .005
Research utilization
Attitude index
Time 2 6.862 .034
Barriers scale
Nurse subscale
Time 3 6.459 .030
Research subscale
Time 3 7.040 <.001 6.610 .005 5.100 .024
Statistically significant p-values are shown in italics.

Table 6. Swedish and Indonesian nursing students’ perceptions of critical thinking, research utilization and barriers to research during their
nursing education over time within groups. Mean differences (M diff) Standard deviations (SD) and p-values (paired t-test) are shown. Only
statistically significant differences are shown.

Sweden (bachelor’s) Indonesia (bachelor’s) Indonesia (diploma)

Questionnaire M diff SD p-values M diff SD p-values M diff SD p-values

Critical thinking
Time 1 and 2 19.726 (9.744) <.001 18.532 (9.299) <.001
Time 1 and 3 2.357 (5.846) .013 18.259 (11.502) <.001 19.898 (10.631) <.001
Research utilization
Attitude index
Time 1 and 2 3.480 (6.081) <.001 2.409 (8.721) .011
Time 1 and 3 3.164 (5.082) <.001
Barriers Scale
Nurse subscale
Time 2 and 3 4.320 (8.749) <.001 2.240 (8.962) .025
Research subscale
Time 2 and 3 3.120 (7.696) <.001 3.888 (6.715) <.001
Presentation subscale
Time 2 and 3 2.771 (7.157) <.001
Statistically significant p-values are shown in italics.

often offered teaching positions immediately after gradua-
tion.4,5,41 Consequently, they have limited clinical experi-
ence,40 which in turn may result in reduced potential
for critically observing reality. In addition, the
Indonesian education system is influenced by the
American curriculum and thus often uses English-language
literature. Moreover, due to limited resources there might
not be enough literature available for the students.
Consequently, the students have to memorize facts that
are read aloud by the teacher. According to Diekelmann
and Smyth it is well known that passive learning methods
like this limit critical thinking development.42 This does
not promote either critical thinking43 or the development
of evidence-based practice, both of which are important
for high-quality patient care. As studies have demonstrated
that critical thinking is related to research utilization, it is
recommended that critical thinking be recognized and
strengthened both in nursing education and clinical prac-
tice.16,44 Several studies have been conducted in order to
develop nursing students’ critical thinking abilities with
36
variable results.45,46 However, problem-based learning
(PBL), simulation, and concept mapping seem to be the
most common educational interventions used to promote
critical thinking development.
Methodological considerations
This study focuses on perceptions of critical thinking, atti-
tudes to and availability of research, as well as research
utilization and barriers to this among nursing students in
two samples from Scandinavia and two samples from
Indonesia during their nursing education. The RUQ and
the Barrier Scale have been validated in several other stu-
dies and were pilot tested for the target groups in this
study. The translation process followed a well-established
procedure,22 which is considered a strength when different
countries are involved. The reliability of the RUQ indexes
and the Barrier subscales was satisfactory for most scales.
The CTQ was specifically developed for the present study
and face and content validity was established. Cronbach’s
alpha values were satisfactory on two occasions and some-
what low on one. The Cronbach’s alpha coefficients pre-
sented in the article are based on the total of all four
samples. If computed within each sample within each
assessment point (72 computations all together), fairly
similar, yet slightly higher, reliability values were obtained.
The response rate is considered high for all samples, with
the exception of the Norwegian sample at T2 and T3.
According to Polit and Beck, the response rates, apart
from two of the Norwegian ones, are satisfactory.47
Students were recruited from a limited number of settings.
Future studies could improve the generalizability by recruit-
ing samples from more settings in different countries.
As the data were self-reported, the assessments of pro-
fessional development may be influenced by the wish to
provide socially desirable responses. This might be a
threat to validity and should be kept in mind when inter-
preting the findings.9
Summary and conclusions
At the end of their education, all samples exhibited positive
attitudes towards research and the main barrier was related
to the setting. The Scandinavian samples reported a higher
inclination towards critical thinking. The Indonesian sam-
ples perceived greater barriers on the Barrier subscales
Nurse and Research. No differences were found between
the samples in terms of research utilization. Significant
changes regarding professional development over time
were found to varying degrees among all samples except
for that from Norway. The Indonesian diploma sample
showed the most changes over time.
It is important for teachers to support nursing students so
that their critical thinking ability is strengthened, thus
enabling them to develop their professional competence.
A suggestion for future research is to focus on strategies
to increase professional development among nursing stu-
dents. It would also be of interest to explore teachers’ level
of competence related to evidence-based practice.
Nordic Journal of Nursing Research 38(1)
Acknowledgements
The authors wish to thank Jari Appelgren for valuable statistical
advice and assistance and Monique Federsel for reviewing the
English language.
Funding
This research received no specific grant from any funding agency
in the public, commercial, or not-for-profit sectors.
Conflict of interest
The authors declare that there is no conflict of interest.
References
1. Kirke-utdannings – og forskningsdepartementet. Gjør din
plikt – krev din rett [Do your duty – demand your rights].
St.meld. nr. 27 (2000–2001). Oslo: Kvalitetsreform av
høyere utdanning, 2001.
2. SFS, 1992. Higher Education Act. The Higher Education
Ordinance, Stockholm, Sweden, http//:www.riksdagen.se/
sv/Dokument-lagar/Lagar/svenskforfattningssamling/
Hogskolelag-19921434_sfs-1992-1434 (1992, accessed 12
December 2014).
3. Hennessy D, Hicks C, Hilan A, et al. A methodology for
assessing the professional development needs of nurses and
midwifes in Indonesia: paper 1 of 3. Hum Resource Health
2006; 4(8): 1–8.
4. SETYOWATI. The health service and nursing education in
Indonesia 2009. Paper presented at the international seminar
After accepting Indonesian care workers in Japanese Society:
Discussing their problems and solutions, Fukuoka City, Japan,
23 May 2009, http://catalog.lib.kyushu-u.ac.jp/handle/2324/
17939/p199.pdf (2009, accessed 20 February 2017).
5. Law of the Republic of Indonesia. 2014: Number 38 on
Nursing Act, http://www.observatorisdmkindonesia.org/wp-
content/uploads/2015/01/3.-Indonesian-Nursing-Act-No.-
38-year-2014 (2014, accessed 20 February 2017).
6. World Health Organization. Nursing and midwifery: human
resources for health. Geneva: Global Standards for the initial
education of professional nurses and midwives, 2009 (WHO/
HRH/HPN/08.6), http://www.who.int/hrh/nursing_midwif-
ery/hrh_global_standards_education.pdf (2009, accessed 20
February 2017).
7. International Council of Nurses. Reforming primary health
care: a nursing perspective. Geneva: International Council
of Nurses, 2012.
8. Institute of Medicine. Health professions education: a
bridge to quality. Washington, DC: National Academies
Press, 2003.
9. Estabrooks CA, Midodzi WK, Cummings GG, et al. The
impact of hospital nursing characteristics on 30-day mortal-
ity. Nurs Res 2005; 54(2): 74–84.
10. Aiken LH, Sloane L, Bruyneel K, et al. Nurse staffing and
education and hospital mortality in nine European countries:
a retrospective observational study. Lancet 2014; 383(9931):
1824–1830.
11. Cronenwett L, Sherwood G, Barnstener J, et al. Quality and
safety education for nurses. Nurs Outlook 2007; 53(3):
122–131.
12. Kajander-Unkuri S, Salminen L, Saarikoski M, et al.
Competence areas of nursing students in Europe. Nurse
Educ Today 2013; 33(6): 625–633.
13. Profetto-McGrath J. Critical thinking and evidence-based
practice. J Prof Nurs 2005; 21(6): 364–371.
Wilde-Larsson et al.
14. Forsman H, Gustavsson P, Ehrenberg A, et al. Research use
in clinical practice: extent and patterns among nurses one and
three years postgraduation. J Adv Nurs 2009; 65(6):
1195–1206.
15. Ajani K and Moez S. Gap between knowledge and practice
in nursing. Proc Soc Behav Sci 2011; 15: 3927–3931.
16. Wangensteen S, Johansson IS, Björkström M, et al. Research
utilization and critical thinking among newly graduated
nurses: predictors for research use. A quantitative cross-sec-
tional study. J Clin Nurs 2011; 20: 2436–2447.
17. Andersson N, Cederfjäll C, Jylli L, et al. Professional roles
and research utilization in pediatric care: newly graduated
nurses’ experiences. Scand Caring Sci 2007; 21: 91–97.
18. Nilsson Kajermo K, Boström AM, Thompson DS, et al. The
Barrier Scale – the barriers to research utilization scale: a
systematic review. Impl Sci 2010; 5(32): 1–22.
19. Facione NC, Facione PA and Sanchez CA. Critical thinking
disposition as a measure of competent clinical judgment: the
development of the California Critical Thinking Disposition
Inventory. J Nurs Educ 1994; 33(8): 345–350.
20. Wangensteen S. Newly graduated nurses’ perception of com-
petence, critical thinking and research utilization. Dissertation,
Karlstad University, Sweden, 2010.
21. Pitt V, Powis D, Levett-Jones T, et al. The influence of crit-
ical thinking skills on performance and progression in a pre-
registration nursing program. Nurs Educ Today 2015;
35: 125–131.
22. European Communities. Guidelines for the development and
criteria for the adoption of health survey instruments.
Luxembourg: European Commission, Eurostat, 2005.
23. Watson G and Glaser EM. Watson Glaser II. Critical
Thinking Appraisal: technical manual and user’s guide. San
Antonio, TX: NCS Pearson, 2009.
24. Champion VL and Leach A. Variables related to research
utilization in nursing: an empirical investigation. J Adv
Nurs 1989; 14: 705–710.
25. Pettengrill MM, Gillies DA and Clark CC. Factors encoura-
ging and discouraging the use of nursing research findings.
IMAGE: J Nurs Schol 1994; 26: 143–147.
26. Humphris D, Hamilton S, O’Halloran P, et al. Do diabetes
nurse specialists utilise research evidence? Pract Diab Int
1999; 16(2): 47–50.
27. Wallin LW, Boström AM, Wikblad K, et al. Sustainability in
changing clinical practice promotes evidence-based nursing
care. J Adv Nurs 2003; 41(5): 509–518.
28. Funk SG, Champagne MT, Wiese RA, et al. BARRIERS:
The barriers to research utilization scale. Appl Nurs Res 1991;
4(1): 39–45.
29. Hommelstad J and Ruland CM. Norwegian nurses’ per-
ceived barriers and facilitators to research. AORN 2004;
79(3): 621–634.
30. PricewatherhouseCoopers Health Research Institute. The
price of excess: identifying waste in healthcare spending,
http://www.hasnet.us/uploads/PWC_The_Price_of_Excess_
Exec_Sum_Exhibit3.pdf (2009, accessed 20 February 2017).
31. Saunders H and Vehviläinen-Julkunen K. The state of readi-
ness for evidence-based practice among nurses: an integrative
review. Int J Nurs Stud 2016; 56: 128–140.
37
32. Forsman H, Wallin L, Gustavsson P, et al. Nursing students’
intentions to use research as a predictor of use one year post
graduation: a prospective study. Int J Nurs Stud 2012;
49: 1155–1164.
33. Boström AM, Ehrenberg A, Gustavsson JP, et al. Registered
nurses’ application of evidenced based practice: a national
survey. J Eval Clin Pract 2009; 15: 1159–1163.
34. Shinnick MA and Woo MA. The effect of human patient
simulation on critical thinking and its predictors in prelicen-
sure nursing students. Nurse Educ Today 2012; 28(1): 41–47.
35. Chau JPC, Chang AM, Lee IFK, et al. Effects of using video-
taped vignettes of enhancing students’ critical thinking ability
in a baccalaureate nursing programme. J Adv Nurs 2001;
36(1): 112–119.
36. Hunter S, Pit V, Croce N, et al. Critical thinking skills of
undergraduate nursing students: description and demo-
graphic predictors. Nurs Educ Today 2014; 34: 809–814.
37. Tan M, Sahin ZA and Özdemir FK. Barriers of research
utilization from the perspective of nurses in eastern Turkey.
Nurs Outlook 2012; 60(1): 44–50.
38. Wang LP, Jiang XL, Wang L, et al. Barriers to and facilita-
tors of research utilization: a survey of registered nurses in
China. PLoS ONE 2013; 8(11): e81908. DOI: 10.1371/
journal.pone.0081908.
39. Hennessy D, Hicks C, Hilan A, et al. The training and devel-
opment needs of nurses in Indonesia: paper 3 of 3. Hum Res
Health 2006; 4(10): 1–14.
40. Valen Gillund M, Rystedt I, Wilde-Larsson B, et al. Building
competence through cross-cultural collaboration in the after-
math of a tsunami: experiences of Indonesian teachers. Nurs
Edu Today 2013; 33: 192–198.
41. Shields L and Hartati LE. Nursing and health care in
Indonesia. J Adv Nurs 2003; 44(2): 209–216.
42. Diekelmann N and Smythe E. Covering content and the
additive curriculum: how can I use my time with students
to best help them learn what they need to know? J Nurs
Edu 2004; 43(8): 341–344.
43. Berland A, Richards J and Lund KD. A Canada–Bangladesh
partnership for nurse education: case study. Int Nurs Rev
2010; 57(3): 352–358.
44. Profetto-McGrath J, Smith KB, Hugo K, et al. Nurse edu-
cators’ critical thinking dispositions and research utilization.
Nurse Edu Pract 2009; 9: 199–208.
45. Carter AG, Creedy DK and Sidebotham M. Efficacy of
teaching methods used to develop critical thinking in nursing
and midwifery undergraduate students: a systematic review
of the literature. Nurs Edu Today 2016; 40: 209–218.
46. Melender H-L, Mattila L-R and Häggman-Laitila A. A sys-
tematic review on educational interventions for learning and
implementing evidence-based practice in nursing education:
the state of evidence. Nord J Nurs Res 2016; 36(1): 3–12.
47. Polit DF and Beck CT. Nursing research: generating and
assessing evidence for nursing practice. Philadelphia, PA:
Wolters Kluwer Health/Lippincott Williams & Wilkins,
2012.