International Journal for Quality in Health Care, 2020, 32(6), 396–404

doi: 10.1093/intqhc/mzaa055
Advance Access Publication Date: 29 May 2020
Research Article

Research Article

Psychometric properties of the Bulgarian version

of Hospital Survey on Patient Safety Culture
DONKA KESKINOVA1 , ROSITSA DIMOVA2 , and
RUMYANA STOYANOVA 2

Downloaded from https://academic.oup.com/intqhc/article/32/6/396/5843378 by guest on 21 July 2023

1
Department of Applied and Institutional Sociology, Faculty of Philosophy and History, University of Plovdiv Paisii
Hilendarski, 24 Tzar Asen str., Plovdiv 4000, Bulgaria , and 2 Department of Health Management and Health Economics,
Faculty of Public Health, Medial University of Plovdiv, 15A Vassil Aprilov Blvd, Plovdiv 4002, Bulgaria

Address reprint requests to: Rumyana Stoyanova, PhD, Department of Health Management and Health Economics, Faculty
of Public Health, Medical University Plovdiv, 15A Vassil Aprilov Blvd, Plovdiv 4002, Bulgaria. Tel: +359-899936048;
E-mail: rumi_stoqnova@abv.bg
Received 4 February 2020; Revised 27 March 2020; Accepted 7 May 2020

Abstract
Objective: To explore the psychometrics of the Bulgarian version of the Hospital Survey on Patient
Safety Culture (B-HSOPSC) and its suitability for use in Bulgaria.
Design: A national web-based cross-sectional survey of the safety patient culture.
Setting: The hospitals’ staffs from 28 administrative areas in the country.
Interventions: Web-based self-administered questionnaire.
Participants: Physicians and non-physicians such as nurses, midwifes, etc., working at hospitals.
Main Outcome Measures: Confirmatory factor analysis (CFA) was performed to assess the psycho-
metric properties of the original US structure. Assessment of construct validity included convergent
validity, discriminant validity, and nomological validity of constructs.
Results: A total of 525 valid cases were included in the analysis. The results of CFA revealed
acceptable values for absolute indices and lower for the incremental index, comparative fit
index. Due to the very low convergence validity, the dimension ‘staffing’ was removed from the
model. Additionally, one item was removed from another dimension. The B-HSOPSC included 11
dimensions and 37 items.
Conclusions: The B-HSOPSC had acceptable levels of global and local fits. Its safety culture
dimensions were sufficiently distinguishable and correlated with outcome variables.

Key words: patient safety, quality measurement, quality management

incorporating: knowledge, values and symbols of patient safety [5].

Introduction
Based on the WHO concept, ‘patient safety’ is a chief component
of quality of medical care and presents a serious global healthcare
issue [1, 2]. The Institute of Medicine defined patient safety as ‘the
prevention of harm to patients’ [3]. According to the Agency for
Healthcare Research and Quality (AHRQ) patient safety is: ‘freedom
from accidental or preventable injuries produced by medical care’ [4].
Patient safety culture (PSC), as an essential part of organizational
culture of health organizations, is a multidimensional phenomenon
Generally, it outlines the efforts of the health organization to secure
safe delivery of healthcare and includes: effective leadership, a state
of co-operation and mutual respect between staff and management,
open communication, continuous education of the hospital staff,
introduction of standard procedures, promoting a sense of justice
(hospital culture, focused on identifying system rather than individual
errors or failures and assuming responsibility for them) and patient-
centered care [6]. The implementation of measures to improve PSC
in healthcare dates back to the end of the 1990s and still presents

© The Author(s) 2020. Published by Oxford University Press in association with the International Society for Quality in Health Care.
All rights reserved. For permissions, please e-mail: journals.permissions@oup.com 396
Psychometrics of the B-HSOPSC • Research Article
a major issue in healthcare research [3]. It has been established that
better measurements scores of PSC were associated with better out-
comes [5]. Currently, the Hospital Survey on Patient Safety Culture
(HSOPSC), developed by the US AHRQ is one of the three instru-
ments, recommended for internal use in the European Union [4]. Until
now, the HSOPSC has been used worldwide in >40 countries [7–10].
In our country, systematic measurement and regulations about
implementation of reporting systems of patient safety is still lacking.
In order to validate the instrument for evaluation of PSC, a study
of its conceptual equivalence and cultural relevance of the Bulgarian
version of the HSOPSC (B-HSOPSC) was performed [11].
The aim of this study is to explore the psychometrics of the
B-HSOPSC and its suitability for use in Bulgaria.
Methods
Study design and population
A national cross-sectional survey was conducted using an internet-
based platform for registration and evaluation of hospital PSC
through the B-HSOPSC. The present study was part of the Medical
University Plovdiv Project No. 11–2017 and was approved by the
university’s research ethics committee (number 05/19.10.2017).
The survey was organized as a multistep process. At step one, 50
out of 545 multi-disciplinary private and public hospitals (represent-
ing 14.0% of all hospitals in the country) were randomly selected
from all 28 administrative areas in the country. During step two
hospital executives and managers were provided by post and e-
mail with information brochures including links to the web-based
platform, alongside with cover letters, introducing the purpose and
expected outcomes of the project. The information packages and
cover letters were intended to aid in achieving the co-operation of
the managers and to encourage the hospital staff to participate in the
survey. At a later stage, follow-up reminder phone calls were made to
the hospital managers.
After obtaining a written consent, the hospital staff had to com-
plete the online version of the B-HSOPSC.
The study contingent included healthcare professionals: physi-
cians and non-physicians such as nurses, midwifes, etc., working at
hospitals in all 28 administrative country’s areas.
The response rate of the questionnaire could not be determined
due to the web-based design of the study. Questions regarding the
hospital name or brand and other demographic details were not
included to avoid staff anxiety.
Data were collected in the period from July to October 2018.
Sample properties
The sample included 545 respondents, of which 61.3% non-
physician medical practitioners and 38.7% physicians. Regarding
work placements, most participants worked at internal medicine
wards (26.8%), followed by surgical words (15.1%). Half of the
participants (50.6%) stated that they work 41–60 h weekly; 7.5%
worked over 61 h. Most participants (38.8%) had worked under
5 years at the respective hospital, 24.1%—had worked 6–10 years
and 27.2%—over 11 years.
Measures
The process of translation, linguistic validation and cultural adap-
tation of the B-HSOPSC questionnaire was described in detail in a
previous article [11].
397
The B-HSOPSC includes 42 questions, grouped in 12 different
dimensions, aimed at measuring PSC. Additionally, the questionnaire
provides two outcome variables—patient safety grade and number
of adverse events reported. Three of the dimensions were measured
based on their frequency response rate (‘feedback and communi-
cation about error’, ‘communication openness’ and ‘frequency of
events reported’). The remaining nine dimensions were rated on the
five-point Likert-type scale with agreement response option from 1
(‘strongly disagree’ or ‘never’) to 5 (‘strongly agree’ or ‘always’). Two
new question items were added to the B-HSOPSC: one with open and
one with closed answer domain. The respondents were asked if they
would use a mandatory error and adverse events reporting system.
The open question referred as to whether the participants were
ready to report medical errors or adverse events from their real-life
medical practice. Subsequently, the most detailed reported cases were
Downloaded from https://academic.oup.com/intqhc/article/32/6/396/5843378 by guest on 21 July 2023
expertly analyzed and made accessible through the online platform
to encourage sharing of adverse event experiences, information and
knowledge as well as avoidance of repeated errors.
The questionnaire scale items are listed in Table 1.
Statistical analyses
The starting point in the analysis was to determine the percentage of
missing values both in terms of respondents and items. Respondents
without answers to >5% of the items were excluded. The remaining
missing values were imputed using expectation–maximization (EM)
algorithm [12]. The next step was to reverse the codes of the nega-
tively worded items.
In order to test whether our data are adequate for factor analysis,
we used the Kaiser–Meyer–Olkin (KMO) measures of sampling
adequacy (MSA) and Bartlett’s test. The KMO-MSA determines
the adequacy of the data for the whole model, the MSA for the
individual variables. For both coefficients, values >0.8 are considered
excellent [12]. The Bartlett’s test of sphericity tests the hypothesis that
correlation matrix is an identity matrix, which is an indicator of the
lack of correlation between the items. The hypothesis must be rejected
if P < 0.05.
A confirmatory factor analysis (CFA) with maximum likelihood
estimates (MLE) was used to investigate whether the dimensions from
the original US version of the HSOPSC were confirmed. MLE is more
efficient and unbiased when the assumption of multivariate normality
is met, but the ratio of respondents to variables >15/1 minimizes the
consequences of not fulfilling it [12].
Validation of the model required establishing an acceptable level
of goodness-of-fit (GoF; global fit) and finding evidence of construct
validity (local fit) [12].
There are many GoF indices, which divided by types into classes
[12–16]. Multiple indices of different types should be used. We
assessed the GoF with the normed Chi-square (Chi2 /df) and the root
mean square error of approximation (RMSEA) from the class of
absolute indices, and the comparative fit index (CFI) from the class of
the incremental indices. This set of indices usually provides sufficient
unique information to evaluate a model, as the standardized root
mean square residual (SRMR) can replace RMSEA [12, 16]. The
following cut-off values were determined to be acceptable for our
data: ratio 3:1 or less for Chi2 /df, 0.07 or less for RMSEA, 0.08 or
less for SRMR, and >0.90 for CFI [12].
The assessment of the construct validity included the following
components: examination of the items reliability (path estimation),
convergent validity, discriminant validity and nomological (predic-
tive) validity.
 398

Table 1 Questionnaire scale items, descriptive statistics of the items and its reliability

Dimension Itema Mean SD % positive answers Item reliability

Safety culture dimension

1. Supervisor/manager expectations and B1 My supervisor/manager says a good word when he/she sees a 3.78 1.03 69.5% 0.57
actions promoting safety job done according to established patient safety procedures
B2 My supervisor/manager seriously considers staff suggestions 3.68 1.04 66.7% 0.69
for improving patient safety
B3r Whenever pressure builds up, my supervisor/manager wants us 3.24 1.10 48.2% 0.06
to work faster, even if it means taking shortcuts
B4r My supervisor/manager overlooks patient safety problems that 3.94 1.05 77.3% 0.42
happen over and over
2. Organizational learning—continuous A6 We are actively doing things to improve patient safety 3.75 1.05 6.4% 0.49
improvement
A9 Mistakes have led to positive changes here 3.49 1.18 61.5% 0.50
A13 After we make changes to improve patient safety, we evaluate 3.60 0.93 66.5% 0.32
their effectivenessb
3. Teamwork within hospital units A1 People support one another in this unit 3.61 0.96 61.9% 0.46
A3 When a lot of work needs to be done quickly, we work 3.81 1.01 71.6% 0.50
together as a team to get the work done
A4 In this unit, people treat each other with respect 3.50 0.98 57.1% 0.58
A11 When one area in this unit gets busy, others help out 3.36 1.14 53.7% 0.38
4. Communication openness C2 Staff will freely speak up if they see something that may 3.88 1.06 68.8% 0.54
negatively affect patient care
C4 Staff feel free to question the decisions or actions of those with 3.24 1.19 43.0% 0.49
more authority
C6r Staff are afraid to ask questions when something does not seem 3.59 1.18 54.7% 0.45
right
5. Feedback and communication about error C1 We are given feedback about changes put into place based on 3.40 1.20 44.8% 0.47
event reports
C3 We are informed about errors that happen in this unit 3.73 1.13 60.2% 0.53
C5 In this unit, we discuss ways to prevent errors from happening 3.92 1.03 68.8% 0.63
again
6. Non-punitive response to error A8r Staff feel like their mistakes are held against them 2.89 1.11 34.3% 0.25
A12r When an event is reported, it feels like the person is being 2.93 1.16 37.0% 0.56
written up, not the problem
A16r Staff worry that mistakes they make are kept in their personnel 3.32 1.07 53.0% 0.30
file
7. Staffing A2 We have enough staff to handle the workload. 2.83 1.27 37.9% 0.41
A5r Staff in this unit work longer hours than is best for patient care 2.19 1.01 12.8% 0.27
A7r We use more agency/temporary staff than is best for patient 3.90 1.16 73.7% 0.06
care
A14r We work in ‘crisis mode’, trying to do too much, too quicklyb 2.35 1.16 20.8% 0.22

Continued
Keskinova et al.

Downloaded from https://academic.oup.com/intqhc/article/32/6/396/5843378 by guest on 21 July 2023

Table 1 Continue

Dimension Itema Mean SD % positive answers Item reliability

8. Hospital management support for patient F1 Hospital management provides a work climate that promotes 3.59 1.12 62.7% 0.52
safety patient safety
F8 The actions of hospital management show that patient safety is 3.61 1.07 60.8% 0.55
a top priority
F9r Hospital management seems interested in patient safety only 3.43 1.15 55.8% 0.44
after an adverse event happens
9. Teamwork across hospital units F2r Hospital units do not coordinate well with each other 3.35 1.16 50.3% 0.33
F4 There is good cooperation among hospital units that need to 3.53 1.10 60.2% 0.54
work together
F6r It is often unpleasant to work with staff from other hospital 3.48 1.05 53.5% 0.44
units
F10 Hospital units work well together to provide the best care for 3.61 1.02 61.5% 0.68
Psychometrics of the B-HSOPSC • Research Article

patients
10. Handoffs and transitions F3r Things ‘fall between the cracks’ when transferring patients 3.70 0.99 65.3% 0.55
from one unit to another
F5r Important patient care information is often lost during shift 3.92 0.94 74.9% 0.51
changes
F7r Problems often occur in the exchange of information across 3.46 1.02 53.5% 0.54
hospital units
F11r Shift changes are problematic for patients in this hospital 4.01 0.90 81.0% 0.49
Outcome dimension
11. The frequency of event reporting D1 When a mistake is made but is caught and corrected before 3.61 1.24 59.6% 0.67
affecting the patient, how often is this reported?
D2 When a mistake is made but has no potential to harm the 3.43 1.23 50.9% 0.88
patient, how often is this reported?
D3 When a mistake is made that could harm the patient, but does 3.75 1.32 62.1% 0.49
not, how often is this reported?
12. Overall perceptions of safety A10r It is just by chance that more serious mistakes do not happen 3,45 1.17 54.5% 0.17
around here
A15 Patient safety is never sacrificed to get more work done 3,75 1.12 72.0% 0.30
A17r We have patient safety problems in this unit 3,76 1.08 71.4% 0.28
A18 Our procedures and systems are good at preventing errors 3,62 0.96 65.7% 0.43
from happening

a
‘r’ after number of the item mean ‘negatively worded item’, the scale of which is reversed in the analysis.
b
These items were paraphrased in Bulgarian.
399

Downloaded from https://academic.oup.com/intqhc/article/32/6/396/5843378 by guest on 21 July 2023

400
Table 2 Model goodness-of-fit
Chi2 /df RMSEA SRMR CFI
Threshold ≤3 ≤0.07 ≤0.08 ≥0.90
Original US model 3.167 0.064 0.060 0.837
BG model 3.088 0.063 0.058 0.869
From items reliability measures, were examined the squared
multiple correlations (SMCs) coefficient with a cut-off value >0.3.
The convergence validity was estimated using the following criteria
and cut-off values: average variance extracted (AVE) with a value
>0.5; construct reliability (CR) and Cronbach’s alpha with a value
>0.6 for both, as measures of internal consistency [12, 17]. The
discriminant validity was evaluated by using heterotrait-monotrait
(HTMT) ratio of correlation. HTMT values close to 1 indicates a
lack of discriminant validity. We determined a cut-off value of 0.9,
associated with liberal discriminant reliability [17, 18].
Finally, to evaluate the nomological validity of the model, Pear-
son’s correlation coefficient between each patient safety dimensions
and the single-item outcome variable ‘patient safety grade’ was
calculated. This correlation is expected to be significant and positive.
All statistical analyses were performed using the software IBM
SPSS Statistics 25 and IBM SPSS AMOS 24 [19] with plugins ‘model
fit’ and ‘master validity’ [20].
Results
Sample characteristics
The missing values of the items do not exceed 3%. Of the 545
participants in the sample, 4% did not respond to three or more
items. After excluding the latter, 525 cases were included in the
analysis. To eliminate the rest missing values, we applied the EM
imputation. Table 1 contains the mean values, standard deviations
and the percentage of positive responses by items.
The value of the KMO-MSA was 0.93. The values of most of the
MSA coefficients were >0.9, but there were two MSA coefficients
<0.8 for items B3 (0.71) and A7 (0.78), values defined as ‘middling’
[12]. The P-value in Bartlett’s test was <0.001. These results indicated
the adequacy of the data for the application of factor analysis.
Original US model
The results of the CFA for the original US structure revealed accept-
able values for absolute indices: Chi2 /df = 3, RMSEA = 0.06,
SRMR = 0.06 and lower for the incremental index CFI = 0.84
(Table 2).
Values for SMC that indicate how well an item variability was
measured by a dimension, was below the fixed threshold of 0.3 for
B3 from dimension 1, A7 and A14 form dimension 7, and A10 from
dimension 12 (Table 1).
Two of the three indices for convergence validity—CR and Cron-
bach’s alpha (Table 3), indicated good fit, except for the dimension 7.
Regarding the third index—AVE, only half of the dimensions reached
a value of >0.5. The lowest was the values of AVE for ‘staffing’ (0.24)
and ‘overall perceptions of safety’ (0.30).
The HTMT ratios of correlation values were in the range 0.19–
0.91 (Table 4). The values could not be calculated for dimension 7
(the presence of negative correlations seems the likely explanation
for this). Figure 1 depicts the extent to which dimensions were
Keskinova et al.
distinguished. HTMT ratio was >0.9, but very close to it (0.91), only
between dimensions 8 and 9. This indicates that the safety culture
dimensions (except ‘staffing’) were sufficiently distinguishable.
With the exception of dimension 7, the Pearson’s correlation coef-
ficients between composite values for dimensions and outcome vari-
able ‘patient safety grade’ were statistically significant and positive
(Table 4). The lowest (0.29) was the correlation between dimension
6 and ‘patient safety grade’. The other coefficients were in the range
0.40–0.64, which revealed a moderate positive correlation.
Bulgarian model
The item-level results MSA and SMC revealed low adequacy and a
weak relationship with the dimensions of B3 and A10. This called
into question that their inclusion in the model. The effect of their
Downloaded from https://academic.oup.com/intqhc/article/32/6/396/5843378 by guest on 21 July 2023
exclusion from dimensions 1 to 12 was checked.
After removal of B3, the three indices for convergence validity
of dimension 1 increased (see Table 3). In most cases, HTMT ratios
of correlation also increased, but remained below the threshold of
0.9 (see Table 4). The situation was different with respect to the
A10. After removing this item, values for two of the three indices
for convergence validity of dimension 12 decreased—CR (0.615) and
Cronbach’s alpha (0.590), with the latter falling below the threshold
of 0.6. The HTMT between dimensions 2 and 12 had increased to
0.963—a value that indicated that the two dimensions were nearly
indistinguishable. Thus, the removal of B3 improved the convergent
validity of dimension 1, whereas the removal of A10 impaired the
convergent and discriminant validity of dimension 12.
At the construct level, dimension ‘staffing’ did not satisfy any of
the criteria for convergent validity. This dimension contains two of
the items: A7 and A14, which were with item reliability of 0.06 and
0.22. Removing the items individually did not change the indices for
convergence validity. Therefore, the removal of a single item from
dimension 7 was not a solution and necessitated the removal of
dimension ‘staffing’ from the model.
All this was the reason for making the following changes to
the original model—removal of item B3 from dimension 1 and
removal of dimension 7. All other dimensions and their contents were
preserved. Thus, the Bulgarian (BG) model of B-HSOPSC included 11
dimensions and 37 items.
The CFA score for the BG model showed better GoF indices
compared with the original US model (Table 2). This is especially
true for CFI, whose value rose from 0.84 to 0.87 and closer to the
acceptable range.
With respect to nomological validity of the BG model, compared
to the US model, there was only a change in the value of the
Pearson’s correlation coefficient between dimension 1 and ‘patient
safety grade’, which increased from 0.42 to 0.49.
Discussion
Using CFA, the original US structure was tested. The dimension
‘staffing’ showed problematic psychometrics. This was due to the
low [−0.15; 0.13] correlation between A7 and other items of the
dimension, which can be explained by the local specific cultural,
economic and health system characteristics. Regardless of presence
of adequate legislation (the introduction of the Employment Encour-
agement Act in 2011), the practice of locum work is still not popular
in Bulgaria. Removing these two items from the dimension would
violate the practice of a minimum of three items per dimension [12].
Therefore, one option in the search for an acceptable model was
Psychometrics of the B-HSOPSC • Research Article 401

Table 3 Convergence validity criteria—CR, AVE and Cronbach’s alpha, for original US model and BG model

Model
Index Original US model BG model

Dimension CR AVE Cronbach’s alpha CR AVE Cronbach’s alpha

Threshold ≥0.6 ≥0.5 ≥0.6 ≥0.6 ≥0.5 ≥0.6
1. Supervisor/manager expectations and 0.73 0.44 0.71 0.79 0.56 0.78
actions promoting safety (‘without B3 in
the BG model’)
2. Organizational learning—continuous 0.70 0.44 0.69 0.70 0.44 0.69
improvement
3. Teamwork within hospital units 0.79 0.48 0.78 0.79 0.48 0.78
4. Communication openness 0.74 0.49 0.74 0.74 0.49 0.74
5. Feedback and communication about error 0.78 0.55 0.78 0.78 0.55 0.78
6. Non-punitive response to error 0.63 0.37 0.61 0.63 0.37 0.61

Downloaded from https://academic.oup.com/intqhc/article/32/6/396/5843378 by guest on 21 July 2023

7. Staffing (‘removed from BG model’) 0.04 0.24 −0.01 − − −
8. Hospital management support for patient 0.75 0.50 0.75 0.75 0.50 0.75
safety
9. Teamwork across hospital units 0.79 0.50 0.78 0.79 0.50 0.78
10. Handoffs and transitions 0.81 0.52 0.81 0.81 0.52 0.81
11. The frequency of event reporting 0.86 0.68 0.84 0.86 0.68 0.84
12. Overall perceptions of safety 0.62 0.30 0.60 0.62 0.30 0.60

Figure 1 Multidimensional scaling (Proxscal) on HTMT profiles. Note: The farther apart are two dimensions in space, the more they distinct that they are from
each other and vice versa.

to remove the dimension as a whole, and the other to move the model had good convergence validity (AVE ≥ 0.5 and CR ≥ 0.7)
other two items to another dimension. We chose the first option and the rest—acceptable (AVE < 0.5 and CR ≥ 0.6). This would
for the following reasons. First, we felt that the content of the ensure comparability of results with other studies (countries) and
remaining dimensions should be retained, eight of which in the BG over time. The fact that a dimension is problematic at present does
 402

Table 4 HTMT ratios of correlation between dimensions and Pearson’s correlation coefficient between dimensions and the ‘patient safety grade’—below the diagonal for the original US
model, above the diagonal for the BG model

Dimension 1 2 3 4 5 6 7 8 9 10 11 12 Patient
safety grade

1. Supervisor/manager expectations 0.67 0.72 0.79 0.81 0.41 − 0.79 0.61 0.64 0.50 0.73 0.49a
and actions promoting safety
(‘without B3 in the BG model’)
2. Organizational 0.64 0.76 0.64 0.66 0.19 − 0.71 0.63 0.45 0.41 0.90 0.48a
learning—continuous
improvement
3. Teamwork within hospital units 0.66 0.76 0.54 0.65 0.42 − 0.69 0.73 0.61 0.39 0.69 0.50a
4. Communication openness 0.80 0.64 0.54 0.90 0.53 − 0.70 0.54 0.56 0.61 0.70 0.47a
5. Feedback and communication 0.77 0.66 0.65 0.90 0.44 − 0.82 0.65 0.59 0.75 0.69 0.52a
about error
6. Non-punitive response to error 0.44 0.19 0.42 0.53 0.44 − 0.49 0.49 0.59 0.38 0.58 0.29a
7. Staffing (‘removed from BG NaN NaN NaN NaN NaN NaN − − − − − −
model’)
8. Hospital management support for 0.78 0.71 0.69 0.70 0.82 0.49 NaN 0.91 0.81 0.59 0.87 0.64a
patient safety
9. Teamwork across hospital units 0.61 0.63 0.73 0.54 0.65 0.49 NaN 0.91 0.89 0.48 0.82 0.56a
10. Handoffs and transitions 0.65 0.45 0.61 0.56 0.59 0.59 NaN 0.81 0.89 0.42 0.71 0.50a
11. The frequency of event reporting 0.47 0.41 0.39 0.61 0.75 0.38 NaN 0.59 0.48 0.42 0.53 0.40a
12. Overall perceptions of safety 0.71 0.90 0.69 0.70 0.69 0.58 NaN 0.87 0.82 0.71 0.53 0.62a
Patient safety grade 0.42a 0.48a 0.50a 0.47a 0.52a 0.29a 0.08 0.64a 0.56a 0.50a 0.40a 0.62a

a
Correlation is significant at the 0.01 level (two-tailed).
NaN—value is not output by the software, ‘-’ not included in the HTMT analysis.
Keskinova et al.

Downloaded from https://academic.oup.com/intqhc/article/32/6/396/5843378 by guest on 21 July 2023

Psychometrics of the B-HSOPSC • Research Article
not mean that it will remain the same in the future. Second, we had
no guarantee that this was not an effect of the limitations of the
study described below. When deciding on the regular application of
B-HSOPSC in Bulgaria, for which we strongly appeal, the construct
validity (in all its aspects) of dimension ‘staffing’ must be verified
and a decision made on its form of presence or removal from the
instrument.
The problem of dimension ‘staffing’ in the Bulgarian survey is
not a precedent. For example, in France the A7 was removed [9]; in
Sweden A7 (and A15) had the least item reliability [21]; in Romania,
the dimension was removed as well due to two other items (A2 and
A5) [22]; in Greece, there were problems with both A7 and other
two (with the smallest items reliability) items in our study—B3 and
A10, which in the final Greek model was combined in one dimension
(together with B4) [23]; an optimal model derived from UK data
resulted in a dimension that linked ‘overall safety perceptions’ and
‘staffing’ [24]. This is the most problematic dimension in HSOPSC—
half of the 62 studies that have used the HSOPSC reported internal
reliability for ‘staffing’ <0.6 [25].
Our study registered lower, but acceptable, convergent validity
for dimensions 6 and 12. A large percentage of studies (21 and
39%, respectively) reported an unacceptable level (<0.6) of internal
reliabilities for these two dimensions [25].
The problem with ‘overall perceptions of safety’ in other studies
was caused by item A15. Dimension 12 was eliminated in Romania
due to A15 [22], in the Netherlands, item A15 was removed from
dimension 12 [26], in Sweden with the least item reliability was
item A15 (and A7) [21]. In our data, the problem came from
item A10. The smallest item reliability of this item was due to the
lack of correlation between A10 and A15 (RA10 and A15 = 0.08;
P = 0.08) and the slightly lower correlation of A10 with the other
two items in the dimension, compared with A15. Regardless of
this, the value of Cronbach’s alpha for dimension 12 is far more
satisfactory compared with that in the previous study [11]. It is
likely attributable to the improved semantics of items A15 and A10,
performed after the conceptual equivalence and cultural relevance
study.
In our study, the lowest (0.03) item reliability for B3 (and A7) was
recorded by ‘supervisor/manager expectations and actions promoting
safety’, dimension with good convergence validity. Excluding this
item from the BG model has increased both global GoF and local
fit.
The HTMT analyst, which was applied to evaluate the extent to
which a construct is distinct from other constructs, revealed strict
discriminant reliability of 4 from 11 dimensions (1,3,6,11), and
acceptable of the others.
As expected, all 11 dimensions correlate with the outcome vari-
able ‘patient safety grade’. The highest correlation of this outcome
variable was ‘hospital management support for patient safety’ (0.64)
and ‘overall perceptions of safety’ (0.62). Of the other dimensions,
only one correlation with the outcome variable was <0.4, which is a
good indicator of the validity of the dimensions.
Just over half (52%) of the 62 studies from 29 countries reported
Cronbach’s alpha, as a measure of internal reliabilities, <0.7 for
at least six HSOPSC dimensions [25]. In the Bulgarian study, these
dimensions were four, and for one of them Cronbach’s alpha was
0.69, which practically limits them to three. In only 18% of studies,
the original 12 HSOPSC dimensions were confirmed [25]. Therefore,
in the present study, the original US model displayed psychometric
characteristics above the average.
403
There are several limitations to our study. First, although the
hospitals were selected at random, the completion of the question-
naire was based on the principle of responders, i.e. we have no
guarantee of random selection among hospital staff. Second, the
small size of the sample did not allow it to be split in half and
cross-validated.
Despite these limitations, the current study further contributes to
the evaluation of the psychometric properties of B-HSOPSC. On one
hand, the study complements the knowledge about the validation
of HSOPSC in a new country, and on the other hand, it creates
conditions for its implementation in Bulgaria. Moreover, the online
survey version avoids the possibility of interviewer’s bias. Confi-
dentiality of respondents was guaranteed by exclusion of questions
regarding demographic characteristics as well as questions regarding
the hospital name, logo or location. Thus, they could be sure that
Downloaded from https://academic.oup.com/intqhc/article/32/6/396/5843378 by guest on 21 July 2023
no information could be used against them. Hence, the responses
received can be considered to have high reliability.
Conclusion
The B-HSOPSC questionnaire is applicable to the Bulgarian popula-
tion with minimal exceptions to its original form. Item B3 and dimen-
sion ‘staffing’ should be removed for the purpose of the research.
Thus, the BG model of B-HSOPSC includes 11 dimensions and 37
items.
In order to test reproducibility of the measured items of the B-
HSOPSC, additional research is needed in the near future.
Acknowledgements
The present study was made possible thanks to the University Project of the
Plovdiv Medical University No.11/2017 titled: ‘Development and implementa-
tion of a web-based platform for registration and evaluation of the level of PSC
in the healthcare system in Bulgaria and conduction of a national representative
study’.
References
1. World Health Organization. WHO Multi-professional Patient
Safety Curriculum Guide: https://apps.who.int/iris/bitstream/handle/
10665/44641/9789241501958_eng.pdf;jsessionid=D5CB73AC21339215
CF691588F6280C78?sequence=1 (14 January 2020, date last accessed).
2. World Health Organization. A Brief Synopsis on Patient Safety, 2010.
http://www.euro.who.int/__data/assets/pdf_file/0015/111507/E93833.
pdf. (14 January 2020, date last accessed).
3. Kohn LT, Corrigan JM, Donaldson MS (eds). To Err Is Human. Building
a Safer Health System. Washington, DC: National Academies Press, 2000.
4. Sorra J, Nieva VF. Hospital Survey on Patient Safety Culture. Agency
for Healthcare Research and Quality publication no. 04–0041. Rockville,
MD: AHRQ, 2004.
5. Nieva VF, Sorra J. Safety culture assessment: a tool for improving patient
safety in healthcare organizations. Qual Saf Health Care 2003;12:ii17–23.
6. Pronovost P, Sexton B. Assessing safety culture: guidelines and recommen-
dations. Qual Saf Health Care 2005;14:231–3.
7. Agency for Healthcare Research and Quality. Hospital Survey on Patient
Safety Culture: https://www.ahrq.gov/sites/default/files/wysiwyg/profe
ssionals/quality-patient-safety/patientsafetyculture/hospital/userguide/ho
spcult.pdf (14 January 2020, date last accessed).
8. Eiras M, Escoval A, Grillo IM et al. The hospital survey on patient safety
culture in Portuguese hospitals: instrument validity and reliability. Int J
Health Care Qual Assur 2014;27:111–22.
404
9. Occelli P, Quenon JL, Kret M et al. Validation of the French version of the
hospital survey on patient safety culture questionnaire. Int J Qual Health
Care 2013;25:459–68.
10. Bodur S, Filiz E. A survey on patient safety culture in primary healthcare
services in Turkey. Int J Qual Health Care 2009;21:348–55.
11. Stoyanova R, Dimova R, Tarnovska M et al. Linguistic validation and
cultural adaptation of Bulgarian version of hospital survey on patient
safety culture (HSOPSC). OAMJMS 2018;6:925–30.
12. Hair JF, Black WC, Babin BJ et al. Multivariate Data Analysis, 2nd edn.
New Jersey: Pearson Prentice Hall, 2009.
13. Hu L, Bentler P. Cutoff criteria for fit indexes in covariance structure anal-
ysis: conventional criteria versus new alternatives. Struct Equ Modeling
1999;6:1–55.
14. Kline RB. Principles and Practice of Structural Equation Modeling, 3rd
edn. New York: Guilford Press, 2005.
15. Brown TA. Confirmatory Factor Analysis for Applied Research. New
York: Guilford Press, 2006.
16. Hooper D, Coughlan J, Mullen M. Structural equation modelling: guide-
lines for determining model fit. EJBRM 2008;6:53–60.
17. Hamid MR, Sami W, Mohmad Sidek MH. Discriminant validity assess-
ment: use of Fornell & Larcker criterion versus HTMT criterion. J Phys
Conf Ser 2017;890:012163.
18. Henseler J, Ringle CM, Sarstedt M. A new criterion for assessing dis-
criminant validity in variance-based structural equation modeling. J Acad
Market Sci 2015;43:115–35.
Keskinova et al.
19. Byrne BM. Structural Equation Modeling with AMOS. Basic Concepts,
Applications, and Programming, 2nd edn. Routledge: Taylor & Francis
Group, 2010.
20. Gaskin J, James M, Lim J. Master Validity Tool. AMOS Plugin In:
Gaskination’s StatWiki, 2019. http://statwiki.kolobkreations.com/index.
php?title=Main_Page.
21. Hedsköld M, Pukk-Härenstam K, Berg E et al. Psychometric proper-
ties of the hospital survey on patient safety culture, HSOPSC, applied
on a large Swedish health care sample. BMC Health Serv Res 2013;
13:332.
22. Tereanu C, Smith SA, Ghelase MS et al. Psychometric properties of
the Romanian version of the hospital survey on patient safety culture
(HSOPS). Maedica 2018;13:34–43.
23. Kapaki V, Souliotis K. Psychometric properties of the hospital survey on
patient safety culture (HSOPSC): findings from Greece. Vignettes Patient
Safety 2017;2:171–89.
Downloaded from https://academic.oup.com/intqhc/article/32/6/396/5843378 by guest on 21 July 2023
24. Waterson P, Griffiths P, Stride C et al. Psychometric properties of the
hospital survey on patient safety culture: findings from the UK. Qual Saf
Health Care 2010;19:e2.
25. Waterson P, Carman EM, Manser T et al. Hospital survey on patient safety
culture (HSPSC): a systematic review of the psychometric properties of 62
international studies. BMJ Open 2019;9:e026896.
26. Smits M, Christiaans-Dingelhoff I, Wagner C et al. The psychometric
properties of the ‘Hospital Survey on Patient Safety Culture’ in Dutch
hospitals. BMC Health Serv Res 2008;8:230–9.