This action might not be possible to undo. Are you sure you want to continue?
journal homepage: www.intl.elsevierhealth.com/journals/ijmi
An evaluation framework for Health Information Systems: human, organization and technology-ﬁt factors (HOT-ﬁt)
Maryati Mohd. Yusof a,∗ , Jasna Kuljis b , Anastasia Papazafeiropoulou b , Lampros K. Stergioulas b
Faculty of Information Science and Technology, Universiti Kebangsaan Malaysia, 43600 Bangi, Selangor, Malaysia School of Information Systems, Computing and Mathematics, Brunel University, Kingston Lane, Uxbridge, Middlesex UB8 3PH, UK
a r t i c l e
i n f o
a b s t r a c t
Background and purpose: The realization of Health Information Systems (HIS) requires rigorous evaluation that addresses technology, human and organization issues. Our review indicates that current evaluation methods evaluate different aspects of HIS and they can be improved upon. A new evaluation framework, human, organization and technology-ﬁt (HOT-ﬁt) was developed after having conducted a critical appraisal of the ﬁndings of existing HIS evaluation studies. HOT-ﬁt builds on previous models of IS evaluation—in particular, the IS Success Model and the IT-Organization Fit Model. This paper introduces the new frame-
Received 15 December 2005 Received in revised form 12 August 2007 Accepted 12 August 2007
Keywords: Information Systems Health Information Systems Evaluation Framework Human factors Organizational factors
work for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological, human and organizational ﬁt. Methods: Literature review on HIS and IS evaluation studies and pilot testing of developed framework. The framework was used to evaluate a Fundus Imaging System (FIS) of a primary care organization in the UK. The case study was conducted through observation, interview and document analysis. Results: The main ﬁndings show that having the right user attitude and skills base together with good leadership, IT-friendly environment and good communication can have positive inﬂuence on the system adoption. Conclusions: Comprehensive, speciﬁc evaluation factors, dimensions and measures in the new framework (HOT-ﬁt) are applicable in HIS evaluation. The use of such a framework is argued to be useful not only for comprehensive evaluation of the particular FIS system under investigation, but potentially also for any Health Information System in general. © 2007 Elsevier Ireland Ltd. All rights reserved.
The beneﬁts derived from a Health Information Systems (HIS) require rigorous evaluation. It is claimed that organizational and social issues are the main components of such a sys-
tem . The more technology, human and organization ﬁt with each other, the greater the potential of HIS. Most existing evaluation studies of HIS focus on technical issues or clinical processes, which do not explain why HIS works well or poorly with a speciﬁc user in a speciﬁc setting [2–7].
DOI of original article:10.1016/j.ijmedinf.2007.08.004. Corresponding author. Tel.: +603 8921 6649. E-mail addresses: firstname.lastname@example.org, email@example.com (M.Mohd. Yusof). 1386-5056/$ – see front matter © 2007 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2007.08.011
A number of studies in Health Informatics have included the concept of ‘ﬁt’ in explaining the interdependent relationship between human. The aforementioned models are presented to explore their applicability in improving on those used in Health Informatics. Theoretical background The proposed human. while for educational system is user acceptance. Highest success factors for Clinical Information Systems (CIS) are: collaboration and cooperation. A case study demonstrates the application of the proposed framework for describing the critical adoption factors of a particular HIS with a particular user in a particular setting. They argued that changes in both technology and health practices affect each other to a similar extent. and documents. the alignment of organization. Section 5 presents the research methodology for applying our framework to this case study. Section 4 discusses an existing Fundus Imaging System in a primary care organization that is used as a research case study. HOT-ﬁt makes use of the IS Success Model in categorizing its evaluation factors. organizational. Human and organizational factors 2. This paper is organized as follows. the IT-Organization Fit Model  is used to complement the IS Success Model by integrating its featured organizational factors and the concept of ﬁt between the human. management assumptions. lack of medical staff sponsorship and alteration of traditional workﬂow patterns. Human.9]).  and from feedback gained from two conference presentations of earlier stages in the research [8.11].11] and the IT-Organization Fit Model . As a result.i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 387 The central purpose of this paper is to show how we derived a framework for HIS evaluation that incorporates comprehensive dimensions and measures of HIS and provides a technological. Barriers to using HIS are also important to consider in HIS evaluation as they explain the failure and success of these systems. human.1. organizational and technical elements should also have a mutual alignment or ‘ﬁt’ in order to ensure successful HIS implementation. namely the IS Success Model [10. A study on factors inﬂuencing success and failure of HIS by a group of medical informaticians identiﬁed 110 success factors and 27 failure criteria from a wide range of socio-technical issues . Culture and process changes are reported to be the barriers to the wider use of health care systems . such as being risk adverse. lack of acceptance. its structure and/or work procedures. Rigorous evaluation of HIS can be undertaken by addressing these factors as well as the ﬁt or alignment between them. In addition. human and organizational factors are as important as technical issues with regards to system effectiveness . while for administrative system is not understanding the organizational context and not foreseeing the extent to which new HIS affects the organization. both models are utilized in constructing HOT-ﬁt. The IS Success Model is adopted because of its comprehensive. The importance of human and organizational factors in the development and implementation of IS has been advocated in the IS literature. organization and technology-ﬁt (HOT-ﬁt) is constructed so that it is capable of being useful in conducting a thorough evaluation study. The IS Success Model has then been identiﬁed as being complementary to another model in fulﬁlling the limitations of existing HIS evaluation frameworks. organizational processes.9]. human and organizational ﬁt. According to Willcocks . namely the IT-Organization Fit Model. machines. The new framework builds on previous work on the review of HIS evaluation [8. Examples of organizational challenges include hospital culture. Kaplan  shows that poor ﬁt between system developers’ goals and clinicians’ cultural values contributes to user reluctance to use Clinical Decision Support . A review of success determinants of Inpatient Clinical IS indicates that the categories for success in the IS Success Model can be used to assess HIS . 2. organization and technology factors [21. patient care philosophies and users’ needs as the introduction of a system affects different dimensions of ﬁt in complex ways . organizational and technological factors. But ﬁrst some background to the human and organizational aspects is necessary for later discussions. It is crucial that HIS ﬁt organizational aspects as well as align with work routines. reluctance to invest much in IT and resistance to change [18. dimensions and measures. discussion and conclusions are given in the last section. This proposed framework. Studies cited in Anderson  identiﬁed a number of barriers to direct physician use of HIS including low level of expertise.  found that the lack of ﬁt among main organizational elements contributes to a large number of system failures in public health. This then forms the basis for our ﬁrst proposed evaluation framework for HIS presented in Section 3. It is also possible that it will assist researchers and practitioners to unfold and understand the perceived complexity of HIS evaluation. Southon et al. tools. The two models are discussed in the last two sub-sections of this section prior to HOT-ﬁt’s development being explained in Section 4. work practices are seen as integrated networks of various related elements such as people.22]. These factors are assessed for six types of HIS. Berg  makes use of the socio-technical approach in HIS evaluation.  propose a model which illustrates the stages of information and system changes and their relative speciﬁc “personal requirements”. Aarts et al. organization and technology-ﬁt evaluation framework was developed after a critical investigation of the existing ﬁndings of HIS and IS evaluation studies (discussed in the preceding paper in this issue of the journal. The ﬁt between technical.19]. Highest failure criteria for CIS is response rate. speciﬁc evaluation categories. It also makes use of two models of IS evaluation. The case study ﬁndings are presented in Section 6. extensive validation and its applicability to HIS evaluation [10. and social factors is analyzed to identify gaps between current health care systems and new system features . Section 2 discusses the theoretical background of the proposed framework. setting goals and courses. technology and human is an important starting point in IT implementation as it is one of the strategies that affect IT investment. In short. Finally.
Within this internal and external ﬁt as its enabler. However. they are linked causally and temporally as success is viewed as a dynamic process instead of a static state. User Satisfaction (recipient response to the use of the output of IS) and Net Beneﬁts (the overall IS impact). For instance. For example. Organizational factors. ). industry and technology. 1). speciﬁc dimensions of IS success or effectiveness and the relationships between them. Fig. 2. The multidimensional relationships among the measures of IS success have been tested extensively in a number of IS studies . the framework might lead to spiral behaviors in both positive and negative directions. IT can be further classiﬁed into system quality and information quality. Effective process management can lead to increased user acceptance and participation in system use. IS Success Model In order to structure numerous results from research on IS success categories. 2 illustrates the concept of ﬁt between the main organizational elements. structure and management process). it does not include organizational factors that are pertinent to IS evaluation. a robust IT infrastructure such as an electronic network and understood standards should be equipped within the organization. organizational corporate strategy (business and IT). The extension of this framework is recommended by adding the organizational factors.388 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 Systems (CDSS). In comparison with existing HIS frameworks. Second. These measures are included in the six system dimensions: System Quality (the measures of the information processing system itself). effective use of the system will result in higher net beneﬁts which lead to more intensive use of the system. and roles and skills. Van der Meijden et al. Information Use (recipient consumption of the output of IS). The model was also identiﬁed as being capable of identifying the main organizational elements which can affect IS as well as emphasizing the essential alignment or ﬁt between them. and leadership are identiﬁed as both enablers and inhibitors of the ﬁt of IT and business . These three prerequisites as well as the internal and external ﬁt may be used to identify the problems in IT implementation. insufﬁcient system use will yield to lower net beneﬁts. Based on the strengths and limitations pointed out in both models. In order to realize the beneﬁts of IT. However. an updated version of this model is presented  (see Fig. Management in the 1990s (MIT90s) is a well-known ITOrganization Fit Model. Based on these studies. 2. organizational structure. three prerequisites are required for successful IT transformation. human (roles and skills) and organization (strategy. IT-Organization Fit Fig.3. Third. Internal ﬁt is accomplished by a dynamic equilibrium of organizational components including business strategy. Moreover. Because of its circularity. the model is comprehensive as it includes the following factors: technology (IT). A model is constructed which consists of six success categories or dimensions. which will in turn help achieve high net beneﬁts and subsequently increase system use. Service Quality (the measures of technical support or service). Similarly. IT-business relations. their dimensions and clinical measures related to the healthcare domain. a comprehensive taxonomy is introduced . This model was relatively new and has not been extensively utilized in healthcare . roles and skills can be associated with use and user satisfaction. which includes both internal and external elements of ﬁt . understanding the business. thereby acting as a disincentive to system use. Executive support. Fig. these factors can be categorized into more detailed dimensions to provide more speciﬁc evaluation dimensions. External ﬁt is achieved by formulating organizational strategy based on environmental trends and changes such as market. 2 – The MIT90s (IT-Organization Fit Model) (adapted from Scott Morton. IT is expected to affect the management process. which are lacking in the IS . Information Quality (the measures of IS output). its strategy.2. the IT-Organization Fit Model highlights the important role of the organization in managing the process of changes that take place during system introduction . DeLone and McLean’s IS model illustrates clear. In contrast. organizational vision and the reasons behind it have to be clear to organizational members to get them prepared for organizational changes and hence reduce the challenges in managing transformation. In addition. In both cases. as proposed by DeLone and McLean . thus impacting on organizational performance and to some degree. management processes.  discovered that a number of measures such as user involvement during system development and organizational culture do not match any of the dimensions of the framework. First. 1 – Information System Success Model (Source: DeLone and McLean ). IT-Organization Fit and the IS Success Model complement each other in presenting a comprehensive evaluation framework. information technology and organizational dimensions have to be aligned with each other.
Information Quality and User Satisfaction. System Use. 3 – Proposed human-organization-technology ﬁt (HOT-ﬁt) framework. Similarly. In the HOT-ﬁt context. These three factors and the impacts of HIS correspond to eight interrelated dimensions of HIS success: System Quality. abstract and subjective. Organizational Structure and Environment are direct antecedents of Net Beneﬁts. • Organizational Structure and Organizational Environment affects System Use. are featured in the IT-Organization ﬁt. 3. speciﬁc evaluation categories. Information Quality. images and prescriptions produced by the system. • Fit between technology. which relies on user knowledge and training. • The level of System Use can affect the degree of User Satisfaction and vice versa. organization and technology. Similarly. Success Model. It makes use of the IS Success Model in categorizing its evaluation factors. • Two-way relationships between these dimensions: Information Quality and System Use. a new evaluation framework is presented in the next section. 3): • Organization factors. Each of these dimensions is associated with a number of evaluation measures. Proposed evaluation framework Satisfaction. Effective System Use yield to higher User Satisfaction as user is able to explore and make full use of system features and functions. • System Use and User Satisfaction are direct antecedents of Net Beneﬁts. since the user’s knowledge in using the system can affect reports. The IS Success Model was extended by the addition of the following features which are explained in the following part of this section (see Fig. dimensions and measures. ﬁt is concerned with the ability of HIS. organization and technology are the essential components of IS. These evaluation dimensions inﬂuenced each other in a temporal and causal way: • System Quality. • One-way relationship between these dimensions: Structure and System Use. Based on the two models explained above. Examples of evaluation measures according to their corresponding dimension and factor are listed in Table 1. In addition. are featured in the IS Success Model. Net Beneﬁts subsequently have impact on organizational Structure and Environment. higher User Satisfaction subsequently motivate/lead user to increase System Use. Information Quality and Service Quality singularly and jointly affect both System Use and User Satisfaction.i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 389 Fig. extensive validation and its applicability to HIS evaluation. for both positive and negative cases. Some of these relationships are two ways: • System Use. • Similarly. human and organization factors. the IT-Organization Fit Model is also used to incorporate the concept of ﬁt between the evaluation factors: human. Organizational Environment and Net Beneﬁts. speciﬁc evaluation dimensions and measures which are lacking in the IT-Organization ﬁt.9]. • New evaluation measures pertinent to HIS and IS in general. the Organizational Environment factors such as government policy and politics can affect Organizational Structure while factors in Organizational Structure will affect the population served in the Organizational Environment. Human. The concept of ﬁt is perceived as complex. the proposed evaluation framework was developed after a critical appraisal of the existing ﬁndings of HIS and IS evaluation studies. The IS Success Model is adopted based on its comprehensive. human (HIS stakeholders and clinical Building on previous studies on the evaluation approach [8. User . Organizational Structure and Net Beneﬁts. Organizational Environment and Net Beneﬁts. Organizational Structure and Environment. Net Beneﬁts subsequently affect System Use and User Satisfaction. their dimensions (Structure and Environment) and evaluation measures (listed in Table 1). It can be viewed in terms of strategic planning (formulating IS plan according to organizational plan) and strategic alignment (managing IT closely with organizational needs) perspectives . Service Quality. Organizational Structure. can inﬂuence the Information Quality. the impacts of HIS are assessed in the net beneﬁts.
expectations/belief. percentage used. inter-organizational relationship.) actual vs. usefulness of system features and functions. effectiveness is deﬁned as the ability of a healthcare organization to continuously accomplish goals using optimum resources within a speciﬁed time. Thus. As mentioned above. it is important to determine whether the system (1) meets the need of the projected users. reliability. consistency and reliability [23. effectiveness and impact of HIS or IT in healthcare settings. Subsequently. decision making quality (analysis. type of information used. security. acceptance. technical support .1. accuracy. Organization User satisfaction Human Satisfaction with speciﬁc functions. reliability. number of records accessed. (2) is convenient and easy to use. Examples of system quality measures are ease of use. quantitative or a combination of both approaches. overall satisfaction.) purpose of use. 3. accuracy. Criteria that can be used for HIS quality are information completeness. the framework could be used to evaluate the performance. motivation to use. management. Service quality Information quality System quality Technology Data accuracy. The following subsections explain the evaluation factors.33. and security [29–33]. top management support. participation). decision making satisfaction Nature. (3) ﬁts the work patterns of the professionals for whom it is intended and the overall health system . attitude. appropriate use. conﬁdence. timeliness. organizational culture. morale. strategy. morbidity. an effective use of a HIS can be attributed to the ﬁt between system ﬂexibility and clinical process (system-human). number of functions used. mediator. knowledge/expertise. population served. use by whom? (direct vs. Availability refers to the up time of HIS while ﬂexibility is concerned with the ability of HIS to adapt to a healthcare setting and integrate with other systems.) recurring use. report acceptance. availability. strategy. mortality. management and communication.34]. information relevancy. frequency of access. leadership. accuracy. Based on its comprehensive dimensions and outcome measures. usefulness. size) culture. error reduction. assurance. Thus. technical support. This framework can be applied using qualitative. ﬂexibility. effectiveness (goal achievement. level of use (general vs. organization-technology) using a number of measures deﬁned in the three factors including system ﬂexibility. as well as the correct utilization of appropriate resources . any data collection methods from both approaches can be employed while conducting the evaluation. champion. Information quality measures can be subjective. convenient and pleasant to use.30. external communication Environment Clinical practice (Job effects. Ease of use assesses whether healthcare professionals regard HIS as satisfactory. organization and technology is illustrated by the bold arrows in Fig. user satisfaction. Measures of Information Quality are concerned with information produced by HIS including patient records. The three evaluation factors can be evaluated through out the whole system development life cycle namely planning. efﬁciency. user attitude. resistance/reluctance. speciﬁc. dimensions and measures in detail. service). politics. legibility. data entry methods Quick responsiveness. ease of learning. reports. design. availability. enjoyment. productivity. conciseness. format. The ﬁt between human. autonomy. localization. operation and maintenance. planning. Net beneﬁts can be anticipated before implementation and apparently evaluated after implementation. system ﬂexibility. Even systems that often work are often not used as anticipated.) efﬁciency. availability. Meanwhile. competition. reported use. System Quality in a healthcare setting measures the inherent features of HIS including system performance and user interface. In this research. implementation. planning. ease of learning. number of reports generated). communication. data currency. government. For example. completeness. legibility. medical sponsorship. empathy. images and prescriptions.) cost Net beneﬁts 3. reliability. system’s ease of use. clinical process. perceived usefulness. task performance. work volume. turnaround time Importance. analysis. nature of use (use for intended purpose. chauffeured use. Database contents. response time. completeness. Effectiveness refers to the accomplishment of speciﬁc goals with accuracy and completeness. human-technology. resource utilization. relevancy. teamwork Structure Financing source. voluntaries of use. amount of connect time. relevance. ease of use.390 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 Table 1 – Examples of the evaluation measures of the proposed framework Amount/duration:(number of inquiries. each phase focuses on different issues. frequency of report requests. Technology The studies on System Quality are often associated with system performance. system usefulness. timeliness.29. software satisfaction. ﬁt can be measured and analyzed from multiple compatibility between human. organization and technology (human-organization. usefulness. clinical outcomes (patient care. as they are derived from the user perspective. user training. time. (type. follow up service. communication. response time. training System use practices) and setting to align with each other.
Apart from these quantitative measures. staff. hierarchy.37]. population served. value and belief have been suggested as inﬂuencing individual’s attitude towards technology. Clinical outcomes are also measured through two criteria: morbidity (the rate of incidence of a disease) and mortality (death rate). User Satisfaction is often used to measure system success. competition. Net beneﬁts 3.33]. HOT-ﬁt combines those human factors in both frameworks in addition to other factors of human factors as mentioned earlier. as well as to test the applicability of the proposed framework in being a useful evaluation tool. User Satisfaction can be related to user’s perceived usefulness and attitudes towards HIS which are inﬂuenced by his/her personal characteristics. managers and IT. Personal characteristics such as age. Organizational impact is the effect of information on organizational performance. The practice is actively engaged in medical and nursing research. Leadership. Organization structure consists of nature including type and size (number of beds). People-oriented theory describes resistance to system results from user’s (groups or individuals) internal factors. Thus. Jiang et al. Examples of these measures include costs reduction. an organization or an entire industry. gender. in association to the factors involved and their relationships. and communication. We make use of the case study to answer our research question. assessment of a system varies across settings and users. Human The IS Success Model includes the use of system and user satisfaction while MIT90s incorporates roles and skills as part of human factors elements. System Use also relates to the person who uses it. training. Net Beneﬁts capture the balance of positive and negative impacts on user. It is associated with performance as well as changes in user task (clinical practice) such as job performance. clinical impacts can also be assessed qualitatively using these measures: quality of care. management and communication. Expectation refers to the anticipation of improved patient care delivery from the use of HIS . politic. The GP practice was established in the 1920s and now serves around 6000 patients who are mainly the elderly. Service quality can be measured through technical support. A case study is conducted in the clinical settings of a primary care organization (PCO) and two of its collaborating specialist hospitals. A number of HIS were already in place. individual Net Beneﬁts can be assessed using job effects. efﬁciency. Interaction theory explains resistance from the interaction between people and system factors. Job insecurity and fear are some examples of interaction resistance. clinical outcomes can be used as a means of measurement. top management support and medical staff sponsorship can also be measured from the organization factors Our research design consists of a case study strategy. and error reduction.3. hospitals or the entire healthcare sector. which is due to fewer medication errors and adverse drug effect (ADE). Resistance can be viewed from one of the following theories: (1) people-oriented. The actual use of the system as a measure of IS success refers to voluntary instead of mandatory use. including telemedicine and General Practice Information Systems (GPIS). The environment of a healthcare organization can be analyzed through its ﬁnancing source. The case study serves dual purposes: (1) to evaluate the adoption factors of HIS in the context of the phenomena under study. assurance. impact on patient care and communication. change in work activity and improved productivity [7. System Use is concerned with the frequency and breadth of HIS inquiries and functions. culture. Knowledge is concerned with computer literacy and skills [7.i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 391 Service Quality is concerned with the overall support delivered by the service provider of HIS or technology.2. The case study is also undertaken to obtain a comprehensive view and understanding of the development process of a HIS described in this section. localization. government. all members of the UK National Health Service (NHS). the reasons for resistance might differ among system types. which includes clinicians. effectiveness. (2) system-oriented and (3) interaction-oriented. In the healthcare context. User satisfaction is deﬁned as the overall evaluation of a user’s experience in using the system and the potential impact of the system. expectation and acceptance or resistance [30. [17. regardless of whether the service is delivered by the internal department of healthcare organization or outsourced to external providers. It is subjective in nature as it depends on whose satisfaction is measured. belief. strategy. speciﬁcally pertaining to tests and drug orders and increased use of generic drug brands and number of consultations and length of waiting lists [7. (2) to validate the proposed HOT-ﬁt evaluation framework. 3. The case of Fundus Imaging System (FIS) 3. 4. system developers. the type of populations being served. politics.23].  regard resistance as an important factor of system success. The use of information output such as reports appears to be one of the most frequent measures to assess the success of IS. autonomy. thus. empathy and follow up service. such as change in communication style and facilitation of information access . their levels of use.4. Organization The nature of a healthcare institution can be examined from its structure and environment . Individual impact is the effect of information on the behavior of the recipient. training and education. knowledge. A system can beneﬁt a single user.30]. quick responsiveness. The case study facilitated the conceptualisation of HIS adoption tending towards success and failure. inter-organizational relationship. improved efﬁciency in patient care delivery.17]. . decision quality. background. As different types of systems are usually related with a particular type of function and user. System oriented theory suggests that resistance results from system design factors or relevant technology including user interface and system characteristics. a group of users. planning and control systems.
5. stafﬁng. In the end 15 participants were found among clinicians. ABC Dr. major system development and improvement decisions with regards to the technology procurement. the FIS is part of this initiative.392 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 This study evaluates the development of a digital Fundus Imaging System (FIS) for diabetic retinopathy. While evaluating FIS. and PCs to transfer patient eye images into patient records which are featured in the GPIS. Participants were identiﬁed from the researcher’s initial contact with individuals known to staff members. who is a GP himself. patient data management. including interviews. The primary care screening aims to improve the patient pathway by providing better patient education—GPs could educate patients during the screening by showing them their eye images and immediately discuss the clinical implications and subsequently improve their diabetic control. and training for using the system and grading the eye images. which included problem identiﬁcation. the system assembles the image. particularly to the elderly patient who represents a signiﬁcant population of the PCO. which requires more advance equipment. a GPIS was also studied in order to investigate the overall adoption factors of HIS of the speciﬁc users in the particular setting. image manipulation. the list of participants is as follows (Table 2). after a Fundus camera was purchased. both from the primary care organization and from the two specialist hospitals involved in the study. as the existing camera does not comply with the guideline of the National Screening Committee. the patient pathway could be more cost effective through reduced time. MNO Dr. In addition. insightful explanation of the study . who is also the sole user of the system. JKL . staff and patients who were associated with FIS. retinal fundoscopy for such patients has been carried out in a hospital or by an optometrist. A subjectivist approach was employed in order to gain an extensive understanding of the healthcare context surrounding the FIS through detailed. questions and concerns) were identiﬁed through a literature review as well as observations made during an immersion. and reﬁnement of the evaluation framework (Fig. For each captured eye image. in addition to the FIS. DEF STU GHI PQR Mr. A group of eye specialists who include Consultant Ophthalmologists and a Reading Centre Advisor was liaised with to obtain expert advice in terms of selecting alternative patient pathways. During this study period. the selection of a research strategy and methods. The screening took place in a separate examination room equipped with a digital Fundus camera with a touch screen display. However. the development of an initial evaluation framework. ABC. Further. framework validation. During this period. Methodology The approach used in this study was that of a subjectivist. purchasing digital camera and relevant software. system evaluation. After identifying an initial group of participants. The recall system in hospital is seen as inefﬁcient in terms of traveling cost and accessibility. participant observation and document/artifact analysis. and timer data into a data ﬁle. A number of data collection methods were employed. Traditionally. A GP had to refer a patient to a hospital to be invited for annual screening. FIS consists of a Fundus camera and digital imaging software that is used to capture the eyes images of patients with diabetes. image transfer. A purposeful. The attempt to implement the FIS was continued by collaborating with a leading specialist hospital. In particular. the retinal screening is temporarily discontinued. the use of a General Practitioner Information Systems was also observed in order to obtain an overall view of the general application and the attitudes of the PCO staff towards Health Informa- Table 2 – List of participants for the case of FIS Participant User (a GP) Senior partner GP Nurse Physiotherapist IT staff Ophthalmologist Consultant Reading Centre Advisor Patients Total Total (N) 1 1 2 5 1 1 1 1 2 15 Initial used in data analysis Dr. The retinal screening of the FIS is done by a GP. The inefﬁciency of traditional screening motivated the partners at the PCO to implement a retinal fundoscopy screening within their practice. effort and expense of travel and double appointments. as well as to establish rapport with relevant stakeholders. VWX Dr. quality assurance and possible number of patient pathways are still under negotiation. All phases were completed. 4) [34. FIS is recognized for its importance in preventing blindness among a large population of diabetic patients and in improving the control of diabetic care. qualitative methods were employed to generate a fuller description of the healthcare setting and its cultural issues and to understand why the system functioned well or poorly in a particular setting. viewing and temporary storage. These data ﬁles are transferred to the GPIS by attaching each image to associated patient. patient data. snowball sampling method  was used in order to gain in-depth information from key informants about the development of the FIS.36]. FIS has functions for image capture. Evaluation problems (issues. a network was built by asking these ﬁrst participants to suggest additional participants for interview. A formative evaluation was undertaken of the adoption of FIS to identify system problems as they emerged and to improve the system as it was developed . Dr. The ﬁeld study was conducted between April and September 2005 by one of the authors. case study strategy employing qualitative methods. The idea to develop the system was initiated by the senior partner. The immersion was carried out to set the general context of the research. Our research approach to the evaluation of the FIS consisted of six iterative phases. which allow patients to have both retinal screening and diabetic health check on the same day. The UK government has recommended that all patients with diabetes have their annual retinal screening using digital retinal fundoscopy .
Participants were aware of the role of the researcher in their clinical settings. General background information on the primary care organization and its clinical and administrative staff were gathered to obtain a holistic and in-depth view of the clinical setting. and analyzed. These codes corresponded to each factor. The original framework in Part I of this paper has been employed initially for the FIS case study. Activities during immersion were recorded in a detailed ﬁeldwork log. the evaluation measures in the interview questions were identiﬁed based on the proposed framework and then linked with the research objectives. Further. During system evaluation. Two tests were used to establish the quality of this empirical research : 1. communication. texts under the same category were compared to identify variations and nuances in meanings. In this study. the framework was eventually validated as well as reﬁned in terms of its evaluation measures based on an appropriate mix of current literature and case study results. Categories were compared to discover connections between themes. The research strategy and methods were selected based on the research problem. facilities. Kaplan ). The data were audio. This is concerned with exposing and reducing subjectivity. by linking data collection questions and measures to research questions . as well as attitudes towards computer and IT literacy. displays. Concept maps were drawn to understand the relationship between the number of concepts involved in the evaluation of FIS. The HOT-ﬁt framework was used as a guideline in the evaluation of the FIS. The data were coded and categorized under similar themes or concepts of the HOT-ﬁt framework and reﬁned through out a series of analyses. meetings.i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 393 Fig. Initial data collection was gathered during the immersion. clinics and nursing homes of the primary care organization and specialist department of the collaborating specialist hospitals. leadership and of the staff in terms of their way of working. The ﬁeld notes were fully transcribed on margin-marked paper. and concept maps). After the conclusion of the study. The margins were used to note any reﬂections. relationships and rapport. clinicians and IT staff that were involved with the system were queried about their system use and patient pathways. Potential bias generally acknowledged in qualitative research approach was overcome by conducting a reliability . conﬂict. vision. A good rapport between the researcher and participants was established at both formal and informal occasions. transcribed into ﬁeld notes. four techniques were used to analyze the results: coding. dimensions and measures described in the HOT-ﬁt framework (see Fig. tion Systems. Data were collected on planned occasions as well as spontaneously in a number of iterative cycles. discussions and social events took place in different departments.and hand-recorded. 4 – Research design (adapted from Friedman and Wyatt . politics. The results showed that the interview questions addressed each of the research questions to a reasonable degree. Based on the HOT-ﬁt framework. patients were queried about their perception about the system. and contextual and narrative analysis . participant observations of daily clinical routines. Reliability: demonstrating that the operation of the study – such as the data collection produced – can be repeated with the same results. 2. Contextual and narrative analyses were done based on the themes assigned to the codes and these are further covered in the next section of this paper. 3). This was achieved through detailed documentation of procedures and appropriate record keeping . culture. An initial evaluation framework was constructed based on the ﬁndings from this ﬁrst phase. themes relevant to HOT-ﬁt and statements which were unclear or needed to be conﬁrmed with the participants. Meanwhile. During observations and face-to-face interviews. This view included aspects of management. Construct validity: establishing correct operational measures for the concepts being studied. analytic memos (such as reﬂection notes. individuals including users.
a number of emerging themes from the framework.2. The service providers of the primary care organization comprise of external vendors and a member of internal IT staff. She had to take the card out and upload the patient’s images to the PC and linked them with the patient records and saved them in a speciﬁc folder. the user was motivated to use the system with the assistance provided by a staff member at the outset of FIS implementation. According to the senior partner. For example. Although she could access their details on the system. and in performing time consuming and tedious tasks such as deleting and transferring ﬁles. DEF. According to Dr.1. When she requested technical support for a particular camera feature. Discussion of the ﬁndings from the Fundus Imaging System case Based on observations that took place during the immersion and pilot study. in practice and for the intended purpose. the external service providers “do not care about doctors’ needs at all. it was difﬁcult to use. the images are good enough. Dr. This purpose was viewed as impractical by the collaborating specialists in terms of cost since it is much cheaper to employ technicians to screen the patient via the FIS. tasks were completed faster and communications between staff and doctors were improved. The relevancy of the eye images lies in their usefulness in educating patients to take control of their diabetes. Service Quality can be observed and determined from the usage of GPIS. simple to use interface and straightforward data entry mode. However. it takes too long for her to print the list of exercises for the patient (2 min). The contents of the memory card will be deleted for future use. Dr. Mr. The remaining staff believe that the use of GPIS and other HIS had assisted them in performing their jobs better. she does not use the system because from her point of view. some of the GPs were not supportive of the system because. 6. a certain fact obtained from a GP was conﬁrmed with a different GP as well as the organization documentation and observation. 6. two aspects were analysed: relevancy and completeness. According to nurse STU. These issues are presented in the next sections. a member of the IT staff. the inefﬁciency of FIS is viewed from its inconvenient storage capacity. the existing camera had to be replaced. Technology The general practice where the study took place is somewhat advanced with its use of modern technology and is well equipped with computing and telecommunications devices. there are some problems related to the less mature technologies involved in its development. A number of the associate GPs dislike the GPIS. which was published after the camera was purchased. Human The Fundus Imaging System (for diabetic retinopathy) was developed for a very speciﬁc clinical purpose—to have a GP user educate patients with diabetes in improving their control of diabetic care. however. So she requires patients to remember their own lengthy list of exercises. although initially most of the users of the primary care organization have minimal IT skills. the ease of use of the Fundus Imaging System (FIS) can be seen immediately as a user friendly. adding “He started giving me all the jargon”. in terms of screening standards. it is very important to produce high-quality. The frequent need to delete and transfer ﬁles from a small size memory card (attached to the camera) to a hard drive when it gets full was seen as tedious and time consuming. In contrast to the GPIS. accurate images to avoid erroneous diagnosis. it is complicated and hard to learn . stored and attached to the patient record. PQR responded by saying “There’s nothing that can be done about it”. It seems that the service providers give little empathy to the clinicians that they are serving.“I hated it!” was her literal response. in their view. Captured images can be manipulated. developing. The user of FIS on the other hand has acknowledged her limited IT skills. their willingness to use the system was a key contributing factor that put the system in place. The internal IT specialist is trusted by the senior partner of the practice.394 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 test and data triangulation. This process resulted in user . MNO who argued. the images are not good enough to detect eye abnormalities. This view was not taken into serious consideration until months after FIS was in place. “He is difﬁcult to deal with but he is always right about something”. According to the Reading Centre Advisor JKL. where a large amount of time and effort were already allocated by the GP in planning. images taken with the existing camera are not as detailed and as accurate compared to those provided by hospitals. organizational and technological factors and net beneﬁts have been identiﬁed. namely human. It is also perceived that the user also receives too little support from the IT staff when technical assistance is needed. the old system that she used in her previous workplace “was template based and made typing and data entering much easier”. Feedback from participants regarding a particular conclusion in the setting studied is also gathered to reduce bias. In addition to system inefﬁciency. all staff members are using a General Practice Information System which features electronic patient records as well as service and management for GPs. physiotherapist GHI has to recall a number of exercises appropriate to a patient’s condition in order to suggest which of them should now be followed by the patient. The user. With regards to the system under investigation (FIS). training and using the system. For example. However. slow response time is also seen as a disincentive for using the GPIS. For example. However. In terms of Information Quality. and the use of the current FIS was temporarily halted because the camera did not comply with the National Screening Committee guideline. 6. As mentioned earlier. ABC said “it is a pain to keep on deleting the [images on the] memory card when it is full and it gets full quickly because it has small storage capacity”. which are organized in terms of the three themes. Triangulation was done through the use of multiple evidences from different sources to conﬁrm the same fact or ﬁnding . In comparison to the GPIS. they just care about their businesses”. Currently. such as copying ﬁles. The barrier to using the system can be seen from the lack of familiarity of the user in using basic ﬁle organization functions. In addition to educating patients effectively.
manual and electronic records are kept in an orderly manner. Another example of unfamiliarity was seen when a pop up window was displayed and the user said. On several occasions. It’s like the parenting role”. sees 30 patients. however. Despite her busy schedule. He does not tell us exactly what is going on and when the service provider meets the staff. In another instance. the organization has an environment that is conducive to work in. I have to guess to identify the causes of the patient’s pain”. the GP. The hospital did not give any referral letter or information that explains the previous medical diagnoses and it is difﬁcult and time consuming to get hold of these documents. we look like idiots!” He also said that. After months of meeting and training. screening and grading would be more expensive than hiring the screener. a GP seek advice from the nurses regarding the critical case of a home care resident and a physiotherapist consulted a GP to check on the effect of a drug–drug interaction. I don’t understand what they are talking about”. physiotherapist GHI stated. although the communication between staff is good. there are some problems with the communication between this primary care and secondary care organization. associate GPs and nurses. the conﬂict of interest between the project leader. Everyone is respected and treated equally regardless of their position. he continued to encourage the GP to change her perception about her role as a sole service provider and convincing her that she is capable of using the system. This situation. She used this example to refer to the initiation of the FIS in this general practice. it is taking a lot of her limited time. a joint meeting was postponed by a few months and this had affected the speed of FIS implementation. “All service providers are rubbish!” Focusing on the external environment of this organization. As nurse STU put it. personality. Three types of communication were observed in this clinical setting: patient–physician. the user ﬁnally took pictures of a nurse’s eyes. you should do more. The communication between the user and eye specialist is limited by their busy schedule. “I hate these messages. “The communication between hospital and the primary care organization is dreadful in terms of patient medical history. has been envisaged by one of the Consultant Ophthalmologists earlier and he had informed the user but it was not taken seriously. the user attended a series of training organized outside the practice. . In her view. She needs to be trained as an accredited screener to be able and eligible to examine the images and inform patients of the initial screening results. using his ‘gentle persuasive’ skill. “you start to criticise your colleague by saying you are not good enough. I would not take the role. He was the leader with a long-term vision and strategy to keep abreast with technology advancement. including the FIS.3. In addition to the rather advanced computing facilities. The teamwork spirit among the staff members can be seen in the decision making process.i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 395 dissatisfaction because in her view. checks the results. a patient received the wrong medical report—the pain was reported to be in the left shoulder instead of the right one. For instance. She believed that the senior partner’s leadership has changed her perception. “This place is different from other places. This is an indication of a typical problem of communication gap between technical staff and users. which includes the senior partner. writes the letters.. there is a common lack of communication between IT staff and clinicians. I was very reluctant because I have a different picture of a GP’s role”. a GP only comes in. The senior partner commented that one of the IT staff “always leaves us in a mess. the user and the eye specialist team in terms of technology procurement. It was not easy for me to accept that role initially. For instance. and universities to exploit new technologies in medicine. research meeting is held on a regular basis to discuss existing and new technology adoption and clinical concern pertinent to technology. Dr. In addition. The IT staff also uses jargon that makes the communication more difﬁcult and confusing to the doctors. it was a big change. The user recalled the time when the camera was not being used after it was purchased. ABC stated that change can be done in two ways: nurtures a person to make him feel that he can do it or it puts him down. have build a strong rapport and good communication between them. ABC pointed out that “. Informal discussions during lunchtime between various staff members. a commonality in their communication is made obvious: patient–physician relationships in terms of interpersonal communication and eye contact are essential during consultations. For example. Usually encouragement will work. Without these factors. According to the user. which include miscommunication and error in reporting. the PCO has considered assigning the role of current user. although each GP and nurse consults their patients in different ways. Dr. Everyone is also encouraged to work as a team in performing their daily tasks. 6. In terms of working style. prompting her to use the system by saying something like “Why don’t you have a play with it?” After a few reluctant initial attempts. This lack of communication has resulted in a difference of understanding about project implementation. The uptake of the FIS is also attributable to medical sponsorship in the practice. and encouragement are the catalytic factors that made possible the adoption and use of the system. . After the senior partner assigned the role of FIS user to her. Organizational readiness has been established in several occasions. training cost and the role of user has also delayed the uptake of FIS. . physician–physician and staff–physician. which results in conﬂicts. motivation. which made her realized that she can develop professionally and change. for example. The senior partner kept on teasing the user. for example. and sends them to hospital if necessary. He also formed a number of collaborations and partnerships with other primary and secondary care organizations. The senior partner really made a difference by creating a teamwork spirit”. to an external screener since the time spent by the GP for training. which was the starting point of diabetic retinopathy screening in this practice. Meanwhile. Organization The senior partner played a major role in shaping the organizational culture and establishing the use of technology in the organization. The initiative to use the system was also encouraged by other staff members. They tend to seek advice from their colleagues who have expertise in an area they themselves do not feel very conﬁdent with.
She has to spend additional hours in the retinal clinic capturing the eye images. Knowledge can be acquired through appropriate training. However. the beneﬁts of primary care screening can be assessed through: (1) patients’ control in their diabetic care. organization and technology Based on these three factors. GPs have such a heavy daily workload. choosing the right people to ﬁll the user role is more important than possessing the required skills. This explains why response time is crucial in the adoption decision. Fit between human. alternatives such as having replacement physicians can be taken to enable the user to attend this training. the communication gap between clinicians and the IT staff is obviously caused by the knowledge gap as well as individual characteristics towards being more sensitive to different stakeholders’ needs. the patient came back. as shown by the uptake of telemedicine. physicians have a very busy schedule. however. On the other hand. some negative effects have also been recognized. which was primarily driven by the willingness of the GP to learn and use it after being persuaded by the senior partner and despite her limited computing literacy. According to Dr. she was late for her surgeries. That is a fantastic way of inﬂuencing patients’ behaviour about managing their own health”. the senior partner. The use of FIS has increased the GP’s work volume. Such organizational culture and leadership has created awareness of technical advancements among the staff members as well as expedited the adoption of HIS. One way to achieve effective communication is through leadership. there have been certain occasions when. as skills can be acquired later. which caused the system to be discontinued temporarily. According to Dr. In this case study. she took up a marvellous opportunity to change her patients’ attitude disorder.396 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 6. ABC. The uptake of FIS was contributed by the user’s strong acceptance and personality to learn using the system (ﬁt between human and technology) and the technical support provided by a staff member that acted as a system champion. (2) equivalency or better treatment in primary care than the traditional screening in terms of speedy treatment and (3) low error rate and patient’s satisfaction. ease of use. it will cause major problems with your vision. As a result. one way to ﬁnd out whether the diabetic control is improved by this sort of patient education is by a blood test in the diabetic clinic. The alignment of the organizational strategy with IT and the existence of up-to-date computing infrastructure have also facilitated the implementation of HIS in the practice. external factors such as government policy can largely affect the viability of HIS. ABC. Net beneﬁts In terms of the net beneﬁts of the new system. In addition to leadership and top management support. In addition. The cause of the problem is poor diabetic control”. The partners of this GP practice envisaged that the implementation of FIS will beneﬁt patient by saving their time and cost of travelling. The same goes with the training issue. Moreover. response time and clinical process. ABC said to one of the patients that “these are your blood vessels. the lack of external ﬁt in FIS was shown in the non-compliance of current FIS equipment with the National guideline. their ﬁt with each other has been recognized.5. it seems that most of the practitioners have acknowledged certain effects on patient care and clinical practice. Communications between all staff at all levels are crucial to ensure that the purposes and beneﬁts of an HIS are understood since communication problem can be costly. “By her (the GP’s) involvement in taking pictures of the patients’ eyes. GPIS and FIS. MNO said.4. Dr. For example. The user was initially reluctant to use the system due to her perception of her role solely as a service provider. The GP changed the patient’s attitude by showing to the patient pictures of his eyes and discussing possible problems that may occur in relation to his diet. Meanwhile. Thus. Arguably. The practice is distinguished by being a research practice and having a good leadership. However. She believes that getting patients see their eye pictures may enable them realize that their diabetic control is not good and they are at a risk of losing their vision. As a result. This shorter patient pathway will also reduce possible error rates as data are stored and access directly within the practice. a lack of internal ﬁt can also be seen between technology and human: storage inconvenience and The case of FIS has demonstrated the importance of having the right user attitude and knowledge in order to be able to use the system effectively and efﬁciently. leadership as well as top management support has proven to be an important starting point and has major inﬂuence in the realization of HIS. It is quite impossible for a GP in the PCO to attend a series of time consuming training sessions while still having to attend her regular surgery sessions. if there’s any problem with them. the mismatch between clinical processes with technology was illustrated by the GP’s busy schedule with time required for training. as well as poor technical support and user need for assistance. It has been recognized that the FIS has the potential to improve the delivery of patient care by providing better patient education. although . medical sponsorship also played a signiﬁcant role in changing user perception and encouraging system use. The barriers to system use are contributed by the user perception. time and effort required from the busy GP user. having lost a couple of kilos and told the GP that she effectively made him change the whole way he thought about his diabetes condition. due to accessibility problems in the FIS. as well as storing and transferring them to the patient notes. In addition. they are so tiny. she has to spend a few hours per month for regular training sessions. Dr. 7. This can be seen from the use of FIS. Furthermore. External ﬁt of the practice is achieved by formulating its strategy according to current IT trends and advances. thus enable them to be screened earlier. For example. Further discussion and conclusions 6. The GP quoted an example of an obese patient who took a lot of liberty with his diet and had very bad diabetic control.
the proposed framework was tested in a real clinical setting. Other potential beneﬁts of FIS can be identiﬁed in terms of patient control of their diabetic care and the provision of equivalent or better treatment in primary care than secondary care organizations in terms of speedy treatment and patient satisfaction.  J. the framework can and should be applied in a ﬂexible way. is proposed using a multidisciplinary approach. J. We thus conclude that human. Inf. The strong willingness of the user to change her perception and clinical practices resulted in the uptake of FIS. Second. it is very important to plan and develop a system and keep it up-to-date in accordance with the latest policy. Int. Hsu. Although our case study focused on a speciﬁc setting. The framework is not the solution to any problem. the proposed evaluation framework is potentially useful to researchers and practitioners for conducting thorough evaluation studies of other HIS or IT applications in healthcare settings. medical sponsorship. As proposed here. user perception and user skills. • The existing evaluation methods do not provide explicit evaluation categories. H. Shaw.  B. user attitude. Int. a careful consideration has to be taken in selecting the most appropriate measures for each data. • The application of the proposed evaluation framework is demonstrated in a real-life. factors contributing to the positive adoption of FIS include: information relevancy. social interactionism. Coiera. Med. J. A clinical information system research agenda for Kaiser Permanente. and social issues. communication can be classiﬁed under organizational structure and environment. problems in using the system can be categorized under of technology factors (ease of use) or human factors (system use).F. practical context where formal evaluation methods have not been or could not have been used. 2003. C. • More work on human and organizational issues is called for as most existing evaluation studies of HIS focus on technical issues or clinical processes which do not explain why HIS work well or poorly with a speciﬁc user in a speciﬁc setting. Shaw. leadership. People. Med. • An evaluation framework for HIS. including FIS. as well as a lack of technical support and limited communication between the technical staff and the collaborating partners of the specialist’s hospital. Kaplan.  E. Kaplan. factors that had inﬂuenced the adoption negatively include: system usefulness.i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398 397 FIS had already been in place and operational. B. Med. references  D. Research needs and priorities in health informatics.  B. 58–59 (2000) 257–289. J. • Insights shed from the ﬁndings of the case study that can be used to inform decision making. N. N. McNair. Inf. Summary points What was known before the study: • There is a large number HIS evaluation frameworks looking at different aspects of these systems. taking into account different contexts and purposes. organization and technology (HOT-ﬁt). Kaplan. Jimson. the classiﬁcation of data analysis according to the three evaluation factors has affected the ﬂow of the narrative approach of presenting the data and can result in confusion to the reader. phases in system development life cycle. Sittig. Future directions in evaluation research: people. organizational and social issues: evaluation as an exemplar. clinical process and external communication with the inter-organizational system (the eye specialist). technical support. Kaplan. Hazlehurst. empathy of service quality. stakeholders’ point of views. Palen. M. The description of the case study and its ﬁndings offer a strong indication of the applicability of the framework to HIS evaluation.  B. For example. . IMIA 2002 Yearbook (2002) 91–102. Hodder Arnold. organizational and technological factors and the ﬁt between them are essential in the realization of FIS. Inf. T. First. The alignment of IT and organizational strategy has led to the initiation of a number of systems. Int. the adoption of FIS was disrupted with the incompatibility of the system with the National Guidelines. Thus. In order to validate its usefulness. Nohr. response time. there are a number of data that share the same evaluation measures. Guide to Health Informatics. which incorporates the concept of ﬁt between human. Evaluating informatics applications-some alternative approaches: theory. a repetition of the same data in different category can be used where necessary. 64 (1) (2001) 39–56. A number of challenges have been encountered while applying the HOT-ﬁt framework in data analysis. P. What the study has added to the knowledge: • Previous work on the evaluation of Information Systems is reviewed. organizational readiness. Meanwhile. FIS can potentially improve healthcare delivery by providing better patient education. its use had to be halted due to the fact that the camera did not comply with the latest National Guidelines. However. J. A number of critical factors to the adoption of FIS have been identiﬁed. Med.  B. organizational. The system put on the test was a Fundus Imaging System in a NHS primary care organization. 43 (3) (2004) 215–231.T. Permanente J.C. Brender. and call for methodological pluralism.L. 64 (1) (2001) 15–37. Acknowledgements We gratefully acknowledge the funding received from both the Public Service Department of Malaysia and from the Universiti Kebangsaan Malaysia (National University of Malaysia) that helped sponsor this study. Methods Inf. and evaluation methods. Otherwise. Evaluating informatics applications—clinical decision support systems literature review. 6 (3) (2002). Hornbrook. it is a structured debating tool that stakeholders can access in order to know their own system health better.
New York.H. Inf. 55 (2) (1999) 87–101. 2005. Jay (Eds. Big Island. Kuhn. Med.G. Springer-Verlag. Med. Qualitative research methods for evaluating computer information systems. 2002. N. B. (Ed.  W.H. Health Management Information Systems: A Handbook for Decision Makers. 2002. Med.K. Maxwell. 1999. H. Managing technology evaluation—techniques and processes. CA.0. Measuring e-commerce success: applying the DeLone & McLean Information Systems Success Model.K. 855–860. Outcome measures in clinical information systems evaluation.G. Anderson. C.J. Thousand Oaks. July 2005. Electron. in: R.J. pp. K. D. News 25 (1) (2002) 16–27. van’t Riet. Bolger Centre for Leadership Development Potomac. Inf. User resistance and strategies for promoting acceptance across system types. Aydin. in: C. Institute for Development Policy and Management. Brender. Proceedings of the 39th Hawaii International Conferences on System Sciences. J. L. Mundy. pp. F. S. Biol. Lippeveld. B.G. L. 45–67.E.  K.G. (2007). Int. AIS 4 (14) (2000) 1–50. Riley.G. Workshop on Issues and Innovation in Routine Health Information in Developing Countries. J. Sauer. Medscape Gen. USA. R. Int. 1 (1) (1999). C.M. E. Evaluating Health Care Information Systems. in: C.  M. Comput. R. S.G. Anderson. Int. Med. L. Aydin. Kaplan.  F. Overview: theoretical perspectives and methodologies for the evaluation of health care information systems. 10 (3) (2003) 235–243. Peel. Int.M. Paul.M. CA. 64 (1) (2001) 1–14. J. in: Proceedings of the 35th Hawaii International                  Conference on System Sciences (HICCS). J. 32 (3) (2002) 209–220. Luftman. A. HI. Inf.  J.).  W. Paul. 255–280. Paul. 69 (2–3) (2003) 197–203. 2005 from: http://www. F. E. Smith. B. Grant.  W.A. 37 (1) (2000) 25–36. Towards a Framework for Health Information Systems Evaluation. Commun. E.E. 2003.A. 1994. Retrieved 2 August.P. 2001. E. Switzerland. D. Stergioulas. Med. J. Patton. Galliers.K. B. Using case studies in research. Case Study Research: Design and Methods. Methods and Applications. 69 (2–3) (2003) 77–82.A. Determinants of success of inpatient clinical information systems: a literature review. pp. Salazar.G. Sage. Kauai. Sauer. J.C. M.C. University of Manchester. Oxford University Press. A. 1096–1100. Anderson.H. Lorenzi. J. Talmon. Scott Morton. 1997. C. Wyatt. Yusof. Inf. J. Thousand Oaks. Stergioulas. 2004.T. McLean. DeLone. M. Southon. A. Troost.). J. Springer-Verlag. . Clearing the way for physicians’ use of clinical information systems. C. Transfer and Management of Patient Information in Diabetic Retinopathy Screening Programmes Version 2. Health Information Systems Evaluation: A focus on Clinical Decision Supports System. pp. Qualitative Research & Evaluation Methods. A.P. Assoc. C. Ammenwerth. Doebbeling. The Corporation of the 1990s. Strategic Information Management: Challenges and Strategies in Managing Information Systems. R. Int. Geneva. J.R. Int. Aydin. Muhanna. Berg. Manage. 1997. 2006. J. Dampney. Sage Publications. Inf. Patient care information systems and health care work: a sociotechnical approach. Organizational ISSUES = change. Bottles. Heeks.J. Health Service Research & Development Series. Inf. Inform.  J. Giuse. Wyatt (Eds. V. 1991. Friedman. Organizational issues in health informatics: a model approach. Maryland. in: Medinfo. Critical choices face healthcare in how to use information technology. 1994. Commerce 9 (1) (2004) 31–47. Open University Press. 45 (1) (2006) 125–136. Am. Organizational evaluation of medical information resources. Methods and Applications.J.T. Integrating quantitative and qualitative methods in patient care information systems evaluation: guidance for the organizational decision maker. Commun. J. Why Health Care Information Systems Succeed or Fail. Hasman. The National Screening Committee (NSC). USA.  M. Buckingham. Lessons from a failed information systems initiative: issues for complex organizations.C. Med. 2004. Med.  J. Med. Factors inﬂuencing success and failure of Health Information System – a pilot delphi study. Med. Med. Int. J. Med.P.J.N. 2003. Routine Health Information Systems: The Glue of a Uniﬁed Health System. Yin. S.K.  M.C. Assessing business-IT alignment maturity. J.H. Proceedings of the XIX International Congress of the European Federation for Medical Informatics (MIE2005). C. J. 1994. Anderson. Thousand Oaks. Berg. Res.uk (2004). Nyakanen. Evaluation Methods in Medical Informatics. Information systems success revisited. Information Technology and Primary Care at VA: Interdisciplinary Partnership Opportunities for Providers. 3 (1) (1992) 60–95. ACM 40 (8) (1997) 83–90. DeLone. Baker (Eds. Shaw.398 i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 7 ( 2 0 0 8 ) 386–398  M.  K. Willcocks.  L. DeLone.R. Inf. ‘CHEATS’: a generic information communication technology (ICT) evaluation framework. Evaluating Health Care Information Systems. Assoc. The Heidelberg Conference: setting an agenda for the IMIA working Group on Health Information Systems. Stergioulas.J. Kaplan. Methods Inf. Rowley. and Researchers. J. Roderer. 42 (4) (2004) 458–462.D. pp.S.  R. Managers. R. 52 (1–3) (1998) 235–242. Med.J. Papazafeiropoulou.E. Hidemma. T. Sol. Southon.J. Van der Meijden. Jay. 4 (2) (1997) 112–124. 2000. G. A.  J. Information Technology in Complex Health Services: organizational impediments to successful technology transfer and diffusion. Aarts. J.Q. New York. Oxford. Jiang. Wright. Inform.nscretinopathy. J.J. Stoop. pp. San Francisco. Talmon. Evaluation Methods in Medical Informatics. Methods Inf.org. Friedman. Sage.S.R. Int. Meeting patients’ needs with patient information systems: potential beneﬁts of qualitative research methods. Berg. P. Med. Sage Publications. in: J. R. Am. M. Inf.L. J. 365–381. J. K. Res. N. Yusof. 5–29. W.). McLean. Thousand Oaks.M. USA.  M. Inf. Investigating evaluation frameworks for health information systems. Yusof.  N. Tange. 55 (1) (1999) 33–46. Manage. G. J. Hawaii. Syst.). Information systems success: the quest for the dependent variable. McLean. New York.  M. Klenin.
This action might not be possible to undo. Are you sure you want to continue?
We've moved you to where you read on your other device.
Get the full title to continue reading from where you left off, or restart the preview.