Professional Documents
Culture Documents
Antimicrobial stewardship
programs: interventions and
associated outcomes
Expert Rev. Anti Infect. Ther. 6(2), 209–222 (2008)
Dimple Patel, Guidelines regarding antimicrobial stewardship programs recommend an infectious diseases-
Wendy Lawson and trained physician and an infectious diseases-trained pharmacist as core members. Inclusion of
B Joseph Guglielmo† clinical microbiologists, infection-control practitioners, information systems experts and
† hospital epidemiologists is considered optimal. Recommended stewardship interventions
Author for correspondence
include prospective audit and intervention, formulary restriction, education, guideline
University of California,
San Francisco School of development, clinical pathway development, antimicrobial order forms and the de-escalation
Pharmacy, 521 Parnassus of therapy. The primary outcome associated with these interventions has been the associated
Avenue, C-152, cost savings; however, few published investigations have taken into account the overall cost of
San Francisco, CA 94143, the intervention. Over the past 5 years, there has been an increased focus upon interventions
USA intended to decrease bacterial resistance or reduce superinfection, including infections
Tel.: +1 415 476 2354 associated with Clostridium difficile colitis. Few programs have been associated with a
Fax: +1 415 476 6632 reduction in antimicrobial drug adverse events. Antimicrobial stewardship programs are
guglielmoj@ becoming increasingly associated with clear benefits and will be integral in the in-patient
pharmacy.ucsf.edu healthcare setting.
www.future-drugs.com 211
Review Patel, Lawson & Guglielmo
www.future-drugs.com 213
Review Patel, Lawson & Guglielmo
14
Antimicrobial guidelines for iv.-to-oral switch were developed and physicians from selected wards were educated. Guidelines were reinforced by clinical pharmacists during
rounds for patients meeting the switch criteria.
15
An antimicrobial classification was developed whereby restricted agents required approval before dispensing and controlled agents were reviewed for their appropriateness
by an infectious disease pharmacist 2 days after being ordered. Recommendations for therapy changes were placed in the patient’s chart. The ordering physician was given a
24 h period to reject or accept. If the 24 h time period was exceeded, acceptance was assumed.
16
Guidelines (pocket cards) for appropriate antimicrobial use, with emphasis on urinary and respiratory infections, were disseminated, lectures on geriatric infectious diseases
were presented, and team members made weekly ward rounds, targeting individual physicians to provide counseling on antimicrobial therapy.
17
Mandatory approval by a staff physician was required for restricted drugs (ceftriaxone, ceftazidime, piperacillin/tazobactam, imipenem/cilastatin and vancomycin). An
educational program was developed for the medical staff, focusing on antimicrobial cost data, appropriate use, indications for early iv.-to-oral switch, and diagnosis and
treatment of common infectious diseases, weekly rounds with medical staff to evaluate antimicrobial therapy, biannual feedback to medical staff regarding antimicrobial costs,
and yearly provision of antimicrobial susceptibilities. Antimicrobial guidelines were disseminated via handbooks and the hospital intranet.
18
An antibiotic cycling protocol was implemented, whereby fluoroquinolones and β-lactams were alternatively cycled every 3 months for empiric therapy of all infections,
except meningitis, endocarditis, sexually transmitted infections or infections with pathogens presumed resistant. Protocol was distributed four-times annually by email to all
medical and pharmacy staff, posted on the hospital intranet and nursing stations. Educational sessions were held for medical, surgical and pharmacy staff prior to
implementation.
19
Education was provided to internal medicine and emergency medicine physicians on the treatment of community-acquired pneumonia and on the potential benefits of
conversion to oral therapy. Patients receiving iv. antibiotics for community-acquired pneumonia were identified daily. Pharmacist recommended or automatically converted
patients meeting criteria for oral therapy to an oral fluoroquinolone.
20
Ceftazidime and cefotaxime were removed from the formulary. The use of ceftriaxone, carbapenems and lipid formulations of amphotericin B was restricted and required
approval by the infectious diseases physician. Cefepime was added to the formulary. An automatic 72-h stop order was placed on vancomycin orders. Ciprofloxacin was
replaced by levofloxacin on the formulary.
21
A hospital antibiogram was introduced. Antibiotic prescription forms were developed. All orders for Third-generation cephalosporins, β-lactam/β-lactamase inhibitors,
glycopeptides and carbapenems were reviewed by the infectious diseases physician and rated for their appropriateness. Feedback was provided to each specialty service at
educational sessions. Educational training sessions were performed monthly for all medical students and residents, and every 4 months for all physicians.
22
All antimicrobial orders by internal medicine residents and general surgery residents were evaluated to determine compliance with hospital guidelines and the AMT provided
recommendations for noncompliant orders. The AMT provided feedback to physicians, including weekly reports on compliance with institutional guidelines and prescriber
acceptance of AMT recommendations, quarterly department-specific reports on compliance and a monthly newsletter outlining the feedback reports.
23
An automatic stop prophylaxis form was implemented for surgery, with extensive education of each surgeon as well as operating room, nursing and pharmacy staff. The
physician could call the pharmacy to request the continuation of an antibiotic if he deemed it necessary.
24
Computerized clinical decision support was added, allowing for modification of existing antimicrobial management program where patients on restricted antimicrobials were
identified and recommendations for therapy changes made if necessary. The software additionally alerted the team of patients with potential for iv.-to-oral conversion, of
potentially unnecessary double coverage or to organisms potentially resistant to current therapy.
25
Education was provided to the medical staff regarding the purpose and objectives of the antimicrobial management program. Pharmacists automatically converted eligible
patients from iv.-to-oral routes for highly bioavailable antimicrobials. Perioperative antimicrobial prophylaxis at 24 h for clean and clean-contaminated surgical procedures was
automatically discontinued. Use of antimicrobials with high risk for adverse events, high cost, high potential for promoting resistance, or drugs of last resort was restricted. All
restricted antimicrobials continued for greater than 48 h required infectious diseases consultation.
26
Eligible patients were automatically converted from iv.-to-oral routes for fluoroquinolones, clindamycin and metronidazole after 24 h. Conversion of iv. antibiotics to oral
antibiotics with similar spectra was recommended by the clinical pharmacists via direct interaction with the prescriber if patients met criteria for switching. The restricted
antibiotic authorization system was reorganized. The pharmacy director reviewed restricted antibiotic orders twice daily with the infection controller.
27
Recommendations on aminoglycoside use were updated, including dosing regimens, indications, and timing of drug level monitoring. Updates were disseminated to all staff
and junior physicians via educational sessions at each ward. Each aminoglycoside order was reviewed, and interventions were provided via counseling to the prescriber when
appropriate.
28
New guidelines for antibiotic prophylaxis were developed. An antibiotic restriction requiring approval of hospital antibiotic center was implemented. All new restricted
antibiotic orders were reviewed by clinical pharmacists. Interventions for inappropriate use were performed via education of prescribers on antimicrobial use and cost, microbial
susceptibilities and conversion to oral agents.
29
Guidelines were revised for duration of treatment of nosocomial pneumonia from 7–14 days and for community-acquired pneumonia from 10–14 days to 5 days.
Carbapenems were removed from recommendations for nosocomial pneumonia to encourage use of piperacillin/tazobactam.
ICU: Intensive-care unit; iv.: Intravenous.
Adverse events Europe: ten out of 11; others: six out of eight). Clinical
TABLE 3 sumarizes those studies that showed the impact of microbiologists and infection-control practitioners were
stewardship intervention on antimicrobial-related adverse events included less frequently (in seven and eight out of 36 of
[22,31]. Changes in the rates of adverse events were expressed reported studies, respectively). A total of 29 studies evaluated
as a relative change between the pre- and postintervention the impact of the intervention on cost outcomes, and 22
time periods. assessed the impact upon bacterial resistance and/or super-
infection due to C. difficile. Of note, only two studies evalu-
ated the impact on the rate of adverse drug events. TABLE 1 dis-
Results plays these studies associated with evaluation of cost
Outcomes associated with antimicrobial stewardship inter- outcomes. Of the identified studies, the majority of studies
ventions are summarized in TABLES 1, 2 & 3. A total of 36 studies examining cost outcomes originated from the USA or
were identified from the time period of interest and, subse- Europe. Interestingly, only seven listed studies provided
quently, included in the analysis. Some studies evaluated information regarding the costs associated with the imple-
more than one of the outcomes of interest. In reviewing the mentation and development of the stewardship intervention
included studies, the composition of the stewardship team program. Of the 29 identified studies, 27 reported a reduc-
could be determined. In 26 out of the 36 included investiga- tion in costs specifically associated with the intervention. In
tions, a pharmacist was part of the stewardship team (USA: most instances, cost savings were generally reported using a
15 out of 17; Europe: six out of 11; others: five out of eight). decrease in total antimicrobial acquisition costs over time,
Physicians were key members of the stewardship interven- while some presented their findings per patient or patient day.
tions in 31 out of 36 of the studies (USA: 15 out of 17; One study evaluating the effectiveness of a parenteral-to-oral
www.future-drugs.com 215
Review Patel, Lawson & Guglielmo
www.future-drugs.com 217
Review Patel, Lawson & Guglielmo
22
A ‘bundle’ approach was implemented in response to a C. difficile outbreak. Education was provided by means of an electronic module. Primary-care nurses were given
authority to order C. difficile testing to expedite early case finding and therapy initiation. Infection-control measures were heightened. A C. difficile-management team was
established to evaluate all toxin-positive patients in near real time. In addition, a formal antimicrobial management program was developed, targeting restriction of clindamycin,
ceftriaxone and levofloxacin, requiring approval for use.
ESBL: Extended-spectrum β-lactamase; iv.: Intravenous; ICU: Intensive-care unit; NP: Not provided (regarding statistical signifcance); NS: Non-significant effect (statistically
significant difference not detected).
switch was unable to demonstrate a reduction in costs over a were prospective antimicrobial order reviews with concurrent
12-week time period [18]. Notably, one study reported an feedback, guideline development and regular physician edu-
increase in antimicrobial acquisition costs associated with the cation. Of note, antimicrobial cycling was associated with a
intervention [22]. In this latter investigation, antimicrobial substantial increase in antimicrobial costs [22].
cycling was performed over a 2-year time period. The investi-
gators observed a 31% increase per 1000 patient-days in two
community hospitals in a healthcare system. Discussion
TABLE 2 summarizes those studies intended to change the rate Numerous organizations, including the IDSA/SHEA, have
of bacterial resistance or superinfection. A total of 22 studies recommended the use of antimicrobial stewardship teams to
were identified that evaluated this study outcome. The ensure the safe, effective and appropriate use of antimicrobi-
majority of the included trials evaluated changes in bacterial als. These recommendations, together with the findings of
resistance, ranging from methicillin-resistant Staphylococcus our review document that infectious diseases-trained physi-
aureus to multidrug-resistant Gram-negative infections. In cians and infectious diseases-trained pharmacists are the most
most studies, changes in susceptibility were reported for mul- commonly used members of the stewardship team. However,
tiple organisms. In six investigations, the primary outcome of it is clear that the contributions of microbiologists, infection-
interest was the rate of CDAD. One study evaluated the control practitioners and computer-system support profes-
impact of the program for both CDAD and changes in the sionals are important as these members constitute the opti-
rate of bacterial resistance. Of the listed studies, most mal team. As others have suggested [2], publication bias prob-
reported positive findings (i.e., a decrease in the rate of bacte- ably exists with respect to the outcomes associated with
rial resistance or CDAD was observed). When reduction in antimicrobial stewardship team interventions. In other
the rate of CDAD was observed, the antimicrobial steward- words, studies with negative results are less likely to be pub-
ship intervention was also generally associated with addi- lished than those associated with positive outcomes. Thus, it
tional changes in infection control. As an example, one medi- is not surprising that the majority of published investigations
cal center noted an increase in the rate of CDAD from 2.7 to report positive results. While the possibility of publication
7.2 per 1000 patient-days [41]. In response, a ‘bundle’ inter- bias probably exists, consistent cost savings, improvements in
ventional approach took place, including education, early and bacterial resistance, and a decreased rate of CDAD have been
increased identification of patients, improved infection-con- associated with such programs. While the published results
trol procedures, development of a CDAD control team and regarding cost outcomes are encouraging, in most instances
antimicrobial control centered upon clindamycin, ceftriaxone the true cost of the intervention is unknown. In order to
and levofloxacin. The results of this bundle approach assess the true ‘net’ cost avoidance, associated with such stew-
included a decrease of 41% in targeted antimicrobial therapy, ardship program interventions, it is clear that these associated
a reduction in the rate of CDAD to 3.0 per 1000 patient program costs must be included in the analysis. Similarly,
days, and a decrease in the percentage of hypervirulent interventions intended to decrease bacterial resistance or hos-
C. difficile from 51% of all CDAD cases to 13%. pital-associated infections should take into account costs
Interestingly, only two trials were identified that assessed the associated with the intervention. The costs and resources
impact of an intervention on antimicrobial adverse events; one required to implement and maintain an intervention are
demonstrated no increase in the observation of adverse events essential components of cost–benefit analysis; such analyses
in a study of antimicrobial cycling, where the primary intent are necessary in establishing sufficient hospital administra-
was a reduction in bacterial resistance [22]. Only one trial used tion support to institute a successful stewardship program.
adverse events as the primary outcome [31]. In this investiga- Resources that should be incorporated in these analyses
tion, a team of infectious diseases physicians, using guideline include stewardship team-member salaries, costs of required
development, education, review and feedback, reduced the rate equipment and computer programs and expenses associated
of nephrotoxicity over 1 year. with the development and distribution of educational materi-
In most studies, pharmacists and physicians were the key als. Changes in bacterial resistance or CDAD are difficult to
members of the antimicrobial stewardship team. The major- unequivocally link solely with antimicrobial stewardship
ity of trials utilized multiple interventions to achieve the pri- interventions. In most instances, changes in infection control
mary outcome. In general, those techniques demonstrating procedures were implemented at the same time as the antimi-
the most benefit in terms of reduction in antimicrobial-related crobial interventions. As an example, the most recent investi-
costs and antimicrobial resistance, and the incidence of CDAD gation regarding the successful control of the hypervirulent
www.future-drugs.com 219
Review Patel, Lawson & Guglielmo
strain of C. difficile used a combination of antimicrobial outcomes associated with such programs has been cost savings;
interventions and enhanced infection control [41]. With however, inclusion of the true costs of the implementation of
respect to interventions intended to reduce antimicrobial- such interventions must also be part of any such analysis. Simi-
associated adverse events, only one investigation was found larly, intervention programs improve bacterial resistance and
and it centered upon aminoglycoside nephrotoxicity. The reduce CDAD, but this is generally only in concert with infec-
paucity of investigations centered upon adverse events is tion-control procedures. Nonetheless, antimicrobial steward-
probably a result of the decreased use of the nephrotoxic and ship programs are associated with clear value and will be
ototoxic aminoglycoside antimicrobials in favor of β-lactams increasingly integral in the inpatient healthcare setting.
and fluoroquinolones. However, over the more recent years
reliance upon these less toxic agents has also been associated
with increases in bacterial resistance and CDAD. In light of Expert commentary
these findings, the use of aminoglycosides may increase. As Antimicrobial stewardship programs should be considered
noted previously, physician education was among those inter- mandatory for any inpatient hospital setting. The optimal
ventions reported to show the most benefit in terms of a stewardship team should include representatives from infec-
reduction in costs, antimicrobial resistance and incidence of tious diseases, pharmacy, microbiology and infection con-
CDAD. One study sought to survey clinicians in order to trol. Hospital administration support is essential in order to
determine their perceptions of various issues related to anti- establish such a team. Vigilant tracking of antimicrobial
microbial resistance. The results showed that clinicians per- costs, resistance pattern and the rate of CDAD, must take
ceived antimicrobial resistance to be less pertinent to their place to allow the team to responsively address and inter-
own institutions, but rather, a complicated problem on a vene. In addition, documentation of the associated out-
national level [42]. On the contrary, antimicrobial resistance is comes should consistently take place in order to confirm the
a widespread issue at the institutional level and clinicians must cost–benefit of such programs.
fully appreciate strategies in order to reduce antimicrobial
resistance selection pressure.
Five-year view
The use of antimicrobial stewardship teams will continue to
Conclusion grow in the acute care setting. In larger hospitals, the likely
This review on the impact of antimicrobial stewardship pro- composition of the team will be infectious diseases-trained
grams underscores and reinforces the conclusions of the physicians and infectious diseases-trained pharmacists.
IDSA/SHEA and others. The past 5 years have been associated However, in smaller facilities, cost efficiencies will probably
with additional studies demonstrating favorable outcome asso- result in the use of internists and pharmacist generalists. In
ciated with stewardship interventions. The most clearly linked contrast to the current situation, clinical microbiologists,
Key issues
• Antimicrobial stewardship programs offer value to acute-care hospital settings.
• Professional societies have established guidelines regarding the development of stewardship programs.
• While the inclusion of microbiology, infection-control and systems-support experts is optimal, the most commonly included members
of the stewardship team are infectious diseases-trained physicians and infectious diseases-trained pharmacists.
• The most likely outcomes associated with stewardship programs are cost avoidance, a reduction in bacterial resistance and a decrease
in Clostridium difficile-associated disease.
• Interventions intended to reduce bacterial resistance and Clostridium difficile-associated disease optimally include concomitant
infection-control interventions.
• The cost of such programs must be included in assessing the value of the intervention as there is currently a paucity of data that clearly
demonstrate a reduction in overall hospital costs.
• The use of informatics will optimally identify antimicrobial usage and bacterial-resistance trends.
www.future-drugs.com 221
Review Patel, Lawson & Guglielmo
17 South M, Royle J, Starr M. A simple 28 McGregor JC, Weekes E, Forrest GN et al. 38 Chang MT, Wu TH, Wang CY, Jang TN,
intervention to improve hospital antibiotic Impact of a computerized clinical decision Huang CY. The impact of an intensive
prescribing. Med. J. Aust. 178(5), 207–209 support system on reducing inappropriate antimicrobial control program in a Taiwanese
(2003). antimicrobial use: a randomized controlled medical center. Pharm. World Sci. 28(4),
18 von Gunten V, Amos V, Sidler AL, Beney J, trial. J. Am. Med. Inform. Assoc. 13(4), 257–264 (2006).
Troillet N, Reymond JP. Hospital 378–384 (2006). 39 Fowler S, Webber A, Cooper BS et al.
pharmacists’ reinforcement of guidelines for 29 Philmon C, Smith T, Williamson S, Successful use of feedback to improve
switching from parenteral to oral antibiotics: a Goodman E. Controlling use of antibiotic prescribing and reduce Clostridium
pilot study. Pharm. World Sci. 25(2), 52–55 antimicrobials in a community teaching difficile infection: a controlled interrupted
(2003). hospital. Infect. Control Hosp. Epidemiol. time series. J. Antimicrob. Chemother. 59(5),
19 Cook PP, Catrou PG, Christie JD, 27(3), 239–244 (2006). 990–995 (2007).
Young PD, Polk RE. Reduction in broad- 30 Schwartzberg E, Rubinovich S, Hassin D 40 Valiquette L, Cossette B, Garant MP, Diab H,
spectrum antimicrobial use associated with no et al. Developing and implementing a Pepin J. Impact of a reduction in the use of
improvement in hospital antibiogram. model for changing physicians’ prescribing high-risk antibiotics on the course of an
J. Antimicrob. Chemother. 53(5), 853–859 habits – the role of clinical pharmacy in epidemic of Clostridium difficile-associated
(2004). leading the change. J. Clin. Pharm. Ther. disease caused by the hypervirulent
20 Lutters M, Harbarth S, Janssens JP et al. 31(2), 179–185 (2006). NAP1/027 strain. Clin. Infect. Dis. 45(Suppl.
Effect of a comprehensive, multidisciplinary, 31 Zahar JR, Rioux C, Girou E et al. 2), S112–S121 (2007).
educational program on the use of antibiotics Inappropriate prescribing of 41 Muto CA, Blank MK, Marsh JW et al.
in a geriatric university hospital. J. Am. aminoglycosides: risk factors and impact of Control of an outbreak of infection with the
Geriatr. Soc. 52(1), 112–116 (2004). an antibiotic control team. J. Antimicrob. hypervirulent Clostridium difficile BI strain in
21 Ruttimann S, Keck B, Hartmeier C, Maetzel Chemother. 58(3), 651–656 (2006). a university hospital using a comprehensive
A, Bucher HC. Long-term antibiotic cost 32 Mach R, Vlcek J, Prusova M, Batka P, “bundle” approach. Clin. Infect. Dis. 45(10),
savings from a comprehensive intervention Rysavy V, Kubena A. Impact of a 1266–1273 (2007).
program in a medical department of a multidisciplinary approach on antibiotic • Example of combined antimicrobial
university-affiliated teaching hospital. Clin. consumption, cost, and microbial resistance stewardship and infection-control
Infect. Dis. 38(3), 348–356 (2004). in a Czech hospital. Pharm. World Sci. interventions toward the reduction of
22 Bruno-Murtha LA, Brusch J, Bor D, Li W, 29(5), 565–572 (2007). Clostridium difficile-associated disease.
Zucker D. A pilot study of antibiotic cycling 33 Meyer E, Buttler J, Schneider C et al. 42 Giblin TB, Sinkowitz-Cochran RL,
in the community hospital setting. Infect. Modified guidelines impact on antibiotic Harris PL et al. Clinicians’ perceptions of the
Control Hosp. Epidemiol. 26(1), 81–87 use and costs: duration of treatment for problem of antimicrobial resistance in health
(2005). pneumonia in a neurosurgical ICU is care facilities. Arch. Intern. Med. 164(15),
23 Davis SL, Delgado G, McKinnon PS. reduced. J. Antimicrob. Chemother. 59(6), 1662–1668 (2004).
Pharmacoeconomic considerations associated 1148–1154 (2007).
with use of intravenous-to-oral moxifloxacin 34 Khan R, Cheesbrough J. Impact of changes
for community-acquired pneumonia. Clin. in antbiotic policy on Clostridium difficile- Affiliations
Infect. Dis. 41(Suppl. 2), S136–S143 (2005). associated diarrhoea (CDAD) over a five-
• Dimple Patel, Pharm.D, BCPS
24 Martin C, Ofotokun I, Rapp R et al. Results year period in a district general hospital.
University of California, San Francisco School
of an antimicrobial control program at a J. Hosp. Infect. 54(2), 104–108 (2003).
of Pharmacy, 521 Parnassus Avenue, C-152,
university hospital. Am. J. Health Syst. Pharm. 35 Lautenbach E, LaRosa LA, Marr AM, San Francisco, CA 94143, USA
62(7), 732–738 (2005). Nachamkin I, Bilker WB, Fishman NO. Tel.: +1 415 514 9323
25 Apisarnthanarak A, Danchaivijitr S, Changes in the prevalence of vancomycin- Fax: +1 415 514 2680
Khawcharoenporn T et al. Effectiveness of resistant enterococci in response to dimple.patel@ucsf.edu
education and an antibiotic-control program antimicrobial formulary interventions:
• Wendy Lawson, BSc Hons. (Pharm)
in a tertiary care hospital in Thailand. Clin. impact of progressive restrictions on use of
Lead Pharmacist, Infectious Diseases,
Infect. Dis. 42(6), 768–775 (2006). vancomycin and third-generation
Hammersmith Hospitals NHS Trust, Du
cephalosporins. Clin. Infect. Dis. 36(4),
26 Arnold FW, McDonald C, Smith S, Newman Cane Road, London, W12 OHS, UK
440–446 (2003).
D, Ramirez JA. Improving antimicrobial use Tel: +44 208 346 1000
in the hospital setting by providing usage 36 O’Connor KA, Kingston M, Fax: +44 208 746 1160
feedback to prescribing physicians. Infect. O’Donovan M, Cryan B, Twomey C, wendy.lawson@imperial.nhs.uk
Control Hosp. Epidemiol. 27(4), 378–382 O’Mahony D. Antibiotic prescribing policy
• B Joseph Guglielmo, Pharm.D
(2006). and Clostridium difficile diarrhoea. Q. J.
University of California, San Francisco School
Med. 97(7), 423–429 (2004).
27 Gomez MI, Acosta-Gnass SI, of Pharmacy, 521 Parnassus Avenue, C-152,
Mosqueda-Barboza L, Basualdo JA. 37 Guglielmo BJ, Dudas V, Maewal I et al. San Francisco, CA 94143, USA
Reduction in surgical antibiotic prophylaxis Impact of a series of interventions in Tel.: +1 415 476 2354
expenditure and the rate of surgical site vancomycin prescribing on use and Fax: +1 415 476 6632
infection by means of a protocol that controls prevalence of vancomycin-resistant guglielmoj@pharmacy.ucsf.edu
the use of prophylaxis. Infect. Control Hosp. enterococci. Jt Comm. J. Qual. Patient Saf.
Epidemiol. 27(12), 1358–1365 (2006). 31(8), 469–475 (2005).