UNIVERSIDAD DE LOS HEMISFERIOS

SISTEMAS DE INFORMACION
Profesor: Francisco Ávila

Nombre: Juan José Espinosa

Laboratorio: Royal Bank of Canada’s Software Woes

Founded in 1864 and chartered in 1869 as the Merchants’ Bank of Halifax, Royal Bank of Canada
(RBC) took its current name in 1901. In 1941, based on its total assets of $1 billion, RBC became
Canada’s largest bank. Twenty years later, RBC installed its first computer, making it the first bank
in Canada to employ such technology. In October 2003, the organization had 60,000 employees,
totaled $413 billion in assets, and served 12 million customers. Including worldwide operations,
RBC has spent $1.6 billion on technology.

On Monday, May 31, 2004, the RBC information technology staff made a programming
upgrade to what has been described as “key banking software.” According to Martin Lippert, RBC’s
vice chairman and CIO, a glitch in the bank’s computer systems “was most likely caused by a single
worker entering ‘a relatively small number’ of incorrect pieces of code during the update.” The
glitch resulted in a massive computer failure that affected millions of banking customers around
the country. Lippert specified that the incorrect code was related to “transit numbers or field
identifiers in a table.” The mistakes worked their way through the bank’s systems quickly, but
conspicuously. RBC had already discovered the problem by six o’clock the next morning.
Unfortunately, recognition of the error wasn’t even half the battle.

In fact, RBC fixed the programming error that had been made on May 31 early on Tuesday,
June 1. Still, several days later, millions of RBC customers were still unable to check their account
balances, had not received their paychecks, or had automatic payments or transfers delayed. The
reactions of customers, as could be expected, were tinged with displeasure. Some customers
merely suffered the inconvenience of not having access to information that they were accustomed
to having at their fingertips all of the time. Other customers, those who count on their paychecks
to get by, had their lives affected more seriously. One such person, a law firm executive assistant
named Andrea Mitchell, was forced to ask her employer for a cash advance to make up for her
temporarily lost paycheck and to meet her personal expenses.

So, if the human programming error was corrected so swiftly, why did RBC continue to
experience problems related to the glitch over the course of a work week? The glitch put into
motion a chain of management and control procedures that exacerbated the problem. Procedures
that were intended to fix the problem instead caused a logjam of activity from which the bank
could not recover immediately.

The first question that most people asked was, Why weren’t backup systems used to keep
business flowing while the main systems were being fixed? As stated on the bank’s Web site,
“Back up facilities exist in case our primary facility is disabled. As a matter of policy, therefore, all
program changes are implemented simultaneously to both the primary and backup facilities.”
Therefore, the same error that compromised the main system also affected the backup system.

According to Lippert, in addition to the programming code update being entered incorrectly,
the new code was not tested properly before it was deployed. RBC policy calls for all new pieces of
software to be tested thoroughly before they are used in production systems.

On Tuesday, June 1, with the knowledge that random duplications of transactions were
occurring in its systems, RBC decided to suspend its end-of-day processing rather than let it run
with incomplete or inaccurate data. When verifying the health of its systems took longer than
expected, RBC was left with a backlog of transactions to process. All the while, new transactions
were pouring in, as they would on a normal business day, and these added to the logjam. On
Wednesday, June 2, the IT department was confident that it could make up the ground lost on
Tuesday and remain current with Wednesday’s transactions. The newer transactions were placed
at the end of queue because of a sequential numbering system, so the backlog had to be cleared
before RBC could process any new transactions. Also slowing down the effort was the bank’s
decision to monitor data processing manually following the glitch to make sure that no coding
errors persisted. Trying to process two days of transactions at once proved to be more difficult
than the bank imagined it would be.

As if the event itself were not bad enough for business, RBC came under criticism for the
way in which it handled the public relations end of the problem. First, RBC made assurance that all
systems and accounts would be normalized by Thursday, June 3. Confident in this assessment, the
bank’s CEO, Gordon Nixon, left for Europe on Wednesday, June 2. When customers were still
experiencing difficulties with their accounts later in the week, the bank appeared weak by not
having its leader available to address customers’ concerns. As a possible reason for Nixon’s
untimely exit from the scene, John Layne, of the crisis management company Contingency
Management Consultants, surmises the common organizational flaw in which lowerlevel
employees are loathe to pass bad news up the corporate ladder. In Nixon’s absence, other RBC
executives took turns addressing the media and the public, though the first public comments on
the glitch did not come forth until late Wednesday afternoon. Having a single representative of the
bank to interface with the public would have inspired more confidence in customers.

Unfortunately for RBC, the cumulative effect of its errors was widespread. About 62,000
government workers in Ontario and 10,000 in New Brunswick didn’t receive their automatic
deposits even if they didn’t do their personal banking at RBC. Their provincial governments did use
RBC to route the payroll, so the deposits didn’t reach the workers’ own banks. Hundreds of
thousands of other workers around Canada ran into similar delays or difficulties accessing their
deposits. The government of Ontario indicated that it would issue physical checks to employees
who still had not received their deposits by the Monday following the initial computer glitch.

RBC’s service disruption had one particularly dangerous side effect, over which the bank had
little control. Opportunistic scam artists seized on the glitch to unleash a phishing scam that
targeted RBC customers, taking advantage of the customers’ frustrations, security fears, or simply
their naïveté. An e-mail message with the subject line “Official information from RBC Royal Bank”
stated that “due to the increased fraudulent activity within our site we are undertaking a review of
member accounts.” The email linked to a page that looked like RBC’s official Web site that
instructed customers “be sure to enter both client card number or business card number &
password otherwise your account will not be verified and your access to the account will be
blocked.” Once RBC customers entered the requested information at that bogus site, hackers
could access their accounts.

RBC announced that it had resumed normal business practices during the week beginning
June 8, 2004. The bank continued to resolve discrepancies from service charges and overdraft
interest that resulted from the disruption and indicated that all accounts should be reflected
accurately in customers’ next statements, by June 30. On June 18, RBC announced that it had
hired Crawford Adjusters Canada as its “administrator for non-banking-related costs and losses in
the bank’s processing disruption.” The bank made claim forms available, by phone and on the
Internet and set a deadline of September 30, 2004 for customers to file claims. RBC was to handle
all claims under $100 itself, leaving larger claims to Crawford. The total loss to RBC was contingent
on the number of claims filed. However, that cost could rise significantly based on a classaction
suit filed in Quebec that requested damages of $500 for each customer that was affected.

In hindsight, analysts agree that RBC handled the technology portion of its problem
effectively. The bank erred mostly in its estimates of how long the recovery period would be. In
addition to issuing a formal apology, RBC created an area of its Web site that was devoted to
detailed explanations of the problem and updates on the progress of restoring its records. The
bank also enlisted IBM Corporation as a consultant to investigate the causes of the problem and
provide guidance on how to avoid such problems going forward. RBC offered refunds to its
customers for any banking fees that they incurred as a result of the computer problems and
agreed to reimburse other financial institutions and their customers for certain expenses that may
have resulted from RBC’s errors. These moves could go a long way toward restoring faith in
customers of the bank, who, in the immediate aftermath, were understandably concerned about
the security of their finances. One client said that she didn’t intend to leave RBC just then, but, “If
it happened again, I may not be so loyal.”

Sources: Ian Austen, “Bank Software Woes Leave Many Canadians Without Money,” New York
Times, June 7, 2004; “RBC Admits Human Error, Scam Artists Act,” Ottawa Business Journal, June
10, 2004; Lindsay Bruce, “RBC Glitch Still Not Ironed Out,” ITWorldCanada.com, June 7, 2004; Chris
Conrath, “Anatomy of a Snafu,” ITWorldCanada.com, June 11, 2004; Paul Waldie, “RBC Calls in
Help,” Globe and Mail, www.theglobeandmail.com, accessed June 19, 2004; Mel Duvall, “RBC’s
Account Imbalance,” Baseline Magazine, July 1, 2004; and CP, “RBC Still Making Up for Glitch,”
London Free Press, www.canoe.ca, accessed June 19, 2004.

CASE STUDY QUESTIONS

1.Provide a summary of the security and control problems experienced by the Royal Bank of
Canada. How did this problem impact the organization and its clients?

En primer lugar, el banco de Canadá es de los más conocidos en el mundo, el principal problema
fue la contraseña, es decir que un empleado digitó una clave que no era la correcta, ocasionando
una serie de problemas en el sistema, éste problema se vio desencadenando otros. El error
humano es muy común, es por eso que muchos de los casos de inseguridad de la red que no fue
con intención, son generados por personas internas. Los errores en la programación generaron
que los clientes no puedan revisar su estado de cuenta, su rol de pagos, o que las empresas no
puedan depositar a sus empleados. Además, ese no era el único problema, sino que también el
sistema suplementario o el de caso de emergencia también falló porque lo que le convertía en un
banco próximo a la quiebra. En mi opinión, también fallaron en no tener un auditor antes que
ocurriera éste evento.

2.What management, organization, and technology factors contributed to the control problems
at RBC?

Los sistemas de seguridad es uno de los activos más importantes en una organización puesto que
si falla las repercusiones pueden ser altas, como ya lo vimos. Ahora bien, el manejo debe ser
cuidadoso. EL manejo dentro de la organización por John Layne fue un poco descuidado al no
prever una crisis de éstas. Se podría decir que la implementación de equipo más moderno ayudo a
solucionar éste problema, en lo que tiene que ver con piezas de software.

3.What was the most critical flaw in the security and control systems in place at RBC? Explain
your answer.

No tener un Sistema de seguridad adecuado y el personal capacitado con el objeto de prever éstas
crisis. Un sistema de seguridad brinda confianza y elimina cierta incertidumbre en los clientes, hay
que destacar que no se puede resguardar la información al 100%. Otro problema fue los
estafadores virtuales, ya que el banco tampoco previó cosas como éstas, ya fueron más problemas
que se ocasionaron.

4.How well did RBC respond to its computer problem? What do you think the bank might have
done differently to prevent the problem and to handle it once it did occur?

RBC, en mi criterio, respondió de manera adecuada y rápida para que no todo el sistema se caiga y
puedan sufrir grandes pérdidas. Como lo habíamos dicho antes, RBC deberá implantar otro
sistema de información que le permita cubrir las necesidades de los clientes en caso de crisis,
además de plataformas de sistemas de seguridad, personal altamente capacitado y una auditoría
constante.

5.If you were a manager at RBC, how would you prevent a problem like this from happening?

EL Sistema de seguridad implementado en mi empresa deberá ser probado por expertos por un
lapso de tiempo antes de ser aplicado en el día-día. Además de lo dicho anteriormente en la cuarta
pregunta. La toma de decisiones es algo muy significativo en estos momentos, puesto que se
debería poner énfasis en los estafadores virtuales, además de una buena comunicación con los
clientes para que esos tipos de problemas no existan o que simplemente no se hagan más
grandes.