PowerPoint Slides to Accompany

A Gift of Fire: Social, Legal, and Ethical

Issues for Computers and the Internet
(2nd Edition)
by

Sara Baase

San Diego State University

PowerPoint slides created by Sherry Clark
Copyright 2003 Prentice Hall

CSC 3250

Topic 4: Can We Trust the Computer?

A Gift of Fire
Can We Trust the Computer?
What Can Go Wrong?
Case Study: The Therac-25
Increasing Reliability and Safety
Perspectives on Failures, Dependence, Risk, and Progress
Computer Models

CSC 3250

Topic 4: Can We Trust the Computer?

What Can Go Wrong?

Facts About Computer Errors
Error-free software is not possible.
Errors are often caused by more than one factor.
Errors can be reduced by following good procedures and professional
practices.

CSC 3250

Topic 4: Can We Trust the Computer?

What Can Go Wrong?

The Roles of People in Computer-related
Problems:
Computer User
At home or work, users should understand the limitations of computers
and the need for proper training and responsible use.

Computer Professional
Understanding the source and consequences of computer failures is
valuable when buying, developing, or managing a complex system.

Educated Member of Society

Personal decisions and political, social, and ethical decisions depend on
understanding computer risks.

CSC 3250

Topic 4: Can We Trust the Computer?

What Can Go Wrong?

Categories of Computer Errors and Failures
Problems for Individuals:
usually in their role as consumers.
who are incorrectly identified by inaccurate law enforcement databases.

System Failures:
affecting large numbers of people and/or costing large amounts of
money.

Safety-Critical Applications:
where people may be injured or killed.

CSC 3250

Topic 4: Can We Trust the Computer?

What Can Go Wrong?

Problems for Individuals
Billing Errors
Lack of tests for inconsistencies and inappropriate amounts.

Database Accuracy Problems

Incorrect information resulting in wrongful treatment or acts.

CSC 3250

Topic 4: Can We Trust the Computer?

What Can Go Wrong?

Problems for Individuals (contd)
Causes:

CSC 3250

Large population.
Human common sense not part of automated processing.
Overconfidence in the accuracy of data from a computer.
Errors in data entry.
Information not updated or corrected.
Lack of accountability for errors.

Topic 4: Can We Trust the Computer?

What Can Go Wrong?

System Failures
Communications:
Telephone, online, and broadcast services.

Business:
Inventory and management software.

Financial:
Stock exchange, brokerages, banks, etc..

Transportation:
Reservations, ticketing, and baggage handling.

CSC 3250

Topic 4: Can We Trust the Computer?

What Can Go Wrong?

System Failures (contd)
Causes:

CSC 3250

Insufficient testing and debugging time.

Significant changes in specifications (during and after project begun).
Overconfidence in system.
Mismanagement of the project.

Topic 4: Can We Trust the Computer?

What Can Go Wrong?

Safety-Critical Applications
Military
Power Plants
Aircraft
Trains
Automated Factories
Medicine
others.

CSC 3250

Topic 4: Can We Trust the Computer?

10

What Can Go Wrong?

Safety-Critical Applications
Causes:

CSC 3250

Overconfidence.
Lack of override features.
Insufficient testing.
Sheer complexity of system.
Mismanagement.

Topic 4: Can We Trust the Computer?

11

Case Study: The Therac-25

Atomic Energy of Canada Limited (AECL)

The Therac-25 was a software-controlled

radiation-therapy machine used to treat
people with cancer. (1985 1987)
Overdoses of radiation
Normal dosage is 100200 rads.
It is estimated that 13,000 and 25,000 rads were given to six people.
Three of the six people died.

CSC 3250

Topic 4: Can We Trust the Computer?

12

Case Study: The Therac-25

Therac-25 Radiation Overdose
Multiple Causes:

CSC 3250

Poor safety design.

Insufficient testing and debugging.
Software errors.
Lack of safety interlocks.
Overconfidence.
Inadequate reporting and investigation of accidents.

Topic 4: Can We Trust the Computer?

13

Why Nuclear
pembangkit listrik tenaga nuklir dapat
mengurangi 155 juta ton metrik CO2 yang
sedianya akan dihasilkan oleh pembangkit
tenaga bahan bakar fosil (minyak dan
batubara).
pembangkit listrik tenaga nuklir yang ada saat
ini dapat menyebabkan hilangnya 59 juta
metrik ton CO2 pada tahun 2020
CSC 3250

Topic 4: Can We Trust the Computer?

14

Increasing Reliability and Safety

What Goes Wrong?
Computer Systems Fail Because:
The job they are doing is inherently difficult, and
The job is done poorly.

Compounding the Reliability Issue:

Developers and users exhibit overconfidence in the system.
Reused system software may not work in different environments.

CSC 3250

Topic 4: Can We Trust the Computer?

15

Increasing Reliability and Safety

Professional Techniques

CSC 3250

Follow good software practices.

Exhibit professional responsibility at all levels of development and use.
Construct well-designed user interfaces.
Take human factors into account.
Include built-in redundancy.
Incorporate self-checking where appropriate.
Follow good testing principals and techniques.

Topic 4: Can We Trust the Computer?

16

Increasing Reliability and Safety

Law and Regulation
Criminal and Civil Penalties:
to recover loss from faulty or unsafe systems.

Liability and Civil Laws:

to provide incentives to produce reliable and safe systems.

Warranties:
to guarantee a certain level of quality.

Federal or State Regulations:

to protect the public.

CSC 3250

Topic 4: Can We Trust the Computer?

17

Increasing Reliability and Safety

Law and Regulation (contd)
Database Accuracy Enforcement:
to protect the public from inaccurate information maintained by private companies and
government.

Mandatory Licensing of Software Developers:

to ensure proper training, competency, and continuing education.

CiscoCertifiedNetworkAssociate(CCNA)
MicrosoftCertifiedSystemsAdministrator(MCSA)
MicrosoftCertifiedTechnologySpecialist(MCTS)
TheMicrosoftCertifiedITProfessional(MCITP)
TheMicrosoftCertifiedProfessionalDeveloper
(MCPD)
MicrosoftCertifiedSystemsEngineers(MCSEs)
MicrosoftCertifiedApplicationDevelopers(MCADs)
CSC 3250

Topic 4: Can We Trust the Computer?

18

Perspectives on Failures,
Dependence, Risk, and Progress
Failures
What are acceptable rates of failures?
How accurate should software be?

Dependence
How dependent on computer systems are our ordinary activities?
How useful are computer systems to our ordinary activities?

Risk and Progress

How do new technologies become safer?
Can progress in software safety keep up with the pace of change in
computer technology?

CSC 3250

Topic 4: Can We Trust the Computer?

19