2010 International Conference on Artificial Intelligence and Computational Intelligence

A Scenario-Based Method for Safety Certification of Artificial Intelligent Software

Guoqi Li, Minyan Lu, and Bin Liu

Department of Reliability and System Engineering,
Beihang University,
Beijing, China
{gqli, lmy, liubin}@buaa.edu.cn

Abstract—Artificial intelligence (AI) is attractive for safety- II. D ESIGN M ETHOD

critical fields. However, there have been few success cases,
for the AI technique is usually lack of determinism and
predictability, which is usually regarded as a disqualifier in a
safety context. Increased researches and supererogatory efforts
are providing to incorporate AI into the safety-critical systems
in recent years. In this paper, we present a scenario-based
method for safety certification, with the method AI modules
of system could be evaluated before invoked, if its trustability
is satisfied, then the program will be performed for safety-
critical systems, otherwise it will be terminated to ask human
assistance or postpone the missions.
Keywords-software trustability; intelligent systems; software
engineering;
I. I NTRODUCTION
Artificial intelligence (AI) is attractive for safety-critical
fields. For example, a typical potential application is un-
manned aerial vehicle, widely used for military purposes and
also widely interested by academic researchers [1]. However,
there have been few success cases, for the AI technique
is usually lack of determinism and predictability, which is
usually regarded as a disqualifier in a safety context[2]. In-
creased researches and supererogatory efforts are providing
to incorporate AI into the safety-critical systems in recent
years [3], [4]. Putting constraints on the adaptability of the
AI [3] and using suitable fault-tolerant design [4] have been
reported as replacements of practicably impossible safety
assurances on AI modules of systems.
In this paper, we present a scenario-based method for
safety certification of AI software. The method derived
from the scenario-based reliability analysis method for
component-based software [5] and specially extended for
AI. With the method, the quantitative safety of the software
with AI modules could be evaluated before invoked, if the
trustability is satisfied, then the program will be performed
for safety-critical systems, otherwise it will be terminated to
ask human assistance or postpone the missions.
In the next section, the philosophy and the detail of the
method are described and a case study on a video navigation
system for a tiny robot is conducted in the section 3. Finally,
draw conclusions and address our future works.
AI research is well invested and it is divided into subfields
that often fail to communicate with each other. For exam-
ple, a program for playing chess and an intelligent agent
for internet search engine may based absolutely different
technique, the corresponding developers may come from
deferent research fields, however they are all typical AI
applications. What is the features of AI and how could it
be incorporated with safety critical fields? In the following
we’ll give our opinions.
A. AI and Scenarios
Thanking about two common kinds of AI technique. One
is program for playing chess. The design of these kinds of
programs typically have 3 steps:
1) Define a objective function as the evaluation criteria.
2) Search all the possible solutions in the limited condi-
tions as possible as the machine can.
3) Select one of the most optimal solution according to
the criteria.
Although there are many additional modules for better
performance, such as general patterns library, the primary
idea is thus.
The other famous AI technique is machine learning.
For the set of all possible behaviors given all possible
inputs is too complex to describe generally in programming
languages, these problems have to solved by algorithms
simulating human beings thinking. Cluster, classification and
regression are typical purpose of machine learning. Cluster
is unsupervised learning, by compare the structure of input
data, extract new knowledge. Classification and regression
are supervised learning and usually produce a model for
prediction, such as neural network, support vector machine
and etc.
Cross-validation, sometimes called rotation estimation, is
a technique for assessing how the results of a machine
learning algorithm will generalize to an independent data
set. It is mainly used in settings where the goal is prediction,
and one wants to estimate how accurately a predictive model
will perform in practice. One round of cross-validation
involves partitioning a sample of data into complementary
subsets, performing the analysis on one subset (called the

978-0-7695-4225-6/10 $26.00 © 2010 IEEE 481

DOI 10.1109/AICI.2010.339
training set), and validating the analysis on the other subset
(called the validation set or testing set). To reduce variability,
multiple rounds of cross-validation are performed using
different partitions, and the validation results are averaged
over the rounds [6].
Could the result of cross-validation be treated as the
reliability of the machine learning module? The answer is
yes, if we have ensured that the input data have the same
semantics with the trailing set or testing set. For the program
of playing chess. Deep Blue, a chess-playing computer
developed by IBM, on May 11, 1997, won a six-game match
by two wins to one with three draws against world champion
Garry Kasparov. However, as far as I known, chess-playing
computer could not win any professional Go player. Go,
known in Chinese as weiqi, is a kind of popular chess in east
Asia. It is inevitable that a chess-playing computer would
lost Lee Chang-ho, who is one of the most famous Go player
in the world. If we ensure that It is Garry Kasparov, but
not Lee Chang-ho, who is sit opposite to AI, then the AI
technique is reliable and trustable.
Consequently, if we could guarantee that the precondition
is suitable for the AI technique, then its result is predictable.
The formal language Z, can used to formally validation the
precondition of a function of a program. The AI precondition
should also be validated for reliability.
We define the precondition of AI as scenarios. Scenarios
are a powerful antidote to the complexity of systems and
analysis [7]. Scenarios vary from brief stories to richly
structured analysis. In the the scenario-based reliability anal-
ysis method for component-based software [5], the sequence
diagram of software components are treated as structural
scenarios. On the contrary, scenarios for AI should emphasis
on environmental or situation scenarios. For example, in the
case study (in the next section), we’ll define the scenarios
for a video navigation component, the light intensity and
visibility are concerned aspects of scenarios for evaluate the
reliability of the AI components.
B. Detail of the Design
The scenario-based reliability analysis method for
component-based software [5] is selected as the high level
reliability evaluation method. Incorporate the AI modules re-
liability evaluation to completely evaluate reliably of intelli-
gent software systems. The scenario-based reliability analy-
sis method for component-based software introduces a prob-
abilistic model and a reliability analysis technique applicable
to high-level designs. The technique is named Scenario-
Based Reliability Analysis (SBRA). SBRA is specific for
component-based software whose analysis is strictly based
on execution scenarios, which is a kind of structural scenar-
ios. Using these scenarios, construct a probabilistic model
named ”Component-Dependency Graph” (CDG). CDGs are
directed graphs that represent components, component re-
liabilities, link and interface reliabilities, transitions, and
482
3&
&RPPXQLFDWHWKURXJK
:LUHOHVV QHWZRUN E\
7&3,3SURWRFRO
$50
&RPPXQLFDWHWKURXJK
566HULDO3RUW
DWPOV
Figure 1. The picture of the tiny mobile robot and its three layer
architecture.
transition probabilities. In CDGs, component interfaces and
link reliabilities are treated as first class elements of the
model. Based on CDGs, an algorithm is also given in the
paper to analyze the reliability of the application as the
function of reliabilities of its components and interfaces.
What’s the paper’s contribution is providing a environ-
ment scenario based method to evaluate reliability of AI
modules and seamless integrated with the reliability analysis
method for component-based software.
III. C ASE S TUDY
A. Introduction of a Video Navigation Tiny Robot
Figure 1 is the Picture of the tiny mobile robot and
its three layer architecture. The object of the system is
automatic navigation by pattern recognition through video
capture. It can recognize the obstacle in the flow of video.
We selected a ARM11 processor (Samsung S3C6410) as
main control processor, and select Windows CE as op-
eration system. Main control processor will be connected
to atml89s52(a kind of 8051 single chip microcomputer)
through RS232 Serial Port, and then atml89s52 control the
motors through a motor drive module. The motion control
program is written in the RAM of atml89s52. Ranging
sensor provide additional assurance to avoid crash. The
ranging sensor connect to atml89s52 directly by I/O port.
Operationally, Main control processor capture and send
digital image signal to PC upper computer through Wireless
network communications by TCP/IP protocol. To implement
automatic navigation, PC upper computer need to do digital
image processing and pattern recognition through digital im-
age signal and send control command to atml89s52 module
through ARM11 module. There is also an interface in the
PC to control the motion of the robot manfully.
Figure 2 is the architecture of the distributed software sys-
tem. Based the above information, the system is component-
based and suit to use the scenario-based reliability analysis
method for component-based software. For the reliability of
AI components, the next subsection will described.
 0DQXDOFRQWURO 9LGHRFDSWXUH
FRPSRQHQW LQ3&
&RPPXQLFDWLRQ VRIWZDUHLQ
,PDJHSURFHVV 7&3,3573LQ DWPOV
$50
3DWWHUQ
UHFRJQLWLRQ &RPPXQLFDWLRQ
56LQ$50
Figure 2. The architecture of the distributed software system.
Figure 3. One of the characters the intelligent system should be recognized.
B. Determine the AI Component reliability with Scenario
Let’s consider a specific mission of the tiny robot. Figure
3 is a Chinese character means ”Stop”. The tiny robot
capture 5 pieces of pictures every second. Then, do image
process on the pictures. If a rectangle is recognized, then
it will do character recognition. There are many Chinese
characters have been learned by the program. If the picture
is recognized, corresponding command will be send to the
mechanical components ultimately.
It is obviously that the recognition accuracy is influenced
by the environment. For example, the light intensity, at
daytime the recognition is about 96.7% and at twilight,
the accuracy is lower. Visibility is another scenario features
influence the recognition accuracy.
We add a light intensity sensor and a software component
to calculate the visibility of the environment to the tiny robot.
Then the recognition accuracy of the AI component could
be predictable. Adding the information to the scenario-based
reliability analysis method for component-based software.
The quantitative reliability of the distributed software with
AI modules could be evaluated before invoked, if the reli-
ability is satisfied, then the program will be performed for
safety-critical systems, otherwise it will be terminated to ask
human assistance or postpone the missions. Additional, we
should validate that the scenarios are completeness and cor-
rectness for the AI algorithm. That is feasible. Completeness,
correctness and etc. are jargons of safety engineers.
483
IV. C ONCLUSION
In this paper, we present a scenario-based method for
safety certification of intelligent systems. With the method,
the quantitative trustability of the software with AI modules
could be evaluated before invoked. For the method, we
should clarify two points:
1) Enumerate all the elements influence the accuracy of
AI.
2) quantify the influences.
We should establish a scenario architecture to complete the
two task. However, in many applications, to finish the two
task many need AI algorithms. Then, it become a recursive
problem. We should avoid the conditions.
This paper is only an exploration on the topic, we’ll give
deep research and timely report in the future.
ACKNOWLEDGMENT
The tiny robot used in the case study is open
source and the source code is available by email now
(gqli@buaa.edu.cn). A website for the robot will be set up in
the recent future for downloading source code and providing
related documents.
R EFERENCES
[1] Stanford Univesity autonomous helicopter. The goal of this
project is to push the state-of-the-art in autonomous helicopter
flight: extreme aerobatics under computer control. Available at
http://heli.stanford.edu/.
[2] R. Varadaraju, ”A Survey of Introducing Ar-
tificial Intelligence Into the Safety Critical
System Software Design Process. Available” at:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.114.8602
[3] A. Mili, G. Jiang, B. Cukic, Y. Liu and R. Ben Ayed, ”Towards
the Verification and Validation of Online Learning Systems:
General Framework and Applications”. In Proceedings of
Hawaii International Conference on System Sciences. 2004.
[4] Z Kurd and T P Kelly, ”Using Safety Critical Artificial Neural
Networks in Gas Turbine Aero-Engine Control”. In Proceed-
ings of the 24 th International Conference on Computer Safety,
Reliability and Security (SAFECOMP’05), September 2005
(Springer Verlag Lecture Notes in Computer Science LNCS).
[5] S. M. Yacoub, B. Cukic and H. H. Ammar, ”Scenario-Based
Reliability Analysis of Component-Based Software”. In pro-
ceedings of the 10th International Symposium on Software Re-
liability Engineering table of contents. EEE Computer Society
Washington, DC, USA. 1999.
[6] Cross-validation (statistics). From Wikipedia, the free encyclo-
pedia. http://en.wikipedia.org/wiki/Cross-validation
[7] Weidenhaupt, K., K. Pohl, M. Jarke, and P. Haumer, ”Scenarios
in System Development: Current Practice”, IEEE Software,
March/April 1998, pp34-45.