You are on page 1of 19

C

Operational Profile Testing: A White Paper

Software Reliability Testing (AKA Operational Profile Testing)


Software reliability testing is testing based on the operational profile1. Test cases are derived from the
OP, executed, and the results evaluated based on pass/fail criteria. Reliability is calculated from results.
In theory, software reliability testing is relatively straightforward. In practice, the concept is far from
trivial. To implement software reliability testing effectively the following elements should be in place. A
project should have the ability to accurately specify software system behavior in the form of a state
machine; this state machine is the operational profile. The project should be able to represent the
operational profile as a behavioral model. A test generation tool should be available that operates on
the profile to automatically produce a statistically significant test sample. The execution of the test
sample should be automated to ensure statistical confidence in the results. Without each of these
elements any project will have difficulty in generating realistic and useful reliability estimates from
software testing. In this section a framework for achieving these elements is introduced.
The content of this section is illustrated through an example in order to provide maximum clarity.
Specification
Following is a prescription for documenting a software specification as a state machine.
1) Identify All Inputs from All Actors Interfacing with the System and All Outputs from the System
Enumerate all possible values that could be received from input source
2) Identify All Operating Variables for the System Under Test (SUT)
These are externally observable parameters that impact a) when an input is possible and/or b) the
required system response based on the variable value. See figure below.
E.g. all switches wired to a control system are considered operating variables

Operating

System
Under Test

Variables

3) Specify Behavior for each Value Variant2 of each Input


For each input value variant specify:
a) Operating variable constraints (variable values MUST be true for input to occur)
b) Nominal required response (response is always observed)
c) All alternative responses based on unique operating variable values (response MAY occur)
d) All operating variable updates resulting from the stimulus and response
1

John D. Musa, Operational Profiles in Software Reliability Engineering, IEEE Software Volume 10 Issue 2, March 1993 pp 14-32

An input variant is one that invokes a different response from the system based on the value of the input

Copyright Unpublished: Cognitive Concepts LLC

Page 1

C
C

Operational Profile Testing: A White Paper

Example: System Under Test is an Elevator Simulator Developed in LabVIEW (National Instruments)

Identify All Interfaces


Input Interfaces
Floor Selector in Cabin
Floor Controls on Wall
Start and Stop Buttons
Time (abstract interface)
Output Interface
Floor Position indicator
For simplicity, the specification will cover only Floors 0 through 3.
1. Identify Inputs
Floor Selection In Cabin
Press Cabin Floor 0
Press Cabin Floor 1
Press Cabin Floor 2
Press Cabin Floor 3
Floor Controls on Wall
Press Floor 0 Up
Press Floor 1 Up
Press Floor 1 Down
Press Floor 2 Up
Press Floor 2 Down
Press Floor 3 Down
Copyright Unpublished: Cognitive Concepts LLC

Page 2

C
C

Operational Profile Testing: A White Paper

Application Buttons
Press Start Button
Press Stop Button
Timers
Travel Timer Expires
Floor Timer Expires
Identify Outputs
Floor Position
Arrive at Floor 0
Arrive at Floor 1 Up
Arrive at Floor 1 Down
Arrive at Floor 2 Up
Arrive at Floor 2 Down
Arrive at Floor 3
Stop at Floor 0
Stop at Floor 1
Stop at Floor 2
Stop at Floor 3
Application Responses
Application Started
Application Stopped
2. Identify Operating Variables
Current Position [0, 0-1, 1, 1-2, 2, 2-3, 3]
Current Direction [Stationary, Up, Down]
Motion Status [not moving, moving]
Travel Timer [inactive, active]
Floor Timer [inactive, active]
High Floor Selected [0, 1, 2, 3, null]
Low Floor Selected [0, 1, 2, 3, null]
Cabin Floor 0 [not selected, selected]
Cabin Floor 1 [not selected, selected]
Cabin Floor 2 [not selected, selected]
Cabin Floor 3 [not selected, selected]
Floor 0 Up [not selected, selected]
Floor 1 Up [not selected, selected]
Floor 1 Down [not selected, selected]
Floor 2 Up [not selected, selected]
Floor 2 Down [not selected, selected]
Floor 3 Down [not selected, selected]
Copyright Unpublished: Cognitive Concepts LLC

Page 3

C
C

Operational Profile Testing: A White Paper

3. Specify Behaviors
Input: Press Cabin Floor 0
Operating Variable Constraints:
Cabin Floor 0 = not selected
Current Position <> 0
Behavior:
IF (Cabin Floor 0 Pressed) System MUST Set Cabin Floor 0 = Selected
AND System MUST set Low Floor Selected = 0
IF (High Floor Selected = Null) System MUST Set High Floor Selected = 0
AND Set Current Direction = Down AND Set Motion Status = Moving
AND Set Travel Timer = active
Input: Press Cabin Floor 1
Operating Variable Constraints:
Cabin Floor 1 = not selected
Current Position <> 1
Behavior:
IF (Cabin Floor 1 Pressed) System MUST Set Cabin Floor 1 = Selected
IF (Low Floor Selected <> 0) System MUST Set Low Floor Selected = 1
IF (High Floor Selected <> 2 or 3) System MUST Set High Floor Selected = 1
IF (Current Direction = Stationary) THEN
[IF (Current Location < 1) System MUST Set Current Direction = Up
ELSE System MUST Set Current Direction = Down]
AND Set Motion Status = Moving and Travel Timer = Active
Input: Press Cabin Floor 2
Operating Variable Constraints:
Cabin Floor 2 = not selected
Current Position <> 2
Behavior:
IF (Cabin Floor 2 Pressed) System MUST Set Cabin Floor 2 = Selected
IF (Low Floor Selected <> 0 or 1) System MUST Set Low Floor Selected = 2
IF (High Floor Selected <> 3) System MUST Set High Floor Selected = 2
IF (Current Direction = Stationary) THEN
[IF (Current Location < 2) System MUST Set Current Direction = Up
ELSE System MUST Set Current Direction = Down]
AND Set Motion Status = Moving and Travel Timer = Active

Copyright Unpublished: Cognitive Concepts LLC

Page 4

C
C

Operational Profile Testing: A White Paper

Input: Press Cabin Floor 3


Operating Variable Constraints:
Cabin Floor 3 = not selected
Current Position <> 3
Behavior:
IF (Cabin Floor 3 Pressed) System MUST Set Cabin Floor 3 = Selected
AND System MUST set High Floor Selected = 3
IF (Low Floor Selected = Null) System MUST Set Low Floor Selected = 3
AND Set Current Direction = Up AND Set Motion Status = Moving
AND Set Travel Timer = active
Input: Press Floor 0 Up
Operating Variable Constraints:
Floor 0 Up = not selected
Current Position <> 0
Behavior:
IF (Floor 0 Up Pressed) System MUST Set Floor 0 Up = Selected
AND System MUST set Low Floor Selected = 0
IF (High Floor Selected = Null) System MUST Set High Floor Selected = 0
AND Set Current Direction = Down AND Set Motion Status = Moving
AND Set Travel Timer = active
Input: Press Floor 1 Up
Operating Variable Constraints:
Floor 1 Up = not selected
Current Position <> 1
Behavior:
IF (Floor 1 Up Pressed) System MUST Set Floor 1 Up = Selected
IF (Low Floor Selected <> 0) System MUST Set Low Floor Selected = 1
IF (High Floor Selected <> 2 or 3) System MUST Set High Floor Selected = 1
IF (Current Direction = Stationary) THEN
[IF (Current Location < 1) System MUST Set Current Direction = Up
ELSE System MUST Set Current Direction = Down]
AND Set Motion Status = Moving and Travel Timer = Active

Copyright Unpublished: Cognitive Concepts LLC

Page 5

C
C

Operational Profile Testing: A White Paper

Input: Press Floor 1 Down


Operating Variable Constraints:
Floor 1 Down = not selected
Current Position <> 1
Behavior:
IF (Floor 1 Down Pressed) System MUST Set Floor 1 Down = Selected
IF (Low Floor Selected <> 0) System MUST Set Low Floor Selected = 1
IF (High Floor Selected <> 2 or 3) System MUST Set High Floor Selected = 1
IF (Current Direction = Stationary) THEN
[IF (Current Location < 1) System MUST Set Current Direction = Up
ELSE System MUST Set Current Direction = Down]
AND Set Motion Status = Moving and Travel Timer = Active
Input: Press Floor 2 Up
Operating Variable Constraints:
Floor 2 Up = not selected
Current Position <> 2
Behavior:
IF (Floor 2 Up Pressed) System MUST Set Floor 2 Up = Selected
IF (Low Floor Selected <> 0 or 1) System MUST Set Low Floor Selected = 2
IF (High Floor Selected <> 3) System MUST Set High Floor Selected = 2
IF (Current Direction = Stationary) THEN
[IF (Current Location < 2) System MUST Set Current Direction = Up
ELSE System MUST Set Current Direction = Down]
AND Set Motion Status = Moving and Travel Timer = Active
Input: Press Floor 2 Down
Operating Variable Constraints:
Floor 2 Down = not selected
Current Position <> 2
Behavior:
IF (Floor 2 Down Pressed) System MUST Set Floor 2 Down = Selected
IF (Low Floor Selected <> 0 or 1) System MUST Set Low Floor Selected = 2
IF (High Floor Selected <> 3) System MUST Set High Floor Selected = 2
IF (Current Direction = Stationary) THEN
[IF (Current Location < 2) System MUST Set Current Direction = Up
ELSE System MUST Set Current Direction = Down]
AND Set Motion Status = Moving and Travel Timer = Active
Copyright Unpublished: Cognitive Concepts LLC

Page 6

C
C

Operational Profile Testing: A White Paper

Input: Press Floor 3 Down


Operating Variable Constraints:
Floor 3 Down = not selected
Current Position <> 3
Behavior:
IF (Floor 3 Down Pressed) System MUST Set Floor 3 Down = Selected
AND System MUST set High Floor Selected = 3
IF (Low Floor Selected = Null) System MUST Set Low Floor Selected = 3
AND Set Current Direction = Up AND Set Motion Status = Moving
AND Set Travel Timer = active
Input: Travel Timer Expires Floor 0
Operating Variable Constraints:
Travel Timer = Active
Current Direction = Down
Motion Status = Moving
Current Position = 0-1
Behavior:
IF (Travel Timer Expires at Floor 0) System MUST do the following:
Set Current Position = 0
Set Motion Status = not moving
Set Travel Timer = not active
Set Floor Timer = active
Set Cabin Floor 0 = not selected
Set Floor 0 Up = not selected
Set Low Floor Selected based on status of other buttons
Set High Floor Selected based on status of other buttons
Input: Floor Timer Expires Floor 0
Operating Variable Constraints:
Floor Timer = Active
Motion Status = Not Moving
Current Position = 0
Behavior:
IF (Floor Timer Expires at Floor 0) System MUST :
Set Floor Timer = not active AND
IF (High Floor Selected > 0) System MUST do the following:
Set Current Direction = Up
Copyright Unpublished: Cognitive Concepts LLC

Page 7

C
C

Operational Profile Testing: A White Paper


Set Current Location = 0-1
Set Motion Status = Moving
Set Travel Timer = Active
ELSE System MUST do nothing else

Input: Travel Timer Expires Floor 1 Up


Operating Variable Constraints:
Travel Timer = Active
Current Direction = Up
Motion Status = Moving
Current Position = 0-1
Behavior:
IF (Travel Timer Expires at Floor 1 Moving UP) System MUST:
IF ((Floor 1 UP OR Cabin Floor 1 = selected )
OR (Floor 1 Down = Selected AND High Floor Selected = 1)) System MUST
Set Current Position = 1
Set Motion Status = not moving
Set Travel Timer = not active
Set Floor Timer = active
Set Cabin Floor 1 = not selected
Set Floor 1 Up = not selected
Set Low Floor Selected based on status of other buttons
Set High Floor Selected based on status of other buttons
ELSE System MUST continue traveling Up and Set Current Position = 1-2
Input: Travel Timer Expires Floor 1 Down
Operating Variable Constraints:
Travel Timer = Active
Current Direction = Down
Motion Status = Moving
Current Position = 1-2
Behavior:
IF (Travel Timer Expires at Floor 1 Moving Down) System MUST:
IF ((Floor 1 Down OR Cabin Floor 1 = selected )
OR (Floor 1 Up = Selected AND High Floor Selected = 1)) System MUST
Set Current Position = 1
Set Motion Status = not moving
Set Travel Timer = not active
Set Floor Timer = active
Set Cabin Floor 1 = not selected
Copyright Unpublished: Cognitive Concepts LLC

Page 8

C
C

Operational Profile Testing: A White Paper


Set Floor 1 Down = not selected
Set Low Floor Selected based on status of other buttons
Set High Floor Selected based on status of other buttons
ELSE System MUST continue traveling Down and Set Current Position = 0-1

Input: Floor Timer Expires Floor 1 Up


Operating Variable Constraints:
Floor Timer = Active
Motion Status = Not Moving
Current Position = 1
Current Direction = Up
Behavior:
IF (Floor Timer Expires at Floor 1) System MUST :
Set Floor Timer = not active
IF (High Floor Selected > 1) System MUST do the following:
Set Travel Timer = Active
Set Motion Status = Moving
Set Current Location = 1-2
ELSE IF (Low Floor Selected < 1) System MUST do following:
Set Floor 1 Down = not selected
Set Travel Timer = Active
Set Motion Status = Moving
Set Current Direction = Down
Set Current Location = 0-1
ELSE System MUST do nothing else
Input: Floor Timer Expires Floor 1 Down
Operating Variable Constraints:
Floor Timer = Active
Motion Status = Not Moving
Current Position = 1
Current Direction = Down
Behavior:
IF (Floor Timer Expires at Floor 1) System MUST :
Set Floor Timer = not active
IF (Low Floor Selected < 1) System MUST do the following:
Set Travel Timer = Active
Set Motion Status = Moving
Set Current Location = 0-1
ELSE IF (High Floor Selected > 1) System MUST do following:
Copyright Unpublished: Cognitive Concepts LLC

Page 9

C
C

Operational Profile Testing: A White Paper


Set Floor 1 Up = not selected
Set Travel Timer = Active
Set Motion Status = Moving
Set Current Direction = Up
Set Current Location = 1-2
ELSE System MUST do nothing else

Input: Travel Timer Expires Floor 2 Up


Operating Variable Constraints:
Travel Timer = Active
Current Direction = Up
Motion Status = Moving
Current Position = 1-2
Behavior:
IF (Travel Timer Expires at Floor 2 Moving UP) System MUST:
IF ((Floor 2 UP OR Cabin Floor 2 = selected )
OR (Floor 2 Down = Selected AND High Floor Selected = 2)) System MUST
Set Current Position = 2
Set Motion Status = not moving
Set Travel Timer = not active
Set Floor Timer = active
Set Cabin Floor 2 = not selected
Set Floor 2 Up = not selected
Set Low Floor Selected based on status of other buttons
Set High Floor Selected based on status of other buttons
ELSE System MUST continue traveling Up and Set Current Position = 2-3
Input: Travel Timer Expires Floor 2 Down
Operating Variable Constraints:
Travel Timer = Active
Current Direction = Down
Motion Status = Moving
Current Position = 2-3
Behavior:
IF (Travel Timer Expires at Floor 2 Moving Down) System MUST:
IF ((Floor 2 Down OR Cabin Floor 2 = selected )
OR (Floor 2 Up = Selected AND High Floor Selected = 2)) System MUST
Set Current Position = 1
Set Motion Status = not moving
Set Travel Timer = not active
Copyright Unpublished: Cognitive Concepts LLC

Page 10

C
C

Operational Profile Testing: A White Paper


Set Floor Timer = active
Set Cabin Floor 2 = not selected
Set Floor 2 Down = not selected
Set Low Floor Selected based on status of other buttons
Set High Floor Selected based on status of other buttons
ELSE System MUST continue traveling Down and Set Current Position = 1-2

Input: Floor Timer Expires Floor 2 Up


Operating Variable Constraints:
Floor Timer = Active
Motion Status = Not Moving
Current Position = 2
Current Direction = Up
Behavior:
IF (Floor Timer Expires at Floor 2) System MUST :
Set Floor Timer = not active
IF (High Floor Selected > 2) System MUST do the following:
Set Travel Timer = Active
Set Motion Status = Moving
Set Current Location = 2-3
ELSE IF (Low Floor Selected < 2) System MUST do following:
Set Floor 2 Down = not selected
Set Travel Timer = Active
Set Motion Status = Moving
Set Current Direction = Down
Set Current Location = 1-2
ELSE System MUST do nothing else
Input: Floor Timer Expires Floor 2 Down
Operating Variable Constraints:
Floor Timer = Active
Motion Status = Not Moving
Current Position = 2
Current Direction = Down
Behavior:
IF (Floor Timer Expires at Floor 2) System MUST :
Set Floor Timer = not active
IF (Low Floor Selected < 2) System MUST do the following:
Set Travel Timer = Active
Set Motion Status = Moving
Copyright Unpublished: Cognitive Concepts LLC

Page 11

C
C

Operational Profile Testing: A White Paper


Set Current Location = 1-2
ELSE IF (High Floor Selected > 2) System MUST do following:
Set Floor 2 Up = not selected
Set Travel Timer = Active
Set Motion Status = Moving
Set Current Direction = Up
Set Current Location = 2-3
ELSE System MUST do nothing else

Input: Travel Timer Expires Floor 3


Operating Variable Constraints:
Travel Timer = Active
Current Direction = Up
Motion Status = Moving
Current Position = 2-3
Behavior:
IF (Travel Timer Expires at Floor 3) System MUST do the following:
Set Current Position = 3
Set Motion Status = not moving
Set Travel Timer = not active
Set Floor Timer = active
Set Cabin Floor 3 = not selected
Set Floor 3 Down = not selected
Set Low Floor Selected based on status of other buttons
Set High Floor Selected based on status of other buttons
Input: Floor Timer Expires Floor 3
Operating Variable Constraints:
Floor Timer = Active
Motion Status = Not Moving
Current Position = 3
Behavior:
IF (Floor Timer Expires at Floor 3) System MUST :
Set Floor Timer = not active AND
IF (Low Floor Selected < 3) System MUST do the following:
Set Current Direction = Down
Set Current Location = 2-3
Set Motion Status = Moving
Set Travel Timer = Active
ELSE System MUST do nothing else
Copyright Unpublished: Cognitive Concepts LLC

Page 12

C
C

Operational Profile Testing: A White Paper

This concludes the specification of the elevator simulator. The resulting specification is a state machine.
It specifies all possible state combinations, all possible transitions between states, and the required
response to all inputs (transitions). It also represents the structure of the operational profile. To
complete the operational profile specification for reliability estimation purposes, relative likelihood
values must be assigned to each of the inputs in the specification. This is addressed later in the section.
The operational profile testing process is illustrated below. The two main elements of the process are
test generation and test execution. In the diagram tests are generated from a behavior model. These
tests are converted to executable scripts which exercise the system under test in its target environment.

Behavior Model
The next step in the reliability testing process is to convert the software specification into the form of a
behavior model that can be operated on to produce test cases. This is best accomplished with a
software tool. The practitioner may select a commercial tool or develop his own. In either case, the
objective is to represent the state machine specification as a model to automatically generate test cases.
Continuing with the Elevator example, a graphical model has been constructed using a commercial tool 3.
The first figure on the next page shows the main operating modes of the elevator simulator. The
elevator may either be Stationary, Going Up, or Going Down. The second figure shows state transitions
while the elevator is in Stationary mode. Using the modeling tool, the entire specification defined
previously is precisely replicated so that the Markov Chain Usage Model (MCUM) contains all of the
conditions and stimuli that the elevator control system may be subjected to during operation.
3

The tool illustrated here, MaTeLo, is for example purposes only; this report does not promote a specific tool

Copyright Unpublished: Cognitive Concepts LLC

Page 13

C
C

Operational Profile Testing: A White Paper

The figure below shows states and transitions while the elevator is Stationary.

Copyright Unpublished: Cognitive Concepts LLC

Page 14

C
C

Operational Profile Testing: A White Paper

Test Generation
The next step in the operational profile testing process is to generate test cases from the MCUM. An
illustration of this step using the MaTeLo Testor tool is provided below. A number of tests are selected.

The results of random test generation are illustrated below. Notice the coverage numbers.

Copyright Unpublished: Cognitive Concepts LLC

Page 15

C
C

Operational Profile Testing: A White Paper

A key part of software reliability testing is considering how large a test sample is sufficient to obtain
reliability estimates that have a high (or at least reasonable) degree of confidence. In the example
above with 10 test cases the elevator project could achieve roughly 60% state and 40% transition
coverage. This level is too low to ensure reliability or to assess reliability with confidence. A general rule
of thumb is to generate and execute enough test cases to achieve 90% state and 90% arc coverage.
Fortunately, with a tool that automatically generates random tests based on the operational profile, we
can experiment with varying numbers of test cases to determine the level of structural model coverage
the sample would achieve. A sample of 100 tests was generated from the elevator model and the
statistical results below were obtained.

Observe that nearly 97% of states and more than 90% of transitions are covered by the 100 tests. If all
of these tests are executed, then we could be confident that the failure rate obtained from executing
those tests would be fairly representative of the population. A larger sample would increase confidence.
The simplified elevator simulator is a relatively basic application. In more complex applications the state
space in a behavioral model will be much larger than this example. Some applications consist of
hundreds of thousands or millions of states. For those systems, achieving 90% structural cover may
require thousands or tens of thousands of test cases. Automation is a necessity for complex systems.
Copyright Unpublished: Cognitive Concepts LLC

Page 16

Operational Profile Testing: A White Paper

Test Adapter Development


Test cases generated from a modeling tool are sequences of abstract actions. The test cases are not
executable. The abstract actions for the Elevator example are list below. They include Input and
Response actions. Many tools and methods exist for creating executable test scripts for this purpose.
Floor Selection In Cabin
Press Cabin Floor 0
Press Cabin Floor 1
Press Cabin Floor 2
Press Cabin Floor 3
Floor Controls on Wall
Press Floor 0 Up
Press Floor 1 Up
Press Floor 1 Down
Press Floor 2 Up
Press Floor 2 Down
Press Floor 3 Down
Application Buttons
Press Start Button
Press Stop Button
Timers
Travel Timer Expires
Floor Timer Expires

Floor Position
Arrive at Floor 0
Arrive at Floor 1 Up
Arrive at Floor 1 Down
Arrive at Floor 2 Up
Arrive at Floor 2 Down
Arrive at Floor 3
Stop at Floor 0
Stop at Floor 1
Stop at Floor 2
Stop at Floor 3
Application Responses
Application Started
Application Stopped

A tool for test adapter creation is not illustrated here; refer to the literature for test execution tools.
A recommended methodology for constructing test scripts associated with operational profile model
actions is described in this section. The approach is simply this: all discrete events in an auto-generated
test sequence should have an associated adapter that is a self-contained executable test function. This
concept is very basic, but its also powerful and enables a project to achieve software reliability testing.
As stated previously, finite state machines for complex systems can be very large. Test cases generated
from those models can be long and highly variable. The sequences of events for almost every test will
be different. As a result, it is not practical to develop and deploy test functions that have dependencies
on other test functions. This means that the order and sequences of events impact the logic in a given
test function; in other words, test actions would need to know the history of previous events.
Maintenance of test cases could become unwieldy. We strongly advise against such a practice.
Test function independence means that a test function can execute correctly every time based upon the
current state of usage of the system when the event occurs. When a test function is called, it first
searches the global variable space of the SUT and determines current usage, then invokes the action. It
also verifies that the system responds correctly to the event given the current (known) usage state.
To illustrate the idea of self-contained test functions well refer to the Elevator example again. Consider
the input Press Cabin Floor 1. This may occur anywhere in a randomly generated test case. When
called, the associated test function should first check the current values of certain operating variables,
Copyright Unpublished: Cognitive Concepts LLC

Page 17

C
C

Operational Profile Testing: A White Paper

specifically Current Location, Current Direction, High Floor Selected and Low Floor Selected. Then it
should trigger the action Press Cabin Floor 1 and implement the logic specified for this action, below.
Behavior:
IF (Cabin Floor 1 Pressed) System MUST Set Cabin Floor 1 = Selected
IF (Low Floor Selected <> 0) System MUST Set Low Floor Selected = 1
IF (High Floor Selected <> 2 or 3) System MUST Set High Floor Selected = 1
IF (Current Direction = Stationary) THEN
[IF (Current Location < 1) System MUST Set Current Direction = Up
ELSE System MUST Set Current Direction = Down]
AND Set Motion Status = Moving and Travel Timer = Active
No matter what Elevator actions have been previously executed in a test sequence, whenever the Press
Cabin Floor 1 Event occurs the above logic is executed and the system response is confirmed.
Lets take one more action, Travel Timer Expires - Floor 2 Up. When this test function is called it must
first query the global variable space for the values of Floor 2 UP, Cabin Floor 2, Floor 2 Down and High
Floor Selected variables. Once this information is obtained it triggers the abstract event (a timer
expiring is tracked internally; there is no concrete trigger from an external interface to the SUT). Then
the test function implements the specified behavior below.
Behavior:
IF (Travel Timer Expires at Floor 2 Moving UP) System MUST:
IF ((Floor 2 UP OR Cabin Floor 2 = selected )
OR (Floor 2 Down = Selected AND High Floor Selected = 2)) System MUST
Set Current Position = 2
Set Motion Status = not moving
Set Travel Timer = not active
Set Floor Timer = active
Set Cabin Floor 2 = not selected
Set Floor 2 Up = not selected
Set Low Floor Selected based on status of other buttons
Set High Floor Selected based on status of other buttons
ELSE System MUST continue traveling Up and Set Current Position = 2-3
With this self-contained concept each of the test functions implement, precisely, the specification of the
action associated with that function. All of the functions perform three generic tasks: 1) obtain current
global state, 2) trigger test action, and 3) verify correct response based on behavior specification.
The test functions are implemented in this manner regardless of the test tool used, the test language, or
the communication channels implemented to interface with the system under test. The test adapter for
each function implements the specified behavior requirements for the action. Each test function is
executable against the SUT.

Copyright Unpublished: Cognitive Concepts LLC

Page 18

C
C

Operational Profile Testing: A White Paper

Test Case Translation


Before any auto-generated test case can be executed, it must be converted to an executable test script.
By employing appropriate tools, this can be achieved fairly easily. A general method that is supported
by some test automation tools involves assigning a unique test function to each transition in the MCUM
associated with an input to the SUT. The test function development environment is natively supported
by the automatic test generation tool. In this way, an auto-generated test is a sequence of test function
calls that is directly executable in the test function execution environment. Auto-generated tests simply
need to be saved in the format that is compatible with the test execution environment. The literature
contains case studies of projects that have successfully utilized commercial test tools for this purpose.
By automatically translating the test cases into executable scripts a large test sample can be created
very quickly and can be executed for the purpose of determining the reliability of the SUT.
Test Execution
The final step in the operating profile testing process is to execute the auto-generated, executable test
sequences. For a given sample, every test in the sample should be executed in the selected test
execution environment. Again, there are many tools to support test execution and test case
management. Refer to the literature for available tools and their applicability.
To support effective and efficient operational profile testing, the test execution environment should
have the capability to report pass/fail status on every event in a test sequence, and every test in a
sample. Offline analysis of test case results should be avoided to the extent possible so that the
maximum information on test result success can be obtained with minimum effort.
The outcome of executing a sample of tests generated from the test model is quantitative information
on test case successes vs. test case failures. This data can be evaluated and manipulated in a number of
ways in order to estimate software reliability.
Assigning Probability Values to Transitions
A Markov Chain Usage Model is a finite state machine with likelihood values assigned to each arc. An
accurate operational profile contains values that closely represent the expected frequency of each of
the events that will occur in actual system operation. Creating these frequency values, or probabilities,
is the final step in operational profile definition.

Copyright Unpublished: Cognitive Concepts LLC

Page 19