Report

on the

Certificate
Z10 15 06 67052 016
Software Tools for Safety Related Development

Simulink Test
Manufacturer
The MathWorks, Inc.
3 Apple Hill Drive
Natick, MA, 01760-2098
USA
Report No. MN86842C
Revision 1.0 dated 2015-06-15
Testing Body
TV SD Rail GmbH
Embedded Systems
Certification Body
TV SD Product Service GmbH
Ridlerstrae 65
80339 Munich

Distribution, copying or any other use of information in this report in part is strictly prohibited.

Revision Log
Rev.
1.0

Date
2015-06-15

Name
S. Waldhausen, M. Braun

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Changes/History
Initial Report for Release R2015b

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 2 of 13

Content

Page

PURPOSE AND SCOPE ...................................................................................................... 4

PRODUCT OVERVIEW ....................................................................................................... 4

2.1

General Description....................................................................................................... 4

2.2

Scope ............................................................................................................................ 5

IDENTIFICATION ................................................................................................................. 6

CERTIFICATION .................................................................................................................. 6

4.1

Standards ...................................................................................................................... 6

4.2

Basis of certification ...................................................................................................... 6

RESULTS ............................................................................................................................ 7
5.1

Software development and quality engineering processes ............................................ 7

5.2

Customer bug reporting processes ................................................................................ 7

5.3
Requirements on software tools in IEC 61508, ISO 26262, and EN 50128.................... 8
5.3.1
General .................................................................................................................. 8
5.3.2
Simulink Test .................................................................................................... 9
5.4

Tool classification and validation according to IEC 61508 ........................................... 10

5.5

EN 50128 .................................................................................................................... 11

5.6
Tool classification and qualification according to ISO 26262 ....................................... 11
5.6.1
Estimation of TD and resulting TCL: ..................................................................... 11
5.6.2
Evaluation of the tool development process ......................................................... 12
5.6.3
Validation of the software tool............................................................................... 12
5.6.4
Summary .............................................................................................................. 12
5.7

IEC 62304 ................................................................................................................... 13

GENERAL CONDITIONS AND RESTRICTIONS ............................................................... 13

SUMMARY AND CERTIFICATE NUMBER ....................................................................... 13

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 3 of 13

1 Purpose and scope

TV SD Rail GmbH evaluated the Simulink Test product of The MathWorks, Inc. The sections
of the MathWorksTM development organization responsible for the Simulink Test product have
been audited to assess their development and quality assurance procedures.
Recurring evaluations focus on processes used by the Simulink Test teams to implement enhancements and modifications, as well as quality engineering, and customer bug reporting processes.
The aim of the assessment was to determine the suitability for use in development processes which
need to comply with IEC 61508, ISO 26262 or EN 50128. The assessment also covered tool classification and tool qualification measures according to ISO 26262.
The basic assessment is documented in the Technical Report MN86843T, recent modifications will
be reported in Modification Reports according to the table below.
Title

Document Name

Date

Revision

Technical Report on Functional Safety

MN86843T -V1.0.pdf

12.06.2015 1.0

2 Product overview
Simulink Test is a verification tool for authoring, managing, and executing systematic, simulationbased tests of the Simulink models.

2.1

General Description

Simulink Test includes a test sequence block to construct test sequences and assessments, and
a test manager to manage and execute tests. It enables functional, baseline, equivalence, and backto-back testing, including software-in-the-loop (SIL) and processor-in-the-loop (PIL). The tool also
allows generating reports, archiving and reviewing test results, rerunning failed tests, and debugging
the component or system under test.

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 4 of 13

2.2

Scope

The testing for the certification of Simulink Test focused on the following use-cases, as described
in the IEC Certification Kit files (see section 3):

[SLTEST_UC1] Development and execution of tests for Simulink models

[SLTEST_UC2] Development and execution of tests for back-to-back testing between model
and code

[SLTEST_UC3] Assessment of test results

[SLTEST_UC4] Generation of test reports

[SLTEST_UC5] Identification of traceability between requirements and tests cases

The assessment covered the following capabilities of the Simulink Test tool, which support the
accomplishment of the above listed use cases:

Development of test harness for subsystem or model testing

Specifying sequence of tests using Test Sequence block
Specifying pass-fail criteria, including tolerances, limits, and temporal conditions
Implementation of baseline, equivalence, and back-to-back testing
Development setup and cleanup scripts for customizing test execution
Authoring, executing, and organizing test cases and their results using Test manager
Automatic report generation to document test outcomes

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 5 of 13

3 Identification
Release

Date

Simulink
Test

Reference Workflow Documentation:

IEC Certification Kit

Simulink Test Reference Workflow, V3.6

Simulink Test ISO 26262 Tool Qualification Package, V3.6

R2015b

Sept. 2015

1.1

4 Certification
4.1

Standards
Standard

Description

IEC 61508-1:2010

Functional Safety of electrical/electronic/programmable electronic

safety-related systems Part 3: General requirements

IEC 61508-3:2010

Functional Safety of electrical/electronic/programmable electronic

safety-related systems Part 3: Software requirements

ISO 26262-8:2011

Road vehicles Functional safety Part 8: Supporting processes

Confidence in the use of software tools

EN 50128:2011

Railway applications Communications, signalling and processing

systems Software for railway control and protection systems

4.2

Basis of certification

Software development, quality engineering, and customer bug reporting processes

Requirements on software tools in IEC 61508, ISO 26262, or EN 50128

Tool classification and validation according to IEC 61508

Tool classification and qualification according to ISO 26262

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 6 of 13

5 Results
5.1

Software development and quality engineering processes

The software development and quality engineering processes applied for Simulink Test have
been audited, no objections were found.
To ensure adherence to the software development and quality engineering processes, as well as to
keep track of quality improvements, the processes to implement enhancements and modifications
are audited once a year by TV SD.
Product versions that are released in between two consecutive audits are subject to a defined approval procedure by TV SD. The procedure includes the following elements:

5.2

The MathWorks, Inc. documents new customer visible features for each release in the corresponding release notes.

The MathWorks, Inc. documents enhancements and new features of each Simulink Test
version in an internal delta report.

Test procedures for enhancements and new features are referenced in the delta report to
document MathWorks internal validation activities for newly developed features.

Customer bug reporting processes

MathWorks reports known critical bugs brought to its attention on its bug report system at
http://www.mathworks.com/support/bugreports/. The bug reports are an integral part of the documentation for each release.
The bug report system provides an interface for customers to view and submit bug reports. Customers can track the status of open bugs. Customers can choose to receive notifications for new or updated bug reports. The bug reports on this web site include internally as well as externally nominated bugs. If applicable, bug reports include provisions for known workarounds or file replacements.
Customers can use the bug report mechanism to nominate bugs. These nominations are processed
and evaluated by The MathWorks, Inc. development organization.

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 7 of 13

5.3

Requirements on software tools in IEC 61508, ISO 26262, and EN 50128

5.3.1 General
ISO 26262, IEC 61508 and EN 50128 in their current versions contain explicit requirements on software tools.
They strongly recommend the application of development tools and provide provisions for using
model-based design for software development. At the same time, they demand to perform an analysis of the tools used, and an analysis on how they are embedded in the development process:
analysis of tool usage (IEC 61508)
analysis of tool use cases (ISO 26262)
analysis on the effect of possible malfunctions of the applied tool(s).
Depending on the outcome of the above analysis, the standards referred to above demand
a) fault mitigation measures (process)
b) the qualification, respectively validation of tools.
These activities should complement each other, and the combination of both shall reduce the number of faults impacting the final product to a minimum.

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 8 of 13

5.3.2 Simulink Test

The verification tool allows the automation of core verification and validation activities for Simulink
models and generated code. The following use cases reflect activities that are required in a software
development process according to the Functional Safety Standards ISO 26262, IEC 61508 and EN
50128:

[SLTEST_UC1] Development and execution of tests for Simulink models

[SLTEST_UC2] Development and execution of tests for back-to-back testing between model
and code

[SLTEST_UC3] Assessment of test results

[SLTEST_UC4] Generation of test reports

[SLTEST_UC5] Identification of traceability between requirements and tests cases

The use cases involve the capabilities of Simulink Test listed in section 2.2:

Development and execution of tests for Simulink models and for back-to-back testing between
model and code:
- Create and execute test harnesses for subsystem or model testing
- Specifying sequence of tests using Test Sequence block
- Specifying pass-fail criteria, including tolerances, limits, and temporal conditions
- Implementation of baseline, equivalence, and back-to-back testing
- Development setup and cleanup scripts for customizing test execution
Assessment and generation of test reports containing simulation and test results:
- Authoring, executing, and organizing test cases and their results using Test manager
- Automatic report generation to document test outcomes
- Identification of traceability between requirements and tests case

The aim of the testing was to certify the involved tool capabilities for the use in development processes which need to comply with IEC 61508, ISO 26262 or EN 50128.

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 9 of 13

5.4

Tool classification and validation according to IEC 61508

Simulink Test is a class T2 off-line support tool.

The following list provides considerations on how tool users are being supported w.r.t. the requirements of IEC 61508-3 clause 7.4.4:

Simulink Test can be integrated with other Model-Based Design and verification tools
from The MathWorks, Inc. (cf. IEC 61508-3, 7.4.4.2, Note 3). A representative combination of
tools is tested at the manufacturers site. (cf. IEC 61508-3, 7.4.4.9, 7.4.4.18 a).

The tool documentation for Simulink Test (cf. IEC 6158-3, 7.4.4.4) is provided with the
product.

Each release of the tool is identifiable (cf. IEC 61508-3, 7.4.4.15 a).

MathWorks reports critical known bugs brought to its attention on its bug report system at
http://www.mathworks.com/support/bugreports/ (cf. IEC 61508-3, 7.4.4.6, Note 1).

The Release Notes provide the version history for Simulink Test. Tool users can assess
available bug reports for different tool versions via the bug reports system (cf. IEC 61508-3,
7.4.4.6, Note 1)

The MathWorks, Inc., as well as 3rd party vendors, offer training courses for MathWorks tools
(cf. IEC 61508-3, 7.4.4.2, Note 6).

The MathWorks, Inc. developed and applied validation suites to validate the model compliance checking and model coverage analysis capabilities. The application of these validation
suites helps to uncover potential bugs in Simulink Test.

Test procedures for enhancements/new features are referenced in the delta report to document MathWorks internal validation activities for newly developed features. The MathWorks,
Inc. validated Simulink Test and provided documentation of this validation to TV SD for
review and approval (cf. IEC 61508-3, 7.4.4.6, 7.4.4.7).

Summary:
All Simulink Test versions listed in section 3 are certified as T2 off-line support tools and are suitable for safety-related use in application development up to SIL 3 according to IEC 61508:2010. The
tools meet the requirements of IEC 61508-3 7.4.4 to the extent applicable to a tool manufacturer.
The certification covers the following capabilities:
Develop test harness and test procedure
Generate test reports containing simulation and test results, including requirement traceability
The tool classification and the assessment of the tool validation activities were carried out by TV
SD.
Tool certification can be claimed by referencing this certification report and the corresponding certificate.

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 10 of 13

5.5

EN 50128

EN 50128:2011 is an application standard derived from IEC 61508. The requirements for software
tools are explicitly derived from the requirements on software tools according to IEC 61508-3:2010.
Due to the equivalences between the two standards no separate testing has been performed with
respect to EN 50128.
Simulink Test are suitable to be used in the development of safety-related software according to
EN 50128:2011 up to SIL 3/4. Tool certification for the versions listed in section 3 can be claimed by
referencing this certification report and the corresponding certificate.

5.6

Tool classification and qualification according to ISO 26262

The tool classification according to ISO 26262 depends on the particular use-cases used during the
development of safety-related application software components.
For Simulink Test, the following use-cases were considered in the tool classification process:

[SLTEST_UC1] Development and execution of tests for Simulink models

[SLTEST_UC2] Development and execution of tests for back-to-back testing between model
and code

[SLTEST_UC3] Assessment of test results

[SLTEST_UC4] Generation of test reports

[SLTEST_UC5] Identification of traceability between requirements and tests cases

Based on these use cases, the tool impact of Simulink Test is TI2.

5.6.1 Estimation of TD and resulting TCL:

Develop test harness and test procedure
[SLTEST_UC1], [SLTEST_UC2]:
Provided that the error prevention or detection measures listed in the reference workflow for
Simulink Test are carried out, the capability of Simulink Test to develop test harness and test
procedure has been classified as TCL1.
Generate test reports containing simulation and test results, including requirement traceability
[SLTEST_UC3], [SLTEST_UC4], [SLTEST_UC5]:
Assuming that there are no systematic measures in the development process to verify the generated
test reports, the tool error detection for the capability of Simulink Test to generate test reports
containing simulation and test results, including requirement traceability is TD3. The resulting tool
confidence level is TCL3.
A combination of the following tool qualification methods was carried out for the capability of
Simulink Test to generate test reports containing simulation and test results, including requirement traceability:
Evaluation of the tool development process
Validation of the software tool
TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 11 of 13

5.6.2 Evaluation of the tool development process

TV SD conducts yearly surveillance audits of the software development and quality engineering processes for Simulink Test.

The MathWorks, Inc. documents new customer visible features for each release in the corresponding release notes. The release notes were submitted to TV SD.

The MathWorks, Inc. documents enhancements and new features for each release to be
qualified in a comprehensive delta report. The delta reports were submitted to TV SD.

5.6.3 Validation of the software tool

The MathWorks, Inc. developed and applied a validation suite for the capability of Simulink
Test to generate test reports containing simulation and test results, including requirement
traceability that can be used to validate these features. The application of this validation suite
helps to uncover potential bugs in Simulink Test. A successful validation is considered as
a means of end-to-end validation of the capability of Simulink Test to generate test reports containing simulation and test results, including requirement traceability. The validation
reports were submitted to TV SD.

Test procedures for enhancements/new features of Simulink Test are referenced in the
delta report to document The MathWorks, Inc. internal validation activities for newly developed features.

5.6.4 Summary
All Simulink Test versions listed in section 3 are qualified for all ASILs according to ISO 26262.
The qualification comprises the following capabilities:
Develop test harness and test procedure
Generate test reports containing simulation and test results, including requirement traceability
The capability of Simulink Test to generate test reports containing simulation and test results,
including requirement traceability has been classified as TCL3 and qualified accordingly.
Provided that the error prevention or detection measures listed in the reference workflow for
Simulink Test are carried out, the capability of Simulink Test to develop test harness and test
procedure has been classified as TCL1. The tool qualification measures have been carried out on a
voluntary basis to provide additional confidence.
The review of the tool classifications and the assessment of the results of the measures applied to
qualify the software tool were carried out by TV SD.
Tool qualification for Simulink Test can be claimed by referencing this certification report and the
corresponding certificate.

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 12 of 13

5.7

IEC 62304

IEC 62304:2006 provides a framework of life cycle processes for the safe design and maintenance
of medical device software.
IEC 62304 does not place specific requirements on software tools, or on the qualification of tools,
but IEC 62304 advises that IEC 61508 can be looked to as a source of methods, tools and techniques that can be used to implement the requirements in IEC 62304 (IEC 62304:2006, C.1).

6 General conditions and restrictions

As a prerequisite to claim tool qualification for Simulink Test according to ISO 26262, the
error prevention or detection measures listed in the respective reference workflows shall be
applied.

7 Summary and certificate number

This report specifies the conditions of use and restrictions required for the application of Simulink
Test by The MathWorks, Inc. on the certificate:

Z10 15 06 67052 016

Munich, 2015-06-15

Technical Certifier
Peter Wei

TV SD Rail GmbH
Embedded Systems
Barthstr. 16
80339 Mnchen
Phone: +49 89 5791-4378; Fax: -2933

Report No.: MN86842C

Revision 1.0
S. Waldhausen
2015-06-15
Page 13 of 13