2013 14th International Workshop on Microprocessor Test and Verification

Verification Methodology of Heterogeneous

DSP+ARM Multicore processors for Multi-core
System on Chip
David Brier, Rama Venkatasubramanian, Sowmya Rangarajan, Abhishek Arun,
David Thompson and Neelima Muralidharan
Multicore and DSP Development Group,
Embedded Processors, Texas Instruments, Dallas, TX USA
dbrier@ti.com

Abstract—Processor complexity continues to evolve, with new

architectures more complex and more tightly intertwined with
the systems in which they operate than previous generations.
Magnifying the individual processor complexity is the need to
create heterogeneous processor clusters which contain multiple
heterogeneous processors (ARM and DSP) with multiple levels of
caches. These processor clusters need to be validated for
functionality and memory coherency across all the levels of
caches. Management of the verification process of these
processor cluster has likewise grown in complexity impacting the
creation and management of tests, of particular interest are the C
and assembly code driven tests which are the primary methods
addressed in this paper. Lessons in test creation from the UVM,
software coding and other previous test management methods
are combined to permit automation of testing for generation of
test suites for processor sub-systems. Key elements of these
methodologies are detailed in this paper.
I. INTRODUCTION
Heterogeneous multicore Soc’s that use a combination of
DSP and ARM multicore processors are extensively used in
many application markets including but not limited to
basestations, multimedia gateway applications and high
performance computing [1]. Extensive integration at the
System on chip (SOC) level leads to heterogeneous processor
clusters that lead to extremely complex processor clusters with
multiple levels of cache. The Texas Instruments KeyStone II
Multicore Architecture [2] provides a unified platform for
integrating ARM and C66x DSP processing cores [3] along
with both hardware/firmware based application-specific
acceleration and high performance I/Os. From a design
verification standpoint, the development team is expected to
validate the functionality of the complex multicore processors
in very short development cycles. This demands a methodical
approach to Design Verification of the heterogeneous
processor clusters. Fig.1 illustrates the Keystone II System on
chip architecture from Texas Instruments [2] and this paper
illustrates the verification methodology for the heterogeneous
multiprocessor cluster in the context of such a multicore Soc.
System Verilog has been extended using the Universal
Verification Methodology (UVM), providing a platform
approach to verification of complex IPs which is both reusable
Fig.1 Block Diagram of Keystone II System on chip from Texas Instruments [2]
and capable of expansion and modification thus making it
highly desirable for reusable IP in large SOCs.
In considering a verification methodology for large multi-
core designs, where not only the number of cores in each SOC
may vary but also being heterogeneous (ARM or DSP), the
lessons which UVM provide can be extended into the practices
for creation of tests code for the CPUs which provide system
stimulus for verification. Given the objective of “UVM like”
structuring of the code which will run in simulations on the
cores there are several issues that must be addressed, where
UVM models are rich in features, some containing many
methods in the classes, code which runs on the CPU in a
simulation must perform the task as efficiently as possible,
therefore the choice is made to follow the examples of UVM in
the creation of library elements and the phasing of the test code
so that test cases using multiple code segments across varying
numbers and types of CPUs may be automated.
Large numbers of test cases are written for verification of
the CPU based sub-system for an SOC, this code, in many
cases contains elements which are similar for each of the CPUs
and in many instances may run on different types of
processors. Identifying “library functions” is an important step
in this methodology just as it is in implementing an effective
UVM like flow. In addition to the lessons which can be taken
from UVM, the adoption of some typical software design
practices can also be adopted and leveraged in performing sub-
system verification. Generators create the thing API
functionality driven by the Test Control File for the test case
allow maintainability as do APIs in software development but

1550-4093/13 $31.00 © 2013 IEEE 112

DOI 10.1109/MTV.2013.32
only generate sufficient code to perform configuration
operations which are needed to support the specific setup.
The final aspect of the methodology is the application of
the test code onto the various sub-system configurations which
are specified using System Verilog Configurations (Build
Configuration). In addition to the sub-system configurations
which are specified for SOCs, the configuration is also
modified for efficient verification reducing test case run times
by depopulating CPU slots to increase simulation performance
and performing conditional code builds to reduce the number
of operations which any processor must execute. Therefore a
management system automates the generation of tests for the
sub-system and will take information provided by the test case
creator about the requirements of each test case and
automatically match these to the configurations which are
specified using the System Verilog Configuration. This
automation allows aggregating code from existing test cases
into newly created test cases in order to build compound tests
while enhancing maintainability and improving the utilization
of existing test cases across various platforms. This
methodology adds another dimension in the randomization of
the verification process by being capable of randomly
populating/depopulating processor slots and in a constrained
random method combining test cases into single simulations by
applying different test cases to each CPU.
II. TEST CONSTRUCTION
Mimicking the structure of a UVM test, each test has
multiple phases which the simulation passes through to
complete the setup, testing and reporting. Unlike UVM which
utilizes the software simulator to pass events from one model
to another test code which runs on a processor sub-system must
rely on the CPU passing flags to a mailbox to communicate
with the other CPUs in the simulation model. Flags must also
be passed to the UVM portion of the testbench in order to
synchronize events which may be driven by BFMs and provide
error management and simulation termination commands. The
test case code based phases are:
• Initialization Phase: init()
• Configuration Phase: config()
• Pre-Run Phase:: prerun()
• Test Run Phase: run()
• Post Run Phase: postrun()
• Result Extraction Phase: extract()
• Test Report Phase: report()
Each of the phases represent discrete activities that are
accomplished in that phase with the Pre-Run and Post Run
phases being the general synchronization points in the test.
Initialization for the different processor types is addressed in
init(), and config(), where in many cases run() is capable of
running on a DSP or ARM class processor. The characteristics
of these tests are captured at the time the test is created to be
utilized in the test allocation process.
113
III. LIBRARIES
Libraries for testing are built and managed to assure that
the code contained within exists only once amongst all of the
libraries and is available to all test cases which might have
need of that code. There are three libraries with clearly defined
objectives. Critical in the control of testing quality is the
assurance of uniqueness of a code unit. This is contradicted
when code is dispersed into the individual test case directories
where slight modifications may be made in order to customize
the code to the specific application. The need for these slight
deviations can be eliminated by planning of the code units to
meet a more generic distributed use model. Functional library
components utilized to build test cases leverages the
centralization to enhance repeatability, as well as
maintainability of all tests. Having a single instance of a well
thought out code unit used for different test cases not only
reduces the effort involved in creating test cases but also
permits inheritance of enhancements and corrections which
occur during the test development time.
A. Functions Library
Functions which are performed to manipulate the DUT in
such a manner as to cause an observable action to occur are
loaded within the Functions Library. The code within this
library is divided into individual sub-directories which are
organized by the codes target platform.
• General Purpose – code which is capable of
running on any CPU.
• DSP Specific – code which utilizes features that
are found in the DSP platforms.
• ARM Specific – code which utilizes features that
are found in the ARM platforms.
Further delineation is necessary when considering code
within one of the above categories, code which runs on an
ARM class processor may need to be further subdivided into
the type of ARM, i.e. ARM9 and even whether the processor
contains certain extensions.
Elements contained within the code library are used as
building blocks for the test cases which are described by a Test
Control File which can aggregate code from the library to
control the DUT in such a manner as to create a sequence of
events which must complete properly to produce a measurable
and correct result. Each of the code divisions mentioned above
may contain code which only manipulates the DUT, checks the
state of the DUT thus making the results visible or manipulate
and check when precise timing or interleaving of events are
required for the testing of specific functions or interconnects.
1) Manipulators
Library code which causes a certain state to exist in the target
is classified as a manipulator. The manipulation may be
loading a memory with a data pattern setting an interface into
a certain state, or causing a sequence of state changes on an
interface. Manipulators may also load a particular value into a
register or set of registers which in turn may cause some
activity in the DUT to take place with the objective of
producing a result which may be checked for correct behavior.
 2) Checkers
Checkers can have a wide range of actions such as simply
checking that data exists in a memory location, the data
matches a specific pattern, or a sequence of patterns which can
be used as proof that an event has occurred. These building
block functions are normally used to observe behavior in a
DUT which has been induced by a Manipulator. Checkers
also will produce error data which may be used by a reporting
function to present a detailed error report for the test.
3) Manipulators with Checking
For complex interactions where specific manipulations and
checking are required code is written which will perform these
interactions. Many of these applications contain tight loops
which may perform many manipulations which must be
followed immediately by a checking operation. Code which is
categorized into this bin should most often represent the entire
test.
B. Test Case Library
Test Cases are maintained in a directory structure which
divides the library by sub-module target and then by operation
so that tests which have a similar goal are contained in one
directory. These Test Case files are also assigned a structured
name which encodes the function, thus they are named in a
manner:
<function_code><sequence_number>
The individual tests cases in this library do not contain any
executable code; the Test Control file contains the list of
threads and specifies the code which makes up those threads,
thus defining the test case. The file also contains the
classification of the test case, permitting tests to be called by
classification; some of the classifications are identified as:
1) Smoke
Smoke tests are those which give a basic indication that the
DUT is working properly. They are not exhaustive but are
chosen because they run quickly and provide a check of all of
the portions of the DUT.
2) Functional
Functional tests comprise the bulk of the testing, these are the
test cases which obtain the functional coverage, they are
similar to the Smoke tests, and the primary difference is most
often run times. These run as a cursory examination of the
functionality of the DUT would consume large amounts of
simulation time.
3) Performance
Performance tests are used to determine the capabilities of the
DUT to execute a particular operation, be that an algorithm,
the transfer of data or some other operation which would have
meaning in the evaluation of the DUT while performing
typical applications operations.
4) Benchmark
Benchmark tests represent those common benchmarks such as
Dhrystone, Whetstone, etc... These are tests which are utilized
114
to generate standardized performance metrics and are not
generally run for design coverage.
C. Conversion Library
Verification of the sub-modules within the DUT is
accomplished at the individual IP unit levels; the primary
functional coverage for the DUT is performed at the sub-
module level in a full UVM environment. These tests emulate
the interface activity between the sub-module and the other
sub-modules with which it interacts. Every bus in this
methodology has a protocol monitor attached for checking
proper transaction activity and thus is capable of recording
transactions. Selecting transaction recordings which can be
translated into CPU driven bus transactions, C code can be
generated which when run on a CPU generates the same
sequence of transactions on the bus pertinent to the test target.
These tests cannot generate the precise sequence timing that
the unit level testing does, and it will not guarantee that the bus
sequences will not have another access sequence interleaved,
but the sequence to the target will remain consistent, thus
making it possible to port many sequential event tests to the
DUT.
IV. TEST CASE MANAGEMENT AND SIMULATION BUILD
ENVIRONMENT
Several issues arise with a large number of test cases and a
multicore sub-system, such as tests which require a particular
type of CPU to run, or certain positions in the architecture may
physically have different functions available. The intent of the
test may also be lost through project attrition as team members
are rotated off, thus demanding a robust DV environment.
When different sub-system architectures are required for new
SOCs, it may require testing which was not done before, with
an automated methodology for test case assignments the
environment comprehends the differences and locations of
specific elements in the sub-system and can allocate the
appropriate testing to each CPU based on type, location of the
core in the processor cluster and interaction with other cores in
the cluster.
A. Automated Test Allocation
Each test case directory must contain a Test Control File
which provides information about the test case, such as the
type of CPU that it can run on, whether it can run
simultaneously on multiple CPUs, and positional restrictions
within the hierarchy, the source code file and parameters which
require modification from the default configurations which are
specified in config() state of the simulation. This Test Control
File is created at the time of test case creation and captures the
intent of the writer for use in allocating tests across multiple
Build Configurations. It permits allocation onto multiple
processors or the code to be use in compound tests as are
present in the different builds which are tested.
Each test case may be comprised of code which is common
to multiple tests, or code which is written specifically for the
test case. Preference is given to re-use of test code where
feasible, but the methodology permits unique code creation and
inclusion in test cases.
 The test allocation methodology permits Test Cases to call
out specific combinations which are permitted, limiting them to
one Test Case, one Configuration and one Build to multiple
combinations, all controlled by the test case developer who is
capable of completely describing any limitations in the Test
Control File and the Test Plan.
1) Test Case
As discussed previously, the Test Case is a control file
which stipulates the code which comprises a thread in the test
and which processor slot it will run on and the resources
required supporting the test.
2) Configuration (Setup)
Configuration setup is a collection of files which identify a
unique setup condition for a parameter in the DUT. For
example, L1D Cache size would be one configuration which
would be stipulated. These may be aggregated and crossed
with the Test Case to create multiple Test Case Scenarios.
3) Build Configuration
The Build Configuration will control the presence of
resources within the DUT utilizing a System Verilog
configuration file. System Verilog configurations can be used
to control which processor slot is populated with an RTL
version of the processor or a BFM which stubs out all the
signals. At the SOC level individual modules may be
eliminated from the simulation or module types may be
exchanged. The objective is to reduce the simulation image
size which improves performance of the simulations without
modifying the structure of the design at the netlist level.
This also is used to release various combinations of Builds
for different end product architectures. This method of control
permits the utilization of the crosses between the Test Case,
Configuration and Build. These are controlled by the Test Plan
and the Test Generator.
B. Test Plan
The Test Plan is used to dictate the legal combinations of
the Test Cases and the Configurations (Setup). Each Test Case
may support multiple Configurations, and the method for
stipulation permits specifying ranges of Configurations using
wild cards, simplifying the specification process.
The Test Plan is used to generate a crossing list of the tests
which shall be run; this listing is exhaustive for the
combinations which are legally possible based upon the inputs
to the Test Plan.
Under configuration control and change tracking a history
of the Test Plan development is maintained for verification
tracking purposes. This allows auditing of the testing which
has been performed on any processor sub-system and also
permits running of additional tests on a sub-system which has
previously verified and updating tracking for complete
histories of the verification process.
C. Test Generation
Test generation is a multi-step process; the methodology is
scripted, thus guaranteeing repeatability of the process.
1) Test List
115
Processing the Test Plan data base generates a Test List
which, was generated according to the crossing rules place on
the Test Plan and is a complete listing of all the Test Cases
crossed with the possible Configurations for the DUT. The list
output when processing the Test Plan is a combination of the
test case name and the configurations which that test case is
crossed with.
2) Generation of Tests
Inputs to the Test Generator builds a test that performs the
functions as called out in the Test Control file with the device
configurations on the build configuration based upon three
categories:
• Test Case (Test Control File)
• Configuration
• Build
Each category permits configuration of the test being
generated to target specific attributes of the DUT. The Test
Generator pulls the test code specified in the Test Control File
from the code library into a build directory for each thread
specified. The generator also utilizes the parameters set in the
Configuration files called out for the test and generates the
code needed to set these configuration parameters in the DUT
saving that into the build directory and lastly maps the test
code onto the build configuration(s) specified in the Build
parameters.
The Test Generator performs an audit of the Test Case,
Configuration and Build parameters to assure that the test
specification is capable of running. For each mapping to a
Build, the generator audits the code called out in the threads to
assure that the compute resources are present in that build for
the code to run. These checks include the number of
processors, one for each thread, the type of processor and the
hierarchical location of the processor.
These audits are in place to assure that the crosses made in
combination with the Test Plan processing and the Test Control
File produce tests which will run properly on the Build
configuration which is being targeted.
3) Test List
The Test Generator outputs two items, the database
containing the test code which is a compilation of code copied
from the code libraries based upon the thread definitions
contained in the Test Control File and the generated
configuration code called out in the configuration specification
in the Test Plan. This list will produce working simulations for
verification purposes, as the generator has audited resources
and configurations for matching the test case requirements,
thus eliminating tests which cannot pass on the Build because
of missing resources.
The final Test List is comprised of a listing of the tests
created following the naming convention enabling the
simulation environment to pull the data bases and allocate the
tests which have been called across the server farm. The
naming convention of the tests is:
test_case-config1-config2…config(n)-build_config
 Thus each test has a unique name and can be mapped into
the data base which tracks the simulations status for each test
for a given design build.
V. TESTING PHILOSPHY
Verification of large complex designs represents a huge
load on the simulation environment, to the point where
simulation performance is significantly degraded. Performing
many of the functional tests at the unit level reduces the
number of test cases which must be created and run at the
device level. Even the CPU Cluster must be broken down into
unit tests and sub-system testing at the Cluster level and the
loading on the simulator for the simulation at the Cluster level
still represents a significant simulation load and thus is
likewise verified in parts.
cluster_1
DSP DSP DSP
Soc L3
ARM ARM ARM
The figure above shows the makeup of the CPU Cluster,
Build configurations are used in simulations to depopulate the
specific blocks within the Cluster to enhance simulation
performance. Verification performed at the unit level for the
sub-modules instantiated in the Cluster is leveraged to reduce
the amount of Cluster level verification require. This reduces
the primary verification at the Processor Cluster level coverage
of sub-module interconnect and functional coverage of events
which require interactions between multiple sub-modules in the
Processor Cluster. The objective is to provide through coverage
of the Processor Cluster while minimizing the time it takes to
develop functional coverage. This is done through analysis or
the design, identification of coverage points and carefully
planning the required resources for each test case created.
Lessons learned in the DV environment which drove the
creation of the UVM methodology within System Verilog are
applied to CPU driven code based verification, the structuring
of the test environment as well as the test generation process
applies many of the items which UVM incorporates and
applies them to the code based verification.
Test code is stored in a Library and the test generator pulls
the code for each individual test directly from that code library
to prevent copying of code into test case directories. This
enhances the maintainability of the test code as well as
representing a more formal structure which can be used to
audit the completeness of the tests and the propagation of
revisions to a particular module of code which is used in
multiple test cases. One motivation; it has not been uncommon
in non-library based environments to discover code from one
test case that has been copied to other test cases, an error, or
deficiency found and that change has not been propagated to
116
the other test cases which utilize the same code. A common
code library using base functions as test building blocks is one
of the primary methods of test development in UVM, adapting
the common library approach strengthens the verification
process in an environment driven by code running on CPUs in
the simulation by assuring the propagation of updates to the
code units.
One significant departure to note is that in UVM most
modules are all feature rich, the classes used have multiple
methods which are utilized for manipulation of the class,
wherein the code written to run on the CPUs is written as
simply as possible for simulation performance purposes. Even
with when writing as such primitive level flexibility of the
code unit is taken into consideration with the objective of
making the code as flexible as possible.
A. Tests
Tests are created to target specific features within the
design, as previously mentioned, basic functionality is not the
primary objective of the testing at the higher levels of
integration and coverage of some functionality is performed as
a means to an end.
1) Interconnect Tests
The first test objectives at the higher hierarchical levels is
bus interconnect testing, this is accomplished by targeting
specific memory or register locations throughout the hierarchy.
Coding library modules in a manner that permits retargeting of
payload addresses and/or transfer sizes allows the same
modules and even threads to be utilized to test many
interconnects within the design. Payload addresses may be
mapped into the L1D Cache, an EMIF, or a System Memory
location. The interconnect testing can be accomplished using
the same code for each location and threads can be modified in
the individual test cases to customize them for the
characteristics of the bus being verified.
A single instantiation of a master element may be
instantiated in each of the slots connecting to the Switch Slave
Ports in separate Build configurations, until all ports have been
verified. Each of these represent a new test case, which is an
adaptation of existing tests cases. Replication of test cases,
deriving executable code form a library maximizes reuse of
code and enhances consistency across all test cases using a
specific code module.
2) Interaction Tests
Interaction between masters must be verified at the
Processor Cluster level, this testing utilizes the same library
code that was used for interconnect testing. Many of the
threads which comprised Interconnect test cases can be utilized
for Interaction testing, in many cases the Test Control file for
the test case will have multiple threads assigned to different
CPUs in the Build. Each of the threads may also be the same
sequence of code units, the Test Generator is capable of
generating code which either places the payload for each thread
in different memory spaces, the same memory but different
addresses or in the same addresses thus generating different
bus and target conflict scenarios.
 3) Complex/Functional Tests
Performance and Benchmarking tests are specified in the
same manner as all other tests, these are treated by the Test
Generator in a manner similar to that of all of the preceding
tests.
In addition, some functional tests which were performed at
sub-module unit level may need to be run at the higher levels
of the design driven by code run on a CPU. These tests are
translated into C code taken from the logging monitors which
are connected to the unit level master bus. These monitors log
bus writes and reads which in turn represent writes and reads
that a CPU would perform. Conversion of these types of tests
is optimal for sequential operations, as timing of bus
transactions for a CPU will normally be quite different from
that of a transaction master in a unit level test.
Conversely as bus transactions are recorded by the bus
monitors, it is possible to replay CPU generated transactions in
the unit level verification environment in order to add test cases
which uncover additional design coverage at the sub-module
level.
B. Test Environment Utilization
Capabilities of the test environment enable the specification
of tests to be run with the following controls:
• Test Case
o Specific Test Case
o Regexp Range of Test Cases
o All Test Cases
• Configuration
o Specific Configuration(s)
o Regexp Range of Configurations
o All Applicable Configurations
• Build
o Specific Build
o Regexp Range of Builds
o All Builds
117
The deterministic nature of the Test Generation allows
reliable generation of tests, with this environment a Test Case
may be regressed against all Configurations and Builds it may
map to as well as when a new Build is derived for a new
product, it may be regressed against all Test Cases and
Configurations which it is capable of supporting.
VI. CONCLUSION
As the complexity of multicore designs increase they
require more judicious application of compute resources by
creating test environments which can automate the mix of
architecture and test case to generate complete testing with
lower expenditures of man power. In facilitating reuse of IP it
is incumbent on the design and verification teams to establish
methodologies which capture intent and in as automated
manner as possible apply that intent to designs.
This methodology provides the capture of intent and the
application of that intent when generating the complete test
from the test case files and applying those tests to the
appropriate architectures. Properly applied and maintained the
data bases which are created in the design and verification
phase of the first sub-system can greatly enhance the yield of a
design and verification team, thus reducing development time
and increasing quality.
REFERENCES
[1] The Case for Heterogeneous Multi-Core SoCs –
http://chipdesignmag.com/display.php?articleId=2090
[2] http://processors.wiki.ti.com/index.php/Keystone_II_multicore
[3] Damodaran, R.; et al., "A 1.25GHz 0.8W C66x DSP Core in 40nm
CMOS," VLSI Design (VLSID), 2012 25th International Conference
on , vol., no., pp.286,291, 7-11 Jan. 2012
[4] Bishnupriya Bhattacharya et-al, Advanced Verification Topics: Cadence
Design Systems.