To appear in the proceedings of the Workshop on Complexity-Effective Design held in conjunction with the 27th annual International Symposium on

Computer Architecture (ISCA-2000), Vancouver, British Columbia, Canada

Toward Complexity-Effective Verification:

A Case Study of the Cray SV2 Cache Coherence Protocol


Dennis Abts David J. Lilja Steve Scott
dabts@cray.com lilja@ece.umn.edu sscott@cray.com

Cray Inc.  University of Minnesota
Chippewa Falls, Wisconsin 54729 Electrical and Computer Engineering
Minnesota Supercomputing Institute
Minneapolis, Minnesota 55455

Abstract
Modern large-scale multiprocessors, capable of scaling to
hundreds or thousands of processors, have proven to be very
difficult to design and verify in a timely manner. In partic-
ular, the verification process, i.e., proving that the design
is functionally correct, is often the most time-consuming
aspect of developing the system. This paper discusses
the methodology and early experiences of verifying the
Cray SV2 cache coherence protocol. This paper proposes
a method of dealing with the verification complexity of
a directory-based coherence protocol. We provide the
framework for a methodology that is built on a formal
model of the coherence protocol, a language, and the RTL
implementation. Finally, we show how this approach was
used to verify the SV2 directory-based coherence protocol
at the architectural level and at the corresponding Verilog
implementation level.
results from accessing the shared address space. The shared-
memory programming model can be viewed as an abstrac-
tion layered upon this message passing substrate.
The FSMs at each level of the memory hierarchy are re-
sponsible for implementing the coherence protocol. The co-
herence protocol is a set of rules that, when followed by each
computing node, ensure a consistent view of the memory
system [4, 5, 6, 7]. The coherence mechanism may be imple-
mented in a variety of ways, through either software, hard-
ware, or a combination of the two. A hardware-based coher-
ence mechanism provides better performance at the expense
of losing the flexibility that is afforded by software-based
approaches. Bus-based systems (such as the SGI Challenge
and Sun Enterprise) use a bus snooping approach. However,
even at modest processor counts, contention for a shared sys-
tem bus can degrade the memory system performance signif-
icantly. A better approach for large-scale systems is to use

Figure 1: The local memory hierarchy at each node in a

1 Introduction DSM machine has several finite state machines (FSMs)
Shared-memory multiprocessors provide both scalability that communicate by exchanging micropackets.
and a flexible programming model. These features, however,
P1 P2 Pn
come at the expense of additional hardware complexity in the
coherent memory subsystem. The Distributed Shared Mem-
ory (DSM) architecture [1, 2, 3] provides a logically shared FSM FSM FSM
L1 $ L1 $ L1 $
address space, although the physical memory is distributed
among the computing nodes. This organization creates an L2 $
FSM
L2 $
FSM
L2 $
FSM
extended memory hierarchy that spans from the load-store
unit of a given processor through multiple levels of cache, FSM FSM FSM
and possibly across multiple nodes that communicate over an Mem Dir Mem Dir Mem Dir
interconnection network. The memory hierarchy is kept con-
sistent by several cooperating finite-state machines (FSMs)
Router Router Router
controlling each level of the memory hierarchy (Figure 1).
These FSMs interact via low-level micropackets. This mes-
sage flow is fundamentally responsible for the implicit trans- Scalable Interconnection Network
ferral of data through the extended memory hierarchy that

1
a directory-based coherence protocol in combination with a
scalable interconnection network. The directory is a data
structure that stores the current state (i.e. access permission)
of a memory block, as well as a list of nodes with access to
the block.
1.1 Motivation
Designing an efficient coherence protocol is very chal-
lenging and, as Lenoski and Weber [1] point out, “unfortu-
nately, the verification of a highly parallel coherence pro-
tocol is even more challenging than its specification.” The
complexity of verifying these systems goes far beyond that
of a typical VLSI circuit. The subtle, yet complex interac-
tions between the FSMs that implement the coherence pro-
tocol at each node makes verification very difficult, time-
consuming and error prone.
To mitigate this complexity, hierarchical design methods
are used, with verification occurring at various levels of ab-
straction (Figure 2). This allows architectural verification
to be performed early in the design process to verify that
a proposed architectural feature of the system is sound. At
this abstract level, a variety of methods, ranging from formal
methods to behavioral simulation, can be employed. The
confidence that the architectural verification provides is lim-
ited by how closely the abstracted model reflects the actual
implementation and the assumptions made during the veri-
fication process. The implementation verification uses tradi-
tional logic simulation techniques to verify the register trans-
fer language (RTL) specification of the design.
Unfortunately, the verification process does not have the
nice step-wise refinement property the design process has.
That is, there is no cohesive method of binding together the
architectural and implementation verification efforts. In this
paper, we propose a methodology that is built on a formal
model of a coherence protocol and show how this approach
was used to verify the SV2 directory-based coherence proto-
Figure 2: Each stage of the design process can be viewed
as a step-wise refinement of the prior stage, and verifica-
tion depicted as a comparison of adjacent stages of the
design process.
Functional  formance evaluation. They implicitly verified aspects of
Specification Architectural
Verification
Design  consuming, expensive, and cannot be performed until late in
Specification Implementation
Verification
Implementation
 very difficult since only limited visibility and primitive tools
(RTL)
Equivalence
Checking
Structural
(gates)
Physical
Design
2
col at the architectural level and at the corresponding Verilog
implementation level.
2 Background and Related Work
2.1 Overview of the Cray SV2 Memory System
The Cray SV2 is a scalable distributed shared memory
(DSM) vector machine targeted at high-end scientific and
technical applications. A single SV2 node consists of four
unique chip types: a scalar/vector processor (P), an external
L2 cache (E), a memory directory (M), and an input/output
(I) chip. Multiple nodes are interconnected using router (R)
chips. The P chip contains an instruction cache and the L1
data cache. The instruction cache coherence is maintained
by software with explicit flushing when the icache contents
are potentially stale, such as after loading a program. The
SV2 cache coherence protocol spans the P, E, and M chips,
where the L1 data cache is in the P chip, the L2 cache is in
the E chip, and the memory directory is in the M chip.
The basic processing element in the SV2 is a multistream
processor (MSP). The memory directory uses a simple bit-
vector directory pointer to track the sharing set of memory
block [4]. The memory directory tracks L2 caches only, not
L1 caches. Since the L2 cache is inclusive of the L1 cache,
we need only track L2 caches at the directory, then each L2
cache maintains a 4-bit inclusion vector with each cache line
to indicate which L1 cache (P chip) is sharing the line.
2.2 Related prior work
Collier [8, 9] developed a suite of high-level tests called
ARCHTEST to reason about the correctness of a multipro-
cessor memory system. Although the test programs them-
selves are relatively simple, the post-processing of the results
is complex. The ARCHTEST suite can be compiled and ex-
ecuted on an existing multiprocessor. Unfortunately, it pro-
vides little assistance in the development of new hardware.
Furthermore, ARCHTEST is unable to detect deadlock, live-
lock, and loss of data coherence.
Dubois et al constructed a prototype [10, 11] of the Stan-
ford DASH [12] multiprocessor using reconfigurable logic
devices (FPGAs). The goal of the project was to pro-
vide a verification vehicle as well as a platform for per-
the DASH multiprocessor by running the SPLASH paral-
lel benchmarks. Unfortunately, this approach is very time-
the design process. Furthermore, finding the design flaws is
are available for probing into the logic design.
Lamport logical clocks have been extended to study the
correctness of memory consistency models [13, 14]. This
time-stamping technique provides a tool for reasoning about
the proper event orderings in a multiprocessor memory sys-
tem. However, it is unclear how this approach can be used to
detect deadlock, live-lock, and fairness. Moreover, since this
analysis must be integrated into an architectural simulation
tool, it can be used to reason about the correctness of only
a particular program execution. Obviously, it is impossible
to conclude that, if the system is correct for a given program
Property 1 If an address,  , is in the “noncached” state at
the directory and there are no messages,  , in-flight in the
interconnection network, then all caches must have address
 in an invalid state.


  !#"$%&')(*,+.-/-/-0
execution, it must be correct for all executions and all pro- 
grams. 132 547698,:;<=?>A@9B@C%5&D')(E,+.-/-!F 2  G  69899+@,B-
More recently, formal verification methods have been
used to validate the coherence protocol of the SGI Ori- where  is an address, H is a  processor
- cache, and  is a
gin2000 [15, 16, 17] and the Sun S3.mp (Sun Scalable message. The function Home  returns the identity of the
Shared-Memory Multiprocessor) [18, 19]. Eiríksson used 9I- for managing the address  .
memory directory responsible
the Symbolic Model Verifier (SMV) [20] to verify an ab- Likewise, the function Dir returns the state of the mem-
stract model of a three node Origin2000 system (two pro- ory directory (access
 permission and sharing set) for a given
cessors and an I/O unit). Pong, et. al. used the Mur  [21] memory directory .
formal verification system to verify an abstracted three node
Property 2 If an address,  , is present in cache, H , then it
S3.mp system. Each of these efforts needed to model the sys-
must be included in the sharing set by the directory.
tem at a high-level of abstraction (ignoring as much detail as
possible) to make the verification tractable. Despite the un-
realistically small verification model, these methods proved
to be very valuable in extracting very subtle design flaws that
would have been difficult, if not impossible, to detect using
traditional logic simulation. Nonetheless, in both cases ex-
tensive logic simulation was performed to verify the actual
RTL implementation.
3 Formal Correctness
Showing that a cache coherence protocol is correct is non-
trivial, as there are many aspects to “correctness,” and the
protocol state space is very large. Our approach is to for-
mally model the protocol and prove that collection of well-
defined, fundamental properties hold over the state space.
We expect that most coherence protocols would require sim-
ilar properties. The properties make no assumptions about
the detailed implementation of the protocol. Rather, we use
generic predicates and functions1 to describe the state of the
caches, directory, and interconnection network.
3.1 Data Coherence
A memory system is coherent if the value returned by a
load is always the value from the “latest” store to the same
memory [6, 22]. On the surface this notion of data coher-
ence appears vague, so this point bears some elaboration. As
UVWX
J ;KMLNOC<*,+@9B-PFRQS NL 895:QS=<*/!#"$%&')(*,+.-/-

The SharingSet predicate returns true if the memory direc-
tory knows that address,  , is present in cache, H . Another
way to look at this is the set of caches with address,  ,
present is a subset ( T ) of the sharing set at the directory.
While these two properties do not explicitly address the
read-the-latest-write aspect of memory coherence, they do
ensure that the memory directory is properly maintaining the
sharing set, an essential ingredient for memory coherence.
Property 2 says that a cache line may still be tracked by
the memory directory, even if it is no longer present in the
cache. For instance, if a cache line is evicted there will be
some transient time between the eviction notice being sent
and the memory directory removing the cache from the shar-
ing set. As such, the cache could receive a “phantom” in-
validate from the directory for a cache line that is no longer
present.
3.2 Forward Progress
Ensuring forward progress requires every memory re-
quest to eventually receive a matching response. Since all
coherent memory transactions occur using request-response
message pairs, we can exploit this fact by formally stating:
Property 3 Each request must have a satisfying response.

YOC<Z?[-!F]\.^_V`O  Oa9b.-
0Q <C8,OdcIO*9b@/[-
a memory request propagates through the memory hierarchy, H 
hardware components, such as arbiters and buffers, will im-
pose a serial ordering on all the memory operations to the Moreover, the forward progress property (Property 3) en-
same address. This linearization of memory events provides capsulates the notion of deadlock and live-lock avoidance
context for the word “latest” in the definition of memory co- by requiring each request to eventually receive a matching
herence. response. Deadlock is the undesirable condition making it
We indirectly capture the notion of data coherence by impossible to transition out of the current global state and
making some assertions about the state of the memory di- live-lock is a cycle
V`X
ofYstates
OC<*9[- that prevents
VW=O forward
Oa9bD- progress.
rectory and caches. The predicates [ and b H evaluate to
a logical true if is a request and is a response, respec-
1 Predicates are designated by bold typeface and evaluate to a logical true
Q <C8,OdcIOZ9b@[-
or false. Functions return a value and are in sans serif typeface tively. Similarly, the predicate  evaluates to

3
Figure 3: An example coherence protocol specification Figure 4: Work-flow diagram describing the steps in-
e L1 cache.
for an
= f Invalid, Exclusive, Shared, Pending g
volved in constructing the protocol verifier.
h = f Invalid, Exclusive, Shared g Coherence Protocol Specification
i/j
k = Invalid
= f PrRead, PrWrite, Inval, ReadResp, GrantExcl g
(*.tbl)
{
i!l e m l k ndo iCp mZq r (action)
Protocol
Invalid PrRead Pending L2(L1ReadReq)
Compiler
Invalid PrWrite Pending L2(GetExclusive)
Invalid Inval Invalid L2(InvalAck) (tbl2m)
Invalid ReadResp - Error(UnexpectedMsg) {
Invalid GrantExcl - Error(UnexpectedMsg)
Exclusive PrRead Exclusive P(ReadResp) Mur 
Exclusive PrWrite Exclusive P(WriteComplete) Specification
Exclusive Inval Invalid L2(InvalAck) (*.m)
Exclusive ReadResp - Error(UnexpectedMsg) {
Exclusive GrantExcl - Error(UnexpectedMsg)
Shared PrRead Shared P(ReadResp) Mur 
Shared PrWrite Pending P(GetExclusive) Compiler
Shared Inval Invalid L2(InvalAck)
(mu)
Shared ReadResp - Error(UnexpectedMsg)
Shared GrantExcl - Error(UnexpectedMsg) {
Pending PrRead Pending Block(PrRead)
Pending PrWrite Pending Block(PrWrite)
Intermediate
Pending Inval Pending Block(Inval)
Specification
Pending ReadResp Shared P(ReadResp) (*.C)
Pending GrantExcl Exclusive P(WriteComplete) {
Compile
b [ and Link
a logical true if satisfies . For example, the predicate
Q C< 8,OdcIOZ?b@[- (g++)
 would consult the transition
b relation for the
coherence protocol to determine if was an expected re- {
[
sponse to request . Clearly, this property ensures forward Protocol Verifier
progress by ensuring that a request is never starved or indef-
initely postponed.
However, the forward progress property makes no claims 3.4 Unexpected Messages
about fairness. For instance, it says nothing about the dis-
tribution of service times or even that requests are serviced A coherence protocol is specified for ,z@/[each
@|N@~}=- level of the
in an equitable manner — these are more implementation- memory hierarchy z as a set of 4-tuples
[ , where the
specific properties that deal with the performance of the current state is . When input|.,zis@/[received,
- then the current}
memory system and not its correctness. state transitions to new state and performs action .
Error conditions should be explicitly defined. Unexpected
3.3 Exclusivity messages will occur only where the protocol is incorrectly
or incompletely specified by the protocol designer. Figure
The coherence protocol enforces some access permis- 3 gives an example protocol specification for an L1 cache,
sions over the shared-memory to ensure that there are never z3€ [‚
where |.9z@is[the
- state of the cache line, is the input
two or more processors with “exclusive” (write) access to message, is the new state resulting from this transi-
the same memory block. This single-writer property can be } z
tion, and is the action that the controller takes. A state, , is
stated as: zƒ€…„`† z‡ †
considered transient if and quiescent if . In
X
Property 4 Two different caches, H and , should never the example shown in Figure 3, the Pending state is transient
have write access to the same address,  , at the same time. and Invalid is quiescent.
J ; 2 aO sW89La<ut9+@9B-PFv
way x  132 OasW89La<ut_9+@zE-
X 4 Verification Methodology
This property ensures that no two processors H and are able
to have memory block  in their local memory hierarchy in 4.1 Formal Verification Model
the “dirty” state 2 at the same time. The input in the verification process is a formal specifi-
2 some protocols use the “dirty” or “exclusive” state, we assume the pred- cation of the cache coherence protocol, as shown in Figure
icate will return true if the cache line is dirty or exclusive. 3, for each level of the memory hierarchy. The objective is

4
Figure 5: Block diagram of the formal verification model, which is described in the Mur description language. The
model is written to be scalable from 1. . . N pseudonodes.

Proc Proc Proc

L1 cache L1 cache L1 cache

tag state data tag state data tag state data

L2 cache L2 cache L2 cache

tag state data tag state data tag state data

tag state sharers tag state sharers tag state sharers

Local Local Local
Memory Memory Memory

memory memory memory

directory directory directory

Network Network Network

Interface Interface Interface
Pseudo Node Pseudo Node Pseudo Node

Virtual Network

to show that the coherence protocol is architecturally sound
by satisfying the correctness properties outlined in Section
3. Once we have established the correctness of the proto-
col at an abstract level, we then would like to show that its
implementation is also correct.
To verify the coherence protocol at an abstract level, we
used the Mur  formal verification environment [21]. The co-
herence protocol is specified as several human-readable text
files. These files are then read by a protocol compiler to au-
tomatically generate the finite-state machine descriptions in
the Mur  description language (Figure 4). The Mur  com-
piler is then used to create the intermediate C++ description
which is compiled and linked into the protocol verifier.
Constructing the formal verification model is a very time-
consuming and difficult task. A detailed understanding of the
design is necessary to be able to abstract away any unneces-
sary details. What details are “unnecessary?” Three types
of abstraction are necessary to make the formal verification
tractable: system scaling, data abstraction, and temporal ab-
straction. System scaling pretends that the system is much
smaller than it really is, while still providing enough detail
to capture the essence of the original design. Data abstrac-
tion seeks to re-encode the meaning of a variable to reduce
the number of reachable states. Temporal abstraction is used
to eliminate the notion of “timing” within the design; that is,
we ignore propagation delay of the actual hardware.
The formal verification model (Figure 5) is scalable from
1 . . . N “pseudonodes”, however, only two pseudonodes are
required for the formal verification. Each pseudonode has
only a single processor and a single L2 cache in each MSP
whereas the real machine has four P chips and four E chips
per MSP. In effect, we are taking a “slice” of an MSP. Each
cache line is only a single-bit, with a single-bit tag. Inter-
mediate hardware structures, such as output request buffers
and transient buffers, are modeled where necessary for accu-
racy. The pseudonodes are connected by a virtual network
with three virtual channels (vc0, vc1 and vc2). The virtual
channel buffers are only one entry deep.
The coherence protocol is blocking (no “retry” nacks are
used) so packets that cannot be processed can put back pres-
sure on the virtual network. Deadlock is avoided by guar-
anteeing that packet dependency chains are acyclic. The
longest dependency chain occurs when (1) the L2 cache
makes a request to the directory, (2) the directory forwards
the request to the owner, and (3) the owner cache sends a
cache-line response back to the directory. Packets between a
sender and receiver pair on a given virtual channel must be
delivered in-order by the virtual network.

5
Figure 6: Producing witness strings from a depth-first search of the state space which are later played back on the actual
hardware.

Protocol Mur  Logic

Verifier ˆ Witness Strings ˆ Simulator
ˆ Accept or
Reject?
(sv2 cc) (*.ws) (Gensim)

(a) using witness strings to refine the abstract formal verification allowing verification of the implementation.

witness
string

S0

(b) a witness string produced by a depth-first search of the state space.

4.2 Verification of the Implementation Model
The formal verification model described above can be
used to automatically verify the coherence protocol at the
architectural level. However, to verify the implementation
itself we must build a connection from this abstract, architec-
tural model to the Verilog implementation model. In verify-
ing the actual implementation of a cache coherence protocol,
we are no longer dealing with an abstract representation of
the design. Instead, the complexity and sheer enormity of a
system comprised of several deep submicron ASICs places
an overwhelming burden on the verification effort.
There are several ways to attack this verification problem.
One possible approach is to construct a testbench around the
from the abstracted verification to be used to verify the im-
plementation. The Mur  formal verification system used to
verify the architectural soundness of the coherence protocol
will enumerate all reachable states and will search the state
space to establish the correctness of the properties specified,
or it will show a counter-example. This search process can be
conducted in three possible ways: 1) breadth-first, 2) depth-
first, or 3) random. We discovered that if we made trivial
modifications to the Mur  source code, we could observe the
nondeterministic firing of rules as the finite-state machines
at each pseudonode interact. We developed a method for
recording these events during a depth-first search of the state
space in Mur  , making it possible to reproduce these inter-
actions on the actual hardware (Figure 6). We refer to the
actual hardware and write some pseudorandom and directed
diagnostics to attempt to expose implementation flaws. This
is a useful exercise and will undoubtedly uncover some im-
plementation errors. However, enumerating all the possible
event orderings and “interesting cases” is extremely difficult
and time consuming.
We feel a better approach is to refine the abstracted for-
mal verification model developed above to allow the results
6
set of events from the start state, ‰Š , to a leaf node as a wit-
ness string since it “witnesses” the execution of the formal
verification model. In a formal sense, we can say that the set
of all witness strings accepted by the formal model defines
the language, ‹ , accepted by the coherence protocol. If the
language ‹ is also accepted by the implementation verifica-
tion, then we have a rigorous connection between the verified
architectural model and its implementation.
Figure 7: An example of a protocol error that was discovered using Mur – had this error gone undetected it would
have resulted in a loss of memory coherence.
Pseudo-Node 1 Pseudo-Node 2

X Y

Shared : E2 Shared : E1
(2) Shared : E1, E2 (6) Shared : E1
(5) Shared : E2 (8) Noncached
8

6
M1 M2
2 2a 5 6a
ReadSharedRsp(X)

)
p(Y
ro
MRead(X)

MDrop(X)

Y)
D
M ad(
Re
M

)
p(Y
edRs
1a 3 4b har
adS
1b
Y : ShClean Re X : ShClean
4a
(1) X : PendingReq
(3) X : ShClean 7
(4) Y : PendingReq
(7) Y : ShClean
E1 E2
4 3a 4c 1c
1
PReadResp(X)

PInvalidate(Y)
PInvalidate(X)
Read(Y)
Read(X)

Proc0 Proc0

5 Results issue more requests, the reachable state space grew expo-
In this section we discuss some preliminary results of the nentially.
SV2 cache coherence verification effort. While the formal When an error is reported by the protocol verifier, we
verification of the abstracted model is complete, the verifica- must determine if the error is a problem with the formal ver-
tion of the implementation is still ongoing. ification model or if it is a genuine protocol design error.
Most of the early errors were a result of modeling errors in
5.1 Architectural Verification the virtual network. However, several protocol design er-
The objective of architectural verification is to show that rors were found one of which would have resulted in a loss
the cache coherence protocol is sound; that is, that it satisfies of memory coherence. Figure 7 shows the protocol error.
the properties outlined in Section 3. The SV2 formal veri- The arrows in Figure 7 show the packets exchanged between
fication model consists of two pseudonodes and the virtual chips, with a square marker indicating the time the packet
network. This model requires 1536 bits of state information. was sent from the chip and a circular marker indicating the
Once the formal verification model was constructed, we be- time the packet was received. Initially, we see that the pro-
gan to verify the protocol in a step-wise manner. First, we al- cessor at pseudonode 1 has the address Y in the ShClean
lowed processors to issue only scalar Read requests. Then, state. A Read(X) request at event time 1 results in an evic-
we allowed processors to issue scalar Read and SWrite op- tion of Y from the E1 cache. The eviction causes a PInvali-
erations. This incremental approach kept debugging rela- date(Y) packet to be sent to the dcache in the processor and
tively simple. However, as we allowed the processors to an MDrop(Y) eviction notice to be sent to the directory.

7
Figure 8: An example of a witness string generated by
the Mur formal verification tool.
--------------------------------------------------
Issuing scalar request Read from Proc_1 of Node_1
--------------------------------------------------
E1(1:ShClean)<---Read(2) [C=0] on vc0 from Proc_1
Quiescent: 1
Note: Evicting addr 1 from the Ecache
E1 sending PInvalidate(1) to Proc_1
E1 sending MDrop(1) to M1 on vc2
E1 sending MRead(2) to M2 on vc0
--------------------------------------------------
M1(Shared)<---MDrop(1) on vc2 from E1
Quiescent: 0
--------------------------------------------------
M2(Noncached)<---MRead(2) on vc0 from E1
Quiescent: 0
--------------------------------------------------
Memory manager rule fired.
--------------------------------------------------
M2(PendMemExclusive)<---MemRExclRsp(2) from MMGR1
Quiescent: 0
M2 sending ReadExclResp(2) to E1 on vc1
--------------------------------------------------
E1(2:PendingReq)<---RExclResp(2) on vc1 from M2
Quiescent: 0
Note: Filling Ecache line...
E1 sending PReadResp(2) to Proc_1
--------------------------------------------------
At the same time, the MRead(X) request is sent to the
directory, which responds with a ReadSharedRsp(X) re-
sponse. Then, at event time 4 the processor issues a Read(Y)
request to the E1 cache. The E1 cache evicts X and sends
a PInvalidate(X) to the dcache in the processor and an
MDrop(X) eviction notice to the directory. At the same
time, the E1 cache sends the MRead(Y) request to the direc-
tory, which responds with a ReadSharedRsp(Y). Finally,
the MDrop(Y) request from the eviction notice sent at time
1b reaches the directory at event time 8. When it receives
the eviction notice, the directory removes E1 from the shar-
ing set and checks to see if there are any remaining sharers.
Since there are none, the directory transitions to the Non-
cached state. At this point, any stores to Y will not be propa-
gated to the E1 cache, resulting in a loss of cache coherence.
While it would have been difficult to construct a test to un-
cover the complex sequence of events that led to this error,
our formal verification approach was able to discover it au-
tomatically.
5.2 Implementation Verification
The objective of the implementation verification is to run
the “witness strings” generated by the Mur  formal verifi-
cation on the Verilog RTL implementation of the hardware.
The Verilog is compiled and simulated using an internally
developed tool called Gensim [23]. The witness strings are
encoded using a high-level verification environment called
Raven [24, 25], which is built around the C/C++ language.
8
Figure 9: The markers inserted in the stimulus from the
witness string shown in Figure 8.
Quiescent
E1(X:ShClean)<---Read(Y) on vc0 from P1 [13]
E1 sending PInvalidate(X) to P1 [14]
E1 sending MDrop(X) to M1 [15]
E1 sending MRead(Y) to M2 [16]
E1(Y:PendingReq)<---RExclResp(Y) on vc1 from M2 [17]
E1 sending PReadResp(Y) to P1 [18]
Quiescent
The witness strings from the Mur  formal verification are
post-processed into stimulus encoded as Raven apply() and
verify() calls.
As an example, Figure 8 shows a snippet from a witness
string as generated by the Mur  formal verification tool.
This string is converted into stimulus for the Verilog simu-
lation model by choosing a “perspective” from which to ob-
serve the events. In this case, we will observe them from the
perspective of the E chip (L2 cache). Alternatively, we could
have chosen to observe them from the memory directory, for
instance. Choosing the E chip perspective implies that we
will be injecting stimulus into the E chip and verifying that
the E chip produces the correct responses. The example in
Figure 8 is a very simple sequence of events where a pro-
cessor is simply making a Read(Y) request. However, this
simple request has a total of six memory transactions associ-
ated with it. Markers are inserted into the stimulus when the
memory system is quiescent, as shown in Figure 9.
6 Conclusion
We have validated this approach by successfully “replay-
ing” the witness strings from the Mur  formal verification
model on the RTL implementation of the E chip. In the pro-
cess we uncovered eight implementation errors relating to
the cache coherence engine in the E chip. Soon, we will be
using this approach to verify the cache coherence engine at
the memory directory (the M chip).
It is commonplace in computer architecture to use trace-
driven simulation to evaluate the performance of an architec-
tural feature. We propose a similar idea for verifying the cor-
rectness of a cache coherence protocol based on a formal ex-
ecution trace generated by executing the formal verification
model. This formal trace file can be encoded into a practical
format that allows it to be simulated on the actual RTL im-
plementation. This technique provides a rigorous method for
bridging the abstraction gap between the architectural verifi-
cation and the RTL implementation verification. While this
work is still ongoing, our initial experience with this pro-
posed method is very promising.
Acknowledgements
This work was supported in part by National Science
Foundation grants no. EIA-9971666 and CCR-9900605.
References
[1] Dan Lenoski and W.D. Weber. Scalable Shared-Memory Mul-
tiprocessing, pages 143–170, 134–140. Morgan Kaufmann
Publishers, 1995.
[2] John Hennessy and David Patterson. Computer Architecture:
A Quantitative Approach, pages 655–749. Morgan Kaufmann
Publishers, second edition, 1996.
[3] David Culler, Jaswinder Pal Singh, and Anoop Gupta. Paral-
lel Computer Architecture: A Hardware/Software Approach,
pages 273–305 and 589–610. Morgan Kaufmann Publishers,
1998.
[4] David J. Lilja. Cache coherence in large-scale shared-memory
multiprocessors: Issues and comparisons. ACM Computing
Surveys, 25(3):303–338, September 1993.
[5] S. V. Adve. Designing Memory Consistency Models for
Shared-Memory Multiprocessors. PhD thesis, Computer Sci-
ences Department, University of Wisconsin-Madison, De-
cember 1993.
[6] S. V. Adve and K. Gharachorloo. Shared memory consistency
models: A tutorial. IEEE Computer, 29(12):66–76, December
1996.
[7] K. Gharachorloo, D. Lenoski, J. Laudon, P. Gibbons,
A. Gupta, and J. L. Hennessy. Memory consistency and
event ordering in scalable shared-memory multiprocessors.
In Proc. 17th Annual Int’l Symp. on Computer Architecture,
ACM SIGARCH Computer Architecture News, page 15, June
1990. Published as Proc. 17th Annual Int’l Symp. on Com-
puter Architecture, ACM SIGARCH Computer Architecture
News, volume 18, number 2.
[8] William W. Collier. Reasoning About Parallel Architectures.
Prentice-Hall, 1992.
[9] W. Collier. Multiprocessor diagnostics home page
www.infomall.org/diagnostics/archtest.html.
[10] L. Barroso, S. Iman, J. Jeong, K. Oner, K. Ramamurthy, and
M. Dubois. RPM: A rapid prototyping engine for multi-
processor systems. IEEE Computer, 28(2):26–34, February
1995.
[11] M. Dubois, J. Jeong, Y. H. Song, and A. Moga. Rapid hard-
ware prototyping on rpm-2: Methodology and experience.
[12] D. Lenoski, K. Gharachorloo, J. Laudon, A. Gupta, J. Hen-
nessy, M. Horowitz, and M. Lam. Design of scalable shared-
memory multiprocessors: The DASH approach. CompCon
Spring 1990.
[13] M. Plakal, D. J. Sorin, A. E. Condon, and M. D. Hill. Lamport
clocks: Verifying A directory cache-coherency protocol. In
Proc. of the 10th ACM Annual Symp. on Parallel Algorithms
and Architectures (SPAA’98), June 1998.
[14] A. Condon, M. Hill, M. Plakal, and D. Sorin. Using lamport
clocks to reason about relaxed memory models. In Proc. of the
5th International Symposium on High-Performance Computer
Architecure (HPCA-5), Jan 1999.
[15] James Laudon and Daniel Lenoski. The SGI origin: A cc-
NUMA highly scalable server. In Proceedings of the 24th
Annual International Symposium on Computer Architecture
(ISCA-97), volume 25,2 of Computer Architecture News,
pages 241–251, New York, June2–4 1997. ACM Press.
[16] Ásgeir Th. Eiríksson, John Keen, Alex Silbey, Swami
Venkataraman, and Michael Woodacre. Origin system design
methodology and experience: 1m-gate asics and beyond. In
COMPCON-97, 1997.
[17] Ásgeir T. Eirı́ksson. Integrating formal verification methods
with A conventional project design flow. In 33rd Design Au-
tomation Conference (DAC’96), pages 666–671, New York,
June 1996. Association for Computing Machinery.
[18] F. Pong, M. Browne, G. Aybay, A. Nowatzyk, and M. Dubois.
Design verification of the S3.mp cache-coherent shared-
memory system. IEEETC, 47(1):135–140, 1998.
[19] Fong Pong and Michel Dubois. Formal verification of com-
plex coherence protocols using symbolic state models. Jour-
nal of the ACM, 45(4):557–587, July 1998.
[20] K. L. McMillan. Symbolic Model Checking, pages 61–85.
Kluwer Academic Publishers, 1993.
[21] D. L. Dill, A, J. Drexler, A. J. Hu, and C. H. Yang. Protocol
verification as a hardware design aid. In International Con-
ference on Computer Design, VLSI in Computers and Proces-
sors, pages 522–525, Los Alamitos, Ca., USA, October 1992.
IEEE Computer Society Press.
[22] L. Lamport. How to make a multiprocessor computer that
correctly executes multiprocess programs. IEEE Transactions
on Computers, 28(9):690–691, 1979.
[23] T. Court. Gensim user manual. Technical report, Cray Inc.,
1996.
[24] D. Abts and M. Roberts. Verifying large-scale multiproces-
sors using an abstract verification environment. In Proceed-
ings of the 36th Design Automation Conference (DAC99),
pages 163–168, June 1999.
[25] D. Abts, M. Roberts, and D. Lilja. A balanced approach to
high-level verification: Performance trade-offs in verifying
large-scale multiprocessors. August 2000. To appear in the
Proceedings of the 2000 International Conference on Parallel
Processing (ICPP-2000).