IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 28, NO.

5, MAY 2017 1403

Path Setup for Hybrid NoC Architectures

Exploiting Flooding and Standby

Flich, Senior Member, IEEE, and Alessandro Cilardo, Senior Member, IEEE
Edoardo Fusella, Jose

Abstract—Future many-core systems will require energy-efficient, high-throughput and low-latency communication architectures.
Silicon Photonics appears today a promising solution towards these goals. The inability of photonics networks to perform inflight
buffering and logic computation suggests the use of hybrid photonic-electronic architectures. In order to exploit the full potential of
photonics, it is essential to carefully design the path-setup architecture, which is a primary source of performance degradation and
power consumption. In this paper, we propose a new path-setup approach which can put allocated circuits in a stand-by state, rapidly
restoring them when needed. Path-setup messages are sent using a flooding routing strategy to enhance the possibility of finding free
optical paths. We compare the proposed approach with a commonly used path-setup strategy as well as some other alternatives
available. The results exhibit encouraging improvements in terms of both performance and energy consumption.

Index Terms—Silicon photonics, path-setup, hybrid photonic-electronic networks-on-chip, parallel architectures, on-chip interconnection
networks

Ç
1 INTRODUCTION

S INCE the last decade, the semiconductor industry has

stopped aiming for solutions based on single, very pow-
erful microprocessors, looking instead for parallel systems
made of an increasing number of cores integrated on a single
die. However, highly parallel chips have to face tight chal-
lenges in terms of energy and heat dissipation. Transistor
power consumption does not scale with the integration den-
sity and performance, causing a serious increase in the over-
all chip power dissipation. In this context, the role of the on-
chip communication infrastructure is critical, as it provides
the required facilities to distribute the computation among
the different cores, largely impacting the overall system per-
formance and power consumption. Networks-on-chip
(NoCs) are a prominent paradigm for organizing the com-
munication in chip multiprocessors (CMPs) due to their scal-
able and modular nature, although they are still constrained
by power dissipation, latency, and achievable bandwidth. In
fact, the power due to the communication architecture is
around 30 percent of the whole die power [1] and is still too
high to meet the expected needs of future systems [2]. On the
other hand, the latency and bandwidth offered by today’s
NoCs does not fully satisfy the needs of CMP applications,
particularly for accessing on-chip memory [2].
Silicon Photonics appears today a promising path to low-
power ultra-high bandwidth on-chip communication.

E. Fusella and A. Cilardo are with the Department of Electrical Engineer-
ing and Information Technologies, University of Naples Federico II, Naples
80125, Italy. E-mail: {edoardo.fusella, acilardo}@unina.it.

J. Flich is with the Department of Computer Engineering (DISCA), Universi-
tat Politecnica de Valencia, Valencia 46071, Spain.
E-mail: jflich@disca.upv.es.
Manuscript received 15 Feb. 2016; revised 8 Oct. 2016; accepted 16 Oct. 2016.
Date of publication 27 Oct. 2016; date of current version 12 Apr. 2017.
Recommended for acceptance by A. Benoit.
For information on obtaining reprints of this article, please send e-mail to:
reprints@ieee.org, and reference the Digital Object Identifier below.
Digital Object Identifier no. 10.1109/TPDS.2016.2622265
1045-9219 ß 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
Nanophotonic waveguides, the photonic counterpart of a
wire, can in fact achieve bandwidths in the order of terabits
per second by exploiting wavelength division multiplexing
(WDM) [3], while broadband electro-optic switches are
capable of routing single-channel data rates up to 40 Gb/s
[4]. In addition, photonic signaling is expected to consume
less power than electrical interconnects. However, photon-
ics is not suitable for implementing traditional buffered
NoCs due to its inability to store and process data without
an optical-electronic-optical conversion. A viable solution
consists of combining electronic and photonic technologies
to build a hybrid network made of two subnetworks: an
electronic packet-switched network (ENoC) for handling
control and short messages and a photonic circuit-switched
network (ONoC) for larger messages or bursty traffic [5].
Circuit switching requires a path-setup protocol to allocate
the required resources to send a message (or burst)
optically. Obviously, the path-setup incurs an overhead in
terms of performance and costs, possibly amortized by
sending large data messages: the setup latency is higher
than the transmission latency and the setup power con-
sumption is much higher than the transmission power con-
sumption. In addition, when two or more transmissions
require the same resource, a conflict arises. Conflicts can be
handled in two different ways: messages are either sent
using the ENoC, or the requestor tries again to setup the
path for the conflicted message. Sending a large message
through the electronic network may however lead to a dras-
tic degradation in terms of latency and power. As a conse-
quence, reducing the number of large messages sent
electronically is a must. Note that, once a path is set, there
are no additional costs or consumption in the electronic net-
work. On the other hand, keeping an optical path unused
results in consuming power in the optical layer as well as
incrementing the probability of conflicts in path setup pro-
cesses. This tradeoff raises the question of when and for
1404 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,
how long to allocate the resources under given power
constraints.
To fully exploit the benefits inherent in photonics, one
needs to design the control network as well as the path-
setup protocol so as to minimize the path-setup overhead
while maximizing the number of concurrent optical paths
and reducing the probability of conflicts. In this paper, we
propose a new path-setup protocol exploiting a flooding
routing strategy and the possibility of putting allocated cir-
cuits in a stand-by state. Compared to traditional path-setup
protocols, the main advantages are listed below:

The flooding routing strategy provides high adap-
tiveness in routing enhancing the possibility of find-
ing free paths. In this way, we can reduce the
number of conflicts that increase the latency and
energy consumption.

By putting allocated circuits in a stand-by state, the
number of path-setup attempts are reduced. In this
way, we can allocate an optical path without per-
forming an entire path-setup phase, thus limiting the
flooding of control messages in the network. This
will further improve the energy efficiency.
We compare the proposed approach with a commonly used
path-setup algorithm and some available alternatives. The
results exhibit encouraging improvements in terms of both
performance and energy consumption.
A few preliminary results of this work were reported in a
previous conference publication [6]. In this paper, we pres-
ent the full methodology and we consider a more detailed
architecture description, improved optical loss and power
estimation models, and a comprehensive experimental eval-
uation involving both synthetic traffic and real applications.
In addition, a formal proof of the deadlock freedom of the
proposed approach is presented. The paper is organized as
follows: Section 2 discusses some related works in the tech-
nical literature. Section 3 introduces the target architecture
and presents the adopted optical loss and energy models.
Section 4 thoroughly describes the path-setup protocol, the
routing algorithm, and the deadlock avoidance policy.
Section 5 presents the experimental setup and the results
achieved. Section 6 provides some final remarks.
2 RELATED WORK
2.1 Photonic Networks-on-Chip
While optical interconnects are widely used to provide com-
munication facilities over long and medium distances [7], the
recent advancements in nanophotonic technologies also
make silicon photonics a promising approach for on-chip
communication. Several works in the recent years have ana-
lyzed the benefits and limits of optics versus electronics.
In [8], a comparison between electrical wires and optical inter-
connects in terms of delay and power is presented. The results
show that optics is advantageous for global signaling, while
electrical wires are more power-effective for shorter link
lengths. Based on the semiconductor technology roadmap,
[9] investigates the requirements that photonic interconnects
should meet in order to outperform traditional electronic
interconnects. In [10] a study of the critical length beyond
which optical interconnect pays off is presented. The results
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
VOL. 28, NO. 5, MAY 2017
show that this length depends on the process technology scal-
ing, e.g., in case of a 22 nm process, it is equal to one-tenth of
the chip edge length. A comparison of different emerging
technologies for on-chip interconnects is carried out in [11].
According to the authors, silicon photonics exhibits the low-
est latency and the highest possible bandwidth in case of
wavelength division multiplexing. In [12] the authors outline
the opportunities and challenges of different emerging inter-
connect technologies, including nanophotonic communica-
tion. They evaluate the possibility of combining three-
dimensional (3D) stacking technology and silicon photonics
to exploit the advantages of both technologies.
In the recent years, several research efforts have focused
on proposing various solutions at the architectural level.
[13] presents a high-bandwidth multiple-wavelength trans-
mission scheme, which essentially exploits the ability to
transmit parallel wavelength channels through a single
microring resonator. Shacham et al. [5] proposes a hybrid
torus-based NoC made up of a photonic circuit-switched
and an electronic packet-switched network. In [14], the
Corona Multiple-Writer Single-Reader (MWSR) photonic
ring is presented. Pan et al. [15] describes Firefly, a clustered
architecture locally connected via an electronic medium and
globally connected via an optical crossbar, partitioned into
multiple logical crossbars arbitrated locally. Mo et al.
[16] proposes a hybrid optical-electronic mesh-based NoC
exploiting hybrid optical-electronic routers able to perform
selective transmission. In [17], the THOE architecture is pro-
posed, a low cost torus-based hybrid optical-electronic NoC
that can be implemented with a lower number of waveguides
and ring resonators compared to [5]. Bahirat and Pasricha
[18] presents METEOR, a complete communication architec-
ture consisting of a reconfigurable electrical mesh coupled
with an optical ring whose waveguides can be configured as a
combination of Single-Writer Multiple-Readers (SWMR) and
Multiple-Writers Multiple-Readers (MWMR). Beux et al.
[19] introduced CHAMELEON, a Single-Writer Single-
Reader optical network-on-chip able to exploit reconfigurabil-
ity both at run- or compile-time in order to open and close
dedicated channels between cores. In [20], a hybrid elec-
tronic/photonic, hybrid-topology NoC, called H2 ONoC, is
presented. Thanks to the novel topology, H2 ONoC achieves a
better energy efficiency compared to torus-based architec-
tures. To reduce the energy consumption and achieve a higher
bandwidth, [21] presents a topology comparison between dif-
ferent architectures and introduces an improved topology for
the solution presented in [5]. Finally, [22] presents PhoNoC-
Map, a computer-aided design tool addressing the design
space exploration of optical NoC mapping solutions. Pho-
NoCMap automatically assigns application tasks to the nodes
of a generic photonic NoC architecture such that either the
worst-case insertion loss or crosstalk noise are minimized.
2.2 Path-Setup
A few electronic NoC architectures employ circuit switch-
ing in order to provide guaranteed service levels. A repre-
sentative example is the ÆTHEREAL [23] NoC, developed
at Philips, that provides guaranteed throughput alongside
best-effort service. A basic version of the path-setup proto-
col relies on four types of control messages: path-setup, path-
ack, path-nack, and path-teardown. Basically, a path-setup
FUSELLA ET AL.: PATH SETUP FOR HYBRID NOC ARCHITECTURES EXPLOITING FLOODING AND STANDBY
message is sent from the source of the communication. The
aim of this message is to reach the destination node while
properly setting the required resources along the path. In
case of success, a path-ack message is generated and sent
back to the source of the communication. On the contrary,
when a path-setup message reaches a router and it is unable
to take an output port due to conflicts with other allocated
circuits, a path-nack is sent back to the source. When a path-
ack reaches its destination, the end-to-end path is ready for
the communication and hence the data message is sent.
When the communication ends, the resources previously
allocated are released by sending a path-teardown message.
The standard path-setup protocol is used-with some
enhancements—in many hybrid electronic—photonic architec-
tures [5], [16], [17], [24]. It is henceforth referred to as the baseline
protocol. To deal with the overhead introduced by the path-
setup as well as the E/O and O/E conversions, existing solu-
tions often use a selective transmission policy injecting in the
optical network only messages with a size higher than a certain
threshold. In addition, a few approaches [5], [17] exploit the
symmetric property of optical paths in order to send back the
path-ack message along the reserved optical path instead of
using the electronic control network. This requires a non-negli-
gible implementation cost since routers should be enhanced
with additional optical components, i.e., microring resonators
and waveguides [17]. In [5], the standard torus network is aug-
mented with optical paths exploiting path multiplicity, so that
the blocking probability is reduced with a consequent benefit
in terms of path setup latency.
However, the congestion on the control layer is the main
drawback of all these approaches. A few works [18], [25], [26]
rely on an alternative ring-based path-setup network that is
able to configure multiple optical switches simultaneously
instead of sequentially. However, this benefit comes at a non-
negligible implementation cost in terms of required wave-
guides and ring resonators. In addition, implementing two
different optical networks, i.e., a control and a data network,
introduces many waveguide crossings due to the planar
nature of the topologies implementable on Silicon. This conse-
quently leads to a raise in crosstalk noise and power loss,
which potentially constrains the network scalability and pre-
vents the whole architecture from operating properly.
To the best of our knowledge this is the first work investi-
gating new path-setup protocols able to put allocated cir-
cuits on a stand-by state, rapidly restoring them when
needed. In addition, to enhance the chance of finding free
paths, path-setup messages rely on a flooding routing strat-
egy. The key contributions in this paper thus are:

a new path-setup protocol;

a routing algorithm designed to take full advantage
of the path-setup protocol;

a formal proof of the deadlock freedom of the pro-
posed routing algorithms for the path-setup protocol
with standby;

a comparison with a commonly used path-setup
approach and a few other alternatives using both
synthetic traffic and real applications.
3 ARCHITECTURE OVERVIEW
The communication architecture is made of two different
layers: an electronic packet-switched network where short
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
1405
Fig. 1. The two building blocks of the Hybrid NoC: (a) The electronic
router. (b) The Cygnus photonic switch.
data and control messages are transmitted, and an optical
circuit-switched network for large messages or bursty traf-
fic. The nodes are organized in a mesh topology.1 The archi-
tecture is composed of k0  k1 tiles, each containing an
electronic router and a photonic switch connected with an
IP core and with the four neighboring tiles. In the following
we will use ðx; yÞ with 0  x  k0  1 and 0  y  k1  1 to
indicate a tile of the network.
Each core is connected to a network interface (NI) own-
ing the necessary logic to perform selective transmission.
Basically, when the head flit of a message is injected in the
NI, it is buffered according to the message size: messages
shorter than a predefined value are stored in a buffer and
then directly sent via the ENoC, while larger or bursty mes-
sages use a different buffer directly connected to the
Optical/Electronic/Optical (O/E/O) interface handling
serialization, deserialization, and OE/EO conversion. Buf-
fers are implemented as FIFO queues. When a large mes-
sage is to be sent, the path-setup phase begins and, if
successful, the message is sent via the ONoC.
Fig. 1a shows the router block diagram. Since this router
is designed to handle only short data and control messages,
it does not require the support for large messages and,
hence, it is optimized for latency and not for throughput. In
addition, it holds the necessary logic to take path-setup
decisions. The network is non-interfering, meaning that two
virtual networks are used in order to prevent data packets
from blocking control packets. This is essential since the
setup procedure must be as fast as possible in order to pre-
vent sending large messages through the ENoC. The two
networks are identical from a topological point of view.
Consequently, each electronic router is coupled with a pho-
tonic switch (PS) that is the basic building block of the ONoC.
Fig. 1b shows the five-port PS considered in our study,
called Cygnus, first presented in [27]. Concerning the elec-
tronic router, there are five bidirectional ports, the four
ports corresponding to the cardinal directions and the local
port connected to the O/E and E/O conversion modules.
Cygnus is a non-blocking switch, meaning that it is always
possible to use simultaneously two different pairs of
1. Notice that, although we rely on a mesh topology, the proposed
approach can be extended to any other topology. In case of topologies
requiring five-ports optical routers, no changes are needed. Differently,
in the other cases, the router data structures and the deadlock avoid-
ance rules should be adapted.
1406 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 28, NO. 5, MAY 2017

input-output ports. In addition, unlike a few previously TABLE 1

proposed switches, it is not optimized for a certain routing Optical Loss Parameters
algorithm. Furthermore, it implements straight default
Parameter Value Ref.
paths, i.e., paths taken by the signal when all the rings are
in an OFF resonance state, thereby crossing a reduced Modulator 0.5 dB [33]
Photodetector 0.1 dB [33]
number of microring resonators (MRs) in ON resonance Coupler 1 dB [33]
with a consequent power gain. As explained in Section 4, Propagation Loss in Silicon 1.5 dB/cm [34]
these features are essential for our approach. By injecting Waveguide Crossing 0.05 dB [35]
electronic carriers into the device, due to free-carrier dis- Waveguide Bend 0.005 dB/90 [34]
persion effect [28], each Path-Setup Unit (PSU) controls the Dropping into a Ring 0.5 dB [28]
MRs in the same tile, in order to ensure that all the micror- Passing by a Ring 0.005 dB [28]
ing resonators are always set properly for a data transmis-
sion. A detailed description of the path-setup protocols photonic signals as they propagate through a waveguide:
and of the PSU is given in Section 4. the optical signal power must be higher than the photode-
tector sensitivity plus the worst-case insertion loss to ensure
3.1 O/E/O Interface a proper detection in the reception stage. In addition, the
The O/E/O interface is made up of several components: total power cannot exceed a certain threshold due to the
serializer, laser source, and driver circuit in the generation nonlinearities of the silicon material. In case of WDM sig-
stage as well as photodetector, transimpedance amplifier nals, these considerations yield the following inequality [21]
(TIA), limiting amplifier (LA), and deserializer in the recep- P  S  ILwc þ 10 log10 N ; (1)
tion stage. The generation stage handles the data transition
from the electronic domain to the optical domain. The data where P is the maximum signal power allowed to be
is first serialized. Then, the driver circuit controls the optical injected considering a threshold for nonlinearities, S is the
modulator device which translates the electrical signal into photodetector sensitivity and N is the total number of
an optical signal through a laser source by modulating the wavelengths.
light intensity according to the data bit values. Conversely, The worst-case insertion loss is given by Equation (2)
the reception stage is responsible for bringing back the opti-
cal signal into the electrical domain. The photodetector ILdB dB dB dB dB
wc ¼ ILmod þ ILdetect þ ILcoup þ ILprop
(2)
turns the light waves into an electrical current that is con- þ ILdB dB dB dB
cross þ ILbend þ ILdrop þ ILpass ;
verted into electrical voltage by TIA. Then, LA brings the
electrical voltage to the proper logic level and last, the digi- with ILdB dB dB dB dB dB dB dB
mod , ILdetect , ILcoup , ILprop , ILcross , ILbend , ILdrop , ILpass
tal signal goes through the deserializer stage. being respectively the loss due to electro-optic modulators,
Serialization/deserialization circuits are key for handling photodetectors, couplers, waveguide propagation, wave-
the electronic/photonic clock mismatch. For the simulation guide crossings, waveguide bends, ring dropping, ring
model of the serializer and deserializer, we rely on the passing in the worst-case scenario. Table 1 shows the uni-
design presented in [29] due to its low power consumption. tary values of insertion loss.
Concerning the light source, we choose a vertical cavity sur- Once the maximum number of exploitable wavelengths
face emitting laser [30] since it can be directly modulated by is defined, it is possible to evaluate the maximum number
the driving current without the need for an ad-hoc optical of wavelength channels and hence the bandwidth as
modulator device and requires reduced power consump-
tion. For the photodetector, we consider a low-voltage ger- Bw ¼ clk  N ; (3)
manium photodetector with a sensitivity of 14:2 dBm at a where clk is the optical clock speed. Note that the network
1012 bit error rate and a data rate of 10 Gbps, as demon- scalability, including the maximum number of wavelength
strated in [31]. This device provides a good tradeoff channels, may be limited by electro-magnetic effects such as
between its sensitivity and the supported data rate. Note the crosstalk noise [36]. In addition, microrings need to pro-
that, the photodetector sensitivity and the BER are highly vide multiwavelength routing capabilities and hence they
interrelated: the receiver BER for different optical signal must be designed with a small free spectral range (FSR), i.e., the
power levels can vary by several orders of magnitude. In spacing between different wavelengths that resonate with the
this paper, we neglected such data integrity aspects as they ring. However, the FSR is inversely proportional to the cir-
are out of the scope of the work. Concerning the TIA-LA cir- cumference of the ring and hence the minimum achievable
cuits, we rely on the design presented in [32] since it FSR is bounded by physical constraints such as the maximum
achieves a lower power consumption than other proposals. ring diameter. This paper neglects such electro-magnetic
effects as well as the physical constraints as it mainly focuses
3.2 Optical Loss and Bandwidth Model on the design of the electronic path-setup architecture.
Electronic networks make use of several distinct wires to
enhance the parallelism of a communication. Differently, by 3.3 Energy Models
exploiting Wavelength-Division Multiplexing each wave- In order to evaluate the energy consumption of the whole
length in an optical communication can carry different data network, we need to consider the consumption of both the
in a single waveguide. In order to achieve this benefit, we ENoC and the ONoC. Concerning the electronic power con-
need to properly deal with insertion loss (IL), which affects sumption, we rely on the method presented in [37]
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
FUSELLA ET AL.: PATH SETUP FOR HYBRID NOC ARCHITECTURES EXPLOITING FLOODING AND STANDBY
TABLE 2
Power Consumption for Electronic
Signaling
Parameter Value
Elink 0.34 pJ/mm/bit
Ebuffer 0.12 pJ/bit
Ecrossbar 0.36 pJ/bit
Estatic 0.35 pJ/bit
previously used in a similar work [5]. The method is based
on the assumption that the following operations are
required at each hop:

reading from the buffer;

taking routing and arbitration decisions;

crossing the inner switch;

going through the link;

writing to a buffer.
As a consequence, the energy consumed by sending a mes-
sage end-to-end is equal to the sum of these energy values
times the number of hops:
Emessage ¼ Ehop  Nhops ;
where Nhops is the number of hops taken by a message and
Ehop is the energy necessary to cross a single hop:
Ehop ¼ ðElink  dlink Þ þ Ebuffer þ Ecrossbar þ Estatic ;
where dlink is the link length between two adjacent routers.
Table 2 reports the values of the energy per bit consumed
for these operations (Ebuffer is the sum of the components
due to reading and writing, while the energy due to routing
and arbitration is neglected) as evaluated in [5].
Differently, when sending a message in the photonic net-
work, the energy is consumed in the O/E/O interface and
in the optical network components along the path. Photonic
signaling benefits from the two proprieties of bit-rate trans-
parency and low loss in optical waveguides, meaning that the
energy consumption in the optical network components,
i.e., waveguides and microring resonators, is independent
of the bitrate and the distance between the two end-points.
The power consumption of a microring resonator depends
on its state: in the OFF state the power is negligible, while in
the ON state it consumes around 10 mW when designed for
switching multiwavelength broadband signals [38]. Conse-
quently, the energy that a message consumes depends on
how long the microrings are in the ON state regardless of
whether they are used or not.
On the other hand, the O/E/O interfaces are made up of
several power hungry components: serializer, deserializer,
driver, VCSEL, photodetector, and TIA-LA circuits. The
power consumptions of the serializer and deserializer are
respectively 2 and 1.6 mW in 90-nm CMOS processes [29].
The power consumptions of the VCSEL driver and TIA-LA
circuits are respectively 2 and 6 mW in 80-nm CMOS pro-
cesses [32]. Since we target a 32-nm process technology, the
above values are scaled linearly to 32 nm. We neglect the
photodetector power consumption since it is some orders of
magnitude lower. Finally, we evaluate the power consump-
tion of the VCSEL as follows. As in [17], we assume the laser
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
1407
TABLE 3
Power Consumption for Photonic
Signaling
Parameter Value
MRON 10 mW
serializer 0.71 mW
deserializer 0.57 mW
driver 0.8 mW
TIA  LA 2.4 mW
VCSEL @4dB 0.32 mW
VCSEL @8dB 0.80 mW
VCSEL @12dB 2 mW
VCSEL @16dB 5.04 mW
source to generate just enough power to ensure a proper
detection of the optical signal at the photodetectors. As a con-
sequence, when generating a new optical signal, we need to
evaluate the power loss the signal is subject to when travel-
ing to the destination. In this way, we can avoid considering
the worst-case insertion loss for every communication.
Table 3 reports the power consumption values for these
operations including the VCSEL power consumption for
four possible loss values assuming 30 percent laser efficiency
(4)
and 14:2 dBm sensitive receivers. Notice that our approach
is independent of the used power consumption values and
optical loss coefficients, which are only indicative and may
be influenced by technology and architectural variables.
(5)
4 PROPOSED PATH-SETUP FOR HYBRID NOCS
4.1 Minimal Flooding Routing
The path-setup protocol is responsible for allocating the
required resources guaranteeing that a required optical com-
munication can take place. A good path-setup protocol
should reduce the number of conflicts, which slow down the
path-setup phase and result in a higher overhead in terms of
performance and power consumption: for every conflict, the
path-setup procedure is interrupted and iterated until the
destination is reachable and the data can be transferred. As a
consequence, less conflicts results in a lower average number
of path-setup attempts per communication.
While the baseline protocol described in Section 2.2 relies
on the XY dimension-order scheme for routing control mes-
sages, the four turns allowed by the XY solution do not per-
mit any adaptiveness in routing, potentially leading to poor
performance. Differently, the proposed approach imple-
ments a minimal path flooding routing algorithm to route
path-setup messages. With the minimal path flooding, each
router checks if it is the target router. If not, the path-setup
message is sent to the neighbors that are closer to the target
router. Each router repeats the process or, in case of a con-
flict, generates a path-nack message.
Flooding leads to exploring paths otherwise inaccessible
using XY routing. However, it requires message replication
and may potentially increase the power consumption as well
as the network congestion perturbing the ENoC perfor-
mance. On the other hand, there are more possibilities to
find a free path, thus avoiding multiple path-setup attempts.
In other words, the differences in the routing approach intro-
duce a tradeoff between the breadth and depth of the path
exploration process and the number of times the exploration
1408 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 28, NO. 5, MAY 2017

is performed. At the same time, in the photonic layer, the

latency is independent of the distance between the two end-
points, while PSs with straight default paths make the energy
consumption depend only on the number of turns. Hence,
optical paths with less turns are preferred.

4.2 Path-Setup Protocol with Standby

As previously described, using a flooding routing algorithm
does not only improve the chances of finding free paths but
it also increases the number of messages in the control net-
work and hence the possibility of congestion. The proposed
path-setup protocol with standby allows using flooding for
routing path-setup messages, and yet it reduces the number
of messages in the network by decreasing the probability of
conflicts in the routers.
The idea is to exploit the deterministic nature of burst
communication, where a source node communicates with a
destination node for a while before switching to a new desti-
nation. As soon as an optical communication finishes, we
put the path in standby instead of tearing it down. In a
standby path, the ring resonators are in OFF mode but the
path is reserved for a further use without the need to per-
form again costly flooding operations.
Implementing the proposed protocol requires two new
messages, i.e., path-standby and path-wakeup, in addition to
the four messages of the baseline protocol: (path-setup, path-
ack, path-nack, and path-teardown). Note that these control
messages contain only the destination address and three
bits specifying the message type.
In order to implement a distributed protocol, we added
in each router a Path-Setup Unit whose functions are:

configuring the switching functions of the corre-
sponding PS;

keeping track of the PS state;

managing path-setup conflicts;

implementing the routing policy for control
messages.
Each PSU owns an internal PS Connectivity Table (PCT) and a
Counter Array (CA) whose structures are shown respectively
in Figs. 2a and 2b. The PCT contains for each output port the
fields State, Input Port, Source Node, and k fields Destination,
where k is the maximum number of paths going through this
port, which can be simultaneously in the standby state. Note
that the maximum number of standby paths in the network
depends on application communication patterns. Assuming
standby paths uniformly distributed over the source nodes, k
PNsrc 1
Ndst
may be evaluated as d i¼0Nsrc i e, where Nsrc is the total
number of source nodes and Ndsti is the number of destination
nodes involved in an optically communication with the ith
source node. The size of these data structures is constant for
the NoC design, regardless of the network size, guaranteeing
scalability. In case of different topologies requiring high-radix
routers, the size of these structures scales linearly with the
radix of the routers.
The PS output ports are in one of five states: Unused,
Reserved, Allocated, Standby, and Woken up. Fig. 2c shows the
finite-state transition diagram for the proposed path-setup
protocol. At the beginning, the state is set to Unused mean-
ing that the port is free for further reservations. When a
path-setup message arrives, the state changes to Reserved and
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
Fig. 2. The data structures located in the Path-Setup Unit for the basic path-
setup protocol. (a) The PS Connectivity Table. (b) The Counter Array.
(c) The finite-state transition diagram for the state of output ports of PSs.
the input port from which the message arrived is stored
in the associated entry. This information is required to guar-
antee that path-ack/path-nack/path-teardown messages are
able to reach the source of the communication in the back-
ward direction. In the Reserved state, when a path-nack mes-
sage arrives, the state goes back to Unused. Otherwise, in
case of path-ack, the state changes to Allocated. In addition,
when a path-ack message reaches a router, the IDs of source
and destination nodes are stored in the corresponding
fields. This is essential for keeping track of the optical paths
in standby state. The Destination node fields are managed
with a Least Recently Used policy keeping track of the
recently used optical connections. An output port in the
Allocated state implies that the necessary ring resonators are
set to establish an optical path between the input and output
ports. Hence, the PS is ready to be used and, as a conse-
quence, it is consuming energy. As soon as the optical com-
munication ends, the source sends a path-standby message
to set the state to standby instead of tearing it down. A
Standby state means that the ring resonators are in OFF
mode but the path is reserved for a further use. The source
node that plans to use a path on standby must send a path-
wakeup message and wait for a path-ack message. In case it
receives a path-nack message, it retries to send a new path-
wakeup. path-nack messages are generated in case of conflicts
since it is possible that another source is trying to tear down
the standby optical path. A new flooding is performed only
after a path-teardown message is received or in case it is nec-
essary to communicate with a new destination.
Paths in Standby state can be used only by the source node
that allocated it for the first time. On the other hand, in order
to avoid source nodes keeping unutilized or standby paths,
which prevent other sources from establishing optical cir-
cuits, a path-setup, not able to reach its destination, is allowed
to tear down standby circuits allocated by other sources. The
path-teardown crosses the network up to the source node of
FUSELLA ET AL.: PATH SETUP FOR HYBRID NOC ARCHITECTURES EXPLOITING FLOODING AND STANDBY 1409

the standby circuit freeing the PCT entry corresponding to

that path. Since the tear-down message could be sent from
any node along the path and not from the destination node,
it is possible that some part of the circuit remains in the
Standby state. In this occurrence, the junk piece of circuit will
never be used again and will be torn down when necessary.
Conflicts arise when path-setup or path-wakeup messages are
not able to reach the next hop since all the required output
ports are Reserved, Allocated, or Woken up and belong to a dif-
ferent optical path. As a consequence, path-teardown mes-
sages will release the resources corresponding to the path in
the Standby state, if any, in order to cross the output port
where the conflict arises. Otherwise, we can either try again
the setup process or send the message via the ENoC. In our
solution we adopted a mixed policy: we try to setup the opti-
cal path for a certain amount of time, after which the message
is sent electronically. In this way, the time that a message can
stay in the queue is bounded, preventing head-of-line block-
ing yet avoiding messages being sent via the ENoC in case of
sporadic conflicts. In case of concurrent path-wakeup and
path-teardown messages related to the same circuit, the path-
wakeup will reach a router where the circuit has already been
torn down. At this time, a path-nack message is generated
and sent back to the source node. Then, a new flooding will
occur, since, in the meanwhile, the path-teardown message
will have reached the source node. Fig. 3. A timing diagram showing an example of possible path-setup
scenarios.
Since we rely on a flooding routing algorithm, it is possi-
ble that a path-setup message needs to be routed to more than
a single output port. Assuming that the path-setup message is reserved by another source node and hence a path-nack mes-
sent through k output ports with k > 1, two resulting sce- sage is generated and sent back. At the next path-setup
narios are possible: the router will receive k path-nack mes- attempt, a path-setup message will be generated, since in the
sages or one path-ack and k  1 path-nack messages. In both meanwhile the standby resources will have been torn down.
cases, we need a synchronization mechanism in order to
avoid leaving junk messages in the network: for each input 4.3 Deadlock Freedom
port, a counter (Fig. 2b) keeps track of how many messages Deadlock is caused by packets waiting on each other in a
coming from that input port were sent. When a path-ack or cycle. In order to ensure deadlock freedom, the routing algo-
path-nack message arrives, the corresponding counter is rithm can be designed so as to prohibit just enough turns to
decremented. Only when the counter is equal to 0, the path- break all of the cycles in the network. Basically, a turn involves
ack or path-nack message is sent back to the previous router a 90-degree change in the traveling direction. In 2D meshes
along the path. In this way, only one message arrives to the there are eight types of turns. A turn is called an NE turn if it
source NI and there are no junk messages left in the network. requires a change of direction from North to East. Similar defi-
Fig. 3 shows a few examples of possible path-setup sce- nitions apply to the other turns, i.e., NW, EN, ES, SE, SW, WN,
narios where the Source Node 0 tries to setup a path to the WS. The drawback is that prohibiting some turns reduces the
Destination Node 0. Fig. 3a illustrates, first, a failed setup adaptiveness of the algorithm. In order to overcome this limi-
since the required output port in Router 2 is allocated by tation, virtual channels can be used [39]: physical channels
another source node, followed by a successful setup. At the may be split into virtual channels, i.e., abstractions that share
end of the optical data transmission, the path is put in the same physical channel, although each virtual channel has
standby. Last, the path is woken up for a further optical its own queue. Basically, virtual channels decouple the alloca-
communication. Differently, in Fig. 3b, the path-setup mes- tion of buffers from the allocation of channels by providing
sage is not able to reach the destination node since a multiple buffers for each physical channel. The cost of imple-
required output port is in a standby state belonging to menting virtual channels is small. Each virtual channel
another optical path. As a consequence, a path-teardown requires its own queue, but the queue size can be as small as a
message is generated and sent to the source of the other single flit. A prohibited turn can be allowed as long as we
optical path to release the resources in standby. Fig. 3c change the virtual channel, effectively breaking potential
shows an example of a failed woken up of a standby path. dependency cycles. In a mesh topology, in case of an arbitrary
Source 0 owns an optical path in standby to reach Destina- routing strategy, the number of forbidden turns in the worst-
tion 0 through Router 0-1-2. However, Source 1 requires case path grows linearly with the network size. Obviously, it
tearing down the optical path in order to reach another des- is not possible to change virtual channel for every invalid
tination. Since the teardown message has not yet reached turn. As a consequence, if a path-setup message cannot cross a
Source 0, Source 0 sends a wake-up message. As soon as the router due to a forbidden turn and the lack of available virtual
wake-up message reaches Router 2, it finds the output port channels, then a path-nack message is generated and sent back.
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
1410 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,
Likewise, when a path-wakeup message is blocked due to a
conflict, a path-nack message is generated and sent back. Simi-
lar considerations apply for path-ack messages. Note that
sending a message in the backward direction introduces 180-
degree turns further exacerbating the deadlock issue.
In the following, we show that the proposed approach is
deadlock free as long as the following rules are met:
Rule 1. As least two virtual channels are used. A higher number
of virtual channels may be used to improve the adaptiveness, in
which case an even number Nvc of virtual channels is required.
Rule 2. Control packets are not allowed to take EN and SW turns
at any nodes located in an even column without changing vir-
tual channel. Control packets are not allowed to take ES and
NW turns at any nodes located in an odd column without
changing virtual channel.
Rule 3. The virtual channels are ordered according to their IDs.
In case of Nvc virtual channels, the IDs range from 0 to
Nvc  1. A message stored on a virtual channel VCi cannot be
forwarded to a virtual channel VCj if j < i, with
0  i  Nvc  1 and 0  j  Nvc  1.
Rule 4. Path-setup, path-wakeup, and path-standby messages are
allowed to use a virtual channels VCi only if 0  i  N2vc  1,
while path-ack, path-nack, and path-teardown messages are
allowed to use a virtual channel VCi only if N2vc  i  Nvc  1.
Rule 1 guarantees that there are enough virtual channels
to allow path-ack/path-nack messages to come back to the
source using the same path of path-setup/path-wakeup mes-
sages in the backward direction. Rule 2 specifies where to
put restrictions and is equivalent to Rules 1-2 of the odd-
even turn model [40]. However, unlike the odd-even turn
model, restrictions are bidirectional meaning that if a packet
crosses a restriction when going to a certain direction,
another packet will cross a restriction when going in the
backward direction. Rule 3 defines a partial order of virtual
channel classes. Rule 4 excludes the occurrence that path-
setup/path-wakeup messages and path-ack/path-nack mes-
sages are waiting on each other in a cycle.
Theorem 1. The minimal flooding routing algorithms for the
path-setup protocol with standby is deadlock free as long as
Rules 1, 2, 3, and 4 are met.
Proof. Assume that there exists a set of packets
p0 ; p1 ; . . . ; pn1 , where piþ1 waits for pi with 0  i  n  2.
A deadlock arises if p0 waits for pn1 , as a circular depen-
dence is generated through the packets. Different cases
need to be considered.
Case 1. All the packets belong to path-setup, path-
wakeup, or path-standby messages. 180-degree turns do
not occur since there are no path-ack, path-nack, or
path-teardown messages. According to Rule 4, the packets
are stored in queues of virtual channel VCi with
0  i  N2vc  1. If Nvc ¼ 2, then all the packets are stored
in the same virtual channel. Since Rule 2 is equivalent to
Rules 1-2 of the odd-even turn model, the reasoning fol-
lows the proof of the odd-even theorem [40]. Differently, if
Nvc > 2, then the packets could be stored in multiple vir-
tual channels. However, according to Rule 3, virtual
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
VOL. 28, NO. 5, MAY 2017
Fig. 4. A few examples of how virtual channels are used to eliminate
deadlocks. The gray and black nodes are respectively sources and
destinations, while the red nodes cannot be crossed due to conflicts. The
bidirectional black arrows indicate routing restrictions. The virtual channel
changes are highlighted and colored according to the message type.
channels are ordered and routing is restricted to visit
channels in increasing order. This will eliminate cycles in
the channel dependency graph and, hence, the routing
function is deadlock free according to Dally’s
Theorem [39].
Case 2. All the packets belong to path-ack, path-nack,
or path-teardown messages. This case can be treated as
Case 1.
Case 3. Packets belong to arbitrary control message.
Suppose packet pi , with 0  i  n  1, belongs to a path-
setup, path-wakeup or path-standby message and packet
pj , with 0  j  n  1, belongs to a path-ack, path-nack,
or path-teardown message.
Suppose that n ¼ 2, then 180-degree turns can occur
since there are both path-setup/path-wakeup and path-
ack/path-nack messages. However, according to Rule 4,
path-setup/path-wakeup/path-standby and path-ack/
path-nack/path-teardown messages are stored in queues
of respectively virtual channel VCi with 0  i  N2vc  1
and VCj with N2vc  j  Nvc  1. These two sets of virtual
channels are disjoint sets, breaking any possible two-cycle
with 180-degree turns. Differently, suppose that n > 2
and i < j. Then, every packet pk with j  k  n  1 must
belong to path-ack/path-nack/path-teardown messages
due to Rules 3, 4. Similarly, every packet ph with 0  h  i
must belong to path-setup/path-wakeup/path-standby
messages. This means that p0 is stored in a virtual channel
VCi with 0  i  N2vc  1 and pn1 is stored in a virtual
channel VCj with N2vc  j  Nvc  1. Consequently, accord-
ing to Rule 3, virtual channels are ordered and routing is
restricted to visit channels in increasing order and, hence,
it is not possible that p0 waits for pn1 . Similar considera-
tions hold true in case of i > j. This will eliminate cycles
in the channel dependency graph and, hence, the routing
function is deadlock free according to Dally’s
Theorem [39]. u
t
Fig. 4 shows an example of how virtual channels are used
in the proposed approach. Consider the communication
between nodes ð0; 5Þ and ð2; 3Þ, and two virtual channels
available. The path-setup message is flooded and it can reach
the destination through multiple paths using virtual
FUSELLA ET AL.: PATH SETUP FOR HYBRID NOC ARCHITECTURES EXPLOITING FLOODING AND STANDBY
TABLE 4
Simulation Parameters
Parameter Value
Chip size (mm ) 2
20  20
Topology size (k0  k1 ) 88
Flit Size (Bytes) 4 Arch-A
Buffer Size (Flits) 6 Arch-B
Electronic Clock Frequency (Ghz) 1 Arch-C
Optical Clock Frequency (Ghz) 10
Photodetector sensitivity (dBm) 14.2
Modulation rate (Gb/s) 10
# Wavelength Channels 20
channel VC0 . To avoid deadlocks, the path-ack message is
sent back on VC1 . Note that in nodes ð1; 5Þ and ð1; 4Þ the
path-setup message is not sent to the South port due to ES
routing restrictions. Similarly, in the communication
between nodes ð3; 4Þ and ð5; 3Þ, the path-setup message is not
able to cross nodes ð3; 3Þ and ð5; 4Þ respectively due to a con-
flict and a router restriction. Path-nack and path-ack messages
are generated and sent back on virtual channel VC1 . Last,
consider the communication between nodes ð1; 1Þ and ð3; 2Þ
and assume here four virtual channels. Due to conflicts on
nodes ð1; 2Þ and ð3; 1Þ, there is only a single path that can be
used. Since there are four virtual channels, we can cross a
single restriction. The path-setup message is sent through
VC0 . As soon as it reaches node ð2; 1Þ, the message switches
to VC1 due to an EN restriction. In the same way, the path-
ack message is sent through VC2 and, in order to cross node
ð2; 1Þ, it exploits VC3 .
5 SIMULATION RESULTS
5.1 Simulation Setup
The simulation environment was set up by modifying
gMemNoCsim, an in-house event-driven cycle-accurate
NoC simulator. The optical bandwidth, the number of
usable wavelengths as well as the energy consumption are
evaluated by incorporating the models provided in Sections
3.2 and 3.3 into the simulator. The main simulation parame-
ters are summarized in Table 4. We targeted a 32-nm pro-
cess technology and we assumed an 8  8 mesh topology
and a 400 mm2 CMP die area. The area footprint of a single
tile is slightly less than 1 mm2 allowing the integration of all
the 64 tiles in the die. This footprint is evaluated by assum-
ing the individual area of modulators and photodetectors
equal to, respectively, 115 and 125 mm2 and the ring diame-
ters of passive filters and broadband comb switches equal
to, respectively, 6 and 200 mm. The hop length, required to
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
D
calculate the propagation loss, is evaluated as ðk0 1Þðk
where D is the die area and k0  k1 is the topology size.
Regarding the ENoC, all the data channels have the same
bus width of 32 bits, equal to a single flit. The routers are
configured to have a buffer depth of 6 flits. The electronic
components in the network are clocked at 1.0 GHz, while
the optical clock speed is conservatively set to 10 Ghz as
in [33]. According to [41], a 10 Gb/s fixed modulation rate
per wavelength is assumed and we consider 20 wavelength
channels. Data messages in the ENoC are routed using XY
dimension-order routing. Of course, different routing algo-
rithms can be used, but, since our goal is to investigate the
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
1411
TABLE 5
Characteristics of the Architectures Analyzed
Approach Path-setup msg #VCs Stand-by
Routing optical paths
Baseline XY 2 NO
Flooding (minimal) 2 NO
Flooding (minimal) 4 NO
Flooding (non-minimal) 4 NO
Proposed Flooding (minimal) 4 YES
control network, this choice provides a simple electronic
infrastructure that is enough to guarantee basic
connectivity.
In order to validate the proposed approach, and to dem-
onstrate the benefits of using a path-setup protocol with
standby, we tested and compared the proposed approach
with a commonly used path-setup algorithm and a few
other alternatives. The comparison is done with both syn-
thetic traffic and real-world applications. Table 5 shows the
characteristics of the five path-setup approaches analyzed.
The baseline approach [5], [16], [17], [24] uses the protocol
introduced in Section 2.2 and XY dimension-order routing
in the control network. Differently, the proposed approach
exploits the path-setup protocol with standby and the mini-
mal flooding routing algorithm. In addition, four virtual
channels are used in order to enhance the routing adap-
tiveness, allowing messages to cross a single forbidden turn
as explained in Section 4. In order to extend our compari-
sons, three other approaches, exploiting the baseline proto-
col, were implemented. Arch-A and Arch-B are equipped
with minimal flooding routing while in Arch-B a single for-
bidden turn is allowed thanks to four virtual channels.
Arch-C is also configured with four virtual channels and, in
addition, it can take advantage of non-minimal paths with a
single non-minimal hop in each path. Non-minimal path
flooding removes the closeness restriction by allowing for
each path a fixed number of non-minimal hops.
5.2 Comparison Under Synthetic Traffic
We first carried out a set of experiments to evaluate the ben-
efits of using the proposed path-setup approach presented
in this article and to analyze its performance and energy
consumption. The main simulation parameters are summa-
rized in Table 6.
Non-bursty traffic is injected at a rate of 0.2 flits/cycle/
NI in order to keep the ENoC busy. All the considerations
in the rest of this section concern burst and control traffic.
Bursty traffic is described by three parameters: the burst
1 1Þ
, message size, the burst injection rate, and the percentage of
nodes that are involved in a burst transfer as sources or des-
tinations. These nodes are randomly selected and during
the simulation are changed in order to provide a scenario
independent of the possible mapping choices. Each node is
characterized by a low, medium, or high value. The values
are chosen so as to have light traffic when all the values are
low and an extremely challenging traffic when all the values
are high. Other traffic behaviours are achieved with a differ-
ent combination of these values. This traffic characterization
is suitable for describing scientific or multimedia applica-
tions where communication patterns are far from full
1412 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,
TABLE 6
Synthetic Traffic Parameters
Parameter
Message Size (Bytes)
Injection Rate (flits/cycle/NI)
Burst Message Size (Bytes) Small
Medium
Large
Burst Injection Rate Low
(flits/cycle/NI) Medium
High
Percentage of source and Low 20% src 20% dst
destination nodes of bursty traffic Medium 40% src 40% dst
High 40% src 60% dst
connectivity [42]. In fact, applications that scale most effi-
ciently to large numbers of IP cores tend to depend on
point-to-point communication patterns where the average
topological degree of communication (TDC)2 ranges
between three and seven distinct destinations [43]. Notice
that this kind of applications may need to handle heavy traf-
fic with large-size messages. For instance, the H:264 video
decoder and the sparse matrix solver applications, analyzed
in the next section, are characterized by an average message
size of 1,280 and 820 bytes, respectively.
Simulations are performed for each combination of the
three values characterizing the bursty traffic. Fig. 5 shows the
average latency for each architecture when varying the simu-
lation parameters values. The latency is divided into three
parts: the latency due to electronic and optical data transfers
and the latency due to the path-setup overhead. It is evaluated
as the average of the latencies of all the burst messages that
reach their destination via the ENoC as well as the ONoC.
The baseline architecture and Arch-C perform ade-
quately for low values of the burst parameters, but as the
burst size, injection rate, and percentage of nodes are
increased, the latency grows faster compared to the other
architectures. In the worst cases, when the burst size and
injection rate are high, the network gets congested. The
main drawback of non-minimal flooding is that routing the
messages to all the router output ports results in reserving a
prohibitive amount of resources. As a consequence, the
number of path setups that can be carried out simulta-
neously decreases drastically, leading to a higher number of
conflicts and hence a higher path-setup overhead. However,
compared to the baseline approach, Arch-C performs better
due to the higher probability of finding free paths. Differ-
ently, Arch-A and Arch-B perform between 30 and 60 per-
cent better than the baseline solution, depending on the size
of the messages and the injection rate. Basically, the gap
increases as the two parameters grow. The ability of safely
crossing forbidden turns in Arch-B, due to the four virtual
channels, leads to an average latency reduction of about
27 percent with a maximum of 60 percent for the high burst
size. All these architectures perform poorly in case of a high
number of nodes sending large-sized messages with a high
injection rate. When these values are high, the network gets
2. The Topological Degree of Communication is defined as the
number of destinations that a given processing element must reach.
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
VOL. 28, NO. 5, MAY 2017
congested. On the contrary, the proposed approach per-
forms better than all the other approaches by exploiting the
path-setup protocol with standby state: the network is never
Value
congested and only in case of all the burst values being
8 high, the latency increases significantly. In other cases,
0.2
when Arch-B incurs high latencies, the proposed approach
256 is able to reach a reduction up to 50 percent.
1,024
Fig. 6 shows the average energy per bit consumption
4,096
for the burst messages. The energy is divided into three
1=300
parts: the energy for sending messages via ENoC, via
1=250
1=200 ONoC, and the energy due to the path-setup overhead.
The ONoC energy includes the energy due to the E/O/E
interface. The energy due to the path-setup is the main
component in case of small and medium size messages:
more than 95 percent of the whole energy in case of small
size messages and more than 80 percent for medium size
messages using the baseline architecture. When the burst
size is large and the injection rate high, a number of mes-
sages are sent via the ENoC due to the congestion on the
control network. This causes an increment of the ENoC
energy that can become the main component even in case
of few messages. The general behavior is that Arch-C per-
forms very poorly due to the large amount of flooding
messages being sent, followed by Arch-B and Arch-A.
The more messages are sent during the path-setup phase,
the more energy is consumed. The baseline consumes less
energy since no message replication is performed. This
trend does not occur in case of large messages, as the
path-setup overhead is independent of the size of the
message and hence it tends to be negligible. Concerning
the baseline and the Arch-C architecture, a high injection
rate leads to a congestion on the control network and
hence to a large increase in the energy consumed by the
ENoC. This holds true for all architectures but Arch-B is
able to route less messages in the ENoC. Different consid-
erations apply to the proposed approach. Its ability to put
an optical circuit in a standby state allows reducing the
number of floodings and hence power consumption.
Compared to the baseline architecture, which does not
perform floodings, it reaches the same values for small-
size messages, while for larger messages it achieves a
reduction greater than a 50 percent.
5.3 Comparison Under Real Applications
In addition to the comparison under synthetic traffic, the dif-
ferent approaches were compared under a real scenario. We
considered four real applications, that are included in the
MCSL realistic NoC traffic benchmark suite [44], namely,
FPPPP, a chemical program performing multi-electron
integral derivatives; H264-720p_dec, a H.264 video decoder
with a resolution of 720p; H264-1080p_dec, a H.264 video
decoder with a resolution of 1080p; and SPARSE, a random
sparse matrix solver for electronic circuit simulations. Their
characteristics are summarized in Table 7. We chose applica-
tions with various characteristics to evaluate the effective-
ness and the scalability of the proposed method for different
traffic scenarios. Note that the MCSL suite provides, for each
application, recorded traffic patterns with detailed commu-
nication traces including the memory space allocation, the
tasks mapping and scheduling.
FUSELLA ET AL.: PATH SETUP FOR HYBRID NOC ARCHITECTURES EXPLOITING FLOODING AND STANDBY 1413

Fig. 5. A comparison between the different path-setup architectures in terms of latency under synthetic traffic.

Fig. 7 shows the average latency and energy consump-
tion for each path-setup approach when implementing the
different applications in a 8  8 mesh topology. As in the
previous section, the figure displays the breakdown of the
latency and energy consumption in terms of path-setup
overhead as well as electronic and optical transmission. In
general, the traffic pattern is less challenging than the
worst-case synthetic traffic and hence the network never
saturates.
Concerning the latency, the baseline architecture per-
forms worse than the other architectures. Arch-C performs
better than the baseline solution, achieving an average
reduction of around 20 percent. This is due the flooding
algorithm with non-minimal paths which increases the
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
probability of finding free paths. A better reduction is pro-
vided by Arch-A and Arch-B, able to reach a reduction of
more than 40 percent compared to the baseline solution.
Notice that the two virtual channels owned by Arch-B give
a slight advantage over Arch-A. Last, the proposed
approach achieves a 45 percent improvement over the base-
line solution while, compared to Arch-A and Arch-B, it
incurs a negligible 3 percent penalty in terms of latency
caused by the protocol with standby.
Differently, concerning the energy consumption, Arch-C
performs poorly due to the non-minimal behaviour of its
flooding routing algorithm: the energy consumption is
around the 50 percent higher than the baseline architecture.
Arch-A consumes an average 30 percent less energy than
1414 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 28, NO. 5, MAY 2017

Fig. 6. A comparison between the different path-setup architectures in terms of energy per bit consumption under synthetic traffic.

the baseline solution, while Arch-B consumes an average 12 architectures. This advantage roughly accounts for a 80 per-
percent less energy than Arch-A. Last, the proposed cent reduction compared to the baseline architecture. The
approach consumes less energy compared to all the other results above point out that the architecture exploiting the
path-setup protocol with standby is highly energy efficient,
TABLE 7 while the latency overhead is negligible.
Characteristics of the Applications Analyzed
6 CONCLUSION
Application #Tasks #Comm Avg msg Avg msg TDC This paper shown the importance of using a suitable
links size (Byte) injection rate
path-setup strategy to enable efficient use of the photonic
FPPPP 334 1,145 204.8 0.00064 0.17 resources in hybrid photonic-electronic networks. We
H264-720p_dec 2,311 13,461 1,280 0.000018 0.10
proposed a new path-setup protocol that can reduce the
H264-1080p_dec 5,191 7,781 1,280 0.000018 0.19
SPARSE 96 67 819.2 0.00031 0.02
path-setup latency and the power consumption due to
its ability to put allocated circuits on a stand-by state. In
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
FUSELLA ET AL.: PATH SETUP FOR HYBRID NOC ARCHITECTURES EXPLOITING FLOODING AND STANDBY
Fig. 7. A comparison between the different path-setup architectures in
terms of latency and energy consumption under real applications traffic.
addition, path-setup messages are sent using a flooding
routing strategy in order to enhance the probability of
finding free optical paths. We analyzed the performance
and energy consumption of the proposed path-setup
strategy and compared it with a commonly used
approach as well as some alternative solutions. The
results show that, as the message sizes and the injection
rates increase, the number of conflicts incurred while
establishing the circuit grows significantly, leading to
performance and energy efficiency degradation. In this
scenario, the proposed approach turned out to reduce
the latency up to 45 percent and the power consumption
up to a 80 percent.
REFERENCES
[1] N. Klrman, et al., “On-chip optical technology in future bus-based
multicore designs,” IEEE Micro, vol. 27, no. 1, pp. 56–66, Jan. 2007.
[2] A. Cilardo and E. Fusella, “Design automation for application-
specific on-chip interconnects: A survey,” VLSI J. Integr., vol. 52,
pp. 102–121, 2016.
[3] E. Fusella and A. Cilardo, “Lighting up on-chip communications
with photonics: Design tradeoffs for optical NoC architectures,”
IEEE Circuits Syst. Mag., vol. 16, no. 3, pp. 4–14, Jul.–Sep. 2016.
[4] A. Biberman, et al., “Broadband silicon photonic electrooptic
switch for photonic interconnection networks,” IEEE Photon. Tech-
nol. Lett., vol. 23, no. 8, pp. 504–506, Apr. 2011.
[5] A. Shacham, K. Bergman, and L. P. Carloni, “Photonic networks-
on-chip for future generations of chip multiprocessors,” IEEE
Trans. Comput., vol. 57, no. 9, pp. 1246–1260, Sep. 2008.
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
1415
[6] E. Fusella, J. Flich, A. Cilardo, and A. Mazzeo, “On the design of a
path-setup architecture for exploiting hybrid photonic-electronic
NoCs,” in Proc. Workshop Exploiting Silicon Photon. Energy-Efficient
High Performance Comput., 2015, pp. 9–16.
[7] R. Ramaswami, K. Sivarajan, and G. Sasaki, Optical Networks: A Prac-
tical Perspective. San Mateo, CA, USA: Morgan Kaufmann, 2009.
[8] P. Kapur and K. C. Saraswat, “Comparisons between electrical
and optical interconnects for on-chip signaling,” in Proc. IEEE Int.
Interconnect Technol. Conf., 2002, pp. 89–91.
[9] M. Haurylau, et al., “On-chip optical interconnect RoadMap:
Challenges and critical directions,” IEEE J. Select. Topics Quantum
Electron., vol. 12, no. 6, pp. 1699–1705, Nov./Dec. 2006.
[10] G. Chen, et al., “Predictions of CMOS compatible on-chip optical
interconnect,” VLSI J. Integr., vol. 40, no. 4, pp. 434–446, 2007.
[11] K.-H. Koo, H. Cho, P. Kapur, and K. C. Saraswat, “Performance
comparisons between carbon nanotubes, optical, and cu for future
high-performance on-chip interconnect applications,” IEEE Trans.
Electron Devices, vol. 54, no. 12, pp. 3206–3215, Dec. 2007.
[12] L. P. Carloni, P. Pande, and Y. Xie, “Networks-on-chip in emerg-
ing interconnect paradigms: Advantages and challenges,” in Proc.
3rd ACM/IEEE Int. Symp. Netw.-on-Chip, 2009, pp. 93–102.
[13] B. A. Small, B. G. Lee, K. Bergman, Q. Xu, and M. Lipson,
“Multiple-wavelength integrated photonic networks based on
microring resonator devices,” J. Opt. Netw., vol. 6, no. 2, pp. 112–
120, 2007.
[14] D. Vantrease, et al., “Corona: System implications of emerging
nanophotonic technology,” in Proc. ACM SIGARCH Comput.
Archit. News, 2008, vol. 36, no. 3, pp. 153–164.
[15] Y. Pan, P. Kumar, J. Kim, G. Memik, Y. Zhang, and A. Choudhary,
“Firefly: Illuminating future network-on-chip with nano-
photonics,” in Proc. ACM SIGARCH Comput. Archit. News, 2009,
vol. 37, no. 3, pp. 429–440.
[16] K. H. Mo, Y. Ye, X. Wu, W. Zhang, W. Liu, and J. Xu, “A hierarchi-
cal hybrid optical-electronic network-on-chip,” in Proc. IEEE Com-
put. Soc. Annu. Symp. VLSI, 2010, pp. 327–332.
[17] Y. Ye, J. Xu, X. Wu, W. Zhang, W. Liu, and M. Nikdast, “A torus-
based hierarchical optical-electronic network-on-chip for multi-
processor system-on-chip,” ACM J. Emerging Technol. Comput.
Syst., vol. 8, no. 1, 2012, Art. no. 5.
[18] S. Bahirat and S. Pasricha, “Meteor: Hybrid photonic ring-mesh
network-on-chip for multicore architectures,” ACM Trans. Embed-
ded Comput. Syst., vol. 13, no. 3s, 2014, Art. no. 116.
[19] S. L. Beux, et al., “Chameleon: Channel efficient optical network-
on-chip,” in Proc. IEEE Des. Autom. Test Europe Conf. Exhibition,
2014, pp. 1–6.
[20] E. Fusella and A. Cilardo, “H2 ONoC: A hybrid optical-electronic
NoC based on hybrid topology,” IEEE Trans. Very Large Scale Integr.
Syst., vol. 99, pp. 1–14, 2016, Doi: 10.1109/TVLSI.2016.2581486.
[21] J. Chan, G. Hendry, A. Biberman, and K. Bergman, “Architectural
exploration of chip-scale photonic interconnection network
designs using physical-layer analysis,” J. Lightwave Technol.,
vol. 28, no. 9, pp. 1305–1315, 2010.
[22] E. Fusella and A. Cilardo, “PhoNoCMap: An application mapping
tool for photonic networks-on-chip,” in Proc. IEEE Des. Autom.
Test Europe Conf. Exhibition, 2016, pp. 289–292.
[23] K. Goossens, J. Dielissen, and A. Radulescu, “Æthereal network
on chip: Concepts, architectures, and implementations,” IEEE
Des. Test Comput., vol. 22, no. 5, pp. 414–421, Sep./Oct. 2005.
[24] M. Petracca, B. G. Lee, K. Bergman, and L. P. Carloni, “Photonic
NoCs: System-level design exploration,” IEEE Micro, vol. 29,
no. 4, pp. 74–85, Jul./Aug. 2009.
[25] S. Pasricha and S. Bahirat, “Opal: A multi-layer hybrid photonic
NoC for 3D ICS,” in Proc. 16th Asia South Pacific Des. Autom. Conf.,
2011, pp. 345–350.
[26] P. Grani and S. Bartolini, “Simultaneous optical path-setup for
reconfigurable photonic networks in tiled CMPS,” in Proc. IEEE
Int. Conf. High Performance Comput. Commun., IEEE 11th Int. Conf.
Embedded Softw. Syst., IEEE 6th Int. Symp. Cyberspace Safety Secur.,
2014, pp. 482–485.
[27] H. Gu, K. H. Mo, J. Xu, and W. Zhang, “A low-power low-cost
optical router for optical networks-on-chip in multiprocessor sys-
tems-on-chip,” in Proc. IEEE Comput. Soc. Annu. Symp. VLSI, 2009,
pp. 19–24.
[28] B. G. Lee, A. Biberman, P. Dong, M. Lipson, and K. Bergman, “All-
optical comb switch for multiwavelength message routing in sili-
con photonic networks,” IEEE Photon. Technol. Lett., vol. 20, no. 10,
pp. 767–769, May2008.
1416 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS,
[29] J. Poulton, et al., “A 14-mw 6.25-gb/s transceiver in 90-nm
CMOS,” IEEE J. Solid-State Circuits, vol. 42, no. 12, pp. 2745–2757,
Dec. 2007.
[30] H. Watanabe, T. Atsushi, K. Takayuki, I. Matsubara, and K. Iwata,
“Vertical cavity surface emitting laser,” U.S. Patent App. 14/447
528, Jul. 30, 2014.
[31] G. Masini, G. Capellini, J. Witzens, and C. Gunn, “A four-channel,
10 Gbps monolithic optical receiver in 130nm CMOS with inte-
grated ge waveguide photodetectors,” in Proc. Conf. Nat. Fiber
Optic Eng. Conf., 2007, Art. no. PDP31.
[32] C. Kromer, et al., “A 100-mw 4 10 gb/s transceiver in 80-nm
cmos for high-density optical interconnects,” IEEE J. Solid-State
Circuits, vol. 40, no. 12, pp. 2667–2679, Dec. 2005.
[33] C. Batten, et al., “Building manycore processor-to-dram networks
with monolithic silicon photonics,” in Proc. 16th IEEE Symp. High
Performance Interconnects, 2008, pp. 21–30.
[34] F. Xia, L. Sekaric, and Y. Vlasov, “Ultracompact optical buffers on
a silicon chip,” Nature Photon., vol. 1, pp. 65–71, 2007.
[35] W. Bogaerts, P. Dumon, D. V. Thourhout, and R. Baets, “Low-loss,
low-cross-talk crossings for silicon-on-insulator nanophotonic
waveguides,” Opt. Lett., vol. 32, no. 19, pp. 2801–2803, 2007.
[36] D. Nikolova, et al., “Scaling silicon photonic switch fabrics for
data center interconnection networks,” Opt. Express, vol. 23, no. 2,
pp. 1159–1175, 2015.
[37] N. Eisley and L.-S. Peh, “High-level power analysis for on-chip
networks,” in Proc. Int. Conf. Compilers Archit. Synthesis Embedded
Syst., 2004, pp. 104–115.
[38] F. Xia, M. Rooks, L. Sekaric, and Y. Vlasov, “Ultra-compact high
order ring resonator filters using submicron silicon photonic wires
for on-chip optical interconnects,” Opt. Express, vol. 15, no. 19,
pp. 11 934–11 941, 2007.
[39] W. J. Dally and C. L. Seitz, “Deadlock-free message routing in
multiprocessor interconnection networks,” IEEE Trans. Comput.,
vol. 100, no. 5, pp. 547–553, May 1987.
[40] G.-M. Chiu, “The odd-even turn model for adaptive routing,”
IEEE Trans. Parallel Distrib. Syst., vol. 11, no. 7, pp. 729–738, Jul.
2000.
[41] J. Chan, G. Hendry, K. Bergman, and L. P. Carloni, “Physical-layer
modeling and system-level design of chip-scale photonic intercon-
nection networks,” IEEE Trans. Comput.-Aided Des. Integr. Circuits
Syst., vol. 30, no. 10, pp. 1507–1520, Oct. 2011.
[42] S. Kamil, L. Oliker, A. Pinar, and J. Shalf, “Communication
requirements and interconnect optimization for high-end scien-
tific applications,” IEEE Trans. Parallel Distrib. Syst., vol. 21, no. 2,
pp. 188–202, Feb. 2010.
[43] J. S. Vetter and F. Mueller, “Communication characteristics of
large-scale scientific applications for contemporary cluster
architectures,” J. Parallel Distrib. Comput., vol. 63, no. 9, pp. 853–
865, 2003.
[44] W. Liu, et al., “A NoC traffic suite based on real applications,” in
Proc. IEEE Comput. Soc. Annu. Symp. VLSI, 2011, pp. 66–71.
Authorized licensed use limited to: Georgia Institute of Technology. Downloaded on September 03,2023 at 07:30:41 UTC from IEEE Xplore. Restrictions apply.
VOL. 28, NO. 5, MAY 2017
Edoardo Fusella received the BS, MS, and PhD
degrees in computer engineering from the Uni-
versity of Naples Federico II, Italy, in 2008, 2011,
and 2015, respectively, where he is currently a
post-doctoral researcher in the Department of
Electrical Engineering and Information Technolo-
gies. He has been a visiting researcher with the
cnica de Vale
Universitat Polite  ncia, Spain and
NEC Europe Ltd., Heidelberg, Germany. His
research interests focus on the domain of many-
core systems, for which he has been investigat-
ing on-chip communication architectures, with emphasis on both elec-
tronic and optical on-chips networks, ranging from design tools to
implementation and physical design.

Flich received the PhD in computer engi-
Jose
neering, in 2001. He is a full professor with UPV,
where he leads the research activities related to
NoCs. He published more than 100 conference
and journal papers, and has served in different
conference program committees (ISCA, PACT,
HPCA, NOCS, ICPP, IPDPS, HiPC, CAC, CASS,
ICPADS, ISCC), as program chair (INA-OCMC,
CAC) and track co-chair (EUROPAR). He has
collaborated with different Institutions (Ferrara,
Naples, Catania, Jonkoping, USC) and compa-
nies (AMD, Intel, Sun). Current research activities focus on routing,
coherency protocols, and congestion management within NoCs. He has
co-invented different routing strategies, reconfiguration and congestion
control mechanisms, some of them with high recognition (RECN and
LBDR for on-chip networks). He is a member of the HiPEAC NoE. He is
coeditor of the book Designing Network-on-Chip Architectures in the
Nanoscale Era. He coordinated the FP7 NaNoC project and the H2020
MANGO project. He is a senior member of the IEEE.
Alessandro Cilardo received the five-year
degree in electronics engineering, magna cum
laude and the PhD degree in computer science
from the University of Naples Federico II, in Janu-
ary 2003 and November 2006, respectively. He is
currently an assistant professor with the Univer-
sity of Naples Federico II. His research activities
focus on electronic design automation, high-per-
formance and embedded computing, as well as
computer arithmetic, with special emphasis on
the application domain of security and cryptogra-
phy-related processing. He is the single or main author of around 70
peer-reviewed papers published in leading scientific journals and confer-
ences. He is a senior member of the Institute of Electrical and Electron-
ics Engineers and a member of the HiPEAC NoE.
" For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/publications/dlib.