Stage-Distributed Time-Division Permutation Routing in a Multistage Optically Interconnected Switching Fabric

Alvaro Cassinelli(1) , Makoto Naruse (2) , Masatoshi Ishikawa(1)

1: University of Tokyo, Dept. Information Physics and Computing, 7-3-1 Hongo Bunkyo-ku, Tokyo 113-0033, Japan. (alvaro@k2.t.u-tokyo.ac.jp) ECOC 2003 /
2: Communications Research Laboratory, 4-2-1 Nukui-kita, Koganei, Tokyo 184-8795, Japan. We4.P.137

Introduction (A) Plane-to-plane guide-wave based optical interconnections (C) Column-controlled buffered MIN architecture
Bandwidth, cross-talk and volume consumption Fiber-based Modules vs. Free-Space
σ Routing Strategy
Exchange switch Shuffle inte rconnection
considerations, all advocate for plane-to-plane optical σ (4) E(1) σ (4) E(1) σ (4) E(1) σ(4) E(1) (2)

interconnects within massively interconnected electronic Multistage architecture 0000

0001
0010
0011
0100
0000
0001
0010
0011
0100
• better efficiency (than holograms) for long-range interconnections.
system as a promising substitute to conventional electronic 0101 0101

• no cross-talk in 3D, just like free-space optics. Conflicts are not resolved individually at the switch level, but globally at the stage level by a “tournament” between all the incoming

Output
0110 0110

Input
parallel computers
0111 0111

circuitry in the near future.

1000

requests to that particular stage. Provided that these request are uniformly directed to any possible output, "votes" leading to the adoption of one of
1000

+
1001 1001
1010 1010
1011 1011

• No space-invariance required. the two possible states of the global-switch will be evenly distributed. Such behaviour takes place for all stages of the network, so that at each
1100 1100

Ω network
1101 1101
1110 1110

switching networks
1111 1111

Multistage Interconnection Networks (MINs) • Theoretically more volume efficient than free-space stage, half of the requests will be dropped and half will be able to pass to the next stage. This means there is an enormous number of discarded
are capable of supporting arbitrary input-output interconnection packets, certainly much bigger than that occurring by internal blocking in a standard SEMIN. However, if one considers a buffered architecture, then
interconnections, and have been long proposed as an Dense optical interconnect: modules • Precise and robust alignment possible… presumably there will be no need to provide it with a large buffer memory, because the packets that have been retained in the buffers are very likely
interesting alternative to the full crossbar and the time- …an optical “3D optical to go forward in the following tournament.
• multiple interleaved permutations possible.
shared bus (in terms of performance, cost and scalability) interconnection folded in 2D wiring” module between
for use as permutation networks for multiprocessors but 2D VLSI arrays. • Maybe “hard” to build? Boring, but not a fundamentally difficult
also as point-to-point switching fabrics for telecom (can be automated, can be done by “layers”, see below). module control
applications. Processor arbitration
arrays • Alignment of both output and input needed.
As a consequence, multistage architectures Optical Multistage • Power dissipation may be a fundamental limitation, but we are far
Buffer 1
…

Bi-permutation
Architecture
…
using optical plane-to-plane regular interconnections from these limits.

module
…
may well represent a theoretically optimal architecture
for use within a massively interconnected systems. Paradigm

…
Buffer N
…

depth of analysis

The work presented here develops on some fundamental e

pl an Stacking layers Stacking planar lightwave circuits to produce 2D Length of transferred
packets/cycle
aspects of such optical multistage architecture: ut 0
guided-wave interconnection modules
p 4 Total length of buffer
in 8 1
5
2 In the case of “column/row-decomposable” processing bi-permutation
(A) Guide-wave based interconnection modules 6
3 permutations (see left), which happen to be (analysis/buffer) module
e
l an the ones required in most regular
While many demonstrator have been built to illustrate the advantages of free-space optics over electronics for dense plane-to-plane tp Routing parameters
interconnections, there has been relatively little research on the use of two-dimensional wave-guide-based interconnections, (or guide- tp
u interconnected MINs, 2D interconnection
(a) ou (b) modules can be easily implemented by
wave based interconnection modules). Yet, these can easily achieve better transmission efficiency than holographic-based interconnections
while almost completely cancelling cross-talk, and contrary to the common belief they may be more volume efficient than free-space optics for stacking layers of printed lightwave circuits
both space-invariant and space-variant interconnects. “Brute” folding and row/column decomposed folding of the perfect containing 1D permutations (see right).
shuffle inter-stage permutation

(B) Transparent column-controlled Optical Multistage Network Performance analysis of a buffered GSMIN
The use of “modular” inter-stage interconnections leads naturally to consider a new paradigm for the optical MIN architecture: interconnection
Experimental results Input pattern (exit VCSEL array) Figures represent performance (normalized amount of satisfied requests) as a function of the input load (computed as the probability of a
modules containing several interleaved global interconnections (i.e. permutations). Addressing of the required permutation can be done by input
output
mechanical displacement of the whole multi-permutation module. request being issued at any input per unit time) for the MIN with stage-distributed switching, and for a standard Shuffle Exchange MIN with
Alignments independent control of switches (both 128x128 large networks). The input traffic follows the uniform request model.
Cascading such modules without intermediate optoelectronic arrays gives a transparent ”globally-stage switched” multistage Recently, we tested this approach by implementing hardware
several 4x4 fiber-based modules each integrating a Output pattern (CCD image)
interconnection network (GSMIN) that can be used as a circuit-switched permutation network for multiprocessor communications (this
architecture represents an original optical implementation of a column-controlled shuffle-exchange MIN). different fixed topology.
“Fair” selection mode (the state of the global-switch is chosen randomly if the poll is a draw)
(C) Buffered column-controlled Optical Multistage Network Automatic alignment of modules, both dynamically Optimal situation: both “depth

input
and statically (pre-aligned ”plug-and-play” of analysis” and “depth of

(VCSELs)
1 1 66

Output
transfer” are equal to the

(CCD)
5
exchangeable blocks), is a critical issue now being

Input

Probability of packet acceptance

Probability of packet acceptance
Column-controlled unbuffered MINs have been extensively studied for circuit-switched applications. We found that an inter-stage buffered Only the older 0.9 0.9
studied.
cros
sba
cros
sb 5 actual buffer size.
column-controlled MIN architecture may also represent an interesting alternative to the well-known Shuffle-Exchange MINs (SEMINs) for point- packets waiting on 0.8 r 0.8 ar 4
4
the queues are
to-point packet-routed communications, both from the point of view of its implementation complexity and simplicity of routing protocol. 0.7 0.7
The performance of a "fair"
given attention in SE SE
3
0.6
MIN 0.6
MIN operated GSMIN is superior to

Buffer size
the selection 3
0.5 0.5 that of a standard ("fair"

Buffer size
phase, and that GS GS 2
2
operated) MIN for buffer sizes
0.4 MIN 0.4 MIN
only these packets 1
0.3 0.3 0 larger than one.
are candidates to Depth Analysis = 1 1
0.2 Depth Analysis = Buffer Size
be transferred 0.2
Depth Transfer = 1 Depth Transfer = Buffer Size Also, for a buffer size equal to
during the transfer 0.1 0.1
0 three, the GSMIN network
phase.

(B) Transparent column-controlled Multistage Network

0 0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
gives better performance than
Input request probability per unit time (λ )
Mechanically Reconfigurable wave-guide-based interconnections
Input request probability per unit time (λ )
the crossbar, even at full-load.

“Alternate” selection mode (the state of the global-switch changes if the poll is even)
Why common control of switches per switching stage (column-control)? When a standard SEMIN and a column-control
Multi-permutation interconnection module paradigm 1
6
5 5
6
SEMIN (GSMIN) are operated in forced alternate

Probability of packet acceptance

Simplicity and Scalability: A whole switching stage needs a unique control signal. A clear improvement in performance is
0.9
cros 4 mode, they become strictly equivalent architectures.
sba 4
seen in the standard SEMIN for buffer 0.8 r Therefore, when a standard SEMIN is operated in
actuators outputs sizes larger than one. 0.7
3
forced alternate mode, its performance roughly

Buffer size
3
degrades to that of an alternate operated GSMIN; on
Can have a “direct” optical implementation: the switching stage and its adjacent inter-stage permutation can be physically 0.6 SEM
IN
This performance may exceed that of the 2 the other hand, when a GSMIN is operated in forced
merged into a unique “permutation module" providing two different interconnection patterns (bi-permutation switching module). 0.5
alternate mode, the performance remains
A small mechanical (or optical ) GSMIN, but it seems to increase slowly 0.4 GSM
I
1
2

inputs (with buffer size), so that when the buffer

N substantially the same.
perturbation produces a drastic 0.3 0
size is equal to five (or four in the case of Depth Analysis = Buffer Size 1
change of the interconnection a 64×64 network), both networks show
0.2
Depth Transfer = Buffer Size 1
6
6
pattern from input to output. roughly the same performance.
0.1 5 5

Probability of packet acceptance

0 0.9 4
0 cros
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 sb ar
4

Two implementation of a column-controlled Multistage Interconnection Network 0.8

Input request probability per unit time (λ )
Mechanically reconfigurable multi-stage architecture based on cascading 0.7
3
Guide-wave based multi-permutation multi-permutation modules 3

Buffer size
interconnection module 0.6 SEM
IN 2

Bi-permutation module “Forced-alternate” selection mode (the state of the global-switch changes periodically) 0.5
2
0.4 GSM
(integrates switching and IN 1
Column-controlled permutation stage) 6
0.3 Size: 128 x 128 0
1 1
Shuffle-Exchange stage 5 0.2 SEMIN (alternate)

Probability of packet acceptance

0.9 SEMIN/GSMIN (forced alternate)
cros
Prototype using interleaved fibers Spanned hypercube using Bi-permutation modules 4 0.1
sba There is no significant change in
0.8 r 0
0

0.7 performance between a buffered 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
Input request probability per unit time (λ )

Buffer size
c4 3
GSMIN with "alternate" selection
Non-integrated bi-permutation module: c3 c3
0.6
and a buffered GSMIN with "forced
c4

0.5
c1 2 alternate" selection. A continual
Channels are multi c2 0.4 Individual control of switches as well as arbitration
"blind" alternation of switching
{c2, id} mode fibers: 0.3 Size: 128 x 128 may be unnecessary on a standard SEMIN if the
c2 states performs just as well.
MFD = 9.5 µ m 1
buffer size is larger than four. Also, when the buffer
= input output
Grad diam. 125
0.2 GSMIN (alternate)
GSMIN (forced alternate) size is equal to three, the 128x128 GSMIN already
0.1
µ m c1 0
0 outperforms a 128x128 full-crossbar.
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
NA: 0.1 Input request probability per unit time (λ )
a b c c b a
Experience with two cascaded modules: four-dimensional hypercube …topology mapped on a plane (optical
connected multiprocessor… interconnects, VLSI integration)
Displacement Output
stages (to CCD)
Four-dimensional “spanned”
hypercube network (16 nodes)
“indirect” implementation obtained by electrically joining A more “direct” Implementation of a column-controlled
using four bi-permutation
individual switches (common control lines a, b and c) of a Multistage Network using multi-permutation switching
Conclusions and further research
modules, each providing a cube
Shuffle-Exchange network (here the Inverse Baseline) modules (the GSMIN network)
Input permutation and the identity.
(from VCSEL array)
{c4, id} A total of 24=16 global
Exit first
module
{c3, id} permutations for the whole
{c2, id} Column-control certainly reduces overall interconnection capacity, but if things are properly designed, the MIN architecture can still
{c1, id} network.
accommodate communication primitives of most static interconnection networks. The interest of such an arrangement lies in the ease of control
A column-controlled MIN can be an efficient (circuit-switched) permutation network. and straightforward implementation. We presented here preliminary experimental results demonstrating a simple optical architecture using
cascaded fiber-based bi-permutation modules. An electro-mechanical system has been developed providing stage-switching times on the order of
Input second milliseconds, making this architecture suitable for reconfigurable, high-bandwidth inter-processor communications. Further research on this topic
module include the feasibility study of dense guided-wave multi-permutation modules by stacking layers of planar lightwave circuits, as well as the
introduction of electro-optical switching.
Output
Displacement pitch for commutation: 125 µ m
Input
(VCSEL array)
Alignment tolerance: ± 5 µ m (half peak power).
A multistage ”spanned” version
input
of most direct network Inter-module Coupling Efficiency: 1.7dB (no additional 3D bi-permutation
optics, matching oil or antireflection coating). module built by cr o s s
topologies (hypercube, cube- stacking planar I4
connected-cycles, etc.) can be C2 or Id lightwave circuits straigh
t
implemented as an unbuffered C1 or Id switching

column-controlled MIN ⇒ Validation of simple cascaded architecture region

input L2 E
architecture. 2
Interconnect N

Interconnect N

Interconnect N
Interconnect 2

Interconnect 1

Interconnect 2

Interconnect 1

Interconnect 2
Interconnect 3
Interconnect 1

Interconnect 3

Interconnect 3

A time-division multiplexing
(TDM) technique can be used to output
select the interconnections at Electro-mechanical switching device
each stage. Id . Id Id . C2 C1 . Id E1 . E2
Possible implementation of an electro-optical bi-permutation
This electro-magnetic actuated module using stacked layers and electro-optically controlled
device can position the module normal coupling between the layers.
Permutation time in both X and Y directions.
appearance period
Time slot

From the routing point of view, simulations showed that a buffered column-controlled MIN would not require excessive buffer size to
achieve respectable performances (under uniform traffic). The path-selection mechanism can be further reduced to the simple alternation of the
The switching speed seems to be limited to the millisecond available stage permutations, without degrading the performance. Under such stage-distributed time-division permutation multiplexing, the SEMIN
range (resonant frequency). and GSMIN fabrics become strictly equivalent routing architectures; hence (provided that buffer size is chosen to be larger than three) this
A column-controlled MIN can emulate a full-crossbar by time-multiplexing analysis-free strategy will provide a very simple arbitration mechanism for standard SEMIN networks. This is an interesting result on its own.
First two modules of a spanned version of a 4-dimensional hypercube were
Micro electro-mechanical actuators (MEMS) may also be an
fabricated using interleaved optical fibers, and the resulting four possible
Despite the reduced permutation capacity of a column-controlled MIN, a permutation network can emulate (by time-multiplexing) a full- interesting alternative when switching latency in the millisecond
interconnections observed.
crossbar provided that the reconfiguration latency remains small with respect to the interconnection request rate, and that the bandwidth is range is tolerable. This arbitration free mechanism may be very appealing for all-optical networks, if optical buffering functions (e.g. delay lines) could be
sufficiently high to accommodate in a single time slot all the data to be transferred. An unbuffered column-controlled MIN may provide enough integrated on the cascaded modules themselves, an issue worth exploring.
optical bandwidth - but the reconfiguration time may be still too slow (using thermo-optical switches for instance).