Custom VLSI Implementations of Neural Networks: Early Attempts

Custom VLSI implementations of neural networks
Early attempts
The idea of making custom analog VLSI implementations of neural networks
dates back to the late 80s - early 90s:
Analog VLSI circuits

for spiking neural networks
[Holler et al. 1989, Satyanarayana et al. 1992, Hammerstrom 1993, Vittoz 1996]
Giacomo Indiveri
Neuromorphic Cognitive Systems group
Institute of Neuroinformatics
University of Zurich and ETH Zurich
General purpose computing
Competing with Intel steamroller
Full-custom analog
implementation
Communication - bandwidth
limited
Neural network accelerator

PC-boards
Difficult to program
Current research
September 17, 2009
Technological progress
Application-specific focus
Power-dissipation/computational
power
G.Indiveri (NCS @ INI)
ICANN09 Tutorial
1 / 78
Silicon neuron designs
Embedded system integration
ICANN09 Tutorial
3 / 78
Silicon neural network characteristics
Many VLSI models of spiking neurons have been developed in the past, and
many are still being actively investigated:
ICANN09 Tutorial
Most designs can be traced back to one of two types of silicon neurons,
4 / 78
Above threshold (strong inversion)
Below threshold (weak inversion)
Mixed analog/digital
Fully analog
Rate-based
Spiking
Real-time
Accelerated-time
Conductance-based
Integrate-and-Fire
Large-scale, event-based
networks
Small-scale, hard-wired
ICANN09 Tutorial
5 / 78
Why subthreshold neuromorphic VLSI
MOSFETs in subthreshold
Vd
Exploit the physics of silicon to reproduce the

bio-physics of neural systems.
Ids
Vg
n-FET subthreshold transfer function
Ids = I0 en Vg /UT eVs /UT eVd /UT
Vs
where
I0 denotes the nFET current-scaling parameter
n denotes the nFET subthreshold slope factor

UT the thermal voltage
Vg the gate voltage, Vs the source voltage, and Vd the drain voltage.
The current is defined to be positive if it flows from the drain to the source
ICANN09 Tutorial
6 / 78
Diffusion and saturation
ICANN09 Tutorial
9 / 78
Exponential voltage dependence

Subthreshold n-FET
Vg
Ids = I0 en Vg /UT eVs /UT eVd /UT
Vd
Ir
-3
-4
10
V
Vs
Ir
Qs
Ids =
Vd
Ids =
If
Qd
subthreshold
10
is equivalent to:
If
Vg
-2
10
I0 e
Ug UVs
T
If
I0 e
-5
U g Ud
T
10
T
-6
10
Ir
If Vds > 4UT the Ir term becomes negligible,

and the transistor is said to operate in the
saturation regime:
Ids (A)
Vs
above threshold
-7
10
-8
10
-9
10
-10
10
-11
10
Ids = I0 en Vg /UT Vs /UT
-12
10
0.5
1.5
2.5
3.5
4.5
Vgs (V)
ICANN09 Tutorial
10 / 78
ICANN09 Tutorial
11 / 78
n-FETs and p-FETs
One, two, and three transistor circuits
In Complementary Metal-Oxide Semiconductor (CMOS) technology, there are

two types of MOSFETs: n-FETs and p-FETs
Vd
Ideal current source
Vs
Vg
Vg
I out
I in
Vin
Vb
Current-mirror
I out
M1
M2
Vd
Vd
Vs
Inverting amplifier
In traditional CMOS circuits, all n-FETs have the common bulk potential (Vb )
connected to Ground (Gnd), and all p-FETs have a common bulk potential
(typically) connected to the power supply rail (Vdd ).
The corresponding (complementary) equation for the p-FET is
Differential pair
I1
V1
Vin
I2
M2
Vout
Vbn
Ids = I0 e
p (Vdd Vg )/UT
(Vdd Vs )/UT
(Vdd Vd )/UT
I2 = I0 e
ICANN09 Tutorial
12 / 78
I1
V1 Vs
V1
V2 Vs
M2
UT
Ib
Vbn
Vs
V2
M3
Vdd
Vdd
M4
M5
M1
V1
V2
x 10
e UT + e UT
V1
I2
M1
I1
M2
I1, I2 (A)
V1
e UT
V2
V1
UT
+e
V2
UT
V2
2UT
ICANN09 Tutorial
0.2
0.1
0
V1V2 (V)
0.1
0.2
(V1 V2 )
Ib
2UT
is a tunable conductance.
0.3
15 / 78
where
0.4
0
0.3
e UT + e UT
Iout gm (V1 V2 )
0.2
e UT
V1
M3
Iout V
out
gm =
V1
I2 = Ib
Vb
In the linear region (|V1 V2 | < 200mV ):
Ib
0.6
V1
Iout = Ib tanh
Vout
I2
Vs
V2
0.8
I1 = Ib
14 / 78
Vb
Iout
I1
I0
ICANN09 Tutorial
I2
UT
Vb
UVs
M1
The transconductance amplifier
Ib = I1 + I2 = I0 e UT
e
V2
The differential-pair
I1 = I0 e
M3
Vs
ICANN09 Tutorial
16 / 78
What is a synapse?
Synapses in the nervous system
In 1897 Charles Sherrington

introduced the term synapse to
describe the specialized
structure at the zone of contact
between neurons as the point
in which one neuron
communicates with another.
Eex (Na+, ...)

Glutammate
Electrical | Chemical
Excitatory | Inhibitory
Vmem
Depressing | Facilitating
Gl
Cmem
AMPA | NMDA
GABA
...
Einh (K+, Cl, ...)

2005 winner of the Science and Engineering
Visualization Challenge.
by G. Johnson, Medical Media, Boulder, CO.
ICANN09 Tutorial
19 / 78
Synaptic transmission
ICANN09 Tutorial
20 / 78
EPSC and EPSP

730
O. SACCHI, M. L. ROSSI, R. CANELLA, AND R. FESCE
In chemical synapses the presynaptic and

postsynaptic membranes are spearated by
extracellular space.
The arrival of a presynaptic action potential
triggers the release of neurotransmitter in the
extracellular space.
The neurotransmitters react with the
postsynaptic receptors and depolarize the cell.
FIG .
2.
Typical synaptic currents at ganglionic synapse evoked by supra-
low the linear fit. This might be partly because of inward

rectification by the ACh-evoked current (Fieber and Adams
1991; Mathie et al. 1990), but it is at least partly accounted
for by the changes in ionic gradients that are associated with
even small ionic flows (especially accumulation of K / ions
in the perineuronal space) (Belluzzi and Sacchi 1990).
In all neurons the decay of EPSC was well fit by a single
exponential function. The average time constant, t, was 7.5
ms at 075 mV (n 14) and was scarcely voltage-dependent:
it decreased exponentially with a constant of 260.4 mV from
0105 to 025 mV (Fig. 3A). EPSC amplitudes decreased in
the average (5 neurons) to 69% of the initial value when
Ca 2/ concentration in the bath was lowered from the usual
5 to 2 mM; the current decay time constants were also somewhat decreased (by 21% in the average). The voltage dependence of the decay time constant and the estimated reversal
potential did not appear to be affected by calcium concentration (Fig. 3B).
maximal preganglionic stimulation.

Superimposed EPSCs recorded in a
Superimposed
excitatory
sympathetic neuron at different membrane potentials between 010 and
NATURE OF THE POSTSYNAPTIC RECEPTOR ACTIVATED BY
0100 mV (in 10 mV steps) under 2-electrode voltage-clamp conditions.
post-synaptic
currents (EPSCs)
Excitatory
post-synaptic
AC . To
correctly model andpotential
reproduce evoked
Cell was held at 050 mV and then repeatedly stepped, at 20 s intervals, to NATURAL
membrane levels at which synaptic current was observed. Note presence synaptic currents, it was important to determine whether or
recorded
in0a
neuron
at different
(EPSP)
in response
multiple
30 mV
tracings. External
Ca concentration was 5 not
of fast I in 010/
conductances
other than thoseto
associated
to the nicotinic
mM.
receptor were activated by nerve-released ACh. In fact, ACh
membrane potentials (from Sacchi et
pre-synaptic
spikes
al.
release
by the presynaptic
nerve(from
terminalsNicholls
is thought toet
gencell was estimated for the rat sympathetic neuron in culture, erate in the ganglion a broad series of side effects, in addition
al.,
1998).
1992).
whereas
the activation-deactivation time constants for putative M- to
the central role of sustaining the fast excitatory transmisH
Chemical synaptic transmission is

characterized by specific temporal dynamics.
2/
Na
currents were 220 ms in the 060/ 030 mV voltage range (Owen

et al. 1990).
ICANN09 Tutorial
21 / 78
sion. Autoreceptors were proposed to control evoked transmitter release (see Starke et al. 1989, for a review). The
evidence for a physiological role in the ganglion seems to
R E S U L G.Indiveri
TS
greatest for the muscarinic autoreceptors, which are pos(NCS @ INI)
ICANN09 be
Tutorial
22 / 78
tulated to mediate a negative feedback loop that reduces
Synaptic current at the sympathetic ganglionic neuron
transmitter mobilization (Koelle 1961; Koketsu and Yamada
Neural network models
VLSI synapses in classical neural networks

The role of the VLSI synapse in implementations of classical neural network
models is that of a multiplier.
Multiplying synaptic circuits have been implemented using a wide range of
analog circuits, ranging from the single MOS-FETs to the Gilbert multiplier.
Iin1
I1
Iin2
I2
In classical neural network theory

signals are (tipically) continuous values that represent the neurons mean
firing rate,
Ib
neurons implement a saturating non-linearity transfer function (S) on the

inputs weighted sum,
Figure: Schematic of half of a Gilbert multiplier. This circuit multiplies Iin1 and Iin2 by Ib
if Iin1 + Iin2 = Ib .
the synapse implements a multiplication between the neurons input

signal (Xi ) and its corresponding synaptic weight (wi ).
ICANN09 Tutorial
VLSI synapses in pulse-based neural networks
IWi
Vi =
Vmem
IWi
Cmem
26 / 78
A linear integrator is a linear low-pass filter. Its impulse response should be a

decaying exponential.
With VLSI and subthreshold MOSFETS its fairly easy to implement
exponential voltage to current conversion, and linear voltage increase or
decrease over time.
Cmem
Vdd
In pulse-based neural networks the weighted contribution of a synapse can be

implemented using a single transistor.
In this case p-FETs implement excitatory synapse, and n-FETs implement
inhibitory synapses.
The synaptic weight can be set by changing the Wi bias voltage or the t
duration.
ICANN09 Tutorial
Linear pulse integrators
Vdd
Wi
24 / 78
ICANN09 Tutorial
27 / 78
Vg
Vdd
Id
Vg(t)
Vc
Id =
d
C dt
Vc
Id (t ) = I0 e UT
(Vdd Vg (t ))
Id(t)
ICANN09 Tutorial
28 / 78
Linear charge-and-discharge integrator
Iw = I0 e
M
Csyn
I
Vw
Vsyn
I = I0 e
Msyn
Mw
Ic = C
Isyn
Iw
Vw
UT
, c ,
(Vdd V )
d
dt
UT
Isyn (t ) =
Csyn UT
Iw = I0 e UT
Csyn UT
, d ,
M
I
I
Vw
(Vdd Vsyn )
Isyn = I0 e
Mpre
Log-domain pulse integrator
Csyn
I = I0 e
Vsyn
Msyn
Mw
UT
Mpre
(t t )
I e+ ci
dt
(charge phase)
syn
ICANN09 Tutorial
29 / 78
The diff-pair integrator (DPI)
d
dt
d
Vsyn
Msyn
Iin
Vthr
Mthr
Isyn
Min
I = I0 e
Ic = C
UT
(Vdd V )
d
dt
(Vdd Vsyn )
Isyn = I0 e
d
dt
d
dt
Isyn + Isyn
+e
UT
Vthr
Iout (ti +t ) =
UT
(Vdd Vsyn )
Isyn =
ICANN09 Tutorial
(Vw Vdd )
UT
30 / 78
Iout
I 1 +
dt
Iout + Iout
Ig
Iin ,
Iout
Ig
Iin
if Iout Ig , Iw I
Isyn
d
dt

ti ti 1
t
t
Ig Iin
1 e + Iout (ti )e , Iout (ti ) = Iout (ti 1 +t )e
I
Response to an arbitrary spike train (t ) = i (t ti ), with t :

Z t
Ig Iin
t
e
e ( )d
Iout (t ) =
I
0
UT
UT
Vsyn
Iw Igain
:
Mean response to spike train of mean frequency

< Iout >=
(Bartolozzi, Indiveri, 2007)
Vsyn
Response to t pulse at time ti :
UT
Vsyn
Iw
Mpre
UT dt
Csyn UT
, Iw0 = I0 e
UT
Iin = Iw
Mw
ICANN09 Tutorial
DPI transfer function:

d
Iout + Iout
dt
Vw
Vsyn
Vw
I0 Iw0
UT
Isyn Iw
Isyn + Isyn =
Isyn + Isyn =
(Vdd Vsyn )
DPI equations
Iw = I0 e
Csyn
(Vdd Vsyn )
Isyn = Isyn
(t t + )
+ di
I e
dt
dt
Isyn = I0 e
(discharge phase)

n
c +d )
c
1
c f ct (
t
d
Isyn (t ) = I0 e
, with f =
, f<
t
c + d t
UT
Isyn
Iw
(Vdd Vsyn )
syn
(Vdd V )
Ic = C
(Vsyn Vw )
31 / 78
Ig Iin
I

t , =
ICANN09 Tutorial
t + ISI
32 / 78
DPI measured response
DPI response to spike-trains
300
200
400
Vw=300mV
Vw=440mV
350
Vw=320mV
Vw=460mV
300
Vw=340mV
EPSC (nA)
150
100
250
200
150
100
50
50
0
0
0.05
0.1
Time (s)
0.15
0
0
0.5
1.5
2
2.5
Time (s)
3.5
100
80
60
40
20
0
ICANN09 Tutorial
33 / 78
Proc. Natl. Acad. Sci. USA 94 (1997)

Short-term depression
50
100
150
Input Frequency (Hz)
200
ICANN09 Tutorial
34 / 78
Short-term depression
Va
Vgain
Va
Vw
M6
M1
M7
0.35
Id
Slow recovery
C2
Vx
0.3
M5
Ir
Vd
Ir
Cd
Vd
M2
C
Vpre
Vd=0.26 V
0.25 Update
Isyn
Vx (V)
EPSC (nA)
250
Vw=420mV
Output Frequency (Hz)
450
l
e
e
f
e
h
M4
Vd=0.28 V
Fast recovery
0.2
Vd=0.3 V
0.15
Vpre
e
n
M3
0.1
0.04
(a)
P,
V =1.30V
w
V =1.33V
w
V =1.35V
120
(C. Rasche, R. Hahnloser, 2001)
ICANN09 Tutorial
35 / 78
0.06
0.08
0.1
0.12
Time (s)
0.14
0.16
(b)
Figure 1: Schematic for a depressing synapse circuit and responses to a regular input spike
(M. Boegerhausen, P. Suter, and S.-C. Liu 2003)
train. (a) Depressing synapse circuit. The voltage @A determines
the synaptic conductance
D

;
C
E

while the synaptic term
or B
is exponential in the voltage, @GF . The subcircuit
DC;E . The presynaptic
5 ( , 5IH , and 5K
J , control
consisting
of transistors,
G.Indiveri
(NCS @ INI)
ICANN09
Tutorial the dynamics of B
36 / 78
5
J
input goes to the gate terminal of
which acts like a switch. When there is a presynaptic
STDP and beyond
Spike-driven learning in VLSI I

Alternative spike-driven learning algorithm
Spike-driven weight change depends on the value of
the post-synaptic neurons membrane potential, and on
its recent spiking activity.
J. Arthur and K. Boahen.

Learning in silicon: Timing is everything.
In Y. Weiss, B. Schlkopf, and J. Platt, editors, Advances in Neural Information Processing Systems 18. MIT Press, Cambridge, MA,
2006.
A. Bofill-i Petit and A. F. Murray.
Synchrony detection and amplification by silicon neurons with STDP synapses.
IEEE Transactions on Neural Networks, 15(5):12961304, September 2004.
Fusi et al. 2000; Brader et al. 2007
E. Chicca, D. Badoni, V. Dante, M. DAndreagiovanni, G. Salina, S. Fusi, and P. Del Giudice.

A VLSI recurrent network of integrateandfire neurons connected by plastic synapses with long term memory.
IEEE Transactions on Neural Networks, 14(5):12971307, September 2003.
Recipe for efficient VLSI implementation

1
bistability: use two synaptic states;
redundancy: implement many synapses that see

the same pre- and post-synaptic activity
P. Hfliger, M. Mahowald, and L. Watts.
stochasticity & inhomogeneity: induce LTP/LTD

only in a subset of stimulated synapses.
ICANN09 Tutorial
G. Indiveri, E. Chicca, and R. Douglas.

A VLSI array of low-power spiking neurons and bistable synapses with spiketiming dependent plasticity.
IEEE Transactions on Neural Networks, 17(1):211221, Jan 2006.
S. Mitra, G. Indiveri, and S. Fusi.
Learning to classify complex patterns using a VLSI network of spiking neurons.
In J.C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems 20, pages 10091016,
Cambridge (MA), 2008. MIT Press.
- Slow learning: only a fraction of the synapses memorize the pattern.

+ The theory is matched to the technology: use binary states, exploit
mismatch and introduce fault tolerance by design.
A spike based learning neuron in analog VLSI.

In M. C. Mozer, M. I. Jordan, and T. Petsche, editors, Advances in neuralinformation processing systems, volume 9, pages 692698.
MIT Press, 1997.
37 / 78
Neurons . . . in a nutshell
ICANN09 Tutorial
38 / 78
ICANN09 Tutorial
42 / 78
Neurons of the world
A quick tutorial
Complexity
Real Neurons
Conductance based models
Integrate and fire models
Rate based models
I
I
Sigmoidal units
Linear threshold units
(adapted from B. Mel, 1994)

ICANN09 Tutorial
40 / 78
Equivalent Circuit
Spike generating mechanism

Eex (Na+, ...)
ENa
Glutammate
gNa
Vmem
Vmem
Gl
Cmem
Einh (K+, Cl, ...)
ICANN09 Tutorial
Cmem
EK
If excitatory input currents are relatively small, the neuron behaves exactly like
a first order low-pass filter.
Gl
gK
GABA
43 / 78
Spike properties
If the membrane voltage increases above a certain threshold, a

spike-generating mechanism is activated and an action potential is initiated.
ICANN09 Tutorial
44 / 78
The F-I curve

Iin=I1
Spike Frequency (F)
Refractory
Period
Refractory Period
Pulse Width
Iin=I2 > I1
Input Current (I)

ICANN09 Tutorial
45 / 78
ICANN09 Tutorial
46 / 78
Hardware implementations of spiking neurons
Conductance-based models of spiking neurons
The first artificial neuron model was proposed in the 1943 by McCulloch and
Pitts. Hardware implementations of this model date almost back to the same
period.
Hardware implementations of spiking neurons are relatively new.
One of the most influential circuits that implements an

integrate and fire (I&F) model of a neuron was the
Axon-Hillock Circuit, proposed by Carver Mead in the late
1980s.
ICANN09 Tutorial
47 / 78
Conductance based Si-Neurons
In 1991 Misha Mahowald and Rodney Douglas proposed a

conductance-based silicon neuron and showed that it had properties
remarkably similar to those of real cortical neurons.
ICANN09 Tutorial
49 / 78
Conductance based Si-Neurons

Silicon neurons measurements
Sodium
Vdd
Vdd
Sodium Current
1V
Vm
+
VNa
Vthr
INaoff
+
GNaoff
[Ca]
INa
INaon
Vthr
ENa
Passive Leak
GNaon
Vm
[Ca]
Vmem
Vdd
Eleak
Vdd
Vmem
Gleak
Passive
+
VK
Potassium
Vthr
Cmem
IK
Vm
IK
GK
[Ca]
Potassium Current
EK
EK
I
50 ms
ICANN09 Tutorial
50 / 78
ICANN09 Tutorial
51 / 78
The Axon-Hillock Circuit
The Axon-Hillock Circuit

voltage
Positive Feedback
Input current
Membrane voltage
Vmem
Vmem
Vout
Vout
Vmem
Vout
Output voltage
Vout
time
Vpw
Reset
Slope = A
Vpw
Vmem
ICANN09 Tutorial
53 / 78
Capacitive Divider
ICANN09 Tutorial
Positive Feedback
Given the change V 2, what is V 1?
C2
Q
Q = C1 V1 + C2 (V1 V2 ) = constant
C1 V1 + C2 (V1 V2 ) = 0
C2
C1 + C2
V1
Vmem
A
C1
V2
time
V2
ICANN09 Tutorial
55 / 78
Vout
Vmem
Vout
Cm
Vpw
voltage
Cfb
V1 =
54 / 78
Positive Feedback
Cfb
Vmem =
V
Cm + Cfb dd
ICANN09 Tutorial
56 / 78
Axon-Hillock Circuit Dynamics

voltage
Cfb
Iin
Vmem
Gain
Vmem
Vout
How to make voltage gain
Vout
Cm
tH
tL
time
Ir
Vpw
tL =
Cfb + Cm
Iin
Vmem =
Frequency Iin
Cfb
Iin
Vdd
tH =
Cfb + Cm
Ir Iin
Vmem =
Cfb
Ir Iin
Vdd
Pulse width 1/Ir for Ir Iin

ICANN09 Tutorial
Whats bad about this?

57 / 78
Power Dissipation
ICANN09 Tutorial
58 / 78
Conductance-based models
Integrate and Fire vs Hodgkin-Huxley
The Axon-Hillock circuit is very compact and allows for

implementations of dense arrays of silicon neurons
Traditionally there have been two main classes of neuron models:
BUT
it has a major drawback: power consumption
During the time when an inverter switches, a large amount
of current flows from Vdd to Gnd.
Integrate and fire (I-C)
ICANN09 Tutorial
59 / 78
Conductance-based (R-C)
ICANN09 Tutorial
61 / 78
Conductance-based models
An ultra low-power generalized I&F circuit
Integrate and Fire vs Hodgkin-Huxley

PositiveFeedback
Adaptation
But recently proposed models bridge the gap between the two:
M11
M5
Iin
Vahp
Vrest
Vthr
M1
DPI
M7
M12
M6
Ifb
M8
Cmem
Leak
M9
M15
M19
M16
M20
Vspk
Imem
M22
Iahp
Vtau
M18
Vmem
M3
M2
Vthr_ahp
M14
Cahp
M10
M4
M17
DPI
Vtau_ahp
Vrf
M21
M13
RefractoryPeriod
(G. Indiveri, P. Livi, ISCAS 2009)
ICANN09 Tutorial
62 / 78
Generalized Integrate and Fire models can account for a very large set of
DPI
neuron sub-threshold equations
d
M5
Vahp
dt
M11
Cmem
M12
+ F (umem )
M18
where F (uDPImem ) is a non-linear function of umem (t ).

Vthr
M2
Vthr_ahp
M1
M6
Vmem
M3
Cmem
Imem
Vtau
M9
M4
M15
M19
M16
M20
15
Vspk
Vthr=0.325 V
Vthr=0.350 V
Vthr=0.375 V
14
12
10
M22
Iahp
Leak
20
Ifb
M8
M7
DPI
Cahp
M10
Vrf
M17
Vtau_ahp
M21
Imem (A)
Vrest
63 / 78
PositiveFeedback
in
M14
M13
10
Imem (A)
Iin
umem =
ICANN09 Tutorial
SPICE simulations
behaviors captured by far more complicated Hodgkin-Huxley models.

Adaptation
8
6
4
RefractoryPeriod
d
dt
Imem + Imem
1
I
I0+1 , ,
Ig Iin
I
0
2
+ (Imem )
6
Time (ms)
10
20
40
60
Time (ms)
80
100
2 + 1
+1
ICANN09 Tutorial
64 / 78
ICANN09 Tutorial
66 / 78
Experimental results
Experimental results
Single spike
Population activity
2.8
Vthr= 0.0V
V = 0.3V
Voltage (V)
2.7
Vthr= 0.6V
2.6
Vthr= 0.9V
Neurons
thr
2.5
30
30
25
25
20
20
15
15
10
10
0.5
1.5
0.5
1.5
0.5
1
Time (s)
1.5
2.4
2.3
Neurons
2.2
2.1
2
1.9
0.01
0.02
0.03
Time (s)
0.04
0.05
ICANN09 Tutorial
30
30
25
25
20
20
15
15
10
10
67 / 78
Spiking multi-neuron architectures
0.5
1
Time (s)
1.5
ICANN09 Tutorial
70 / 78
Spikes and Address-Event Systems

1
Vmem (V)
0.8
0.6
0.4
Encode
Decode
0.2
0
0
0.05
0.1
Time (s)
0.15
0.2
1
2
3
Address Event Bus

1 23 2
Networks of I&F neurons with adaptation,

refractory period, etc.
Inputs
1
2
3
12
Outputs
Source
Chip
Synpases with realistic temporal dynamics
Destination
Chip
Address-Event
representation of
action potential
Winner-Take-All architectures
Spike-based plasticity mechanisms
Action Potential
ICANN09 Tutorial
72 / 78
ICANN09 Tutorial
73 / 78
15
10
5
Distributed multi-layer networks
0
180
120
60
60
120
180
Rotation around y axis (degrees)
Figure 1 Tuned units in inferotemporal cortex. A monkey was trained to recognize

a three-dimensional paperclip from all viewpoints (pictured at top). The graph
shows tuning to the multiple parameters characterizing each view summarized in
terms of spike rate versus rotation angle of three neurons in anterior inferotemporal
cortex that are view-tuned for the specific paperclip. (The unit corresponding to the
green tuning curve has two peaks to a view of the object and its mirror view.) A
combination of such view-tuned neurons (Fig. 2) can provide view-invariant, object
specific tuning as found in a small fraction of the recorded neurons. Adapted from
Logothetis et al.13.
AMS 0.35 m
3.9mm 2.5mm
128
28 128
4 128
32 128 | . . . | 1 4096
Technology:
Size:
Neurons:
AER plastic synapses:
AER non-plastic synapses
Dendritic tree multiplexer:
ICANN09 Tutorial
Summary
Vg
Ir
Ir
M11
M5
Vahp
Vrest
Vthr
M2
M1
DPI
Vthr_ahp
M7
M6
M9
M4
V4/PIT
V4
V1
V1
Figure 2 A model of visual learning. The model summarizes in quantitative terms

other models and many data about visual recognition in the ventral stream pathway
in cortex. The correspondence between the layers in the model and visual areas is
an oversimplification. Circles represent neurons and arrows represent connections
@ INI)
between them;G.Indiveri
the dots signify (NCS
other neurons
of the same type. Stages of neurons
with bell-shaped tuning (with black arrow inputs), that provide example-based
learning and generalization, are interleaved with stages that perform a max-like
operation3 (denoted by red dashed arrows), which provides invariance to position
and scale. An experimental example of the tuning postulated for the cells in the
layer labelled inferotemporal in the model is shown in Fig. 1. The model accounts
well for the quantitative data measured in view-tuned inferotemporal cortex cells10
(J. Pauls, personal communication) and for other experiments55. Superposition of
gaussian-like units provides generalization to three-dimensional rotations and
together with the soft-max stages some invariance to scale and position. IT,
infratemporal cortex, AIT, anterior IT; PIT, posterior IT; PFC, prefrontal cortex.
Adapted from M. Riesenhuber, personal communication.
ICANN09 Tutorial
75 / 78
Thank you for your attention

Additional information available at:
769
http://ncs.ethz.ch/
2004 Nature Publishing Group
Vd
If
Qd
M18
M15
Cmem
Leak
IT
Ifb
M8
Imem
M19
Vspk
M16
M20
M22
Iahp
Vtau
AIT
M14
M12
Vmem
M3
Ident.
PositiveFeedback
Adaptation
Iin
Qs
Categ.
If
Vg
Vs
AER INPUT Y
PFC
NATURE | VOL 431 | 14 OCTOBER 2004 | www.nature.com/nature
Vd
Vs
types of algorithm developed within learning theory corresponds to

networks that combine the activities of units, each broadly tuned to
one of the examples (Box 1). Theory (see references in Box 1) shows
that a combination of broadly tuned neurons those that respond
to a variety of stimuli, although at sub-maximal firing rates might
generalize well by interpolating among the examples.
In visual cortex, neurons with a bell-shaped tuning are common.
Circuits in infratemporal cortex and prefrontal cortex, which combine activities of neurons in infratemporal cortex tuned to different
objects (and object parts) with weights learned from experience, may
underlie several recognition tasks, including identification and
categorization. Computer models have shown the plausibility of this
scheme for visual recognition and its quantitative consistency with
many data from physiology and psychophysics25 .
Figure 2 sketches one such quantitative model, and summarizes a
set of basic facts about cortical mechanisms of recognition established
over the last decade by several physiological studies of cortex68. Object
recognition in cortex is thought to be mediated by the ventral visual
pathway running from primary visual cortex, V1, over extrastriate
visual areas V2 and V4 to the inferotemporal cortex. Starting from
simple cells in V1, with small receptive fields that respond preferably to
oriented bars, neurons along the ventral stream show an increase in
78
receptive field size as well as in74
the/complexity
of their preferred stimuli.
At the top of the ventral stream, in the anterior inferotemporal cortex,
neurons respond optimally to complex stimuli such as faces and other
objects. The tuning of the neurons in anterior inferotemporal cortex
probably depends on visual experience919. In addition, some neurons
show specificity for a certain object view or lighting condition13,18,2022.
For example, Logothetis et al.13 trained monkeys to perform an object
recognition task with isolated views of novel three-dimensional objects
(paperclips; Fig. 1). When recording from the animals' inferotemporal
cortex, they found that the great majority of neurons selectively tuned
to the training objects were view-tuned (see Fig. 1) to one of the training
objects. About one tenth of the tuned neurons were view-invariant,
consistent with an earlier computational hypothesis23.
communication
AER OUTPUT
A minimum-size chip implementing a

reconfigurable AER neural network.
Neurons and synapses have realistic
temporal dynamics. Local circuits at
The basic problem with these models is, of course, generalization:
each synapse implement
the bi-stable
a look-up table cannot deal with new events, such as viewing a face
from the side rather than the front, and it cannot learn in the predicspike-based plasticity mechanism.
tive sense described earlier. One of the simplest and most powerful
Indiveri, Fusi, 2007
which several models have been suggested24; see also review in this
issue by Abbott and Regehr, page 796), since it depends only on
passive experience of the visual inputs. However, the weights of the
combination (see Fig. 3) depend on learning the task and require at
least some feedback (see Box 2).
Thus, generalization in the brain can emerge from the linear combination of neurons tuned to an optimal stimulus effectively
defined by multiple dimensions25,23,26. This is a powerful extension of
Analog
processing, asynchronous
digital
the older computation-through-memory
models of vision
and
motor control. The question now is whether the available evidence
supports the existence of a similar architecture underlying generalization in domains other than vision.
AER INPUT X
Spikes per seco
A spike-based learning chip
239
20
M17
DPI
Cahp
M10
Vtau_ahp
Vrf
M21
M13
RefractoryPeriod
ICANN09 Tutorial
77 / 78
ICANN09 Tutorial
78 / 78

Custom VLSI Implementations of Neural Networks: Early Attempts

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Custom VLSI Implementations of Neural Networks: Early Attempts

Uploaded by

Copyright:

Available Formats

Custom VLSI implementations of neural networks

Analog VLSI circuits

General purpose computing

Competing with Intel steamroller

Neural network accelerator

September 17, 2009

Silicon neuron designs

G.Indiveri (NCS @ INI)

Embedded system integration

Silicon neural network characteristics

G.Indiveri (NCS @ INI)

Above threshold (strong inversion)

Below threshold (weak inversion)

G.Indiveri (NCS @ INI)

Why subthreshold neuromorphic VLSI

Exploit the physics of silicon to reproduce the

n-FET subthreshold transfer function

Ids = I0 en Vg /UT eVs /UT eVd /UT

n denotes the nFET subthreshold slope factor

Diffusion and saturation

G.Indiveri (NCS @ INI)

Exponential voltage dependence

Ids = I0 en Vg /UT eVs /UT eVd /UT

If Vds > 4UT the Ir term becomes negligible,

Ids = I0 en Vg /UT Vs /UT

G.Indiveri (NCS @ INI)

G.Indiveri (NCS @ INI)

n-FETs and p-FETs

One, two, and three transistor circuits

In Complementary Metal-Oxide Semiconductor (CMOS) technology, there are

Ideal current source

G.Indiveri (NCS @ INI)

G.Indiveri (NCS @ INI)

In the linear region (|V1 V2 | < 200mV ):

The transconductance amplifier

G.Indiveri (NCS @ INI)

Synapses in the nervous system

In 1897 Charles Sherrington

Eex (Na+, ...)

Einh (K+, Cl, ...)

G.Indiveri (NCS @ INI)

G.Indiveri (NCS @ INI)

EPSC and EPSP

O. SACCHI, M. L. ROSSI, R. CANELLA, AND R. FESCE

In chemical synapses the presynaptic and

Typical synaptic currents at ganglionic synapse evoked by supra-

low the linear fit. This might be partly because of inward

maximal preganglionic stimulation.

Chemical synaptic transmission is

currents were 220 ms in the 060/ 030 mV voltage range (Owen

G.Indiveri (NCS @ INI)

Neural network models

VLSI synapses in classical neural networks

In classical neural network theory

neurons implement a saturating non-linearity transfer function (S) on the

the synapse implements a multiplication between the neurons input

VLSI synapses in pulse-based neural networks

A linear integrator is a linear low-pass filter. Its impulse response should be a

In pulse-based neural networks the weighted contribution of a synapse can be

G.Indiveri (NCS @ INI)

Linear pulse integrators

G.Indiveri (NCS @ INI)

G.Indiveri (NCS @ INI)

Response to an arbitrary spike train (t ) = i (t ti ), with t :

Pulse width 1/Ir for Ir Iin