Simple Hardware Implementation of Neural Networks For Instruction in Analog Electronics

Session 1793
Simple Hardware Implementation of Neural Networks

for Instruction in Analog Electronics
Kenneth J. Soda and Daniel J. Pack

Department of Electrical Engineering
United States Air Force Academy
In light of the growing predominance of microprocessors and embedded

electronic systems, instruction in basic analog and digital electronic circuits has come to
appear less interesting and important to contemporary students of electrical engineering.
Despite the continuing importance of foundation circuit concepts, curricula across the
country are reducing their emphasis in required courses or shifting them into optional
courses. In hopes of mitigating this trend, we discuss a circuit system which applies
traditional analog and digital MOSFET sub-circuits into a meaningful contemporary
system, the neural network. Neural networks offer a unique approach for processing
complex data streams without the need for digital processors. Constructed in a fashion
which mimics biological nervous systems, these networks are finding applications in
signal processing, control and object recognition. In many cases, a properly prepared
neural network can function faster than a comparable microprocessor based system, with
lower power consumption and lower level of complexity. Despite their potential and
relative conceptual simplicity, it has been difficult to present electronic neural networks
in a form convenient for the university classroom or electronics laboratory setting. In this
paper we describe an approach for implementing a neural network though which many
major analog and digital MOSFET circuit concepts can be illustrated and demonstrated.
This approach is amenable to realization in discrete electronic modules through which
associated laboratory exercises and design projects may be created. Furthermore, the
same concepts can be extended into Very Large Scale Integration (VLSI), where the
limitations of component count and performance can be overcome and addressed to a far
greater degree.
Introduction
The fundamental motivation to study neural networks is based on the belief that humans
make better decisions than machines due to our abilities to process information in parallel.
By treating a large amount of data while extracting and processing relevant contextual
data from diverse source simultaneously, we are believed to fuse the necessary
information to arrive at fairly sophisticated decisions.
The idea of parallel distributed processing models received significant attention when
Minsky showed a number of applications of connected networks called perceptrons1 in
Page 9.1096.1
1
M. Minsky and S. Papert, Perceptrons, The MIT Press, Cambridge, MA, 1969.
Proceedings of the 2004 American Society for Engineering Education Annual Conference & Exposition
Copyright © 2004, American Society for Engineering Education
Page 1
1960’s. Although perceptrons solved interesting problems, they failed to attract a larger
scholarly community due to their inability to extend their uses to more complex problems.
The field of neural networks was re-energized in the late 70’s and early 80’s [2] when
previously considered as unsolved problems using connected networks such as the
exclusive-OR problem was solved by a network of neurons arranged with multiple layers.
The introduction of multiple layered neural networks allowed researchers with
capabilities to extract contextual information from large data, perform pattern recognition
tasks including text and object recognition, and control complex dynamic systems.
The function of a neural network is determined by the number of input and output
neurons, the number of neurons for each network layer, the number of layers, the
connection weights, and the network configurations. In most cases, configurations,
connections among neurons, and connection weights stay constant throughout system
operations. For a feedfoward network - our interest - the designer of a neural network
considers the input patterns and desired output patters to determine the number of input
layer neurons, the hidden layer neurons, and the output layer neurons as shown in the
following figure. The signal propagates from the left to the right in the network shown in
Figure 1. The signals coming out of the output layer nodes are used to make a variety of
decisions.
For example, if pixel values of a camera image are the input signals entering the input
layer, the output layer values can indicate the identity of an object shown in the camera
image. If sensor values of a machine are used as the input signals, appropriate control
signals correspond to the outputs of the neural network. If combinational logic states are
used as the input signals, the neural network can model desired logic function to produce
appropriate output logic states. The math involved in modeling neurons and generating
outputs for various network configurations is readily available in the literature2.
Once we determine the number of neurons for each layer, based on the number of inputs
and outputs of a system, and the type of function used to process input signals for each
neuron, typically a sigmoid function, the connection weights between neurons are used to
‘program’ the behavior of a network. The connection weights govern the importance of
the output signals of neurons as the input signals of connected neurons. For example, if a
particular connection weight between two neurons is zero, the output of the first neuron
does not make any contribution to the input of the connected neuron. In the feedfoward
neural network, neurons in two adjacent layers are linked together with connection
weights. A part of the designer’s job is to find appropriate connection weights for a
neural network to produce desired output signals.
Finding the appropriate set of connection weights for a neural network is done through a
learning process. The learning process involves comparing desired outputs and current
outputs of a network and adjusting the connection weight values to reduce the difference.
The adjustment process is repeated until the difference between the desired and actual
network outputs is reduced to an acceptably low value. Several techniques are available
Page 9.1096.2
2
I. Aleksander and H. Morton, “An Introduction to Neural Computing,” Chapman and Hall, London, 1990.
Page 2
to reduce the error; the most used method is called delta method3, where the difference
between the desired and the actual output values are used to compute the incremental
changes necessary in the connection weights starting from the output layer to the input
layer of a network.
Connection Weights
Output Layer
Input Layer Hidden Layer
Figure 1. Typical Feedforward Neural Network
The contemporary nature of neural networks makes them highly interesting and
motivating for students engaged in the study of analog electronics. However, the
complexity of the circuits makes them difficult to realize in the university electronics
laboratory setting. Indeed, neural networks of complexity sufficient for object
recognition and signal processing require integrated circuit implementation. We have
identified a hardware architecture which implements the classic feed-forward network in
a comparatively simple way. This architecture not only permits students to explore
network operation directly, it also serves as an excellent vehicle to instruct and reinforce
key analog circuit design concepts.
In this paper, we describe the Switchable-Sign Synapse Architecture for realization of

electronic neural networks. The nature of major network components and their
interconnections are detailed. We illustrate the pedagogical relationship between each
network component and basic operation of Field Effect Transistors and other key analog
circuit and digital circuit concepts. Finally we illustrate the manner in which small
networks can be implemented using discrete commercial components and assembly
technologies available modern university electronics laboratories.
3
D.E. Rumelhart, J. L. McClelland, Parallel Distributed Processing, Explorations in the Microstructure of
Page 9.1096.3
Cognition, vol.1: Foundations, The MIT Press, Cambridge, MA, 1986.
Page 3
Network Implementation
Many circuit methodologies exist through which the general architecture of a feed-
forward neural network may be implemented. After some research, we have identified
the Switchable-Sign-Synapse Architecture (SSSA) of Fakhraie and Smith4 as a
straightforward approach for network realization which exercises many traditional analog
and digital electronics concepts.
The central elements of a feedforward neural network are the neurons represented by the
array of circles in Figure 1. In the SSSA implementation, the signals between layers of
neurons are represented by potentials. Each neuron is further divided into synapses, each
of which accepts a single weighted input either from a signal source or the preceding
layer of neurons. The contribution of each synapse is represented by a scaled current.
All synapse current contributions are summed, scaled and converted back to a voltage
value. This signal represents the corresponding neuron’s output and is passed on to the
next network layer.
Details of an SSSA architecture neuron are described in greater detail in Figure 2. Each
synapse is created with a single n-channel Metal Oxide Silicon Field Effect Transistor
(MOSFET). This FET pulls current in from one of two shared signal lines named In+
and In- depending upon the state of the synapse’s memory (SRAM) cell. The Current
Accumulation and Scaling Cell (CASC) produces the scaled output current proportional
to the arithmetic difference between currents pulled in on the In+ line and the In- line.
This net current is reconverted to a voltage through the sigmoid circuit. Although Figure
1 implies a single signal connection, the magnitude of current pulled down by each
synapse is actually controlled by both a weight voltage Wx and a signal input voltage
VInx. The state of a digital memory cell determines to which of the two input lines each
synapse current is contributed. It is this feature from which the Switchable-Sign Synapse
Architecture derives its name. It is important that each synapse is capable of either aiding
or opposing the response of its neuron. While this feature makes the teaching process
somewhat more complex, it also permits SSSA networks to correctly learn more complex
behaviors. Once taught a particular function, the weight voltages, Wx, and states of each
synapse logic cell, Sign-x in each neuron embody the response of a network to a given set
of input voltage signals, VIn_x.
A key advantage of the SSS architecture is its modularity. The neuron illustrated below
can be expanded to accept more input signals through the addition of more synapse units,
each connected in parallel to the In+ and In- signal lines. Each synapse and sigmoid cell
can be identical. Only scaling changes in the Current Accumulation and Scaling Cells
are necessary as the complexity of each neuron increases. Indeed, this methodology is
naturally implemented via CMOS integrated circuit technology where repetition of circuit
elements greatly eases design and reduces the cost of fabrication.
4
S. M. Fakhraie and K. C. Smith, “VLSI-Compatible Implementation of Artificial Neural Networks”,
Page 9.1096.4
Kluwer International Series in Engineering and Computer Science, SECS 382, ISBN 0792398254, Kluwer
Academic Publishing, Boston, 1997.
Page 4
Sign-x In+ In-
SRAM
Wx
SyMOS
VIn_x
Synapse x
Current
Accumulating IDiff Sigmoid Vout
and Scaling Cell
Cell
Sign-1
SRAM
W1
SyMOS
VIn1
Synapse 1
Figure 2 Block diagram of the SSSA neuron.
Neuron Component Implementation
The implementation of each of the major neuron components may be achieved through
circuits commonly the subject of electrical engineering instruction in analog and digital
circuits. As such, the study of SSSA networks can serve both as a source of
reinforcement and motivation for students engaged in the study of electronics. We
highlight the pedagogical relationships as part of our discussion of neuron components
below.
The Synapse
Each synapse in the SSSA network architecture is composed of two types of digital logic
circuits and a single floating gated MOSFET, called a SyMOS. The interconnection of
these components is illustrated in Figure 3. The conducting characteristics of the two
CMOS transmission gates is controlled by the state of a 1 bit digital logic storage element,
in this case implemented with a single SRAM cell. The drain current of the SyMOS
transistor Mx is therefore connected through low resistance to either to the IIn+ or IIn-
signal lines. The floating gate SyMOS transistor is operated in the saturated mode, with
drain current depending upon the relative value of the gate potentials VInx, VWx and
gate capacitances C1X, C2X and CGX.
Page 9.1096.5
Page 5
In saturation, MX’s drain current is controlled primarily by its gate-source potential,
which in turn is controlled via a capacitive voltage divider. Through straightforward
circuit analysis it can be shown that the static threshold voltage of MX is given by:
VthInx =
(C1x + C 2 x + CTotal )Vth − CTotal V −
C 2x
VWx
B
C1x C1x C1x
and the corresponding drain current of MX given by:
(
I D _ MX = k n' VInX − VthInx )
2
where CTotal is the sum of all capacitance connected to the floating gate node and VB is
the FET’s substrate or body potential and k’n is the FET gain factor.
IIn+ IIn-
1-Bit Q
Sign
SRAM Cell
Clk
QNot
C1x
VInx
VB
Wx
Mx
C2x
VSS
Figure 3. Detailed view of an SSSA synapse. The common logic level representation for
the transmission gate is used here. For simplicity, details of the SRAM cell are not
shown. The possible flow paths for Mx’s drain current are indicated by dark lines.
Page 9.1096.6
Page 6
The synapse contains illustrations of several key digital and analog circuit concepts.
Transmission gates5 are commonly studied because of their application to CMOS digital
logic circuits, especially Pass Transistor Logic gates and flip flops. The study of digital
static memory traditionally includes the dynamic behavior of the SRAM cell6. The
operation of the floating gate transistor reinforces saturated mode conduction
characteristics of the MOSFET7 as well as the influence of the substrate or body potential
upon the threshold voltage of the device8. The SyMOS also reinforces the concept of
quasi-static voltage division using capacitors. The latter concept is often discussed as
part of a first course on electric circuits9 or as part of the study of scaled digital to analog
voltage converters10.
VDD
M1 M2 M5 M6
IIn- M3 M4 IIn+
IDiff
VSS
Figure 4. The SSSA architecture Current Accumulation and Scaling Cell. All synapse
IIn- signals within a particular neuron are connected to the heavily shaded IIn- line of this
in this circuit. Likewise synapse IIn+ signals are connected to the heavily shaed IIn+ line.
The net current either flows into or out as IDiff.
5
See for example A. S. Sedra and K. C. Smith, “Microelectronic Circuits”, 4th Ed., Oxford University Press,
New York, 1998, pg 438-441.
6
See for example reference 2, pg 1116-1112.
7
8
9
See for example J. W. Nilsson and S. A. Riedel, “Electric Circuits”, 5th Ed., Addison-Wesley Publishing
Page 9.1096.7
Co., Reading MA, 1996, pp 229-30.

10
Page 7
The Current Accumulation and Scaling Cell
The details of the Current Accumulation and Scaling Cell are illustrated in Figure 4
above. Current contributed by many synapses is summed onto either the IIn+ or IIn-
inputs of this circuit. The current mirrors created by M1-M2 and M5-M6 reflect these
currents into a third current mirror created by M3-M4. The net current, IDiff either flows
into or out of the node shared by the drains of M6 and M3. The relative weight currents
in this circuit can be adjusted through the gain factors of the corresponding transistors.
The simple MOSFET current mirrors used here are commonly applied in analog circuits,
both as biasing current sources and as active loads11. These circuits are only slightly
different than BJT current mirrors, another common topic in the study of analog
electronics12.
The Sigmoid Cell
The final component of the SSSA neuron is the sigmoid cell. It provides an output
voltage which is proportional to the direction and magnitude of the current IDiff. Just as
over-stimulation of a biological neuron results in a saturation of response (e.g. the
adjustment over time of the eye condition of bright illumination), so must the sigmoid
cell also present a saturated voltage behavior at large positive and large negative IDiff.
This is accomplished through the use of three MOSFETs whose characteristics overlap.
When the magnitude of IDiff is small, transistor MP2 tends to conducts since it has the
largest negative gate potential. The small currents correspond to the triode region of
operation where the relationship between IDiff and VDS of MP2 is nearly linear. At
these small output voltages, both MN1 and MP2 remain cutoff. When large currents
enter the cell, transistor MP1 becomes an additional conduction path. In this condition,
positive values of MP1s source potential or Vout are required to make MP1 “turn on”.
MP1 is easily saturated due its drain’s connection to negative supply voltage. Any
increased current entering the cell can be accommodated by increased conduction
through MP1, without significantly increasing Vout. Large currents leaving the cell are
likewise provided by MN1, but will require VOut to become negative. MN1 becomes
saturated as outward flowing current increases without significantly decreasing Vout.
The operation of this circuit provides an excellent reinforcement of the fundamental dc

operation of enhancement mode MOSFETs13. The relationship between drain-source /
gate-source potential and the mode of FET operation are key to understanding the three
distinct regions of the sigmoid circuit’s operation. The sigmoid is also an excellent
illustration of the concept of causality in device operation. Students are conventionally
taught that potential differences cause drain current. However, students must also realize
that forcing a drain current through a FET will cause it to settle to a predictable set of
terminal voltages. The idea that currents will cause terminal voltages is useful in
building an intuitive sense of linear MOS circuit operation, particularly when active loads
are involved.
11
Page 9.1096.8
12
See for example reference 2, pg 508-514 and 522-527.
13
Page 8
VDD
MN1
VSS
IDiff VOut
MP2
MP1
VSS VSS
Figure 5. The CSSA implementation of the Sigmoid Circuit. MP2 creates a linearly
changing Vout at small magnitudes of IDiff. MP1 creates a saturated positive output
voltage for large IDiff entering the cell, MN1 a saturated negative voltage for large
currents leaving the cell.
Discrete Hardware Implementation
In order to illustrate and reinforce the operation of the SSSA neural network, working
laboratory realizations are an important pedagogical tool. We describe here the
methodology through which small scale neural networks can be realized using discrete
components. The applicability of SSSA to CMOS integrated circuits is obvious. While
discrete n and p-channel MOSFETs can be easily be interconnected to create VLSI
CMOS circuits, some modifications to the SSSA approach are necessary to realize the
intended function in discrete form. Furthermore, there are practical considerations which
limit the scale and performance of discrete networks.
Size vs Complexity Choices for Discrete Networks

Just as in biological networks, greater sophistication in response requires more extensive
networks. The SSS architecture could require some thirty transistors to realize a single
synapse neuron. Hence a discrete implementation for instructional demonstration might
be more easily realized if its size were reduced. A good compromise between capability
and component count is the so called 2 X 1 network illustrated in Figure 6 below. This
network has only an input and output layer. Each neuron must be capable of accepting
just two inputs. Since the neurons are identical, such a configuration reduces the scope of
the design problem. At this level of sophistication, behaviors with fair complexity can be
Page 9.1096.9
implemented, including the once troublesome exclusive-OR logic function.

Page 9
In1 N1
Out
N3
In2 N2
Figure 6. The simple 2X1 neural network.
Additional Discrete Network Simplifications

To further reduce the complexity of discrete neural networks, we can presume that
changes to the synapse input conditions Sign-x and VWx will be made manually. This
permits us to replace the transmission gates and SRAM cell in each synapse into a one
single-pole double-throw switch. We also presume the creation of weight voltage VWx
through large valued potentiometer placed between the positive and negative power
supply voltages.
This approach leaves only the analog parts of the circuit and to be implemented through
discrete MOSFETs. The significant difference between integrated and discrete
implementations of these networks is the ability to control the gain factor, ‘k’. Discrete
MOSFETs are generally manufactured to meet the demands of power control circuits
which require large drain currents. As such, their gain factors are large. Large currents
can present power dissipation problems which can make demonstration circuits unreliable.
The magnitude of the threshold voltage of discrete MOSFETs tends to range from one to
several volts in order to limit the sensitivity of power control circuits to noise. These
high thresholds will make it more difficult for the sigmoid circuit to reach a saturated
condition making the network response less abrupt than is desirable.
We have identified two solutions through which these effects can be limited. The
effective gain factor of an FET can be reduced by tying two devices in series, drain to
source, with their gates connected together. The effect gate length is theoretically
doubled, thus cutting the gain factor in two. We have also identified commercial discrete
FETs with threshold voltages of less than 1 volt. ZETEX markets the ZXM61P02F and
ZXM61NO2F complementary p and n-channel MOSFETs which possess nominal
threshold voltages magnitudes of 0.6V. Their SOT23 packages are large enough to be
handled, yet small enough that compact printed circuit implementations are possible.
Other Implementation Considerations

Due to their well controlled manufacturing processes, our experience has shown discrete
Page 9.1096.10
MOSFETs are enough alike in properties that the current accumulation cell can be
realized without significant difficulties. This leaves only the sigmoid unit as presenting
Page 10
the possibility for undesirable behavior. Our experience has shown that a sufficiently
saturated behavior can be achieved using discrete MOSFETs in the configuration
illustrated in Figure 5. However, adjustments to the width of the linear regions its I-V
characteristic can be achieved by independently controlling the gate voltage on each
transistor through a large valued potentiometer.
The realization of discrete neural networks should include some consideration of the
method for component interconnection. We recommend the use of a rudimentary printed
circuit board in any hardware realization of these networks. The surface mount package
in which our MOSFET of choice is delivered demands this interconnection technique.
We have also found it relatively simple to construct identical copies of the simple two
input neurons on standard printed circuit cards. The interconnection of these neuron
cards into the complete network can then be accomplished with a second style printed
circuit board or protoboard.
In order to make the operation of the simple 2X1 network interesting to students, we
recommend the use of simple pin photo-detector circuits for the creation of the network
input signals. Once a set of proper weight voltages has been identified, the sensitivity of
the network to illumination of the detectors to a set of light-dark patterns can be
demonstrated. Responses mimicking the truth table of an AND, OR, NAND, NOR,
EXOR or EXNOR logic gate can be programmed and demonstrated. However, the
network’s sensitivity to “almost true” inputs can also be shown, thus differentiating
neural network responses from that or digital logic gates. This is the feature of neural
networks which gives them an advantage over digital systems; the ability to provide an
indication that a pattern observed by the system sensors bears a resemblance to that
which it is programmed to detect.
Summary
We have described in detail the Switchable-Sign Synapse Architecture for realization of

hardware neural networks. These electronic systems serve as meaningful and
contemporary vehicles through which instruction in basic MOSFET analog and digital
electronic circuits may be accomplished. This architecture realizes neurons with the
smallest number of electronic components, is known to “learn” its desired function with
the smallest number of iterations, and to be capable of learning complex behaviors. We
have further described techniques through which a small scale neural network using this
technology may be implemented using discrete components. These demonstration
circuits may be simplified so that the state of each synapse and magnitude of each
synapse weight is controlled manually. These demonstrators provide a vehicle through
which students can examine the detailed operation of neural networks in the university
electronic laboratory setting.
Page 9.1096.11
Page 11
KENNETH J. SODA. Dr. Soda is the first permanent civilian faculty member of the USAF Academy’s
Department of Electrical Engineering. He holds an advanced degree from University of Illinois, Urbana-
Champaign. Dr. Soda is the 1997 Recipient of the Tau Beta Pi Teacher of the Year Award (Colorado Zeta
Chapter) and the 1988 Recipient of the USAF Academy Outstanding Educator Award.
DANIEL J. PACK. Dr. Pack received the Bachelor of Science degree in Electrical Engineering from
Arizona State University, 1988. He received the Master of Science in Engineering Sciences from Harvard
University in 1990 and the Ph.D. degree from the School of Electrical and Computer Engineering at Purdue
University, West Lafayette, Indiana. Daniel Pack is a recipient of the Outstanding Academy Educator
Award, Tau Beta Pi Faculty Award, Digital Corporation Scholarship, Magoon Teaching Award, and NSF
Travel Scholarship. He is a member of Eta Kappa Nu (Electrical Engineering Honorary), Tau Beta Pi
(Engineering Honorary), IEEE, and ASEE.
Page 9.1096.12
Page 12

Simple Hardware Implementation of Neural Networks For Instruction in Analog Electronics

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Simple Hardware Implementation of Neural Networks For Instruction in Analog Electronics

Uploaded by

Copyright:

Available Formats

Session 1793

Simple Hardware Implementation of Neural Networks

Kenneth J. Soda and Daniel J. Pack

In light of the growing predominance of microprocessors and embedded

Input Layer Hidden Layer

Figure 1. Typical Feedforward Neural Network

In this paper, we describe the Switchable-Sign Synapse Architecture for realization of

Cognition, vol.1: Foundations, The MIT Press, Cambridge, MA, 1986.

Figure 2 Block diagram of the SSSA neuron.

Neuron Component Implementation

and the corresponding drain current of MX given by:

Co., Reading MA, 1996, pp 229-30.

The Sigmoid Cell

The operation of this circuit provides an excellent reinforcement of the fundamental dc

Discrete Hardware Implementation

Size vs Complexity Choices for Discrete Networks

implemented, including the once troublesome exclusive-OR logic function.

Figure 6. The simple 2X1 neural network.

Additional Discrete Network Simplifications

Other Implementation Considerations

We have described in detail the Switchable-Sign Synapse Architecture for realization of

You might also like