Neurocomputing

NN-Models

Neurocomputing
Prof. Dr.-Ing. Andreas König
Institute of Integrated Sensor Systems ISE

Dept. of Electrical Engineering and Information Technology

Technische Universität Kaiserslautern

Fall Semester 2006

© Andreas König Slide 3-1

Neurocomputing
NN-Models

Course Contents:
1. Introduction
2. Rehearsal of Artificial Neural Network models relevant for
implementation and analysis of the required computational steps
3. Analysis of typical ANN-applications with regard to computational
requirements
4. Aspects of simulation of ANNs and systems
5. Efficient VLSI-implementation by simplification of the original
algorithms
6. Derivation of a taxonomy of neural hardware
7. Digital neural network hardware
8. Analog and mixed-signal neural network hardware
9. Principles of optical neural network hardware implementation
10. Evolvable hardware overview
11. Summary and Outlook

© Andreas König Slide 3-2

1
 Neurocomputing
Chapter Contents NN-Models

1. Analysis of typical ANN-applications with regard to

computational requirements
3.1 Image compression by vector quantization
3.2 Optical character recognition
3.3 Image compression by facial feature recognition
3.4 Face recognition
3.5 Visual and acoustical inspection tasks
3.6 Summary

© Andreas König Slide 3-3

Neurocomputing
Image Compression by VQ NN-Models

¾ Typical Neural Network Applications:

• Control (Adaptive Control)

• Communication (Data Compression)
• Identification and Access Control
• Fingerprint Compression and Recognition
• Optical Character and Handwritten Word Recognition (zip codes,
cheques, credit card slips, car number plates, ...)
• Visual Inspection for Automated Industrial Quality Control
• Satellite and Aerial Image Evaluation, Recognition, and
Surveillance
• Medical Image and Signal Analysis
• Prediction in Finance
• Optimization Problems (Scheduling, Resource Allocation, ...)
• Data Analysis, Data Base Mining
• General Pattern Recognition

© Andreas König Slide 3-4

2
 Neurocomputing
Image Compression by VQ NN-Models

¾ Separation in three different needs for real-time processing:

• Real-time requirements of the data processing steps in the

operation of the network with fixed weights
• Time requirements for (off-line) network training by the user
• Real-time requirements for on-line adaptation of network
parameters

¾ Compression of data (Images, speech, text) is an important issue to

save channel or memory capacity
¾ Commonly applied techniques:

• Lossless techniques, such as, e.g., run-lenght coding (1.5 to 4)

• Lossy techniques, such as transform and quantization coding used
in JPEG (40) and related algorithms
• Vector quantization is a lossy technique based on groupwise
compression or quantization of data exploiting local correlations
© Andreas König Slide 3-5

Neurocomputing
Image Compression by VQ NN-Models
¾ For data compression in communication of images (video phone) or
speech, e.g., vector quantization can be applied:

Codebook Codebook

© Andreas König Slide 3-6

3
 Neurocomputing
Image Compression by VQ NN-Models

¾ Illustration of codebook by two-dimensional example

x2 Codebook

x1

¾ The SOM could be used for codebook creation, coding, and adaptation

© Andreas König Slide 3-7

Neurocomputing
Image Compression by VQ NN-Models

¾ Computational assessment of real-time requirements for video phone

data, restricting to just the luminance component:
pixel pixel
352 × 288 = 101376
frame frame
¾ Assuming 25 frames/s:

pixel frames pixel bit

101376 × 25 = 534400 = 20276200
frame s s s
¾ Number n of 64k bit/s channels required: n = 316,8
¾ Choosing 8x8 pixel blocks for vector quantization gives:

(352 / 8)× (288 / 8) blocks = 1584 blocks

frame frame

blocks frames blocks

1584 × 25 = 39600
frame s s
© Andreas König Slide 3-8

4
 Neurocomputing
Image Compression by VQ NN-Models

¾ The available processing time for each block is:

1 s µs
t= ≅ 25
39600 block block
¾ Assuming a codebook size of 1024, the sequential time for block
comparison computes to:
25 µs
tcod = = 24,4ns
1024 block
¾ A block consists of 8x8=64 pixels, each requiring subtraction, squaring,
and accumulation:
24.4 ns
tcod _ pix = = 375 ps
64 block

¾ (Still) worth to consider parallel implementation

¾ Fast codebook generation or even more adaptation would support this, too

© Andreas König Slide 3-9

Neurocomputing
Image Compression by VQ NN-Models

¾ Limits: compr. 50, 35-38 dB

¾ Commonly, VQ is applied
together with some transform
coding in a more complex
codec
¾ Example of CVPR group, Uni
Bonn, with Wavelet transform
¾ DFT, DCT, PCA, or non-linear
transformations are applicable
¾ VQ is used for transform
coefficients
¾ Errors are less visible after
inverse transformation
¾ Better subjective image quality

¾ More demanding and complex

processing scheme

© Andreas König Slide 3-10

5
 Neurocomputing
Image Compression by VQ NN-Models

¾ Concept of a Teleconference System suggested by CVPR group, Uni

Bonn with codebook adaptation:

© Andreas König Slide 3-11

Neurocomputing
Image Compression by VQ NN-Models

¾ Research concept of a codec including the SYNAPSE-1 neurocomputer

pursued by CVPR group, Uni Bonn with codebook adaptation:

© Andreas König Slide 3-12

6
 Neurocomputing
Image Compression by VQ/MLP NN-Models

¾ Two ANN in a hybrid VQ/transform coding approach (ICANN´91):

¾ Training of MLP in autoassociative mode and bottleneck topology

¾ Training SOM on MLP hidden layer output for VQ
¾ Applying forward MLP/SOM-VQ and inverse MLP/SOM-VQ:

© Andreas König Slide 3-13

Neurocomputing
Image Compression by VQ/MLP NN-Models

¾ Tentative computational assessment of real-time requirements:

¾ Assuming 64x8 input/hidden layer neurons in the MLP amounts to 512

MAC and 8 NL applications per Block in the forward transformation

¾ Assuming 25µs/Block amounts to 48ns/MAC requirement followed by

the VQ computation

¾ 8x64 hidden/output layer neurons in the MLP amount to 512 MAC and
8 NL applications per Block in the inverse transformation

¾ It must be distinguished, whether separate HW resources are available

and pipelining is possible

¾ Adaptation now involves the MLP transform mapping and the

codebook !
© Andreas König Slide 3-14

7
 Neurocomputing
OCR NN-Models

Prof. A. König
Integrierte Sensorsysteme
TU Kaiserslautern

67663 Kaiserslautern

MLP
Segmentation
of field and of
chars/digits

© Andreas König Slide 3-15

Neurocomputing
OCR NN-Models

¾ Hypothesis: 1 letter or slip has to be processed per second and

segmentation/preprocessing requires 0.5s
¾ Further, 10 digits have to be recognized per letter/slip
s letter ms
0.5 × = 50
letter 10 × digits digit
¾ First layer of ANN computes 256x64 synapses, amounts to 16384 MAC
¾ Additionally, 64 NL applications required
¾ Second layer of ANN computes 64x64 synapses (4096 MAC)
¾ Additionally, 64 NL applications required
¾ Last layer of ANN computes 64x64 synapses (640 MAC)
¾ Additionally, NL optional max-of-10 required
¾ Amounts to 21120 MAC, 128 NL, and 1 max-of-10 computation
µs
¾ Time required per MAC: ≅ 2.4
MAC
¾ In (easy) DSP reach
© Andreas König Slide 3-16

8
 Neurocomputing
OCR NN-Models

© Andreas König Slide 3-17

Neurocomputing
OCR NN-Models

¾ Application of a complex ANN, denoted as LeNET for OCR:

¾ By Le Cun, Denker,
Henderson, Howard, Hubbard,
Jackel (Neural Computation 1
(1989))
¾ Layers(Neurons:
256x768x192x30x10
¾ Amounts (theoretically) to:
350,124 weights !
¾ Concept of weight sharing in
lower levels reduces to 9,960
¾ Further, concept of receptive
fields employed
¾ Last layers fully connected !
¾ Practically applied and
dedicated mixed-signal HW
build by AT&T
© Andreas König Slide 3-18

9
 Neurocomputing
Image Compression by FFR NN-Models

¾ Multiple ANNs in a hierarchical recognition based coding approach of

BT (ICANN´91):

¾ Approach exploits properties of human perception

¾ Facial features are much more important than background details
¾ Speaker independent facial recognition approach using MLPs
¾ MLPs dedicated to left/right eye and mouth, 144 coarse search positions
per frame, 4x4 field, candidates for fine search, rule-based post processing
¾ Training data extracted by extensive human labor from image sequences !
¾ MLPs trained a priori, forward phase implemented in dedicated hardware
for real-time scanning (HANNIBAL-chip of BT)
© Andreas König Slide 3-19

Neurocomputing
Image Compression by FFR NN-Models

¾ Hierarchical facial feature detection approach, elaborating robust

detection regions:

¾ High resolution coding of facial feature regions, coarse coding of

background
© Andreas König Slide 3-20

10
 Neurocomputing
Face Recognition NN-Models
¾ Recognition or (biometric) identification of persons is becoming more
and more important task
¾ Many methods, e.g., fingerprint recognition, are employed
¾ Vision-based face recognition has been investigated for long time:

¾ Data scaled to 30x30 pixels

¾ Normalized (eye location)
¾ 30x30/40/10 MLP trained in
[Zell 94]
¾ Accuracy: 97%
¾ However, simple approach
not generalizing easily
¾ Products: ZN Bochum (v.d.
Malsburg, Würtz et al.)
Jets, Elastic Graph Matching
¾ „Conventional“ neural HW
somewhat inappropriate for
algorithms
© Andreas König Slide 3-21

Neurocomputing
Visual & Acoustical Inspection NN-Models
¾ Tablette blister inspection for completeness in production speed:

• Empty blister chamber due

to filling/assembly problem
• Detection and discarding
before foil sealing required

Stichprobe V arianz,1 V isor,1 Weight

(Input),1 Watcher,1

ii ii 19
15
69128 47
575 33
20618540
129 135612
126
21
42
55
13071
45 62 29
66
87
18 48
677
34
16
118
57
17
114115
127 68 65 27
41
197
9 051 119
73
10031
112 92
737104
116
70 90
5958
83372
44
101
117
14
91
60
98
30 11
125
76
842839123
113
879
96
53 99 2 2263
121
94 124
80
52 49
81 7835
3824
108
47432
46
105
109
2586
88 120
89 54
122 50
111
3623 110
102 67103
43
106
107
16493
26 95
i ii 82
RNN (L),1

Speichern
(Datei),1
i
i

ANN classifier
© Andreas König Slide 3-22

11
 Neurocomputing
Visual & Acoustical Inspection NN-Models
¾ Acoustical inspection and sound classification by a neural classifier
(NEUROKLA/MEDAV GmbH (1993):

¾ Measurement device & system

¾ Integrated from off-the-shelf
conventional computing
components
¾ Real-time acquisition, FFT &
feature extraction, learning &
classification

¾ Based on simple percetron-

like network and delta-rule
¾ Network trained by feature
data from recorded
samples

© Andreas König Slide 3-23

Neurocomputing
Visual & Acoustical Inspection NN-Models
¾ Feature extraction and classification , e.g., for glass inspection (SENSIG):

¾ Acoustical tasks usually

less demanding than vision
¾ In easy reach for (flexible)
DSP systems

¾ Similar applications: E-
Nose, E-Tongue, other
multisensor applications
¾ Different integration needs
¾ On-line adaptive filtering
has special requirements
© Andreas König Slide 3-24

12
 Neurocomputing
Summary NN-Models

Inspection Man-Machine
Interface
Automotive
Applications
Applications
of
ofIntegrated
IntegratedSensor
Sensor
Systems
Systems
Biometry Robot

Vision

© Andreas König Slide 3-25

Neurocomputing
Summary NN-Models

¾ Neurocomputing in technical application was demonstrated for a

selection of relevant tasks

¾ Evidently, the applications predominantly are in need of hybrid

architectures and system implementations

¾ Signal processing, feature extraction, and multiple network model

application is often required

¾ Real-time requirements do not justify dedicated hardware implemen-

tation in all cases, programmable off-the-shelf systems preferable

¾ Integration needs can give rise to renewed interest in dedicated HW

In the next step, the required user interaction for human-based

configuration, observation, and interfering/interaction in neural
network learning for successful application will be regarded

© Andreas König Slide 3-26

13