1332 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—II: EXPRESS BRIEFS, VOL. 64, NO.
11, NOVEMBER 2017
Energy-Efficient Design of Processing Element for
Convolutional Neural Network
Yeongjae Choi, Student Member, IEEE, Dongmyung Bae, Jaehyeong Sim, Student Member, IEEE,
Seungkyu Choi, Minhye Kim, Student Member, IEEE, and Lee-Sup Kim, Senior Member, IEEE
Abstract—Convolutional neural network (CNN) is the most
prominent algorithm for its wide usage and good performance.
Despite the fact that the processing element (PE) plays an impor-
tant role in CNN processing, there has been no study focusing on
PE design optimized for state-of-the-art CNN algorithms. In this
brief, we propose an optimal PE implementation including a data
representation scheme, circuit block configurations, and control
signals for energy-efficient CNN. To validate the excellence of this
brief, we compared our proposed design with several previous
methods, and fabricated a silicon chip. The software simulation
results demonstrated that we can reduce 54% of data bit lengths
with negligible accuracy loss. Our optimization on PE achieves to
save computing power up to 47%, and an accelerator exploiting
our method shows superior results in terms of power, area, and
external DRAM access.
Index Terms—Neuromorphic computing, convolutional neural
network, processing element, neural network processor.
I. I NTRODUCTION
ONVOLUTIONAL neural network (CNN) is the most
C advanced network that is accurate enough to be practi-
cally utilized in user-level applications, such as visual analysis,
image/speech recognition, and natural language processing.
Its correctness and versatility have attracted the interest of
researchers who have been working on developing better CNN
models for wider applications. At the same time, with the wave
of IoT and smart mobile devices, there is a strong need for
energy-efficient CNN hardware in resource-hungry environ-
ments where powerful GPUs and general purpose processors
are hardly used. Hence, designing a dedicated accelerator for
CNN is particularly crucial.
A fundamental operation of CNN is multiplication and
accumulation (MAC). It is admittedly simple, but iterations
Manuscript received December 23, 2016; revised March 16, 2017;
accepted April 1, 2017. Date of publication April 6, 2017; date of cur-
rent version November 1, 2017. This work was supported by the National
Research Foundation of Korea through the Korea Government (MSIP) under
Grants NRF-2014R1A2A1A05004316 and 2010-0028680. This brief was
recommended by Associate Editor C.-T. Cheng. (Corresponding author:
Lee-Sup Kim.)
The authors are with the School of Electrical Engineering, Korea
Advanced Institute of Science and Technology, Daejeon 34141,
South Korea (e-mail: yjchoi@vlsi2.kaist.ac.kr; baedm12@vlsi2.kaist.ac.kr;
jhsim@mvlsi.kaist.ac.kr; skchoi@vlsi2.kaist.ac.kr; mhkim@mvlsi.kaist.ac.kr;
leesup@kaist.ac.kr).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TCSII.2017.2691771
1549-7747 c 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: Chungbuk National Univ. Downloaded on April 03,2020 at 09:48:06 UTC from IEEE Xplore. Restrictions apply.
of MAC process are massive. Especially in image recogni-
tion, over ten billion of MAC computations are necessary to
perform a highly accurate CNN network [1]. This induces huge
energy consumption on logical operations. Meanwhile, data
transfer in the on/off chip memory system becomes the other
main source of energy consumption. This problem is criti-
cal in a dedicated accelerator in that current CMOS process
technology cannot provide sufficient on-chip SRAMs to store
CNN data which is megabyte-scale. Therefore, a large amount
of data communication between a chip and external DRAM
are compulsory, which causes enormous energy dissipation.
In sum, the energy crisis that can occur due to both logical
computation and memory access must be overcome to effi-
ciently employ CNN in a consumer device with an embedded
accelerator.
Several previous studies suggested dedicated processors for
CNN. Du et al. [2] proposed a 2-D systolic array CNN
processor with an inter-processing element (PE) data prop-
agation scheme, and the memory bandwidth was successfully
reduced. In [3], tile-based computing was proposed to smartly
manage data movements inside the processor. Neuromorphic
system effectively utilizing bit truncation technique was intro-
duced in [4]. Further to these, Pullini et al. [5] presented Mia
Wallace, MPSoC designed to flexibly execute various CNN
models in low power.
While effective CNN architectures have been actively stud-
ied, research on PEs themselves, the core parts of the system,
has not been conducted. In this brief, we argue that the key
solution of the energy problem requires an efficient data rep-
resentation scheme with appropriate optimizations of a PE.
Since all calculations of CNN algorithm occur inside the PE,
careful consideration on the PE design is essential to modify
data representation methods while maintaining an acceptable
accuracy. Our contributions to CNN PE design are summarized
by following three points.
1) We analyze inputs/outputs of PE blocks during CNN
processing, and propose optimized scheme to perform
state-of-the-art CNN networks. By exploiting the obser-
vation that the dynamic range of two PE inputs are
distinct, we introduce a heterogeneous representation
(HR). To express the feature efficiently, Significant bits
securing encoding (SSE) is presented.
2) We verify that CNN arithmetic operations based on the
signed magnitude are better in terms of energy consump-
tion than the conventional two’s complement method,
and utilize the signed magnitude system in our PE
design.
CHOI et al.: ENERGY-EFFICIENT DESIGN OF PE FOR CNN
Fig. 1. Overall architecture of our processor and the proposed PE with HR
scheme.
3) An optimized PE design suitable for performing all
layers in the CNN is introduced. Utilizing our HR
method, we design a PE to execute state-of-the-art CNN
algorithms in an energy-efficient manner.
From the evaluation results, we showed that our work can
effectively reduce the external memory requirement as well as
operating power of a CNN accelerator.
The remainder of this brief is structured as follows.
Section II describes intuitions on our data representation and
details of the proposed schemes. In Section III, a concrete PE
design and data path controls are explained. Evaluation set-
tings and results are presented in Section IV. Then, we make
our conclusions in Section V.
II. O PTIMIZATIONS ON THE DATA R EPRESENTATION
A CNN accelerator should be designed with consideration
not only of energy-efficiency but also network flexibility for
diverse applications. Thus, analysis and implementation must
cover the worst possible scenario to correspond to every kind
of network. PE generally takes two inputs, features and ker-
nels, from their designated data paths. Compared to the kernels
typically bounded within [−1, +1] during the network train-
ings, the dynamic ranges of the features are extremely broad.
Previous approaches have utilized either floating point (FLP)
or variants of fixed point (FXP) to express all kinds of data.
From the fact that two input paths show totally dissimilar pat-
terns of values, we can get an intuition to use different data
types suitable for each path.
A. Heterogeneous Representation (HR)
Due to the high complexity and energy consumption of the
FLP MAC block, FXP MAC is more energy-efficient. On the
other hand, the FXP-based system lacks the ability to express
a wide range of values, which is essential in CNN convolu-
tion. Hence, to exploit the advantages of both FXP’s efficiency
and FLP’s flexibility, we assign our novel data encoding on
expressing the feature while using dual-mode FXP system on
Authorized licensed use limited to: Chungbuk National Univ. Downloaded on April 03,2020 at 09:48:06 UTC from IEEE Xplore. Restrictions apply.
1333
Fig. 2. Brief outline and example cases of SSE. An SSE number consists
of three parts: sign bit, most significant position (MSP), and significands.
Algorithm 1 SSE Generation
1: Given (l+1) bits FXP feature F = sfl−1 fl−2 · · · f1 f0 , where
s and fi denote the sign bit and ith magnitude bit, respec-
tively. Bits are represented in a signed magnitude system.
2: Detect the first 1, the most significant position (MSP),
fm , by checking bits from fl−1 to fmax(g,l−2n ) . (n is the
number of bits indicating MSP. g is allocated bits for the
significand.) If there is no 1 within the searching range,
MSP becomes 0.
3: Save the position of the first 1, MSP, in binary n bits.
pn−1 pn−2 · · · p0 = m − max(g, l − 2n ) + 1
4: Save g bits next to the MSP in significands.
qg−1 qg−2 · · · q0 = fm−1 fm−2 · · · fm−g
5: Final encoding format.
{s, pn−1 pn−2 · · · p0 , qg−1 qg−2 · · · q0 } : (1 + n + g) bits.
kernels, constant, and accumulation register. Fig. 1 illustrates
the architecture of our processor and the outline of HR.
1) Significant Bits Securing Encoding (SSE): As comput-
ing results are accumulated in the register of each PE until
an assigned convolution window is finished, the data is com-
pressed only when the window process is over. Because of
the error-resiliency inherent in CNN algorithms, a reasonable
approximation scarcely affects the overall accuracy. Since we
exploit FXP MAC for the internal data of PE, an appropriate
encoding scheme for the FXP values is necessary. From the
simple but powerful principle that bits near the MSB are of
importance, we developed Significant bits securing encoding
(SSE) to effectively describe the features of CNN algorithms
(Fig. 2). The difference between SSE and FLP lies in the
medium which they use to compress the original data into sig-
nificands. FLP controls ‘the position of a decimal point’ using
exponent, but SSE utilizes ‘the relative position of the first one’
compared to the original MSB of full fixed point word. By lim-
iting the representable range appropriate for CNN algorithms,
SSE can reduce hardware implementation cost compared to
the FLP system. The detail of SSE generation is described in
Algorithm 1.
2) Dual-Mode Fixed Point: Besides the feature data, there
are kernels and constants in convolution, and we use the dual-
mode fixed point to express them. As mentioned before, their
dynamic range is strictly narrow so that the powerful FLP
style is excessive. Furthermore, assigning the FXP system to
one of the data paths enables an easy implementation of the
1334 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—II: EXPRESS BRIEFS, VOL. 64, NO. 11, NOVEMBER 2017
FXP MAC block. As all data go through multiplication at
first, SSE features can be simply joined to FXP MAC during
the multiplication with the aid of a small shifter. Since the
majority of data entering this path are kernels of convolution
or fully connected layers, only the lower half corresponding to
the fractional part are usually turned on. When large constants
are rarely needed, full bits including the integer section are
employed. This flexible utilization does not require complex
controls because the granularity is large, and the occurrence
of configuration change is scarce.
This FXP path is imperative for carrying out minor lay-
ers such as normalization and avg pooling. Most prior works
have just concentrated on the convolution layer so that only
MAC between features and kernels is supported in the PE.
Nevertheless, other layers include arithmetic operations of
two features, which demand either extra computing blocks
or communications with the external host processor to han-
dle the process. The overheads of both options are too heavy.
In the case of mobile/IoT platform particularly, it might be
impossible to incorporate a host-processor providing enough
computing performance. A dual-mode FXP path could then
be utilized for accurate calculations between two features
by forwarding values in the accumulation register to MAC.
Fig. 1 shows the internal path of PE directly connecting the
accumulation register and MAC.
B. Signed Magnitude Arithmetic
In addition to the convenience of encoding features, signed
magnitude shows further advantage in CNN processing.
Han et al. [6] displayed the distribution graph of kernels in.
The distribution is typically symmetric with an axis of sym-
metry at zero, whereas features are always above zero. Hence,
there are numerous changes in multiplication results from neg-
ative to positive and vice versa. These bit flips of kernels are
even propagated through the MAC block to the accumulation
register, affecting the overall circuits of the PE.
Commonly, processing systems employ two’s complement
arithmetic because it is better at handling both positive and
negative numbers than one’s complement or the signed magni-
tude expression. However, frequent transitions of sign in small
magnitude numbers cause massive but needless bit flips in
two’s complement schemes in contrast with the signed mag-
nitude system (Fig. 7(a)). Switching power from the flips
deteriorates overall power consumption in data buses, regis-
ters, and PEs. Also considering the fact that a typical CNN
PE already has a comparator for pooling operations [2], signed
magnitude representation does not incur additional circuit
overhead in our system. As a result, we chose the signed
magnitude as the basic data format instead.
III. PE D ESIGN
A. Heterogeneous Data Paths
When SSE feature enters the PE, multiplication is conducted
first. Applying heterogeneous data paths to FXP MAC, simple
shift and multiplication circuits are necessary. As described in
Fig. 3 (b), bit shifting according to MSP after the multiplica-
tion between the significands of SSE and FXP data naturally
performs SSE decoding. A crucial advantage here is that the
Authorized licensed use limited to: Chungbuk National Univ. Downloaded on April 03,2020 at 09:48:06 UTC from IEEE Xplore. Restrictions apply.
Fig. 3. (a) PE design (b) SSE to FXP interflow during multiplication, and
(c) SSE encoder.
length of multiplication becomes much shorter than the basic
FXP PE structure (l × l operation turns into l × (g + 1).) This
greatly reduces the power consumption spent on multiplica-
tion. A simple one-detection circuit and a shifter comprise
the SSE encoding block (Fig. 3 (c)), which incurs negligible
power and area overheads compared to the savings thanks to
SSE coding. The overhead of the decoding section could be
cut down additionally by applying a gating technique, which
stops decoding before the convolution window is finished.
B. Processing Element Design
The bit length of the internal PE paths is irrelevant to
the amount of data transmission between on-chip SRAM
and PEs because of the decoding and encoding procedures.
Consequently, we could exploit FXP bits which cover the
total dynamic ranges of CNN computing. From the analysis
results plotted in Fig. 6(a), 24 bits are sufficient to express the
internal values. To realize a dual-mode FXP without tough
wiring efforts, 12 bits, half of the total 24 bits, are connected
between the on-chip SRAM and the PE. Kernel transmission
just uses the lower 12 bits of FXP register in the PE. In the
case of a constant, signals from the SRAM are saved in the
upper and lower 12 bits sequentially during two clocks.
Two input paths of the HR system are not interchangeable.
Hence, we have to modify the basic MAC circuits exploited in
previous works to support all operations in CNN algorithms
other than convolution. As a consequence, our PE includes
a constant register and MUXes for the elaborate controls of
arithmetic. Our PE design is illustrated in Fig. 3(a).
Convolution and Fully Connected Layer: Both types of lay-
ers are composed of intensive MACs with a constant addition.
Fig. 4 explains how to run MACs and a constant addition in
our PE. To deal with the constant addition, the value from
the accumulation register is multiplied by 1. Then, a constant
register is set and connected to the addition block.
Pooling Layer: Max or average pooling is commonly
exploited in state-of-the-art CNN models. When data in the
pooling window goes into the PE sequentially, successive com-
parisons between the accumulation register and SSE inputs
perform max pooling. In our design, the comparator can
work for both signed magnitude arithmetic and max com-
parison in the same manner. Only the first 11 bits of the
internal PE paths are activated during the max pooling process.
CHOI et al.: ENERGY-EFFICIENT DESIGN OF PE FOR CNN
Fig. 4. Data path controls in each arithmetic operation.
Fig. 5. Representation schemes used in comparisons: Heterogeneous rep-
resentation, floating point baseline (FLPB), fixed point baseline (FXPB),
Minifloat (MF) [7] and Dynamic fixed point (DFXP) [7].
Average pooling can be decomposed into convolutions and
a constant multiplication which correspond to accumulation
of features in the pooling window and a multiplication with
1/‘pooling window size’, respectively.
Normalization Layer: In [2], the decomposition of the
normalization layer into several consecutive layers, such as
addition, multiplication, division, and exponential operation,
was discussed. Excluding additions and multiplications, oth-
ers are involved with non-linear operations, demanding costly
components. To deal with the non-linear computing, a look-
up-table (LUT) for SSE was designed. Based on the accuracy
simulations, an optimized LUT for our CNN benchmarks is
implemented. When non-linear calculations are necessary, SSE
from the previous layer enters the LUT, and output comes out
through the LUT buffer.
IV. E VALUATION
A. Evaluation Settings
To analyze data and measure the accuracy of CNN, we
used a Caffe simulator [8] and ran 1000 test images from the
ILSVRC dataset [9]. All arithmetic operations in the Caffe
Authorized licensed use limited to: Chungbuk National Univ. Downloaded on April 03,2020 at 09:48:06 UTC from IEEE Xplore. Restrictions apply.
1335
Fig. 6. Accuracy running on 32 bits floating point set as a baseline. Relative
accuracy of benchmark for: (a) FXPB; (b) Minifloat; (c) SSE in HR system.
were processed with our customized function. The majority
of CNN networks from recent studies are variants of several
state-of-the-art CNN models: AlexNet [10], GoogleNet [11],
VGG-16 [1], and VGG-19. Therefore, we organize them as
our benchmark.
To validate the effect of our proposal in a CNN accelerator,
hardware cost simulations for 5 types of data representation
schemes were executed (Fig. 5). RTL-level functional exper-
iments were conducted using a Cadence NCVerilog. For a
concrete power simulation, the Synopsys PrimeTime PX tool
was utilized. A chip was fabricated in Samsung 65nm CMOS
process.
B. Accuracy Evaluation
Accuracy curves of Fig. 6 show that our HR is the most
preferable. As articulated in Section II, important charac-
teristics on expressing kernels and features are contrasting.
Fig. 6(a) states that CNN accuracy is limited by dynamic
ranges of feature. On the other hand, the accuracy of Minifloat
is restricted by the length of bits storing kernel information.
This can be verified in Fig. 6(b) that the accuracy of Minifloat
starts to deteriorate in longer bit width compared to that of
HR, which uses dual-mode FXP to represent kernels. Dynamic
fixed point also aims to take advantages of both FXP and FLP
systems. However, it needs at least 13 bits to handle CNN
benchmark same to ours [12]. In sum, 11 bits of SSE are suf-
ficient in HR system, and this is reduction in the number of
bits (Fig. 6(c)). Due to the stochastic behaviors of CNN algo-
rithm, small accuracy improvements can be observed when
using 11-bit SSE compared to the 24-bit fixed point case.
1336 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS—II: EXPRESS BRIEFS, VOL. 64, NO. 11, NOVEMBER 2017
TABLE I
C OMPARISONS A MONG DATA R EPRESENTATION S CHEMES
Fig. 7. (a) Distributions of bit flips on accumulation register for Alexnet.
(b) Power and area comparison of PEs. FXP baseline utilizes 24-bit FXP, and
the rest are based on HR scheme.
Fig. 8. (a) Chip die photo. (b) Measurement system using FPGA.
C. Hardware Evaluation
Processing element: In spite of additional control signals
and MUXes for HR data paths, the bit reduction and simpli-
fied multiplication of our suggestion make remarkable power
and area savings. We implement a conventional PE including
a FXPB MAC, the proposed PE, and an SSE PE using two’s
complement instead of a signed magnitude system. Fig. 7(b)
shows that the best power consumption was achieved by our
PE in signed magnitude. Due to extra control signals and data
paths, a very small area overhead is added for the signed
magnitude system, but it is negligible to affect the design
choice.
Comparisons in accelerator level: Five accelerators based
on each data representation (Fig. 5) were synthesized to have
equal number of PEs (128) and size of SRAM (312KB). The
minimum number of bits showing accuracy degradation under
3% were utilized. CNN data extracted from Caffe were prop-
agated to the synthesized accelerators. Table I illustrates our
achievements. External DRAM accesses of kernels are pro-
portional to the bit length so that it is possible to reduce
15 ∼ 54% of kernel transmission. In the case of feature,
DRAM accesses only arise when intermediate data between
layers exceed the capacity of on-chip feature SRAM. The
dataflow used in DRAM access simulation is based on tile-
based computing introduced in [3]. As a result, the decrease in
data bit length more significantly affects the amount of feature
DRAM access (up to 60%).
Authorized licensed use limited to: Chungbuk National Univ. Downloaded on April 03,2020 at 09:48:06 UTC from IEEE Xplore. Restrictions apply.
TABLE II
C HIP C HARACTERISTIC
Chip measurement: A functionality of our silicon chip was
tested. Its performance was measured (Fig. 8(b)), and is shown
in Table II. Though the test chip was fabricated in small scale
to deal with simpler CNN networks, result comparisons in
Table I demonstrate the effect of our work in actual silicon
level.
V. C ONCLUSION
Despite the significance of the PE in a CNN accelerator,
optimization has not been considered enough in previous stud-
ies. In this brief, we intensively researched state-of-the-art
CNN networks and proposed an efficient HR system including
SSE. To support all operations in CNN algorithms, detailed PE
data paths and control signals based on HR were implemented.
From the evaluation, our SSE is able to compress the origi-
nal feature by 54% with less than 3% accuracy loss, which
leads to 60% reduction of external DRAM accesses. Our PE
design for HR shows outstanding improvements, which con-
sumes 47% less power and occupies 16% less area. Overall
comparisons in accelerator level demonstrate that HR system
is superior to other representation methods.
ACKNOWLEDGMENT
Test chip fabrication was supported by MPW of IDEC.
R EFERENCES
[1] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” CoRR, vol. abs/1409.1556, 2014.
[2] Z. Du et al., “Shidiannao: Shifting vision processing closer to the sen-
sor,” in Proc. 42nd Annu. Int. Symp. Comput. Archit. (ISCA), Portland,
OR, USA, 2015, pp. 92–104.
[3] J. Sim et al., “14.6 A 1.42tops/w deep convolutional neural network
recognition processor for intelligent IoE systems,” in Proc. IEEE Int.
Solid-State Circuits Conf. (ISSCC), San Francisco, CA, USA, Jan. 2016,
pp. 264–265.
[4] S. Venkataramani, A. Ranjan, K. Roy, and A. Raghunathan, “AxNN:
Energy-efficient neuromorphic systems using approximate computing,”
in Proc. Int. Symp. Low Power Electron. Design (ISLPED), 2014,
pp. 27–32.
[5] A. Pullini et al., “A heterogeneous multi-core system-on-chip for energy
efficient brain inspired computing,” IEEE Trans. Circuits Syst. II, Exp.
Briefs, to be published, doi: 10.1109/TCSII.2017.2652982.
[6] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing
deep neural networks with pruning, trained quantization and Huffman
coding,” CoRR, vol. abs/1510.00149, 2015.
[7] P. Gysel, “Ristretto: Hardware-oriented approximation of convolutional
neural networks,” CoRR, vol. abs/1605.06402, 2016.
[8] Y. Jia et al., “Caffe: Convolutional architecture for fast feature embed-
ding,” CoRR, vol. abs/1408.5093, 2014.
[9] O. Russakovsky et al., “ImageNet large scale visual recognition chal-
lenge,” Int. J. Comput. Vis., vol. 115, no. 3, pp. 211–252, 2015.
[10] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classifica-
tion with deep convolutional neural networks,” in Proc. NIPS, 2012,
pp. 1106–1114.
[11] C. Szegedy et al., “Going deeper with convolutions,” in Proc. Comput.
Vis. Pattern Recognit. (CVPR), Boston, MA, USA, 2015, pp. 1–9.
[12] P. Judd, J. Albericio, T. Hetherington, T. M. Aamodtand, and
A. Moshovos, “Stripes: Bit-serial deep neural network computing,” in
Proc. 49th Annu. IEEE/ACM Int. Symp. Microarchit. (MICRO), Taipei,
Taiwan, Oct. 2016, pp. 1–12.