1642 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO.

5, MAY 2018

Data and Hardware Efficient Design for

Convolutional Neural Network
Yue-Jin Lin and Tian Sheuan Chang , Senior Member, IEEE

Abstract— Hardware design of deep convolutional neural net-

works (CNNs) faces challenges of high computational complexity
and data bandwidth as well as huge divergence in different
CNN network layers, in which the throughput of the convolu-
tional layer would be bounded by available hardware resource,
and throughput of the fully connected layer would be bounded
by available data bandwidth. Thus, a highly flexible and efficient
design is desired to meet these needs. This paper presents an
end-to-end CNN accelerator that maximizes hardware utilization
with run-time configurations of different kernel sizes. It also
minimizes data bandwidth with the output first strategy to
improve the data reuse of the convolutional layers by up to
300× ∼ 600× compared with the non-reused case. The whole
CNN implementation of the target network is generated optimally
for both hardware and data efficiency under design resource Fig. 1. Computational complexity and data amount of the popular networks,
constraints, which can be run-time reconfigured by the layer where C1, C2.. denote the convolutional layer, and F6, F7…denote the fully
optimized parameters to achieve real-time and end-to-end CNN connected layer.
acceleration. An implementation example for AlexNet consumes
a 1.783 M gate count for 216 MACs and a 142.64 kb internal
buffer with TSMC 40 nm process, and achieves 99.7 and 61.6 f/s
under 454 MHz clock frequency for the convolutional layers and
the whole AlexNet, respectively.
Index Terms— Convolutional neural networks (CNNs),
hardware design.

I. I NTRODUCTION

I N RECENT years, deep convolutional neural net-

works (CNNs) are widely adopted in the recogni-
tion [2]–[6], detection [7]–[11], tracking [14], [15], scene
labeling [17], [18], pose estimation [21], [22], industrial
inspection [23], [24], and other computer vision fields because
of it unprecedented accuracies that are even better than
human accuracy [5]. However, computations of convolutional Fig. 2. Distribution of computation and data of different networks.
neural network demands over several Giga Multiplications-
and-Accumulations (GMACs) and millions of data amount per variations of these demands between different layers cause
layer as shown in Fig. 1. The high computational complexity the regular hardware CNN implementation more challenging.
and data amount demands significant hardware and bandwidth Fig. 2 shows the demands of complexity and data access
resources for real time applications. Furthermore, the high in different layers. This figure shows that the convolutional
layer is dominated by computation and the fully connected
Manuscript received June 16, 2017; revised August 11, 2017,
September 5, 2017, and September 22, 2017; accepted October 2, 2017. Date layer is dominated by data access. Fig. 3 shows that the
of publication November 7, 2017; date of current version April 2, 2018. data distributions among layers is quite different even these
This work was supported in part by the Ministry of Science and Technology, are all convolutional layers. In which, the last few layers
Taiwan, under Grant 104-2221-E-009-171-MY2 and Grant 105-2119-M-
009-009, and in part by the Research of Excellence Program under Grant with smaller image sizes have more weights. Thus, an effi-
106-2633-E-009-001. This paper was recommended by Associate Editor cient CNN implementation needs a flexible design with layer
T. Serrano-Gotarredona. (Corresponding author: Tian Sheuan Chang.) adaptive approach to accommodate these variations. In brief,
Y.-J. Lin is with Novatek, Taiwan.
T. S. Chang is with the Department of Electronics Engineering, the trade-off between flexibility, performance, and cost is the
National Chiao Tung University, Taiwan (e-mail: f34150@yahoo.com.tw; major design consideration on the CNN implementation.
tschang@mail.nctu.edu.tw) To address these complexity and bandwidth issues, several
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org. works [13], [26]–[33] have been proposed. However, most
Digital Object Identifier 10.1109/TCSI.2017.2759803 of these works focus on the convolutional part only and
1549-8328 © 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
LIN AND CHANG: DATA AND HARDWARE EFFICIENT DESIGN FOR CNN
Fig. 3. Distribution of different data type of different networks.
Fig. 4. Typical CNN structure.
fail to address layer specific needs in a hardware and data
efficient way. Thus, we proposed a CNN accelerator design
with the following contributions:
• An efficient output first strategy to reuse data of the
convolutional layers by up to 300× ∼ 600× compared
to the non-reused case.
• An optimized CNN accelerator with run-time configu-
rations of multiple kernel sizes to maximize hardware
utilization.
• An example implementation of this architecture on
AlexNet [2] to achieve state-of-the-art throughput and
area efficiency.
The rest of the paper is organized as following. Section II
gives a short overview of the CNN algorithm. Section III
shows the related works on CNN hardware designs.
Section IV presents the analysis of computational complexity
and data access. Based on this analysis, Section V shows our
proposed hardware design and its generators. The implementa-
tion results and comparisons are shown in Section VI. Finally,
this paper is concluded in Section VII.
II. OVERVIEW OF T HE CNN A LGORITHM
Fig. 4 shows an example of a typical CNN structure, which
includes the convolutional layer, pooling layer, fully connected
layer, and activation function. All these layers would be
implemented in this paper.
The convolutional layer uses numerous trained filters
inspired from the receptive fields of the visual cortex to extract
features from low level local ones (edge, corner, line…) to
high level abstract ones (category…) [35]. The convolutional
layer behaves sparser than a pure neural network because
of its weight sharing property. This weight sharing property
makes convolutional layer more relieved from high weight
bandwidth than that in a pure neural network. The equation
Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
1643
Fig. 5. Computation of the convolutional layer.
Fig. 6. An example of the maximum pooling.
of a convolutional layer is defined by

Ouputo (r, c) = Weight o,i ⊗ Inputi (S ∗ r, S ∗ c) + Biaso
i
+Bias
 o
= Weighto,i (x, y)
i x y
∗ Inputi (S ∗ r + x, S ∗ c + y) + Biaso (1)
where “Input” is the input feature map, “Output” is the output
feature map, and “Weight” is the filter kernel. The output is
the sum of the all convolutional results and the bias term. The
successive convolution will have a stride distance S (stride of
shifting window) between each other. Fig. 5 shows an example
with S equal to two.
The pooling layer executes the down-sampling operation
that not only reduces the feature map size but also improves the
translational invariance of features. The typical pooling types
are maximum pooling and average pooling that calculate the
maximum value and the average value from the corresponding
kernels respectively. Fig. 6 shows an example of maximum
pooling that has the kernel size 2×2 and the stride size two.
The fully connected layer uses lots of trained weights to
decide the category likelihood for the classification. The fully
connected layer is just a pure neural network which has numer-
ous weight data and thus demands high weight bandwidth. The
equation of a fully connected layer is defined by

Ouput (o) = Weight (o, i) ∗ Input (i) + Bias (o) (2)
i
where “Input” is the input node, “Output” is the output node,
and “Weight” is the weight synapse. The output is the sum of
all convolution results and the bias term as shown in Fig. 7.
Activation function tries to represent abstractly the expected
fire rate of the biological neurons [36]. Moreover, adding
non-linear activation functions after convolutional layers and
fully connected layers makes whole network more possible to
approach the real model than just a linear composition model.
1644 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 5, MAY 2018
Fig. 7. Computation of the fully connected layer.
Some popular functions include the sigmoid function ( 1+e1 −x )
and the hyperbolic tangent function (tanh(x)). Recently, the
rectified linear unit, abbreviated as ReLU [37] (max(0, x)), can
have much faster training convergence [2], and thus becomes
more popular after AlexNet adopts this function. In this paper,
we will also use ReLU as our choice of the activation function.
III. R ELATED W ORK
A. FPGA Designs
nn-X is a series of designs [16], [19], [27], [38] that can
translate the software based networks to related hardware
configuration sequences. This first design, CNP [38], can
execute all networks but face the difficulty of scalability on
higher network performance. Then the NeuFlow design [16]
on FPGA and its ASIC counterpart [19] was designed to be
more scalable because of its runtime reconfigurable dataflow
architecture. The revised version of NeuFlow, nn-X [27], used
a large filter kernel size for implementation, and thus had lower
hardware utilization on the small filter kernel cases.
Beyond computation, [28] first proposed a memory-centric
design methodology that tried to minimize external com-
munication and buffer size. To handle external communica-
tion and buffer size more efficiently, they used HLS (high
level synthesis) accelerator template to find the optimized
access solution. Reference [29] had taken a step further to
optimize not only communication but also computation with
the HLS tool by the roofline model. However, the hardware
structure was optimized for only a fixed network and the
convolutional layer and it was not optimized for other kinds
of networks. Reference [30] implemented VGG [4] with
improved bandwidth and hardware utilization by the dynamic-
precision data quantization method. However, this implemen-
tation was dedicated to 3x3 convolutions and impossible to
execute other networks with different kernel size.
B. ASIC Designs
The DaDianNao series [12], [13], [39]–[41] minimized the
communication traffic by a tiling strategy and internal buffer.
In which, the most largest scale design, DaDianNao [13],
stored all weights in the on-chip eDRAM to reduce
off-chip weight access at its most. However, the modern
CNNs are too large to store all weights in the on-chip
memory. Origami [25], [31] implemented a scene labeling
network [17] with fixed 7x7 kernel size. Besides, it only
implemented the convolutional layer. EIE [26] solved the high
Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
Fig. 8. An example to map a non-unit stride convolutional kernel to multiple
unit stride convolutional kernels. Input and kernel are divided into four parts.
The parts with same symbols will be associated for convolution operation.
weight bandwidth on the fully connected layer by model
compression. Reference [32] used the dual-range multiply-
accumulate (DRMAC) to have low power convolution opera-
tion. However, it only tested on a small size dataset, MNIST,
as an example. Eyeriss [33] was a spatial array architecture
that can be mapped to wide range kernels (1 ∼ 32 × 1 ∼ 12).
Moreover, Eyeriss proposed a row stationary (RS) [42] data
flow that includes both output stationary [12], [28] and weight
stationary [27], [30], [31] to maximize the internal and exter-
nal reuse and minimize the movement energy consumption.
Envision [34] proposed a precision scalable design to achieve
ultra low power.
In summary, various designs have been proposed, but none
has a systematic analysis and approach on the design problem
to ease design generation for different networks.
IV. D ESIGN A NALYSIS
A. All Unit-Stride Convolutional Kernels
Hardware design of CNN faces high computational com-
plexity, data access and network variations. For CNN, one
of the major variations comes from different kernel sizes
and non-unit stride length. Too many possible kernel sizes
make convolutional layers hard to analyze and implement.
Convolutional kernels with non-unit stride length lead to lower
1 1
throughput about stride2 since true output data are only stride2
of all output data. Thus, to simplify design and ease analysis,
we propose to map all non-unit stride convolutional kernels
to the decomposed unit stride convolutional kernels.
Fig. 8 shows an example that maps one 6x6 convolu-
tional kernel with stride two to four decomposed 3x3 con-
volutional kernels with stride one. For a  KxK kernel

with
 K  stride
 S, the new kernel size will be K
S + i ×
S + j , i f (K %S == 0) i, j = 0, elsei, j = 0 · · · S − 1.
For a 6x6 kernel with stride 2, the new mapped kernel size
will be four (3x3). For a 7x7 kernel with stride 2 as used
in GoogleNet and ResNet, the new mapped kernel size will
be (3x3, 3x4, 4x3, 4x4). This will result in a fully usable
output. However, above nonsquare kernel size is bad for
implementation. Thus, we use zero padding forthe  kernels
 and
have a new formula for mapped kernel size as KS × KS .
Thus, for a 7×7 kernel with stride 2, we will have four (4×4)
kernels. Note this will result in slightly lower hardware uti-
lization (roughly around 70%), but is still better than hardware
utilization of the original stride one (roughly 1 S 2 , usually
<= 1/4). Thus, we will discuss only the stride one case for the
convolutional layer in the following. The original input with
index (x, y) is also divided
   to multiple smaller one with new
index (x’, y’) = ( Sx , Sy ). Each smaller input corresponds
LIN AND CHANG: DATA AND HARDWARE EFFICIENT DESIGN FOR CNN
Fig. 9. Illustration of the symbols in the convolutional model.
TABLE I
S YMBOL D EFINITIONS OF THE C ONVOLUTIONAL M ODEL
to the one new filter kernel at the same relative position for
convolution as in Fig. 8.
With above conversion, each new mapped convolution will
be executed as the normal convolution. To prepare the input
data without non-contiguous memory access, one way is to
load the original input as usual to a local input line buffer and
distribute input to different filter kernels. Their output will be
combined to generate the final output. The drawback of this
approach is extra line buffer cost, but this approach can be
integrated into current design seamlessly without extra control
burden or non-contiguous memory access.
B. Symbol Definitions for Analysis
To ease the following analysis, we adopt the symbol defi-
nitions in a tile based model as shown in Fig. 9 and Table I.
The tile based model divides the input/output/batch into non-
overlapped rectangular partition units, denoted as tiles, to ease
processing. In this analysis, the results of the fully connected
layer will be derived as the simplified version of the convo-
lutional layer because the fully connected layer is the special
case of convolutional layer (R = C = K =Kp = Sp = 1) if
the kernel size is 1x1.
C. Analysis of Computational Complexity
For one output pixel, we need all input maps (I ) and
weight kernels (K ∗ K ) as input and I ∗ K 2 MACs. Therefore,
the number of total MACs for O ∗ R ∗ C output pixels is
Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
1645
TABLE II
D ATA R EUSE P OLICY
O RC ∗ I K 2 and the computational complexity of convolu-
tional layer is I O RC K 2 . Therefore the computational com-
plexity of fully connected layer is I ∗ O.
D. Analysis of Data Access
To evaluate the degree of data reuse, we use average data
access as an index
Amount o f data per batch
Aver age data access =
layer (i nput + out put + weight)
(3)
This index measures the number of data access per unit
data. In which, the denominator is amount of input, output
and weight per batch when only accessed once. These three
quantities only depend on the network parameters, and thus
the denominator of the right hand side in Eq. (3) is actually
a fixed normalization factor. The numerator in Eq. (3) is the
final accessed input, output and weight based on different data
reuse policy and strategy as discussed below. Note, without
loss of generality, we assume that all these data have the same
precision to ease our analysis.
1) Data Reuse Policy: Table II shows the general data reuse
policy for input, output, and weight. Input data access can
be reduced by To because To different output maps reuse
these input data. For output, local accumulated partial sums
access can be reduced by Ti because different Ti input maps
can share these partial sums. Weight data access can be
reduced by Tr Tc Tb since Tr Tc pixels and Tb batches can reuse
these weight data. For the fully connected layer, we can set
Tr = Tc = 1asaspecialcase, which has much lower data reuse
possibility for weight data access.
2) Data Reuse Strategy: Based on above data reuse policy,
we can maximize one of them first as our data reuse strategy
such that we can access one of data type for only one time.
Others would basically follow the data reuse policy mentioned
above.
Table III summarizes the total data amount per batch for
different data reuse strategies. To illustrate different strategies,
Fig. 10 shows the average data access of the convolutional
layer for different strategies with AlexNet as an example.
It shows that the output first strategy has almost half of
average data access compared to the one in the input first
strategy and quarter of average data access compared to the
one in the weight first strategy. The reason that the output
first strategy has lower data amount than others is due to
the pooling layer (factor S12 in Table III) and localized partial
p
accumulated sum (avoid factor 2 in Table III for write and read
1646 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 5, MAY 2018
TABLE III
D ATA A MOUNT PER BATCH F OR T HREE D IFFERENT
D ATA R EUSE S TRATEGIES
Fig. 10. Average data access of the convolutional layer for different data
reuse strategies.
TABLE IV
E FFICIENCY R ATIO OF D ATA R EUSE F OR S INGLE BATCH C ONVOLUTIONAL
L AYERS OF THE P OPULAR N ETWORKS
partial sum) to save extra write out and read in accesses. Thus,
we choose this strategy in this paper. Though this strategy
has been used in others works, e.g. [33], [34], our analytical
equation helps find the best design point under the resource
constraints through program search.
Table IV shows the data reuse efficiency compared to the
non-reuse case (three (input + weight + partial sum) data per
MAC and data amount of output first reuse), which is around
300× to 600× depending on the network structures.
V. S YSTEM A RCHITECTURE
Fig. 11 shows the whole architecture of our CNN acceler-
ator, which includes three data buffers and address generators
for input, weight and output, respectively, a configurable
processing engine and partial result buffer. This CNN acceler-
ator can realize an end-to-end implementation which includes
the convolutional layer, pooling layer, fully connected layer,
and ReLU layer. Moreover, the convolutional layer, pooling
layer, and fully connected layer can share the same processing
engine to exploit hardware efficiency.
Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
Fig. 11. The architecture of the convolutional neural network accelerator.
Fig. 12. Data flow of the CNN accelerator.
Fig. 12 shows the data flow of the CNN accelerator has two
operating modes: the convolutional mode (convolutional layer,
pooling layer, and ReLU layer) and fully connected mode
(fully connected layer, ReLU layer, and 1x1 convolutional
layer). In which, the 1x1 convolutional layer is regarded as
the fully connected layer with many batches. For inference
of a CNN, this accelerator is operated layer by layer. The
accelerator will be run-time reconfigured according to its layer
structure through configuration data. The output of intermedi-
ate layers will be stored to external memory and read back as
next layer input.
For the proposed accelerator, we adopt the output first
strategy to design the data flow of the convolutional layer.
In this design, we set Ti = 1 to minimize input buffer size
because the overall data amount per batch is independent
of Ti 3 4 in the output first strategy. That is, we just compute
one input map in a certain computational time.
Fig. 13 shows the data flow of the convolutional layer, which
includes the initial stage for the first input map convolution,
the final stage for the last one, and the intermediate stage
for the others. In the initial stage, we set the initial partial
sums to be the bias terms. Then in the intermediate stage,
the partial sums will be stored in the partial result buffer except
the case of no pooling layer. In above exceptional case, we will
save the final convolutional outputs into the output data buffer,
which will be stored to the external memory afterwards. Above
exceptional case also applies to the final stage. In addition,
the final data path depends on if the ReLU layer exists.
The hardware of the convolutional layer can execute the
pooling layer as well if the operation is average or maximum
LIN AND CHANG: DATA AND HARDWARE EFFICIENT DESIGN FOR CNN
Fig. 13. Data flow of the convolutional layer.
Fig. 14. Two modes for the pooling layer.
Fig. 15. Data flow of the pooling layer.
instead of convolution. The average operation is a special
case of the convolution operation whose kernel weights are
1
all kernelsize2 . In addition, we can also execute the maximum
operation by reconstructing the original accumulator to the
max mode as shown in Fig. 14. For the MAC mode, out = in
∗ w + acc_in. For the MAX mode, out = sign(acc_in – in) ?
in : acc_in;
Fig. 15 shows the data flow of the average and maximum
pooling. Its input is the convolutional outputs from the partial
result buffer. In addition, the weight andfilter in will be
constants that depends on the pooling type. For example,
the weight is 14 or 19 and the filter in is 0 in 2x2 or 3x3 average
pooling, respectively. In the maximum pooling, the weight and
the filter in are −1 and negative maximum, which is invalid
to the true maximum value, as shown in Fig. 15. All the final
results will be put in the output data buffer to be stored to the
external memory after the pooling operations.
To execute the fully connected layer on the same hard-
ware of the convolutional layer, the data input of the fully
connected layer has to be filled in the MAC output registers
Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
1647
Fig. 16. Multiple 2D convolvers.
of the processing engine for self-accumulations, which needs
extra ker nel_si ze2 latency. To reduce the extra fill-in latency,
we execute self-accumulations on multiple input nodes for
one certain computational time. The data flow of the fill-in
procedure of MAC output registers depends on the stage, that
is, the initial stage to fill in the zeros, the intermediate stage
to fill in the partial sums from the partial result buffer, and
the final stage to fill in the partial sums similar to that of
the convolutional layer. In the above fill-in procedure, we also
move the previous partial results to the partial result buffer at
the same time for the initial and intermediate stage, and to the
output data buffer for the final stage. The final data path also
depends on whether the ReLU layer exists.
A. CNN Processing Engine
The basic architecture of our CNN processing engine (PE)
is a multiple 2D convolvers [38] as shown in Fig. 16. In this
architecture, we compute one input map and multiple output
maps at the same time since the overall data amount per
batch is independent from Ti and smaller for lager To (input
data ∝ O
To ) in the proposed output first strategy. This mul-
tiple 2D convolvers can support different kernel sizes which
include 1x1, 3x3, 4x4, and 5x5. Larger kernel sizes are decom-
posed into smaller ones. Here the 1x1 convolutional layer
is realized by the fully connected layer with many batches.
Fig. 17 shows how to restructure PEs to fit different kernel
sizes. For example, we can restructure 27 MACs to three
3x3 kernels (27 MACs), one 4x4 kernel (16 MACs), or one
5x5 kernel (25 MACs). With decomposition and restructuring,
we can maximize hardware utilization.
B. Buffer Design and the Corresponding Bandwidth
The buffer design and the required data bandwidth for
input, weight and output data types has significant effects
on hardware utilization. To ease analysis without loss of
generality, we analyze the hardware utilization and average
data access of one type of data buffer by assuming the infinity
size for the other two types of data buffers. This analysis could
be regarded as the upper bound of the hardware utilization and
the lower bound of data access under ideal conditions. Besides,
to be more concrete, we assume AlexNet as our example for
the following analysis.
Fig. 18 to Fig. 20 shows analysis results of different types
of data buffers by assuming 108 MACs and 16 bits data. The
results show that the upper bound of the hardware utilization
is higher for the ping-pong buffer and the average data access
is lower for the single buffer in the input, output, and weight
cases respectively. Thus, we choose the ping-pong buffer to
hide the latency of the data access and maximize hardware
utilization.
1648 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 5, MAY 2018
Fig. 17. An example of PE connections for 27 MACs, (a) 3x3 convolution
mode, (b) 4x4 convolution mode, (c) 5x5 convolution mode, and (d) fully
connected mode.
Fig. 18. Input data buffer effect on (a) hardware utilization, and (b) average
data access.
The optimal input and output buffer size is determined by
the convolutional layers because the data per tile is always
one for the fully connected layer and the fully connected layer
needs only smaller size of input and output data buffer.
The weight buffer size of the convolutional layer is just
the number of hardware MACs (denoted as M∗ ). For fully
connected layer, the weight buffer size, Ti M∗ , cannot improve
the hardware utilization anymore after the upper bound due
to the limited weight bandwidth. In short, this buffer size
is relative insensitive once exceeding a certain level. Despite
this, weight buffer size matters for the fully connected layers
more than the convolutional layers. Thus, we will use the fully
connected layers to decide the optimized weight buffer size.
For weight bandwidth problem, we assume a large scale
hardware with 432 MACs, small scale hardware with
54 MACs, the high bandwidth case with 9 weight loading
per cycle, and the low bandwidth case with 1 weight loading
Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
Fig. 19. Output data buffer effect on (a) hardware utilization, and (b) average
data access.
Fig. 20. Weight data buffer effect on (a) hardware utilization, and (b) average
data access.
Fig. 21. Weight bandwidth effect on the convolutional layer.
Fig. 22. Weight bandwidth effect on the fully connected layer.
per cycle for the analysis in Fig. 21. Fig. 21 shows that we
need higher weight bandwidth or/and more batches for larger
scale convolutional layer hardware. On the other hand, for the
fully connected layer, this problem is getting worse, which
also exists even in the smaller scale hardware. In the analysis
of Fig. 22, we assume 108 MACs, the high bandwidth case
with 27 weight loading per cycle, and the low bandwidth case
with 1 weight loading per cycle. Fig. 22 shows that the fully
connected layers need much higher weight bandwidth or/and
larger batches.
C. Generator
Based on above basic architecture and analysis, we use a
generator to accept input configurations and design constraints
and generate the optimized design and its parameters. These
parameters include the static Verilog parameters and dynamic
LIN AND CHANG: DATA AND HARDWARE EFFICIENT DESIGN FOR CNN
Fig. 23. The flow of the generator.
Fig. 24. An example for the design space exploration of AlexNet C1.
TABLE V
T HE C ONFIGURATIONS OF THE E XAMPLE I MPLEMENTATION
configuration data to provide the trade-off between the per-
formance, cost, and flexibility of the CNN accelerator design.
Fig. 23 shows the proposed generator flow that includes the
hardware parameters search such as final adopted buffer size,
PE numbers and weight bandwidth, and the corresponding
Verilog and configuration data generation.
In the generator flow, we will find the optimal tile config-
uration by the two following cost functions to make a trade
off between hardware utilization and data access number. The
first one is hardware cost first, which is our first priority, as
H + λ1 ∗ D, where λ1 = λ ∗ total MACs/total data.
The second one is data access first, which is the second
priority, as
D + λ2 ∗ H, where λ2 = λ ∗ total data/total MACs.
In above equations, H is the hardware utilization represented
by the cycle count per batch and D is data access represented
Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
1649
TABLE VI
T HE I MPLEMENTATION R ESULTS OF THE BATCH -4 A LEX N ET C ASE
TABLE VII
C OMPARISONS ON THE C ONVOLUTIONAL L AYERS
OF THE B ATCH -4 A LEX N ET C ASE
by data amount per batch. λ1 and λ2 come from λ, which is
a optimization parameter to trade-off the hardware utilization
and data access.
Fig. 24 shows an example for the design space explo-
ration of AlexNet C1 on the constraints of 108 MACs and
120KB (# of data=60000) internal buffer size. This figure
shows that the design point a has the best hardware utilization
solution (λ = 0) and the design point b has the best data reuse
solution (λ = ∞). The final choice will depend on the user
constraints and preference.
VI. I MPLEMENTATION R ESULTS
To demonstrate our design, we use the same area cost,
data access per frame, and the targeted network (batch-4
AlexNet) as those in Eyeriss [33] to be design constraints
for a detailed comparison. Our implementation is designed by
Verilog and synthesized with TSMC 40nm CMOS technology
process, which consumes 1.783M gate count for 216 MACs
and 142.64 KB internal buffer as shown in Table V. This
implementation can achieve 99.7 fps and 61.6 fps for the
convolutional layers and all layers of AlexNet respectively
under 454 MHz clock rate as shown in Table VI.
1650 IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS–I: REGULAR PAPERS, VOL. 65, NO. 5, MAY 2018

TABLE VIII
C OMPARISONS B ETWEEN CNN A CCELERATORS

Table VII shows comparisons on the convolutional layers
of the batch-4 AlexNet case. Our design has much higher
throughput and lower area cost than that of Eyeriss [33].
Thus the area efficiency of our design is higher even after
technology scaling. This is because our connection is more
regular that makes lower hardware cost. For the bandwidth
comparison, we use uncompressed bandwidth in [33] since
their compression method can be adopted here as well. Under
this condition, our design needs smaller internal buffer size
to obtain almost the same data access per frame. In addition,
our design has also higher area and bandwidth efficiency than
those in [33]. Thus our data reuse strategy indeed relieves the
data bandwidth.
Table VIII shows the comparisons between different CNN
accelerators. In this table, we only list the designs implemented
with ASIC processes instead of FPGA chips due to space
limitation. For fair comparison, we have scaled all throughput
to the same process node. In which, area cost in [25] is
the smallest one because it had fixed connections and only
implemented convolutional layers. Thus, it has the highest
area efficiency but with the limited application. Except this,
the area efficiency of our design is the highest one among
these works. The reason for this high efficiency is the proposed
architecture design with high hardware utilization and optimal
design generator under design constraints.
VII. C ONCLUSION
In this paper, we propose a run-time reconfigurable
CNN architecture that can handle non-unit stride kernels and
process different kernel types efficiently to overcome the large
computational complexity and data amount of the CNN algo-
rithm. The overall design exploits layer specific characteristics
for the buffer design and data bandwidth. The hardware design
adopts a tile based model and the output first strategy to
reuse data of the convolutional layers with 300× ∼ 600×
better efficiency than the non-reused case. Designs for different
networks can be generated by a CNN generator for a target
network that is optimal under the given hardware resources
and data bandwidth. An example design for AlexNet with
TSMC 40nm process consumes 1.783M gate count for
216 MACs and 142.64 KB internal buffer size, and achieves
61.6 fps under the 454 MHz clock frequency, which shows
better area and bandwidth efficiency compared to the other
state-of-the-art works.

Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
LIN AND CHANG: DATA AND HARDWARE EFFICIENT DESIGN FOR CNN
R EFERENCES
[1] W. Qadeer, R. Hameed, O. Shacham, P. Venkatesan, C. Kozyrakis, and
M. A. Horowitz, “Convolution engine: Balancing efficiency & flexibility
in specialized computing,” in Proc. ACM Int. Symp. Comput. Archit.,
2013, pp. 24–35.
[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classifica-
tion with deep convolutional neural networks,” in Proc. NIPS, 2012,
pp. 1097–1105.
[3] C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEE
CVPR, Jun. 2015, pp. 1–9.
[4] K. Simonyan and A. Zisserman. (2014). “Very deep convolutional
networks for large-scale image recognition.” [Online]. Available:
https://arxiv.org/abs/1409.1556
[5] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proc. IEEE CVPR, Jun. 2016, pp. 770–778.
[6] S. Ji, W. Xu, M. Yang, and K. Yu, “3D convolutional neural networks
for human action recognition,” IEEE Trans. Pattern Anal. Mach. Intell.,
vol. 35, no. 1, pp. 221–231, Jan. 2013.
[7] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and
Y. LeCun. (2014). “OverFeat: Integrated recognition, localization
and detection using convolutional networks.” [Online]. Available:
https://arxiv.org/abs/1312.6229
[8] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature
hierarchies for accurate object detection and semantic segmentation,”
in Proc. IEEE CVPR, Jun. 2014, pp. 580–587.
[9] T. He, W. Huang, Y. Qiao, and J. Yao, “Text-attentional convolutional
neural network for scene text detection,” IEEE Trans. Image Process.,
vol. 25, no. 6, pp. 2529–2541, Jun. 2016.
[10] H. Li, Z. Lin, X. Shen, J. Brandt, and G. Hua, “A convolutional neural
network cascade for face detection,” in Proc. IEEE CVPR, Jun. 2015,
pp. 5325–5334.
[11] D. Tomè, F. Monti, L. Baroffio, L. Bondi, M. Tagliasacchi, and
S. Tubaro, “Deep convolutional neural networks for pedestrian detec-
tion,” Signal Process., Image Commun., vol. 47, pp. 482–489, Sep. 2016.
[12] Z. Du et al., “ShiDianNao: Shifting vision processing closer to the
sensor,” in Proc. ISCA, Jun. 2015, pp. 92–104.
[13] Y. Chen et al., “DaDianNao: A machine-learning supercomputer,” in
Proc. MICRO, 2014, pp. 609–622.
[14] Y. Chen, X. Yang, B. Zhong, S. Pan, D. Chen, and H. Zhang,
“CNNTracker: Online discriminative object tracking via deep convo-
lutional neural network,” Appl. Soft Comput., vol. 38, pp. 1088–1098,
Jan. 2016.
[15] H. Li, Y. Li, and F. Porikli, “Robust online visual tracking with a single
convolutional neural network,” in Proc. ACCV, 2014, pp. 194–209.
[16] C. Farabet, B. Martini, B. Corda, P. Akselrod, E. Culurciello, and
Y. LeCun, “NeuFlow: A runtime reconfigurable dataflow processor for
vision,” in Proc. IEEE CVPRW, Jun. 2011, pp. 109–116.
[17] L. Cavigelli, M. Magno, and L. Benini, “Accelerating real-time embed-
ded scene labeling with convolutional networks,” in Proc. DAC, 2015,
p. 108.
[18] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva, “Learning
deep features for scene recognition using places database,” in Proc.
NIPS, 2014, pp. 487–495.
[19] P.-H. Pham, D. Jelaca, C. Farabet, B. Martini, Y. LeCun, and
E. Culurciello, “NeuFlow: Dataflow vision processing system-on-a-
chip,” in Proc. MWSCAS, 2012, pp. 1044–1047.
[20] F. Conti and L. Benini, “A ultra-low-energy convolution engine for fast
brain-inspired vision in multicore clusters,” in Proc. IEEE Des. Autom.
Test Eur. Conf., Mar. 2015, pp. 683–688.
[21] W. Yang, W. Ouyang, H. Li, and X. Wang, “End-to-end learning
of deformable mixture of parts and deep convolutional neural net-
works for human pose estimation,” in Proc. IEEE CVPR, Jun. 2016,
pp. 3073–3082.
[22] T. Pfister, J. Charles, and A. Zisserman, “Flowing convnets for
human pose estimation in videos,” in Proc. IEEE ICCV, Dec. 2015,
pp. 1913–1921.
[23] D. Weimer, B. Scholz-Reiter, and M. Shpitalni, “Design of deep con-
volutional neural network architectures for automated feature extraction
in industrial inspection,” CIRP Ann.-Manuf. Technol., vol. 65, no. 1,
pp. 417–420, 2016.
[24] X. Bian, S. N. Lim, and N. Zhou, “Multiscale fully convolutional
network with application to industrial inspection,” in Proc. WACV, 2016,
pp. 1–8.
[25] L. Cavigelli and L. Benini. (Dec. 2015). “Origami: A 803
GOp/s/W convolutional network accelerator.” [Online]. Available:
https://arxiv.org/abs/1512.04295
Authorized licensed use limited to: MANIPAL INSTITUTE OF TECHNOLOGY. Downloaded on October 08,2022 at 07:29:36 UTC from IEEE Xplore. Restrictions apply.
1651
[26] S. Han et al., “EIE: Efficient inference engine on compressed deep neural
network,” in Proc. ISCA, 2016, pp. 243–254.
[27] V. Gokhale, J. Jin, A. Dundar, B. Martini, and E. Culurciello,
“A 240 G-ops/s mobile coprocessor for deep neural networks,” in Proc.
IEEE CVPRW, Jun. 2014, pp. 682–687.
[28] M. Peemen, A. A. Setio, B. Mesman, and H. Corporaal, “Memory-
centric accelerator design for convolutional neural networks,” in Proc.
ICCD, 2013, pp. 13–19.
[29] C. Zhang, P. Li, G. Sun, Y. Guan, B. Xiao, and J. Cong, “Optimizing
FPGA-based accelerator design for deep convolutional neural networks,”
in Proc. ISFPGA, 2015, pp. 161–170.
[30] J. Qiu et al., “Going deeper with embedded FPGA platform for
convolutional neural network,” in Proc. ISFPGA, 2016, pp. 26–35.
[31] L. Cavigelli, D. Gschwend, C. Mayer, S. Willi, B. Muheim, and
L. Benini, “Origami: A convolutional network accelerator,” in Proc.
GLSVLSI, 2015, pp. 199–204.
[32] J. Sim, J.-S. Park, M. Kim, D. Bae, Y. Choi, and L.-S. Kim,
“A 1.42TOPS/W deep convolutional neural network recognition proces-
sor for intelligent IoE systems,” in IEEE Int. Solid-State Circuits Conf.
(ISSCC) Dig. Tech. Papers, Jan./Feb. 2016, pp. 264–265.
[33] Y.-H. Chen, T. Krishna, J. S. Emer, and V. Sze, “Eyeriss: An
energy-efficient reconfigurable accelerator for deep convolutional neural
networks,” in IEEE Int. Solid-State Circuits Conf. (ISSCC) Dig.
Tech. Papers, Jan./Feb. 2016, pp. 262–263.
[34] B. Moons, R. Uytterhoeven, W. Dehaene, and M. Verhelst, “Envi-
sion: A 0.26-to-10TOPS/W subword-parallel dynamic-voltage-accuracy-
frequency-scalable Convolutional Neural Network processor in 28nm
FDSOI,” in IEEE Int. Solid-State Circuits Conf. (ISSCC) Dig.
Tech. Papers, Feb. 2017, pp. 246–247.
[35] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolu-
tional networks,” in Proc. ECCV, 2014, pp. 818–833.
[36] X. Glorot, A. Bordes, and Y. Bengio, “Deep sparse rectifier neural
networks,” in Proc. AISTATS, 2011, p. 275.
[37] V. Nair and G. E. Hinton, “Rectified linear units improve restricted
boltzmann machines,” in Proc. ICML, 2010, pp. 807–814.
[38] C. Farabet, C. Poulet, J. Y. Han, and Y. LeCun, “CNP: An FPGA-based
processor for convolutional networks,” in Proc. FPL, 2009, pp. 32–37.
[39] T. Chen et al., “Diannao: A small-footprint high-throughput accelerator
for ubiquitous machine-learning,” in Proc. ASPLOS, 2014, pp. 269–284.
[40] D. Liu et al., “PuDianNao: A polyvalent machine learning accelerator,”
in Proc. ASPLOS, 2015, pp. 369–381.
[41] T. Chen et al., “A high-throughput neural network accelerator,” IEEE
Micro, vol. 35, no. 3, pp. 24–32, May 2015.
[42] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial architecture for
energy-efficient dataflow for convolutional neural networks,” in Proc.
ISCA, 2016, pp. 367–379.
Yue-Jin Lin received the M.S. degrees in electronics
engineering from National Chiao Tung University,
Hsinchu, Taiwan, in 2016. He is currently with
Novatek, Hsinchu. His current research interests are
image processing, deep learning, and digital inte-
grated circuits.
Tian Sheuan Chang (S’93–M’06–SM’07) received
the B.S., M.S., and Ph.D. degrees in electronic
engineering from National Chiao Tung Univer-
sity (NCTU), Hsinchu, Taiwan, in 1993, 1995, and
1999, respectively.
From 2000 to 2004, he was a Deputy Manager
with Global Unichip Corporation, Hsinchu. In 2004,
he joined the Department of Electronics Engineer-
ing, NCTU, where he is currently a Professor.
In 2009, he was a Visiting Scholar with Imec,
Belgium. His current research interests include
system-on-a-chip design, VLSI signal processing, and computer architecture.
Dr. Chang has received the Excellent Young Electrical Engineer from the
Chinese Institute of Electrical Engineering in 2007, and the Outstanding
Young Scholar from the Taiwan IC Design Society in 2010. He has been
actively involved in many international conferences as an organizing commit-
tee or technical program committee member.