Energy-efficient Scheduling Method with Cross-loop

Model for Resource-limited CNN Accelerator Designs

Kaiyi Yang, Shihao Wang, Jianbin Zhou and Takeshi Yoshimura
Graduate School of Information, Production and Systems, Waseda University
Kitakyushu, Japan
yangkaiyi1222@fuji.waseda.jp, wshh1216@moegi.waseda.jp

Abstract—The state-of-the-art customized accelerators of

convolution neural networks (CNN) have achieved high
throughput while the huge amount of data movements still
remains as the dominant part of the total energy costs. In this
paper, we propose an energy-efficient scheduling approach to find
an efficient dataflow that minimizes data movements with limited
hardware resource budgets. In detail, two-level nested loop
transformations are proposed to separate memory and computing
resource constraints. This allows us to fully exploit the potential of
available memory resources for reducing off-chip memory traffic.
Further, the proposed cross-loop model is capable of figuring out
the data locality across nested loops in CNN algorithms. Finally,
energy-delay production is employed as the evaluation criteria to
balancing energy and throughput performance. The experimental
results show our cross-loop model can reduce the off-chip data
movements by 11-21% and achieve the theoretical optimum.
Therefore, the proposed scheduling method can increase the
energy efficiency by at least 8.7 times.
Keywords—convolutional neural network; memory bandwidth;
energy efficiency; loop tiling; loop transformation; accelerator
I. INTRODUCTION
Deep convolutional neural networks (CNN) have shown
their great performance in many computer vision tasks in recent
years, like AlexNet [1], VGG [2] and ResNet [3]. Among these
networks, convolutional computations take the major part of the
total workload for feature extractions. There is a trend that the
convolution layers will increase more for a better performance
at the expense of higher computational complexities.
The increasing computational complexity of the state-of-the-
art CNNs has urged many researchers to find hardware
accelerators for speeding up these processes. With the
techniques like high-level synthesis, it greatly reduces the design
period for high parallel CNN accelerators. However, the large
amount of convolutions also involves huge number of memory
accesses to move various operands between memories and the
processor core. These data movements have been regarded as
the major part of the total power consumption [4][5].
There are many research groups trying to find energy-
efficient ASIC/FPGA-based CNN accelerators by reducing the
data movements, which are tied tightly to the chosen dataflow.
Some works [6][9] use the loop parameters of CNN algorithms
to model the memory bandwidth requirement and try to find
efficient scheduling methods by loop transformations under
limited resource budgets. [7] gives an improved model that can
i 1C l ih i b dl
accurately represent the bandwidth. [4] solves this problem from
hardware aspects by building a memory hierarchy and try to
allocate computations at register level as many as possible.
Although these works can improve the energy efficiency, there
is a lack of systematical analysis for evaluating the performance
so the results may be the local optimal solutions.
This paper introduces a novel model and scheduling method
for finding an energy-efficient dataflow given a limited resource
budgets. The contributions are concluded as below:
(a) With the same constraints, the proposed cross-loop
model can achieve theoretically minimal memory accesses to
obtain 11-21% reduction of memory bandwidth requirements.
(b) Besides, scheduling is based on two-level nested loop
transformations to exploit the potential of on-chip resources.
8.7x energy-efficiency improvements can be achieved.
(c) This approach can be used systematically analyze most
of the existing CNN accelerators as it covets data parallelism in
various levels including batch level.
II. BACKGROUND AND CHANLLENGES
A. CNN overviews
Generally, CNNs contain multiple layers where
convolutional layers contain the major of the computations. For
convolutional layers, the input/output of each image are
organized as channels of two dimensional input/output feature
maps (ifmaps/ofmaps). For each pair of ifmaps and ofmaps, a
unique convolutional kernel exists for feature extractions. Each
ofmap is the sum up of features generated from all ifmaps.
Several images are organized as a batch, in which kernels are
invariant. They can be represented as the nested loops as Fig. 1.
We don’t unroll the convolution operations as they are usually
designed to be processed in parallel [5] or processed
continuously [8] in many customized CNN accelerators.

978-1-4673-6853-7/17/$31.00 ©2017 IEEE


Fig. 2. General architecture of customized CNN accelerators.
We define the nested loops as control loops with three
attributes (order, range and tile). The order and range of control
loops mean the iterative order starting from the outermost loop
and the trip counts as shown in Fig. 1. Tile is used to define as a
subset of computations in blue color and introduced in Sec. II.B.
B. Design Chanllenges and Previous Works
A general FPGA/ASIC-based CNN accelerator system is
shown in Fig. 2 with limited on-chip resources budgets. Memory
resource constraint limits the maximum available on-chip
SRAM, which can be used to exploit data locality and reduce
DRAM accesses for energy saving (200x versus 6x energy cost
according to [4]). Meanwhile, Computing resource constraint
limits the maximum number of PEs in parallel, which directly
affects the throughput performance of CNN accelerators.
The design challenges are finding an energy-efficient
accelerator under the two constraints. Especially, as the
FPGA/ASIC designs can easily stack PEs for high parallelism,
the remaining difficulty is finding an dataflow that can reduce
the memory bandwidth requirements as data movements
consume the major part of energy costs. Loop tiling is widely
employed in prior works like [6][7][9] and is solved as a
mathematical optimization problem as below (k is kernel size):
‫݈݁݀݋ܯܹܤ݉ܽݎܦ݁ݖ݅݉݅݊݅ܯ‬൫ܶ௕ ǡ ܶ௜ ǡ ܶ௢ ǡ ܶ௫ ǡ ܶ௬ ൯
‫݈݄ܶ݁݅ݓ‬௕ ܶ௜ ሺܶ௫ ൅ ݇ െ ͳሻ൫ܶ௬ ൅ ݇ െ ͳ൯ ൅
ܶ௕ ܶ௢ ܶ௫ ܶ௬ ൅ ܶ௜ ܶ௢ ൑ ‫݁ܿݎݑ݋ݏܴ݁ݕݎ݋݉݁ܯ‬
ܶ௕ ܶ௜ ܶ௢ ܶ௫ ܶ௬ ൑ ‫݁ܿݎݑ݋ݏܴ݁݃݊݅ݐݑ݌݉݋ܥ‬
Although this method can find an efficient dataflow, we
could find a better solution with the hardware-oriented
consideration. Firstly, the above constraints actually regard on-
chip SRAM and processor core as an indivisible system,
resulting in local optimum result. Secondly, existing models
overestimate the DRAM bandwidth as they can’t fully exploit
on-chip SRAM. Thirdly, the energy efficiency is affected not
only by DRAM BW, but also by the SRAM part and processing
time. These issues will result in local optimum results by
previous works and we are challenged by considering these
hardware-oriented thinking into our designs.
III. HARDWARE-ORIENTED SCHEDULING METHOD
This section presents a detailed discussion on the proposed
scheduling methodology for solving the three challenges
mentioned in the end of last sections and concludes these
optimizations in hardware views in Sec. III.D.
for each TileD in loopTransformation(B):
// Calculate DRAM BW
DramBW = cross_loop_model(TileD) Sec.III.A
for each TileS in loopTransformation(TileD):
// Calculate SRAM BW Sec.III.B
SramBW = cross_loop_model(TileS)
thisEDP = calcEDP(DramBW,SramBW,TileD,TileS)
// Record results with minimum EDP Sec.III.C
if thisEDP<minEDP:
optRes = thisRes // May contain TileD, TileS, ...
return optRes
Fig. 3 Pseudocode of the hardware-oriented scheduling algorithm
A. Framework of Scheduling Methodology
Two-level nested loop transformations form the framework
of our proposed scheduling algorithm as shown in Fig. 3. We
define TileD and TileS to represent the candidates of each loop
transformation level. TileD is generated with the memory
resource constraint while TileS is limited by the computing
resource constraint. Instead of combing two constraints together,
we separate them to enhance the exploration space. We first
guarantee the DRAM bandwidth is minimized with available
memory resources due to its huge energy cost. Then, we explore
an dataflow within TileD to achieve minimal SRAM bandwidth
traffic as well as fully utilizing the computing resources.
Notice that the loop transformations contain not only loop
tiling but also loop reordering. All possible combinations of
order and tile attributes are considered. Unlike a predefined
order for control loops, the traversing of possible orders can
fully exploit the data locality in CNN algorithms to further
reduce the memory bandwidth together with the proposed cross-
loop model in Sec. III.B.
B. Cross-loop Model of Memory Bandwidth
When given the attributes of order and tile, many works [6][9]
use a simple model to model the memory bandwidth. This model
is a production of the bandwidth per tile and the number of tiles.
‫ܤ‬
‫ ݈݁݀݋ܯܹܤ‬ൌ ሺ͓݈ܶ݅݁‫ݏ‬ሻ ൈ ‫ܹܤ‬Ȁ݈ܶ݅݁ ൌ ෑ ൈ ‫ܹܤ‬Ȁ݈ܶ݅݁
݈ܶ݅݁
஻ ஻ ஻ ஻ ஻
ൌ ቒ ್ ቓ ቒ ೔ ቓ ቒ ೚ ቓ ቒ ೣቓ ඄ ೤ ඈ ൈ ‫ ܹܤ‬Τ݈ܶ݅݁ (1)
்್ ்೔ ்೚ ்ೣ ்೤
where: ࢏ࢌ࢓ࢇ࢖࢙ǣ ‫ ܹܤ‬Τ݈ܶ݅݁ ൌ ܶ௕ ܶ௜ ሺܶ௫ ൅ ݇ െ ͳሻሺܶ௫ ൅ ݇ െ ͳሻ
࢕ࢌ࢓ࢇ࢖࢙ǣ ‫ ܹܤ‬Τ݈ܶ݅݁ ൌ ʹ ൈ ܶ௕ ܶ௜ ܶ௫ ܶ௬ 
࢑ࢋ࢘࢔ࢋ࢒ǣ ‫ ܹܤ‬Τ݈ܶ݅݁ ൌ ܶ௜ ܶ௢
We claim that this model may over-estimate the bandwidth
requirement as it ignores the data locality among tiles. While [7]
proposed inter-tile model to eliminate the locality between
adjacent tiles, there still exist potentials for data reuses in outer
control loops.
Cross-loop model is proposed to provide an accurate
estimation so as to fully exploit the caching ability of on-chip
memories. This model starts with the simple model and keeps
removing the overlaps between the operands of a new tile and
the buffered data inside on-chip SRAM. The removal is based
on the conception of Reuse Potential (RP) defined in Table I.
Reuse potential is marked to every subscripts of each operand
with three candidates, including none, half and full. It refers how

Fig. 4. An example of half reuse potential (RP)
Fig. 5. Pseudocode of the function cross_loop_model.
many overlapped data exist inside on-chip SRAM. Full RP
means all data of the current tile are already cached while none
RP is the case all the data have to be fetched from off-chip
DRAM. Specially, there exist cases (half RP) for ifmaps where
only part of the data is inside SRAM as shown in Fig. 4.
Table I. Reuse Potentials of the indexes for control loops
Data Type
Reuse Potential
b i o x y active PEs. An improper tiling may not fully utilize the available
ifmaps[b][i][x][y] none none full half half
ofmaps[b][o][x][y] none full none none none
kernels[i][o] full none none full full
We give the pseudocode of our cross-loop model in Fig. 5.
This function receives a tile candidate after loop transformation
and calculate BW by the simple model in Equation (1). Then,
two optimizations are executed sequentially: (Opt.A) Traverse
control loops from innermost to outermost. As long as the RP is
not none, we can remove the overlapped data according to Table
II. Moreover, the traversing of each operand is interrupted as
long as the RP is not full. Even half RP can’t guarantee the data
to be reused by the following outer control loops. (Opt.B) Notice
that ofmaps in Equation (1) is double as we have extra reading
operations compared to other operands. We can omit the reading
accesses only if all the calculations involved by a specific
ofmaps are executed continuously. As shown in Fig. 5, we have
to be sure the innermost control loop has a subscript of ifmaps
and the range of ifmaps is equal to the number of ifmaps channel
of current convolutional layer.
Table II shows how to remove overlaps considering current
control loop has a range of B and a tile is T. With full RP, all the
tiles share the same data so we just need to divide the bandwidth
by the tile numbers. Half RP occurs only in ifmaps according to
Table I. The locality exists across the whole range B. Instead of
considering this control loop is divided into many tiles, we can
rewrite the bandwidth by imaging this control loop is just one
tile.
Table II. Removal of overlapped data in cross-loop model
Reuse Removal Method
Potential Before After
full ‫ܹܤ‬Ȁ‫݈݁݅ݐ‬
‫ܤ‬
඄ ඈ ൈ ‫ܹܤ‬Ȁ‫݈݁݅ݐ‬ ‫ܤ‬൅݇െͳ
half ܶ ൈ ‫ܹܤ‬Ȁ‫݈݁݅ݐ‬
ܶ൅݇െͳ
C. Memory-centric Energy-Delay Procduction
Instead of considering only off-chip DRAM bandwidth in
prior works, this work also involves the energy cost of on-chip
SRAM to give a systematical analysis. The power consumption
can be written as the weighted sum of these two sources of
energy consumptions as below where the second term represents
the energy costs of SRAM. B means the parameters of the
current convolutional layer.
‫ ݐݏ݋ܿݕ݃ݎ݁݊ܧ‬ൌ ʹͲͲ ൈ ‫ܹܤ‬ሺ‫݉ܽݎܦ‬ሻ ൅ ͸ ൈ ‫ܹܤ‬ሺܵ‫݉ܽݎ‬ሻ
‫ܤ‬
ൌ ʹͲͲ‫ܹܤ‬ሺ‫݉ܽݎܦ‬ሻ ൅ ͸ ൈ ෑ ൈ ‫ܹܤ‬ሺܵ‫݉ܽݎ‬ሻȀ‫݈݁݅ݐ‬
݈ܶ݅݁‫ܦ‬
Moreover, we employ the energy-delay production as the
cost function to guarantee the loop transformation not only
achieves a high energy efficiency, but also fully utilize PEs. The
delay is calculated with the following equation:
‫ ݕ݈ܽ݁ܦ‬ൌ ሺ͓݈ܶ݅݁ܵሻ ൈ ሺ݈݀݁ܽ‫݈ܵ݁݅ܶݎ݁݌ݕ‬ሻ
‫ܤ‬ ݈ܶ݅݁‫ܦ‬ ͳ
ൌ  ൬ෑ ෑ ൰ൈ൬ ൰
݈ܶ݅݁‫ܦ‬ ݈ܶ݅݁ܵ ‫ܧܲ݁ݒ݅ݐܿܣ‬
The delay of each TileS is defined as the reciprocal of the
computing resources. For example, when To is tiled as 3 in TileS
and we prefer at most two ofmaps are processed in parallel, half
of the available PEs will not be active in the last iteration of loop
To, resulting in achieving only 75% of its peak throughput.
D. Hardware-oriented Analysis
Firstly, two-level nested loop transformation can fully
exploit the potential of on-chip resources compared to prior
works. The twisted constraints in Sec. II.B may cause a local
optimal. When a tiling candidate is constrained by computing
resources, on-chip memory resources may be under-utilized.
The unused memory can actually be used to buffering more data
for reducing DRAM bandwidth but this possibility is ignored by
prior works. Two-level nested loop transformation scheduling
can eliminate this problem as it divided these two constraints
into two sub-tasks. There is no dependency existing in these two
sub-tasks so a better solution can be expected by our approach.
Secondly, we introduce loop reordering, cross-loop model,
parallelism in batch level and energy-delay production to
enlarge the solution space. Most of them will guarantee the same
hardware design complexity as prior works because they just try
to find a more energy-efficient dataflow. Only the half reuse
potential in cross-loop model will require the on-chip SRAM to
 Off-chip Bandwidth Traffic (MByte/image)
Table III. Experimental Results with different methods and Configs.
5.6 This Work
DATE'15 [7] Scheduling Methods
FPGA'15 [6]
Metrics
4.6
Theoriotical Opt.
A B C
(b,i,o,x,y)
TileD (b,i,o,x,y) (b,i,o,x,y)
3.6 (4,8,128,13,13)
(2,64,64,13,13) (1,16,32,13,13)
(b,x,y,i,o)
TileS Para(x=8,y=64) Para(x=16,y=32)
2.6 (1,1,1,8,64)
BW(Dram) 1.8MB/imgae 4.19MB/image 10.5MB/image
1.6
BW(Sram) 26.8MB/imge 624.0MB/image 625.1MB/image
0.6
Active
200 300 400 500 600 700 800 900 1000 1100 512 512 512
On-chip Memory Constraint (KByte)
# of MAC
Active
Fig. 6 DRAM bandwidth versus memory resource constraint. 3.3Mb 2.8Mb 435.7Kb
SRAM
be controlled more sophisticatedly. Actually, a circular buffer EDP 1x 8.7x 11.1x
with a registered index can support this case like [8].
V. CONCLUSION
IV. EXPERIMENTAL RESULTS This paper provides an energy efficient scheduling approach
for find an optimal dataflow that can minimize data movements
A. Evaluation of Cross-loop model for DRAM Bandwidth from memories. Based on the understanding of hardware
DRAM bandwidth requirement is the most expensive part in behaviors, we propose several optimization schemes including
CNN accelerators. Fig. 6 evaluates the performance of our cross- two-level nested loop transformation, cross-loop model and
loop model. The test bench is the 3rd layer of AlexNet with a introduce energy-delay production as the cost function. These
loop range of (4,256,384,13,13) for (b,i,o,x,y) as used in [4] as schemes can generate more efficient dataflow and improve the
it is the most nested layer inside AlexNet. energy-efficiency by at 8.7 times.
The line of opt. in Fig. 6 means the theoretically minimum
DRAM bandwidth by assuming an infinite size of on-chip ACKNOWLEDGMENT
memories. Our cross-loop model achieves 11-21% more power- This paper is partly supported by KAKENHI (26420323)
efficiency than the inter-tile model [7] as we consider reuse and by Waseda University Graduate Program for Embodiment
possibilities across control loops. Generally, 1MB on-chip Informatics (FY2013-FY2019).
memories can reduce data movements to the theoretical minimal.
REFERENCES
B. Evalutaion of the Proposed Scheduling Method
We conduct several experiments to show how this approach [1] A. Krizhevsky et al., “ImageNet Classification with Deep Convolutional
can find efficient dataflow as well as fully exploiting on-chip Neural Networks,” in Proc. Adv. Neural Inf. Process. Syst., vol. 25, pp.
memories. In [6][9], Vertex7 485t is chosen as the platform with 1097-1105, 2012.
2800 DSPs (equivalent to 560 32-bit multiplier– [2] K. Simonyan et al., “Very Deep Convolutional Networks for LargeScale
accumulator/MAC) and 37Mb on-chip memories. We use these Image Recognition,” CoRR, vol. abs/1409.1556, pp. 1–14, Sep. 2014.
constraints for fair comparisons and evaluate three scheduling [3] K. He et al., “Deep residual learning for image recognition”, in Proc.
methods in Table III: (A) Scheduling by this work; (B) Two- IEEE Conf. Comput. Vis. Pattern Recognit., pp. 770-778, 2016.
level nested loop transformation with simple model in [6]; (C) [4] Y.-H. Chen et al., “Eyeriss: A Spatial Architecture for Energy-Efficient
Dataflow for Convolutional Neural Networks,” in Proc. 43rd Annu. Int.
Scheduling by [6]. Notice that in B/C computing parallelism of Symp. Comput. Archit., pp. 367-379, June 2016.
subscripts x and y are considered.
[5] S. Wang et al., “Chain-NN: An Energy-Efficient 1D Chain Architecture
Table III proves that our approach A needs much fewer for Accelerating Deep Convolutional Neural Networks,” in Proc. Design,
Autom. Test Eur. Conf. Exhibit., 2017.
memory accesses than B/C with different tiling parameters. In C,
a possible solution is usually constrained by available [6] C. Zhang et al., “Optimizing FPGA-based Accelerator Design for Deep
Convolutional Neural Networks,” In Proc. ACM/SIGDA Int. Symp.
computing resources while memories are not fully exploited. Field-Programmable Gate Arrays, pp. 161-170, 2015.
Even though we have 37Mb available memories, only 436Kb [7] M. Peemen et al., “Inter-tile reuse optimization applied to bandwidth
are used in C. Thus we conduct B with our proposed two-level constrained embedded accelerators,” in Proc. Design, Autom. Test Eur.
nested loop transformation to omit this issue as discussed in Sec. Conf. Exhibit., pp. 169-174, 2015.
III.D. B can use much more memories to reduce 60% redundant [8] T. Chen et al., “DianNao: A Small-Footprint High-Throughput
DRAM bandwidth requiremets than C. This partly explains why Accelerator for Ubiquitous Machine-Learning,” in Proc. ACM Int. Conf.
the half of memory reourses are not utilized in [6]. In A, we Archit. Support Program. Lang. Oper. Syst., pp. 269-284, 2014.
further enable the otpimization schemes with cross-loop model [9] M. Motamedi et al., "Design space exploration of FPGA-based Deep
Convolutional Neural Networks," in Proc. 21st Asia and South Pacific
and multiple parallelisms to reduce the data locality comapred Design Autom. Conf., pp. 575-580, 2016.
to B. The scheduling produces the optimal parallelism as shown
in TileD and TileS. This solution shows that 57.0% of DRAM
and 95.7% of SRAM accesses are reduced and we can improve
the energy-delay production by at least 8.7 times.