IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO.
1, JANUARY 2023
ECBC: Efficient Convolution
via Blocked Columnizing
Tianli Zhao , Qinghao Hu, Xiangyu He, Weixiang Xu , Jiaxing Wang,
Cong Leng, and Jian Cheng , Member, IEEE
Abstract— Direct convolution methods are now drawing
increasing attention as they eliminate the additional stor-
age demand required by indirect convolution algorithms
(i.e., the transformed matrix generated by the im2col convolution
algorithm). Nevertheless, the direct methods require special
input–output tensor formatting, leading to extra time and mem-
ory consumption to get the desired data layout. In this article,
we show that indirect convolution, if implemented properly,
is able to achieve high computation performance with the help of
highly optimized subroutines in matrix multiplication while avoid
incurring substantial memory overhead. The proposed algorithm
is called efficient convolution via blocked columnizing (ECBC).
Inspired by the im2col convolution algorithm and the block
algorithm of general matrix-to-matrix multiplication, we propose
to conduct the convolution computation blockwisely. As a result,
the tensor-to-matrix transformation process (e.g., the im2col
operation) can also be done in a blockwise manner so that it
only requires a small block of memory as small as the data
block. Extensive experiments on various platforms and networks
validate the effectiveness of ECBC, as well as the superiority of
our proposed method against a set of widely used industrial-level
convolution algorithms.
Index Terms— Convolutional neural networks (CNNs), direct
convolution, high performance computing for mobile devices,
im2col convolution, memory-efficient convolution (MEC).
I. I NTRODUCTION
C ONVOLUTIONAL neural networks (CNNs) have
achieved great success in various areas, including image
recognition [1]–[3], object detection [4]–[6], and semantic
segmentation [7], [8]. Recently, as the emerging of high-end
mobile devices, more and more deep learning applications
are migrating from desktops to these edge devices. How-
ever, the high memory and computation demands of convo-
lution obstacle the application of CNNs on these platforms,
where memory and power resources are largely constrained.
A high-performance routine that accelerates the convolution
computation on mobile devices is of demanding.
Manuscript received 8 December 2020; revised 30 March 2021; accepted
25 June 2021. Date of publication 19 July 2021; date of current version
5 January 2023. This work was supported in part by the National Natural Sci-
ence Foundation of China under Grant 61972396, in part by the National Key
Research and Development Program of China under Grant 2020AAA0103402,
and in part by the Strategic Priority Research Program of the Chinese
Academy of Sciences under Grant XDA27040300. (Corresponding author:
Jian Cheng.)
Tianli Zhao, Xiangyu He, and Jian Cheng are with the Institute of
Automation, Chinese Academy of Sciences, Beijing 100080, China (e-mail:
jcheng@nlpr.ia.ac.cn).
Qinghao Hu, Weixiang Xu, Jiaxing Wang, and Cong Leng are with the
National Laboratory of Pattern Recognition, Beijing 100190, China.
Color versions of one or more figures in this article are available at
https://doi.org/10.1109/TNNLS.2021.3095276.
Digital Object Identifier 10.1109/TNNLS.2021.3095276
2162-237X © 2021 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
433
Early endeavors accelerate convolution in an indirect man-
ner. These approaches either transform the input and kernel
tensors into another domain [9], [10] or transform the input
tensor into a special format [11], [12] for efficient convolution
computation.
Though widely applied, the indirect algorithms introduce
substantial additional memory footprint as the transforma-
tion process requires reshaping and selectively duplicating
parts of the input/kernel/output tensors. The problem becomes
more critical on embedded devices, where storage resources
are largely constrained [13], [14]. Previous works elimi-
nate these memory demands through directly performed con-
volution [13]–[16]. By carefully designing the loop tiling,
ordering, and data layout, direct convolution1 is able to
achieve comparable performance or even outperform indirect
methods.
While achieving overall good performance, existing direct
convolution methods also suffer from two critical problems.
First, in order to optimize the register blocking and memory
access, these methods require the input and output to be
stored in an odd format. Specifically, the input and out-
put tensor with shape N × C × H × W are split as
N × (C/x) × H × W × x for convolution computation. The
practical use of these methods requires either a large number of
operations implemented for this data layout or additional time
and memory overhead for data format transformation [17].
What is worse, the x factor can be different across different
layers [15], [16] so that the reorganization must be carried out
for each convolution layer. Second, because of the complicated
computation loops of convolution, the memory access during
computation is not totally continuous, which is not friendly to
the popular hierarchical memory architecture in modern CPUs,
leading to further performance degradation.
In this article, we show that indirect convolution, if imple-
mented properly, is able to retain high computation perfor-
mance with the help of highly optimized subroutines in matrix
multiplication while avoiding substantial memory overhead
during the whole computation. The proposed algorithm is
called efficient convolution via blocked columnizing (ECBC).
Inspired by the block algorithm of general matrix-to-matrix
multiplication (GEMM), instead of performing the whole
convolution, we conduct the computation blockwisely. As a
result, the im2col transformation can also be done in a
blockwise manner. The block algorithm, together with memory
1 Direct convolution is defined as the convolution algorithm that is imple-
mented directly with nested loops.
 434 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 1, JANUARY 2023
Fig. 1. Performance of the proposed ECBC and the direct convolution algo-
rithm ZMC [13] normalized to im2col convolution algorithm on convolution
layers of AlexNet [1]. ZMC [13] is able to get higher performance compared to
the im2col convolution algorithm because it eliminates the memory overhead
required by the im2col procedure, while ECBC further outperforms ZMC due
to its better memory access pattern.
sharing, can greatly reduce the memory overhead in indirect
methods. We demonstrate that, with proper loop ordering,
block size selection, and delicately designed kernel tensor
layout, the continuity and reusability of memory access can
be improved. The optimized memory access is friendly to the
memory hierarchy architecture of CPUs, which also improves
the computation efficiency. Compared to previous indirect
methods, ECBC significantly reduces the memory overhead
and enjoys better memory access. Compared to direct methods,
ECBC does not assume any special data layout of input–output
tensors, so there is no need to transform the data layout of
input–output tensors before convolution.
To further illustrate the advantages of ECBC, in Fig. 1,
we show the performances of the state-of-the-art direct con-
volution algorithm zero memory overhead direct convolution
(ZMC) [13] and our proposed ECBC on convolution layers
of AlexNet [1]. The benchmarks are conducted on 4× ARM
Cortex-A17 cores and normalized by im2col + OpenBLAS.
ZMC is able to run faster than the im2col convolution
algorithm by approximately 2× because it eliminates the
additional packing overhead, while the proposed ECBC further
outperforms ZMC by at least 10% (conv5) and up to 60%
(conv1). The performance improvement is mainly gained by
better continuity of memory access of ECBC. What is more,
we achieve this performance at the cost of only a small area
of shared additional memory overhead.
In summary, the main contributions of this article are
threefold.
1) We present ECBC and show that indirect convolution
methods can achieve high computation performance
with highly optimized subroutines in matrix multiplica-
tion while avoid incurring substantial memory overhead.
2) We propose proper loop ordering, block size selection,
and kernel layout for convolution computation using
our ECBC. The continuity and reusability of memory
access can be improved without additional memory
consumption.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
3) Extensive experiments on various platforms and net-
works validate the effectiveness of ECBC as well as
the superiority of our proposed method against a set of
widely used industrial-level convolution algorithms. For
example, on 4× ARM Cortex-A17 cores, ECBC is at
least 2.5× and up to 5× faster compared to the im2col
convolution algorithm on a wide range of CNN models,
and the memory overhead is saved by at least 30% and
up to 75%.
II. R ELATED W ORKS
As the growing popularity of CNNs, efficient computing of
convolution has long been a major concern of researchers.
Some works reduce the computation and memory com-
plexity of convolutions with model compression, such as
pruning [18]–[22], quantization [23]–[27], and low-rank
decomposition [28]–[30]. While achieving high compression
or acceleration ratio, these methods accelerate convolution
computation at the expense of more training cost and accu-
racy loss. Some other works focus on accelerating the com-
putation of full-precision convolutions so that there is no
demand for further fine-tuning of models and at the same
time retaining the accuracy of models. Early efforts accel-
erate convolution in an indirect manner. These methods can
be roughly divided into two categories: domain-transformed-
based methods and spatial-domain-based methods. Domain-
transformed-based methods, such as fast Fourier transform
(FFT)-based convolution [9] and Winograd-based convolu-
tion [10], [31]–[34] transform the input/kernel tensors into
another domain, in which the convolution can be calculated
with less floating-point operations, thus improving the com-
putation efficiency. Spatial-domain-based methods, such as
im2col-based convolution [11], [35], convert convolution into
matrix multiplication found in basic linear algebra subpro-
grams [36] so that highly optimized libraries [37]–[40] can be
utilized. To achieve this goal, these methods need to first dupli-
cate subpacks of the input tensor into a large flat matrix, incur-
ring additional memory overhead for the transformed input.
The main problem with these methods is that the addi-
tional transformation operation incurs nontrivial memory or
computation overhead to the system. The problem becomes
more critical on embedded devices, where memory, compu-
tation, and power resources are largely limited [13], [14].
Thus, many recent works have focused on memory-efficient
convolution (MEC) algorithms, which means to maximize
the computation performance under some memory constraints.
Cho and Brand [12] took the first step toward this direction.
They propose to reduce the memory overhead during computa-
tion with multiple small matrix multiplication operations, each
with a smaller patch of input data. This method still needs
additional memory overhead for the transformed input data.
Thereby, directly implemented convolution, which is able to
eliminate the extra storage demands during computation, has
approached to the mind of scholars [13]–[16]. These methods
optimize the computation by carefully reorganizing the nested
loops of convolution. Their performances are largely relied
on architecture-specific data layouts of input–output/kernel
tensors. This implies that the practical use of these algorithms
 ZHAO et al.: ECBC
TABLE I
C ONVOLUTION R ELATED N OTATIONS U SED IN T HIS A RTICLE
needs either large number of operations implemented under
these data layouts or additional memory and operation over-
heads for data layout transformation. Dukhan et al. [17] pro-
posed an indirect convolution algorithm specifically optimized
for NHWC data layout. It performs the convolution computa-
tion with the help of indirect GEMM and a relatively small
im2col buffer, which contains only the pointers to the rows of
input data. More recently, Gural and Murmann [41] proposed
memory optimal direct convolution (MOC) for deployment of
CNNs on microcontrollers with only 2-kB memory. The main
idea is to reuse the memory of input data during computation
of output. This method is proved to be memory optimal but not
optimized for computation efficiency. Some other solutions,
such as TVM [42], FlexTensor [43], and Ansor [44], propose
to generate high-performance tensor computation routines with
the approach of just-in-time code generation and automatic
code tuning. These approaches need to take a long time to
search good routines in a large search space for a specific
convolution/platform.
III. P RELIMINARIES
Our work is closely related to the im2col convolution
algorithm and the block algorithm of GEMM. In this section,
we first describe the notations in this article and then review
these two algorithms briefly.
A. Notations
Table I shows the main notations related to a convolution
operation. We use capital characters to denote tensors and
matrices and lowercase characters to denote integers. We use
a set of integers included in a pair of square brackets ([i, j ])
to denote index and a capital character followed by an index
( A[i, j ]) to denote a specific element of a matrix or tensor.
If the index [i, j ] is out of range of dimensions of A, then
we define A[i, j ] = 0. We define add operation between
two indices as [i, j ] + [ p, q] = [i + p, j + q]. A sequence
{a, a + 1, . . . , b − 1} is denoted as a : b. We use sequences
to denote a submatrix, for example, A[i : i + p, j : j + q]
denotes a p × q submatrix of the matrix A with the top-left
corner A[i, j ]. A single : is used to denote all the elements of
a dimension. For example, A[:, j : j + q] denotes a submatrix
of A, starting at A[0, j ] with the same rows as A and columns
of q.
c ,n c
For a matrix A, we use Am i, j to denote the subblock at
the i th row and j th column when A is blocked along the two
dimensions with block size m c and n c , respectively. Namely,
c ,n c
Ami, j = A[i ∗ m c : i ∗ m c + m c , j ∗ n c : j ∗ n c + n c ] and
nc
A:,i is used to denote the submatrix A[:, i ∗ n c : i ∗ n c + n c ].
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
435
B. Im2col-Based Convolution
Im2col-based convolution is perhaps the most well-known
convolution method, which is widely used in deep learning
frameworks such as Caffe [35], Mxnet [45], and Tensor-
flow [46]. Fig. 2(a) shows the main idea of im2col-based
convolution algorithm. It first transforms the input tensor I
into a flat matrix I ∗ . As the kernel K sliding through both
dimensions of I with strides sh and sw , the corresponding
patch of I is vectorized and duplicated into a column of I ∗ .
For example, the subpatch with blue dashed line borders is
vectorized and duplicated to the first column of I ∗ , and the
subpatch with green dashed line borders is vectorized and
duplicated to the second column of I ∗ . The kernel tensor K
is directly reinterpreted as a oc × i c kh kw matrix K ∗ . Then,
the convolution results can be calculated with GEMM between
K ∗ and I ∗ , i.e., O = K ∗ × I ∗ . The output O can be
automatically stored in oc × oh × ow order after GEMM, so no
reordering is needed.
For ease of explanation, we define an operator im2col. Tak-
ing the index [h, w] of I ∗ as input, the im2col operator
generates the corresponding index [i c, i h, i w] of I so that
[i c, i h, i w] = i m2col([h, w]) ⇒ I ∗ [h, w] = I [i c, i h, i w].
Apparently, the index [i c, i h, i w] can be easily computed
by the following equations:
 
h
ic = kd = mod (h, kh kw )
k k
 h w
kd
kh = kw = mod (kd, kw )
kw
 
w
oh = ow = mod (w, ow )
ow
i h = − ph + kh × dh + oh × sh
i w = − pw + kw × dw + ow × sw . (1)
C. Block Algorithm of GEMM
1) Overview: GEMM is the most important operation in
linear algebra. There have been many libraries providing
high-performance implementations of GEMM based on CPUs,
such as OpenBLAS [38], Eigen [40], and MKL [39]. The
efficiency of these implementations mainly depends on the
block algorithm of GEMM [47]–[49]. Fig. 2(b) shows the
overall pipeline of this algorithm.
For two matrices A ∈ Rm× p and B ∈ R p×n , GEMM
computes the matrix multiplication between A and B: C =
A × B. In the first stage, both A and B are blocked along the
p dimension with factor pc , resulting in  p/ pc  subproblems
of general panel-to-panel multiplication [GEPP; refer to the
middle of Fig. 2(b)]. During this stage, the corresponding
m × pc submatrix of A, say Â, is duplicated in a continuous
area of memory, namely a tensor A∗ of special format [refer
to the arrow labeled by packa in Fig. 2(b)], which will
be described in detail in Section III-C2. After that, each
GEPP problem is further blocked along the n dimension with
the factor n c for cache blocking, creating multiple problems
of general panel-to-block multiplication [GEPB; refer to the
middle of Fig. 2(b)]. In this stage, the corresponding pc × n c
 436 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 1, JANUARY 2023

Fig. 2. (a) Computation pipeline of im2col-based convolution algorithm. When the kernel window slides through the input tensor I , each corresponding
subpatch of I is vectorized and duplicated into one column of a flat buffer I ∗ . In this way, the results of convolution can be computed through matrix
multiplication between the kernel matrix K ∗ and the flat buffer matrix I ∗ . (b) Computation pipeline of the block algorithm of GEMM. The computation is
first blocked along the p dimension with factor pc and, then, the n dimension with factor n c for cache blocking. The subsequent computations are further
decomposed along the m dimension with factor m r and, finally, the n c dimension with factor nr for register blocking.

submatrix, say B̂ of B, is duplicated into a contiguous area of
memory B ∗ [refer to the arrow labeled by packb in Fig. 2(b)].
The blocking factors n c and pc are carefully selected such that
the memory block B ∗ resides in the L2 cache and is reused
during the computation of GEPB.
Most mobile devices are based on reduced instruction set
computer (RISC) architecture, which means that data must be
loaded into registers before operations are done, and single
instruction multiple data (SIMD) are also a widely utilized
technique. For this reason, register blocking is further con-
sidered. More specifically, each problem of GEPB is further
blocked along the m dimension with the factor m r and finally
the n c dimension with the factor nr . The blocking factors m r
and nr are selected according to the number and bit width
of (vector) registers. In this way, the whole computation is
finally decomposed into multiple inner product computations
between one m r × pc micro panel of A and one pc × nr
micro panel of B, which is highly optimized assembly. Due to
register blocking, during each computation of inner product,
each of the data is loaded into registers from memory just
once and reused for computation for some many times (nr
times for the micro panel of A and m r times for the micro
panel of B). What is more, multiple output elements can
be calculated simultaneously with SIMD instructions, which
further improves the computation performance.
2) Packing: It is generally acknowledged that, because of
the widely used memory hierarchy architecture in modern
CPUs, accessing memory in a continuous way during the
computation is much more efficient than noncontinuous
memory access. For the purpose of continuous memory
access during computation of GEMM, the packing operation
is actually needed. To this end, two areas of shared memory
are needed, i.e., tensor A∗ with size of (m/m r ) × pc × m r
and tensor B ∗ with size (n c /nr ) × pc × n r . By packing the
corresponding elements of A and B into these two tensors
of A, denoted as Â, is packed into A∗ . In this process,
 is blocked along the m dimension with the factor m r ,
generating multiple m r × pc micro panels. These m/m r 
micro panels are stored in A∗ one by one in column-major
order.2 Similarly, one pc × n c submatrix of B, denoted as B̂,
is organized into multiple pc × nr micro panels and stored
in B ∗ one by one in row-major order. In this way, during the
multiplication of submatrices of A and B, memory access of
A∗ and B ∗ is actually continuous [47].
For ease of presentation, we formally define two operators
packa and packb so that
[h, w] = packa([m, pc, mr ]) ⇒ A∗ [m, pc, mr ] = Â[h, w]
[h, w] = packb([nc, pc, nr ]) ⇒ B ∗ [nc, pc, nr ] = B̂[h, w].
The operators packa and packb can be defined by the
following equations:
packa([m, pc, mr ]) = [m × m r + mr, pc]
packb([nc, pc, nr ]) = [ pc, nc × nr + nr ]. (2)
IV. M ETHODOLOGY
The main drawback of im2col convolution algorithm is that
it needs a large area of memory with size i c kh kw oh ow for
the storage of the transformed input tensor, which is several
times larger than the size of feature map. In this section,
we introduce the main idea of ECBC. Our method only needs
a small area of shared memory to store the packed inputs, and
it can be generalized to arbitrary convolutional architectures.
A. Computation Pipeline
Fig. 3 briefly describes the computation pipeline of ECBC.
The proposed algorithm is a principled improvement over
2 Row-major order and column-major order are two commonly used format

with a specific organization, the memory access during the for storing multidimensional tensors in linear memory. In row-major order,
adjacent elements in one row of the matrix are stored in a continuous area of
GEPB computation shown at the bottom of Fig. 2(b) is memory, whereas in column-major order, adjacent elements in one column of
completely continuous. For instance, one m × pc submatrix the matrix are stored in a continuous area of memory.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
 ZHAO et al.: ECBC 437

Algorithm 1 ECBC Algorithm

Require: Input tensor I , packed input kernel K ∗∗ , shared
tensor Is , block factors pc , n c , m r , n r
Ensure: Output tensor O
1: Reinterpret O as a oc × oh ow matrix
2: Initialize O as all zeros
3: for p ∈ [0 : i c k h k w ] in steps of pc do
4: pc = M I N( pc , i c kh kw − p)
5: for n ∈ [0 : oh ow ] in steps of n c do
6: nc = M I N(n c , oh ow − n)
7: Reshape Is as a  nc n r  × pc × n r tensor
8: for i ∈ [0 :  nc nr
] in parallel do
9: for j ∈ [0 : pc] do
10: for k ∈ [0 : nr ] do
11: [i h∗ , i w∗ ] = packb([i, j, k])
12: [c, h, w] = i m2col([ p + i h∗ , n + i w∗ ])
13: Is [i, j, k] = I [c, h, w]
Fig. 3. Overview of ECBC. The red arrows illustrate the computation 14: end for
flow of im2col-based convolution with GEMM implemented with the block
algorithm described in Section III-C. It first packs the input tensor I into a
15: end for
large matrix I ∗ with im2col and then packs each subblock of I ∗ into the shared 16: end for
tensor Is . While the arrow labeled by packing denotes ECBC’s computation 17: for mr ∈ [0 : oc ] in steps of m r in parallel do
flow, for each subblock of I ∗ , we directly pack the corresponding elements
of the input tensor I into the shared tensor Is , thus deprecating the significant
18: for nr ∈ [0 : nc] in steps of nr do
memory overhead needed by I ∗ . 19: O[mr : mr + m r , n + nr : n + nr + nr ]+ =
K ∗∗ [ ppc , mr m r , :, :] × Is [ n r , :, :]
nr

20: end for

im2col-based convolution. Red arrows in Fig. 3 show the
computation flow of im2col-based convolution. The input
tensor I with dimension i c × i h × i w is first unfolded and
duplicated into a flat matrix I ∗ with dimension i c kh kw × oh ow
with im2col operation, and then, the output tensor O can
be obtained by GEMM between the kernel matrix K and
the transformed inputs I ∗ . Following the block algorithm of
GEMM, the matrix I ∗ is blocked into multiple pc × n c
submatrices, and the computation of GEMM is decomposed
into multiple subproblems as
 p ∗ p ,n
O:,n cj = K :,ic × Ii, j c c . (3)
i B. Data Layout of Kernels
p ∗ p ,n
In each iteration, before computing K :,ic × Ii, j c c , the cor-
∗ p ,n
responding submatrix Ii, j c c is packed into the shared
(n c /nr ) × pc × nr tensor Is for continuous memory access.
An interesting observation is that in the summation illus-
trated in (3), for a specific index i, j , the corresponding sub-
∗ p ,n ∗ p ,n
matrix Ii, j c c is multiplied just once. Specifically, after Ii, j c c
pc
is multiplied by K :,i , it will never be used in the subsequent
computation. In other words, for different indices i and j ,
the corresponding submatrices are decoupled. This indicates
that there is no need to store these submatrices separately
as what is done in the im2col-based convolution algorithm.
Instead of packing the input tensor I into a flat matrix I ∗ and
∗ p ,n
then packing each submatrix Ii, j c c to the shared tensor Is ,
we directly pack the corresponding subpatch of I with respect
∗ p ,n
to the submatrix Ii, j c c into the shared tensor Is in each
iteration (the arrow labeled with packi ng in Fig. 3). After
p K [i, j, k, l] = K ∗ [i, j kh kw + kkw + l].
Is is filled, we multiply Is by the corresponding submatrix K :,ic
of the kernel tensor and accumulate the results to the cor-
responding submatrix O:,n cj of the output tensor. In this way,
the transformed inputs I ∗ are entirely deprecated. The only
21: end for
22: end for
23: end for
24: Reinterpret O as a oc × oh × ow tensor
needed extra memory is the fixed size shared tensor Is , which
is much smaller than I ∗ . Results in the final algorithm are
shown in Algorithm 1. We will describe the algorithm in detail
in the following.
Recall that in the block algorithm of GEMM, the left matrix
(e.g., the kernel matrix in convolution) is supposed to be
blocked and packed in a special format for continuous memory
access during computing. The reformatting is very important
to the computation performance but incurs additional memory
and time cost. Note that once trained, elements of the kernel
matrix are fixed. Thus, here in ECBC, we address the problem
by blocking and packing the kernel in advance. With proper
kernel layout, inference can be conducted without introducing
extra memory and time cost. To achieve this goal and fulfill
the requirement of ECBC, a simple but novel storage format
of the kernel tensor is proposed.
Fig. 4 shows the proposed data layout of kernel data. First,
the kernel tensor K of size oc × i c × kh × kw is reinterpreted
as a oc × i c kh kw matrix K ∗ , i.e.,
(4)
K ∗ is then packed into a tensor K ∗∗ with size
(i c kh kw )/ pc  × oc /m r  × pc × m r . The matrix K ∗ is
first blocked along the column dimension with factor pc ,

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
 438 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 1, JANUARY 2023
Fig. 4. Proposed data layout of the kernel data. The original kernel tensor K
with shape oc × ic × kh × kw is first flattened into a oc × ic kh kw matrix K ∗ .
The matrix is then blocked along the column dimension with factor n c and
finally the row dimension with factor m r , generating multiple micro panels
with size m r × n c . These micro panels are stored in memory one by one.
The fastest dimension is m r , followed by n c . Note that this process does
not incur any extra memory overhead because the kernel tensors are fixed in
CNN models after training so that they can be preorganized in advance.
generating (i c kh kw )/ pc  submatrices with size oc × pc . These
submatrices are stored in K ∗∗ one by one, with the same
format as that of A∗ described in Section III-C, i.e.,
     modern CPUs, parallelism is also considered in our algorithm.
w h
K ∗ [h, w] = K ∗∗ , , w% pc , h%m r . (5)
pc mr
C. Packing of Input Data
As what has been described in Section III-C, the right
matrix of GEMM (e.g., the input data in convolution) should
also be blocked and packed for continuous memory access.
Different from the case of kernel data, the blocking and
packing operations of the input data cannot be done in
advance because they are dependent on the input image of
CNN models and thus are different for different computations.
For this reason, we apply the blocking and packing operations
on-the-fly.
Fortunately, through proper organization of computation
loops, this procedure does not introduce much memory cost.
In Algorithm 1, the key insight lies in that, each iteration of the
first two loops (e.g., the loops of lines 3 and 5) is related to a
∗ p ,n
pc × n c submatrix I p,nc c of I ∗ , and this submatrix will never
be used in the subsequent iterations. Thus, instead of storing
all of these submatrices separately in memory, we only pack
the corresponding pc × n c submatrix into a shared tensor,
say Is , at the very beginning of each iteration. In this way,
the additional memory overhead is reduced to the size of Is
(e.g., pc × n c ), which is far below that of the other indirect
convolution algorithms.
D. Blocking Factors
The blocking factors pc , n c , m r , and nr are carefully
selected to match the architecture parameters of CPUs, such
as the cache size, the number, and length of SIMD registers.
In particular, the shared tensor Is should match the L2 cache
size. In each iteration, after packing pc × n c elements of I
into Is , it will be stored into L2 cache and reused during
the loop in lines 17–21. Line 19 performs the inner product
between one micro panel of K ∗∗ with size m r × pc and one
micro panel of Is with size pc × n r , which is highly optimized
and implemented assembly with SIMD instructions. In this
procedure, the m r × nr intermediate results of O are stored
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
into SIMD registers and reused during the whole iteration,
so the blocking factors m r and nr are upper bounded by the
number and length of SIMD registers in CPUs. The factor pc
should be as large as possible to reuse these m r × nr register
values sufficiently. However, to avoid stalls caused by cache
miss, we must ensure that the m r × pc micro panel of the
kernel K ∗∗ along with the nr × pc micro panel of Is should
match the L1 cache size, so that in each iteration in line 17,
the corresponding m r × pc micro panel of K ∗∗ resides in
L1 cache and is reused during the loop in line 18. In this
respect, n c should be as large as possible to make full use of
these L1 cache values. What is more, the loop in line 8 for
packing procedure and line 17 for multiplication calculation
can be done in parallel, which further improves computation
efficiency.
E. Parallelism
In order to make full use of multicore architectures in
Note that it is not sufficient to parallelize the out-most two
loops. Since different iterations of these two loops need dif-
ferent subpatches of the input data (e.g., pc × n c submatrices
of I ∗ ). This means that one distinct shared tensor Is will be
required for each thread in order for completing the subsequent
computations, leading to multiple times of additional memory
overhead. Recall that in Section IV-D, it is expected that the
shared tensor Is should be stored into L2 cache, which is
shared across many cores in the context of mobile CPUs.
However, these multiple shared tensors will introduce further
additional memory overhead and may even exceed the size of
L2 cache, leading to performance degradation. What is worse,
different iterations of the out-most loop will accumulate the
intermediate results into the same area of the output data.
This means that parallelization of this loop will cause write
conflict between different threads, thus incurring time penalty
on system performance.
For the reasons described above, in our algorithm, the par-
allelization is applied on the inner iterations. To be more
detailed, the packing of the input data is parallelized along
the n c dimension and the subsequent computation is paral-
lelized along the oc dimension. This means that each thread
is assigned to a block of channels of the output data for
computation. In this way, only one area of shared tensor Is is
required, and the L2-cache data are shared across multithreads.
In each iteration of one specific thread, the corresponding
m r × pc micro panel of K ∗∗ is fetched in its private L1 cache
and reused for computation.
F. Analysis
The main goal of our method is to optimize the memory
access during computation. In this section, we analyze and
compare our method with other algorithms in terms of conti-
nuity of memory access and overall memory ahead.
1) Continuity of Memory Access: It is a common agree-
ment that accessing memory in a continuous manner is more
efficient than noncontinuous memory access because of the
popular utilized hierarchical memory architecture in modern
 ZHAO et al.: ECBC
Fig. 5. Storage format and memory access order of (a) directly implemented
convolution algorithm [13] and (b) proposed ECBC. During computation of
ECBC, the memory access is totally continuous, which is friendly to cache,
resulting in its high computation performance.
CPUs. This is very important to the system performance.
In this section, we mainly focus on the continuity of memory
access during computation (e.g., lines 17–21 in Algorithm 1).
Fig. 5 shows the comparison of storage format and memory
access order of the start-of-the-art directly implemented convo-
lution algorithm [13] and the proposed ECBC. In the method
of [13], the input data are first blocked along the channel
dimension with some factor cblk , arising (i c /cblk ) blocks of
data, each with shape cblk × i h × i w . All of these blocks
are stored in memory one by one, and the fastest dimension
is the blocked channels cblk , followed by i h and i w . While
during computation, the fastest accessing dimension is along
 tions [14]–[16], the blocking factor x can be different for
the input channel i c with another blocking factor cblk , which is
larger than cblk . In this context, during the first cblk iterations of
computation, elements at the first cblk channels [e.g., the ones
with blue colors at the bottom of Fig. 5(a)] are accessed.
In this stage, the memory access is continuous. However,
at iteration (cblk + 1), the program tries to load data at the
(cblk + 1)th channel [e.g., the ones with orange colors at
the bottom of Fig. 5(a)]. These data are not continuous with
the first cblk ones, which will cause cache miss, and thereby
leading to performance degradation.
While in ECBC, the core computation is the multiplication
between one m r × pc micro panel of the kernel tensor and one
pc × n c submatrix of the input tensor, which is packed into
the shared tensor Is . Recall that each pc × n c submatrix of
the input data is blocked along the n c dimension into (n c /nr )
micro panels and stored in memory one by one, and the
fastest dimension is the blocked column. During computation
illustrated in lines 18–20 of Algorithm 1 and the bottom of
Fig. 3, these micro panels are multiplied by the corresponding
kernel data one by one. The storage order of data is the same
as the order they are accessed, which means that the memory
access of ECBC is totally continuous.
2) Memory Overhead: In this section, we analyze and
compare the memory overhead of different convolution algo-
rithms. For all the convolution algorithms, the total memory
overhead Ralg contains four terms: the memory needed by the
input Rin , output Rout , parameters Rker , and additional memory
overhead Rext
Ralg = Rin + Rout + Rker + Rext (6)
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
439
and the memory needed by the input–output and kernel tensors
is the same for all the algorithms, despite that the data layout
of the kernel tensor is different across different algorithms
Rin = i c i h i w
Rout = oc oh ow
Rker = oc i c kh kw . (7)
We start from the extra memory overhead of the
im2col-convolution algorithm [11], [35]. The algorithm needs
a buffer for the storage of the matrix generated by the im2col
procedure. The size of the transformed tensor is
im2col
Rext = i c k h k w × oh ow . (8)
We now analyze the additional memory overhead of
MEC [12]. The main idea of the algorithm can be seen
as to apply the im2col operation on only one dimension,
instead of both the oh and ow dimension as in the im2col
convolution algorithm. Following the analysis of the original
paper, the needed extra memory overhead is
MEC
Rext = i c (i h + 2 ph )kw ow . (9)
The additional memory overhead of direct convolution
algorithms depends on the actual scenario because these
algorithms are implemented under some special data for-
mat. Between which the most commonly used format is
(C/x) × H × W × x [13]–[16]. In some implementa-
input–output tensors. In this case, additional memory overhead
with the same size as the input tensor is needed for data
format transformation. For other implementations such as
Zhang et al.’s method [13], the x factor is predefined and
keeps the same between the input and output tensors, so no
additional memory is needed for computation.
Finally, we analyze the additional memory overhead
of ECBC. In our algorithm, the only needed extra memory
is the shared tensor Is with size pc × n c
ECBC
Rext = pc × n c . (10)
pc and n c are architecture-specific parameters. In all the
implementations of our experiments, we set pc = n c = 256,
which is fixed and much smaller than the size of input–output
tensors, and can be shared between all the convolution layers
in a model.
V. E XPERIMENTAL R ESULTS
To evaluate the proposed algorithm, in this section,
we present the comparison between ECBC and other widely
used industrial-level convolution algorithms on various plat-
forms and networks.
Experiments are conducted on two platforms with different
ARM architectures:
1) 2× ARM Cortex-A72 CPUs, with frequency up to
1.8 GHz. ARM V8 architecture;
2) 4× ARM Cortex-A17 CPUs, with frequency up to
1.8 GHz. ARM V7 architecture.
 440 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 1, JANUARY 2023

Fig. 6. Comparison of single-thread computation performance of different algorithms on convolution layers. The notes under each group of bars denote the
configuration of convolutions. For example, the note (64, 56, 3)(128, 7, 1) denotes a 7 × 7 convolution with 64 input channels and 128 output channels. The
spatial size of the input tensor is 56 × 56. Also, the stride and padding of the convolution are 1 and 3, respectively. (a) Speedup with respect to im2col +
OpenBLAS of different algorithms on convolution layers on a single A72 core. (b) Speedup with respect to im2col + OpenBLAS of different algorithms on
convolution layers on a single A17 core.

A. Experimental Setup and padding) appearing frequently in CNN models. We do not

We implement ECBC on ARM CPUs in C++ with
OpenMP for parallel computing. In all of our experiments,
the blocking factors pc and n c are all set to 256. m r and
nr are different on different architectures because the number
of registers is different. On ARM Cortex-A72 cores, we set
m r = 4, nr = 16, while on ARM Cortex-A17 cores, we set
m r = 4, nr = 8.
We compare our proposed ECBC with the popular im2col
convolution algorithm. We extract the implementation of the
im2col convolution algorithm from the popular deep learn-
ing framework Caffe [35] and utilize OpenBLAS [37] and
Eigen [40] as high-performance matrix multiplication routine.
We also present a comparison of ECBC with the convolution
implementations provided by NNPACk [50] and TFLite [51],
which are two of the most popular frameworks for CNN
acceleration on mobile devices. Besides, we also compare
ECBC with the MEC [12] and the state-of-the-art direct
convolution algorithm ZMC [13]. Both of these convolution
algorithms are not open-sourced, so we reimplemented them
by ourselves.
B. Results on Convolution Layers
We conduct 22 convolution layers with various configura-
tions (input–output channels, spatial size, kernel size, stride,
conduct experiments on 1 × 1 convolutions because they are
exactly equivalent to GEMM, where there will be no difference
between our method and the traditional blocking algorithm
of GEMM described in Section III-C. We will report the
computation performance and memory overhead of different
algorithms on these convolution layers in this section.
To evaluate the computation efficiency, we run a convo-
lution layer with each algorithm 100 times and record the
average run time. We report the speed up of each algorithm
normalized to im2col + OpenBLAS. Both single-thread and
multithread performances are presented. The results are sum-
marized in Figs. 6 and 7. The notes under each group of bars
denote the convolution configurations. The gray dashed lines
are the performance of im2col + OpenBLAS. We can see
that ECBC achieves the best performance under all the cases
excluding seven convolutions on 2× ARM Cortex-A72 cores,
where ECBC performs slightly inferior compared to ZMC.
Fig. 6(a) shows the performance comparison between ECBC
and other methods on a single ARM Cortex-A72 CPU. Com-
pared to im2col convolution algorithm, ECBC is approxi-
mately 1.2× and sometimes nearly 2× faster. ZMC performs
better than im2col convolution on most layers, while it still
performs inferior compared to ECBC. Specifically, ECBC out-
performs ZMC by about 10%, and the improvement achieves
up to 40% under some configurations.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
 ZHAO et al.: ECBC 441

Fig. 7. Comparison of single-thread computation performance of different algorithms on convolution layers. The notes under each group of bars denote the
configuration of convolutions. For example, the note (64, 56, 3)(128, 7, 1) denotes a 7 × 7 convolution with 64 input channels and 128 output channels. The
spatial size of the input tensor is 56 × 56. Also, the stride and padding of the convolution are 1 and 3, respectively. (a) Speedup with respect to im2col +
OpenBLAS of different algorithms on convolution layers on 2× A72 cores. (b) Speedup with respect to im2col + OpenBLAS of different algorithms on
convolution layers on 2× A17 cores. (c) Speedup with respect to im2col + OpenBLAS of different algorithms on convolution layers on 3× A17 cores.
(d) Speedup with respect to im2col + OpenBLAS of different algorithms on convolution layers on 4× A17 cores.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
 442 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 1, JANUARY 2023

Fig. 8. Comparison of different convolution algorithms in terms of run time and memory overhead. (a) Computation performance on resnet18 models with
different widths. All the performances are measured on a single ARM Cortex A72 Core. Lower is better. (b) Memory overhead of resnet18 models with
different widths. Lower is better. (c) Tradeoff between run time and memory overhead on resnet18 models with different widths. Lower is better.

Fig. 6(b) shows the performance comparison between TABLE II

ECBC and other methods on a single ARM Cortex-A17 CPU. RUN T IME OF D IFFERENT C ONVOLUTION A LGORITHMS ON
D IFFERENT CNN M ODELS . T HE VALUES IN TABLES A RE
In this case, the advantages of all the convolution algorithms IN M ILLISECONDS . (a) R ESULTS ON A S INGLE A17
are more obvious. NNPACK and TFLite perform better than C ORE . (b) R ESULTS ON 4× A17 C ORES .
the im2col convolution, and the improvement factor is approxi- (c) R ESULTS ON A S INGLE A72 C ORE .
(d) R ESULTS ON 2× A72 C ORES
mately 1.5×, while ECBC still performs the best among all the
algorithms. Compared to the im2col convolution algorithms,
ECBC is more than 2× faster on all the convolution layers.
Fig. 7(a)–(d) shows the comparisons of computation effi-
ciency of different algorithms in multithread context. Overall,
the performance improvement of our algorithm is greater than
that in a single-thread case. For example, on a single ARM
Cortex-A17 CPU, ECBC is faster than im2col + OpenBLAS
by a factor of less than 2.5×, while on 4× ARM Cortex-
A17 cores, the performance improvement is more than 2.5×
on most layers. On conv3, the improvement factor even
achieves more than 3×. Similar conclusions also hold on ARM
Cortex-A72.
In general, the performance improvement is more obvious
on ARM Cortex-A17 cores. For example, by comparison of
Fig. 6(a) and (b), we can find that on a single ARM Cortex-
A72 CPU, ECBC performs about 1.25× better than im2col +
OpenBLAS, while the improvement is nearly 2.5× on a single
ARM Cortex-A17. The results shown in Fig. 7(a) and (d) show
similar trends. We can see that on 2× ARM Cortex-A72 cores,
ECBC is about 1.25× faster than im2col + OpenBLAS, while
it is overall more than 2.5× and up to 3× faster on 4×
ARM Cortex-A17 cores. The main reason is that ARM Cortex-
A17 CPU is more limited by memory access and bandwidth,
so it gains more benefits from memory access optimization. see that ECBC performs the best against all the algorithms
We believe that the performance gap between ECBC and in all the cases except on GoogleNet and InceptionV3 on
these baseline implementations can be larger on more low-end 2× ARM Cortex-A72 cores, where ECBC performs slightly
devices, which may be a possible further direction of our work. slower compared to ZMC.
Similar to the results of experiments conducted on stand-
alone convolution layers, overall, the performance improve-
C. Results on CNNs ment of ECBC is more obvious when running with multiple
For comprehensive evaluation, we run and compare the threads or on relatively more bandwidth-constrained platform
performance of different algorithms on real models [1]–[3], (the ARM Cortex-A17 CPU with ARMV7 architecture). For
[52], [53]. For each model, we measure and report the summa- example, by comparison of Table II(a) and (b), ECBC is over-
tion of run time of its convolution layers. The results are shown all nearly 2.5× faster than the im2col convolution algorithm on
in Table II, and the run time is shown in milliseconds. We can a single ARM Cortex-A17 core, while when running with 4×

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
 ZHAO et al.: ECBC
TABLE III
D ETAILED C OMPUTATION P ERFORMANCE OF D IFFERENT C ONVOLUTION
A LGORITHMS ON C ONVOLUTION L AYERS OF A LEX N ET [1]. T HE
VALUES IN TABLES A RE IN M ILLISECONDS . (a) P ERFORMANCE ON
A S INGLE A17 C ORE . (b) P ERFORMANCE ON 4× A17 C ORES .
(c) P ERFORMANCE ON A S INGLE A72 C ORE .
(d) P ERFORMANCE ON 2× A72 C ORES
cores, ECBC is always more than 2.5× faster. The speedup
even achieves 4× on GoogleNet and 5× on resnet50. The
same trends also hold on ARM Cortex-A72 cores.
Table II(a) and (b) support that the advantages of ECBC
are more obvious on relatively more bandwidth-constrained
platforms. Specifically, ECBC is overall 1.2× faster compared
to the im2col convolution on a single ARM Cortex-A72, while
the speedup is more than 2× on a single ARM Cortex-A17.
Table II(b) and (d) also shows the same results, where ECBC
is 1.2–1.5× faster than the im2col convolution algorithm on
2× A72 cores, while the improvement is more than 2.5× and
up to 5× on 4× A17 cores.
In Table III, we show the detailed run time of different con-
volution algorithms on each convolution layer of AlexNet [1].
The values in tables are in milliseconds. We can see that ECBC
performs the best under all the cases excluding on conv4 and
conv5 on 2× A72 cores. An interesting fact that is noteworthy
is that, the performance of ECBC is stable and efficient, that is
to say, the performance of ECBC is not bad under all the cases,
while other methods may perform well under some cases and
suffer poor performance under some other cases. For example,
the performance of ZMC [13] is comparable with ECBC on
conv4 and conv5 on A72 CPUs and even faster than ECBC
by 0.21 and 0.82 ms relatively on these two layers on 2×
A72 cores, while on other layers or platforms, it performs
inferior compared to ECBC. On conv1, ECBC is even more
than 1.5× faster than ZMC on all the platforms. The same
trends also hold for the performance of other algorithms. For
example, the performance of MEC [12] is similar to ECBC
on conv1 on A72 CPUs, and ECBC outperforms MEC by a
large margin on other layers or platforms.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
443
Fig. 9. Normalized memory overhead of different convolution algorithms
on real CNN models. In this figure, all the memory overhead includes the
memory needed for the input–output/parameters of the models and additional
memory needed by various algorithms for computation are considered. Lower
is better.
D. Memory Overhead
In Section IV-F, we analyze the memory overhead of
different convolution algorithms theoretically. This section will
prove the memory efficiency of ECBC through experiments.
In Fig. 8, we show the tradeoff among run time and memory
overhead of different algorithms on resnet18 models with
different widths. In particular, in Fig. 8(a), we show the run
time in milliseconds of different convolution algorithms on
resnet18 models with different widths. Latency of all the
models is measured on a single A72 core, and we can see that
ECBC consistently achieves the best performance under all the
cases. Fig. 8(b) shows the memory overhead of different con-
volution algorithms. In this figure, all the memory overhead,
including the memory required by convolution inputs/outputs,
parameters of networks, and extra memory overhead required
during computation, are considered. Note that the memory
overhead of convolution implementation of nnpack and tflite
is not shown in this figure. We mainly compare the memory
overhead of ZMC [13], which needs the least memory among
all the algorithms because it does not introduce any additional
memory overhead at all. We can see that the memory overhead
of ECBC is obviously lower than im2col and MEC and almost
the same with ZMC under all the cases. To further prove that
ECBC is able to achieve a better tradeoff between run time
and memory overhead, in Fig. 8(c), we plot the summation
of run time (in milliseconds) and memory overhead (in mega
parameters) of different algorithms (lower is better). We can
see that compared to other convolution algorithms, ECBC
consistently performs superior in terms of tradeoff between
latency and memory overhead.
Fig. 9 shows the memory overhead of different convolution
algorithms on different CNN models. Similar to Fig. 8, in this
figure, all the memory overhead includes the memory needed
for storage of the inputs/outputs/parameters of models, and
additional memory needed by various algorithms for compu-
tation of convolutions is considered. Lower is better. From
Fig. 9, we can see that ECBC is able to save at least 30%
and up to 75% memory overhead compared to the im2col
convolution algorithm. Also, the memory needed by ECBC
is almost the same as that of ZMC [13]. In fact, compared
 444 IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS, VOL. 34, NO. 1, JANUARY 2023
to ZMC, a small piece of additional memory is needed by
ECBC for the storage of the shared tensor Is . While, as what
has been mentioned at the end of Section IV-F, the size of
Is is fixed and independent of the convolution configurations,
it can be shared across all the convolutions in a model, thereby
largely reducing the total memory overhead.
VI. C ONCLUSION
In this article, we propose a simple and efficient convo-
lution algorithm for mobile devices. Instead of transforming
the whole input tensor before computing, we reduce the
memory overhead through blocking and memory sharing.
We show that, by carefully designing the loop order and
selecting the block size, the continuity and reusability of
memory access can be improved, and memory overhead can
also be largely reduced through memory sharing. Evaluated
on multiple networks and platforms, our method gets high
computation and memory efficiency compared to previous
algorithms. The results support that ECBC gains more benefits
on low-end devices because they are more limited by memory
resources and thus more sensitive to memory optimization.
What is more, ECBC also shows great scalability in that the
performance improvement is more obvious when in multi-
thread context. Moreover, the performance of ECBC is stable
and efficient. It performs not bad under a various range of
platforms, convolutions, and CNN models.
The main idea of ECBC is to conduct the convolution
computation blockwisely so that the tensor transformation
operation can also be done in a blockwise manner, and
the memory overhead can be thus largely reduced with
memory sharing. A possible further direction of this work
is to optimize the memory cost of domain-transformation
convolution methods (i.e., FFT-based [9] or Winograd-based
[10] convolution algorithms) with the similar idea, which will
further improve the practical utility of these computation effi-
cient while memory-hungry convolution algorithms on mobile
applications.
R EFERENCES
[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classifica-
tion with deep convolutional neural networks,” in Advances in Neural
Information Processing Systems, vol. 25, F. Pereira, C. J. C. Burges,
L. Bottou, and K. Q. Weinberger, Eds. Red Hook, NY, USA:
Curran Associates, 2012, pp. 1097–1105.
[2] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” in Proc. 3rd Int. Conf. Learn. Represent.
(ICLR), 2015, pp. 1–14.
[3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for
image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.
(CVPR), Jun. 2016, pp. 770–778.
[4] W. Liu et al., “SSD: Single shot multibox detector,” in Proc. Eur. Conf.
Comput. Vis. (ECCV), 2016, pp. 21–37.
[5] J. Redmon and A. Farhadi, “YOLOv3: An incremental improve-
ment,” 2018, arXiv:1804.02767. [Online]. Available: http://arxiv.org/abs/
1804.02767
[6] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-
time object detection with region proposal networks,” in Proc. 28th Int.
Conf. Neural Inf. Process. Syst. (NIPS), vol. 1. Cambridge, MA, USA:
MIT Press, 2015, pp. 91–99.
[7] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks
for semantic segmentation,” in Proc. IEEE Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jun. 2015, pp. 3431–3440.
Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.
[8] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing
network,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
Jul. 2017, pp. 2881–2890.
[9] T. Highlander and A. Rodriguez, “Very efficient training of convo-
lutional neural networks using fast Fourier transform and overlap-
and-add,” 2016, arXiv:1601.06815. [Online]. Available: http://arxiv.
org/abs/1601.06815
[10] A. Lavin and S. Gray, “Fast algorithms for convolutional neural net-
works,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR),
Jun. 2016, pp. 4013–4021.
[11] K. Chellapilla, S. Puri, and P. Simard, “High performance convolutional
neural networks for document processing,” in Proc. 10th Int. Workshop
Frontiers Handwriting Recognit., 2006, pp. 1–7.
[12] M. Cho and D. Brand, “MEC: Memory-efficient convolution for deep
neural network,” in Proc. 34th Int. Conf. Mach. Learn. (ICML), 2017,
pp. 815–824.
[13] J. Zhang, F. Franchetti, and T. M. Low, “High performance zero-memory
overhead direct convolutions,” in Proc. 35th Int. Conf. Mach. Learn.
(ICML), 2018, pp. 5776–5785.
[14] E. Georganas et al., “Anatomy of high-performance deep learning
convolutions on SIMD architectures,” in Proc. Int. Conf. High Perform.
Comput., Netw., Storage Anal., Nov. 2018, pp. 830–841.
[15] E. Georganas et al., “Harnessing deep learning via a single building
block,” in Proc. IEEE Int. Parallel Distrib. Process. Symp. (IPDPS),
May 2020, pp. 222–233.
[16] Y. Liu, Y. Wang, R. Yu, M. Li, V. Sharma, and Y. Wang, “Optimizing
CNN model inference on CPUs,” in Proc. USENIX Annu. Tech. Conf.,
2019, pp. 1025–1040.
[17] M. Dukhan, “The indirect convolution algorithm,” CoRR,
vol. arXiv:1907.02129, pp. 1–10, Jul. 2019.
[18] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing
deep neural networks with pruning, trained quantization and Huffman
coding,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2016, pp. 1–14.
[19] W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li, “Learning structured
sparsity in deep neural networks,” in Proc. Adv. Neural Inf. Process.
Syst., 2016, pp. 2074–2082.
[20] S. Guo, Y. Wang, Q. Li, and J. Yan, “DMCP: Differentiable Markov
channel pruning for neural networks,” in Proc. IEEE Conf. Comput. Vis.
Pattern Recognit. (CVPR), Jun. 2020, pp. 1539–1547.
[21] J. Yu, L. Yang, N. Xu, J. Yang, and T. S. Huang, “Slimmable
neural networks,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2019,
pp. 1–12.
[22] J. Yu and T. Huang, “Universally slimmable networks and improved
training techniques,” in Proc. Int. Conf. Comput. Vis. (ICCV), Oct. 2019,
pp. 1803–1811.
[23] Z. Yang et al., “Searching for low-bit weights in quantized neural
networks,” in Proc. Int. Conf. Neural Inf. Process. Syst. (NeurIPS), 2020,
pp. 1–12.
[24] Z. Liu, Z. Shen, M. Savvides, and K.-T. Cheng, “ReactNet: Towards
precise binary neural network with generalized activation functions,” in
Proc. 16th Eur. Conf. Comput. Vis. (ECCV), 2020, pp. 143–159.
[25] Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K.-T. Cheng, “Bi-
Real Net: Enhancing the performance of 1-bit CNNs with improved
representational capability and advanced training algorithm,” in Proc.
Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 722–737.
[26] P. Wang and J. Cheng, “Fixed-point factorized networks,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017,
pp. 4012–4020.
[27] C. Leng, Z. Dou, H. Li, S. Zhu, and R. Jin, “Extremely low bit neural
networks: Squeeze the last bit out with ADMM,” in Proc. 32th AAAI
Conf. Artif. Intell. (AAAI), 2018, pp. 1–8.
[28] E. L. Denton, W. Zaremba, J. Bruna, Y. LeCun, and R. Fergus,
“Exploiting linear structure within convolutional networks for efficient
evaluation,” in Proc. 38th Conf. Neural Inf. Process. Syst. (NeurIPS),
2014, pp. 1269–1277.
[29] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional
neural networks with low rank expansions,” in Proc. Brit. Mach. Vis.
Conf., 2014, pp. 1–13.
[30] V. Lebedev, Y. Ganin, M. Rakhuba, I. V. Oseledets, and V. S. Lempitsky,
“Speeding up convolutional neural networks using fine-tuned CP-
decomposition,” in Proc. 3rd Int. Conf. Learn. Represent. (ICLR), 2015,
pp. 1–11.
[31] S. Winograd, Arithmetic Complexity of Computations, vol. 43, no. 2.
Philadelphia, PA, USA: Society for Industrial and Applied Mathematics,
1980.
 ZHAO et al.: ECBC 445

[32] Z. Jia, A. Zlateski, F. Durand, and K. Li, “Optimizing N-dimensional, Qinghao Hu received the B.E. degree in computer
Winograd-based convolution for manycore CPUs,” in Proc. 23rd science from Northwestern Polytechnical University,
ACM SIGPLAN Symp. Princ. Pract. Parallel Program., Feb. 2018, Xi’an, China, in 2014, and the Ph.D. degree from the
pp. 109–123. National Laboratory of Pattern Recognition, Insti-
[33] A. Xygkis, D. Soudris, L. Papadopoulos, S. Yous, and D. Moloney, tute of Automation, Chinese Academy of Sciences,
“Efficient winograd-based convolution kernel implementation on edge Beijing, China, in 2019.
devices,” in Proc. 55th Design Autom. Conf. (DAC), Jun. 2018, pp. 1–6. He currently works as a Research Assistant at
[34] Z. Jia, A. Zlateski, F. Durand, and K. Li, “Towards optimal Winograd the Institute of Automation, Chinese Academy of
convolution on manycores,” in Proc. SysML, 2018, pp. 1–3. Sciences. His current research interests include deep
[35] Y. Jia et al., “Caffe: Convolutional architecture for fast feature embed- neural network compression and acceleration, hash-
ding,” in Proc. ACM Int. Conf. Multimedia, Nov. 2014, pp. 675–678. ing, and quantization.
[36] J. J. Dongarra, J. D. Croz, and S. Hammarling, “A set of level 3 basic
linear algebra subprograms,” ACM Trans. Math. Softw., vol. 16, no. 1,
pp. 1–17, 1990.
[37] Z. Xianyi, W. Qian, and Z. Yunquan, “Model-driven level 3 BLAS Xiangyu He received the B.E. degree in information
performance optimization on Loongson 3A processor,” in Proc. IEEE security from the Beijing University of Posts and
18th Int. Conf. Parallel Distrib. Syst. (ICPADS), Dec. 2012, pp. 684–691. Telecommunications, Beijing, China, in 2017. He is
[38] Q. Wang, X. Zhang, Y. Zhang, and Q. Yi, “AUGEM: Automatically currently pursuing the Ph.D. degree with the Insti-
generate high performance dense linear algebra kernels on x86 CPUs,” tute of Automation, Chinese Academy of Sciences,
in Proc. Int. Conf. High Perform. Comput., Netw., Storage Anal. (SC), Beijing.
Nov. 2013, pp. 17–21. His current research interests include deep learning
[39] Intel. (2020). MKL: Intel Math Kernel Library. [Online]. Avail- and image retrieval.
able: https://software.intel.com/content/www/us/en/develop/tools/math-
kernel-library.html
[40] G. Guennebaud et al. (2010). Eigen V3. [Online]. Available: http://eigen.
tuxfamily.org
[41] A. Gural and B. Murmann, “Memory-optimal direct convolutions for
maximizing classification accuracy in embedded applications,” in Proc. Weixiang Xu received the B.E. degree in automa-
36th Int. Conf. Mach. Learn. (ICML), 2019, pp. 2515–2524. tion from Northeastern University, Shenyang, China,
[42] T. Chen et al., “TVM: An automated end-to-end optimizing compiler in 2018. He is currently pursuing the Ph.D. degree
for deep learning,” in Proc. 12th USENIX Symp. Operating Syst. Design with the National Laboratory of Pattern Recogni-
Implement. (OSDI), 2018, pp. 578–594. tion, Institute of Automation, Chinese Academy of
[43] S. Zheng, Y. Liang, S. Wang, R. Chen, and K. Sheng, “Flextensor: An Sciences, Beijing, China.
automatic schedule exploration and optimization framework for tensor His current research interests include compression
computation on heterogeneous system,” in Proc. Architectural Support of deep learning and high-performance computing.
Program. Lang. Operating Syst. (ASPLOS), 2020, pp. 859–873.
[44] L. Zheng et al., “Ansor: Generating high-performance tensor programs
for deep learning,” in Proc. 14th USENIX Symp. Operating Syst. Design
Implement. (OSDI), 2020, pp. 863–879.
[45] T. Chen et al., “MXNet: A flexible and efficient machine learning library
for heterogeneous distributed systems,” in Proc. Neural Inf. Process. Jiaxing Wang received the B.E. degree in School
Syst. Workshop Mach. Learn. Syst., 2015, pp. 1–6. of Control and Computer Engineering, North China
[46] M. Abadi et al., “Tensorflow: A system for large-scale machine learn- Electric Power University, Beijing, China, in 2015.
ing,” in Proc. 12th USENIX Symp. Operating Syst. Design Implement. He is currently pursuing the Ph.D. degree with
(OSDI), 2016, pp. 265–283. the Institute of Automation, Chinese Academy of
[47] K. Goto and R. A. van de Geijn, “Anatomy of high-performance matrix Sciences, Beijing.
multiplication,” ACM Trans. Math. Softw., vol. 34, no. 3, pp. 12:1–12:25, His current research interests include efficient deep
May 2008. learning and Bayesian methods.
[48] T. M. Smith, R. van de Geijn, M. Smelyanskiy, J. R. Hammond, and
F. G. V. Zee, “Anatomy of high-performance many-threaded matrix
multiplication,” in Proc. IEEE 28th Int. Parallel Distrib. Process. Symp.,
May 2014, pp. 1049–1059.
[49] F. G. Van Zee and R. A. van de Geijn, “BLIS: A framework for rapidly
instantiating BLAS functionality,” ACM Trans. Math. Softw., vol. 41, Cong Leng received the B.E. degree in automation
no. 3, pp. 1–33, Jun. 2015. from Central South University, Changsha, China,
[50] M. Dukhan. NNPACK. (2020). [Online]. Available: https://github. in 2011, and the Ph.D. degree in pattern recognition
com/Maratyszcza/NNPACK and intelligent systems from the Institute of Automa-
[51] (2020). TFLITE. [Online]. Available: https://www.tensorflow.org/lite tion, Chinese Academy of Sciences, Beijing, China,
[52] C. Szegedy et al., “Going deeper with convolutions,” in Proc. IEEE in 2016.
Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2015, pp. 1–9. His current research interests include machine
[53] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking learning, deep learning, and their applications in
the inception architecture for computer vision,” in Proc. IEEE Conf. computer vision and data mining.
Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 2818–2826.

Tianli Zhao received the B.E. degree in engineering
physics from Tsinghua University, Beijing, China,
in 2016. He is currently pursuing the Ph.D. degree
with the National Laboratory of Pattern Recogni-
tion, Institute of Automation, Chinese Academy of
Sciences, Beijing.
His research focuses on compression and accelera-
tion of deep learning and high-performance comput-
ing of deep learning algorithms on mobile platforms.
Jian Cheng (Member, IEEE) received the B.S. and
M.S. degrees in mathematics from Wuhan Univer-
sity, Wuhan, China, in 1998 and 2001, respectively,
and the Ph.D. degree in pattern recognition and
intelligent systems from the Institute of Automa-
tion, Chinese Academy of Sciences, Beijing, China,
in 2004.
He is currently a Professor with the Institute of
Automation, Chinese Academy of Sciences. His cur-
rent major research interests include deep learning,
computer vision, and chip design.

Authorized licensed use limited to: INDIAN INSTITUTE OF TECHNOLOGY DELHI. Downloaded on February 01,2024 at 19:30:01 UTC from IEEE Xplore. Restrictions apply.