深度学习CNN模型在CPU上的优化论文指导
需积分: 12 172 浏览量
2022-10-13
18:49:11
上传
评论
收藏 1.5MB PDF 举报
Optimizing CNN Model Inference on CPUs
Yizhi Liu
∗
, Yao Wang
∗
, Ruofei Yu, Mu Li, Vin Sharma, Yida Wang
Amazon Web Services
{yizhiliu, wayao, yuruofei, mli, vinarm, wangyida}@amazon.com
Abstract
The popularity of Convolutional Neural Network (CNN) mod-
els and the ubiquity of CPUs imply that better performance of
CNN model inference on CPUs can deliver significant gain
to a large number of users. To improve the performance of
CNN inference on CPUs, current approaches that treat the
model as a graph mostly rely on the use of high-performance
libraries such as Intel MKL-DNN and some basic graph-level
optimizations, which is restrictive and misses the opportu-
nity to optimize the end-to-end inference pipeline as a whole.
This paper presents a more comprehensive approach of CNN
model inference on CPUs that employs a full-stack and sys-
tematic scheme of optimizations. The proposed solution op-
timizes the operations as templates, which enables further
improvement of the performance via operation- and graph-
level joint optimization. Experiments show that the proposed
solution achieves up to 3.45
×
lower latency for CNN model
inference than the current state-of-the-art implementations on
various kinds of popular CPUs.
1 Introduction
The growing use of Convolutional Neural Network (CNN)
models in computer vision applications makes this model
architecture a natural focus for performance optimization
efforts. Similarly, the widespread deployment of CPUs in
servers, clients, and edge devices makes this hardware plat-
form an attractive target. Therefore, performing CNN model
inference efficiently on CPUs is of critical interest to many
users.
The performance of CNN model inference on CPUs leaves
significant room for improvement. Performing a CNN model
inference is essentially executing a computation graph con-
sisting of operations. In practice, people normally use high-
performance kernel libraries (e.g. Intel MKL-DNN [25] and
OpenBlas [47]) to obtain decent performance for CNN opera-
tions. While these libraries tune very carefully for common
∗
Equal contribution
operations with normal input data shapes (e.g. 2D convolu-
tions), they only focus on the (mostly, convolution) operations
but miss the opportunities to further optimize the end-to-end
model inference from the graph level. The graph-level opti-
mization is often handled by the deep learning frameworks,
e.g. TensorFlow [5] and MXNet [8].
However, the graph-level optimization such as operation fu-
sion and data layout planing that a framework can do is limited
because the operation implementation is already predefined
in the third-party libraries. Therefore, the optimizations in the
frameworks do not work in concert with the optimizations in
the kernel library, which leaves significant performance gains
unrealized in practice. Furthermore, different CPU architec-
tures rely on different high-performance libraries and integrat-
ing a library into a deep learning framework requires error-
prone and time-consuming engineering effort. For instance,
integrating MKL-DNN into MXNet involved 9732 lines of
additions and 8234 lines of deletions of the code over a period
of 6 months
1
, without considering a number of following
patches. Lastly, although those libraries are highly optimized,
they present as third-party plug-ins, which may introduce con-
tention issues with other libraries in the framework. As an
example, TensorFlow originally used the Eigen library [4]
to handle computation on CPUs. Later on, MKL-DNN was
also introduced. As a consequence, at runtime MKL-DNN
threads compete with Eigen threads, resulting in performance
loss. In summary, this kind of framework-specific approach
for CNN model inference on CPUs is inflexible, cumbersome,
and sub-optimal.
Because of the constraint imposed by the framework, opti-
mizing the performance of CNN model inference end-to-end
without involving a framework (i.e. a framework-agnostic
method) is of obvious interest to many deep learning prac-
titioners. Recently, Intel launched a universal CNN model
inference engine called OpenVINO Toolkit [16]. This toolkit
optimizes CNN models in the computer vision domain on
Intel processors (mostly x86 CPUs) and claims to achieve
better performance than the deep learning frameworks alone.
1
https://github.com/apache/incubator- mxnet/pull/8302
1
arXiv:1809.02697v2 [cs.DC] 11 Jan 2019
剩余15页未读，继续阅读
资源评论
