MetaPruning-MetaLearningforAutomaticNeuralNetworkChannelPruning.pdf资源-CSDN文库

Model

需积分: 37 103 浏览量 2019-10-28 22:04:53 上传评论 1 收藏 4.39MB PDF 举报

资源推荐

资源详情

资源评论

MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning

Zechun Liu

Haoyuan Mu

Xiangyu Zhang

Zichao Guo

Xin Yang

Tim Kwang-Ting Cheng

Jian Sun

Hong Kong University of Science and Technology

Tsinghua University

Megvii Technology

Huazhong University of Science and Technology

Abstract

In this paper, we propose a novel meta learning ap-

proach for automatic channel pruning of very deep neural

networks. We ﬁrst train a PruningNet, a kind of meta net-

work, which is able to generate weight parameters for any

pruned structure given the target network. We use a sim-

ple stochastic structure sampling method for training the

PruningNet. Then, we apply an evolutionary procedure to

search for good-performing pruned networks. The search is

highly efﬁcient because the weights are directly generated

by the trained PruningNet and we do not need any ﬁne-

tuning at search time. With a single PruningNet trained

for the target network, we can search for various Pruned

Networks under different constraints with little human par-

ticipation. Compared to the state-of-the-art pruning meth-

ods, we have demonstrated superior performances on Mo-

bileNet V1/V2 and ResNet. Codes are available on https:

//github.com/liuzechun/MetaPruning.

1. Introduction

Channel pruning has been recognized as an effective

neural network compression/acceleration method [32, 22, 2,

3, 21, 52] and is widely used in the industry. A typical prun-

ing approach contains three stages: training a large over-

parameterized network, pruning the less-important weights

or channels, ﬁnetuning or re-training the pruned network.

The second stage is the key. It usually performs iterative

layer-wise pruning and fast ﬁnetuning or weight reconstruc-

tion to retain the accuracy [17, 1, 33, 41].

Conventional channel pruning methods mainly rely

on data-driven sparsity constraints [28, 35], or human-

designed policies [22, 32, 40, 25, 38, 2]. Recent AutoML-

style works automatically prune channels in an iterative

mode, based on a feedback loop [52] or reinforcement

learning [21]. Compared with the conventional pruning

This work is done when Zechun Liu and Haoyuan Mu are interns at

Megvii Technology.

Figure 1. Our MetaPruning has two steps. 1) training a Prun-

ingNet. At each iteration, a network encoding vector (i.e., the

number of channels in each layer) is randomly generated. The

Pruned Network is constructed accordingly. The PruningNet takes

the network encoding vector as input and generates the weights

for the Pruned Network. 2) searching for the best Pruned Net-

work. We construct many Pruned Networks by varying network

encoding vector and evaluate their goodness on the validation data

with the weights predicted by the PruningNet. No ﬁnetuning or

re-training is needed at search time.

methods, the AutoML methods save human efforts and can

optimize the direct metrics like the hardware latency.

Apart from the idea of keeping the important weights in

the pruned network, a recent study [36] ﬁnds that the pruned

network can achieve the same accuracy no matter it inher-

its the weights in the original network or not. This ﬁnding

suggests that the essence of channel pruning is ﬁnding good

pruning structure - layer-wise channel numbers.

However, exhaustively ﬁnding the optimal pruning struc-

ture is computationally prohibitive. Considering a network

with 10 layers and each layer contains 32 channels. The

possible combination of layer-wise channel numbers could

be 32

. Inspired by the recent Neural Architecture Search

(NAS), speciﬁcally One-Shot model [5], as well as the

weight prediction mechanism in HyperNetwork [15], we

arXiv:1903.10258v3 [cs.CV] 14 Aug 2019

propose to train a PruningNet that can generate weights for

all candidate pruned networks structures, such that we can

search good-performing structures by just evaluating their

accuracy on the validation data, which is highly efﬁcient.

To train the PruningNet, we use a stochastic structure

sampling. As shown in Figure 1, the PruningNet generates

the weights for pruned networks with corresponding net-

work encoding vectors, which is the number of channels

in each layer. By stochastically feeding in different net-

work encoding vectors, the PruningNet gradually learns to

generate weights for various pruned structures. After the

training, we search for good-performing Pruned Networks

by an evolutionary search method which can ﬂexibly incor-

porate various constraints such as computation FLOPs or

hardware latency. Moreover, by directly searching the best

pruned network via determining the channels for each layer

or each stage, we can prune channels in the shortcut without

extra effort, which is seldom addressed in previous chan-

nel pruning solutions. We name the proposed method as

MetaPruning.

We apply our approach on MobileNets [24, 46] and

ResNet [19]. At the same FLOPs, our accuracy is 2.2%-

6.6% higher than MobileNet V1, 0.7%-3.7% higher than

MobileNet V2, and 0.6%-1.4% higher than ResNet-50.

At the same latency, our accuracy is 2.1%-9.0% higher

than MobileNet V1, and 1.2%-9.9% higher than MobileNet

V2. Compared with state-of-the-art channel pruning meth-

ods [21, 52], our MetaPruning also produces superior re-

sults.

Our contribution lies in four folds:

• We proposed a meta learning approach, MetaPruning, for

channel pruning. The central of this approach is learn-

ing a meta network (named PruningNet) which gener-

ates weights for various pruned structures. With a sin-

gle trained PruningNet, we can search for various pruned

networks under different constraints.

• Compared to conventional pruning methods, MetaPrun-

ing liberates human from cumbersome hyperparameter

tuning and enables the direct optimization with desired

metrics.

• Compared to other AutoML methods, MetaPruning can

easily enforce constraints in the search of desired struc-

tures, without manually tuning the reinforcement learn-

ing hyper-parameters.

• The meta learning is able to effortlessly prune the chan-

nels in the short-cuts for ResNet-like structures, which

is non-trivial because the channels in the short-cut affect

more than one layers.

2. Related Works

There are extensive studies on compressing and accel-

erating neural networks, such as quantization [54, 43, 37,

23, 56, 57], pruning [22, 30, 16] and compact network de-

sign [24, 46, 55, 39, 29]. A comprehensive survey is pro-

vided in [47]. Here, we summarize the approaches that are

most related to our work.

Pruning Network pruning is a prevalent approach for

removing redundancy in DNNs. In weight pruning, people

prune individual weights to compress the model size [30,

18, 16, 14]. However, weight pruning results in unstruc-

tured sparse ﬁlters, which can hardly be accelerated by

general-purpose hardware. Recent works [25, 32, 40, 22,

38, 53] focus on channel pruning in the CNNs, which re-

moves entire weight ﬁlters instead of individual weights.

Traditional channel pruning methods trim channels based

on the importance of each channel either in an iterative

mode [22, 38] or by adding a data-driven sparsity [28, 35].

In most traditional channel pruning, compression ratio for

each layer need to be manually set based on human ex-

perts or heuristics, which is time consuming and prone to

be trapped in sub-optimal solutions.

AutoML Recently, AutoML methods [21, 52, 8, 12] take

the real-time inference latency on multiple devices into ac-

count to iteratively prune channels in different layers of a

network via reinforcement learning [21] or an automatic

feedback loop [52]. Compared with traditional channel

pruning methods, AutoML methods help to alleviate the

manual efforts for tuning the hyper-parameters in channel

pruning. Our proposed MetaPruning also involves little hu-

man participation. Different from previous AutoML prun-

ing methods, which is carried out in a layer-wise prun-

ing and ﬁnetuning loop, our methods is motivated by re-

cent ﬁndings [36], which suggests that instead of selecting

“important” weights, the essence of channel pruning some-

times lies in identifying the best pruned network. From

this prospective, we propose MetaPruning for directly ﬁnd-

ing the optimal pruned network structures. Compared to

previous AutoML pruning methods [21, 52], MetaPruning

method enjoys higher ﬂexibility in precisely meeting the

constraints and possesses the ability of pruning the channel

in the short-cut.

Meta Learning Meta-learning refers to learning from

observing how different machine learning approaches per-

form on various learning tasks. Meta learning can be used

in few/zero-shot learning [44, 13] and transfer learning [48].

A comprehensive overview of meta learning is provided

in [31]. In this work we are inspired by [15] to use meta

learning for weight prediction. Weight predictions refer to

weights of a neural network are predicted by another neural

network rather than directly learned [15]. Recent works also

applies meta learning on various tasks and achieves state-of-

the-art results in detection [51], super-resolution with arbi-

trary magniﬁcation [27] and instance segmentation [26].

Neural Architecture Search Studies for neural archi-

tecture search try to ﬁnd the optimal network structures

剩余9页未读，继续阅读

评论收藏

内容反馈

librahfacebook

粉丝: 198
资源: 2

MetaPruning-Meta Learning for Automatic Neural Network Channel P...

最新资源

MetaPruning-Meta Learning for Automatic Neural Network Channel P...

模型压缩之通道剪枝channel-pruning-master.zip

pytorch-pruning.zip

Python-ChannelPruningforAcceleratingVeryDeepNeuralNetworks

Python-MetaSR一个用于超分辨率的放大任意网络CVPR2019PyTorch

Meta-分析导论（我就是跟着它自学的饿）

win超分辩重建Meta-SR

An Overview of Multi-Task Learning in Deep Neural Networks.pdf

-Convolution Neural Network for Eart.pdf

前馈神经网络（Feedforward neural network）.pdf

CNN神经网络23-mgic-multigrid-in-channels-neural-network-architectures.pdf

ABCPruner:Pytorch实施的论文被IJCAI 2020接受-通过自动结构搜索进行通道修剪

An FPGA-Based Resource-Saving Hardware Accelerator for Deep Neural Network.pdf

Deep-Learning-with-TensorFlow-Explore-neural-networks-with-Python.pdf.pdf

07Neural Network with Memory.pdf

Neural Network Toolbox Getting Started Guide

Top- N Recommendation with A Neural Co-Attention Model.pdf

YouTube推荐系统Paper[2016]-Deep Neural Networks for YouTube Recommendations.pdf

藏经阁-That Learns From a Neural Network Huge Graph-38.pdf

Deep Learning: Practical Neural Networks with Java 完整高清英文azw3版

Research on transformer protection based on wavelet neural network and FPGA.pdf

Matlab Neural Network Toolbox documentation.pdf

神经网络剪枝的研究进展状态（Neural Network Pruning）.pdf

论文研究-Elman-style Process Neural Network and Its Learning Algorithm.pdf

ISSCC2021_Session_15V_Compute-in-Memory Processors for Deep Neural Networks.pdf

YOLOv8-deepsort 实现智能车辆目标检测+车辆跟踪+车辆计数

Transformer模型实现长期预测并可视化结果（附代码+数据集+原理介绍）

YOLOv8网络结构图，自制visio文件，yolov8.vsds，需要的自取，在原有的基础上直接改就行了

最新资源