1.deep+learning课本-三巨头(中)1
需积分: 0 119 浏览量
2022-08-03
15:48:36
上传
评论
收藏 18.84MB PDF 举报
Chapter 8
Optimization for Training Deep
Models
Deep learning algorithms involve optimization in many contexts. For example,
performing inference in models such as PCA involves solving an optimization
problem. We often use analytical optimization to write proofs or design algorithms.
Of all of the many optimization problems involved in deep learning, the most
difficult is neural network training. It is quite common to invest days to months of
time on hundreds of machines in order to solve even a single instance of the neural
network training problem. Because this problem is so important and so expensive,
a specialized set of optimization techniques have been developed for solving it.
This chapter presents these optimization techniques for neural network training.
If you are unfamiliar with the basic principles of gradient-based optimization,
we suggest reviewing Chapter . That chapter includes a brief overview of numerical4
optimization in general.
This chapter focuses on one particular case of optimization: finding the param-
eters
θ
of a neural network that significantly reduce a cost function
J
(
θ
), which
typically includes a performance measure evaluated on the entire training set as
well as additional regularization terms.
We begin with a description of how optimization used as a training algorithm
for a machine learning task differs from pure optimization. Next, we present several
of the concrete challenges that make optimization of neural networks difficult. We
then define several practical algorithms, including both optimization algorithms
themselves and strategies for initializing the parameters. More advanced algorithms
adapt their learning rates during training or leverage information contained in
274
剩余230页未读，继续阅读
评论0