TrainingIssues&CNNDevelopment资源-CSDN文库

需积分: 7 35 浏览量 2022-06-14 14:16:34 上传评论收藏 338KB PDF 举报

资源推荐

资源详情

资源评论

TrainingIssues&CNNDev elopment

GradientExploding&Vanishing

avoidgradientexploding

weightinitialization,suchthat| wi|≤1ingeneral

weightre-normalizationduringtraining

rescalingxto|x|≤1

reducegradientvanishing

Weightinitialization:Xavier’smethod

Varianceofsignalacrosslayerdoesnotchange!

Varianceofbackwardgradientsignalacrosslayerdoesnotchange!

activation---noReLU

activation---Re LU

n上一层神经元的个数

m反向传播上一层神经元个数

Mini-BatchIssue

Is sueofmini-batch

Differentmini-batchdataoftenhavedifferentdistributions

Causeddifferentmini-batchinputdistributionsforeverylayer!

Distributionofon eminibatch changesovertimeforalayer!

Eachlayerneedstocontinuouslyadapttonewdistributions

Batchnor malization

reducesvarietiesofneurons’inputs/o utputs,i.e.,reducinglayer’srepresentationpower.

Solution:considerγkandβkaspartofmodelparameters

ResNetandItsextensions

Prob lemofdeeper networks

thelatterlayerchangesconsequently

morelaye rscausedlarge rtrainingandtesterror!

Deepernetworknotoverfitting,buthardertooptimize

so l ution:usenetworklayertolearnresidualmappingra therthan

directlytolearnadesiredunderlyingmapping!

IfH(x)isidentitymapping,itiseasiertopushresidualtozerothantofitanidentitymappingbyastackofnonlinearlayers

WhyRes Netswor kbetter? Train/updateea chlayereasier,byskipconnections

ResNetextens ions

DenseNets:den selyskipconnectionswithineachblock;Fewerkernel s(parameters)ineachlayer,why?

WideResNets:morekernelsineachlaye r,withfewerlayers

ResNeXt:divideResNetblockintosmallertransformations,thenaggregate.

SENet:Squeeze-and-ExcitationNetworks

PNASNet:progressiveneuralarchitecturesearch

本内容试读结束，登录后可阅读更多

下载后可阅读完整内容，剩余0页未读，立即下载

内容反馈

溢le

粉丝: 0
资源: 2

最新资源

资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！点击此处反馈

feedback-tip