两层神经网络全连接层

所需积分/C币:39 2018-07-24 20:47:59 1.17MB PDF
收藏 收藏
举报

cs231n作业,我们已经完成了一个两层的全连接神经网络的设计,但是有些简单, 并且还没有模块化,因为那里的损失函数和梯度我们是用一个函数来计算出来的。因 此,我们希望可以设计更复杂的网络,以便于我们可以完成不同类型层的设计,然后 将它们集成到不同结构的模型中
16 print('difference: ' ,rel_error(out, correct_out)) 输出结果如下 1 testing affine forward function 2 difference:9.76984772881e-10 误差竟然达到e-10,还是挺不错的。接下来我们就继续计算反向传播。 1.12反向传播 反向传播接收来自上一层输岀的梯度值和εache值,返回输入和权重的梯度,形式如 1 def layer_backward(dout, cache x,w,z,out=cache 2345 dx= return dx dw 实现代码如下 1 def affine_backward( dout, cache 2 dout:(N,M);x:(N,d1,…dk)xrsp中我们会将x转化为(N,D),其中 D=d 1**d k :(D, M); dx: (D d k); dw:(D, M); db: (M,) x.w. b=cache dx, dw. db=none. none. none N=x shape[oI 6 x rsp=. reshape(N,-1) dx=dout dot(wT) dx=dx reshape(kx. shape) 9 dw=x_ rsp.T dot(dout) 10 db=np sum(dout, axis=0) 11 return dx. dw. db 注:如果不清楚反向传播过程中如何利用利用链式法则进行计算,可以参阅我们上 次的作业 同样,下面我们来检验下 1 X=np. random randn(10, 2, 3) W=np. random randn(6, 5) 3 b=np. random randn(5) 4 dout=np. random randn(10, 5) 6 dx_numeval_numerical_gradientarray(lambda x: affine_forward(x, W, b) [O,x, dout) 7 dw_num=eval_numerical_gradient_array (lambda w: affine_forward(x, w, b) o],W, dout) 8 db_num=eval_numerical_gradient_array(lambda b: affine_ forward (x,w, b) LOl,b, dout) 9 0_, cache=affine_forward (x, W, b) 11 dx, dw, db=affine_backward dout, cache) 12 13 print(test affine_backward function 14 print(dx error: ',rel_error(dx_num, dx) 15 print(dw error: , rel_error(dw_num, dw)) 16 print(db error: ', rel_error(db_num, db ) 输出结果如下 1 test affine backward function 2 dx error:2.5214819994e-19 3 dw error:9.11372672216e-11 4 db error:3.57189937616e-11 bingo,误差还是很小的。这样我们就完成了一层映射层的前向和反向传播两个过程。 12Relu层 12.1前向传播 代码如下 1 def relu_forward(x) out=none 2345 out=x(>=0) cache=x return out cache 检验代码如下 1 X=np linspace (-0.5, 0.5, num=12). reshape(3, 4) 2 out, -relu forward(x) 3 correct_out np array([[ o 9 4 9 9.94545455 0.13636364,], [8.22727273,9,31818182,9.40909891, 6 print(testing relu_ forward function: ' 7 print('difference: ' , rel_error(out, correct_out)) 输出结果如下: 1 testing relu_forward function: 2 difference:4.9999979892e-98 122反向传播 代码如下 1 def relu_backward(dout, cache): 2 dx. x=none cache dx=(x>=0)*dout return dx 检验代码如下 1 X=np. random randn (10, 10) 2 dout=np. random randn (*X shape 3 4 dx_num=eval_numerical_gradient_array (lambda x:relu_forward(x)[o], X, dout) 5_,cache=relu_forward(x) 6 dx=relu_backward ( dout, cache) 8 print('testing relu_backward functin 9 print('dx error:,relerror(dx_num, dx)) 输出结果如下: 1 testing relu_backward functin 2 dx error:3.27561649471e-12 到现在,我们差不多已经完成了基础层的构建。但是,实践中,一般映射层后面会接 Reu层,我们这里把两层合为一层来看看: 1 def affine_relu_forward (x,W, b): a, fc cache=affine forward(x, w, b) out, relu cache=relu_ forward (a) 456 cache=(fc cache, relu cache) return out, cache 7 def affine_relubackward(dout, cache) 8 fc cache, relu cache=cache da=relu backward (dout, relu cache 10 dx, dw, db=affine backward (da, fc cache) 11 return dx, dw, db 梯度检验代码如下: 1 X=np. random randn( 2, 3, 4) 2 W=np. random randn (12, 10) 3 b=np. random randn (10) 4 dout=np. random randn(2, 10 5 6 out, cache=affine_relu_forward (x, w, b) 7 dx, dw, db=affine_relu_ backward ( dout, cache) 9 dx_num=eval_numerical_gradient_array (lambda x: affine_relu_forward(x, w, b) [o],x, dout) 10 dw_num=eval_numerical_gradient_array(lambda w: affine_relu_forward (x,w, b) [Ol,W, dout 11 db_num=eval_numerical_gradient_array (lambda b: affine_relu_forward(x, w, b) o,b, dout 12 13 print(testing affine_relu_forward: ' 14 print('dx_error: rel_error(dx_num, dx)) 15 print(dw error: ',relerror(dw_num, dw )) 16 print('db error:,, rel_error(db_num, db)) 运行结果如下 1 testing affine relu forward 2 dx error:1.58523685885e-19 3 dw error:3.68669862446e-18 4 db error:1.04868325611e-18 13损失函数层 在上一个作业中,我们已经实现了sm和 softmax线性分类器,现在我们来实现了sm 和 softmax层。 代码如下 1 def svm_ loss(x,y): ': param x:(N,C),x[i,j]表示第i个输入是第j个的分数 param y:(N,), x[i]的标签: return:1oss,dx 3 N=x shape[o] 4 correct class scores=x[np arange(n),y] margins=np. maximum(0, X-correct class scores[:,np. newaxis]+1.0) 6 margins[np arange(n,y]=0 loss=np sum (margins)/N num_pos=np sum(margins >0, axis=1) 10 dx=np zeros like(x) 11 dx[ margins>0]=1 12 dx[np arange(n), y]-num_pos 13 dx/=N 14 return loss. dx 15 16 def softmax_loss(x, y) probs=np. exp(x-np. max(x, axis=1, keepdims=True)) 18 probs/=np sum(probs, axis=l, keepdims=True) 19 N=x shape[e] 20 loss=-np sum(np. log(probs [np arange(N),y1))/N 21 dx=probs. copy 22 dx[np arange(n),y]-=1 23 dx/=N 24 eturn loss, dx 我们检验下我们做的是不是正确,代码如下 1 num classes, num inputs=10,50 2 X =0.001*np. random randn (num_inputs, num_classes) 3 y=np. random randint(num_ classes, size=num_inputs) 5 dx_num=eval_numerical_gradient (lambda x: svm_loss(x, y)[01,x, verbose=False) loss, dx=svm loss(x, y) 7 print(testing svm_loss: ' 8 print("loss: ',loss) 9 print(dx error: ,,rel_error(dx_num, dx)) 18 11 dx_num=eval_numerical_gradient(lambda x: softmax_loss(x, y) 0],, verbose=False) 12 loss, dx=softmax_ loss(x, y) 13 14 print('\testing softmax_loss: ' 15 print('loss: ',loss) 16 print('dx error:', rel_error(dx_num, dx)) 运行结果如下 testing svm loss: 21o55:9.00834578116 3 dx error:1.40215660067e-9 5 testing softmax_ loss 610ss:2.38262818984 7 dx error:8.19589239336e-89 OK,现在我们已经完成了基础层的构建,下面我们就可以通过这些层的连接来组成我 们想要的结构。 2两层神经网络 下面我们就使用我们原来构建的基础层来搭建一个两层的神经网络。 我们定义了一个τ WOLayerNet代的类来实现一个两层神经网络的搭建,代码如下: 1 class TwoLayer Net(object) def init(self, input dim=3*32*32, hidden dim=100, num classes=10, weight scal e=1e-3,reg=8.8): self params= 345 self. reg=reg self params[W1]=weight scale*np. random. rand (input dim, hidden dim) self params['bl]=np zeros(hidden dim) self params[ W2]=weight scale*np. random randn(hidden dim, num classes) self params[b2 ]=np zeros(num classes) 18 def loss(self, x, y=none 11 scores=None 12 ar1_out, ar1 cache=affine relu forward(x, self params[ W1l, self params[ 'b 1]) 13 a2 out, a2 cache=affine forward(ar1 out, self params[ W2],self params[b2 ]) 14 scores=a2 out 15 16 if y is None: 17 return scores 18 19 loss, grads=0, 0 20 21 loss, dscores=softmax loss(scores,y) 22 loss=loss+0. 5*self. reg*np sum(self params[ W1 ] self params[ 'W1])+0.5*s elf. reg*np sum(self params[ W2]*self params[ ']) 23 dx2, dw 2, db2=affine backward(scores, a2 cache) 24 grads[ W2]=dw2+self. reg*self params[W2I 25 grads[b2]=db2 26 27 dxl, dwl, dbl=affine relu forward (dx2, ar1 cache) 28 grads['W1']=dwl+self. reg*self params['W1' 29 grads[ b1]=db1 30 31 return loss, grads 基本就同前面的内容,这里我们就不对代码进行解释了 同样,我们来检验下我们搭建的网络。代码如下 1N,D,H,C=3 59 2 X= np. random randn(N, D) 3 y=np. random randint(C, size=N) 5 std= 1e-2 6 model=TwoLayerNetinput_dim=D, hidden_dim=H, num_classes=C, Weight scale=std) 8 print ( ' Testing initialization .' 9 W1_std abs(model params[W1]. std ()-std) 10 b1 model params['b1'I 11 W2_std abs(model params[W2'I std()-std 12 b2= model params['b2' 13 assert W1_std std/ 10,First layer weights do not seem right 14 assert np. all(b1 ==0),First layer biases do not seem right 15 assert W2_std std/ 10, 'Second layer weights do not seem right 16 assert np. all(b2 = 0),'Second layer biases do not seem right 17 18 print ('Testing test-time forward pass 19 model params['W1']=np linspace (-07, 0.3, num=D*H).reshape(D, H) 20 model params["b1']=nplinspace(-01,0.9, num=H) 21 model params[ W2']= nplinspace(-03, 0.4, num=H*C).reshape(H, c) 22 model params[b2']=nplinspace(-09,0.1, num=c) 23X=nplinspace(-5.5,4.5,num=N*D).reshape(D,N).T 24 scores model loss(X) 25 correct_scores =np. asarray( 26[[11.53165108,12.2917344,13.0518171,13.81190102,14.57198434, 15.33206765,16.89215996] 27 [12.05769898,12.74614185,13.43459113,14.1238412,14.81149128, 15.49994135,16.18839143], 28[12.5837387,13.295471713.81736455,14.43418138,15.05922 15.66781586,16.2846319]) 29 scores_diff =np. abs(scores- correct_scores). sum() 30 assert scores_diff le-6,'Problem with test-time forward pass 31 32 print ('Testing training loss(no regularization)') 3 y=np. asarray(e,5, 11) 341055,gads=moel.1055(X,y) 35 correct1oss=3.4702243556 36 assert abs(loss - correct_loss)< le-10,'Problem with training-time loss 37 38 model. reg =1.0 39 losS, grads modelloss(X, y) 40 correct1oss=26.5948426952 41 assert abs(loss correct_ loss)< 1e-10,'Problem with regularization Oss 42 43 for reg in [0.0,0.7] 44 print ('Running numeric gradient check with reg =' reg) 45 model. reg = reg 46 loss, grads= model loss(X, y) 输出结果如下

...展开详情
试读 35P 两层神经网络全连接层
立即下载 低至0.43元/次 身份认证VIP会员低至7折
    抢沙发
    一个资源只可评论一次,评论内容不能少于5个字
    关注 私信 TA的资源
    上传资源赚积分,得勋章
    最新推荐
    两层神经网络全连接层 39积分/C币 立即下载
    1/35
    两层神经网络全连接层第1页
    两层神经网络全连接层第2页
    两层神经网络全连接层第3页
    两层神经网络全连接层第4页
    两层神经网络全连接层第5页
    两层神经网络全连接层第6页
    两层神经网络全连接层第7页
    两层神经网络全连接层第8页
    两层神经网络全连接层第9页
    两层神经网络全连接层第10页
    两层神经网络全连接层第11页

    试读已结束,剩余24页未读...

    39积分/C币 立即下载 >