MATLAB强化学习_QLearning网格迷宫问题_程序包

%I thought what I'd do was I'd pretend I was one of those deaf-mutes, or should I? clear all; %风速初始化 Windyworld.windx=zeros(7,10); Windyworld.windy=zeros(7,10); Windyworld.windy(:,4:9)=Windyworld.windy(:,4:9)+1; %%Windyworld.windy(:,7:8)=Windyworld.windy(:,7:8)+1; targetxy=[6,8]; %!!注意：第一个坐标为y坐标，第二个坐标为x坐标 alpha=0.5; gamma=0.9; lambda=0.6; Tloop=6000; %总学习循环次数 mark=zeros(1,Tloop); %记录是否成功 %迭代为二重时间循环 Q_func=zeros(7,10,4); %!!三维值函数矩阵：（z=1：+x）（z=2：-x）（z=3：+y）（z=4：-y） Q_func(:,:,1)=0.2; %size(B),ndims(B) Q_func(targetxy(1),targetxy(2),:)=0; %目标值的所有Q函数始终为0 for Ts=1:Tloop %Ts=study time %单次运动初始化 rolexy=[4,1]; E_sa=zeros(7,10,4); result(Ts).Q_func=zeros(7,10,4); result(Ts).trace=zeros(40,3); result(Ts).trace(1,:)=([1,rolexy(1),rolexy(2)]); for Tm=1:40 %按照策略获得到达下一步的动作 if Tm==1 act=tcegreedy(Ts,Q_func(rolexy(1),rolexy(2),:)); else act=actnext; end nextxy=movement(act,rolexy,Windyworld); %TD算法进行策略值迭代 %计算reward if nextxy(1)==targetxy(1)&&nextxy(2)==targetxy(2) %到达目标 reward=5; else if nextxy(1)<1||nextxy(1)>7||nextxy(2)<1||nextxy(2)>10 %超出边界 reward=-1; else reward=0; end %reward不考虑超出步数的问题 end %LambdaSarsa方式进行Q函数更新，确定下一步动作的同时更新Q函数 %在nextxy状态下选择动作并计算Qnext if nextxy(1)<1||nextxy(1)>7||nextxy(2)<1||nextxy(2)>10 %超出边界 Qnext=0; else actnext=tcegreedy(Ts,Q_func(nextxy(1),nextxy(2),:)); Qnext=Q_func(nextxy(1),nextxy(2),actnext); end %计算delta,更新E_sa delta=reward+gamma*Qnext-Q_func(rolexy(1),rolexy(2),act); E_sa(rolexy(1),rolexy(2),act)=E_sa(rolexy(1),rolexy(2),act)+1; %全局更新Q函数 Q_func=Q_func+alpha*delta*E_sa; E_sa=gamma*lambda*E_sa; %更新坐标 rolexy=nextxy; result(Ts).trace(Tm+1,:)=([Tm+1,rolexy(1),rolexy(2)]); %判断是否跳出本episode if rolexy(1)==targetxy(1)&&rolexy(2)==targetxy(2) mark(Ts)=1; break; else if rolexy(1)<1||rolexy(1)>7||rolexy(2)<1||rolexy(2)>10 break; end end end result(Ts).Q_func=Q_func; end Avegain=zeros(1,Tloop); for i=1:Tloop Avegain(i)=sum(mark(1:i))/i; end plot(1:Tloop,Avegain);