Matlab强化学习_网格迷宫问题_QLearning算法

%I thought what I'd do was I'd pretend I was one of those deaf-mutes, or should I? clear all; %风速初始化 Windyworld.windx=zeros(7,10); Windyworld.windy=zeros(7,10); Windyworld.windy(:,4:9)=Windyworld.windy(:,4:9)+1; targetxy=[6,8]; %!!注意：第一个坐标为y坐标，第二个坐标为x坐标 alpha=0.5; gamma=0.99; Tloop=6000; %总学习循环次数 mark=zeros(1,Tloop); %记录是否成功 %迭代为二重时间循环 Q_func=zeros(7,10,4); %!!三维值函数矩阵：（z=1：+x）（z=2：-x）（z=3：+y）（z=4：-y） Q_func(:,:,2)=0.2; %size(B),ndims(B) Q_func(targetxy(1),targetxy(2),:)=0; %目标值的所有Q函数始终为0 for Ts=1:Tloop %Ts=study time %单次运动初始化 rolexy=[4,1]; result(Ts).Q_func=zeros(7,10,4); result(Ts).trace=zeros(40,3); result(Ts).trace(1,:)=([1,rolexy(1),rolexy(2)]); for Tm=1:40 %按照策略获得到达下一步的动作 act=tcegreedy(Ts,Q_func(rolexy(1),rolexy(2),:)); nextxy=movement(act,rolexy,Windyworld); %TD算法进行策略值迭代 %计算reward if nextxy(1)==targetxy(1)&&nextxy(2)==targetxy(2) %到达目标 reward=5; else if nextxy(1)<1||nextxy(1)>7||nextxy(2)<1||nextxy(2)>10 %超出边界 reward=-1; else reward=0; end %reward不考虑超出步数的问题 end %计算下一步的策略函数最大值 %Qlearning方式进行Q函数更新，更新Q值中所用的s'状态下动作与实际在s'状态下做出动作不一定相同（因为e-greedy的存在） if nextxy(1)<1||nextxy(1)>7||nextxy(2)<1||nextxy(2)>10 %超出边界 Q1=0; else Q1=max(Q_func(nextxy(1),nextxy(2),:)); end Q_func(rolexy(1),rolexy(2),act)=(1-alpha)*Q_func(rolexy(1),rolexy(2),act)+alpha*(reward+gamma*Q1); %更新坐标 rolexy=nextxy; result(Ts).trace(Tm+1,:)=([Tm+1,rolexy(1),rolexy(2)]); %判断是否跳出本episode if rolexy(1)==targetxy(1)&&rolexy(2)==targetxy(2) mark(Ts)=1; break; else if rolexy(1)<1||rolexy(1)>7||rolexy(2)<1||rolexy(2)>10 break; end end end result(Ts).Q_func=Q_func; end Avegain=zeros(1,Tloop); for i=1:Tloop Avegain(i)=sum(mark(1:i))/i; end plot(1:Tloop,Avegain);