MasteringtheGameofGowithoutHumanKnowledge（机器学习英文著作）.pdf资源-CSDN文库

需积分: 5 173 浏览量 2022-12-02 14:12:59 上传评论收藏 2.43MB PDF 举报

资源详情

资源评论

Mastering the Game of Go without Human Knowledge

David Silver*, Julian Schrittwieser*, Karen Simonyan*, Ioannis Antonoglou, Aja Huang, Arthur

Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy

Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, Demis Hassabis.

DeepMind, 5 New Street Square, London EC4A 3TW.

*These authors contributed equally to this work.

A long-standing goal of artiﬁcial intelligence is an algorithm that learns, tabula rasa, su-

perhuman proﬁciency in challenging domains. Recently, AlphaGo became the ﬁrst program

to defeat a world champion in the game of Go. The tree search in AlphaGo evaluated posi-

tions and selected moves using deep neural networks. These neural networks were trained

by supervised learning from human expert moves, and by reinforcement learning from self-

play. Here, we introduce an algorithm based solely on reinforcement learning, without hu-

man data, guidance, or domain knowledge beyond game rules. AlphaGo becomes its own

teacher: a neural network is trained to predict AlphaGo’s own move selections and also the

winner of AlphaGo’s games. This neural network improves the strength of tree search, re-

sulting in higher quality move selection and stronger self-play in the next iteration. Starting

tabula rasa, our new program AlphaGo Zero achieved superhuman performance, winning

100-0 against the previously published, champion-defeating AlphaGo.

Much progress towards artiﬁcial intelligence has been made using supervised learning sys-

tems that are trained to replicate the decisions of human experts

1–4

. However, expert data is often

expensive, unreliable, or simply unavailable. Even when reliable data is available it may impose a

ceiling on the performance of systems trained in this manner

. In contrast, reinforcement learn-

ing systems are trained from their own experience, in principle allowing them to exceed human

capabilities, and to operate in domains where human expertise is lacking. Recently, there has been

rapid progress towards this goal, using deep neural networks trained by reinforcement learning.

These systems have outperformed humans in computer games such as Atari

6, 7

and 3D virtual en-

vironments

8–10

. However, the most challenging domains in terms of human intellect – such as the

game of Go, widely viewed as a grand challenge for artiﬁcial intelligence

– require precise and

sophisticated lookahead in vast search spaces. Fully general methods have not previously achieved

human-level performance in these domains.

AlphaGo was the ﬁrst program to achieve superhuman performance in Go. The published

version

, which we refer to as AlphaGo Fan, defeated the European champion Fan Hui in October

2015. AlphaGo Fan utilised two deep neural networks: a policy network that outputs move prob-

abilities, and a value network that outputs a position evaluation. The policy network was trained

initially by supervised learning to accurately predict human expert moves, and was subsequently

reﬁned by policy-gradient reinforcement learning. The value network was trained to predict the

winner of games played by the policy network against itself. Once trained, these networks were

combined with a Monte-Carlo Tree Search (MCTS)

13–15

to provide a lookahead search, using the

policy network to narrow down the search to high-probability moves, and using the value net-

work (in conjunction with Monte-Carlo rollouts using a fast rollout policy) to evaluate positions in

the tree. A subsequent version, which we refer to as AlphaGo Lee, used a similar approach (see

Methods), and defeated Lee Sedol, the winner of 18 international titles, in March 2016.

Our program, AlphaGo Zero, differs from AlphaGo Fan and AlphaGo Lee

in several im-

portant aspects. First and foremost, it is trained solely by self-play reinforcement learning, starting

from random play, without any supervision or use of human data. Second, it only uses the black

and white stones from the board as input features. Third, it uses a single neural network, rather

than separate policy and value networks. Finally, it uses a simpler tree search that relies upon this

single neural network to evaluate positions and sample moves, without performing any Monte-

Carlo rollouts. To achieve these results, we introduce a new reinforcement learning algorithm that

incorporates lookahead search inside the training loop, resulting in rapid improvement and precise

and stable learning. Further technical differences in the search algorithm, training procedure and

network architecture are described in Methods.

1 Reinforcement Learning in AlphaGo Zero

Our new method uses a deep neural network f

with parameters θ. This neural network takes as an

input the raw board representation s of the position and its history, and outputs both move probabil-

ities and a value, (p, v) = f

(s). The vector of move probabilities p represents the probability of

selecting each move (including pass), p

= P r(a|s). The value v is a scalar evaluation, estimating

the probability of the current player winning from position s. This neural network combines the

roles of both policy network and value network

into a single architecture. The neural network

consists of many residual blocks

of convolutional layers

16, 17

with batch normalisation

and

rectiﬁer non-linearities

(see Methods).

The neural network in AlphaGo Zero is trained from games of self-play by a novel reinforce-

ment learning algorithm. In each position s, an MCTS search is executed, guided by the neural

network f

. The MCTS search outputs probabilities π

π of playing each move. These search proba-

bilities usually select much stronger moves than the raw move probabilities p of the neural network

(s); MCTS may therefore be viewed as a powerful policy improvement operator

20, 21

. Self-play

with search – using the improved MCTS-based policy to select each move, then using the game

winner z as a sample of the value – may be viewed as a powerful policy evaluation operator. The

main idea of our reinforcement learning algorithm is to use these search operators repeatedly in

a policy iteration procedure

22, 23

: the neural network’s parameters are updated to make the move

probabilities and value (p, v) = f

(s) more closely match the improved search probabilities and

self-play winner (π

π, z); these new parameters are used in the next iteration of self-play to make the

search even stronger. Figure 1 illustrates the self-play training pipeline.

The Monte-Carlo tree search uses the neural network f

to guide its simulations (see Figure

2). Each edge (s, a) in the search tree stores a prior probability P (s, a), a visit count N(s, a),

and an action-value Q(s, a). Each simulation starts from the root state and iteratively selects

moves that maximise an upper conﬁdence bound Q(s, a) + U(s, a), where U(s, a) ∝ P (s, a)/(1+

N(s, a))

12, 24

, until a leaf node s

is encountered. This leaf position is expanded and evaluated just

剩余41页未读，继续阅读

评论收藏

内容反馈

Mastering the Game of Go without Human Knowledge（机器学习英文著作）.pdf

评论0

最新资源

Mastering the Game of Go without Human Knowledge（机器学习英文著作）.pdf

评论0

最新资源

相关推荐

Mastering the game of Go without human knowledge原文

Mastering the Game of Go without Human Knowledge

Mastering the game of Go without human knowledge翻译1

mastering the game of Go

Mastering the game of Go with deep neural networks and tree search 中英文

Mastering the game of Go with deep neural networks and tree search

mastering the game of go with deep neural networks and tree search.pdf

Mastering the FreeRTOS™ Real Time Kernel.pdf

python机器学习教程-从零开始掌握Python机器学习：十四步教程.pdf

Mastering matplotlib.pdf.pdf

Mastering_IPython_4.0.pdf.pdf

Mastering the DMA and IOMMU APIs.pdf

Mastering-Python.pdf.pdf

Mastering Unity 2017 Game Development with C# - Second Edition.pdf

MASTERING_DOCKER.pdf

mastering cmake.pdf

Mastering the C++17 .pdf

Mastering.Git.2016.4.pdf

相关实用应用程序（Windows可用）

免费可用的ChatGPT网页版.zip

ChatGPT使用总结：150个ChatGPT提示词模板（完整版）

chromedriver-win64.zip

全国计算机二级WPSoffice精选350道选择题题库（含答案）.pdf

HAI-2024斯坦福AI指数报告（中文译版）.pdf

哈尔滨工业大学-ChatGPT调研报告-2023.3.6-94页.pdf

2023泛娱乐社交出海手册-ZEGO即构科技

4个亲测好用的ChatGPT4渠道

民宿网站