DouZero斗.地主AI

# [ICML 2021] DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning <img width="500" src="https://raw.githubusercontent.com/kwai/DouZero/main/imgs/douzero_logo.jpg" alt="Logo" /> [](https://github.com/kwai/DouZero/actions/workflows/python-package.yml) [](https://badge.fury.io/py/douzero) [](https://pepy.tech/project/douzero) [](https://pepy.tech/project/douzero) [](https://opensource.org/licenses/Apache-2.0) [](https://colab.research.google.com/github/daochenzha/douzero-colab/blob/main/douzero-colab.ipynb) [中文文档](README.zh-CN.md) DouZero is a reinforcement learning framework for [DouDizhu](https://en.wikipedia.org/wiki/Dou_dizhu) ([斗地主](https://baike.baidu.com/item/%E6%96%97%E5%9C%B0%E4%B8%BB/177997)), the most popular card game in China. It is a shedding-type game where the player’s objective is to empty one’s hand of all cards before other players. DouDizhu is a very challenging domain with competition, collaboration, imperfect information, large state space, and particularly a massive set of possible actions where the legal actions vary significantly from turn to turn. DouZero is developed by AI Platform, Kwai Inc. (快手). * Online Demo: [https://www.douzero.org/](https://www.douzero.org/) * :loudspeaker: New Version with Bid（叫牌版本）: [https://www.douzero.org/bid](https://www.douzero.org/bid) * Run the Demo Locally: [https://github.com/datamllab/rlcard-showdown](https://github.com/datamllab/rlcard-showdown) * Video: [YouTube](https://youtu.be/inHIi8sej7Y) * Paper: [https://arxiv.org/abs/2106.06135](https://arxiv.org/abs/2106.06135) * Related Project: [RLCard Project](https://github.com/datamllab/rlcard) * Related Resources: [Awesome-Game-AI](https://github.com/datamllab/awesome-game-ai) * Google Colab: [jupyter notebook](https://github.com/daochenzha/douzero-colab/blob/main/douzero-colab.ipynb) * Unofficial improved versions of DouZero by the community: [[DouZero ResNet]](https://github.com/Vincentzyx/Douzero_Resnet) [[DouZero FullAuto]](https://github.com/Vincentzyx/DouZero_For_HLDDZ_FullAuto) * Zhihu: [https://zhuanlan.zhihu.com/p/526723604](https://zhuanlan.zhihu.com/p/526723604) * Miscellaneous Resources: Have you heard of data-centric AI? Please check out our [data-centric AI survey](https://arxiv.org/abs/2303.10158) and [awesome data-centric AI resources](https://github.com/daochenzha/data-centric-AI)! **Community:** * **Slack**: Discuss in [DouZero](https://join.slack.com/t/douzero/shared_invite/zt-rg3rygcw-ouxxDk5o4O0bPZ23vpdwxA) channel. * **QQ Group**: Join our QQ group to discuss. Password: douzeroqqgroup * Group 1: 819204202 * Group 2: 954183174 * Group 3: 834954839 * Group 4: 211434658 * Group 5: 189203636 **News:** * Thanks for the contribution of [@Vincentzyx](https://github.com/Vincentzyx) for enabling CPU training. Now Windows users can train with CPUs. <img width="500" src="https://douzero.org/public/demo.gif" alt="Demo" /> ## Cite this Work If you find this project helpful in your research, please cite our paper: Zha, Daochen et al. “DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning.” ICML (2021). ```bibtex @InProceedings{pmlr-v139-zha21a, title = {DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning}, author = {Zha, Daochen and Xie, Jingru and Ma, Wenye and Zhang, Sheng and Lian, Xiangru and Hu, Xia and Liu, Ji}, booktitle = {Proceedings of the 38th International Conference on Machine Learning}, pages = {12333--12344}, year = {2021}, editor = {Meila, Marina and Zhang, Tong}, volume = {139}, series = {Proceedings of Machine Learning Research}, month = {18--24 Jul}, publisher = {PMLR}, pdf = {http://proceedings.mlr.press/v139/zha21a/zha21a.pdf}, url = {http://proceedings.mlr.press/v139/zha21a.html}, abstract = {Games are abstractions of the real world, where artificial agents learn to compete and cooperate with other agents. While significant achievements have been made in various perfect- and imperfect-information games, DouDizhu (a.k.a. Fighting the Landlord), a three-player card game, is still unsolved. DouDizhu is a very challenging domain with competition, collaboration, imperfect information, large state space, and particularly a massive set of possible actions where the legal actions vary significantly from turn to turn. Unfortunately, modern reinforcement learning algorithms mainly focus on simple and small action spaces, and not surprisingly, are shown not to make satisfactory progress in DouDizhu. In this work, we propose a conceptually simple yet effective DouDizhu AI system, namely DouZero, which enhances traditional Monte-Carlo methods with deep neural networks, action encoding, and parallel actors. Starting from scratch in a single server with four GPUs, DouZero outperformed all the existing DouDizhu AI programs in days of training and was ranked the first in the Botzone leaderboard among 344 AI agents. Through building DouZero, we show that classic Monte-Carlo methods can be made to deliver strong results in a hard domain with a complex action space. The code and an online demo are released at https://github.com/kwai/DouZero with the hope that this insight could motivate future work.} } ``` ## What Makes DouDizhu Challenging? In addition to the challenge of imperfect information, DouDizhu has huge state and action spaces. In particular, the action space of DouDizhu is 10^4 (see [this table](https://github.com/datamllab/rlcard#available-environments)). Unfortunately, most reinforcement learning algorithms can only handle very small action spaces. Moreover, the players in DouDizhu need to both compete and cooperate with others in a partially-observable environment with limited communication, i.e., two Peasants players will play as a team to fight against the Landlord player. Modeling both competing and cooperation is an open research challenge. In this work, we propose Deep Monte Carlo (DMC) algorithm with action encoding and parallel actors. This leads to a very simple yet surprisingly effective solution for DouDizhu. Please read [our paper](https://arxiv.org/abs/2106.06135) for more details. ## Installation The training code is designed for GPUs. Thus, you need to first install CUDA if you want to train models. You may refer to [this guide](https://docs.nvidia.com/cuda/index.html#installation-guides). For evaluation, CUDA is optional and you can use CPU for evaluation. First, clone the repo with (if you are in China and Github is slow, you can use the mirror in [Gitee](https://gitee.com/daochenzha/DouZero)): ``` git clone https://github.com/kwai/DouZero.git ``` Make sure you have python 3.6+ installed. Install dependencies: ``` cd douzero pip3 install -r requirements.txt ``` We recommend installing the stable version of DouZero with ``` pip3 install douzero ``` If you are in China and the above command is too slow, you can use the mirror provided by Tsinghua University: ``` pip3 install douzero -i https://pypi.tuna.tsinghua.edu.cn/simple ``` or install the up-to-date version (it could be not stable) with ``` pip3 install -e . ``` Note that Windows users can only use CPU as actors. See [Issues in Windows](README.md#issues-in-windows) about why GPUs are not supported. Nonetheless, Windows users can still [run the demo locally](https://github.com/datamllab/rlcard-showdown). ## Training To use GPU for training, run ``` python3 train.py ``` This will train DouZero on one GPU. To train DouZero on multiple GPUs. Use the following arguments. *