具有研究友好功能的深度强化学习算法的高质量单文件实施（PPO、DQN、C51、DDPG、TD3、SAC、PPG）

# CleanRL (Clean Implementation of RL Algorithms) [<img src="https://img.shields.io/badge/license-MIT-blue">](https://github.com/vwxyzjn/cleanrl) [](https://github.com/vwxyzjn/cleanrl/actions/workflows/tests.yaml) [](https://docs.cleanrl.dev/) [<img src="https://img.shields.io/discord/767863440248143916?label=discord">](https://discord.gg/D6RCjA6sVT) [<img src="https://img.shields.io/youtube/channel/views/UCDdC6BIFRI0jvcwuhi3aI6w?style=social">](https://www.youtube.com/channel/UCDdC6BIFRI0jvcwuhi3aI6w/videos) [](https://github.com/psf/black) [](https://pycqa.github.io/isort/) [<img src="https://img.shields.io/badge/%F0%9F%A4%97%20Models-Huggingface-F8D521">](https://huggingface.co/cleanrl) [](https://colab.research.google.com/github/vwxyzjn/cleanrl/blob/master/docs/get-started/CleanRL_Huggingface_Integration_Demo.ipynb) CleanRL is a Deep Reinforcement Learning library that provides high-quality single-file implementation with research-friendly features. The implementation is clean and simple, yet we can scale it to run thousands of experiments using AWS Batch. The highlight features of CleanRL are: * 📜 Single-file implementation * *Every detail about an algorithm variant is put into a single standalone file.* * For example, our `ppo_atari.py` only has 340 lines of code but contains all implementation details on how PPO works with Atari games, **so it is a great reference implementation to read for folks who do not wish to read an entire modular library**. * 📊 Benchmarked Implementation (7+ algorithms and 34+ games at https://benchmark.cleanrl.dev) * 📈 Tensorboard Logging * 🪛 Local Reproducibility via Seeding * 🎮 Videos of Gameplay Capturing * 🧫 Experiment Management with [Weights and Biases](https://wandb.ai/site) * 💸 Cloud Integration with docker and AWS You can read more about CleanRL in our [JMLR paper](https://www.jmlr.org/papers/volume23/21-1342/21-1342.pdf) and [documentation](https://docs.cleanrl.dev/). CleanRL only contains implementations of **online** deep reinforcement learning algorithms. If you are looking for **offline** algorithms, please check out [corl-team/CORL](https://github.com/corl-team/CORL), which shares a similar design philosophy as CleanRL. > ℹ️ **Support for Gymnasium**: [Farama-Foundation/Gymnasium](https://github.com/Farama-Foundation/Gymnasium) is the next generation of [`openai/gym`](https://github.com/openai/gym) that will continue to be maintained and introduce new features. Please see their [announcement](https://farama.org/Announcing-The-Farama-Foundation) for further detail. We are migrating to `gymnasium` and the progress can be tracked in [vwxyzjn/cleanrl#277](https://github.com/vwxyzjn/cleanrl/pull/277). > ⚠️ **NOTE**: CleanRL is *not* a modular library and therefore it is not meant to be imported. At the cost of duplicate code, we make all implementation details of a DRL algorithm variant easy to understand, so CleanRL comes with its own pros and cons. You should consider using CleanRL if you want to 1) understand all implementation details of an algorithm's varaint or 2) prototype advanced features that other modular DRL libraries do not support (CleanRL has minimal lines of code so it gives you great debugging experience and you don't have do a lot of subclassing like sometimes in modular DRL libraries). ## Get started Prerequisites: * Python >=3.7.1,<3.11 * [Poetry 1.2.1+](https://python-poetry.org) To run experiments locally, give the following a try: ```bash git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl poetry install # alternatively, you could use `poetry shell` and do # `python run cleanrl/ppo.py` poetry run python cleanrl/ppo.py \ --seed 1 \ --env-id CartPole-v0 \ --total-timesteps 50000 # open another terminal and enter `cd cleanrl/cleanrl` tensorboard --logdir runs ``` To use experiment tracking with wandb, run ```bash wandb login # only required for the first time poetry run python cleanrl/ppo.py \ --seed 1 \ --env-id CartPole-v0 \ --total-timesteps 50000 \ --track \ --wandb-project-name cleanrltest ``` If you are not using `poetry`, you can install CleanRL with `requirements.txt`: ```bash # core dependencies pip install -r requirements/requirements.txt # optional dependencies pip install -r requirements/requirements-atari.txt pip install -r requirements/requirements-mujoco.txt pip install -r requirements/requirements-mujoco_py.txt pip install -r requirements/requirements-procgen.txt pip install -r requirements/requirements-envpool.txt pip install -r requirements/requirements-pettingzoo.txt pip install -r requirements/requirements-jax.txt pip install -r requirements/requirements-docs.txt pip install -r requirements/requirements-cloud.txt ``` To run training scripts in other games: ``` poetry shell # classic control python cleanrl/dqn.py --env-id CartPole-v1 python cleanrl/ppo.py --env-id CartPole-v1 python cleanrl/c51.py --env-id CartPole-v1 # atari poetry install -E atari python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4 python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4 python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4 python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4 # NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux) poetry install -E envpool python cleanrl/ppo_atari_envpool.py --env-id BreakoutNoFrameskip-v4 # Learn Pong-v5 in ~5-10 mins # Side effects such as lower sample efficiency might occur poetry run python ppo_atari_envpool.py --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3 # procgen poetry install -E procgen python cleanrl/ppo_procgen.py --env-id starpilot python cleanrl/ppg_procgen.py --env-id starpilot # ppo + lstm poetry install -E atari python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4 ``` You may also use a prebuilt development environment hosted in Gitpod: [](https://gitpod.io/#https://github.com/vwxyzjn/cleanrl) ## Algorithms Implemented | Algorithm | Variants Implemented | | ----------- | ----------- | | ✅ [Proximal Policy Gradient (PPO)](https://arxiv.org/pdf/1707.06347.pdf) | [`ppo.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py), [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppopy) | | | [`ppo_atari.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari.py), [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_ataripy) | | [`ppo_continuous_action.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_continuous_action.py), [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_continuous_actionpy) | | [`ppo_atari_lstm.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_lstm.py), [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_lstmpy) | | [`ppo_atari_envpool.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool.py), [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_envpoolpy) | | [`ppo_atari_envpool_xla_jax.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax.py), [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_envpool_xla_jaxpy) | | [`ppo_atari_envpool_xla_jax_scan.py`](https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_atari_envpool_xla_jax_scan.py), [docs](https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_atari_envpool_xla_jax_scanpy)) | | [`ppo_procgen