给力的PyTorch深度强化学习库

# Tianshou's Mujoco Benchmark We benchmarked Tianshou algorithm implementations in 9 out of 13 environments from the MuJoCo Gym task suite^{[[1]](#footnote1)}. For each supported algorithm and supported mujoco environments, we provide: - Default hyperparameters used for benchmark and scripts to reproduce the benchmark; - A comparison of performance (or code level details) with other open source implementations or classic papers; - Graphs and raw data that can be used for research purposes^{[[2]](#footnote2)}; - Log details obtained during training^{[[2]](#footnote2)}; - Pretrained agents^{[[2]](#footnote2)}; - Some hints on how to tune the algorithm. Supported algorithms are listed below: - [Deep Deterministic Policy Gradient (DDPG)](https://arxiv.org/pdf/1509.02971.pdf), [commit id](https://github.com/thu-ml/tianshou/tree/e605bdea942b408126ef4fbc740359773259c9ec) - [Twin Delayed DDPG (TD3)](https://arxiv.org/pdf/1802.09477.pdf), [commit id](https://github.com/thu-ml/tianshou/tree/e605bdea942b408126ef4fbc740359773259c9ec) - [Soft Actor-Critic (SAC)](https://arxiv.org/pdf/1812.05905.pdf), [commit id](https://github.com/thu-ml/tianshou/tree/e605bdea942b408126ef4fbc740359773259c9ec) - [REINFORCE algorithm](https://papers.nips.cc/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf), [commit id](https://github.com/thu-ml/tianshou/tree/e27b5a26f330de446fe15388bf81c3777f024fb9) - [Natural Policy Gradient](https://proceedings.neurips.cc/paper/2001/file/4b86abe48d358ecf194c56c69108433e-Paper.pdf), [commit id](https://github.com/thu-ml/tianshou/tree/844d7703c313009c4c364edb4018c91de93439ca) - [Advantage Actor-Critic (A2C)](https://openai.com/blog/baselines-acktr-a2c/), [commit id](https://github.com/thu-ml/tianshou/tree/1730a9008ad6bb67cac3b21347bed33b532b17bc) - [Proximal Policy Optimization (PPO)](https://arxiv.org/pdf/1707.06347.pdf), [commit id](https://github.com/thu-ml/tianshou/tree/6426a39796db052bafb7cabe85c764db20a722b0) - [Trust Region Policy Optimization (TRPO)](https://arxiv.org/pdf/1502.05477.pdf), [commit id](https://github.com/thu-ml/tianshou/tree/5057b5c89e6168220272c9c28a15b758a72efc32) - [Hindsight Experience Replay (HER)](https://arxiv.org/abs/1707.01495) ## EnvPool We highly recommend using envpool to run the following experiments. To install, in a linux machine, type: ```bash pip install envpool ``` After that, `make_mujoco_env` will automatically switch to envpool's Mujoco env. EnvPool's implementation is much faster (about 2\~3x faster for pure execution speed, 1.5x for overall RL training pipeline in average) than python vectorized env implementation, and it's behavior is consistent to gym's Mujoco env. For more information, please refer to EnvPool's [GitHub](https://github.com/sail-sg/envpool/) and [Docs](https://envpool.readthedocs.io/en/latest/api/mujoco.html). ## Usage Run ```bash $ python mujoco_sac.py --task Ant-v3 ``` Logs is saved in `./log/` and can be monitored with tensorboard. ```bash $ tensorboard --logdir log ``` You can also reproduce the benchmark (e.g. SAC in Ant-v3) with the example script we provide under `examples/mujoco/`: ```bash $ ./run_experiments.sh Ant-v3 sac ``` This will start 10 experiments with different seeds. Now that all the experiments are finished, we can convert all tfevent files into csv files and then try plotting the results. ```bash # generate csv $ ./tools.py --root-dir ./results/Ant-v3/sac # generate figures $ ./plotter.py --root-dir ./results/Ant-v3 --shaded-std --legend-pattern "\\w+" # generate numerical result (support multiple groups: `--root-dir ./` instead of single dir) $ ./analysis.py --root-dir ./results --norm ``` ## Example benchmark <img src="./benchmark/Ant-v3/offpolicy.png" width="500" height="450"> Other graphs can be found under `examples/mujuco/benchmark/` For pretrained agents, detailed graphs (single agent, single game) and log details, please refer to [https://cloud.tsinghua.edu.cn/d/f45fcfc5016043bc8fbc/](https://cloud.tsinghua.edu.cn/d/f45fcfc5016043bc8fbc/). ## Offpolicy algorithms ### Notes 1. In offpolicy algorithms (DDPG, TD3, SAC), the shared hyperparameters are almost the same, and unless otherwise stated, hyperparameters are consistent with those used for benchmark in SpinningUp's implementations (e.g. we use batchsize 256 in DDPG/TD3/SAC while SpinningUp use 100. Minor difference also lies with `start-timesteps`, data loop method `step_per_collect`, method to deal with/bootstrap truncated steps because of timelimit and unfinished/collecting episodes (contribute to performance improvement), etc.). 2. By comparison to both classic literature and open source implementations (e.g., SpinningUp)^{[[1]](#footnote1)}^{[[2]](#footnote2)}, Tianshou's implementations of DDPG, TD3, and SAC are roughly at-parity with or better than the best reported results for these algorithms, so you can definitely use Tianshou's benchmark for research purposes. 3. We didn't compare offpolicy algorithms to OpenAI baselines [benchmark](https://github.com/openai/baselines/blob/master/benchmarks_mujoco1M.htm), because for now it seems that they haven't provided benchmark for offpolicy algorithms, but in [SpinningUp docs](https://spinningup.openai.com/en/latest/spinningup/bench.html) they stated that "SpinningUp implementations of DDPG, TD3, and SAC are roughly at-parity with the best-reported results for these algorithms", so we think lack of comparisons with OpenAI baselines is okay. ### DDPG | Environment | Tianshou (1M) | [Spinning Up (PyTorch)](https://spinningup.openai.com/en/latest/spinningup/bench.html) | [TD3 paper (DDPG)](https://arxiv.org/abs/1802.09477) | [TD3 paper (OurDDPG)](https://arxiv.org/abs/1802.09477) | | :--------------------: | :---------------: | :----------------------------------------------------------: | :--------------------------------------------------: | :-----------------------------------------------------: | | Ant | 990.4±4.3 | ~840 | **1005.3** | 888.8 | | HalfCheetah | **11718.7±465.6** | ~11000 | 3305.6 | 8577.3 | | Hopper | **2197.0±971.6** | ~1800 | **2020.5** | 1860.0 | | Walker2d | 1400.6±905.0 | ~1950 | 1843.6 | **3098.1** | | Swimmer | **144.1±6.5** | ~137 | N | N | | Humanoid | **177.3±77.6** | N | N | N | | Reacher | **-3.3±0.3** | N | -6.51 | -4.01 | | InvertedPendulum | **1000.0±0.0** | N | **1000.0** | **1000.0** | | InvertedDoublePendulum | 8364.3±2778.9 | N | **9355.5** | 8370.0