2. Approximation in Value Space - Rollout Algorithms
2.1. Deterministic Discrete Spaces Finite Ho rizon Problems . . p. 14 8
2.2. Approximation in Value Space . . . . . . . . . . . . . p. 157
2.3. Rollout Algorithms for Discrete Optimization . . . . . . . p. 158
2.3.1. Cost Improvement with Rollout - Sequential Consistency, . .
Sequential Improvement . . . . . . . . . . . . . . p. 163
2.3.2. The Fortified Rollout Algorithm . . . . . . . . . . . p. 170
2.3.3. Using Multiple Base Heuristics - Parallel Rollout . . . p. 173
2.3.4. Simplified Rollout Algorithms . . . . . . . . . . . . p. 174
2.3.5. Truncated Rollout with Terminal Cost Approximation . p. 175
2.3.6. Model-Free Rollout . . . . . . . . . . . . . . . . p. 1 76
2.4. Rollout and Approximation in Value Space with Multistep . . . .
Lookahead . . . . . . . . . . . . . . . . . . . . . . p. 180
2.4.1. Iterative Deepening Using Forward Dynamic . . . . . . . .
Programming . . . . . . . . . . . . . . . . . . . p. 186
2.4.2. Incremental Multistep Rollout . . . . . . . . . . . . p. 188
2.5. Constraine d Forms of Rollout Algorithms . . . . . . . . p. 190
2.5.1. Constrained Rollout for Discrete Optimization and Integer . .
Programming . . . . . . . . . . . . . . . . . . . p. 202
2.6. Small Stage Costs and Long Horiz on - Continuous-Time . . . . .
Rollout . . . . . . . . . . . . . . . . . . . . . . . p. 20 6
2.7. Stochastic Rollout and Monte Carlo Tree Search . . . . . p. 214
2.7.1. Simplified Rollout and Policy Iteration . . . . . . . . p. 218
2.7.2. Certainty Equivalence Approximations . . . . . . . . p. 219
2.7.3. Simulation-Based Implementation of the Rollout . . . . . . .
Algorithm . . . . . . . . . . . . . . . . . . . . . p. 220
2.7.4. Variance Reduction in Rollout - Compa ring Advantages p. 223
2.7.5. Monte Carlo Tree Search . . . . . . . . . . . . . . p. 22 6
2.7.6. Randomized Policy Improve ment by Monte Carlo . . . . . .
Tree Search . . . . . . . . . . . . . . . . . . . . p. 229
2.8. Rollout for Infinite-Spaces Pro blems - Optimiza tion . . . . . . .
Heuristics . . . . . . . . . . . . . . . . . . . . . . p. 230
2.8.1. Rollout for Infinite-Spaces Deterministic Problems . . . p. 230
2.8.2. Rollout Based on Stochastic Programming . . . . . . p. 234
2.8.3. Sto chastic Programming with Certainty Equivalence . . p. 237
2.9. Multiagent Rollout . . . . . . . . . . . . . . . . . . p. 23 8
2.9.1. Asynchronous and Autonomous Multiagent Rollout . . p. 249
2.10. Rollout for Bayesian Optimization and Sequential . . . . . . .
Estimation . . . . . . . . . . . . . . . . . . . . . p. 253
2.11. Adaptive Control by Rollout with a POMDP . . . . . . . . .
Formulation . . . . . . . . . . . . . . . . . . . . . p. 264
2.12. Rollout for Minimax Control . . . . . . . . . . . . . p. 272
2.13. Notes, Sources, and Exercise s . . . . . . . . . . . . . p. 280
