理解多任务策略优化中的默认策略_TowardsanUnderstandingofDefaultPoliciesin资源-CSDN文库

版权申诉

26 浏览量 2022-01-27 06:44:15 上传评论收藏 748KB PDF 举报

在多任务策略优化中，默认策略的理解是强化学习领域中的一个重要课题。近年来，深度强化学习的许多成就都归功于正则化策略优化（Regularized Policy Optimization，RPO）算法，这些方法在多个领域表现出色。RPO算法的核心思想是训练智能体最大化累积奖励，并惩罚与参考或默认策略的行为偏差。在单任务环境中，RPO有深厚的理论基础，与自然梯度、信任区域和变分方法有紧密联系。然而，当涉及到多任务设置时，对于默认策略的理想的性质和影响的理解还相对有限，而这个领域正变得越来越重要，因为研究趋势转向培养更具有普适性的智能体。这篇论文旨在填补这一知识空白，通过形式化地将默认策略的质量与其对优化过程的影响关联起来，为多任务学习提供一个有原则性的RPO算法，并保证其性能。在介绍部分，作者指出适当的正则化是深度强化学习广泛成功的关键因素。RPO方法通过最大化奖励并最小化偏离有用行为的成本（通常以默认策略的形式编码）来训练智能体。这种方法不仅易于扩展，而且与函数近似兼容，已被证明可以减轻深度RL方法的高样本复杂性，使其成为解决高维度问题的理想选择。例如，像Monte Carlo Tree Search (MCTS)和Proximal Policy Optimization (PPO)等算法就成功地应用了这一思想。论文的主要贡献在于建立默认策略与优化效果之间的理论联系。通过这种方式，研究人员能够设计出一种新的RPO算法，该算法在多任务学习中具有强大的性能保证。这可能包括确保收敛性、稳定性和泛化能力的提升。此外，这种理论框架可能有助于揭示如何选择或学习有效的默认策略，以便在多任务环境中提供更好的指导。在具体实现上，论文可能会探讨如何在不同任务之间共享学习，以及如何利用每个任务的特定信息来改进默认策略。这可能涉及在策略网络中引入额外的结构，如任务嵌入或注意力机制，以允许智能体灵活适应任务的变化。同时，为了确保算法的效率和实用性，可能会有对计算资源和内存使用的优化考虑。这篇论文的工作对于深入理解多任务强化学习中的策略优化至关重要。它不仅为实际应用提供了理论支持，还有助于推动未来研究，探索更复杂的环境和任务中的智能决策制定。随着对默认策略的深入理解，我们有望开发出更加高效、鲁棒的智能体，能够在各种任务和环境中展现出强大的泛化能力。

资源推荐

资源详情

资源评论

Towards an Understanding of Default Policies

in Multitask Policy Optimization

Ted Moskovitz

Gatsby Unit, UCL

ted@gatsby.ucl.ac.uk

Michael Arbel

Universit´e Grenoble Alpes, Inria, CNRS

michael.n.arbel@gmail.com

Jack Parker-Holder

University of Oxford

jackph@robots.ox.ac.uk

Aldo Pacchiano

Microsoft Research

apacchiano@microsoft.com

Abstract

Much of the recent success of deep reinforcement learning has been driven by regularized

policy optimization (RPO) algorithms, with strong performance across multiple domains. In

this family of methods, agents are trained to maximize cumulative reward while penalizing

deviation in behavior from some reference, or default policy. In addition to empirical success,

there is a strong theoretical foundation for understanding RPO methods applied to single tasks,

with connections to natural gradient, trust region, and variational approaches. However, there is

limited formal understanding of desirable properties for default policies in the multitask setting,

an increasingly important domain as the ﬁeld shifts towards training more generally capable

agents. Here, we take a ﬁrst step towards ﬁlling this gap by formally linking the quality of the

default policy to its eﬀect on optimization. Using these results, we then derive a principled RPO

algorithm for multitask learning with strong performance guarantees.

1 Introduction

Appropriate regularization has been a key factor in the widespread success of policy-based deep

reinforcement learning (RL) (Levine, 2018; Furuta et al., 2021). The key idea underlying such

regularized policy optimization (RPO) methods is to train an agent to maximize reward while

minimizing some cost which penalizes deviations from useful behavior, typically encoded as a default

policy. In addition to being easily scalable and compatible with function approximation, these

methods have been shown to ameliorate the high sample complexity of deep RL methods, making

them an attractive choice for high-dimensional problems (Berner et al., 2019; Espeholt et al., 2018).

A natural question underlying this success is why these methods are so eﬀective. Fortunatley,

there is a strong foundation for the formal understanding of regularizers in the single-task setting.

These methods can be seen as approximating a form of natural gradient ascent (Kakade, 2002; Pac-

chiano et al., 2020; Moskovitz et al., 2021), trust region or proximal point optimization (Schulman

et al., 2015, 2017), or variational inference (Levine, 2018; Haarnoja et al., 2018; Marino et al., 2020;

Abdolmaleki et al., 2018), and thus are well-understood by theory (Agarwal et al., 2020).

However, as interest has grown in training general agents capable of providing real world utility,

there has been a shift in emphasis towards multitask learning. Accordingly, there are a number

of approaches to learning or constructing default policies for regularized policy optimization in

multitask settings (Galashov et al., 2019; Teh et al., 2017; Goyal et al., 2019, 2020). The basic idea

is to obtain a default policy which is generally useful for some family of tasks, thus oﬀering a form

of supervision to the learning process. However, there is little theoretical understanding of how the

choice of default policy aﬀects optimization. Our goal in this paper is to take a ﬁrst step towards

bridging this gap, asking:

What properties does a default policy need to have in order to improve optimization on new tasks?

arXiv:2111.02994v2 [cs.LG] 6 Nov 2021

This is a nuanced question. The choice of penalty, structural commonalities among the tasks

encountered by the agent, and even the distribution space in which the regularization is applied

have dramatic eﬀects on the resulting algorithm and the agent’s performance characteristics.

In this work, we focus on methods using the Kullback-Leibler (KL) divergence with respect to

the default policy, as they are the most common in the literature. We ﬁrst consider this form of

regularized policy optimization applied to a single task, with the goal of understanding how the

relationship between the default and optimal policies for a given problem aﬀect optimization. We

then generalize these results to the multitask setting, where we not only quantify the advantages of

this family of approaches, but also identify its limitations, both fundamental and algorithm-speciﬁc.

In the process of garnering new understanding of these algorithms, our results also imply a new

framework through which to understand families of tasks. Because diﬀerent algorithms are sensitive

to diﬀerent forms of structure, this leads to another guiding question, closely tied to the ﬁrst:

What properties does a group of tasks need to share for a given algorithm to provide a measurable

beneﬁt?

It’s clear that in order to be eﬀective, any multitask learning algorithm must be applied to a

task distribution with some form of structure identiﬁable by that algorithm: if tasks have nothing in

common, no understanding gained from one task will be useful for accelerating learning on another.

Algorithms may be designed to accommodate—or learn—a broader array of structures, but at

increased computational costs. In high-dimensional problems, function approximation mandates

new compromises. In this paper, which we view as a ﬁrst step towards understanding these trade-

oﬀs, we make the following contributions:

• We show the error bound and iteration complexity for optimization using an α-optimal default

policy, where sub-optimality is measured via distance from the optimal policy for a given task.

• From these results, we derive a principled RPO algorithm for multitask learning, which we

term total variation policy optimization (TVPO). We show that popular multitask KL-based

algorithms can be seen as approximations of TVPO and demonstrate the strong performance

of TVPO on simple tasks.

• We oﬀer novel insights on the optimization characteristics—both limitations and advantages—

of common multitask RPO frameworks in the literature.

2 Regularized Policy Optimization

Reinforcement learning In reinforcement learning (RL), an agent learns how to act within its

environment in order to maximize its performance on a given task or tasks. We model a task as

a ﬁnite Markov decision process (MDP; (Puterman, 2010)) M = (S, A, P, r, γ, ρ), where S, A are

ﬁnite state and action spaces, respectively, P : S × A → ∆(S) is the state transition distribution,

r : S × A → [0, 1] is a reward function, γ ∈ [0, 1) is the discount factor, and ρ ∈ ∆(S) is the

starting state distribution. ∆(·) is used to denote the simplex over a given space. We also assume

access to a restart distribution for training µ ∈ ∆(S) such that µ(s) > 0 ∀s ∈ S, as is common in

the literature (Kakade and Langford, 2002; Agarwal et al., 2020). The agent takes actions using

a stationary policy π : S → ∆(A), which, in conjunction with the transition dynamics, induces a

distribution over trajectories τ = (s

, a

)

∞

t=0

The value function V

: S → R

measures the expected discounted cumuluative reward obtained

by following π from state s, V

(s)

= E



∞

t=0

r(s

, a

)|s

= s



, where the expectation is with

respect to the distribution over trajectories induced by π in M. We overload notation and deﬁne

(ρ)

= E

∼ρ

)] as the expected value for initial state distribution ρ. The action-value and

advantage functions are given by

(s, a)

= E

∞

t=0

r(s

, a

)|s

= s, a

= a

, A

(s, a)

= Q

(s, a) − V

(s).

By d

, we denote the discounted state visitation distribution of π with starting state distribution

µ, so that

(s) = E

∼µ

(1 − γ)

∞

t=0

= s|s

)

(2.1)

where d

= E

∼µ



(s)



. The goal of the agent is to adapt its policy so as to maximize its value,

i.e., optimize max

(ρ). We use π

∈ argmax

(ρ) to denote the optimal policy and V

and

as shorthand for V

and Q

, respectively.

Policy parameterizations In practice, this problem typically takes the form max

θ∈Θ

, where

{π

|θ ∈ Θ} is a class of parametric policies. In this work, we primarily consider the softmax policy

class, which may be tabular or complete (able to represent any stochastic policy), as in the case of

the tabular softmax

(a|s) =

exp(θ

s,a

)

∈A

exp(θ

s,a

)

, (2.2)

where θ ∈ R

|S|×|A|

, or restricted, where π

(a|s) ∝ exp(f

(s, a)), with f

: S×A → R some parametric

function class (e.g., a neural network).

The general form of the regularized policy optimization (RPO) objective function is given by

(θ)

= V

(µ) − λΩ(θ), (2.3)

where Ω is some convex regularization functional. Gradient ascent updates proceed according to

(t+1)

= θ

(t)

+ η∇

(θ

(t)

). (2.4)

For simplicity of notation, from this point forward, for iterative algorithms which obtain successive

estimates of parameters θ

(t)

, we denote the associated policy and value functions as π

(t)

and V

(t)

respectively. The choice of Ω plays a signﬁcant role in algorithm design and practice, as we discuss

below. It’s also important to note that the error bounds and convergence rates we derive are

based on the basic policy gradient framework in Appendix Algorithm 2, in which update Eq. (2.4)

is applied every after a ﬁxed number B of trajectories {τ

}

b=1

is sampled from the environment.

Therefore, the iteration complexities below are proportional to the associated sample complexity.

3 Related Work

Single-task learning The majority of the theoretical (Agarwal et al., 2020; Grill et al., 2020) and

empirical (Schulman et al., 2015, 2017; Abdolmaleki et al., 2018; Pacchiano et al., 2020) literature

has focused on the use of RPO in a single-task setting, i.e., applied to a single MDP M. The

majority of these methods place a soft or hard constraint on the Kullback-Leibler (KL) divergence

between the updated policy at each time step and the current policy, maximizing an objective of

the form

(π

, π

) =

∞

t=0

∼d

∼π

(·|s

)



G(s

, a

) − λKL(π

(·|s

), π

(·|s

))



(3.1)

where G : S ×A → R is typically the Q- or advantage function and π

, π

∈ {π

, π

} (Furuta et al.,

2021). At each update, then, the idea is to maximize reward while minimizing the regularization

cost. From a theoretical perspective, such methods can often be framed as a form of approximate

variational inference, with either learned (Abdolmaleki et al., 2018; Song et al., 2019; Peng et al.,

2021; Nair et al., 2021; Peters et al., 2010) or ﬁxed (Todorov, 2007; Toussaint and Storkey, 2006;

Rawlik et al., 2013; Fox et al., 2016) π

. When π

≈ π

, we can also understand such approaches as

approximating the natural policy gradient (Kakade, 2002), which is known to accelerate convergence

(Agarwal et al., 2020). Similarly, regularizing the objective using the Wasserstein distance (Pacchi-

ano et al., 2020) rather than the KL divergence produces updates which approximate those of the

Wasserstein natural policy gradient (Moskovitz et al., 2021). Other approaches can be understood

as trust region or proximal point methods (Schulman et al., 2015, 2017; Touati et al., 2020), or even

model-based approaches (Grill et al., 2020). It’s also important to note the special case of entropy

regularization, where Ω(θ) = −E

s∼U

H[π

(·|s)] = E

s∼U

KL(π

(·|s), U

) (where U

denotes the

uniform distribution over a space X) which is perhaps the most common form of RPO (Haarnoja

et al., 2018; Levine, 2018; Mnih et al., 2016; Williams and Peng, 1991; Schulman et al., 2018) and has

been shown to aid optimization by encouraging exploration and smoothing the objective landscape

(Ahmed et al., 2019).

Multitask learning Less common in the literature are policy regularizers designed explicitly for

multitask settings. In many multitask RL algorithm which apply RPO, shared task structure is

leveraged in other forms (e.g., importance weighting), and the regularizer itself doesn’t reﬂect shared

information (Espeholt et al., 2018; Riedmiller et al., 2018). However, in cases where the penalty

is designed for multitask learning, the policy is penalized for deviating from a more general task-

agnostic default policy meant to encode behavior which is generally useful for the family of tasks

at hand. The use of such a behavioral default is intuitive: by distilling the common structure of the

tasks the agent encounters into behaviors which have shown themselves to be useful, optimization

on new tasks can be improved through with the help of prior knowledge. For example, some

approaches Goyal et al. (2019, 2020) construct a default policy by marginalizing over goals g for

a set of goal-conditioned policies π

(a|s) =

P (g)π

(a|s, g). Such partitioning of the input into

goal-dependent and goal-agnostic features can be used to create structured internal representations

via an information bottleck (Tishby et al., 2000), shown empirically to improve generalization.

In other multitask RPO algorithms, the default policies are derived from a Bayesian framework

which views π

as a prior (Wilson et al., 2007; O’Donoghue et al., 2020). Still other methods learn

online through distillation (Hinton et al., 2015) by minimizing KL(π

, π) with respect to π

(Galashov et al., 2019; Teh et al., 2017). When π

is preserved across tasks but π

is re-initialized,

learns the average behavior across task-speciﬁc policies. However, to our knowledge, there has

been no investigation of the formal optimization properties of explicitly multitask approaches, and

basic questions remain unanswered.

4 A Basic Theory for Default Policies

At an intuitive level, the question we’d like to explore is: What properties does a default policy need

in order to improve optimization? By “improve” we refer to a reduction in the error at convergence

with respect to the optimal value function or a reduction in the number of updates required to reach

a given error threshold. To begin, we consider perhaps the simplest default: the uniform policy.

The proofs for this section are provided in Appendix C.

4.1 Log-barrier regularization

For now, we’ll restrict ourselves to the direct softmax parameterization (Eq. (2.2)) with access to

exact gradients. Our default is a uniform policy over actions, i.e.: π

(a|s) = U

, resulting in the

objective

(θ)

= V

(µ) − λE

s∼U

[KL(U

, π

(·|s))]

≡ V

(µ) +

|S||A|

s,a

log π

(a|s),

(4.1)

where we have dropped terms that are constant with respect to θ. Importantly, it’s known that

even this default policy has beneﬁcial eﬀects on optimization by erecting a log-barrier against low

values of π

(a|s). This barrier prevents gradients from quickly dropping to zero due to exponential

scaling, leading to a polynomial convergence rate

. We now brieﬂy restate convergence error and

iteration complexity results for this case, due to Agarwal et al. (2020):

Lemma 4.1 (Error bound for log-barrier regularization). Suppose θ is such that k∇

(θ)k

≤ 

opt

with 

opt

≤

2|S||A|

. Then we have for all starting state distributions ρ,

(ρ) ≥ V

(ρ) −

2λ

1 − γ



∞

We brieﬂy comment on the term



∞

(in which the division refers to component-wise divi-

sion), known as the distribution mismatch coeﬃcient, which roughly quantiﬁes the diﬃculty of the

exploration problem faced by the optimization algorithm. We note that µ is the starting distribu-

tion used for training/optimization, while the ultimate goal is to perform well on the target starting

state distribution ρ. The iteration complexity is given below.

Lemma 4.2 (Iteration complexity for log-barrier regularization). Let β

8γ

(1−γ)

2λ

|S|

. Starting

from any initial θ

(0)

, consider the updates Eq. (2.4) with λ =

(1−γ)



∞

and η = 1/β

. Then for all

starting state distribution ρ, we have

min

t≤T

(ρ) − V

(t)

(ρ)} ≤  whenever T ≥

320|S|

|A|

(1 − γ)





∞

These results will act as useful reference points for the following investigation. At a minimum,

we’d like a default policy to provide guarantees that are at least as good as those of log-barrier

regularization.

4.2 Regularization with an α-optimal policy

To understand what properties are required of the default policy, we place an upper-bound on the

suboptimality of π

via the TV distance. For each s ∈ S, we have

(π

(·|s), π

(·|s)) ≤ α(s) (4.2)

Our regularized objective is

(θ) = V

(µ) − λE

s∼U

[KL(π

(·|s), π(·|s))]

≡ V

(µ) +

|S|

s,a

(a|s) log π

(a|s)

(4.3)

for starting state distribution µ ∈ ∆(S). We then have

∂J

(θ)

∂θ

s,a

1 − γ

(s)π

(a|s)A

(s, a) +

|S|

(π

(a|s) − π

(a|s)).

(4.4)

Our ﬁrst result presents the error bound for ﬁrst-order stationary points of the π

-regularized

objective.

It remains an open question whether entropy regularization, which is gentler in penalizing low probabilities,

produces a polynomial convergence rate.

剩余29页未读，继续阅读

评论收藏

内容反馈

版权申诉

易小侠

粉丝: 6568
资源: 9万+

理解多任务策略优化中的默认策略_Towards an Understanding of Default Policies in

最新资源

理解多任务策略优化中的默认策略_Towards an Understanding of Default Policies in

TextFlow Towards Better Understanding of Evolving Topics in Text

UniDT鲁棒优化算法____The_implementation_on_Towards_Scala_WDRO.zip

Analogies Explained_Towards Understanding Word Embeddings 词嵌入1

Image Embedding of PMU Data for Deep Learning towards Transient Disturbance

迈向6G物联网最新进展、用例和开放式挑战_Towards 6G Internet of Things Recent Advanc

走向人工智能决策科学实证研究综述_Towards a Science of Human-AI Decision Making A

Exploring_The_Maze_Of_Memory_Layouts_Towards_Exploits.pdf

「企业安全」MI-LXC：a_first_step_towards_a_free_CyberRange_？ - 技术分析.

信息安全_数据安全_Towards participatory IoT monito.pdf

Towards_the_Detection_of_Inconsistencies_in_Public_Security_V

信息安全_数据安全_Towards a More Secure Operating .pdf

NOX: Towards an Operating System for Networks

Practical_Guide_to_LTE－A，_VoLTE_and_IoT_－_Paving_the_way_towards_5G(1).pdf

A break in the clouds_towards a cloud definition

Physician leadership development_ towards multidisciplin

Wheeled Mobile Robotics. From Fundamentals Towards Autonomous Systems

TOWARDS THE HARMONIZATION OF SAFETY ASSESSMENT METHODS OF AUTOMA

面向全球和无限连接——专用NGSO卫星星座在未来天地网络中的作用_Towards Global and Limitless Co

面向单阶段弱监督语义分割_Towards Single Stage Weakly Supervised Semantic Seg

Cobalt Strike下载

计算机系统-笔记-HUN2021级

北京邮电大学计算机考研复试笔试资料

cs1.6老版本供下载

合成孔径雷达的经典成像算法cs(matlab)仿真代码（吐血整理，内容全，注释全）

港大CS（MSC）面试整理

合成孔径雷达RD CS OmegaK算法点目标仿真.rar

计算机科学导论原书第二版答案.zip

Cobalt-Strike-4.5

cobaltstrike4.3.zip

最新资源