CurricuLLM: Automatic Task Curricula Design for Learning Complex
Robot Skills using Large Language Models
Kanghyun Ryu
1
, Qiayuan Liao
1
, Zhongyu Li
1
, Koushil Sreenath
1
, Negar Mehr
1
Abstract— Curriculum learning is a training mechanism in
reinforcement learning (RL) that facilitates the achievement of
complex policies by progressively increasing the task difficulty
during training. However, designing effective curricula for a
specific task often requires extensive domain knowledge and
human intervention, which limits its applicability across various
domains. Our core idea is that large language models (LLMs),
with their extensive training on diverse language data and
ability to encapsulate world knowledge, present significant
potential for efficiently breaking down tasks and decomposing
skills across various robotics environments. Additionally, the
demonstrated success of LLMs in translating natural language
into executable code for RL agents strengthens their role in gen-
erating task curricula. In this work, we propose CurricuLLM,
which leverages the high-level planning and programming capa-
bilities of LLMs for curriculum design, thereby enhancing the
efficient learning of complex target tasks. CurricuLLM consists
of: (Step 1) Generating sequence of subtasks that aid target
task learning in natural language form, (Step 2) Translating
natural language description of subtasks in executable task
code, including the reward code and goal distribution code, and
(Step 3) Evaluating trained policies based on trajectory rollout
and subtask description. We evaluate CurricuLLM in various
robotics simulation environments, ranging from manipulation,
navigation, and locomotion, to show that CurricuLLM can aid
learning complex robot control tasks. In addition, we validate
humanoid locomotion policy learned through CurricuLLM in
real-world. The code is provided in https://github.com/
labicon/CurricuLLM
I. INTRODUCTION
Deep reinforcement learning (RL) has achieved notable
success across various robotics tasks, including manipula-
tion [1], navigation [2], and locomotion [3]. However, RL re-
quires informative samples for learning, and obtaining these
from a random policy is highly sample-inefficient, especially
for complex tasks. In contrast, human learning strategies
differ significantly from random action trials; they typically
start with simpler tasks and progressively increase difficulty.
Curriculum learning, inspired by this structured approach of
learning, aims to train models in a meaningful sequence [4],
gradually enhancing the complexity of the training data [5]
or the tasks themselves [6]. Particularly in RL, curriculum
learning improves training efficiency by focusing on simpler
tasks that can provide informative experiences to reach more
complex target task, instead of starting from scratch [6].
Although effective, designing a good curriculum is chal-
lenging. Manual curriculum design often necessitates the
*This work is supported by the National Science Foundation, under grants
ECCS-2438314 CAREER Award, CNS-2423130, and CCF-2423131
1
Mechanical Engineering, University of California Berkeley
{kanghyun.ryu, qiayuanl, zhongyu li, koushils,
negar}@berkeley.edu
Design a curriculum for
humanoid locomotion
Prompts: Environment Description, Robot
Description, Target Task Description, …
Sequential Training throughout Curriculum
Fine-tuning Policy
(Step 2)
Task Code Sampling
def compute_rewards(self):
# balance reward
…
# velocity reward
(Step 3)
Optimal Policy Selection
Policy 1:
base_lin_vel: [-0.202 -0.114 0.017]
base_ang_vel: [-0.001 -0.004 0.084]
Policy from
Preceding Subtask
(Step 1)
Curriculum Design
Basic Stability Learning
Learn to Walk
Increase Speed
Target Task
Sequentially Trained Policies
Fig. 1: CurricuLLM takes natural language description of environ-
ments, robots, and target task that we wish the robot to learn, and
then generates a sequence of subtasks. In each subtasks, it samples
different task codes and evaluates the resulting trained policy to
find the policy which is best aligned within the current subtask.
These iterations are repeated throughout the curriculum subtasks to
sequentially train a policy that reaches complex target task.
costly intervention of human experts [7], [8], [9] and is
typically restricted to a limited set of predefined tasks [10].
Consequently, several works focused on automatic curricu-
lum learning (ACL). To generate task curricula, ACL re-
quires the ability of both determining subtasks aligned with
the target task, ranking the difficulty of each subtask, and
organizing them in ascending order of difficulty [11]. How-
ever, autonomously evaluating the relevance and difficulty
of these subtasks remains unresolved. As a result, ACL has
been limited to initial state curricula [12], [13], goal state
curricula [14], or environment curricula [10], [15], rather
than task-level curricula.
Meanwhile, in recent years, large language models
(LLMs) trained on extensive collections of language
data [16], [17], [18] have been recognized as reposito-
ries of world knowledge expressed in linguistic form [19].
Leveraging this world knowledge, LLMs have demonstrated
their capabilities in task planning [20] and skill decompo-
sition for complex robotic tasks [21], [22]. Furthermore,
the programming skills of LLMs enabled smooth integration
between high-level language description and robotics through
API call composition [23], [24], simulation environment
generation [25], [26], or reward design [27], [28].
In this paper, we introduce CurricuLLM, which leverages
the reasoning and coding capabilities of LLMs to design
curricula for complex robotic control tasks. Our goal is to
autonomously generate a series of subtasks that facilitate
the learning of complex target tasks without the need for
extensive human intervention. Utilizing the LLM’s task de-
composition and coding, CurricuLLM autonomously gener-
ates sequences of subtasks along with appropriate reward
arXiv:2409.18382v1 [cs.RO] 27 Sep 2024
剩余12页未读，继续阅读
资源评论
