Instruction Tuning for Large Language Models A Survey.pdf
需积分: 5 103 浏览量
2024-06-13
21:07:02
上传
评论
收藏 4.71MB PDF 举报
Instruction Tuning for Large Language Models: A Survey
Shengyu Zhang
♠
, Linfeng Dong
♠
, Xiaoya Li
♣
, Sen Zhang
♠
Xiaofei Sun
♠
, Shuhe Wang
♣
, Jiwei Li
♠
, Runyi Hu
♠
Tianwei Zhang
▲
, Fei Wu
♠
and Guoyin Wang
♦
Abstract
This paper surveys research works in the
quickly advancing field of instruction tuning
(IT), a crucial technique to enhance the
capabilities and controllability of large
language models (LLMs). Instruction tuning
refers to the process of further training LLMs
on a dataset consisting of (INSTRUCTION,
OUTPUT) pairs in a supervised fashion,
which bridges the gap between the next-word
prediction objective of LLMs and the users’
objective of having LLMs adhere to human
instructions. In this work, we make a
systematic review of the literature, including
the general methodology of IT, the construction
of IT datasets, the training of IT models, and
applications to different modalities, domains
and application, along with analysis on
aspects that influence the outcome of IT (e.g.,
generation of instruction outputs, size of the
instruction dataset, etc). We also review the
potential pitfalls of IT along with criticism
against it, along with efforts pointing out
current deficiencies of existing strategies and
suggest some avenues for fruitful research.
1 Introduction
The field of large language models (LLMs)
has witnessed remarkable progress in recent
years. LLMs such as GPT-3 (Brown et al.,
2020b), PaLM (Chowdhery et al., 2022), and
LLaMA (Touvron et al., 2023a) have demonstrated
impressive capabilities across a wide range of
natural language tasks (Zhao et al., 2021; Wang
et al., 2022b, 2023a; Wan et al., 2023; Sun et al.,
2023c; Wei et al., 2023; Li et al., 2023a; Gao et al.,
2023a; Yao et al., 2023; Yang et al., 2022a; Qian
et al., 2022; Lee et al., 2022; Yang et al., 2022b;
Gao et al., 2023b; Ning et al., 2023; Liu et al.,
♠
Zhejiang University,
♣
Shannon.AI,
▲
Nanyang
Technological University,
♦
Amazon
Email: sy_zhang@zju.edu.cn
Project link:
https://github.com/xiaoya-li/
Instruction-Tuning-Survey
2021b; Wiegreffe et al., 2021; Sun et al., 2023b,a;
Adlakha et al., 2023; Chen et al., 2023). One of the
major issues with LLMs is the mismatch between
the training objective and users’ objective: LLMs
are typically trained on minimizing the contextual
word prediction error on large corpora; while
users want the model to "follow their instructions
helpfully and safely" (Radford et al., 2019; Brown
et al., 2020a; Fedus et al., 2021; Rae et al., 2021;
Thoppilan et al., 2022)
To address this mismatch, instruction tuning
(IT) is proposed, serving as an effective technique
to enhance the capabilities and controllability
of large language models. It involves further
training LLMs using (INSTRUCTION, OUTPUT)
pairs, where INSTRUCTION denotes the human
instruction for the model, and OUTPUT denotes the
desired output that follows the INSTRUCTION. The
benefits of IT are threefold: (1) Finetuning an LLM
on the instruction dataset bridges the gap between
the next-word prediction objective of LLMs and
the users’ objective of instruction following; (2)
IT allows for a more controllable and predictable
model behavior compared to standard LLMs. The
instructions serve to constrain the model’s outputs
to align with the desired response characteristics
or domain knowledge, providing a channel for
humans to intervene with the model’s behaviors;
and (3) IT is computationally efficient and can help
LLMs rapidly adapt to a specific domain without
extensive retraining or architectural changes.
Despite its effectiveness, IT also poses
challenges: (1) Crafting high-quality instructions
that properly cover the desired target behaviors is
non-trivial: existing instruction datasets are usually
limited in quantity, diversity, and creativity; (2)
there has been an increasing concern that IT only
improves on tasks that are heavily supported in
the IT training dataset (Gudibande et al., 2023);
and (3) there has been an intense criticism that
IT only captures surface-level patterns and styles
arXiv:2308.10792v4 [cs.CL] 9 Oct 2023
剩余30页未读，继续阅读
资源评论
