ScalingInstruction-FinetunedLanguageModels.pdf资源-CSDN文库

人工智能

需积分: 1 13 浏览量 2023-05-18 14:42:54 上传评论收藏 1.49MB PDF 举报

资源推荐

资源详情

资源评论

Scaling Instruction-Finetuned Language Models

Hyung Won Chung

∗

Le Hou

∗

Shayne Longpre

∗

Barret Zoph

†

Yi Tay

†

William Fedus

†

Yunxuan Li Xuezhi Wang Mostafa Dehghani Siddhartha Brahma

Albert Webson Shixiang Shane Gu Zhuyun Dai Mirac Suzgun Xinyun Chen

Aakanksha Chowdhery Alex Castro-Ros Marie Pellat Kevin Robinson

Dasha Valter Sharan Narang Gaurav Mishra Adams Yu Vincent Zhao

Yanping Huang Andrew Dai Hongkun Yu Slav Petrov Ed H. Chi

Jeﬀ Dean Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le

Jason Wei

∗

Google

Abstract

Finetuning language models on a collection of datasets phrased as instructions has been shown to improve

model performance and generalization to unseen tasks. In this paper we explore instruction ﬁnetuning

with a particular focus on (1) scaling the number of tasks, (2) scaling the model size, and (3) ﬁnetuning on

chain-of-thought data. We ﬁnd that instruction ﬁnetuning with the above aspects dramatically improves

performance on a variety of model classes (PaLM, T5, U-PaLM), prompting setups (zero-shot, few-shot, CoT),

and evaluation benchmarks (MMLU, BBH, TyDiQA, MGSM, open-ended generation, RealToxicityPrompts).

For instance, Flan-PaLM 540B instruction-ﬁnetuned on 1.8K tasks outperforms PaLM 540B by a large margin

(+9.4% on average). Flan-PaLM 540B achieves state-of-the-art performance on several benchmarks, such as

75.2% on ﬁve-shot MMLU. We also publicly release Flan-T5 checkpoints,

which achieve strong few-shot

performance even compared to much larger models, such as PaLM 62B. Overall, instruction ﬁnetuning is a

general method for improving the performance and usability of pretrained language models.

The cafeteria had 23 apples

originally. They used 20 to

make lunch. So they had 23 -

20 = 3. They bought 6 more

apples, so they have 3 + 6 = 9.

The cafeteria had 23 apples

originally. They used 20 to

make lunch. So they had 23 -

20 = 3. They bought 6 more

apples, so they have 3 + 6 = 9.

The cafeteria had 23 apples

originally. They used 20 to

make lunch. So they had 23 -

20 = 3. They bought 6 more

apples, so they have 3 + 6 = 9.

The cafeteria had 23 apples

originally. They used 20 to

make lunch. So they had 23 -

20 = 3. They bought 6 more

apples, so they have 3 + 6 = 9.

(B)

Language

model

Please answer the following question.

What is the boiling point of Nitrogen?

-320.4F

Answer the following question by

reasoning step-by-step.

The cafeteria had 23 apples. If they

used 20 for lunch and bought 6 more,

how many apples do they have?

The cafeteria had 23 apples

originally. They used 20 to

make lunch. So they had 23 -

20 = 3. They bought 6 more

apples, so they have 3 + 6 = 9.

Q: Can Georey Hinton have a

conversation with George Washington?

Give the rationale before answering.

Georey Hinton is a British-Canadian

computer scientist born in 1947. George

Washington died in 1799. Thus, they

could not have had a conversation

together. So the answer is “no”.

Instruction netuning

Chain-of-thought netuning

Inference: generalization to unseen tasks

Multi-task instruction netuning (1.8K tasks)

Figure 1: We ﬁnetune various language models on 1.8K tasks phrased as instructions, and evaluate them on unseen tasks.

We ﬁnetune both with and without exemplars (i.e., zero-shot and few-shot) and with and without chain-of-thought,

enabling generalization across a range of evaluation scenarios.

∗

Equal contribution. Correspondence: [email protected].

†

Core contributor.

Public checkpoints: https://github.com/google-research/t5x/blob/main/docs/models.md#flan-t5-checkpoints.

arXiv:2210.11416v5 [cs.LG] 6 Dec 2022

1 Introduction

An important goal of artiﬁcial intelligence is to develop models that can generalize to unseen tasks. In natural

language processing (NLP), pretrained language models have made signiﬁcant progress towards this goal,

as they can perform tasks given natural language descriptions (Brown et al., 2020, inter alia). Further progress

has been made by ﬁnetuning language models on a collection of tasks phrased as instructions, which enables

models to respond better to instructions and reduces the need for few-shot exemplars (Ouyang et al., 2022;

Wei et al., 2021; Sanh et al., 2021, inter alia).

In this paper we advance instruction ﬁnetuning in several ways. First, we study the impact of scaling

on instruction ﬁnetuning. Our experiments show that instruction ﬁnetuning does scale well with the

number of tasks and the size of the model. Their respective scaling behaviors suggest that future research

should scale up the number of tasks and the size of the model even further. Second, we study the eﬀect of

ﬁnetuning on the ability of the models to perform reasoning tasks. Our experiments show that whereas

prior instruction ﬁnetuning methods that do not include chain-of-thought (CoT; Wei et al., 2022b) severely

degrade performance on CoT evaluations, adding just nine CoT datasets into the ﬁnetuning mixture enables

better performance on all evaluations.

Based on these ﬁndings, we train Flan-PaLM by using a 540B-parameter model, increasing the number of

ﬁnetuning tasks to 1.8K, and including CoT data. Flan-PaLM outperforms PaLM, achieving new state-of-the-

art on several benchmarks. For instance, Flan-PaLM’s improved reasoning abilities enable it to leverage CoT

and self-consistency (Wang et al., 2022c) to achieve 75.2% on Massive Multi-task Language Understanding

(MMLU; Hendrycks et al., 2020). Flan-PaLM also has improved multilingual abilities compared to PaLM, such

as 14.9% absolute improvement on one-shot TyDiQA (Clark et al., 2020) and 8.1% on arithmetic reasoning

in under-represented languages (Shi et al., 2022). In human rater evaluations, Flan-PaLM substantially

outperforms PaLM on a challenging set of open-ended generation questions, suggesting improved usability.

Moreover, we found that instruction ﬁnetuning also improves performance across several responsible AI

evaluation benchmarks.

In addition, we also instruction-ﬁnetune Flan-T5 models (80M to 11B). These checkpoints have strong zero-

shot, few-shot, and CoT abilities, outperforming prior public checkpoints such as T5 (Raﬀel et al., 2020). For

example, Flan-T5 11B outperforms T5 11B by double-digit improvements and even outperforms PaLM 62B on

some challenging BIG-Bench tasks (Srivastava et al., 2022). Overall, our results underscore how instruction

ﬁnetuning can improve performance across a range of models, prompting setups, and evaluation tasks.

- Random 25.0

- Average human rater 34.5

May 2020 GPT-3 5-shot 43.9

Mar. 2022 Chinchilla 5-shot 67.6

Apr. 2022 PaLM 5-shot 69.3

Oct. 2022

Flan-PaLM 5-shot 72.2

Flan-PaLM 5-shot: CoT + SC 75.2

- Average human expert 89.8

Jun. 2023 forecast (Hypermind) 73.2

Jun. 2024 forecast (Hypermind) 75.0

Jun. 2023 forecast (Metaculus) 82.7

Jun. 2024 forecast (Metaculus) 87.6

Table 1: Average 5-shot MMLU scores (%) for 57 tasks with model and human accuracy comparisons

(Hendrycks et al., 2020). Forecasts were made in July 2022 by competitive human forecasters, regarding a sin-

gle model (Steinhardt, 2021); see

https://prod.hypermind.com/ngdp/en/showcase2/showcase.html?sc=

JSAI

and

https://www.metaculus.com/questions/11676/mmlu-sota-in-2023-2025/

. CoT + SC: chain-

of-thought prompting with self-consistency (Wang et al., 2022b).

T0-SF

Commonsense reasoning

Question generation

Closed-book QA

Adversarial QA

Extractive QA

Title/context generation

Topic classication

Struct-to-text

…

55 Datasets, 14 Categories,

193 Tasks

Mun

Natural language inference Closed-book QA

Code instruction gen. Conversational QA

Program synthesis Code repair

Dialog context generation …

69 Datasets, 27 Categories, 80 Tasks

CoT (Reasoning)

Arithmetic reasoning Explanation generation

Commonsense Reasoning Sentence composition

Implicit reasoning …

9 Datasets, 1 Category, 9 Tasks

Natural

Instructions v2

Cause eect classication

Commonsense reasoning

Named entity recognition

Toxic language detection

Question answering

Question generation

Program execution

Text categorization

…

372 Datasets, 108 Categories,

1554 Tasks

❖ A Dataset is an original data source (e.g. SQuAD).

❖ A Task Category is unique task setup (e.g. the SQuAD dataset is congurable for multiple task categories such as

extractive question answering, query generation, and context generation).

❖ A Task is a unique <dataset, task category> pair, with any number of templates which preserve the task category (e.g.

query generation on the SQuAD dataset.)

Finetuning tasks

Held-out tasks

MMLU

Abstract algebra Sociology

College medicine Philosophy

Professional law …

57 tasks

BBH

Boolean expressions Navigate

Tracking shued objects Word soing

Dyck languages …

27 tasks

TyDiQA

Information

seeking QA

8 languages

MGSM

Grade school

math problems

10 languages

Figure 2: Our ﬁnetuning data comprises 473 datasets, 146 task categories, and 1,836 total tasks. Details for

the tasks used in this paper is given in Appendix F.

2 Flan Finetuning

We instruction-ﬁnetune on a collection of data sources (Figure 2) with a variety of instruction template

types (Figure 3). We call this ﬁnetuning procedure Flan (Finetuning language models; Wei et al., 2021) and

prepend “Flan” to the resulting ﬁnetuned models (e.g., Flan-PaLM).

We show that Flan works across several

model sizes and architectures (Table 2).

2.1 Finetuning Data

Task mixtures.

Prior literature has shown that increasing the number of tasks in ﬁnetuning with instructions

improves generalization to unseen tasks (Wei et al., 2021; Sanh et al., 2021, inter alia). In this paper we scale

to 1,836 ﬁnetuning tasks by combining four mixtures from prior work: Muﬃn, T0-SF, NIV2, and CoT, as

summarized in Figure 2.

Muﬃn

(80 tasks)

comprises 62 tasks from Wei et al. (2021) and 26 new tasks that

we added in this work, including dialog data (Byrne et al., 2019; Anantha et al., 2021; Dai et al., 2022) and

program synthesis data (Yasunaga and Liang, 2020; Li et al., 2022).

T0-SF (193 tasks)

comprises tasks from

T0 (Sanh et al., 2021) that do not overlap with the data used in Muﬃn (SF stands for “sans Flan”).

NIV2

(1554 tasks) comprises tasks from Wang et al. (2022c).

We use “Flan” to refer to our ﬁnetuning procedure. “FLAN” is a model in Wei et al. (2021).

Multi-task finetuning with instructions.

We removed 44 tasks related to MMLU (Hendrycks et al., 2020), since MMLU is used for evaluation.

Answer the following

yes/no question.

Can you write a whole

Haiku in a single tweet?

Answer the following yes/no question

by reasoning step-by-step.

Can you write a whole Haiku in a

single tweet?

Q: Answer the following

yes/no question.

Could a dandelion suer

from hepatitis?

A: no

Q: Answer the following

yes/no question.

Can you write a whole Haiku

in a single tweet?

Q: Answer the following yes/no question by

reasoning step-by-step.

Could a dandelion suer from hepatitis?

A: Hepatitis only aects organisms with livers.

Dandelions don’t have a liver. The answer is no.

Q: Answer the following yes/no question by

reasoning step-by-step.

Can you write a whole Haiku in a single tweet?

A haiku is a japanese

three-line poem.

That is sho enough

to t in 280

characters. The

answer is yes.

A haiku is a japanese

three-line poem.

That is sho enough

to t in 280

characters. The

answer is yes.

yes

With chain-of-thoughtWithout chain-of-thought

Instruction

without

exemplars

Instruction

with exemplars

Figure 3: Combinations of ﬁnetuning data formats in this work. We ﬁnetune with and without exemplars,

and also with and without chain-of-thought. In addition, we have some data formats without instructions

but with few-shot exemplars only, like in Min et al. (2022) (not shown in the ﬁgure). Note that only nine

chain-of-thought (CoT) datasets use the CoT formats.

Chain-of-thought ﬁnetuning mixture.

The fourth ﬁnetuning data mixture (reasoning) involves CoT anno-

tations, which we use to explore whether ﬁnetuning on CoT annotations improves performance on unseen

reasoning tasks. We create a new mixture of nine datasets from prior work for which human raters manually

wrote CoT annotations for a training corpus. These nine datasets include tasks such as arithmetic reasoning

(Cobbe et al., 2021), multi-hop reasoning (Geva et al., 2021), and natural language inference (Camburu et al.,

2020). We manually compose ten instruction templates per task. A data card is in Appendix F.

Templates and formatting.

For Muﬃn, T0-SF, and NIV2, we use instructional templates for each task as

given by the creators of the mixtures. For CoT, we manually write around ten instruction templates for each of

the nine datasets. To create few-shot templates, we write a variety of exemplar delimiters (e.g., "Q:"/"A:") and

apply them randomly at the example level. An example of formatting for both with and without exemplars,

as well as with and without CoT, is shown in Figure 3.

2.2 Finetuning procedure

In this paper, we apply instruction ﬁnetuning across a broad range of model families, including T5 (Raﬀel

et al., 2020), PaLM (Chowdhery et al., 2022), and U-PaLM (Tay et al., 2022b). These model families span a

range of sizes, from Flan-T5-small (80M parameters), to PaLM and U-PaLM (540B parameters). For each

model, we apply the same training procedure, except for a few hyperparameters: learning rate, batch size,

dropout, and ﬁnetuning steps. We use a constant learning rate schedule and ﬁnetune using the Adafactor

optimizer (Shazeer and Stern, 2018). We use packing (Raﬀel et al., 2020) to combine multiple training

examples into a single sequence, separating inputs from targets using an end-of-sequence token. Masking is

applied to prevent the tokens from attending to others across the packed example boundary. The number

of ﬁnetuning steps, learning rate, batch size, and dropout for each model are given in Appendix E. For

each model, we use a single checkpoint for all evaluations; the optimal step was chosen based on periodic

evaluations (every 2k to 10k steps depending the model size) of the held-out tasks, and we used the same

number of checkpoint steps across all ablation runs for a given model. Notably, the amount of compute used

for ﬁnetuning is only a small fraction relative to the training compute, as shown in Table 2. For example, we

only use 0.2% of the pre-training compute to instruction-ﬁnetune Flan-PaLM 540B (approximately 512 v4

TPU chips for 37 hours). We use the JAX-based T5X framework (Bradbury et al., 2018; Roberts et al., 2022).

Params Model

Architecture

Pre-training

Objective

Pre-train

FLOPs

Finetune

FLOPs

% Finetune

Compute

80M Flan-T5-Small encoder-decoder span corruption 1.8E+20 2.9E+18 1.6%

250M Flan-T5-Base encoder-decoder span corruption 6.6E+20 9.1E+18 1.4%

780M Flan-T5-Large encoder-decoder span corruption 2.3E+21 2.4E+19 1.1%

3B Flan-T5-XL encoder-decoder span corruption 9.0E+21 5.6E+19 0.6%

11B Flan-T5-XXL encoder-decoder span corruption 3.3E+22 7.6E+19 0.2%

8B Flan-PaLM decoder-only causal LM 3.7E+22 1.6E+20 0.4%

62B Flan-PaLM decoder-only causal LM 2.9E+23 1.2E+21 0.4%

540B Flan-PaLM decoder-only causal LM 2.5E+24 5.6E+21 0.2%

62B Flan-cont-PaLM decoder-only causal LM 4.8E+23 1.8E+21 0.4%

540B Flan-U-PaLM decoder-only preﬁx LM + span corruption 2.5E+23 5.6E+21 0.2%

Table 2: Across several models, instruction ﬁnetuning only costs a small amount of compute relative to

pre-training. T5: Raﬀel et al. (2020). PaLM and cont-PaLM (also known as PaLM 62B at 1.3T tokens):

Chowdhery et al. (2022). U-PaLM: Tay et al. (2022b).

2.3 Evaluation protocol

Evaluation benchmarks

. We focus on performance on held-out tasks which were not included as part of the

ﬁnetuning data. We are interested in Flan-PaLM’s overall capabilities on world knowledge and reasoning

tasks. Thus, we evaluate the model on a range of diﬀerent benchmarks, including multilingual ones. We

do not use the evaluation set from Brown et al. (2020) since almost all of those tasks have training sets that

are included in our ﬁnetuning mixture. Instead, we use the following challenging benchmarks, for which

current language models still perform well below expert human raters.

(1) MMLU

(Hendrycks et al., 2020)

includes exam questions from 57 tasks such as mathematics, history, law, and medicine.

(2) BBH

includes

23 challenging tasks from BIG-Bench (Srivastava et al., 2022) for which PaLM performs below an average

human rater (Suzgun et al., 2022).

(3) TyDiQA

(Clark et al., 2020) is a question-answering benchmark

across 8 typologically diverse languages.

(4) MGSM

(Shi et al., 2022) is a multilingual benchmark of math

word problems from Cobbe et al. (2021) manually translated into 10 languages. These benchmarks were also

used in the PaLM paper (Chowdhery et al., 2022), which did not ﬁnd any meaningful data contamination

with pre-training data, consistent with data contamination analyses in previous work (Brown et al., 2020;

Wei et al., 2021; Du et al., 2022). Responsible AI evaluations are discussed in Appendix C.

Evaluation methods and metrics.

For MMLU and BBH, we evaluate both the ability to directly predict the

answer via direct prompting, where the model directly gives the answer (Brown et al., 2020; Srivastava et al.,

2022), as well as via chain-of-thought (CoT) prompting, where the model must provide a reasoning chain

before giving the ﬁnal answer (Wei et al., 2022b). For TyDiQA, we only measure direct prompting exact-match

score, since highlighting the portion of a passage with the correct answer may not require sophisticated

reasoning. For MGSM, we only measure CoT prompting accuracy since direct prompting has very low

performance. For all benchmarks we use the given few-shot exemplars, with the number of exemplars

following prior work: ﬁve-shot for MMLU, three-shot for BBH, one-shot for TyDiQA, and 8-shot for MGSM.

For a given model we also report a single “normalized average” metric, following the “normalized preferred

metric” in BIG-Bench (Srivastava et al., 2022).

Our normalized average metric is the macro-average over

six normalized scores: MMLU-Direct, MMLU-CoT, BBH-Direct, BBH-CoT, TyDiQA-Direct, and MGSM-CoT.

Results for all tasks in each benchmark are reported in Appendix D. Some Responsible AI benchmarks use

additional methods for generative tasks described in Appendix C.

A normalized metric scales an evaluation number with respect to a task-speciﬁc lower bound such as random guessing baseline for

a multiple choice question. For example, if random guessing produces 50% accuracy and the max accuracy of 100%, then a raw accuracy

of 55% would be be normalized to 10%, and a raw accuracy of 45% would be normalized to -10% since it is worse than random.

剩余53页未读，继续阅读

评论收藏

内容反馈

Java老徐

粉丝: 1514
资源: 2045

Scaling Instruction-Finetuned Language Models.pdf

最新资源

Scaling Instruction-Finetuned Language Models.pdf

Language Models as Knowledge Bases.pdf

Fine-Tuning Language Models from Human Preferences.pdf

Language Models for Dialog Applications.pdf

Evaluating Large Language Models Trained on Code.pdf

aws-java-sdk-autoscaling-1.11.277.jar

Python库 | aws-cdk.aws-autoscaling-common-1.32.1.tar.gz

Python库 | mypy-boto3-autoscaling-plans-1.17.0.0.tar.gz

PyPI 官网下载 | aws-cdk.aws-autoscaling-common-0.30.0.tar.gz

Large Language Models are Zero-Shot Reasoners.pdf

Safety Assessment of Chinese Large Language Models.pdf

Multimodal Chain-of-Thought Reasoning in Language Models.pdf

LLaMA: Open and Efficient Foundation Language Models.pdf

Finetuned Language Models are Zero-Shot Learners.pdf

PyPI 官网下载 | aws-cdk.aws-autoscaling-common-1.30.0.tar.gz

PyPI 官网下载 | mypy-boto3-autoscaling-plans-1.18.40.tar.gz

PyPI 官网下载 | mypy-boto3-autoscaling-plans-1.15.12.1.tar.gz

PyPI 官网下载 | mypy-boto3-autoscaling-plans-1.19.6.tar.gz

PyPI 官网下载 | mypy-boto3-autoscaling-plans-1.14.42.0.tar.gz

Language Models are Multilingual Chain-of-Thought Reasoners.pdf

2005.14165.pdf Language Models are Few-shot Learners

gpt3-Language-Models-are-Few-Shot-Learners.pdf

Mapping Language Models to Grounded Conceptual Spaces.pdf

Language Models：Past, Present, and Future.pdf

PyPI 官网下载 | mypy-boto3-autoscaling-plans-1.17.25.0.tar.gz

PyPI 官网下载 | mypy-boto3-autoscaling-plans-1.14.55.1.tar.gz

PyPI 官网下载 | mypy-boto3-autoscaling-plans-1.10.39.0.tar.gz

Python库 | aws-cdk.aws-autoscaling-hooktargets-1.16.3.tar.gz

Python库 | aws-cdk.aws-autoscaling-hooktargets-1.32.0.tar.gz

最新资源