ChatGPT背后的大模型最新有哪些？最新最全《Transformer预训练模型分类》论文，pdf.pdf

TRANSFORMER MODELS: AN INTRODUCTION AND CATALOG

Xavier Amatriain

Los Gatos, CA 95032

xavier@amatriain.net

February 17, 2023

ABSTRACT

In the past few years we have seen the meteoric appearance of dozens of models of the Transformer

family, all of which have funny, but not self-explanatory, names. The goal of this paper is to offer

a somewhat comprehensive but simple catalog and classification of the most popular Transformer

1.3 What are Transformers used for and why are they so popular . . . . . . . . . . . . . . . . . . . . . . 5

1.4 RLHF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.5 Diffusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 The Transformers catalog 8

2.1 Features of a Transformer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1.1 Pretraining Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1.2 Pretraining Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2.1.3 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.2 Catalog table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.3 Family Tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2.5.5 BERT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.5.6 Big Bird . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.5.7 BlenderBot3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.5.8 BLOOM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.5.9 ChatGPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.5.10 Chinchilla . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.5.11 CLIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.5.12 CM3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.5.19 DQ-BART . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.5.20 ELECTRA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.5.21 ERNIE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2.5.22 Flamingo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.5.23 Gato . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.5.24 GLaM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

2.5.25 GLIDE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.5.26 Global Context ViT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.5.27 Gopher . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.5.28 GopherCite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

2.5.35 GPT-NeoX-20B . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.5.36 HTML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.5.37 Imagen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

2.5.38 Jurassic-1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.5.39 LAMDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.5.40 mBART . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2.5.41 Megatron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.5.42 Minerva . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.5.43 MT-NLG (Megatron TouringNLG) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.5.44 OPT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.5.45 PalM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.5.46 Pegasus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.5.52 Switch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.5.53 T5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.5.54 Trajectory Transformers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.5.55 Transformer XL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

2.5.56 Turing-NLG . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.5.57 ViT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.5.58 Wu Dao 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.5.59 XLM-RoBERTa . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

2.5.60 XLNet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

3 Further reading 31

2

over the 2–3 years prior. Up until that point however, attention was just one of the mechanisms used by these models,

which were mostly based on LSTM (Long Short Term Memory)[

3

] and other RNN (Recurrent Neural Networks)[

4

]

variations. The key insight of the Transformers paper was that, as the title implies, attention could be used as the only

mechanism to derive dependencies between input and output.

It is beyond the scope of this blog to go into all the details of the Transformer architecture. For that, I will refer you

to the original paper above or to the wonderful The Illustrated Transformer

3

post. That being said, we will briefly

describe the most important aspects since we will be referring to them in the catalog below. Let’s start with the basic

architectural diagram from the original paper, and describe some of the components.

sublayer, which is another multi-head attention layer over the output of the Encoder. Besides, the other multi-head layer

in the decoder is masked to prevent attention to subsequent positions.

1

2

3

specific attention function called scaled dot-product attention. I will refer you again to the The Illustrated Transformer

4

post for many more details on how the attention mechanism works, but will reproduce the diagram from the original

paper in Figure 2 so you get the main idea

There are several advantages of attention layers over recurrent and convolutional networks, the two most important

being their lower computational complexity and their higher connectivity, especially useful for learning long-term

dependencies in sequences.

1.3 What are Transformers used for and why are they so popular

The original transformer was designed for language translation, particularly from English to German. But, already

adapt to other tasks, a.k.a. Transfer learning. Pretrained Transformer models can adapt extremely easily and quickly to

tasks they have not been trained on, and that has huge advantages. As an ML practitioner, you no longer need to train

a large model on a huge dataset. All you need to do is re-use the pretrained model on your task, maybe just slightly

4

5