3.2 Autoregressive models . . . . . 28
3.3 Gradient descent . . . . . . . . 30
3.4 Backpropagation . . . . . . . . 34
3.5 Training protocols . . . . . . . 38
3.6 Training data . . . . . . . . . . 41
II Deep models 43
4 Model components 44
4.1 The notion of layer . . . . . . . 45
4.2 Linear layers . . . . . . . . . . . 47
4.3 Activation functions . . . . . . 56
4.4 Pooling . . . . . . . . . . . . . . 59
4.5 Dropout . . . . . . . . . . . . . 62
4.6 Normalizing layers . . . . . . . 64
4.7 Skip connections . . . . . . . . 68
4.8 Attention layers . . . . . . . . . 71
4.9 Token embedding . . . . . . . . 78
4.10 Positional encoding . . . . . . . 79
5 Architectures 81
5.1 Multi-Layer Perceptrons . . . . 82
5.2 Convolutional networks . . . . 84
5.3 Attention models . . . . . . . . 91
4 140
