大模型量化技术GPTQ

Published as a conference paper at ICLR 2023

GPTQ: ACCURATE POST-TRAINING QUANTIZATION

FOR GENERATIVE PRE-TRAINED TRANSFORMERS

Elias Frantar

∗

IST Austria

Saleh Ashkboos

ETH Zurich

Torsten Hoefler

ETH Zurich

Dan Alistarh

IST Austria & NeuralMagic

ABSTRACT

Generative Pre-trained Transformer models, known as GPT or OPT, set them-

selves apart through breakthrough performance across complex language mod-

we address this challenge, and propose GPTQ, a new one-shot weight quantiza-

tion method based on approximate second-order information, that is both highly-

accurate and highly-efficient. Specifically, GPTQ can quantize GPT models with

175 billion parameters in approximately four GPU hours, reducing the bitwidth

down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the

uncompressed baseline. Our method more than doubles the compression gains rel-

ative to previously-proposed one-shot quantization methods, preserving accuracy,

allowing us for the first time to execute an 175 billion-parameter model inside a

single GPU for generative inference. Moreover, we also show that our method

can still provide reasonable accuracy in the extreme quantization regime, in which

Pre-trained generative models from the Transformer (Vaswani et al., 2017) family, commonly known

as GPT or OPT (Radford et al., 2019; Brown et al., 2020; Zhang et al., 2022), have shown break-

through performance for complex language modelling tasks, leading to massive academic and prac-

tical interest. One major obstacle to their usability is computational and storage cost, which ranks

among the highest for known models. For instance, the best-performing model variants, e.g. GPT3-

175B, have in the order of 175 billion parameters and require tens-to-hundreds of GPU years to

train (Zhang et al., 2022). Even the simpler task of inferencing over a pre-trained model, which is

our focus in this paper, is highly challenging: for instance, the parameters of GPT3-175B occupy

326GB (counting in multiples of 1024) of memory when stored in a compact float16 format. This

exceeds the capacity of even the highest-end single GPUs, and thus inference must be performed

Nahshan et al., 2021), which compress the model in one shot, without retraining, would be very

appealing. Unfortunately, the more accurate variants of such methods (Li et al., 2021; Hubara et al.,

2021; Frantar et al., 2022) are complex and challenging to scale to billions of parameters (Yao et al.,

∗

Published as a conference paper at ICLR 2023

2022). To date, only basic variants of round-to-nearest quantization (Yao et al., 2022; Dettmers

et al., 2022) have been applied at the scale of GPT-175B; while this works well for low compression

targets, e.g., 8-bit weights, they fail to preserve accuracy at higher rates. It therefore remains open

a few hours, and precise enough to compress such models to 3 or 4 bits per parameter without

significant loss of accuracy. For illustration, GPTQ can quantize the largest publicly-available mod-

els, OPT-175B and BLOOM-176B, in approximately four GPU hours, with minimal increase in

perplexity, known to be a very stringent accuracy metric.

Further, we show that our model can also provide robust results in the extreme quantization regime,

in which models are quantized to 2 bits per component, or even ternary values. On the practical

side, we develop an execution harness which allows us to execute the resulting compressed models

efficiently for generative tasks. Specifically, we are able to run the compressed OPT-175B model

for the first time on a single NVIDIA A100 GPU, or using only two more cost-effective NVIDIA

A6000 GPUs. We also implement bespoke GPU kernels which are able to leverage compression for

This high degree of compression may appear natural, as these networks are overparametrized; yet,

as we discuss in our detailed analysis of results, compression induces non-trivial tradeoffs between

the accuracy of the language modeling (perplexity), bit-width, and the size of the original model.

We hope that our work will stimulate further research in this area, and can be a further step towards

making these models available to a wider audience. In terms of limitations, our method currently

does not provide speedups for the actual multiplications, due to the lack of hardware support for

mixed-precision operands (e.g. FP16 x INT4) on mainstream architectures. Moreover, our current

results do not include activation quantization, as they are not a significant bottleneck in our target

scenarios; however, this can be supported using orthogonal techniques (Yao et al., 2022).

2 RELATED WORK

1

Published as a conference paper at ICLR 2023

trained model using modest resources, typically a few thousand data samples and a few hours of

computation. Post-training approaches are particularly interesting for massive models, for which

full model training or even finetuning can be expensive. We focus on this scenario here.

layers. (See Section 3 for more details.) The AdaRound method (Nagel et al., 2020) computes a

data-dependent rounding by annealing a penalty term, which encourages weights to move towards

grid points corresponding to quantization levels. BitSplit (Wang et al., 2020) constructs quantized

values bit-by-bit using a squared error objective on the residual error, while AdaQuant (Hubara et al.,

2021) performs direct optimization based on straight-through estimates. BRECQ (Li et al., 2021)

introduces Fisher information into the objective, and optimizes layers within a single residual block

jointly. Finally, Optimal Brain Quantization (OBQ) (Frantar et al., 2022) generalizes the classic

Optimal Brain Surgeon (OBS) second-order weight pruning framework (Hassibi et al., 1993; Singh

& Alistarh, 2020; Frantar et al., 2021) to apply to quantization. OBQ quantizes weights one-by-one,

in order of quantization error, always adjusting the remaining weights. While these approaches can

et al., 2022)— carefully select quantization granularity, e.g., vector-wise, they ultimately just round

weights to the nearest (RTN) quantization level, in order to maintain acceptable runtimes for very

large models. ZeroQuant further proposes layer-wise knowledge distillation, similar to AdaQuant,

but the largest model it can apply this approach to has only 1.3 billion parameters. At this scale,

LLM.int8() observes that activation outliers in a few feature dimensions break the quantization

of larger models, and proposes to fix this problem by keeping those dimensions in higher preci-

sion. Lastly, nuQmm develops efficient GPU kernels for a specific binary-coding based quantization

scheme.

Relative to this line of work, we show that a significantly more complex and accurate quantizer can

the error incurred by quantizing a single weight. Since the corresponding objective is a quadratic,

Published as a conference paper at ICLR 2023

whose Hessian is H

the greedy-optimal weight to quantize next, which we denote by w

update of all weights in F , denoted by δ

applied recursively inside each block: the white

middle column is currently being quantized.

The original OBQ method quantizes rows of W

independently, in a specific order defined by the

corresponding errors. By contrast, we will aim

to quantize the weights of all rows in the same

order, and will show that this typically yields

results with a final squared error that is simi-

lar to the original solutions. As a consequence,

the set of unquantized weights F and similarly

the layer inputs X

F

, which are the same for all

rows, and not on any weights. Therefore, we

have to perform the update of H

−1

F

Equation (3) only d

times, once per column,

rather than d

times, once per weight. This

ditional major problems need to be addressed.

ratio. For example, Equation (3) needs to update all elements of a potentially huge matrix using just a