Rethinking the Role of Demonstrations:
What Makes In-Context Learning Work?
Sewon Min
1,2
Xinxi Lyu
1
Ari Holtzman
1
Mikel Artetxe
2
Mike Lewis
2
Hannaneh Hajishirzi
1,3
Luke Zettlemoyer
1,2
1
University of Washington
2
Meta AI
3
Allen Institute for AI
{sewon,alrope,ahai,hannaneh,lsz}@cs.washington.edu
{artetxe,mikelewis}@meta.com
Abstract
Large language models (LMs) are able to in-
context learn—perform a new task via infer-
ence alone by conditioning on a few input-
label pairs (demonstrations) and making pre-
dictions for new inputs. However, there has
been little understanding of how the model
learns and which aspects of the demonstra-
tions contribute to end task performance. In
this paper, we show that ground truth demon-
strations are in fact not required—randomly
replacing labels in the demonstrations barely
hurts performance on a range of classification
and multi-choce tasks, consistently over 12 dif-
ferent models including GPT-3. Instead, we
find that other aspects of the demonstrations
are the key drivers of end task performance, in-
cluding the fact that they provide a few exam-
ples of (1) the label space, (2) the distribution
of the input text, and (3) the overall format of
the sequence. Together, our analysis provides
a new way of understanding how and why
in-context learning works, while opening up
new questions about how much can be learned
from large language models through inference
alone.
1 Introduction
Large language models (LMs) have shown impres-
sive performance on downstream tasks by simply
conditioning on a few input-label pairs (demonstra-
tions); this type of inference has been referred to as
in-context learning (Brown et al., 2020). Despite in-
context learning consistently outperforming zero-
shot inference on a wide range of tasks (Zhao et al.,
2021; Liu et al., 2021), there is little understanding
of how it works and which aspects of the demon-
strations contribute to end task performance.
In this paper, we show that ground truth demon-
strations are in fact not required for effective in-
context learning (Section 4). Specifically, replac-
ing the labels in demonstrations with random labels
barely hurts performance in a range of classifica-
tion and multi-choice tasks (Figure 1). The result
Figure 1: Results in classification (top) and multi-
choice tasks (bottom), using three LMs with varying
size. Reported on six datasets on which GPT-3 is eval-
uated; the channel method is used. See Section 4 for
the full results. In-context learning performance drops
only marginally when labels in the demonstrations are
replaced by random labels.
is consistent over 12 different models including the
GPT-3 family (Radford et al., 2019; Min et al.,
2021b; Wang and Komatsuzaki, 2021; Artetxe
et al., 2021; Brown et al., 2020). This strongly
suggests, counter-intuitively, that the model does
not rely on the input-label mapping in the demon-
strations to perform the task.
Further analysis investigates which parts of
demonstrations actually do contribute to the perfor-
mance. We identify possible aspects of demonstra-
tions (e.g., the label space and the distribution of
the input text) and evaluate a series of variants of
the demonstrations to quantify the impact of each
(Section 5). We find that: (1) the label space and
the distribution of the input text specified by the
demonstrations are both key to in-context learn-
ing (regardless of whether the labels are correct
for individual inputs); (2) specifying the overall
format is also crucial, e.g., when the label space
is unknown, using random English words as la-
bels is significantly better than using no labels; and
arXiv:2202.12837v2 [cs.CL] 20 Oct 2022
