Under review as a conference paper at ICLR 2024
CURIOSITY-DRIVEN RED-TEAMING FOR LARGE LAN-
GUAGE MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
Large language models (LLMs) hold great potential for various natural language
applications but risk generating incorrect or toxic content. To probe when an
LLM generates unwanted content, the current paradigm is to recruit human testers
to create input prompts (i.e., test cases) designed to elicit unfavorable responses
from LLMs. This procedure is called red teaming. However, relying solely on
human testers can be both expensive and time-consuming. Recent works automate
red teaming by training LLMs (i.e., red team LLMs) with reinforcement learning
(RL) to maximize the chance of eliciting undesirable responses (i.e., successful
test cases) from the target LLMs being evaluated. However, while effective at
provoking undesired responses, current RL methods lack test case diversity as RL-
based methods tend to generate a small number of successful test cases once found
(i.e., high-precision but low diversity). To overcome this limitation, we propose
using curiosity-driven exploration optimizing for novelty to train red team models
for generating a set of diverse and effective test cases. We evaluate our method by
performing red teaming against LLMs in text continuation and instruction following
tasks. Our experiments show that curiosity-driven exploration achieves greater
diversity in all the experiments compared to existing RL-based red team methods
while maintaining effectiveness. Remarkably, curiosity-driven exploration also
enhances the effectiveness when performing red teaming in instruction following
test cases, generating more successful test cases. Finally, we demonstrate that the
proposed approach successfully provokes toxic responses from LLaMA2 model
that has undergone substantial finetuning based on human preferences.
WARNING: This paper contains model outputs which are offensive in nature.
1 INTRODUCTION
Large language models (LLM) have achieved unprecedented success in question-answering, virtual
assistance, summarization, and other applications of natural language processing (NLP). A big issue
in deploying LLMs is the potential generation of misinformation and vulnerable content (Lee, 2016).
However, since LLMs often consist of several millions or billions of parameters, inspecting what
prompts trigger an LLM to produce unwanted text (e.g., toxic, hateful, or untruthful) is challenging.
Ideally, before an LLM is deployed, it should be tested to ensure it cannot be prompted to produce an
undesired response. The current paradigm (Ganguli et al., 2022) for testing models relies on human
testers to design test cases (i.e., prompts) that elicit unwanted responses from the target LLM. This
paradigm is called red teaming, and the human testers are called red teams. Since relying solely
on human red teams is costly and time-consuming, a promising alternative is to automate test case
generation using another LLM (Perez et al., 2022).
The idea is to train a red-team LLM (Perez et al., 2022) (which is a different model than the target
LLM) using reinforcement learning (RL) (Sutton & Barto, 2018). Assuming access to a function
that can score how unwanted a particular piece of text is, RL treats the red-team LLM as a policy to
generate test cases that maximize the likelihood of unwanted responses generated from the target
LLM. Existing RL-based methods (Perez et al., 2022) can create effective test cases that provoke
undesired responses from the target LLM. However, these test cases lack diversity, resulting in low
coverage of the span of prompts that can result in undesirable responses. Insufficient coverage implies
1
剩余25页未读，继续阅读
资源评论
