Open-Vocabulary Video Anomaly Detection
Peng Wu
1
, Xuerong Zhou
1
, Guansong Pang
2
*
, Yujia Sun
3
, Jing Liu
3
, Peng Wang
1∗
, Yanning Zhang
1
1
Northwestern Polytechnical University,
2
Singapore Management University,
3
Xidian University
{xdwupeng, zxr2333}@gmail.com, gspang@smu.edu.sg, yjsun@stu.xidian.edu.cn
neouma@163.com, {peng.wang, ynzhang}@nwpu.edu.cn
Abstract
Current video anomaly detection (VAD) approaches with
weak supervisions are inherently limited to a closed-set set-
ting and may struggle in open-world applications where
there can be anomaly categories in the test data unseen
during training. A few recent studies attempt to tackle a
more realistic setting, open-set VAD, which aims to de-
tect unseen anomalies given seen anomalies and normal
videos. However, such a setting focuses on predicting frame
anomaly scores, having no ability to recognize the specific
categories of anomalies, despite the fact that this ability is
essential for building more informed video surveillance sys-
tems. This paper takes a step further and explores open-
vocabulary video anomaly detection (OVVAD), in which we
aim to leverage pre-trained large models to detect and cate-
gorize seen and unseen anomalies. To this end, we propose
a model that decouples OVVAD into two mutually comple-
mentary tasks – class-agnostic detection and class-specific
classification – and jointly optimizes both tasks. Particu-
larly, we devise a semantic knowledge injection module to
introduce semantic knowledge from large language models
for the detection task, and design a novel anomaly synthesis
module to generate pseudo unseen anomaly videos with the
help of large vision generation models for the classification
task. These semantic knowledge and synthesis anomalies
substantially extend our model’s capability in detecting and
categorizing a variety of seen and unseen anomalies. Exten-
sive experiments on three widely-used benchmarks demon-
strate our model achieves state-of-the-art performance on
OVVAD task.
1. Introduction
Video anomaly detection (VAD), which aims at detecting
unusual events that do not conform to expected patterns,
has become a growing concern in both academia and indus-
try communities due to its promising application prospects
*
Corresponding Authors
in, such as, intelligent video surveillance and video content
review. Through several years of vigorous development,
VAD has made significant progress with many works con-
tinuously emerging.
Traditional VAD can be broadly classified into two
types based on the supervised mode, i.e., semi-supervised
VAD [17] and weakly supervised VAD [38]. The main dif-
ference between them lies in the availability of abnormal
training samples. Although they are different in terms of
supervised mode and model design, both can be roughly
regarded as classification tasks. In the case of semi-
supervised VAD, it falls under the category of one-class
classification, while weakly supervised VAD pertains to bi-
nary classification. Specifically, semi-supervised VAD as-
sumes that only normal samples are available during the
training stage, and the test samples which do not conform
to these normal training samples are identified as anomalies,
as shown in Fig. 1(a). Most existing methods essentially en-
deavor to learn the one-class pattern, i.e., normal pattern, by
means of one-class classifiers [50] or self-supervised learn-
ing technique, e.g. frame reconstruction [9], frame predic-
tion [17], jigsaw puzzles [44], etc. Similarly, as illustrated
in Fig. 1(b), weakly supervised VAD can be seen as a binary
classification task with the assumption that both normal and
abnormal samples are available during the training phase
but the precise temporal annotation of abnormal events are
unknown. Previous approaches widely adopt a binary clas-
sifier with the multiple instance learning (MIL) [38] or Top-
K mechanism [27] to discriminate between normal and ab-
normal events. In general, existing approaches of both
semi-supervised and weakly supervision VAD restricts their
focus to classification and use corresponding discriminator
to categorize each video frame. While these practices have
achieved significant success on several widely-used bench-
marks, they are limited to detecting a closed set of anomaly
categories and are unable to handle arbitrary unseen anoma-
lies. This limitation restricts their application in open-world
scenarios and poses a risk of increasing missing reports,
as many real-world anomalies in actual deployment are not
present in the training data.
This CVPR paper is the Open Access version, provided by the Computer Vision Foundation.
Except for this watermark, it is identical to the accepted version;
the final published version of the proceedings is available on IEEE Xplore.
18297
剩余10页未读，继续阅读
资源评论
