没有合适的资源?快使用搜索试试~ 我知道了~
零镜头学习旨在通过学习视觉语义嵌入空间来对看不见的图像类别进行分类。 在大多数情况下,传统方法采用分离的两步流水线,该流水线从预先训练的CNN模型中提取图像特征。 然后利用固定的图像特征来学习嵌入空间。 这导致缺乏针对零镜头学习任务的图像特征的特定结构语义信息。 在本文中,我们提出了一种端到端可训练的深度语义结构约束模型来解决此问题。 所提出的模型包含图像特征结构约束和语义嵌入结构约束,目的是学习保留结构的图像特征并赋予学习的嵌入空间以更强的泛化能力。 借助语义结构信息,该模型获得了更多的零散学习辅助线索。 最新的性能证明了我们提出的方法的有效性。
资源推荐
资源详情
资源评论
![zip](https://img-home.csdnimg.cn/images/20210720083736.png)
![rar](https://img-home.csdnimg.cn/images/20210720083606.png)
![](https://csdnimg.cn/release/download_crawler_static/15761210/bg1.jpg)
Deep Semantic Structural Constraints for Zero-Shot Learning
Yan Li
∗ 1,2
, Zhen Jia
∗ 1,2
, Junge Zhang
1,2
, Kaiqi Huang
1,2,3
, Tieniu Tan
1,2,3
1
CRIPAC & NLPR, Institute of Automation, Chinese Academy of Sciences
2
University of Chinese Academy of Sciences
3
CAS Center for Excellence in Brain Science and Intelligence Technology
yan.li@cripac.ia.ac.cn, {zhen.jia, jgzhang, kqhuang, tnt}@nlpr.ia.ac.cn
Abstract
Zero-shot learning aims to classify unseen image categories
by learning a visual-semantic embedding space. In most
cases, the traditional methods adopt a separated two-step
pipeline that extracts image features from pre-trained CNN
models. Then the fixed image features are utilized to learn
the embedding space. It leads to the lack of specific structural
semantic information of image features for zero-shot learn-
ing task. In this paper, we propose an end-to-end trainable
Deep Semantic Structural Constraints model to address this
issue. The proposed model contains the Image Feature Struc-
ture constraint and the Semantic Embedding Structure con-
straint, which aim to learn structure-preserving image fea-
tures and endue the learned embedding space with stronger
generalization ability respectively. With the assistance of se-
mantic structural information, the model gains more auxiliary
clues for zero-shot learning. The state-of-the-art performance
certifies the effectiveness of our proposed method.
Introduction
As one of the most basic problems in the computer vision
area, image classification methods gain huge progress in re-
cent years with the impressive development of deep learn-
ing. Although ResNet (He et al. 2016), an outstanding repre-
sentation of the Convolutional Neural Network (CNN) clas-
sification models, gets the top-5 error rate as low as 3.57%
on ImageNet classification task, its classification ability is
still limited to the image categories in the training dataset.
The limitation that models can only classify image cate-
gories within the training set, restricts them to become more
intelligent as human beings. For a simple example, human
beings are able to classify different kinds of animals by just
reading their descriptions rather than seeing them. More and
more researchers try to break through this limitation by in-
troduce Zero-Shot Learning (ZSL) into image classification
(Lampert, Nickisch, and Harmeling 2009; Frome et al. 2013;
Norouzi et al. 2013; Socher et al. 2013; Fu et al. 2015;
Akata et al. 2015; Romera-Paredes and Torr 2015; Bucher,
Herbin, and Jurie 2016; Akata et al. 2016; Huang, Loy,
and Tang 2016; Changpinyo et al. 2016; Xian et al. 2017;
Morgado and Vasconcelos 2017).
∗
The first two authors contributed equally to this work.
Copyright
c
2018, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
Zero-shot learning seeks to make image classification
models able to classify image categories which never ap-
pear in the training dataset. In the zero-shot learning task,
we refer to the image categories in the training set as seen
classes and those in the test set as unseen classes. The
category characteristic of unseen classes are learned from
the side information, i.e., the semantic features of the im-
ages. The commonly used side information could be the hu-
man annotated attribute features of images (Lampert, Nick-
isch, and Harmeling 2009; Akata et al. 2016), the text
descriptions of the image categories (Reed et al. 2016),
word vectors of the category labels (Frome et al. 2013;
Norouzi et al. 2013) and so on.
A large number of previous state-of-the-art methods focus
on building a common space where image features and se-
mantic features are embedded (Frome et al. 2013; Socher et
al. 2013; Akata et al. 2015; Romera-Paredes and Torr 2015;
Akata et al. 2016). The embedding space is built on the
correspondence between the seen images and their seman-
tic features. Then in the test stage, unseen image features
will be mapped to the embedding space where the classifi-
cation method, such as nearest neighbour (NN) search, can
be operated easily. Most of these methods adopt a separated
two-step pipeline, i.e., extracting image features from pre-
trained CNN models and using fixed image features to learn
the embedding space.
However, we argue that separating the image feature ex-
traction and the embedding space construction harms the
ZSL models severely. The separation leads to the result
that models cannot regulate the image features for the spe-
cific ZSL task during training. What’s more, the image fea-
tures extracted from a fixed pre-trained CNN model will not
capture the plentiful semantic information in the side in-
formation. The semantic information of human annotated
attributes, text descriptions or word vectors constructs the
semantic structure of a specific category. We believe that
combining the learning of image features and embedding
space in an end-to-end manner, meanwhile, incorporating
the structural information into the whole learning process
would contribute to much better zero-shot performance.
In this paper, we come up with a new Deep Semantic
Structural Constraints (DSSC) model for zero-shot learning
looking forward to training the model in an end-to-end style
and using the semantic structural information to supervise
![](https://csdnimg.cn/release/download_crawler_static/15761210/bg2.jpg)
Input Image
CNN Model
Image
Feature
x
(x)
W
W (x)
T
Projected Image Feature
in the Semantic Space
softmax
L
Semantic space
deer
cow
fox
giraffe
antelope
horse
Projection
Matrix
IFSc
L
SESc
L
Figure 1: The framework of Deep Semantic Structural Constraints (DSSC) model. The Image Feature Structure constraint
(IFSc) and the Semantic Embedding Structure constraint (SESc) use the structural information of semantic space to supervise
the model’s image feature extraction and embedding space construction respectively.
the image feature extraction and embedding space construc-
tion. The contributions of this work are as follows:
• Distinguished from most of the current work, the DSSC
model is end-to-end trainable. Benefit from the end-to-
end training, the image feature extraction can be brought
into the training stage. We can add specific supervision to
train the model and make the features more suitable for
the ZSL task.
• In DSSC model, we set the Image Feature Structure con-
straint (IFSc) specially in the image feature extraction
stage. The constraint makes the model automatically learn
structure-preserving image representations. The learned
image features rather than the fixed features extracted
from pre-trained CNN models are beneficial to the con-
struction of semantic embedding space. To the best of our
knowledge, this is the first attempt to add specific image
feature extraction constraint in ZSL models.
• For the construction of embedding space, the DSSC
model contains the Semantic Embedding Structure con-
straint (SESc) to learn a better embedding space for the
unseen classes by relaxing the commonly used strict soft-
max loss. The embedding space directly learned on seen
classes will not generalize the unseen classes well due to
the domain shift problem (Fu et al. 2015). The semantic
embedding structure constraint can endue the model with
much better generalization capability for the ZSL task.
• As shown by the experiments, the DSSC model gets the
state-of-the-art performance on the representative datasets
of zero-shot learning.
Related Work
Zero-shot learning aims to classify images of unseen classes
which is an impossible challenge for traditional image clas-
sification models.
In recent years, researchers try to build a common embed-
ding space for image and their semantic features by learning
a compatibility function between them. The DeViSE model
(Frome et al. 2013) and the ALE model (Akata et al. 2016)
learn linear transformation matrices by solving a hinge rank-
ing loss function. R. Socher et al. operate a two-layer neu-
ral network to map unseen images to the semantic feature
space in (Socher et al. 2013). The ESZSL model (Romera-
Paredes and Torr 2015) adopts a principled designed Frobe-
nius norm regularizer making itself simple but efficient. The
SJE model (Akata et al. 2015) combines several compatibil-
ity function linearly to form a joint embedding which cap-
tures the non-redundant information from different aspects
of side information. Most recently, the SAE model (Kodirov,
Xiang, and Gong 2017) draws successful lessons from linear
auto-encoder. Besides the encoder mapping images to the se-
mantic space, it employs a decoder to make the model able
to reconstruct the original visual feature from the mapped
semantic feature. Similar to DeViSE and ALE, N. Karessli
et al. optimize a hinge ranking loss in (Karessli et al. 2017),
but they introduce the gaze information as side information
to the ZSL task innovatively.
In addition to learning the compatibility function directly,
hybrid models (Norouzi et al. 2013; Changpinyo et al. 2016)
bring us another way to think about zero-shot learning. The
basic idea of hybrid models is to use the composition of seen
classes to classify the unseen images. The ConSE model
(Norouzi et al. 2013) convexly combines the image classi-
fication probabilities of seen classes to determine the un-
seen classification result. The SynC model (Changpinyo et
al. 2016) synthesizes the classifiers of agent classes as the
classifier of the unseen class. Almost all these methods sep-
arate the image feature extraction and embedding space con-
struction. They use fixed image features when building the
embedding space. This trend leads to the image features’
lack of the semantic information in ZSL task.
Similar to our work, the SCoRe model (Morgado and
Vasconcelos 2017) also puts forward an end-to-end training
pipeline for ZSL. The authors add two semantic constraints
at the end of a CNN classification model to supervise the
attribute prediction and the category classification respec-
tively. Although the SCoRe model is end-to-end trainable
and deployed with specific constraints for ZSL task, it does
not consider any semantic structural constraints in the image
剩余7页未读,继续阅读
资源评论
![avatar-default](https://csdnimg.cn/release/downloadcmsfe/public/img/lazyLogo2.1882d7f4.png)
![avatar](https://profile-avatar.csdnimg.cn/default.jpg!1)
weixin_38553478
- 粉丝: 7
- 资源: 924
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![voice](https://csdnimg.cn/release/downloadcmsfe/public/img/voice.245cc511.png)
![center-task](https://csdnimg.cn/release/downloadcmsfe/public/img/center-task.c2eda91a.png)
最新资源
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
![feedback](https://img-home.csdnimg.cn/images/20220527035711.png)
![feedback](https://img-home.csdnimg.cn/images/20220527035711.png)
![feedback-tip](https://img-home.csdnimg.cn/images/20220527035111.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![dialog-icon](https://csdnimg.cn/release/downloadcmsfe/public/img/green-success.6a4acb44.png)