【免费】深度学习图像检索(CBIR):十年之大综述_图像检索技术综述资源-CSDN文库

图像检索

综述论文

5星 · 超过95%的资源需积分: 0 156 浏览量 2020-12-05 17:46:53 上传评论 6 收藏 2.46MB PDF 举报

资源推荐

资源详情

资源评论

A DECADE SURVEY OF CONTENT BASED IMAGERETRIEVAL USING DEEP LEARNING 1

A Decade Survey of Content Based Image

Retrieval using Deep Learning

Shiv Ram Dubey

Abstract—The content based image retrieval aims to ﬁnd the similar images from a large scale dataset against a query image.

Generally, the similarity between the representative features of the query image and dataset images is used to rank the images for

retrieval. In early days, various hand designed feature descriptors have been investigated based on the visual cues such as color,

texture, shape, etc. that represent the images. However, the deep learning has emerged as a dominating alternative of hand-designed

feature engineering from a decade. It learns the features automatically from the data. This paper presents a comprehensive survey of

deep learning based developments in the past decade for content based image retrieval. The categorization of existing state-of-the-art

methods from different perspectives is also performed for greater understanding of the progress. The taxonomy used in this survey

covers different supervision, different networks, different descriptor type and different retrieval type. A performance analysis is also

performed using the state-of-the-art methods. The insights are also presented for the beneﬁt of the researchers to observe the

progress and to make the best choices. The survey presented in this paper will help in further research progress in image retrieval

using deep learning.

Index Terms—Content Based Image Retrieval; Deep Learning; Convolutional Neural Networks; Survey; Supervised and

Unsupervised Learning.

1 INTRODUCTION

MAGE retrieval is a well studied problem of image matching

where the similar images are retrieved from a database w.r.t. a

given query image [1], [2]. Basically, the similarity between the

query image and the database images is used to rank the database

images in decreasing order of similarity [3]. Thus, the performance

of any image retrieval method depends upon the similarity com-

putation between images. Ideally, the similarity score computation

method between two images should be discriminative, robust and

efﬁcient. The easiest way to compute the similarity between two

images is to ﬁnd the sum of absolute difference of corresponding

pixels in both the images, i.e., L

distance. This method is also

referred as the template matching. However, this approach is not

robust against the image geometric and photometric changes,

such as translation, rotation, viewpoint, illumination, etc. It is

demonstrated in Fig. 1 with the help of two pictures of the same

category of Corel dataset [4] and corresponding representative

intensity values of a window. Another problem with this approach

is that it is not efﬁcient due to the high dimensionality of the

image which leads to the high computation requirement to ﬁnd

the similarity between the query and database images.

1.1 Hand-crafted Descriptor based Image Retrieval

In order to make the retrieval robust to geometric and photometric

changes, the similarity between images is computed based on the

content of images. Basically, the content of the images (i.e., the

visual appearance) in terms of the color, texture, shape, gradient,

etc. are represented in the form of a feature descriptor [6].

The similarity between the feature vectors of the corresponding

images is treated as the similarity between the images. Thus, the

S.R. Dubey is with the Computer Vision Group, Indian Institute of Informa-

tion Technology, Sri City, Chittoor, Andhra Pradesh-517646, India (e-mail:

shivram1987@gmail.com, srdubey@iiits.in).

Fig. 1: Comparing pixels of two regions (images are taken from

Corel-database [4]). The presented raw intensity values are only

indicative not actual. The uses of raw intensity values for the

image similarity computation is not a good idea as it is not robust

against the geometric and photometric changes. This ﬁgure has

been originally appeared in [5].

performance of any content based image retrieval (CBIR) method

heavily depends upon the feature descriptor representation of the

image. Any feature descriptor representation method is expected to

have the discriminating ability, robustness and low dimensionality.

Fig. 2 illustrates the effect of descriptor function in terms of

its robustness. The rotation and scale hybrid descriptor (RSHD)

function [7] is used to show the rotation invariance between an

image taken from Corel-dataset [4] and its rotated version. It can

be seen in Fig. 2 that the raw intensity values based comparison

does not work, however the descriptor based comparison works

given that the descriptor function is able to capture the relevant

information from the image. Various feature descriptor represen-

tation methods have been investigated to compute the similarity

between the two images for content based image retrieval. The

feature descriptor representation utilizes the visual cues of the

images selected manually based on the need [8], [9], [10], [11],

[12], [13], [14], [15], [16], [17], [18], [19]. These approaches

are also termed as the hand-designed or hand-engineered feature

description. Moreover, generally these methods are unsupervised

as they do not need the data to design the feature representation

method. Various survey has been also conducted time to time to

present the progress in content based image retrieval, including

arXiv:2012.00641v1 [cs.CV] 23 Nov 2020

A DECADE SURVEY OF CONTENT BASED IMAGERETRIEVAL USING DEEP LEARNING 2

Fig. 2: Depicting rotation robustness of descriptor function. The

image is taken from Corel-database [4]. The 1

image is 90

rotated version of 2

image in counter-clockwise direction.

The rotation and scale invariant hybrid descriptor (RSHD) [7]

is used as the descriptor function in this example. In spite of

having the differences in the intensity values at the corresponding

pixels between both images, the feature descriptors are very much

similar. This ﬁgure has been originally appeared in [5].

Fig. 3: The pipeline of state-of-the-art feature representation is

replaced by the CNN based feature representation with increased

discriminative ability and robustness.

[2] in 2000, [20] in 2002, [21] in 2004, [22] in 2006, [23] in

2007, [24] in 2008, [25] in 2014 and [26] in 2017. The hand-

engineering feature for image retrieval was a very active research

area. However, its performance was limited as the hand-engineered

features are not able to represent the image characteristics in an

accurate manner.

1.2 Distance Metric Learning based Image Retrieval

The distance metric learning has been also used very extensively

for feature vectors representation [27]. It is also explored well

for image retrieval [28]. Some notable deep metric learning based

image retrieval approaches include Contextual constraints distance

metric learning [29], Kernel-based distance metric learning [30],

Visuality-preserving distance metric learning [31], Rank-based

distance metric learning [32], Semi-supervised distance metric

learning [33], etc. Generally, the deep metric learning based

approaches have shown the promising retrieval performance com-

pared to hand-crafted approaches. However, most of the existing

deep metric learning based methods rely on the linear distance

functions which limits its discriminative ability and robustness to

represent the non-linear data for image retrieval. Moreover, it is

also not able to handle the multi-modal retrieval effectively.

1.3 Deep Learning based Image Retrieval

From a decade, a shift has been observed in feature representation

from hand-engineering to learning-based after the emergence of

deep learning [34], [35]. This transition is depicted in Fig. 3 where

the convoltional neural networks based feature learning replaces

Fig. 4: Taxonomy used in this survey to categorize the existing

deep learning based image retrieval approaches.

the state-of-the-art pipeline of traditional hand-engineered feature

representation. The deep learning is a hierarchical feature repre-

sentation technique to learn the abstract features from data which

are important for that dataset and application [36]. Based on the

type of data to be processed, different architectures came into

existence such as Artiﬁcial Neural Network (ANN)/ Multilayer

Perceptron (MLP) for 1-D data [37], [38], [39], Convolutional

Neural Networks (CNN) for image data [40], [41], [42], and

Reurrent Neural Networks (RNN) for time-series data [43], [44],

[45]. The CNN features off-the-shelf have shown very promising

performance for the object recognition and retrieval tasks in terms

of the discriminative power and robustness [34]. A huge progress

has been made in this decade to utilize the power of deep learning

for content based image retrieval [46], [47], [48], [49]. Thus,

this survey mainly focuses over the progress in state-of-the-art

deep learning based models and features for content based image

retrieval from its inception. A taxonomy of state-of-the-art deep

learning approaches for image retrieval is portrayed in Fig. 4.

The major contributions of this survey, w.r.t. the existing

literature, can be outlined as follows:

1) As per my best knowledge, this survey can be seen as the

ﬁrst of its kind to cover the deep learning based image

retrieval approaches very comprehensively in terms of

evolution of image retrieval using deep learning, different

supervision type, network type, descriptor type, retrieval

type and other aspects.

2) In contrast to the recent reviews [47], [28], [48], this

survey speciﬁcally covers the progress in image retrieval

using deep learning progress in 2011-2020 decade rather

than hand-crafted and distance metric learning based

approaches. Moreover, we provide a very informative tax-

onomy (refer Fig. 4) with wide coverage of existing deep

learning based image retrieval approaches as compared to

the recent survey [49].

3) This survey enriches the reader with the state-of-the-art

image retrieval using deep learning methods with analysis

from various perspectives.

4) This paper also presents the brief highlights and impor-

tant discussions along with the comprehensive compar-

isons on benchmark datasets using the state-of-the-art

deep learning based image retrieval approaches (Refer

Table 3, 4, and 5).

A DECADE SURVEY OF CONTENT BASED IMAGERETRIEVAL USING DEEP LEARNING 3

TABLE 1: The summary of large-scale datasets for deep learning

based image retrieval.

Dataset Year #Classes Training Test Image Type

CIFAR-10 [50] 2009 10 50,000 10,000 Object Category Images

NUS-WIDE [51] 2009 21 97,214 65,075 Scene Images

MNIST [52] 1998 10 60,000 10,000 Handwritten Digit Images

SVHN [53] 2011 10 73257 26032 House Number Images

SUN397 [54] 2010 397 100,754 8,000 Scene Images

UT-ZAP50K [55] 2014 8 42,025 8,000 Shoes Images

Yahoo-1M [56] 2015 116 1,011,723 112,363 Clothing Images

ILSVRC2012 [57] 2012 1,000 ∼1.2 M 50,000 Object Category Images

MS COCO [58] 2014 80 82,783 40,504 Common Object Images

MIRFlicker-1M [59] 2010 - 1 M - Scene Images

Google Landmarks

[60]

2017 15 K ∼1 M - Landmark Images

Google Landmarks

v2 [61]

2020 200 K 5 M - Landmark Images

This survey is organized as follows: the background is pre-

sented in Section 2 in terms of the datasets and evaluation

measures; the evolution of deep learning based image retrieval

is compiled in Section 3; the categorization of existing approaches

based on the supervision type, network type, descriptor type, and

retrieval type are discussed in Section 4, 5, 6, and 7, respectively;

Some other aspects are highlighted in Section 8; the performance

comparison of the popular methods is performed in Section 9;

conclusions and future directions are presented in Section 10.

2 B ACKGROUND

In this section the background is presented in terms of the

commonly used performance evaluation metrics and benchmark

retrieval datasets.

2.1 Retrieval Evaluation Measures

In order to judge the performance of image retrieval approaches,

precision, recall and f-score are the common evaluation metrics.

The mean average precision (mAP ) is very commonly used in the

literature. The precision is deﬁned as the percentage of correctly

retrieved images out of the total number of retrieved images. The

recall is another performance measure being used for image re-

trieval by computing the percentage of correctly retrieved images

out of the total number of relevant images present in the dataset.

The f-score is computed from the harmonic mean of precision and

recall as (2 × precision × recall)/(precision + recall). Thus,

the f-score provides a trade-off between precision and recall.

2.2 Datasets

With the inception of deep learning models, various large-scale

datasets have been created to facilitate the research in image

recognition and retrieval. The details of large-scale datasets are

summarized in Table 1. Datasets having various types of images

are available to test the deep learning based approaches such as

object category datasets [50], [57], [58], scene datasets [51], [54],

[62], digit datasets [52], [53], apparel datasets [55], [56], landmark

datasets [60], [61], etc. The CIFAR-10 dataset is very widely

used object category datset [50]. The ImageNet (ILSVRC2012),

a large-scale dataset, is also an object category dataset with more

than a million number of images [57]. The MS COCO dataset

[58] created for common object detection is also utilized for image

retrieval purpose. Among scene image datasets commonly used for

retrieval purpose, the NUS-WIDE dataset is from National Univer-

sity of Singapore [51]; the Sun397 is a scene understanding dataset

from 397 categories with more than one lakh images [54], [63];

and the MIRFlicker-1M [62] dataset consists of a million images

downloaded from the social photography site Flickr. The MNIST

dataset is one of the old and large-scale digit image datasets [52]

consisting of optical characters. The SVHN is another digit dataset

[53] from the street view house number images which is more

complex than MNIST dataset. The shoes apparel dataset, namely

UT-ZAP50K [55], consists of roughly 50K images. The Yahoo-

1M is another apparel large-scale dataset used in [56] for image

retrieval. The Google landmarks dataset is having around a million

landmark images [60]. The extended version of Google landmarks

(i.e., v2) [61] contains around 5 million landmark images. There

are more datasets used for retrieval in the literature, such as Corel,

Oxford, Paris, etc., however, these are not the large-scale datasets.

The CIFAR-10, MNIST, SVHN and ImageNet are the widely used

datasets in majority of the research.

3 EVOLUTION OF DEEP LEARNING FOR CONTENT

B ASED IMAGE RETRIEVAL (CBIR)

The deep learning based generation of descriptors or hash codes is

the recent trends large-scale content based image retrieval, due

to its computational efﬁciency and retrieval quality [28]. The

deep learning driven features led to the improved retrieval quality.

Recently, it has received increasing attention to utilize the features

for image retrieval using end-to-end representation learning. In

this section, a journey of deep learning models for image retrieval

from 2011 to 2020 is presented. A chronological overview of

different methods is illustrated in Fig. 5. Rest of this section

highlights the selected methods in chronological manner.

3.1 Chronological Overview: 2011 - 2015

3.1.1 2011-2013

Among the initial attempts, in 2011, Krizhevsky and Hinton have

used a deep autoencoder to map the images to short binary codes

for content based image retrieval (CBIR) [64]. Kang et al. (2012)

have proposed a deep multi-view hashing to generate the code

for CBIR from multiple views of data by modeling the layers

with view-speciﬁc and shared hidden nodes [65]. In 2013, Wu

et al. have considered the multiple pretrained stacked denoising

autoencoders over low features of the images [66]. They also ﬁne

tune the multiple deep networks on the output of the pretrained

autoencoders and integrated to generate the multi-modal similarity

function for image retrieval.

3.1.2 2014

In an outstanding work, Babenko et al. (2014) have utilized

the activations of the top layers of a large convolutional neural

network (CNN) as the descriptors (neural codes) for image re-

trieval application [67] as depicted in Fig. 6. A very promising

performance has been recorded using the neural codes for image

retrieval even if the model is trained on un-related data. The

retrieval results are further improved by re-training the model

over similar data and then extracting the neural codes as the

descriptor. They also compress the neural code using principal

component analysis (PCA) to generate the compact descriptor.

In 2014, Wang et al. have investigated a deep ranking model by

learning the similarity metric directly from images [68]. Basically,

they have employed the triplets to capture the inter-class and intra-

class image differences to improve the discriminative ability of the

learnt latent space as the descriptor.

A DECADE SURVEY OF CONTENT BASED IMAGERETRIEVAL USING DEEP LEARNING 4

Fig. 5: A chronological view of deep learning based image retrieval methods depicting its evolution from 2011 to 2020.

Fig. 6: The illustration of the neural code generation from a

convolutional neural network (CNN) [67]. The outputs of layer

5, layer 6 and layer 7 are used to generate the neural code. This

ﬁgure has been originally shown in [67].

3.1.3 2015

In 2015, Lai et al. have used a deep architecture consisting

of a stack of convolution layers to produce the intermediate

image features [69]. They have generated the hash bits from the

different branches of the intermediate image features. The triplet

ranking loss is also utilized to incorporate the inter-class and intra-

class differences in [69] for image retrieval. Zhang et al. (2015)

have developed a deep regularized similarity comparison hashing

(DRSCH) by training a deep CNN model in an end-to-end fashion

to simultaneously optimize the discriminative image features and

hash functions [70]. They have weighted each bit unequally to

make bit-scalable and to prune the redundant bits.

3.2 Chronological Overview: 2016 - 2020

3.2.1 2016

In 2016, Gordo et al. have pooled the relevant regions to form the

descriptor with the help of a region proposal network to prioritize

the important object regions leading to better retrieval performance

[71]. Song et al. (2016) have learnt the lifted structure embedding

by computing the lifted structure loss between the CNN and the

original features [72]. Zhu et al. have proposed a supervised

deep hashing network (DHN) by learning the important image

representation for hash codes and controlling the quantization

error [73]. At the same time, Cao et al. have introduced a deep

quantization network (DQN) which is very similar to the DHN

model [74]. The CNN based features are aggregated by Husain

and Bober (2016) with the help of rank-aware multi-assignment

and direction based combination [75]. Zhong et al. have added a

sigmoid layer before the loss layer of a CNN to learn the binary

code for CBIR [76].

3.2.2 2017

In 2017, Cao et al. have proposed HashNet deep architecture to

generate the hash code by a continuation method [77]. It learns

the non-smooth binary activations using the continuation method

to generate the binary hash codes from imbalanced similarity

data. Gordo et al. (2017) have shown that the noisy training data,

inappropriate deep architecture and suboptimal training procedure

are the main hurdle to utilize the deep learning for image retrieval

[78]. They have performed the cleaning step to improve the

dataset and utilized the Siamese network for learning the image

representations and reported the mean average precision (%) of

94.7, 96.6, and 94.8 over Oxford 5k, Paris 6k and Holidays

datasets, respectively. Different masking schemes such as SUM-

mask and MAX-mask are used in [79] to select the prominent

CNN features for image retrieval. A bilinear network with two

parallel CNNs is also used as the compact feature extractors for

CBIR [80] and reported the mean average precision of 95.7% on

Oxford5K and 88.6% on Oxford105K datasets with feature vector

of 16-length.

3.2.3 2018

In 2018, Cao et al. have investigated a deep cauchy hashing

(DCH) model for binary hash code with the help of a pairwise

cross-entropy loss based on Cauchy distribution [81]. Su et al.

have employed the greedy hash by transmitting the gradient as

intact during the backpropagation for hash coding layer which

uses the sign function in forward propagation [82]. Thus, it

maintains the discrete constraints, while avoiding the vanishing

gradient problem. Yuan et al. (2018) have trained the network

directly via policy gradient to maximize the reward expectation

of similarity preservation using the generated binary codes [83].

A series expansion is used to treat the binary optimization of the

hash function as the differentiable optimization which minimizes

the objective discrepancy caused by relaxation [84]. Wu et al.

(2018) have investigated a deep index-compatible hashing (DICH)

method [85] by minimizing the number of similar bits between the

binary codes of inter-class images.

3.2.4 2019

In 2019, a deep incremental hashing network (DIHN) is proposed

by Wu et al. [127] to directly learn the hash codes corresponding

剩余19页未读，继续阅读

评论收藏

内容反馈

weixin_41476291

2021-03-22

非常感谢！

syp_net

粉丝: 158
资源: 1196

深度学习图像检索(CBIR): 十年之大综述

最新资源

深度学习图像检索(CBIR): 十年之大综述

基于深度学习的图像检索研究

深度图像检索: 2012到2020综述论文

cbir 基于内容的图像检索

CBIR 基于内容的图像检索概述

基于内容的图像检索CBIR HOUGH

基于深度学习的图像检索系统 (1).pdf

py-cbir:python中基于内容的图像检索演示

基于深度学习的图像检索系统（CNN）

CNN实现的图像检索

CBIR 基于内容的图像搜索 源代码

基于深度学习特征的图像检索方法.pdf

深度哈希图像检索综述

基于深度学习的图像检索研究.pdf

图像检索综述1

CBIR基础文章

cbir:使用视觉和文本特征的图像检索

Python-Oxford5kParis6k大规模图像检索基准

基于多层 CNN 特征的图像检索技术

基于特征点匹配技术的图像检索方法

android--图像检索

基于内容的图像检索技术的现状和发展趋势研究

基于深度学习的服装图像检索方法

基于深度学习的图像检索研究 (1).pdf

基于深度学习的跨模态检索综述

基于多尺度深度学习的商品图像检索.pdf

基于深度哈希学习的商标图像检索研究

图像检索-caltech101-keras-python

CNN图像检索

最新资源

CBIR 基于内容的图像搜索源代码