《对比监督学习》2020综述论文_深度对比学习综述资源-CSDN文库

需积分: 50 146 浏览量 2020-11-05 20:39:19 上传评论 4 收藏 5.18MB PDF 举报

资源推荐

资源详情

资源评论

A SURVEY ON CONTRASTIVE SELF-SUPERVISED LEARNING

Ashish Jaiswal

The University of Texas at Arlington

Arlington, TX 76019

ashish.jaiswal@mavs.uta.edu

Ashwin Ramesh Babu

The University of Texas at Arlington

Arlington, TX 76019

ashwin.rameshbabu@mavs.uta.edu

Mohammad Zaki Zadeh

The University of Texas at Arlington

Arlington, TX 76019

mohammad.zakizadehgharie@mavs.uta.edu

Debapriya Banerjee

The University of Texas at Arlington

Arlington, TX 76019

debapriya.banerjee2@mavs.uta.edu

Fillia Makedon

The University of Texas at Arlington

Arlington, TX 76019

makedon@uta.edu

November 3, 2020

ABSTRACT

Self-supervised learning has gained popularity because of its ability to avoid the cost of annotating

large-scale datasets. It is capable of adopting self-deﬁned pseudo labels as supervision and use the

learned representations for several downstream tasks. Speciﬁcally, contrastive learning has recently

become a dominant component in self-supervised learning methods for computer vision, natural

language processing (NLP), and other domains. It aims at embedding augmented versions of the

same sample close to each other while trying to push away embeddings from different samples. This

paper provides an extensive review of self-supervised methods that follow the contrastive approach.

The work explains commonly used pretext tasks in a contrastive learning setup, followed by different

architectures that have been proposed so far. Next, we have a performance comparison of different

methods for multiple downstream tasks such as image classiﬁcation, object detection, and action

recognition. Finally, we conclude with the limitations of the current methods and the need for further

techniques and future directions to make substantial progress.

Keywords

contrastive learning

self-supervised learning

discriminative learning

image/video classiﬁcation

object

detection · unsupervised learning · transfer learning

1 Introduction

The advancements in deep learning have elevated it to become one of the core components in most intelligent systems in

existence. The ability to learn rich patterns from the abundance of data available today has made deep neural networks

(DNNs) a compelling approach in the majority of computer vision (CV) tasks such as image classiﬁcation, object

detection, image segmentation, activity recognition as well as natural language processing (NLP) tasks such as sentence

classiﬁcation, language models, machine translation, etc. However, the supervised approach to learning features from

labeled data has almost reached its saturation due to intense labor required in manually annotating millions of data

samples. This is because most of the modern computer vision systems (that are supervised) try to learn some form of

image representations by ﬁnding a pattern between the data points and their respective annotations in large datasets.

Works such as GRAD-CAM [

] have proposed techniques that provide visual explanations for decisions made by a

model to make them more transparent and explainable.

arXiv:2011.00362v1 [cs.CV] 31 Oct 2020

A PREPRINT - NOVEMBER 3, 2020

Generative models gained its popularity after the introduction of Generative Adversarial Networks (GANs) [

] in

2014. The work later became the foundation for many successful architectures such as CycleGAN [4], StyleGAN [5],

PixelRNN [

], Text2Image [

], DiscoGAN [

], etc. These methods inspired more researchers to switch to training deep

learning models with unlabeled data in an self-supervised setup. Despite their success, researchers started realizing

some of the complications in GAN-based approaches. They are harder to train because of two main reasons: (a)

non-convergence–the model parameters oscillate a lot and rarely converge, and (b) the discriminator gets too successful

that the generator network fails to create real-like fakes due to which the learning cannot be continued. Also, proper

synchronization is required between the generator and the discriminator that prevents the discriminator to converge and

the generator to diverge.

Figure 3: Top-1 classiﬁcation accuracy of different contrastive learning methods against baseline supervised method on

ImageNet

Unlike generative models, contrastive learning (CL) is a discriminative approach that aims at grouping similar samples

closer and diverse samples far from each other as shown in ﬁgure 1. To achieve this, a similarity metric is used to

measure how close two embeddings are. Especially, for computer vision tasks, a contrastive loss is evaluated based

on the feature representations of the images extracted from an encoder network. For instance, one sample from the

training dataset is taken and a transformed version of the sample is retrieved by applying appropriate data augmentation

techniques. During training referring to ﬁgure 2, the augmented version of the original sample is considered as a

positive sample, and the rest of the samples in the batch/dataset (depends on the method being used) are considered

negative samples. Next, the model is trained in a way that it learns to differentiate positive samples from the negative

ones. The differentiation is achieved with the help of some pretext task (explained in section 2). In doing so, the model

learns quality representations of the samples and is used later for transferring knowledge to downstream tasks. This

idea is advocated by an interesting experiment conducted by Epstein [

] in 2016, where he asked his students to draw a

dollar bill with and without looking at the bill. The results from the experiment show that the brain does not require

complete information of a visual piece to differentiate one object from the other. Instead, only a rough representation of

an image is enough to do so.

Most of the earlier works in this area combined some form of instance-level classiﬁcation approach[

][

] with

contrastive learning and were successful to some extent. However, recent methods such as SwAV [

], MoCo [

], and

A PREPRINT - NOVEMBER 3, 2020

SimCLR [

] with modiﬁed approaches have produced results comparable to the state-of-the-art supervised method on

ImageNet [

] dataset as shown in ﬁgure 3. Similarly, PIRL [

], Selﬁe [

], and [

] are some papers that reﬂect the

effectiveness of the pretext tasks being used and how they boost the performance of their models.

2 Pretext Tasks

Pretext tasks are self-supervised tasks that act as an important strategy to learn representations of the data using pseudo

labels. These pseudo labels are generated automatically based on the attributes found in the data. The learned model

from the pretext task can be used for any downstream tasks such as classiﬁcation, segmentation, detection, etc. in

computer vision. Furthermore, these tasks can be applied to any kind of data such as image, video, speech, signals,

and so on. For a pretext task in contrastive learning, the original image acts as an anchor, its augmented(transformed)

version acts as a positive sample, and the rest of the images in the batch or in the training data act as negative samples.

Most of the commonly used pretext tasks are divided into four main categories: color transformation, geometric

transformation, context-based tasks, and cross-modal based tasks. These pretext tasks have been used in various

scenarios based on the problem intended to be solved.

2.1 Color Transformation

Figure 4: Color Transformation as pretext task [

]. (a) Original (b) Gaussian noise (c) Gaussian blur (d) Color

distortion (Jitter)

Color transformation involves basic adjustments of color levels in an image such as blurring, color distortions, converting

to grayscale, etc. Figure 4 represents an example of color transformation applied on a sample image from the ImageNet

dataset [15]. During this pretext task, the network learns to recognize similar images invariant to their colors.

2.2 Geometric Transformation

A geometric transformation is a spatial transformation where the geometry of the image is modiﬁed without altering

its actual pixel information. The transformations include scaling, random cropping, ﬂipping (horizontally, vertically),

etc. as represented in ﬁgure 5 through which global-to-local view prediction is achieved. Here the original image is

considered as the global view and the transformed version is considered as the local view. Chen et. al. [

] performed

such transformations to learn features during pretext task.

剩余19页未读，继续阅读

评论收藏

内容反馈

syp_net

粉丝: 158
资源: 1196

《对比监督学习》2020综述论文

监督学习论文

最新的对比自监督学习（Contrastive Self-supervised Learning）综述论文

自监督学习：生成和对比方法综述

《深度半监督学习》综述论文

最新《知识蒸馏》2020综述论文（来自悉尼大学）

电子科大最新《深度半监督学习》综述论文（2021版）

《监督学》课程综述-论文.zip

TAMU首篇《图神经网络自监督学习》综述论文

深度对比学习综述论文PDF

深度学习中的无监督学习方法综述.pdf

强化学习综述论文

很棒的对比自我监督学习：很棒的对比自我监督学习论文的完整列表

8篇半监督学习相关论文

最新「无监督网络表示学习」综述论文

自监督视频表征学习综述.docx

《图对抗机器学习》2020综述论文.pdf

2019-2020必看的十篇【深度学习领域综述论文】.zip

《知识图谱:构建到应用》综述论文（2020年）

深度学习多源领域自适应综述论文.pdf

半监督学习研究综述 半监督

很棒的自我监督论文：自我监督学习的纸质银行

最新资源

半监督学习研究综述半监督