ContrastiveMultiviewCoding笔记_coding教程资源-CSDN文库

需积分: 31 176 浏览量 2020-12-18 20:10:19 上传评论收藏 10.59MB PDF 举报

资源详情

资源评论

Contrastive Multiview Coding

Yonglong Tian

MIT CSAIL

yonglong@mit.edu

Dilip Krishnan

Google Research

dilipkay@google.com

Phillip Isola

MIT CSAIL

phillipi@mit.edu

Abstract

Humans view the world through many sensory channels,

e.g., the long-wavelength light channel, viewed by the left

eye, or the high-frequency vibrations channel, heard by the

right ear. Each view is noisy and incomplete, but impor-

tant factors, such as physics, geometry, and semantics, tend

to be shared between all views (e.g., a “dog” can be seen,

heard, and felt). We investigate the classic hypothesis that

a powerful representation is one that models view-invariant

factors. We study this hypothesis under the framework of

multiview contrastive learning, where we learn a represen-

tation that aims to maximize mutual information between

different views of the same scene but is otherwise compact.

Our approach scales to any number of views, and is view-

agnostic. We analyze key properties of the approach that

make it work, ﬁnding that the contrastive loss outperforms

a popular alternative based on cross-view prediction, and

that the more views we learn from, the better the resulting

representation captures underlying scene semantics. Our ap-

proach achieves state-of-the-art results on image and video

unsupervised learning benchmarks. Code is released at:

http://github.com/HobbitLong/CMC/.

1. Introduction

A foundational idea in coding theory is to learn com-

pressed representations that nonetheless can be used to re-

construct the raw data. This idea shows up in contemporary

representation learning in the form of autoencoders [

] and

generative models [

], which try to represent a data

point or distribution as losslessly as possible. Yet lossless

representation might not be what we really want, and indeed

it is trivial to achieve – the raw data itself is a lossless rep-

resentation. What we might instead prefer is to keep the

“good” information (signal) and throw away the rest (noise).

How can we identify what information is signal and what is

noise?

To an autoencoder, or a max likelihood generative model,

a bit is a bit. No one bit is better than any other. Our con-

jecture in this paper is that some bits are in fact better than

others. Some bits code important properties like seman-

tics, physics, and geometry, while others code attributes that

we might consider less important, like incidental lighting

✓

<latexit sha1_base64="q/a21S0rSvzWx6csIj2G7GwCFa0=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lE0GPRi8cKthaaEDbbSbt0swm7E6GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvyqQw6LrfTmVtfWNzq7pd29nd2z+oHx51TZprDh2eylT3ImZACgUdFCihl2lgSSThMRrfzvzHJ9BGpOoBJxkECRsqEQvO0Ep+HBY+jgBZ6E3DesNtunPQVeKVpEFKtMP6lz9IeZ6AQi6ZMX3PzTAomEbBJUxrfm4gY3zMhtC3VLEETFDMb57SM6sMaJxqWwrpXP09UbDEmEkS2c6E4cgsezPxP6+fY3wdFEJlOYLii0VxLimmdBYAHQgNHOXEEsa1sLdSPmKacbQx1WwI3vLLq6R70fTcpnd/2WjdlHFUyQk5JefEI1ekRe5Im3QIJxl5Jq/kzcmdF+fd+Vi0Vpxy5pj8gfP5AxNYkbE=</latexit>

✓

<latexit sha1_base64="VhLeRtSRrYmdxiVlEYJPRKRDDDY=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Ac0oWy2m3bpZhN2J0IJ/RtePCji1T/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6nhUijeRoGS91LNaRxK3g0nd3O/+8S1EYl6xGnKg5iOlIgEo2glPxrkPo450kFjNqjW3Lq7AFknXkFqUKA1qH75w4RlMVfIJDWm77kpBjnVKJjks4qfGZ5SNqEj3rdU0ZibIF/cPCMXVhmSKNG2FJKF+nsip7Ex0zi0nTHFsVn15uJ/Xj/D6CbIhUoz5IotF0WZJJiQeQBkKDRnKKeWUKaFvZWwMdWUoY2pYkPwVl9eJ51G3XPr3sNVrXlbxFGGMziHS/DgGppwDy1oA4MUnuEV3pzMeXHenY9la8kpZk7hD5zPHxTdkbI=</latexit>

2 V

<latexit sha1_base64="H+m5d1VWYwFqrNMt4Ah4AnBsMpg=">AAAB9HicbVBNS8NAEJ3Ur1q/qh69LBbBU8mKoMeiF48V7Ae0MWy2m3bpZhN3N4US+ju8eFDEqz/Gm//GTZuDtj4YeLw3w8y8IBFcG9f9dkpr6xubW+Xtys7u3v5B9fCoreNUUdaisYhVNyCaCS5Zy3AjWDdRjESBYJ1gfJv7nQlTmsfywUwT5kVkKHnIKTFW8iaP3MeozyVq+9iv1ty6OwdaJbggNSjQ9Ktf/UFM04hJQwXRuofdxHgZUYZTwWaVfqpZQuiYDFnPUkkipr1sfvQMnVllgMJY2ZIGzdXfExmJtJ5Gge2MiBnpZS8X//N6qQmvvYzLJDVM0sWiMBXIxChPAA24YtSIqSWEKm5vRXREFKHG5lSxIeDll1dJ+6KO3Tq+v6w1boo4ynACp3AOGK6gAXfQhBZQeIJneIU3Z+K8OO/Ox6K15BQzx/AHzucPUGCRIg==</latexit>

2 V

<latexit sha1_base64="WTzTlGfGzUWV/jN8CFNM+vGS7s8=">AAAB9HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48VbCu0MWy2k3bpZhN3N4VS+ju8eFDEqz/Gm//GbZuDtj4YeLw3w8y8MBVcG9f9dgpr6xubW8Xt0s7u3v5B+fCopZNMMWyyRCTqIaQaBZfYNNwIfEgV0jgU2A6HNzO/PUKleSLvzThFP6Z9ySPOqLGSP3rkQY10uSStoBaUK27VnYOsEi8nFcjRCMpf3V7CshilYYJq3fHc1PgTqgxnAqelbqYxpWxI+9ixVNIYtT+ZHz0lZ1bpkShRtqQhc/X3xITGWo/j0HbG1Az0sjcT//M6mYmu/AmXaWZQssWiKBPEJGSWAOlxhcyIsSWUKW5vJWxAFWXG5lSyIXjLL6+SVq3quVXv7qJSv87jKMIJnMI5eHAJdbiFBjSBwRM8wyu8OSPnxXl3PhatBSefOYY/cD5/AFNwkSQ=</latexit>

<latexit sha1_base64="oO91WuXFyjBY+3N9hG7fR4rfJLk=">AAAB7HicbVBNS8NAEJ34WetX1aOXxSJ4KokIeix68VjBtIU2ls120y7dbMLuRKihv8GLB0W8+oO8+W/ctjlo64OBx3szzMwLUykMuu63s7K6tr6xWdoqb+/s7u1XDg6bJsk04z5LZKLbITVcCsV9FCh5O9WcxqHkrXB0M/Vbj1wbkah7HKc8iOlAiUgwilbynx5Ez+tVqm7NnYEsE68gVSjQ6FW+uv2EZTFXyCQ1puO5KQY51SiY5JNyNzM8pWxEB7xjqaIxN0E+O3ZCTq3SJ1GibSkkM/X3RE5jY8ZxaDtjikOz6E3F/7xOhtFVkAuVZsgVmy+KMkkwIdPPSV9ozlCOLaFMC3srYUOqKUObT9mG4C2+vEya5zXPrXl3F9X6dRFHCY7hBM7Ag0uowy00wAcGAp7hFd4c5bw4787HvHXFKWaO4A+czx+JGY59</latexit>

<latexit sha1_base64="guyt78QnHSsnalutsxfj0JjsQ4w=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48VTFtoY9lsJ+3SzSbsboRa+hu8eFDEqz/Im//GbZuDtj4YeLw3w8y8MBVcG9f9dgpr6xubW8Xt0s7u3v5B+fCoqZNMMfRZIhLVDqlGwSX6hhuB7VQhjUOBrXB0M/Nbj6g0T+S9GacYxHQgecQZNVbynx54r9YrV9yqOwdZJV5OKpCj0St/dfsJy2KUhgmqdcdzUxNMqDKcCZyWupnGlLIRHWDHUklj1MFkfuyUnFmlT6JE2ZKGzNXfExMaaz2OQ9sZUzPUy95M/M/rZCa6CiZcpplByRaLokwQk5DZ56TPFTIjxpZQpri9lbAhVZQZm0/JhuAtv7xKmrWq51a9u4tK/TqPowgncArn4MEl1OEWGuADAw7P8ApvjnRenHfnY9FacPKZY/gD5/MHip2Ofg==</latexit>



<latexit sha1_base64="q4WCp+vNSk6GA1s6N0x45fCcZO8=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GOpF48V7Ac0oWy203TpZhN2N0IJ/RFePCji1d/jzX/jts1BWx8MPN6bYWZemAqujet+O6WNza3tnfJuZW//4PCoenzS0UmmGLZZIhLVC6lGwSW2DTcCe6lCGocCu+Hkbu53n1BpnshHM00xiGkk+YgzaqzU9Zs8ivx8UK25dXcBsk68gtSgQGtQ/fKHCctilIYJqnXfc1MT5FQZzgTOKn6mMaVsQiPsWyppjDrIF+fOyIVVhmSUKFvSkIX6eyKnsdbTOLSdMTVjverNxf+8fmZGt0HOZZoZlGy5aJQJYhIy/50MuUJmxNQSyhS3txI2pooyYxOq2BC81ZfXSeeq7rl17+G61mgWcZThDM7hEjy4gQbcQwvawGACz/AKb07qvDjvzseyteQUM6fwB87nDyI/j2w=</latexit>



<latexit sha1_base64="l9ZhmQEFCunp9RwAEQItJ26mlG8=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GOpF48V7Ac0oWy2k3TpZhN2N0Ip/RFePCji1d/jzX/jts1BWx8MPN6bYWZemAmujet+O6WNza3tnfJuZW//4PCoenzS0WmuGLZZKlLVC6lGwSW2DTcCe5lCmoQCu+H4bu53n1BpnspHM8kwSGgsecQZNVbq+k0ex/5sUK25dXcBsk68gtSgQGtQ/fKHKcsTlIYJqnXfczMTTKkynAmcVfxcY0bZmMbYt1TSBHUwXZw7IxdWGZIoVbakIQv198SUJlpPktB2JtSM9Ko3F//z+rmJboMpl1luULLloigXxKRk/jsZcoXMiIkllClubyVsRBVlxiZUsSF4qy+vk85V3XPr3sN1rdEs4ijDGZzDJXhwAw24hxa0gcEYnuEV3pzMeXHenY9la8kpZk7hD5zPHyVHj24=</latexit>

<latexit sha1_base64="axFzcoBaHbWATKajATY6NWO1CV4=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBg5REBD0WvXhswX5AG8pmO2nXbjZhdyOU0F/gxYMiXv1J3vw3btsctPXBwOO9GWbmBYng2rjut1NYW9/Y3Cpul3Z29/YPyodHLR2nimGTxSJWnYBqFFxi03AjsJMopFEgsB2M72Z++wmV5rF8MJME/YgOJQ85o8ZKjYt+ueJW3TnIKvFyUoEc9X75qzeIWRqhNExQrbuemxg/o8pwJnBa6qUaE8rGdIhdSyWNUPvZ/NApObPKgISxsiUNmau/JzIaaT2JAtsZUTPSy95M/M/rpia88TMuk9SgZItFYSqIicnsazLgCpkRE0soU9zeStiIKsqMzaZkQ/CWX14lrcuq51a9xlWldpvHUYQTOIVz8OAaanAPdWgCA4RneIU359F5cd6dj0VrwclnjuEPnM8fcwmMsA==</latexit>

✓

<latexit sha1_base64="zQwMUXQQ2w367YTABSaCx2nivVQ=">AAAB83icbVBNS8NAEN3Ur1q/qh69LBbBU0lU0GPRi8cK9gOaUDbbSbt0swm7E6GE/g0vHhTx6p/x5r9x2+agrQ8GHu/NMDMvTKUw6LrfTmltfWNzq7xd2dnd2z+oHh61TZJpDi2eyER3Q2ZACgUtFCihm2pgcSihE47vZn7nCbQRiXrESQpBzIZKRIIztJIf9XMfR4CsfzntV2tu3Z2DrhKvIDVSoNmvfvmDhGcxKOSSGdPz3BSDnGkUXMK04mcGUsbHbAg9SxWLwQT5/OYpPbPKgEaJtqWQztXfEzmLjZnEoe2MGY7MsjcT//N6GUY3QS5UmiEovlgUZZJiQmcB0IHQwFFOLGFcC3sr5SOmGUcbU8WG4C2/vEraF3XPrXsPV7XGbRFHmZyQU3JOPHJNGuSeNEmLcJKSZ/JK3pzMeXHenY9Fa8kpZo7JHzifPxZikbM=</latexit>

✓

<latexit sha1_base64="n2CaaxZoLkQxpfbeT53tx7kPMqU=">AAAB83icbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Ac0oWy2m3bpZhN2J0IJ/RtePCji1T/jzX/jts1BWx8MPN6bYWZemEph0HW/ndLG5tb2Tnm3srd/cHhUPT7pmCTTjLdZIhPdC6nhUijeRoGS91LNaRxK3g0nd3O/+8S1EYl6xGnKg5iOlIgEo2glPxrkPo450kFjNqjW3Lq7AFknXkFqUKA1qH75w4RlMVfIJDWm77kpBjnVKJjks4qfGZ5SNqEj3rdU0ZibIF/cPCMXVhmSKNG2FJKF+nsip7Ex0zi0nTHFsVn15uJ/Xj/D6CbIhUoz5IotF0WZJJiQeQBkKDRnKKeWUKaFvZWwMdWUoY2pYkPwVl9eJ52ruufWvYdGrXlbxFGGMziHS/DgGppwDy1oA4MUnuEV3pzMeXHenY9la8kpZk7hD5zPHxfnkbQ=</latexit>

<latexit sha1_base64="M7m4hocqbGeXGgakLxdWsFDBQ8A=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8cK9gPaWDbbTbt0swm7E6GG/gYvHhTx6g/y5r9x2+agrQ8GHu/NMDMvSKQw6LrfTmFldW19o7hZ2tre2d0r7x80TZxqxhsslrFuB9RwKRRvoEDJ24nmNAokbwWjm6nfeuTaiFjd4zjhfkQHSoSCUbRS4+lB9M575YpbdWcgy8TLSQVy1Hvlr24/ZmnEFTJJjel4boJ+RjUKJvmk1E0NTygb0QHvWKpoxI2fzY6dkBOr9EkYa1sKyUz9PZHRyJhxFNjOiOLQLHpT8T+vk2J45WdCJSlyxeaLwlQSjMn0c9IXmjOUY0so08LeStiQasrQ5lOyIXiLLy+T5lnVc6ve3UWldp3HUYQjOIZT8OASanALdWgAAwHP8ApvjnJenHfnY95acPKZQ/gD5/MHjCGOfw==</latexit>

<latexit sha1_base64="OZSe5FjkW+cQ+ylxJrqrA7doVXU=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48VTFtoY9lsJ+3SzSbsboRa+hu8eFDEqz/Im//GbZuDtj4YeLw3w8y8MBVcG9f9dgpr6xubW8Xt0s7u3v5B+fCoqZNMMfRZIhLVDqlGwSX6hhuB7VQhjUOBrXB0M/Nbj6g0T+S9GacYxHQgecQZNVbynx54r9YrV9yqOwdZJV5OKpCj0St/dfsJy2KUhgmqdcdzUxNMqDKcCZyWupnGlLIRHWDHUklj1MFkfuyUnFmlT6JE2ZKGzNXfExMaaz2OQ9sZUzPUy95M/M/rZCa6CiZcpplByRaLokwQk5DZ56TPFTIjxpZQpri9lbAhVZQZm0/JhuAtv7xKmhdVz616d7VK/TqPowgncArn4MEl1OEWGuADAw7P8ApvjnRenHfnY9FacPKZY/gD5/MHjaWOgA==</latexit>

2 V

<latexit sha1_base64="uxZqoTOKoLsYr9ruMLIOiz8PdA8=">AAAB9HicbVBNSwMxEJ2tX7V+VT16CRbBU9m1gh6LXjxWsB/Qrks2zbah2WRNsoWy9Hd48aCIV3+MN/+NabsHbX0w8Hhvhpl5YcKZNq777RTW1jc2t4rbpZ3dvf2D8uFRS8tUEdokkkvVCbGmnAnaNMxw2kkUxXHIaTsc3c789pgqzaR4MJOE+jEeCBYxgo2V/PEjC2qoxwRqBbWgXHGr7hxolXg5qUCORlD+6vUlSWMqDOFY667nJsbPsDKMcDot9VJNE0xGeEC7lgocU+1n86On6MwqfRRJZUsYNFd/T2Q41noSh7Yzxmaol72Z+J/XTU107WdMJKmhgiwWRSlHRqJZAqjPFCWGTyzBRDF7KyJDrDAxNqeSDcFbfnmVtC6qnlv17i8r9Zs8jiKcwCmcgwdXUIc7aEATCDzBM7zCmzN2Xpx352PRWnDymWP4A+fzB1aAkSY=</latexit>

2 V

<latexit sha1_base64="ThSQ3X4V8NFm1C31LnrV6TpJX+U=">AAAB9HicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48VbCu0MWy2k3bpZhN3N4VS+ju8eFDEqz/Gm//GbZuDtj4YeLw3w8y8MBVcG9f9dgpr6xubW8Xt0s7u3v5B+fCopZNMMWyyRCTqIaQaBZfYNNwIfEgV0jgU2A6HNzO/PUKleSLvzThFP6Z9ySPOqLGSP3rkQY10uSStoBaUK27VnYOsEi8nFcjRCMpf3V7CshilYYJq3fHc1PgTqgxnAqelbqYxpWxI+9ixVNIYtT+ZHz0lZ1bpkShRtqQhc/X3xITGWo/j0HbG1Az0sjcT//M6mYmu/AmXaWZQssWiKBPEJGSWAOlxhcyIsSWUKW5vJWxAFWXG5lSyIXjLL6+S1kXVc6veXa1Sv87jKMIJnMI5eHAJdbiFBjSBwRM8wyu8OSPnxXl3PhatBSefOYY/cD5/AFmQkSg=</latexit>

✓

2 V

<latexit sha1_base64="aVeMGIr31EA8b+RIJUiMu4YPpfA=">AAAB9HicbVBNSwMxEJ2tX7V+VT16CRbBU9mIUI9FLx4r2A9o1yWbZtvYbHZNsoWy9Hd48aCIV3+MN/+NabsHbX0w8Hhvhpl5QSK4Nq777RTW1jc2t4rbpZ3dvf2D8uFRS8epoqxJYxGrTkA0E1yypuFGsE6iGIkCwdrB6Gbmt8dMaR7LezNJmBeRgeQhp8RYyRs/PPoY9bhELR/75YpbdedAqwTnpAI5Gn75q9ePaRoxaaggWnexmxgvI8pwKti01Es1SwgdkQHrWipJxLSXzY+eojOr9FEYK1vSoLn6eyIjkdaTKLCdETFDvezNxP+8bmrCKy/jMkkNk3SxKEwFMjGaJYD6XDFqxMQSQhW3tyI6JIpQY3Mq2RDw8surpHVRxW4V311W6td5HEU4gVM4Bww1qMMtNKAJFJ7gGV7hzRk7L86787FoLTj5zDH8gfP5A1HukSM=</latexit>

<latexit sha1_base64="duUJZRL7BAW9fYFkJEfi2hnUSE4=">AAAB7HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPRi8cKpi20tWy2m3btZhN2J0IN/Q1ePCji1R/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLK6tr6RnGztLW9s7tX3j9omDjVjPsslrFuBdRwKRT3UaDkrURzGgWSN4PR9dRvPnJtRKzucJzwbkQHSoSCUbSS/3T/0PN65YpbdWcgy8TLSQVy1Hvlr04/ZmnEFTJJjWl7boLdjGoUTPJJqZManlA2ogPetlTRiJtuNjt2Qk6s0idhrG0pJDP190RGI2PGUWA7I4pDs+hNxf+8dorhZTcTKkmRKzZfFKaSYEymn5O+0JyhHFtCmRb2VsKGVFOGNp+SDcFbfHmZNM6qnlv1bs8rtas8jiIcwTGcggcXUIMbqIMPDAQ8wyu8Ocp5cd6dj3lrwclnDuEPnM8fip+Ofg==</latexit>

Matching views

<latexit sha1_base64="PqZpBI/5PUTD03tg2Et+F0fHoPs=">AAAB9XicbVA9TwJBEJ3DL8Qv1NJmIzGxInc0WhJtbEwwkY8ETrK3DLBhb++yuwchF/6HjYXG2Ppf7Pw3LnCFgi+Z5OW9mczMC2LBtXHdbye3sbm1vZPfLeztHxweFY9PGjpKFMM6i0SkWgHVKLjEuuFGYCtWSMNAYDMY3c795hiV5pF8NNMY/ZAOJO9zRo2Vnu6pYUMuB2TMcaK7xZJbdhcg68TLSAky1LrFr04vYkmI0jBBtW57bmz8lCrDmcBZoZNojCkb0QG2LZU0RO2ni6tn5MIqPdKPlC1pyEL9PZHSUOtpGNjOkJqhXvXm4n9eOzH9az/lMk4MSrZc1E8EMRGZR0B6XCEzYmoJZYrbWwkbUkWZsUEVbAje6svrpFEpe27Ze6iUqjdZHHk4g3O4BA+uoAp3UIM6MFDwDK/w5kycF+fd+Vi25pxs5hT+wPn8AX6Ckn0=</latexit>

Unmatching view

<latexit sha1_base64="Adr4auocLSgzEl3Yo4wZFJ+ig6s=">AAAB+HicbVA9TwJBFHznJ+IHp5Y2G4mJFbmj0ZJoY4mJByRAyN7yDjbs7V129zBI+CU2Fhpj60+x89+4wBUKTrLJZOa9vNkJU8G18bxvZ2Nza3tnt7BX3D84PCq5xycNnWSKYcASkahWSDUKLjEw3AhspQppHApshqPbud8co9I8kQ9mkmI3pgPJI86osVLPLQUypoYNuRyQMcfHnlv2Kt4CZJ34OSlDjnrP/er0E5bFKA0TVOu276WmO6XKcCZwVuxkGlPKRnSAbUsljVF3p4vgM3JhlT6JEmWfNGSh/t6Y0ljrSRzaSZtyqFe9ufif185MdN2dcplmBiVbHooyQUxC5i2QPlfIjJhYQpniNithQ6ooM7aroi3BX/3yOmlUK75X8e+r5dpNXkcBzuAcLsGHK6jBHdQhAAYZPMMrvDlPzovz7nwsRzecfOcU/sD5/AHLupMo</latexit>

Figure 1: Given a set of sensory views, a deep representation is

learnt by bringing views of the same scene together in embedding

space, while pushing views of different scenes apart. Here we show

and example of a 4-view dataset (NYU RGBD [

]) and its learned

representation. The encodings for each view may be concatenated

to form the full representation of a scene.

conditions or thermal noise in a camera’s sensor.

We revisit the classic hypothesis that the good bits are the

ones that are shared between multiple views of the world,

for example between multiple sensory modalities like vision,

sound, and touch [

]. Under this perspective “presence of

dog” is good information, since dogs can be seen, heard,

and felt, but “camera pose” is bad information, since a cam-

era’s pose has little or no effect on the acoustic and tactile

properties of the imaged scene. This hypothesis corresponds

to the inductive bias that the way you view a scene should

not affect its semantics. There is signiﬁcant evidence in

the cognitive science and neuroscience literature that such

view-invariant representations are encoded by the brain (e.g.,

[

]). In this paper, we speciﬁcally study the setting

where the different views are different image channels, such

as luminance, chrominance, depth, and optical ﬂow. The fun-

damental supervisory signal we exploit is the co-occurrence,

in natural data, of multiple views of the same scene. For

example, we consider an image in Lab color space to be a

paired example of the co-occurrence of two views of the

scene, the L view and the ab view: {L, ab}.

Our goal is therefore to learn representations that capture

arXiv:1906.05849v4 [cs.CV] 11 Mar 2020

information shared between multiple sensory channels but

that are otherwise compact (i.e. discard channel-speciﬁc

nuisance factors). To do so, we employ contrastive learning,

where we learn a feature embedding such that views of the

same scene map to nearby points (measured with Euclidean

distance in representation space) while views of different

scenes map to far apart points. In particular, we adapt the

recently proposed method of Contrastive Predictive Coding

(CPC) [

], except we simplify it – removing the recurrent

network – and generalize it – showing how to apply it to

arbitrary collections of image channels, rather than just to

temporal or spatial predictions. In reference to CPC, we term

our method Contrastive Multiview Coding (CMC), although

we note that our formulation is arguably equally related to

Instance Discrimination [

]. The contrastive objective in

our formulation, as in CPC and Instance Discrimination,

can be understood as attempting to maximize the mutual

information between the representations of multiple views

of the data.

We intentionally leave “good bits” only loosely deﬁned

and treat its deﬁnition as an empirical question. Ultimately,

the proof is in the pudding: we consider a representation

to be good if it makes subsequent problem solving easy, on

tasks of human interest. For example, a useful representation

of images might be a feature space in which it is easy to learn

to recognize objects. We therefore evaluate our method by

testing if the learned representations transfer well to standard

semantic recognition tasks. On several benchmark tasks, our

method achieves results competitive with the state of the

art, compared to other methods for self-supervised repre-

sentation learning. We additionally ﬁnd that the quality of

the representation improves as a function of the number of

views used for training. Finally, we compare the contrastive

formulation of multiview learning to the recently popular

approach of cross-view prediction, and ﬁnd that in head-to-

head comparisons, the contrastive approach learns stronger

representations.

The core ideas that we build on: contrastive learning,

mutual information maximization, and deep representation

learning, are not new and have been explored in the literature

on representation and multiview learning for decades [

]. Our main contribution is to set up a framework to

extend these ideas to any number of views, and to empirically

study the factors that lead to success in this framework. A

review of the related literature is given in Section 2; and Fig.

1 gives a pictorial overview of our framework. Our main

contributions are:

•

We apply contrastive learning to the multiview setting,

attempting to maximize mutual information between

representations of different views of the same scene (in

particular, between different image channels).

•

We extend the framework to learn from more than two

views, and show that the quality of the learned represen-

tation improves as number of views increase. Ours is

the ﬁrst work to explicitly show the beneﬁts of multiple

views on representation quality.

•

We conduct controlled experiments to measure the ef-

fect of mutual information estimates on representation

quality. Our experiments show that the relationship

between mutual information and views is a subtle one.

•

Our representations rival state of the art on popular

benchmarks.

•

We demonstrate that the contrastive objective is supe-

rior to cross-view prediction.

2. Related work

Unsupervised representation learning is about learning

transformations of the data that make subsequent problem

solving easier [

]. This ﬁeld has a long history, starting with

classical methods with well established algorithms, such as

principal components analysis (PCA [37]) and independent

components analysis (ICA [

]). These methods tend to

learn representations that focus on low-level variations in

the data, which are not very useful from the perspective of

downstream tasks such as object recognition.

Representations better suited to such tasks have been

learnt using deep neural networks, starting with seminal

techniques such as Boltzmann machines [

], autoen-

coders [

], variational autoencoders [

], generative adver-

sarial networks [

] and autoregressive models [

]. Nu-

merous other works exist, for a review see [

]. A power-

ful family of models for unsupervised representations are

collected under the umbrella of “self-supervised” learning

[

]. In these models, an input

the model is transformed into an output

, which is sup-

posed to be close to another signal

(usually in Euclidean

space), which itself is related to

in some meaningful way.

Examples of such

X/Y

pairs are: luminance and chromi-

nance color channels of an image [

], patches from a single

image [

], modalities such as vision and sound [

] or the

frames of a video [

]. Clearly, such examples are numerous

in the world, and provides us with nearly inﬁnite amounts

of training data: this is one of the appeals of this paradigm.

Time contrastive networks [

] use a triplet loss framework

to learn representations from aligned video sequences of

the same scene, taken by different video cameras. Closely

related to self-supervised learning is the idea of multi-view

learning, which is a general term involving many different

approaches such as co-training [

], multi-kernel learning

[

] and metric learning [

]; for comprehensive surveys

please see [

]. Nearly all existing works have dealt with

one or two views such as video or image/sound. However, in

many situations, many more views are available to provide

training signals for any representation.

The objective functions used to train deep learning based

representations in many of the above methods are either

reconstruction-based loss functions such as Euclidean losses

in different norms e.g. [

], adversarial loss functions [

]

that learn the loss in addition to the representation, or con-

trastive losses e.g. [

] that

take advantage of the co-occurence of multiple views.

Some of the prior works most similar to our own (and

inspirational to us) are Contrastive Predictive Coding (CPC)

[57], Deep InfoMax [31], and Instance Discrimination [79].

These methods, like ours, learn representations by contrast-

ing between congruent and incongruent representations of

a scene. CPC learns from two views – the past and future –

and is applicable to sequential data, either in space or in time.

Deep Infomax [

] considers the two views to be the input

to a neural network and its output. Instance Discrimination

learns to match two sub-crops of the same image. CPC and

Deep InfoMax have recently been extended in [

] and [

]

respectively. These methods all share similar mathematical

objectives, but differ in the deﬁnition of the views. Our

method differs from these works in the following ways: we

extend the objective to the case of more than two views, and

we explore a different set of view deﬁnitions, architectures,

and application settings. In addition, we contribute a unique

empirical investigation of this paradigm of representation

learning.

The idea of contrastive learning has also started to spread

over many other tasks in various other domains [

76, 48, 38, 75].

3. Method

Our goal is to learn representations that capture informa-

tion shared between multiple sensory views without human

supervision. We start by reviewing previous predictive learn-

ing (or reconstruction-based learning) methods, and then

elaborate on contrastive learning within two views. We show

connections to mutual information maximization and extend

it to scenarios including more than two views. We consider

a collection of

views of the data, denoted as

,...,V

For each view

, we denote

as a random variable repre-

senting samples following v

⇠P(V

3.1. Predictive Learning

Let

and

represent two views of a dataset. For in-

stance,

might be the luminance of a particular image and

the chrominance. We deﬁne the predictive learning setup

as a deep nonlinear transformation from

through la-

tent variables

, as shown in Fig. 2. Formally,

z = f(v

)

and

ˆv

= g(z)

, where

and

represent the encoder and decoder

respectively and ˆv

is the prediction of v

given v

. The pa-

rameters of the encoder and decoder models are then trained

using an objective function that tries to bring

ˆv

“close to”

. Simple examples of such an objective include the

loss functions. Note that these objectives assume in-

dependence between each pixel or element of

given

Figure 2: Predictive Learning vs Contrastive Learning. Cross-view

prediction (

Top

) learns latent representations that predict one view

from another, with loss measured in the output space. Common

prediction losses, such as the

and

norms, are unstructured, in

the sense that they penalize each output dimension independently,

perhaps leading to representations that do not capture all the shared

information between the views. In contrastive learning (

Bottom

representations are learnt by contrasting congruent and incongruent

views, with loss measured in representation space. The red dotted

outlines show where the loss function is applied.

i.e.,

p(v

)=⇧

p(v

)

, thereby reducing their ability

to model correlations or complex structure. The predictive

approach has been extensively used in representation learn-

ing, for example, colorization [

] and predicting sound

from vision [58].

3.2. Contrastive Learning with Two Views

The idea behind contrastive learning is to learn an embed-

ding that separates (contrasts) samples from two different

distributions. Given a dataset of

and

that consists of

a collection of samples

}

i=1

, we consider contrast-

ing congruent and incongruent pairs, i.e. samples from the

joint distribution

x ⇠ p(v

)

x = {v

}

, which we

call positives, versus samples from the product of marginals,

y ⇠ p(v

)p(v

) or y = {v

}, which we call negatives.

We learn a “critic” (a discriminating function)

✓

(·)

which is trained to achieve a high value for positive pairs and

low for negative pairs. Similar to recent setups for contrastive

learning [

], we train this function to correctly select

a single positive sample

out of a set

S = {x, y

,...,y

}

that contains k negative samples:

contrast

= E

log

✓

(x)

✓

(x)+

i=1

✓

)

(1)

To construct

, we simply ﬁx one view and enumerate pos-

itives and negatives from the other view, allowing us to

､ …!

”

rewrite the objective as:

contrast

=  E

,...,v

k+1

}

log

✓

({v

})

k+1

j=1

✓

({v

})

(2)

where

is the number of negative samples

for a given

sample

. In practice,

can be extremely large (e.g., 1.2

million in ImageNet), and so directly minimizing Eq. 2 is

infeasible. In Section 3.4, we show two approximations that

allow for tractable computation.

Implementing the critic

We implement the critic

✓

(·)

a neural network. To extract compact latent representations

and

, we employ two encoders

✓

(·)

and

✓

(·)

with

parameters

✓

and

✓

respectively. The latent representions

are extracted as

= f

✓

)

= f

✓

)

. We compute

their cosine similarity as score and adjust its dynamic range

by a hyper-parameter ⌧:

✓

({v

})=exp(

✓

) · f

✓

)

✓

)k·kf

✓

⌧

) (3)

Loss

contrast

in Eq. 2 treats view

as anchor and

enumerates over

. Symmetrically, we can get

contrast

by anchoring at V

. We add them up as our two-view loss:

L(V

)=L

contrast

+ L

contrast

(4)

After the contrastive learning phase, we use the representa-

tion

, or the concatenation of both,

]

, depending

on our paradigm. This process is visualized in Fig. 1.

Connecting to mutual information

The optimal critic

⇤

✓

is proportional to the density ratio between the joint distri-

bution

p(z

)

and the product of marginals

p(z

)p(z

)

(proof provided in supplementary material):

⇤

✓

({v

}) /

p(z

)

p(z

)p(z

)

p(z

)

p(z

)

(5)

This quantity is the pointwise mutual information, and its

expectation, in Eq. 2, yields an estimator related to mutual

information. A formal proof is given by [57, 62], which we

recapitulate in supplement, showing that:

I(z

; z

)  log(k) L

contrast

(6)

where, as above,

is the number of negative pairs in sample

set

. Hence minimizing the objective

maximizes the

lower bound on the mutual information

I(z

; z

)

, which is

bounded above by

I(v

; v

)

by the data processing inequality.

The dependency on

also suggests that using more negative

samples can lead to an improved representation; we show

that this is indeed the case (see supplement). We note that

recent work [

] shows that the bound in Eq. 6 can be very

weak; and ﬁnding better estimators of mutual information is

an important open problem.

(a) Core View

(b) Full Graph

<latexit sha1_base64="BTRfsG7MuUcz7Tb1mYDXTk3FGhE=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPRi8eK9gPaUDbbTbt0swm7E6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTKUw6HnfTmltfWNzq7xd2dnd2z9wD49aJsk0402WyER3Qmq4FIo3UaDknVRzGoeSt8Px7cxvP3FtRKIecZLyIKZDJSLBKFrpodX3+27Vq3lzkFXiF6QKBRp996s3SFgWc4VMUmO6vpdikFONgkk+rfQyw1PKxnTIu5YqGnMT5PNTp+TMKgMSJdqWQjJXf0/kNDZmEoe2M6Y4MsveTPzP62YYXQe5UGmGXLHFoiiTBBMy+5sMhOYM5cQSyrSwtxI2opoytOlUbAj+8surpHVR872af39Zrd8UcZThBE7hHHy4gjrcQQOawGAIz/AKb450Xpx352PRWnKKmWP4A+fzB9fVjX4=</latexit>

<latexit sha1_base64="NdtrmwDhkKHNNnS1oER0xSZxDQA=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmnXqp5b9e6vKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDZWY1/</latexit>

<latexit sha1_base64="OZ7/eGjiaAd6iFmhE9KemdqLXmM=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0m0oMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh1b/sl+uuFV3DrJKvJxUIEejX/7qDWKWRlwhk9SYrucm6GdUo2CST0u91PCEsjEd8q6likbc+Nn81Ck5s8qAhLG2pZDM1d8TGY2MmUSB7YwojsyyNxP/87ophtd+JlSSIldssShMJcGYzP4mA6E5QzmxhDIt7K2EjaimDG06JRuCt/zyKmldVD236t3XKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDa3Y2A</latexit>

<latexit sha1_base64="RGbnccGESBHMfrtDKDZHJvcqDa8=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkUI9FLx4r2g9oQ9lsJ+3SzSbsboQS+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ7dzvPKHSPJaPZpqgH9GR5CFn1FjpoT2oDcoVt+ouQNaJl5MK5GgOyl/9YczSCKVhgmrd89zE+BlVhjOBs1I/1ZhQNqEj7FkqaYTazxanzsiFVYYkjJUtachC/T2R0UjraRTYzoiasV715uJ/Xi814bWfcZmkBiVbLgpTQUxM5n+TIVfIjJhaQpni9lbCxlRRZmw6JRuCt/ryOmlfVT236t3XKo2bPI4inME5XIIHdWjAHTShBQxG8Ayv8OYI58V5dz6WrQUnnzmFP3A+fwDcYY2B</latexit>

<latexit sha1_base64="UQ8RRPt2NN02WZInLZUJLciw6fw=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbRqVc+teg9XlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwAKKI2f</latexit>

<latexit sha1_base64="fHYLD/bLdDeenuRAHuRzLEbLywI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lU0GPRi8eK9gPaUDbbSbt0swm7m0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU1HGqGDZYLGLVDqhGwSU2DDcC24lCGgUCW8Hobua3xqg0j+WTmSToR3QgecgZNVZ6HPcue+WKW3XnIKvEy0kFctR75a9uP2ZphNIwQbXueG5i/Iwqw5nAaambakwoG9EBdiyVNELtZ/NTp+TMKn0SxsqWNGSu/p7IaKT1JApsZ0TNUC97M/E/r5Oa8MbPuExSg5ItFoWpICYms79JnytkRkwsoUxxeythQ6ooMzadkg3BW355lTQvqp5b9R6uKrXbPI4inMApnIMH11CDe6hDAxgM4Ble4c0Rzovz7nwsWgtOPnMMf+B8/gALrI2g</latexit>

<latexit sha1_base64="+c4psOKK6zsb24Fz44doEjLIVSY=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mkoMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0m/1i9X3Kq7AFknXk4qkKPRL3/1BjFLI66QSWpM13MT9DOqUTDJZ6VeanhC2ZgOeddSRSNu/Gxx6oxcWGVAwljbUkgW6u+JjEbGTKPAdkYUR2bVm4v/ed0Uwxs/EypJkSu2XBSmkmBM5n+TgdCcoZxaQpkW9lbCRlRThjadkg3BW315nbSuqp5b9R5qlfptHkcRzuAcLsGDa6jDPTSgCQyG8Ayv8OZI58V5dz6WrQUnnzmFP3A+fwANMI2h</latexit>

<latexit sha1_base64="iwb6UOVPBQMaW/UYkl1KcM9pxNQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48V7Qe0oWy2m3bpZhN2J4US+hO8eFDEq7/Im//GbZuDtj4YeLw3w8y8IJHCoOt+O4WNza3tneJuaW//4PCofHzSMnGqGW+yWMa6E1DDpVC8iQIl7ySa0yiQvB2M7+Z+e8K1EbF6wmnC/YgOlQgFo2ilx0nf65crbtVdgKwTLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjks1IvNTyhbEyHvGupohE3frY4dUYurDIgYaxtKSQL9fdERiNjplFgOyOKI7PqzcX/vG6K4Y2fCZWkyBVbLgpTSTAm87/JQGjOUE4toUwLeythI6opQ5tOyYbgrb68TlpXVc+teg/XlfptHkcRzuAcLsGDGtThHhrQBAZDeIZXeHOk8+K8Ox/L1oKTz5zCHzifPwikjZ4=</latexit>

Figure 3: Graphical models and information diagrams [

] associ-

ated with the core view and full graph paradigms, for the case of

views, which gives a total of

learning objectives. The numbers

within the regions show how much “weight” the total loss places on

each partition of information (i.e. how many of the 6 objectives that

partition contributes to). A region with no number corresponds to

weight. For example, in the full graph case, the mutual information

between all

views is considered in all

objectives, and hence is

marked with the number 6.

3.3. Contrastive Learning with More than Two

Views

We present more general formulations of Eq. 2 that can

handle any number of views. We call them the “core view”

and “full graph” paradigms, which offer different tradeoffs

between efﬁciency and effectiveness. These formulations

are visualized in Fig. 3.

Suppose we have a collection of

views

,...,V

The “core view” formulation sets apart one view that we want

to optimize over, say

, and builds pair-wise representations

between

and each other view

,j > 1

, by optimizing

the sum of a set of pair-wise objectives:

j=2

L(V

) (7)

A second, more general formulation is the “full graph” where

we consider all pairs

(i, j),i6= j

, and build





relationships

in all. By involving all pairs, the objective function that we

optimize is:

1i<jM

L(V

) (8)

Both these formulations have the effect that information is

prioritized in proportion to the number of views that share

that information. This can be seen in the information dia-

grams visualized in Fig. 3. The number in each partition of

the diagram indicates how many of the pairwise objectives,

L(V

)

, that partition contributes to. Under both the core

剩余15页未读，继续阅读

评论收藏

内容反馈

Contrastive Multiview Coding笔记

评论0

最新资源

Contrastive Multiview Coding笔记

评论0

最新资源

相关推荐

颜色分类leetcode-CMC:[ECCV2020]“对比多视图编码”，还包含MoCo和InstDis的实现

[论文阅读笔记]Contrastive Boundary Learning for Point Cloud Segment

Max-Margin Contrastive Learning.pdf

PyPI 官网下载 | contrastive_learner-0.0.7.tar.gz

Algorithm-Noise-Contrastive-Estimation-NCE-for-pyTorch.zip

Contextualized Spatio-Temporal Contrastive Learning with Self-S

Python库 | pixel-level-contrastive-learning-0.0.1.tar.gz

CONTRASTIVE LEARNING OF STRUCTURED WORLD MODELS

Contrastive Representation Distillation

Contrastive Self-supervised Learning for Graph Classification

VideoMoCo Contrastive Video Representation Learning With.pdf

Contrastive Data and Learning for Natural Language Processing.pd

Article Analysis(AA): A Simple Framework for Contrastive Learning of Visual Representations

A Contrastive Study of Euphemisms in English and Chinese.zip

PyPI 官网下载 | contrastive-0.1.0.tar.gz

contrastive-learner:一个简单易用的pytorch包装器，可在任何神经网络上进行对比式自我监督学习

Adaptive-Soft-Contrastive-Learning_ICPR2022-main.zip

再介绍一篇最新的Contrastive Self-supervised Learning综述论文 .rar

第十五届蓝桥杯大赛软件赛省赛-C/C++ 研究生组

单链表的逆置的概要介绍与分析

VRPTW 的 Solomon 标准测试数据集

数学建模国赛：无人机遂行编队飞行中的纯方位无源定位分析

最值得收藏的 数据结构 全部知识点思维导图整理(王道考研), 附带经典题型整理.emmx

利用SVM（支持向量机）进行图像分割/提取-MATLAB

PSG 3D 三维测绘系统

变分模态分解（VMD）代码

最新版Notepad++十六进制查看的插件x64HexEditor0.9.12

origin2021下载免费分享

最值得收藏的数据结构全部知识点思维导图整理(王道考研), 附带经典题型整理.emmx