Please make sure to register to Dropbox to edit this document!
Date:June 27, 2020
Title: A Simple Framework for Contrastive Learning of Visual Representations
Authors: Ting Chen, Simon Kornblith, Mohammad Norouzi, Geoffrey Hinton
Abstract:This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We simplify recently proposed contrastive self-supervised learning algorithms without requiring specialized architectures or a memory bank. In order to understand what enables the contrastive prediction tasks to learn useful representations, we systematically study the major components of our framework. We show that(1) composition of data augmentations plays a critical role in defining effective predictive tasks,(2) introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and(3) contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning. By combining these findings, we are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet. A linear classifier trained on self-supervised representations learned by SimCLR achieves 76.5% top-1 accuracy, which is a 7% relative improvement over previous state-of-the-art, matching the performance of a supervised ResNet-50. When fine-tuned on only 1% of the labels, we achieve 85.8% top-5 accuracy, outperforming AlexNet with 100X fewer labels.
While reading the paper, we encourage you to post your notes/comments/summaries of what you understood from the paper(Use the sections below to determine where notes/comments/summaries should go). You can also include your questions below.
Discussion 🤓
Introduction ⚡
Discuss the motivation and objectives of this paper at a high level. As we read through the paper we can all take notes on the points we found important to emphasize and have further discussion about.
Contrastive learning is a framework that learns similar/dissimilar representations from data that are organized into similar/dissimilar pairs.
Contrastive methods aim to learn representations by enforcing similar elements to be equal and dissimilar elements to be different.
Annotating data is tedious and costly. Meanwhile, unlabeled data is widely available or can be collected with little effort. Self-supervised learning aims to automate the labeling process and leverage the abundance of unlabeled data.
If we can understand what enables contrastive learning we can improve those components for better model.
Agenda/Housekeeping
Discussion 🤓