Contrastive learning has achieved remarkable success in computer vision, however it is built on instance-level discrimination which leaves the valuable intra-class correlation in dataset unexploited. Current semantic clustering methods are proven to be helpful but they would suffer from the error accumulated in the iteration process without ground-truth guidance. In an attempt to remedy the clustering error accumulation when utilizing intra-class correlation for contrastive learning, we propose an online Contrastive Visual Clustering (CVC) method with two actions: gathering instances with highly similar feature embeddings, and penalizing instances being clustered with low confidence. CVC can integrate with not only contrastive learning but also arbitrary self-supervised learning frameworks simply as a plugin. Under various experiment settings, we show that CVC improves the linear classification performance by a large margin for models pre-trained with self-supervised representation learning, in both image and video scenarios. The code is available at https://github.com/yliu1229/CVC. •A contrastive clustering method CVC is proposed to improve contrastive learning.•CVC is shown to be a generic method.•Experiments show CVC improves linear classification performances by a large margin.