Fabric defect detection plays an important role in the textile production process, but there are still some challenges in detecting defects rapidly and accurately. In this paper, we propose a powerful detection method for automatic fabric defect detection using a deep convolutional neural network (CNN). It consists of three main steps. First, the fabric image is decomposed into local patches and each local patch is labelled. Then the labelled patches are transmitted to the pretrained deep CNN for transfer learning. Finally, defects are detected during the inspection phase by sliding over the whole image using the trained model, and the category and position of each defect is obtained. The proposed method is validated on two public and one self‐made fabric database. The experimental results demonstrate that our method significantly outperforms selected state‐of‐the art methods in terms of both quality and robustness.