The deck pavement, as an important structure of the bridge, cushions the wheel actions on the bridge, prevents the main girder from rain erosion, and ensures the flatness and slip resistance for vehicles. The asphalt concrete has been widely used in bridge decks. However, the conventional crack detection methods cannot identify the defects on asphalt concrete bridge deck accurately and efficiently, due to the dark color of the deck and the complexity, different types of defects. The objective of this paper is to develop a weakly supervised network for the segmentation and detection of cracks in asphalt concrete deck. Firstly, the data were differentiated by the autoencoder, and the unlabeled data features were highlighted, so that the original data autonomously generate a weakly supervised start point for convergence. Secondly, the features were classified by k-means clustering (KMC). Thirdly, the cracks in the bridge deck defects images were subjected to semantic segmentation under weak supervision. A dataset of six types of defects on asphalt concrete bridge deck which was set up the defects in the dataset were labeled manually. The experimental results show that the proposed achieved outstanding segmentation effects on all six types of defects was better than the other existed methods reported in the references.