This paper mainly improves the visual geometry group network-16 (VGG-16), which is a classic convolutional neural network (CNN), to classify the surface defects on cement concrete bridges in an accurate manner. Specifically, the number of fully connected layers was reduced by one, and the Softmax classifier was replaced with a Softmax classification layer with seven defect tags. The weight parameters of convolutional and pooling layers were shared in the pre-trained model, and the rectified linear unit (ReLU) function was taken as the activation function. The original images were collected by a road inspection vehicle driving across bridges on national and provincial highways in Jiangxi Province, China. The images on surface defects of cement concrete bridges were selected, and divided into a training set and a test set, and preprocessed through morphology-based weight adaptive denoising. To verify its performance, the improved VGG-16 was compared with traditional shallow neural networks (NNs) like the backpropagation neural network (BPNN), support vector machine (SVM), and deep CNNs like AlexNet, GoogLeNet, and ResNet on the same sample dataset of surface defects on cement concrete bridges. Judging by mean detection accuracy and top-5 accuracy, our model outperformed all the contrastive methods, and accurately differentiated between images with seven classes of defects such as normal, cracks, fracturing, plate fracturing, corner rupturing, edge/corner exfoliation, skeleton exposure, and repairs. The results indicate that our model can effectively extract the multi-layer features from surface defect images, which highlights the edges and textures. The research findings shed important new light on the detection of surface defects and classification of defect images.