The CrackNet, an efficient architecture based on the Convolutional Neural Network (CNN), is proposed in this article for automated pavement crack detection on 3D asphalt surfaces with explicit objective of pixel‐perfect accuracy. Unlike the commonly used CNN, CrackNet does not have any pooling layers which downsize the outputs of previous layers. CrackNet fundamentally ensures pixel‐perfect accuracy using the newly developed technique of invariant image width and height through all layers. CrackNet consists of five layers and includes more than one million parameters that are trained in the learning process. The input data of the CrackNet are feature maps generated by the feature extractor using the proposed line filters with various orientations, widths, and lengths. The output of CrackNet is the set of predicted class scores for all pixels. The hidden layers of CrackNet are convolutional layers and fully connected layers. CrackNet is trained with 1,800 3D pavement images and is then demonstrated to be successful in detecting cracks under various conditions using another set of 200 3D pavement images. The experiment using the 200 testing 3D images showed that CrackNet can achieve high Precision (90.13%), Recall (87.63%) and F‐measure (88.86%) simultaneously. Compared with recently developed crack detection methods based on traditional machine learning and imaging algorithms, the CrackNet significantly outperforms the traditional approaches in terms of F‐measure. Using parallel computing techniques, CrackNet is programmed to be efficiently used in conjunction with the data collection software.