In order to relieve the problem of a false and missed detection of casting defects in X-ray detection, a robust detection method based on vision attention mechanism and deep learning of feature map is proposed. The ray images are used as input sequence, the false detection is eliminated by the intra-frame attention strategy, and the missed detection is excluded by the inter-frame deep convolution neural network (DCNN) strategy. In the intra-frame detection stage, the center-peripheral difference method is proposed to simulate the difference operation of biological vision; the suspicious defect area is directly detected according to the gradient threshold in this stage. In the inter-frame learning stage, the convolution neural network is established based on deep learning strategy to extract defect feature from a suspicious defect area; a deep learning feature vector is obtained in this stage. The similarity degree of the suspicious defect area is computed by a feature vector; a casting defect is tracked by the similarity matching of the suspicious defect in continuous frames; then, the false defects (such as noise) is excluded after defect tracking. The experimental results show that the false rate and missed rate for detection of casting defects are less than 4%, and the accuracy of the defect detection is more than 96%, which proves the robustness of the proposed method.