•An integrated CAD system of deep learning detection and classification is proposed aiming to improve the diagnostic performance of breast lesions from the entire digital X-ray mammograms.•Breast lesions are accurately detected via YOLO detector, with F1-scores of 99.28% for DDSM and 98.02% for INbreast.•The capability of the YOLO detector boosted the modified InceptionResNet-V2 classifier achieving promising diagnosis performance with overall accuracies of 97.50% for DDSM and 95.32% for INbreast.•The proposed deep learning CAD system is able to detect and classify breast lesions in a single mammogram in less than 0.025 s. Deep learning detection and classification from medical imagery are key components for computer-aided diagnosis (CAD) systems to efficiently support physicians leading to an accurate diagnosis of breast lesions. In this study, an integrated CAD system of deep learning detection and classification is proposed aiming to improve the diagnostic performance of breast lesions. First, a deep learning YOLO detector is adopted and evaluated for breast lesion detection from entire mammograms. Then, three deep learning classifiers, namely regular feedforward CNN, ResNet-50, and InceptionResNet-V2, are modified and evaluated for breast lesion classification. The proposed deep learning system is evaluated over 5-fold cross-validation tests using two different and widely used databases of digital X-ray mammograms: DDSM and INbreast. The evaluation results of breast lesion detection show the capability of the YOLO detector to achieve overall detection accuracies of 99.17% and 97.27% and F1-scores of 99.28% and 98.02% for DDSM and INbreast datasets, respectively. Meanwhile, the YOLO detector could predict 71 frames per second (FPS) at the testing time for both DDSM and INbreast datasets. Using detected breast lesions, the classification models of CNN, ResNet-50, and InceptionResNet-V2 achieve promising average overall accuracies of 94.50%, 95.83%, and 97.50%, respectively, for the DDSM dataset and 88.74%, 92.55%, and 95.32%, respectively, for the INbreast dataset. The capability of the YOLO detector boosted the classification models to achieve a promising breast lesion diagnostic performance. Such prediction results should help to develop a feasible CAD system for practical breast cancer diagnosis. Display omitted