Deep learning applied to computer vision has different applications in agriculture, medicine, marketing, meteorology, etc. In agriculture, plant diseases can cause significant yield and quality losses. The treatment of these diseases depends on accurate and rapid classification. Olive leaf diseases are a problem that threatens the crop quality of olive growers. The objective of this work was to classify olive leaf diseases with Deep Learning in olive crops of the La Yarada-Los Palos area in the Tacna region, Peru. Disease classification is a critical task, nevertheless, for the most common diseases in the region: virosis, fumagina, and nutritional deficiencies, there is no dataset to train deep learning models. Due to the latter, a novel dataset of RGB olive leaf images is elaborated and published. Then, an extensive comparative ex-perimental study was conducted using all possible configurations of Learning from Scratch, Transfer Learning, Fine-Tuning, and Data Augmentation state-of-the-art methods to train a modified VGG16 architecture for the classification of Olive Leaf Diseases. It was demonstrated experimentally: (i) The ineffectiveness of Data Augmentation when the model Learning from Scratch, (ii) A high improvement by using Transfer Learning vs Learning from Scratch, (iii) Similar performance using Transfer Learning vs Transfer Learning + Fine-Tuning vs Transfer Learning + Data Augmentation, and (iv) Very high improvement using Transfer Learning + Fine-Tuning + Data Augmentation. This led us to a Deep Learning Model with an accuracy of 100%, 99.93%, and 100% in the training, validation, and test sets and F1-Score on the validation set of 1, 0.9901, and 0.9899 in the Nutritional Deficiences, Fumagina, and Virosis olive leaf diseases respectively. Replication of the results is ensured by publishing the novel dataset and the final model on GitHub.