•A Grapevine yellows (GY) disease detection system was developed.•It can distinguish GY disease from other diseases with similar symptoms.•Six neural network architectures were utilized and evaluated.•This automatic tool is crucial to avoid missing GY-positive plants.•It offers an end-to-end detection of GY. Grapevine yellows (GY) are a significant threat to grapes due to the severe symptoms and lack of treatments. Conventional diagnosis of the phytoplasmas associated to GYs relies on symptom identification, due to sensitivity limits of diagnostic tools (e.g. real time PCR) in asymptomatic vines, where the low concentration of the pathogen or its erratic distribution can lead to a high rate of false-negatives. GY’s primary symptoms are leaf discoloration and irregular wood ripening, which can be easily confused for symptoms of other diseases making recognition a difficult task. Herein, we present a novel system, utilizing convolutional neural networks, for end-to-end detection of GY in red grape vine (cv. Sangiovese), using color images of leaf clippings. The diagnostic test detailed in this work does not require the user to be an expert at identifying GY. Data augmentation strategies make the system robust to alignment errors during data capture. When applied to the task of recognizing GY from digital images of leaf clippings—amongst many other diseases and a healthy control—the system has a sensitivity of 98.96% and a specificity of 99.40%. Deep learning has 35.97% and 9.88% better predictive value (PPV) when recognizing GY from sight, than a baseline system without deep learning and trained humans respectively. We evaluate six neural network architectures: AlexNet, GoogLeNet, Inception v3, ResNet-50, ResNet-101 and SqueezeNet. We find ResNet-50 to be the best compromise of accuracy and training cost. The trained neural networks, code to reproduce the experiments, and data of leaf clipping images are available on the internet. This work will advance the frontier of GY detection by improving detection speed, enabling a more effective response to the disease.