Nowadays, the physicians usually predict functional outcomes of stroke based on clinical experiences and big data, so we wish to develop a model to accurately identify imaging features for predicting functional outcomes of stroke patients. Using magnetic resonance imaging of ischemic and hemorrhagic stroke patients, we developed and trained a VGG-16 convolutional neural network (CNN) to predict functional outcomes after 28-day hospitalization. A total of 44 individuals (24 men and 20 women) were recruited from Taoyuan General Hospital and China Medical University Hsinchu Hospital to enroll in the study. Based on “modified Rankin Scale (mRS)” and “National Institutes of Health Stroke Scale (NIHSS)” assessments, men, women, and mixed men and women were trained separately to evaluate the differences of the results, and we have shown that VGG-16 demonstrated high accuracy in predicting the functional outcomes of stroke patients. The new deep-learning approach has provided an automated decision support system for personalized recommendations and treatments, assisting the physicians to predict functional outcomes of stroke patients in clinical practice. Graphical abstract