Classifying concrete defects during a bridge inspection remains a subjective and laborious task. The risk of getting a false result is approximately 50% if different inspectors assess the same concrete defect. This is significant in the light of an over-aging bridge stock, decreasing infrastructure maintenance budgets and catastrophic bridge collapses as happened in 2018 in Genoa, Italy. To support an automated inspection and an objective bridge defect classification, we propose a three-staged concrete defect classifier that can multi-classify potentially unhealthy bridge areas into their specific defect type in conformity with existing bridge inspection guidelines. Three separate deep neural pre-trained networks are fine-tuned based on a multi-source dataset consisting of self-collected image samples plus several Departments of Transportation inspection databases. We show that this approach can reliably classify multiple defect types with an average mean score of 85%. Our presented multi-classifier is a contribution towards developing a mostly or fully inspection schema for a more cost-effective and more objective bridge inspection. •The presented method can automatically multi-classify concrete bridge defects on image patches in accordance with existing inspection guidelines.•The cross-learning strategy of using a pre-trained network and refine this to domain-specific knowledge can be successfully applied for concrete bridge surface defects.•The multi-classifier can adapt to local variations and consider possible (and impossible) defect combinations.