This article implements the state‐of‐the‐art deep learning technologies for a civil engineering application, namely recognition of structural damage from images. Inspired by ImageNet Challenge and the development of computer hardware, the concept of Structural ImageNet is proposed herein with four naïve baseline recognition tasks: component type identification, spalling condition check, damage level evaluation, and damage type determination. A relatively small number of images (2,000) are selected from the Structural ImageNet and manually labeled according to the four recognition tasks. In order to avoid overfitting, Transfer Learning (TL) based on VGGNet (Visual Geometry Group) is introduced and applied using two different strategies, namely feature extractor and fine‐tuning. Two experiments are designed based on properties of these two strategies to find the relative optimal model parameters and scope of application. Models obtained by both strategies indicate the promising recognition results and different application potentials where feature extractor and fine‐tuning can be respectively used for preliminary analysis and for further improvement. These results also reveal the potential uses of deep TL in image‐based structural damage recognition.