Structural components with different scales normally show different fatigue behaviors, which are virtually dominated by defects originated from multiple sources, including manufacturing processes. This paper reviews three types of size effects (statistical, geometrical, technological) as well as their recent advances in metal fatigue, aiming to provide a guide for fatigue strength assessment of engineering components containing defects, inclusions and material inhomogeneity. Firstly, the background of inherent defects and defect-based failure mechanism are briefly outlined, and fatigue failure analysis based on fracture mechanics as well as statistics theory are emphasized. Then, two approaches commonly applied in statistical size effect modeling, i.e. critical defect method and weakest link method, are elaborated. In addition, the highly stressed volume method is introduced for considering the geometrical size effects, and the technological (production and surface) size effect is briefly overviewed. Finally, further directions on size effect in metal fatigue under defects are explored.