•Effects of microstructure and defect on fatigue behavior in Ti-6Al-4V is quantitatively characterized.•A modelling strategy is designed to highlight notch morphology/orientation effect in Ti-6Al-4V.•Roundness for notch morphology is linearly correlated with fatigue accumulation around defect.•Corner angle defined for notch orientation reflects the heterogeneous features of microstructure. Fatigue accumulation in dual-phase Ti-6Al-4V is strongly associated with the combined effects of microstructure and defects in itself, which is experimentally challenging to be characterized and quantified. Based on crystal plasticity finite element method (CPFEM), the present work aims to address the coordinated effect of microstructure and defect on fatigue behavior in Ti-6Al-4V, highlighting the influence of morphology and orientation of defects, in conjunction with the designated idealized microstructure. In light of these considerations, a modelling strategy is proposed to focus on defect morphology and orientation in polycrystalline RVE models. Firstly, the geometric roundness parameter is adopted to characterize the diverse defect morphology and is quantitatively correlated with the characteristic of fatigue accumulation in Ti-6Al-4V. Secondly, a liner relationship with respect to corner angle and plastic deformation is formulated to characterize the influence of defect orientation, reflecting the microstructural heterogeneity in dual-phase Ti-6Al-4V. The proposed modelling strategy overcomes the experimental limitation on characterizing the complex effect of microstructure and defect, holding the potential of revealing fatigue mechanism around defects in metal alloys.