Microstructural aspects have fundamental influences on the fatigue crack characteristics of materials. In this paper, effects of inclusions, grain boundaries (GBs) and grain orientations on the fatigue crack initiation and propagation behavior in a 2524-T3 aluminum alloy have been investigated using in-situ scanning electron microscope (SEM) fatigue testing and electron back scattering diffraction (EBSD). The results show that, potential fatigue cracks tend to nucleate along coarse and closely spaced inclusion particles or high-angle GBs. Coarse inclusion particles drastically accelerate local crack growth rates. A model of series crack growing stages is given based on the observation of initiation and growth of cracks at the inclusion region. GBs serve to impede the crack tip from propagation and cause large angle crack deflections, which greatly affects local crack propagation behaviors. In addition, fatigue crack shows a strong tendency to propagate transgranularly grains with high Schmid factors (SFs) and avoid grains with low SFs.