The heterogeneous deformation of polycrystalline metals inherently originates from the intergranular deformation incompatibility. This paper proposes physical parameters related to the crystal orientations, the Schmid factor of the most activated slip system, and the misorientation angle to characterize the deformation incompatibility between the adjacent grains couple. A comprehensive multiscale investigation is conducted to reveal the mechanism from intergranular deformation incompatibility to fracture initiation at grain boundaries. At the specimen scale, experimental and numerical uniaxial tensile tests are performed on smooth and pre-notched dog-bone specimens to achieve different loading paths on the materials. The heterogeneous fields of stress triaxiality explains the heterogeneous size of the dimples observed in fractography. At the grain scale, electron backscatter diffraction analysis is conducted to characterize the microstructural properties around the nucleated voids within the materials. Voids are captured at the grain boundaries with directions parallel to the loading direction and intergranular deformation incompatibility is characterized using the proposed parameters. Simulations on the plastic deformation of realistic microstructures are performed to clarify the phenomenon. The results reveal that the fluctuation in stress triaxiality at grain boundaries is ascribed to intergranular deformation incompatibility, leading to fracture initiation at these sites. The relationships between the proposed physical parameters of intergranular deformation incompatibility and fluctuation in stress triaxiality are summarized in all circumstances. Finally, the ductile damage at the grain scale is predicted by the Rice-Tracey model, and the results show that the effects of microstructures on heterogeneous plastic deformation and stress state can be well considered. Display omitted •Intergranular deformation incompatibility is characterized by physical parameters.•Heterogeneous distribution of voids at grain boundary is captured.•Intergranular deformation incompatibility causes high triaxiality at grain boundary.•Ductile damage in the microstructure is predicted by Rice-Tracey model.