Grain refinement is a very effective method to improve the mechanical properties of materials and attracts widespread interests among researchers. However, the grain size effect on the mechanical properties is still unclear due to the undesirable microstructure in ultrafine grained (UFG) materials. In the present work, series of ideal materials with average grain sizes range from 0.7 μm to 30.0 μm containing high fraction of high angle grain boundaries (HAGBs), equiaxed grains and low density of dislocations were produced by friction stir processing (FSP). It was found that the Hall-Petch relationships could be classified into three stages as the grain size reduced from coarse grain to UFG regimes, which were decided by the strengthening mechanisms during tensile deformation. The strengthening effect of HAGBs (71 MPa⋅μm1/2) was almost three times of low angle grain boundaries (25 MPa⋅μm1/2), resulting in the positive deviation of Hall-Petch slope in fine grain regime by the increased specific surface area of HAGBs. The further positive deviation of the Hall-Petch slope in UFG regime was affected by the occurrence of an extra dislocation source limited strengthening mechanism, which was up to 29 MPa and reached to about 20% of the yield strength. The increased recovery rate of dislocations at HAGBs contributed to the decrease of mobile dislocation density, leading to the losing of work hardening in UFG regime during tensile deformation.