•Break down of H-P effect.•GB absorption of deformation is the main mechanism.•The area of plastic zone depends on the GB. The hardness measured by the nanosize indenter under atomic indentation is examined for the cases of nanocrystalline nickel by means of molecular dynamics (MD) simulations. The grain size effect observed is different from the one by uniform deformation or deep indentation. The results show that hardness can only show inverse Hall–Petch (H-P) effect, no H-P effect is observed with the grain size up to 40nm. Grain boundary (GB) absorption of the localized strain is the main deformation mechanism when the indenter size and the depth both come to nano size. The area of plastic zone generated beneath the tip is strongly dependent on the GB density, sample with small grain size results in larger plastic area, which leads to the softer response of hardness.