•A new stress invariant based isotropic ductile fracture criterion is developed.•The criterion is further extended to describe the direction dependency of ductile fracture.•A new out-of-plane shear specimen is proposed to study the through-thickness property of the sheet.•Hybrid experimental-numerical method is used to determine fracture strains from in-plane to out-of-plane.•The new criterion is able to predict the anisotropic fracture behaviors of a DP980 sheet with high accuracy. Sheet metals usually exhibit varying degrees of anisotropy due to formation of texture during rolling process. The grain structures and distributions in sheet normal direction may also be different from that within the sheet plane. For example, martensite banding structures are usually observed in the middle of thickness for advanced high strength steels. The particular martensite morphology and inhomogeneous distribution will greatly affect the deformation and fracture behavior under out-of-plane direction. It is necessary to model the anisotropic fracture of sheet metals from in-plane to out-of-plane. In this work, a new stress invariant based ductile fracture criterion was developed by introducing a stress triaxiality and normalized third invariant dependent function h(η, ξ) and further extended to an anisotropic form through the linear transformation of stress tensor. Parametric study showed that the new anisotropic criterion can describe the direction dependency of ductile fracture in both strain and stress spaces. A series of in-plane fracture tests including tension with a central hole, notched tension, V-bending test, Nakajima test and in-plane shear as well as a new out-of-plane shear test were conducted to study the anisotropic fracture behavior of DP980 sheet metals over a variety of stress states. Hybrid experimental-numerical method is used to determine the fracture strains and the loading paths to fracture. The results show that slight in-plane anisotropy of fracture responses exists for this DP980 sheet, while the out-of-plane shear fracture stress is approximately 11% lower than that under the in-plane shear condition. The new anisotropic ductile fracture criterion is able to predict the anisotropic fracture behaviors of this DP980 sheet with high accuracy.