This work is focused on a quantitative procedure for estimating the generally unfavourable effects that incorrectly drilled holes, characterized by the initial clearance between a rivet and a hole, have on the fatigue life of riveted joints. The solution is based on an analytical approach using the stress-severity-factor concept. An experimental programme with riveted-joint specimens characterized by low-load transfer factors was realized in the Czech Aerospace Research Centre (VZLU) test lab under constant amplitude loading. The holes for rivet joints with 4-mm diameters were prepared with the clearance in a range of 0.0–0.16 mm. Force-controlled riveting was applied using a constant pressure force to form the driven head. To prevent fretting events between the joined parts, their anodized contact surfaces were lubricated with MOLYKA, plastic grease with molybdenum disulphide and graphite. The experimental data showed that the load-transfer factor and the fatigue life depend on the initial clearance between a rivet and a hole. The presented procedure introduced the hole-filling factor, integrated in the stress-severity-factor concept as a function of the initial clearance between a rivet and a hole.