Pathogenic mutations in BRCA1 and BRCA2 are associated with high risks of breast and ovarian cancer. However, penetrance estimates for mutation carriers have been found to vary substantially between studies, and the observed differences in risk are consistent with the hypothesis that genetic and environmental factors modify cancer risks for women with these mutations. Direct evidence that this is the case has emerged in the past decade, through large-scale international collaborative efforts. Here, we describe the methodological challenges in the identification and characterisation of these risk-modifying factors, review the latest evidence on genetic and lifestyle/hormonal risk factors that modify breast and ovarian cancer risks for women with BRCA1 and BRCA2 mutations and outline the implications of these findings for cancer risk prediction. We also review the unresolved issues in this area of research and identify strategies of clinical implementation so that women with BRCA1 and BRCA2 mutations are no longer counselled on the basis of ‘average’ risk estimates.