Purpose To identify factors associated with (perceived) access to health care among (ex-)breast cancer patients during the COVID-19 pandemic. Methods Cross-sectional study within a large prospective, multicenter cohort of (ex-)breast cancer patients, i.e., UMBRELLA. All participants enrolled in the UMBRELLA cohort between October 2013 and April 2020 were sent a COVID-19-specific survey, including the Hospital Anxiety and Depression Scale (HADS) questionnaire. Results In total, 1051 (66.0%) participants completed the survey. During COVID-19, 284 (27.0%) participants reported clinically relevant increased levels of anxiety and/or depression, i.e., total HADS score greater than or equal to 12. Participants with anxiety and/or depression reported statistically significant higher barriers to contact their general practitioner (47.5% vs. 25.0%, resp.) and breast cancer physicians (26.8% vs. 11.2%, resp.) compared to participants without these symptoms. In addition, a higher proportion of participants with anxiety and/or depression reported that their current treatment or (after)care was affected by COVID-19 compared to those without these symptoms (32.7% vs. 20.5%, resp.). Factors independently associated with symptoms of anxiety and/or depression during COVID-19 were pre-existent anxiety (OR 6.1, 95% CI 4.1-9.2) or depression (OR 6.0, 95% CI 3.5-10.2). Conclusion During the COVID-19 pandemic, (ex-)breast cancer patients with symptoms of anxiety and/or depression experience higher barriers to contact health care providers. Also, they more often report that their health care was affected by COVID-19. Risk factors for anxiety and/or depression during COVID-19 are pre-existent symptoms of anxiety or depression. Extra attention-including mental health support-is needed for this group.