Purpose During the COVID-19 pandemic, teleconsultations have increasingly been used to reduce physical contact and thus risk of infection. This study investigated how patients with cancer experienced the COVID-19 pandemic and how they perceived the change from in-person consultations to telephone consultations in an oncology outpatient clinic. The aim was to provide insights that could optimize the future use of teleconsultations in cancer care. Methods This qualitative study included 15 patients with colorectal, breast, gynecological, lung, or prostate cancer treated at the outpatient clinic at the Department of Clinical Oncology and Palliative Care, Zealand University Hospital, Denmark in June or July 2020. Data were collected through semi-structured individual interviews and analyzed by thematic analysis. Results Patients with cancer experienced social, psychological, and organizational consequences of the COVID-19 pandemic related to their cancer care. Not all patients were comfortable with telephone consultations. Six themes were identified: (1) double burden as a consequence of simultaneous cancer and the COVID-19 pandemic, (2) parameters for patient satisfaction with telephone consultations, (3) the importance of relatives attending consultations, (4) loss of information and nuances during telephone consultations, (5) the impact of physicians’ language and communicative skills during telephone consultations, and (6) patients’ suggestions for future telephone consultations. Conclusion Beyond the COVID-19 pandemic, it is important that hospitals offering teleconsultations involve patients’ preferences, consider for which patients and consultations the solution is suitable, which technology to use, how to prepare patients and relatives, and how to provide physicians with the necessary communicative skills.