The abscopal effect could theoretically be potentiated when combined with immunomodulating drugs through increased antigen production. The optimal dosing and schedule of radiotherapy with immunotherapy are unknown, although they are actively investigated in laboratory and clinical models. Clinical data in patients treated for metastatic disease with both modalities may guide future studies. This is a single-institution retrospective review of all patients treated with stereotactic body radiotherapy (SBRT)/stereotactic radiosurgery (SRS) and immunomodulating therapy within 6 months before or after SBRT/SRS for metastatic cancer. Clinical and tumor characteristics were recorded, as well as SBRT/SRS details, immunotherapy details, and survival. Log-rank tests on Kaplan-Meier curves for overall survival (OS) that were calculated from the end of SBRT/SRS were used in univariate analysis and Cox proportional hazards regression for multivariate analysis. A total of 125 patients were identified who met the inclusion criteria; 70 received SBRT, and 57 received SRS. Eighty-three patients were treated for non-small cell lung cancer, 7 patients for small cell lung cancer, and 35 patients for other cancers, with the most common one being melanoma. Fifty-three percent of patients received nivolumab, 29% pembrolizumab, 13% atezolizumab, 5% other. Twenty percent received immunotherapy before SBRT/SRS, 39% during SBRT/SRS, 41% after. Eighty-six patients had died by the time of the analysis; the median OS for the whole cohort was 9.7 months. Patients who had completed immunotherapy prior to SBRT/SRS had worse OS than those who received concurrent therapy or immunotherapy after SBRT/SRS, with a difference in median OS of 3.6 months vs. 13.0 months (p = 0.010) that was retained on multivariate analysis (p = 0.011). There was no significant difference in OS between patients receiving SRS vs. SBRT (p = 0.20), sex (p = 0.53), age >62 years (p = 0.76), or lung primary vs. others (p = 0.73) on univariate or multivariate analysis. When comparing before/concurrent to after/concurrent administration, there is a difference in survival with after/concurrent survival of 8.181 months and before survival of 13.010 months, but this was not significant (p = 0.25). OS appears to be worse in patients who complete immunotherapy prior to SBRT/SRS compared to those receiving it concurrently or after. The design of this retrospective review may be prone to lead time bias, although the difference in median survival is longer than the 6-month window before SBRT/SRS and could only account for part of this difference. Further analysis into causes of death and toxicity and prospective studies are needed to confirm the results of this analysis.