Research is scant regarding differential effects of specific types of recreational drugs use on antiretroviral therapy adherence among women, particularly to single-tablet regimens (STR). This is increasingly important in the context of marijuana legalization. We examined the effects of self-reported substance use on suboptimal (<95%) adherence in the Women’s Interagency HIV Study, 2003–2014. Among 1799 HIV-infected women, the most prevalent substance used was marijuana. In multivariable Poisson GEE regression, substance use overall was significantly associated with suboptimal adherence (adjusted prevalence ratio, aPR = 1.20, 95% CI 1.10–1.32), adjusting for STR use, socio-demographic, behavioral, and clinical factors. Among STR users, compared to no drug use, substance use overall remained detrimental to ART adherence (aPR = 1.61, 95% CI 1.24–2.09); specifically, both marijuana (aPR = 1.48, 95% CI: 1.11–1.97) and other drug use (aPR = 1.87, 95% CI 1.29–2.70) predicted suboptimal adherence. These findings highlight the need to intervene with drug-using women taking antiretroviral therapy to maintain effective adherence.