Physical activity (PA) reduces the risk for a number of chronic diseases including heart disease, hypertension, hyperlipidemia, and diabetes mellitus type 2. However, most Americans do not meet expert recommendations for exercise, and minorities and low-income persons are the most inactive. Community-based approaches to promoting PA include primary care exercise referral programs. This study examines patient characteristics associated with utilization of a community health center-based exercise referral program. Adult female patients of a community health center with an affiliated fitness center, in Boston, MA, were included in the study if they received a referral to the fitness center from their primary care provider. Demographic and medical information was abstracted from the medical chart, and fitness records were abstracted to measure activation of a fitness center membership (creation of an account denoting at least an initial visit) and utilization over time. Overall, 503 (40 %) of the 1,254 referred women in the study sample activated their membership. Black women were almost 60 % more likely to activate their membership (adjusted OR 1.6, 95 % CI 1.2–2.2), and women with higher comorbidity counts were almost 45 % more likely to activate (adjusted OR 1.4, 95 % CI 1.0–2.0). Once activated, a minority of women participated at levels likely to improve cardiometabolic fitness. Of the 503 activations, 96 (19 %) had no participation, 359 (71 %) had low participation, and only 48 (10 %) had high participation. No independent predictors of participation were identified. These findings suggest that program design may benefit from developing activation, initial participation, and retention strategies that address population-specific barriers.