Active surveillance is the preferred management of low risk prostate cancer. Cancer specific anxiety during active surveillance remains under studied. We evaluated long-term anxiety in men on active surveillance to determine whether interventions must be tailored to improve adherence. A total of 413 men enrolled in active surveillance at a single tertiary care center completed quality of life surveys as part of routine care. A modified version of the MAX-PC (Memorial Anxiety Scale for Prostate Cancer) was used to determine cancer specific anxiety. Generalized estimating equations were applied to evaluate the association between anxiety and the duration on surveillance. Additionally, we examined associations between anxiety and patient age, marital status, Gleason score, the number of positive cores, family history and overall health. Median patient age was 61 years, median prostate specific antigen at diagnosis was 4.4 ng/ml and 95% of the patients had Gleason 6 disease. Median time from the initiation of active surveillance to the last survey was 3.7 years. There was a 29% risk of reporting cancer specific anxiety within year 1. Anxiety significantly decreased with time (OR 0.87, 95% CI 0.79–0.95, p = 0.003). Pathological and demographic characteristics were not associated with anxiety after adjusting for time on surveillance. In men undergoing active surveillance we observed a moderate risk of cancer specific anxiety which significantly decreases with time. Those considering conservative management can be informed that, although it is common to experience some anxiety initially, most patients rapidly adjust and report low anxiety levels within 2 years.