The start of retirement is an important stage in an (older) adult's life and can affect physical activity (PA) and/or sedentary behaviors, making it an ideal period to implement health interventions. To identify the most optimal timing of such interventions it is important to determine how PA and sedentary behaviors change not only when making the transition to retirement, but also during the first years of retirement. The main study aim was to examine whether PA and sedentary behaviors change differently in retiring adults compared with recently retired adults. A second aim was to examine potential moderating effects of gender and educational level. A longitudinal study was conducted in Ghent, Belgium. Baseline measurements took place in 2012-2013 and follow-up data were collected 2 years later. In total, 446 adults provided complete data at both time points. Of the participants 105 adults were not retired at baseline but retired between baseline and follow-up (i.e. retiring) and 341 were already retired at baseline (i.e. recently retired). All participants completed a questionnaire on PA, sedentary behaviors, socio-demographic factors and physical functioning. Repeated measures MANOVAs were conducted in SPSS 22.0. to analyze the data. Leisure-time cycling increased over time in retiring adults, but decreased in recently retired adults (p < 0.01). (Voluntary) work-related walking and moderate-to-vigorous PA decreased strongly in retiring adults, while slight increases were found in recently retired adults (p < 0.001 and p < 0.01). Passive transport decreased more strongly in recently retired than in retiring adults (p < 0.05), and computer use increased more in retiring adults than in the recently retired group (p < 0.001). Low-educated recently retired adults had the strongest decrease in walking for transport (p < 0.05) and strongest increase in TV viewing time (p < 0.01) and computer use (p < 0.10). For gender, almost no moderating effects were found. Future interventions should focus on PA and/or specific sedentary behaviors in retiring adults, but should definitely include long-term follow-up, as recently retired adults seem to be prone to lapse into an unhealthy lifestyle. Specific attention should be paid to low-educated adults as they are particularly susceptible to a decrease in PA and increased TV viewing time and computer use.