Background: Japan is one of the world’s largest tobacco epidemic countries but few studies have focused on socioeconomic inequalities. We aimed to examine whether socioeconomic inequalities in smoking have reduced in Japan in recent times. Methods: We analyzed data from the Comprehensive Survey of Living Conditions, a large nationally representative survey conducted every 3 years (n ≈ 700,000 per year) in Japan, during 2001–2016. Age-standardized smoking prevalence was computed based on occupational class and educational level. We calculated smoking prevalence difference (PD) and ratio (PR) of (a) manual workers versus upper non-manual workers and (b) low versus high educational level. The slope index of inequality (SII) and relative index inequality (RII) by educational level were used as inequality measures. Results: Overall smoking prevalence (25–64 years) decreased from 56.0% to 38.4% among men and from 17.0% to 13.0% among women during 2001–2016. The PD between manual and upper non-manual workers (25–64 years) increased from 11.9% (95% confidence interval CI, 11.0–12.9%) to 14.6% (95% CI, 13.5–15.6%) during 2001–2016. In 2016, smoking prevalence (25–64 years) for low, middle, and highly educated individuals were 57.8%, 43.9%, and 27.8% for men, and 34.7%, 15.9%, and 5.6% for women, respectively. SII and RII by educational level increased among both sexes. Larger socioeconomic differences in smoking prevalence were observed in younger generations, which suggests that socioeconomic inequalities in smoking evolve in a cohort pattern. Conclusions: Socioeconomic inequalities in smoking widened between 2001 and 2016 in Japan, which indicates that health inequalities will continue to exist in near future.