Purpose The present study aimed to examine the association between adherence to the Japanese diet and the subsequent risk of all-cause and cause-specific mortality using a large-scale cohort from settings all over Japan. Methods We analyzed data from a cohort study of 92,969 Japanese adults aged 45–74 years, covering 11 public health center areas nationwide. We collected dietary information using a validated 147-item food frequency questionnaire. Adherence to the Japanese diet consisting of eight components (high intake of rice, miso soup, seaweeds, pickles, green and yellow vegetables, fish, and green tea; low intake of beef and pork) was assessed using 8-item Japanese Diet Index (JDI8) score, with scores ranging from 0 to 8. The Cox proportional hazards model was used to estimate the hazard ratio (HR) and 95% confidence interval (CI) for all-cause and cause-specific mortality. Results During a median follow-up of 18.9 years, we documented 20,596 deaths. A higher JDI8 score was significantly associated with a lower risk for all-cause and cardiovascular disease (CVD) mortality. The multivariable-adjusted HR of all-cause and CVD mortality for the highest JDI8 score group (score of 6–8) versus the lowest JDI8 score group (score of 0–2) were 0.86 (95% CI 0.81–0.90, P trend < 0.001), and 0.89 (95% CI 0.80–0.99, P trend = 0.007), respectively. Conclusions Adhering to the Japanese diet, as assessed by the JDI8, was associated with a decreased risk of all-cause and CVD mortality among adults living in multiple areas across Japan.