Aims. We trace the evolution and the star formation history of passive red galaxies, using a subset of the VIMOS Public Extragalactic Redshift Survey (VIPERS). The detailed spectral analysis of stellar populations of intermediate-redshift passive red galaxies allows the build up of their stellar content to be followed over the last 8 billion years. Methods. We extracted a sample of passive red galaxies in the redshift range 0.4 <z< 1.0 and stellar mass range 10 < log(M sub(star)/M sub(middot in circle)) < 12 from the VIPERS survey. The sample was selected using an evolving cut in the rest-frame U?V color distribution and additional cuts that ensured high quality. The spectra of passive red galaxies were stacked in narrow bins of stellar mass and redshift. We use the stacked spectra to measure the 4000 A break (D 4000) and the H delta Lick index (H delta sub(A)) with high precision. These spectral features are used as indicators of the star formation history of passive red galaxies. We compare the results with a grid of synthetic spectra to constrain the star formation epochs of these galaxies. We characterize the formation redshift-stellar mass relation for intermediate-redshift passive red galaxies. Results. We find that at z~ 1 stellar populations in low-mass passive red galaxies are younger than in high-mass passive red galaxies, similar to what is observed at the present epoch. Over the full analyzed redshift range 0.4 < z< 1.0 and stellar mass range 10 < log(M sub(star)/M sub(middot in circle)) < 12, the D 4000 index increases with redshift, while H delta sub(A) gets lower. This implies that the stellar populations are getting older with increasing stellar mass. Comparison to the spectra of passive red galaxies in the SDSS survey (z~ 0.2) shows that the shape of the relations of D 4000 and H delta sub(A) with stellar mass has not changed significantly with redshift. Assuming a single burst formation, this implies that high-mass passive red galaxies formed their stars at z sub(form)~ 1.7, while low-mass galaxies formed their main stellar populations more recently, at z sub(form)~ 1. The consistency of these results, which were obtained using two independent estimators of the formation redshift (D 4000 and H delta sub(A)), further strengthens a scenario in which star formation proceeds from higher to lower mass systems as time passes, i.e., what has become known as the downsizing picture.