Aims. We compute photometric redshifts in the fourth public release of the Canada-France-Hawaii Telescope Legacy Survey. This unique multi-colour catalogue comprises $u^*, g', r', i', z'$ photometry in four deep fields of 1 deg2 each and 35 deg2 distributed over three wide fields. Methods. We used a template-fitting method to compute photometric redshifts calibrated with a large catalogue of 16 983 high-quality spectroscopic redshifts from the VVDS-F02, VVDS-F22, DEEP2, and the zCOSMOS surveys. The method includes correction of systematic offsets, template adaptation, and the use of priors. We also separated stars from galaxies using both size and colour information. Results. Comparing with galaxy spectroscopic redshifts, we find a photometric redshift dispersion, $\sigma_{\Delta z/(1+z_{\rm s})}$, of 0.028–0.30 and an outlier rate, $|\Delta z| \ge 0.15\times (1+z_{\rm s})$, of 3–4% in the deep field at $i'_{\rm AB}$ < 24. In the wide fields, we find a dispersion of 0.037–0.039 and an outlier rate of 3–4% at $i'_{\rm AB}$ < 22.5. Beyond $i'_{\rm AB}$ = 22.5 in the wide fields the number of outliers rises from 5% to 10% at $i'_{\rm AB}$ < 23 and $i'_{\rm AB}$ < 24, respectively. For the wide sample the systematic redshift bias stays below 1% to $i'_{\rm AB}$ < 22.5, whereas we find no significant bias in the deep fields. We investigated the effect of tile-to-tile photometric variations and demonstrated that the accuracy of our photometric redshifts is reduced by at most 21%. Application of our star-galaxy classifier reduced the contamination by stars in our catalogues from 60% to 8% at $i'_{\rm AB}$ < 22.5 in our field with the highest stellar density while keeping a complete galaxy sample. Our CFHTLS T0004 photometric redshifts are distributed to the community. Our release includes 592891 ($i'_{\rm AB}$ < 22.5) and 244701 ($i'_{\rm AB}$ < 24) reliable galaxy photometric redshifts in the wide and deep fields, respectively.