We use data taken as part of Hubble Space Telescope (HST)/Wide Field Camera 3 (WFC3) observations of the Cosmic Assembly Near-infrared Deep Extragalactic Legacy Survey (CANDELS) to identify massive and evolved galaxies at 3 < z < 4.5. This is performed using the strength of the Balmer break feature at rest-frame 3648 A, which is a diagnostic of the age of the stellar population in galaxies. Using the WFC3 H-band-selected catalog for the CANDELS GOODS-S field and deep multi-waveband photometry from optical (HST) to mid-infrared (Spitzer) wavelengths, we identify a population of old and evolved post-starburst galaxies based on the strength of their Balmer breaks (Balmer break galaxies, BBGs). The galaxies are also selected to be bright in rest-frame near-IR wavelengths and hence massive. We identify a total of 16 BBGs. Fitting the spectral energy distribution of the BBGs shows that the candidate galaxies have average estimated ages of ~800 Myr and average stellar masses of ~5 x 10 super(10) M sub(middot in circle), consistent with being old and massive systems. Two of our BBG candidates are also identified by the criteria that are sensitive to star-forming galaxies (Lyman break galaxy selection). We find a number density of ~3.2 x 10 super(-5) Mpc super(-3) for the BBGs, corresponding to a mass density of ~2.0 x 10 super(6) M sub(middot in circle) Mpc super(-3) in the redshift range covering the survey. Given the old age and the passive evolution, it is argued that some of these objects formed the bulk of their mass only a few hundred million years after the big bang.