It is possible to infer the mass and spin of the remnant black hole from binary black hole mergers by comparing the ringdown gravitational wave signal to results from studies of perturbed Kerr spacetimes. Typically, these studies are based on the fundamental quasinormal mode of the dominantℓ=m=2harmonic. By modeling the ringdown of accurate numerical relativity simulations, we find, in agreement with previous findings, that the fundamental mode alone is insufficient to recover the true underlying mass and spin, unless the analysis is started very late in the ringdown. Including higher overtones associated with thisℓ=m=2harmonic resolves this issue and provides an unbiased estimate of the true remnant parameters. Further, including overtones allows for the modeling of the ringdown signal for all times beyond the peak strain amplitude, indicating that the linear quasinormal regime starts much sooner than previously expected. This result implies that the spacetime is well described as a linearly perturbed black hole with a fixed mass and spin as early as the peak. A model for the ringdown beginning at the peak strain amplitude can exploit the higher signal-to-noise ratio in detectors, reducing uncertainties in the extracted remnant quantities. These results should be taken into consideration when testing the no-hair theorem.