We calculate the two-photon exchange corrections to electron-proton scattering with nucleon and \(\Delta\) intermediate states. The results show a dependence on the elastic nucleon and nucleon-\(\Delta\)-transition form factors used as input which leads to significant changes compared to previous calculations. We discuss the relevance of these corrections and apply them to the most recent and precise data set and world data from electron-proton scattering. Using this, we show how the form factor extraction from these data is influenced by the subsequent inclusion of physical constraints. The determination of the proton charge radius from scattering data is shown to be dominated by the enforcement of a realistic spectral function. Additionally, the third Zemach moment from the resulting form factors is calculated. The obtained radius and Zemach moment are shown to be consistent with Lamb shift measurements in muonic hydrogen.